ConsoleProgressDisplay wrote every progress frame to sys.stdout. Progress
is diagnostic output, so stderr is the correct stream for it — tqdm and
most progress renderers default there for the same reason — and stdout
must stay clean for programs that carry a machine-readable protocol on
it. The stdio MCP servers put newline-delimited JSON-RPC on stdout, where
an interleaved progress bar makes a response body unparseable (#1134).
ConsoleProgressDisplay now takes an optional stream, defaulting to
stderr. The stream is resolved per write rather than captured at
construction, so a later rebinding of sys.stderr (pytest capture, for
instance) is honoured. All writes and the four bare flushes route through
it, and the emoji-capability probe now inspects that stream rather than
stdout, so a cp1252 stderr still degrades correctly.
The existing cp1252 tests in tests/deduplication/test_deduplication.py
patched sys.stdout to assert emoji auto-disabling; they now patch the
stream progress is actually written to. Their intent is unchanged.
Closes#1134 (point 1 only; the SEMANTICA_KG_PATH persistence and README
items remain with @akaszubski)
* fix(dedup): never merge entities with different explicit types (fixes#1137)
The duplicate candidate confidence scoring only rewarded same-type pairs
but never penalized different-type pairs, so a Person 'Alice' and an
Organization 'Acme' (different id, type, and name) passed the confidence
threshold and were merged, silently dropping one entity. Add a type guard:
when both entities carry a non-empty type and they differ, the pair is
never a duplicate candidate (confidence 0, reason 'type_mismatch').
Untyped entities and genuinely duplicate same-type pairs keep their
previous behavior. Regression tests cover all three cases.
* fix(dedup): honor Entity.type and exclude mismatch structurally (review fixes)
Two gaps from code review (#1149):
1. _get_entity_value mapped object 'type' exclusively to .label, which
Entity objects never have — their type lives on .type. The mismatch
guard therefore never saw the type of Entity objects, and differently
typed objects could still merge. Read .type first, fall back to .label.
2. The mismatch branch returned a normal candidate with confidence 0.0,
but detection filters with >= confidence_threshold, and 0.0 is a
documented valid threshold, so mismatches slipped through. Exclude
type_mismatch candidates structurally at both filter sites regardless
of threshold.
Adds tests for Entity objects with different types and for
confidence_threshold=0.0. 94 dedup tests pass.
---------
- bug_001: top_k_per_entity now uses OR semantics — keep a candidate if
EITHER entity is under quota, preventing high-quality candidates being
silently dropped when a popular counterpart saturates its quota
- bug_002: validate max_results and top_k_per_entity at construction;
negative or non-int values raise ValueError instead of silent empty output
- bug_003: validate min_similarity in [0.0, 1.0] at construction;
out-of-range values raise ValueError
- bug_004: harden ConflictDetector method='relationship' normalization —
always produces List[Dict] before calling detect_relationship_conflicts
- quality_001: update detect_duplicates + incremental_detect docstrings to
reflect configurable sort_by field (not hardcoded 'confidence')
- quality_002: add _normalize_entity_id helper (always str) used in both
_apply_result_limits and _build_duplicate_groups for consistent ID handling
Backward compatible: callers not using new params see no behavior change.
58 tests pass (0 failures)
Fixes#534
- New __init__ params: max_results, top_k_per_entity, min_similarity, sort_by
- _apply_result_limits: drop below min_similarity, sort by sort_by field,
enforce top_k_per_entity per entity, cap at max_results globally
- Wired into detect_duplicates() and incremental_detect()
- 30 new tests in TestResultLimiting; full suite 42/42 passed
Closes#531
- Replace 5 direct sys.stdout.write() calls in ConsoleProgressDisplay.update()
with self._safe_write() so emoji/block characters are encoded safely on
Windows cp1252 consoles
- Add TestProgressTrackerEncoding regression tests (3 cases) covering
_safe_write, pipeline header, and auto emoji-disable on cp1252
- Robust ID extraction in CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer
- Support for direct Entity objects and dictionaries as node identifiers
- Improved Entity hashability in utils/types.py
- Added integration test to verify fix and prevent regression
dex0shubham