- llms.md: use showcase models (llama-3.3-70b-versatile, gpt-4o) in
provider examples and use-case tables; clarify defaults vs recommended
in Defaults and Reproducibility section
- split.md: document that chunk_size is in characters with migration note
- ingest.md: add Note that glob patterns are not supported by ingest()
- explorer-setup.md: remove hardcoded "1.5 seconds" timing claim
- cli-setup.md: expand semantica-worker description with concrete usage
- mcp_server.md: clarify turtle/ttl are aliases for the same RDF format
- semantic_extract.md: remove emoji from code comments
Consolidate dual import (module alias + from-import) to a single
`import ... as progress_module` alias and qualify all references.
Replace bare `BaseException` catch with `Exception` in the thread
runner helper.
Co-Authored-By: Zohaib Hassnain <zohaib179949@gmail.com>
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
Deep-copy **options in ingest_examples and batch_public_apis so that
mutable values (e.g. params dicts) are not shared across iterations.
Add rate_limit_delay to the config_only_key strip list in ingest_public_api
so it is not forwarded twice when passed via kwargs.
- Extract _discover_modules() to scan benchmarks/ at runtime so the
--module choices list stays accurate as directories are added or
removed; eliminates the stale context_graph_effectiveness entry and
the missing infrastructure entry from the original implementation
- Add an existence guard before passing the resolved path to pytest so
a valid-looking choice that maps to a missing directory fails fast
with a clear error instead of silently collecting 0 tests and exiting 0
- Print the active module filter to the console so users can confirm
the filtered scope in runner output
CLI section:
- Intro updated to mention startup dashboard and Rich polish
- Data In: added semantica watch examples; removed --watch flag from ingest
(watch is now its own command)
- Developer Tools: new subsection covering init, doctor, changelog, shell,
info with representative examples
What's New in v0.5.0:
- Added Modern CLI Experience subsection listing all 11 improvements:
startup dashboard, grouped help, doctor, init, watch, changelog, shell,
progress bars, elapsed timing, error cards, Windows UTF-8 fix
In TOML, declaring [project.urls] inside the [project] block causes all
subsequent key-value pairs (including dependencies = [...]) to be parsed
as project.urls.* keys, producing:
ValueError: invalid pyproject.toml config: project.urls.dependencies
must be string
Fix: move [project.urls] to after the dependencies array closes and before
[project.optional-dependencies], which is the correct TOML position for a
sub-table of [project].
Elapsed timing
- CLIContext._start records time.perf_counter() at context creation
- _ok() appends elapsed seconds to every success message automatically
Structured error cards
- _show_error_card() renders a red-bordered Rich Panel with title, detail,
and an actionable hint line
- _ERROR_HINTS maps common exception types to fix suggestions
- _run_with_error_handling() now routes all errors through the card renderer
instead of raising plain click.ClickException
Rich progress bars
- kg build: per-source Progress bar (SpinnerColumn + BarColumn +
MofNCompleteColumn + TimeElapsedColumn) when multiple --source flags given;
single-source path keeps the spinner
- ingest: spinner added (was missing entirely); shows filename and recursive flag
semantica changelog
- Hits GitHub releases API via stdlib urllib; compares latest tag against
__version__; renders release notes in a rounded Panel; --json supported
semantica doctor
- Checks: Python version, semantica/rich versions, graph store reachability,
vector store importability, LLM provider env vars, config file, log dir
- Rich table with ✓/⚠/✗ per check; summary error/warning count at bottom
semantica init
- Interactive wizard: graph backend, vector backend, optional LLM key
- Writes ~/.semantica/config.yaml via yaml.dump; --force to overwrite
semantica watch
- Wraps watchdog Observer; matches configurable glob patterns; auto-ingests
on created/modified events; graceful Ctrl+C shutdown
- Guards ImportError with pip install semantica[watch] hint
_HELP_SECTIONS updated to surface init, doctor, changelog, watch
- description: rewritten to lead with the accountability/provenance
angle and name concrete capabilities; drops emoji which render
inconsistently across PyPI clients
- keywords: expanded from 8 to 23 terms covering modern search queries
(ai-agents, llm, graph-rag, decision-intelligence, provenance, etc.)
- classifiers: added Information Analysis, Text Processing::Linguistic,
Database Engines/Servers, Information Technology audience
- [project.urls]: new section with Homepage, Documentation, Repository,
Changelog, Bug Tracker, Discord — shown prominently on the PyPI page
and drive clicks to GitHub/docs
- optional-dependencies: added watch = [watchdog>=3.0.0]; bundled into all
Prevents UnicodeEncodeError on the default cp1252 code page when Rich
renders box-drawing characters and emoji in the startup banner and panels.
Placed before all other imports so Click and Rich capture the already-
reconfigured streams. Uses reconfigure() (Python 3.7+) which modifies the
existing TextIOWrapper in-place rather than replacing sys.stdout.
- _BANNER: ASCII art shown when `semantica` is run with no subcommand
- _show_startup: dashboard panel with Graph Store / Vector Store / Profile
status cards; suppressed under --quiet and --json
- RichGroup: click.Group subclass that renders --help with grouped sections
(Data Ingestion, Intelligence, Knowledge Graph, Analytics, Export & Viz,
Infrastructure, Services, Tools) plus a Quick Start block
- main decorator: cls=RichGroup + invoke_without_command=True to wire both
- `semantica shell`: interactive REPL that dispatches subcommands while
sharing the parent CLIContext; supports readline on Unix for line editing
- Guard parse_cmd spinner with `fmt == "json"` (default format) to prevent
Rich status output from polluting machine-readable stdout in piped usage
- Remove unused `Rule` import from cli.py
- Remove unused `_orig_print` variable in verify_rich_cli.py
- Unify semantica.cli import style in verify_rich_cli.py; use cli_mod.main
The docs validation workflow runs python docs_check.py with no pip
install step, so rich is not available. Wrap the rich import in a
try/except ModuleNotFoundError and fall back to plain print() calls
so the script works in both environments:
- With rich installed: coloured pass/FAIL output
- Without rich (CI): plain text pass/FAIL output, same exit codes
## Summary
Overhaul the CLI and all library modules to produce polished, modern
terminal output comparable to tools like uv, gh, and cargo. Rich was
already a declared dependency but barely used — this commit wires it
throughout every layer.
## Changes by layer
### semantica/cli.py — visual overhaul
- Add imports: `box`, `Panel`, `Rule`, `Syntax`, `Text` from Rich
- Add 7 style constants (`_BRAND`, `_KEY`, `_VAL`, `_DIM`, `_SUCCESS`,
`_WARN_STY`, `_TABLE_BOX`) for a consistent colour palette
- `_ok()` now prefixes output with a green ✓ checkmark
- New `_info()` helper (neutral · bullet, respects --quiet)
- New `_warn()` helper (yellow ⚠ prefix, never suppressed)
- New `_pprint()` helper: renders dicts/lists as syntax-highlighted JSON
(Rich Syntax, monokai theme) instead of raw Python repr; strings
pass through unchanged; respects --quiet
- `info` command: banner replaced with a rounded Rich Panel showing
version + tagline; component table uses SIMPLE_HEAD box
- All 7 table sites updated: `box=SIMPLE_HEAD`, `show_edge=False`,
consistent `_KEY`/`_VAL` column styles (KG Stats, Reasoning Engines,
Recent Decisions, Configured Backends, Backup Info, MCP Tools)
- `_run_build()`: `console.status(spinner="dots")` wraps the blocking
build call; skipped under --quiet / --json
- `parse`, `extract`, `embed generate`, `reason run`, `reason explain`,
`deduplicate`: each wraps its long-running operation in a status
spinner, guarded by --quiet / --json
- All 30+ `console.print(result)` calls replaced with `_pprint()`
- All raw `[yellow]Warning:[/yellow]` and "not running" patterns
replaced with the new `_warn()` / `_WARN_STY` style
### semantica/explorer/__init__.py
- Error messages use `Console(stderr=True)` with `[bold red]Error:[/bold red]`
- Graph loading wrapped in `console.status()` spinner
- Startup info replaced with a cyan-bordered Rich Panel showing URL,
API docs, and health endpoint
### Library internals — replace print() with structured logger calls
All modules below had active `print()` calls that bypassed the logging
framework, corrupted spinners, and polluted stdout in piped/programmatic
use. All replaced with appropriate `self.logger.*` calls:
- `semantica/kg/graph_builder.py` — 23 calls: entity resolution
progress, graph structure steps, GraphStore persistence timing, and
the two `='*60` completion banners → `self.logger.info/debug()`
- `semantica/semantic_extract/methods.py` — 4 verbose-mode debug
prints → `logger.debug()`
- `semantica/semantic_extract/relation_extractor.py` — progress +
error prints → `self.logger.debug/warning()` with `exc_info`
- `semantica/semantic_extract/triplet_extractor.py` — same pattern
- `semantica/semantic_extract/semantic_network_extractor.py` — batch
error prints → `self.logger.warning/error()`
- `semantica/semantic_extract/coreference_resolver.py` — error print
→ `self.logger.error()`
- `semantica/semantic_extract/providers.py` — debug print →
`self.logger.debug()`
### Tooling
- `benchmarks/benchmarks_runner.py`: Rule banner, ✓/✗/⚠ status lines,
Rule separators around regression alert
- `benchmarks/infrastructure/compare.py`: removed manual ANSI escape
codes; comparison output is now a Rich Table with SIMPLE_HEAD;
summary uses coloured Rule + styled SUCCESS/FAILURE messages
- `cookbook/advanced/snowflake_ingestion_examples.py`: `_section()`
helper using Rule; tabular data rendered as Rich Table; result lines
use ✓/✗/⚠ prefixes; logger.error already present, retained
- `docs_check.py`: `pass`/`FAIL` lines use `[bold green]` /
`[bold red]`; summary uses styled output
## Tests
- `tests/test_cli_commands.py`: fix 3 pre-existing mock mismatches
- `test_kg_stats_json_with_mock`: mock now uses `compute_metrics()`
(the method the code actually calls) instead of `get_statistics()`
- `test_dry_run_not_needed_extract_is_read_only` and
`test_stdin_input`: mock now provides `NERExtractor`,
`RelationExtractor`, `TripletExtractor`, `EventDetector`
(the classes the code imports) instead of `SemanticAnalyzer`
Result: 230/230 tests pass (was 227/230)
- `tests/verify_rich_cli.py`: new verification script; exercises all
14 command groups (92 --help checks, table rendering, dry-run
formatting, --json mode, _pprint helper); 111 pass, 0 fail
- Wire --confidence, --model, --temporal flags to extractors via a flat
extractor_config dict (min_confidence, llm_model, include_temporal)
instead of the unused kwargs dict and sectioned to_dict() spread
- Pass confidence_threshold=confidence directly to RelationExtractor
which exposes it as a named parameter alongside **config
- Remove dead SemanticAnalyzer import and unreachable else branch from
extract; unsupported modes now consistently raise ClickException
- Add _serialize_extract_result() to convert dataclass/list results to
plain dicts so JSON and YAML output is machine-readable, not str()
- Fix kg_stats: remove graph={} arg from compute_metrics() so it uses
the analyzer's loaded graph instead of always computing on empty data
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Covers all 22 command groups introduced in issue #568:
global flags, data in, processing, KG, intelligence (reason/decision/temporal),
provenance, validation, ontology, export, visualize, orchestration
(pipeline/store/backup), services (server/explorer/mcp), and shell completion.
Each section shows real invocation examples rather than flag tables.
P1 — runtime-breaking API mismatches:
- decision record/list/query/trace/similar/impact/check: all six decision
commands now call decision_methods / decision_query using a GraphStore
from _get_graph_store(cli_ctx) instead of passing config= kwargs that
don't exist on the underlying API signatures.
- embed index: load vectors from the Parquet/JSON file into List[np.ndarray]
before calling create_index(), which expects vectors not a file path string.
P2 — stub implementations replaced with real logic:
- backup sync: now collects local data sources via _collect_backup_sources
and performs an incremental copy (skips files whose dst mtime >= src mtime).
- backup restore: detects .enc / tar.gz / .tar / directory, decrypts SEM1
format when --enc, extracts tar archives with leading prefix stripped, or
copies directory trees back to cwd.
P3 — correctness bugs:
- backup create: archive now includes actual config/ontology/store data files
via _collect_backup_sources; manifest records the file list.
- extract: --output now works for all formats (table/rdf/yaml), not only JSON.
- backup create: empty keyfile now raises a clear error instead of silently
producing an unencrypted archive.
- normalize: use Path.is_file() instead of Path.exists() to avoid accidentally
reading a directory that matches the input text.
- visualize: without --output, emit to stdout; do not silently write kg.html.
Minor:
- _setup_cli_logging: replace opaque _ = (quiet, json_output, exc) tuple
with del to suppress unused-variable lint.
- reason list: try to source engines from the reasoning module registry;
fall back to the hardcoded list.
- deduplicate --action report: use method="pairwise" to produce individual
pair objects with similarity scores, distinct from --action detect.
- tests: remove mixed import (from semantica.cli import main) — all 192
runner.invoke calls now use cli_module.main as CodeQL flagged.
- tests: add two focused embed-index regression tests that verify vectors
are loaded from the file before create_index is called.
- embed search: embed query text before calling search_vectors (was passing
raw string to query_vector positional arg, causing TypeError on every call)
- ontology version: import OntologyVersionManager not OntologyVersioning
(symbol never existed; command always failed even with package installed)
- ingest --watch: forward watch flag into _ingest() kwargs (was accepted
but silently dropped, so --watch had no effect)
- store migrate: replace fake success stub with honest ClickException pointing
to the export+embed-index workaround (no bulk-dump API exists in vector store layer)
Expands semantica/cli.py from a 2-command stub into a complete terminal
interface covering every capability described in issue #568, and ships
253 tests covering all new commands, flags, and error paths.
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
- Remove incorrect # pragma: no cover from _run_with_error_handling
generic Exception branch (test_runtime_errors_are_click_safe already
covers it via the monkeypatched RuntimeError path)
- Add _require_ctx() guard: converts None ctx.obj into a clean
ClickException instead of an AttributeError (protects standalone_mode=False
/ library-use callers); apply to info, kg_build, build_alias commands
- Rename serve group -> services to avoid collision with the future
`semantica server` flat command specified in issue #568; update docstring
to document planned subcommand layout
- Fix command-level config logging: re-call setup_logging() with the
command-level config logging section when -c is used (setup_logging
clears handlers before adding, so no accumulation risk)
- Fix missing log_level_override in command_ctx: global --log-level was
silently dropped when a per-command -c config was present, breaking
the override chain for any nested _build_runtime_config calls
- Add return-shape docstring on _run_build documenting the expected
build_knowledge_base() return dict structure
- Add type annotation to runner fixture (-> CliRunner) so Pylance
correctly types runner.invoke() -> Result across all test functions
- Expand test suite: 25 -> 32 tests
* test_info_command_shows_framework_components
* test_info_command_shows_config_path_when_supplied
* test_log_level_global_override_stores_in_context
* test_command_config_preserves_global_log_level_override
* test_build_result_with_stats_shows_source_count
* test_build_result_without_stats_shows_generic_success
* test_build_result_none_shows_generic_success
* test_require_ctx_raises_click_exception_on_none
* test_require_ctx_returns_ctx_unchanged
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename logo PNG to semantica-logo.png (lowercase, hyphenated)
- Update docs.json logo (light/dark) and favicon to reference new PNG
- Replace legacy purple favicon with new teal brain neural network icon
- llms.md: replace non-exported Anthropic/Ollama imports with LiteLLM provider-prefix pattern; replace ReasoningEngine with Reasoner; replace create_provider with LiteLLM in YAML config example and tip
- concepts.md: replace ReasoningEngine with Reasoner/ReteEngine/GraphReasoner; fix DatalogReasoner.reason() to evaluate()/query(); replace TemporalKnowledgeGraph with TemporalGraphQuery; replace DistanceCalculator with SimilarityCalculator; replace EntityDeduplicator with DuplicateDetector/EntityMerger
- kg.md: replace non-exported build_knowledge_graph with method_registry.execute()
- semantic_extract.md: replace Anthropic import with LiteLLM
- index.md: replace Anthropic/Ollama imports with LiteLLM
- modules.md: fix TemporalKnowledgeGraph, DistanceCalculator, OntologyManager, ReasoningEngine, DatalogEngine, start_explorer, create_provider across code examples and module index table
- triplet_store.md: replace non-exported NamespacePrefixManager with semantica.ontology.NamespaceManager
Replace plain markdown in every docs/reference/ file and docs/concepts.md with
rich Mintlify JSX components — CardGroup, Steps, Tabs, AccordionGroup, Tip,
Warning, Note, and CodeGroup — for a consistent, navigable, production-grade
developer experience.
Frontmatter, intro, heading, Tabs, and Module Map all said "three-layer"
while the architecture-overview.svg and its alt text showed four layers.
Adds Layer 3 (Intelligence: KG, vector store, ontology, triplet store,
embeddings) and renumbers the former Layer 3 Application to Layer 4.
Diagrams (docs/assets/img/diagrams/):
- architecture-overview.svg: 4-column layered architecture
- pipeline-flow.svg: 8-step numbered pipeline flow
- kg-structure.svg: entity/relation graph with typed nodes and labeled edges
- graphrag-flow.svg: dual-path retrieval (vector + graph) to LLM to grounded answer
- extraction-pipeline.svg: NER/Relation/Coreference fan-out to Triplet Generator
- agent-context-flow.svg: AgentContext hub with VectorStore and ContextGraph
- reasoning-chain.svg: forward-chaining inference with explanation path
Wordmark logo (light + dark SVG variants):
- Green rounded-square S icon + Semantica text in green
- docs.json updated to use wordmark SVGs for light and dark modes
Pages updated with diagrams:
- index.md, architecture.md, quickstart.md, concepts.md
- reference/kg.md, reference/pipeline.md, reference/semantic_extract.md
- reference/context.md, reference/reasoning.md
- installation.md: revert card title back to "Getting Started" to match
the Tip text that already links to it by that name
- getting-started.md: restore pip install semantica[all] code block that
was removed in the original PR; users need the copy-paste snippet even
when the Installation guide is the canonical reference; also standardize
link text to "Installation" (was "Installation guide")
- index.md: add Installation card as first entry in "Start Here" CardGroup
so the prose ("install first, then open Quickstart") is backed by an
actual card to click
- quickstart.md: standardize link text to "Installation" (was "Installation guide")
mint export fails with 'file does not exist' for pages named 'contributing'
and 'license' — these are reserved by Mintlify's GitHub integration layer.
Renamed to contributing-guide.md and project-license.md and updated all
nav entries and cross-links throughout the docs.
Also adds .gitattributes LF rules to prevent CRLF issues from Windows devs.
- Validate docs structure with docs_check.py (Python)
- Validate Mintlify build with mint validate (Node 20 LTS)
- Export static site with mint export, deploy to GitHub Pages
- Deploy job skipped on PRs (validate-only for branches)
- docs.yml: replace mkdocs build/deploy with python docs_check.py;
Mintlify deployment is handled by its own GitHub App
- ci.yml: remove dead paths-ignore refs to deleted mkdocs.yml and
requirements-docs.txt
- OntologyManager: remove red error banner on HTTP 500; always fall back
to empty state silently (error banners reserved for user actions only)
- AlignmentsTab: remove offline-backend warning when both registry and
alignments requests fail; show empty form silently
- ShaclStudio: fix Monarch tokenizer crash — [@] character class prevents
Monaco from misinterpreting @prefix/@base as language-property refs;
wrap beforeMount in try/catch so any Monaco setup failure cannot crash
the React tree
Decision workspace:
- Add AbortController per loadChain() call; abort previous request when a
new decision is selected, preventing stale out-of-order chain responses
- Guard all setState calls with signal.aborted so unmounted component
state updates are skipped; cancel in-flight request on unmount via a
dedicated cleanup effect
SPARQL workspace:
- Guard results table on both result.rows && result.columns to prevent
runtime crash when backend omits columns field
- Use (result.columns ?? []) inside rows.map() to satisfy TypeScript
narrowing inside the closure
- Add .catch() to clipboard.writeText() — silently swallows permission
errors (query remains visible in the editor as fallback)
- Fix CSV export anchor: append to body before click, remove after, to
ensure cross-browser compatibility
Import/Export workspace:
- Fix download anchor: append to document.body before a.click() and
remove afterwards, matching the standard compatible pattern
Lineage workspace:
- Replace 🔗 emoji empty-state icon with lucide-react Link2 for
consistent theming and sizing
Diff & Merge workspace:
- Add "Sample preview" banner above the mock diff table so users know
the displayed fields are illustrative until the backend is connected
OntologyManager:
- Restore non-blocking warning (flash message) when HTTP response is
non-OK and not a 404; network errors (backend down) stay silent
AlignmentsTab:
- When both registry and alignments promises reject, surface a soft
error banner so users know data is missing rather than just empty
Four issues raised in code review:
- Mode.JSON retry now strips response_format from create_kwargs before
calling json_client.chat.completions.create, preventing incompatible
kwargs from being forwarded to a client configured for a different mode.
- Add exc_info=True to the generate_structured fallback warning in the
manual repair loop so the gateway rejection traceback is visible in
production logs, consistent with the other warnings added in this PR.
- Remove the duplicate is_available definition in GroqProvider. Python
silently kept only the second definition; the first (with diagnostic
branching) was dead code and could cause confusion on future edits.
- Validate base_url scheme in OpenAIProvider._init_client. Non-HTTP(S)
schemes (file://, ftp://, javascript:, etc.) are now rejected with a
ValueError at init time, preventing SSRF if base_url originates from
configuration rather than hardcoded values.
Add 3 new tests: SSRF scheme rejection, valid-URL acceptance, and
exc_info presence on the generate_structured fallback warning (20/20 pass).
Update CHANGELOG.md with full description of all fixes under [Unreleased].
Three bugs caused NERExtractor to silently return pattern-based entities
even when method="llm" was configured:
1. exc_info=True missing on method-failure warning in NERExtractor —
the root exception was swallowed, making the gateway error invisible
in logs even with DEBUG enabled.
2. OpenAIProvider.generate_structured always sent response_format=json_object
to the API. Custom/enterprise gateways (Qwen, LLaMA proxies, internal
gateways) often reject this parameter, causing both the instructor path
and the manual repair loop to fail with the same error on every retry.
3. generate_typed manual repair loop had no fallback when generate_structured
itself raised — it retried the same failing call up to max_retries times,
then propagated the error, triggering _extract_fallback (pattern extraction).
Fixes:
- Add exc_info=True to the method-failure warning so the full traceback
appears in logs and users can diagnose the root cause.
- Skip response_format=json_object in OpenAIProvider.generate_structured
when base_url is set (custom endpoint), since standard OpenAI gateways
don't require it and third-party ones reject it.
- In the generate_typed manual repair loop, catch generate_structured
failures and immediately retry via plain generate() + _parse_json,
breaking the retry-the-same-failing-call loop for custom gateways.
Also adds 17 targeted regression tests covering all three bug paths,
including the exact gateway configuration reported in the issue.
- bug_001: top_k_per_entity now uses OR semantics — keep a candidate if
EITHER entity is under quota, preventing high-quality candidates being
silently dropped when a popular counterpart saturates its quota
- bug_002: validate max_results and top_k_per_entity at construction;
negative or non-int values raise ValueError instead of silent empty output
- bug_003: validate min_similarity in [0.0, 1.0] at construction;
out-of-range values raise ValueError
- bug_004: harden ConflictDetector method='relationship' normalization —
always produces List[Dict] before calling detect_relationship_conflicts
- quality_001: update detect_duplicates + incremental_detect docstrings to
reflect configurable sort_by field (not hardcoded 'confidence')
- quality_002: add _normalize_entity_id helper (always str) used in both
_apply_result_limits and _build_duplicate_groups for consistent ID handling
Backward compatible: callers not using new params see no behavior change.
58 tests pass (0 failures)
Fixes#534
- New __init__ params: max_results, top_k_per_entity, min_similarity, sort_by
- _apply_result_limits: drop below min_similarity, sort by sort_by field,
enforce top_k_per_entity per entity, cap at max_results globally
- Wired into detect_duplicates() and incremental_detect()
- 30 new tests in TestResultLimiting; full suite 42/42 passed
Fixes#533
- Removes duplicate `detect_conflicts` definition that was silently overridden,
causing AttributeError for callers passing `method=` or `property_name=` kwargs
- Merges dispatcher logic into the surviving method with `method="all"` default
supporting: "all", "value", "property", "type", "relationship", "temporal",
"logical", "entity"
- Fixes `method="relationship"` incorrectly defaulting `relationships` to the
entities list; now defaults to `[]` with dict normalization
- Removes unreachable dead code block after try/except raise in
`detect_entity_conflicts`
Closes#531
- Replace 5 direct sys.stdout.write() calls in ConsoleProgressDisplay.update()
with self._safe_write() so emoji/block characters are encoded safely on
Windows cp1252 consoles
- Add TestProgressTrackerEncoding regression tests (3 cases) covering
_safe_write, pipeline header, and auto emoji-disable on cp1252
test_retry_logic.py injected sys.modules["openai"] = MagicMock() at module
level so providers.py could be imported without the real openai package.
Those mocks were never restored, leaving openai (and spacy, instructor etc.)
as MagicMock objects for the entire test session. This caused
test_pr482_deepseek_openai tests to receive a MagicMock when importing
openai.OpenAI, making MagicMock(spec=OpenAI) raise InvalidSpecError.
Fix: save original sys.modules entries before injection and restore them
immediately after the semantica imports that needed the mocks complete.
The mock objects remain bound inside the already-imported provider module,
so test_retry_logic tests are unaffected; other test modules now see the
real packages again.
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
subprocess.CompletedProcess[str] as a return annotation is not subscriptable
at runtime on Python 3.8, causing test collection to abort before any tests
run. Adding PEP 563 deferred evaluation makes all annotations strings at
import time, restoring 3.8 compatibility without changing behaviour on 3.9+.
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.
Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.
Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Backend (semantica/explorer/routes/ontology.py):
- suggest-alignments: add TF-IDF character-ngram embeddings via sklearn
(SimilarityCalculator-compatible cosine scoring) so embedding_similarity
is populated in results; combined score = 0.4*label + 0.6*embedding when
available, falling back to label-only when sklearn is absent
- suggest-alignments: add token-overlap prefilter before SequenceMatcher so
zero-Jaccard pairs are skipped without computing full similarity; add
_MAX_ENTITIES_PER_SIDE=500 per-ontology cap on top of the existing
_MAX_ANALYSIS_NODES global cap
- suggest-alignments: remove dead try/except OntologyEngine.create_alignment
block that always failed silently (no TripletStore configured); replace
with a comment explaining the intentional ephemeral-only storage model
- health: replace O(alignments x entities) any() scans for alignment coverage
with O(1) set membership checks via assessed_ids
- shacl/validate: run rdflib.Graph().parse(format='turtle') syntax check on
the submitted Turtle before returning; invalid syntax now raises 422 instead
of returning a misleading unavailable/success response
Frontend:
- AlignmentsTab: add pairwise alignment matrix section that groups recorded
alignments by (source_ontology, target_ontology) pair; each cell shows
color-coded relation badges per RELATION_COLORS; clicking a badge populates
the create/edit form for quick editing; matrix is shown when at least two
ontologies are loaded
- ShaclStudio: add selectedShapeId state and fullShacl ref; each shape row in
the library is now a clickable button that extracts its Turtle block from
the full SHACL and pre-populates the Monaco editor; a "View all" toggle
restores the full SHACL; selected shape ID is shown in the editor header
- GraphWorkspace: fix viewMode race in external focus effect — call
setSelectedNodeId directly instead of going through focusNode(), which
captured a stale viewMode in its closure; remove focusNode from the
dependency array since it is no longer called
Tests (14 passing, was 11):
- Add test_suggest_alignments_returns_embedding_similarity: asserts
embedding_similarity is non-null when sklearn is available
- Add test_shacl_validate_rejects_invalid_turtle_syntax: asserts 422 on
syntactically invalid Turtle
- Add test_health_alignment_coverage_uses_set_lookup: asserts alignment
dimension score is non-zero after recording an alignment, verifying the
O(1) set lookup path works correctly end-to-end
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Backend:
- Replace false conforms=True SHACL stub with status=unavailable always;
live validation cannot be wired until OntologyEngine.validate_graph is
connected to a data graph — a stub that returns conforms=True misleads
users editing shapes
- Cap node/edge fetches in health, suggest-alignments, and SHACL generation
at _MAX_ANALYSIS_NODES (5 000) with a logger.warning when the graph
exceeds the limit; unbounded limit=999_999 fetches cause OOM on large graphs
- Set SHACL health dimension score to 0.0 (was 70.0) when status=unavailable;
exclude unavailable dimensions from the total_score average so they neither
inflate nor deflate the result
- Allow alignments to reference external/unloaded URIs (e.g. schema.org)
without raising 404; label falls back to URI fragment or caller-supplied
source_label/target_label fields added to OntologyAlignmentRequest
- Fix _alignment_id to use uuid.NAMESPACE_OID instead of NAMESPACE_URL;
the composite key is not a URL
- Fix _summarize_shapes to normalise \r\n before splitting on .\n so shape
parsing works correctly on Windows line endings
Frontend:
- Wrap handleSave/handleSuggest/handleRemove/handleAcceptSuggestion in
useCallback in AlignmentsTab for consistency with sibling components
- Add ephemeral-storage banner in AlignmentsTab warning that alignments are
session-memory-only and not persisted across restarts
- Fix exportReport in HealthTab to append/remove anchor from document before
clicking and defer URL.revokeObjectURL to avoid Blob URL leak in some browsers
- Derive health dimension grid column count from health.dimensions.length
instead of the hardcoded repeat(5, ...) that breaks if the backend adds
or removes a dimension
- Add minimal Monarch tokenizer for the Monaco turtle language registration
in ShaclStudio so prefix declarations, IRIs, SHACL properties, comments,
and string literals are syntax-highlighted; previously the editor rendered
as plain text despite theme rules being defined
Tests (11 passing, was 5):
- Rename test_shacl_validate_has_stable_contract to
test_shacl_validate_returns_unavailable and assert status == unavailable
- Add test_shacl_validate_rejects_empty_turtle (expects 422)
- Add test_health_returns_404_for_unknown_ontology
- Add test_health_shacl_dimension_is_zero_when_unavailable with total_score check
- Add test_delete_unknown_alignment_returns_404
- Add test_alignment_upsert_is_idempotent (verifies ID stability and created_at
preservation across updates)
- Add test_alignment_accepts_external_uri (verifies no 404 for schema.org URIs)
- Relax test_alignment_suggestions_are_ranked label assertions to substring
checks so the test survives similarity algorithm changes
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
- Fix domain_uri/range_uri always being truthy strings
- Only create rdfs:domain/rdfs:range edges when domain/range are non-empty strings
- Add proper validation with .strip() to handle whitespace-only values
- Apply fix to both 'data' and 'text' mode ontology creation
- Prevents pollution of graph with invalid edges to namespace root
Fixes issue where empty domain/range values like '' or None would still create
edges pointing to namespace root (e.g., 'https://ex/#/') instead of being
properly omitted.
The pattern `<[^>]+>\s+<[^>]+>` in _detect_format() was flagged by CodeQL
(py/polynomial-redos, CWE-1333/730/400) as a polynomial regular expression
on uncontrolled user data.
The `<...>` branch was already unreachable — strings starting with '<' return
'xml' two lines above — but CodeQL does not track that control flow path.
Fix: replace the entire re.match() call with plain startswith / 'in' checks:
- N-Triples with URI subjects are already handled by the XML branch.
- Only blank-node-subject N-Triples (_:word <uri> ...) need detection here,
which is correctly expressed as startswith('_:') and ' <' in stripped.
- Removed the now-unused `import re`.
Closes security advisory #23.
Bug 4 — Upload format misdetected:
- Added xml→'xml' and json→'json-ld' to the extension→format map so
.xml and .json files are no longer misidentified as turtle.
- Changed the fallback from '|| "turtle"' to '?? ""' (empty string for
unknown extensions) so the backend _detect_format() runs instead of
blindly assuming turtle for any unrecognised extension.
- Omit the format key entirely from the load request body when no format
was detected, letting the backend auto-detect from content heuristics.
- Added .n3 to the file picker accept list and dropzone hint text.
Bug 1 — Broken registry filters:
fetchRegistry no longer sends format/kind values (owl/skos/internal/external)
as the status query param; those filters are applied client-side via
filteredEntries which already had the correct logic. Only the text search
param q is delegated to the backend.
Bug 2 — Toggle/refresh URI corruption:
Removed removesuffix('/toggle') and removesuffix('/refresh') from
toggle_ontology and refresh_ontology. Starlette's route regex already
strips the literal suffix from the captured path param; the removesuffix
call was a no-op for normal URIs but corrupted any ontology URI that
legitimately ends with /toggle or /refresh.
Bug 3 — SSRF in URL fetch:
Added _validate_fetch_url() which rejects non-http/https schemes and
resolves the hostname to block private, loopback, link-local, reserved,
and multicast addresses before requests.get() is called. Applied to all
three fetch sites: preview, load, and refresh.
Bug 5 — Inconsistent XML hardening:
_parse_rdf_sync now calls _safe_parse_rdf() from
semantica/explorer/utils/rdf_parser.py instead of g.parse() directly,
applying the existing defusedxml-based XXE protection for RDF/XML inputs.
Bug 6 — Search scans whole graph:
search_entities now calls session.search(q, limit*6) which hits the
GraphSearchIndex instead of fetching up to 999,999 nodes and doing a
linear Python substring scan. Results are post-filtered by _SEARCHABLE_TYPES
and entity_type before being returned up to the requested limit.
- Replace invalid inset-left with inset: 0 0 0 72px on ::before at <=680px
- Add matching mobile inset fix to ::after (was still at 88px)
- Merge duplicate .landing-capability-band CSS rule blocks into one
- Fix non-standard font-weight: 850 -> 800 on .landing-launcher-item-title
- Remove unused eyebrow field from LandingAction type and all data entries
- Extract static 42-dot SVG preview array to module-level PREVIEW_DOTS constant
Co-Authored-By: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
- Align _coerce_embedding_vector inner dict-probe key list with
_extract_node_embeddings outer key list (add 'embeddings', reorder to
generic-first) so nested embedding dicts resolve consistently.
- Add TODO comment on _extract_node_embeddings to cache per-session
graph revision and avoid O(N) re-scan on every semantic request.
- Add deprecation docstrings to legacy path-segment routes
(/node/{id}/path and /node/{id}/semantic-neighborhood) documenting
the known slash-in-ID limitation and pointing to the query-param
alternatives.
- Extract _FakeSimilarity to module level so it is shared without
duplication across test classes.
- Rewrite test_legacy_semantic_neighborhood_still_works_for_simple_ids
as a fully isolated TestClient session instead of mutating the
shared module-scoped 'client' fixture, preventing cross-test
state pollution.
- Extract _make_slash_node_session helper to reduce boilerplate in the
slash-safe route tests.
Co-Authored-By: ZohaibHassan16 <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <98801504+KaifAhmad1@users.noreply.github.com>
- Fix dead `if (anchorNodeId)` conditional in buildHeatmapRenderSnapshot
(anchor is always truthy past the early-return guard on line 263)
- Replace O(n) array .includes() with WeakMap-cached Set.has() in
resolveDistanceNodeStyle heatmap path — prevents per-node O(n) scan
during every Sigma reducer pass on large graphs
- Rename GraphDistanceBucketCounts.threeHop → threeHopPlus across
types.ts, graphSceneState.ts, and GraphWorkspace.tsx so the field
name reflects that it accumulates distance ≥ 3, not exactly 3;
update status-strip labels to "3+ hop" accordingly
- Restore hasMetrics guard in PathDistanceIntelPanel to suppress the
empty metric grid <div> when a path result carries no optional metrics
Co-Authored-By: ZohaibHassan16 <zohaib@hawksight.ai>
Co-Authored-By: KaifAhmad1 <mohammadk78600@gmail.com>
- Extract ENTITY_SHAPE_ALIASES and classifyEntityShape into a shared
graphEntityShape.ts utility — resolveEntityShape was duplicated with
divergent signatures in useLoadGraph.ts and graphSceneState.ts; both
now import from one place so aliases can never drift
- graphSceneState.resolveEntityShape falls back to classifyEntityShape
for nodes created programmatically that bypass useLoadGraph
- Fix graphTheme.ts indentation around fullGraphStructure,
fullGraphStructureLayer, and interaction — closing braces were at
wrong indent levels making the nesting visually misleading
- Add comment on fullGraphStructureLayer.mode explaining it is
intentionally "off" as a staged-rollout gate (flip to "auto" to enable
cross-community canvas curve rendering)
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Merge origin/main (Distance Intelligence #502) into feat/explorer-visual-refresh.
Conflict was in the viewModeItems useMemo: the PR's new cluster-based toolbar
structure diverged from main's coreToolbarGroups additions.
Resolution:
- Keep PR's viewModeItems as a clean 3-item segmented control (Full/Grouped/Focused)
- Port Distance Intelligence controls (ego mode, heatmap, structural/semantic overlay)
into a new distanceToolbarItems useMemo that slots into the cluster toolbar as a
"Distance" cluster, visible only when a node is selected
- Wire distanceToolbarItems into toolbarClusters between "local-structure" and
"analysis" clusters
- All other Distance Intelligence additions (state vars, BFS helpers, useEffects,
ego depth slider, GraphInspectorPanel onFocusNode prop) merged cleanly
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
KaifAhmad1