Promotes the Unreleased changelog section (Databricks connector, SQLite
vector store, SPARQL CONSTRUCT templates, JenaStore named-graph support)
to 0.6.0 and syncs version references across pyproject.toml, __init__.py,
and docs.
- TemporalGraphQuery.query_time_range() and RDFExporter.export() both
expect {entities/relationships} (or {relationships} with source_id/
target_id keys), not ContextGraph.to_dict()'s {nodes, edges} shape.
Map the output before passing it in, and add an actual temporally-
bounded edge to the Temporal Intelligence example so the query has
something to find.
- add_causal_relationship() only accepts relationship_type values of
CAUSED, INFLUENCED, or PRECEDENT_FOR; replace the invented "triggers"/
"enables" values used in Decision Intelligence and the audit-trail
recipe, which would otherwise raise ValueError immediately.
PipelineBuilder.add_step() returns the created PipelineStep, not the
builder, so chaining .add_step().add_step() raised AttributeError.
Only connect_steps() and set_parallelism() return the builder and can
be chained.
Consolidates the platform reference into a single, premium README with
collapsible module/recipe sections so the docs and the deep-dive reference
no longer live in two places. Every code example was checked against the
actual semantica/ source and corrected where the API had drifted:
resolve_conflicts, register_source, ValidationResult.valid, clean_data,
execute_pipeline, ParquetExporter/LPGExporter/ReportGenerator calls,
graph.to_dict(), TemporalGraphQuery/TemporalNormalizer usage, the
Reasoner/ExplanationGenerator API, and the REST endpoint paths. Also
removed duplicated titles, snippets, and repeated example scenarios that
had crept in during the merge.
The CodeQL Analyze Python job failed on the #757 merge commit with
ECONNRESET while streaming the CodeQL bundle download in
codeql-action/init's "Setup CodeQL tools" step. This is unrelated to
the merged code — it's a known, currently-unaddressed gap in
codeql-action: the download error is retryable but the action doesn't
retry it internally (confirmed via codeql-action's issue tracker and
changelog).
Since a `uses:` step can't be wrapped by a shell-level retry action,
Initialize CodeQL now runs up to 3 times, cascading to the next
attempt only if the previous one failed, so the common case (success
on attempt 1) costs nothing extra.
Dataset.remove() on a bare 3-tuple resolves context=None internally,
which the underlying store treats as a wildcard and deletes the
matching triple from every graph, not just the default graph the
docstring promises. Pass self.graph.default_graph explicitly as the
context so delete_triplet stays scoped to the default graph, matching
the isolation guarantee default_union=False is meant to provide.
Also corrects a misleading comment: SPARQLStore is graph_aware=True
too, so graph-awareness isn't what requires SPARQLUpdateStore here —
it's SPARQLStore being read-only (.add()/.remove() raise TypeError).
Adds regression tests and a CHANGELOG entry for PR #757.
Updates the tagline, adds Ontology Management/SKOS to the feature pills, swaps
the yellow-highlight subhead for a cleaner italic style, and surfaces a
regulated-domains teaser linking to the existing "Built for High-Stakes
Domains" section.
Trims the README from a full module/API dump into a scannable pitch (hero,
why-Semantica, quick start, architecture, decision intelligence, one flagship
audit-trail recipe) and moves the exhaustive per-module reference, extra
recipes, and full integrations matrix into a new PLATFORM_REFERENCE.md.
1. Endpoint derivation regression: Detect if self.endpoint already contains
a Fuseki service suffix (/query, /update, /sparql) to prevent double-appending
(e.g., /ds/query/query). If it does, derive the base and construct both
paths properly.
2. Misleading serialize warning: Limit the named-graph data loss warning
to single-graph serializer formats (turtle, xml, n3, etc.). Multi-graph
formats (trig, nquads, nt) will correctly serialize all graphs without warning.
3. Zero-added error misdiagnosis: Track malformed triples accurately in
add_triplets(). If every triplet fails the local validation (ValueError/
AttributeError), raise a formatting-oriented ProcessingError instead of
assuming a store connectivity issue.
Includes comprehensive regression tests for all three cases.
Migrates JenaStore from rdflib.Graph to rdflib.Dataset with default_union=False
explicitly set, per maintainer-confirmed architecture for issue #756.
Changes:
- _initialize_graph: construct self.graph as Dataset(default_union=False) for
the in-memory path, and Dataset(store=SPARQLUpdateStore(...), default_union=False)
for the remote path. SPARQLUpdateStore.graph_aware=True satisfies Dataset's
hard requirement. Both paths verified against rdflib source.
- add_triplets: accept and honor graph= option. When supplied, Dataset.graph(uri)
creates/retrieves the named-graph context and the triple is written via a
4-tuple (which SPARQLUpdateStore maps to INSERT DATA { GRAPH <uri> { ... } }).
When graph= is omitted, the 3-tuple path routes to Dataset's default graph,
preserving pre-migration semantics exactly.
- serialize: add WARNING log when named-graph content would be silently dropped
by a single-graph serializer (turtle/xml/n3). Log includes triple count and
recommends trig/nquads formats. No warning when only the default graph is used.
- create_model: document that triplet_count now counts triples across all graphs
(default + named) as a consequence of this migration. Semantics shift made
visible, not silent.
- delete_triplet: document that graph= parity is a known gap, deferred to a
future follow-up per maintainer's stated scope (add_triplets only).
Decisions applied:
1. triplet_count semantics shift: documented in create_model docstring
2. delete_triplet graph= parity: explicitly out of scope, noted in docstring
3. Existing store.graph=Graph() tests: left unchanged; new tests added
to cover the real _initialize_graph path
Tests added (TestJenaStoreDatasetMigration):
- test_initialize_graph_produces_dataset_not_graph
- test_initialize_graph_dataset_has_default_union_false
- test_add_triplets_with_graph_option_writes_to_named_graph
- test_add_triplets_without_graph_option_writes_to_default_graph
- test_add_triplets_named_graph_isolated_from_default_query
- test_serialize_logs_warning_when_named_graph_content_present
- test_serialize_no_warning_when_only_default_graph_used
Also updated test_add_triplets_remote_endpoint_fires_insert_data_via_update_store
to patch Dataset instead of Graph (the remote path now creates Dataset(store=...)).
Full suite: 269 passed, 0 failed (tests/triplet_store/ + tests/pipeline/)
The exception-propagation comment and Raises docstring in
execute_construct_template stated that add_triplets signals failure
exclusively via a returned dict. This became stale after the JenaStore fix
(previous commit) which introduced ProcessingError propagation for complete
batch failures.
Updated to document both paths:
- dict-based failure: success=False in returned dict (BlazegraphStore, RDF4J, etc.)
- raised ProcessingError: JenaStore full-batch failure now raises directly
No logic changed. 262 tests pass.
The remote-endpoint path in _initialize_graph was instantiating the read-only
rdflib SPARQLStore, causing every add_triplets() call against a remote Fuseki
endpoint to silently fail: SPARQLStore.add() raises TypeError which was swallowed
by the broad except Exception per-triplet handler and returned as success=True/added=0.
Changes:
- Import SPARQLUpdateStore alongside SPARQLStore
- _initialize_graph: use SPARQLUpdateStore(query_endpoint=<base>/query,
update_endpoint=<base>/update) per standard Fuseki REST API conventions
- Fix constructor: self.endpoint=config.get('endpoint') always returned None
because the named positional 'endpoint' param captures the kwarg before **config;
now uses endpoint or config.get('endpoint')
- Narrow per-triplet except to (ValueError, AttributeError); add ProcessingError
when entire batch fails to prevent misleading success=True/added=0 return
Tests added (TestJenaStoreRemoteEndpointUsesUpdateStore): 4 new test cases
Full suite: 262 passed (tests/triplet_store/ + tests/pipeline/)
_format_object_for_ntriples decided IRI vs. literal purely from the
presence of datatype/lang metadata, defaulting anything without it to
<obj>. Any plain literal object (e.g. typical NER/extraction output
like "Alice", or an untyped Turtle literal round-tripped through the
new CONSTRUCT path) was wrapped as an invalid IRI instead of a quoted
literal, diverging from BlazegraphStore's _is_uri_value-first check.
Port _is_uri_value from BlazegraphStore so RDF4JStore checks whether
the object is actually URI-shaped before falling back to literal
handling, with a plain-quoted-literal fallback instead of <obj>.
Extends CONSTRUCT support to JenaStore, which uses rdflib.Graph natively rather
than an HTTP protocol - CONSTRUCT results come as native 3-tuples with no
Accept-header/parsing dance needed, unlike Blazegraph/RDF4J.
- CONSTRUCT-aware execute_sparql: reuses shared sparql_escaping.CONSTRUCT_QUERY_RE,
extracts datatype/language from rdflib Literal objects into the same 4-tuple
metadata contract used by Blazegraph/RDF4J
- Non-CONSTRUCT path (SELECT/ASK) confirmed byte-for-byte unchanged (Property 9)
- execute_construct_template confirmed backend-agnostic against JenaStore, zero
changes needed
- Named-graph support explicitly out of scope - JenaStore wraps a single
rdflib.Graph with no named-graph concept; add_triplets continues to silently
ignore graph= exactly as before. Tracked separately as a follow-up issue
requiring a Graph -> ConjunctiveGraph/Dataset migration.
Extends the Blazegraph-only CONSTRUCT support from #322 (commit 4f2c6c82's
approved pattern) to RDF4JStore:
- CONSTRUCT-aware execute_sparql: Accept: text/turtle, rdflib Turtle parsing,
4-tuple (s, p, o, metadata) contract with datatype/language preservation
- Named-graph writes via RDF4J's REST context parameter, N-Triples-encoded
(angle-bracket-wrapped IRI), confirmed against RDF4J's Protocol.java source
- graph=None preserves existing behavior exactly (no context param sent,
not context=null - verified as a distinct, deliberate choice)
- _CONSTRUCT_QUERY_RE moved to sparql_escaping.py as a shared, backend-agnostic
constant; Blazegraph now delegates to it, zero behavioral change confirmed
- execute_construct_template (construct_templates.py) required zero changes -
confirmed backend-agnostic via end-to-end integration tests against RDF4JStore
29 new tests, full suite 245/245 passing. Jena support remains out of scope
for this PR - tracked separately in #754's remaining scope.
Replace remaining Hawksight-AI GitHub org links and the old
semantica-dev noreply email with the current semantica-agi org
and kaif@getsemantica.ai contact, so security/support contacts
match pyproject.toml.
Implements #322: ConstructTemplate/ParameterDescriptor/ConstructTemplateRegistry
with injection-safe {{param}} rendering, Blazegraph CONSTRUCT-aware execute_sparql
extension, execute_construct_template (render->execute->parse->persist), and a
construct_template pipeline step. RDF4J/Jena support deferred to a follow-up issue.
Closes#322
Documents the 9 pre-existing test failures caused by never-implemented
kg.ProvenanceTracker compatibility methods, the deprecation fix, and
the follow-up migration guide addition in this PR.
Every deprecation warning added in this PR (and the class docstring)
points to docs/migration/kg-provenance-tracker.md, but the file was
never added, so the reference was dead. Adds the guide with a
method-mapping table to semantica.provenance.ProvenanceManager.
Add graph-apache-age extra (psycopg2-binary) which was previously
undeclared despite age_store.py depending on it, wire it into
graph-all, and document install commands for FalkorDB/AGE/Neptune
alongside Neo4j. Note that RDF triple stores need no extra since they
talk SPARQL over HTTP via the core `requests` dependency. Also align
README's "Triplet Stores" table label to "Triple Stores (RDF)" to
match the standard term used elsewhere in the docs, while keeping the
TripletStore interface name in backticks.
Semantica already ships both an RDF triplet-store stack (Blazegraph,
Apache Jena, Eclipse RDF4J via a unified TripletStore/SPARQL interface)
and an LPG graph-store stack (Neo4j, FalkorDB, Apache AGE, AWS Neptune
via Cypher), but the README only surfaced the LPG side. Add a hero
highlight line, a "What Semantica gives you" bullet, and split the
Features-at-a-Glance table row so both formats and all backends are
named explicitly.
- get_table_lineage() gains include_column_lineage=True, resolving
per-column upstream/downstream references via Unity Catalog's
column-lineage API (one request per column, opt-in)
- DatabricksConnector.connect() now reuses an already-open connection
instead of opening a second one; ingest_table()/ingest_query() only
close the connection they opened themselves, so using the ingestor
as a context manager no longer leaks the connection opened by
__enter__
- get_table_schema()/get_table_lineage()/list_tables() now validate
both catalog and schema are resolved before calling Unity Catalog,
matching list_tables()'s existing catalog check
- 8 new regression tests (35 total)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
Review follow-up: only append archived_history_id to used_entities when
explicit_parent_supplied is True. Previously it was appended unconditionally,
so the no-explicit-parent re-track path ended up with the same history id in
both parent_entity_id and used_entities, duplicating the reference in
get_lineage() output.
- get_lineage() aggregated metadata by iterating trace_lineage()'s BFS
order and calling dict.update() on each entry, so ancestor metadata
(now reachable via derived_from chains) could overwrite the queried
entity's own metadata on conflicting keys. Reverse the iteration so
the queried entity (always lineage_entries[0]) is applied last and
wins, matching the documented "most recent entry's metadata takes
precedence" intent.
- track_entity()'s derived_from guard only accepted a concrete dict,
silently ignoring other collections.abc.Mapping implementations
(e.g. types.MappingProxyType). Switch the isinstance check to
Mapping so any mapping-like metadata is honored.
Addresses Qodo review findings on PR #741.
track_entity() only auto-linked a parent by looking up `source` as an
existing entity_id, so two entities sharing a real source URL (e.g. a
document and a decision derived from it) never got connected, and
metadata["derived_from"] was stored but never consulted by any linking
or traversal code.
track_entity() now treats metadata["derived_from"] as an explicit
parent link (unless parent_entity_id was already passed directly), so
the existing BFS in trace_lineage() picks it up for free.
Closes#735
Rule is a mutable dataclass, so an already-registered rule's priority
could change after being added; the dedup early-return skipped the
priority re-sort, so re-adding a rule after mutating its priority
left self.rules stale relative to that change. The duplicate branch
now re-sorts before returning, matching the append path.
- add_rule()'s duplicate-skip path now logs at warning level instead
of debug, so a skipped duplicate is visible by default rather than
silent in typical logging configs
- The duplicate-rule log message now stringifies conditions via
map(str, ...) before joining, since Rule.conditions is List[Any]
and non-string entries would otherwise raise TypeError
add_rule() unconditionally appended to self.rules, so re-running the
same setup code on an existing Reasoner instance (e.g. re-executing a
Jupyter cell) duplicated every rule; forward_chain() would then match
the duplicated rules but silently return no new results since the
conclusions were already in self.facts, with no error or warning.
add_rule() now compares an incoming rule's rule_type, conditions, and
conclusion against existing rules and returns the existing Rule
instead of appending a duplicate, keeping repeated add_rule() calls
with the same definition idempotent.
* Add shacl extra to pyproject.toml (fixes#736)
* docs(changelog): add entry for shacl extra fix (#736)
by @Sameer6305
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
'By default, initializing ... with storage_path=...' read as if passing
storage_path were the default, contradicting the very next sentence
about the no-argument in-memory default. Rephrased so the in-memory
default isn't undercut by the first sentence.
* docs: improve SHACL validation guide onboarding and workflow guidance
* docs: fix SHACL validation implementation mismatches
* docs: fix stale violation URIs and drop unused imports in SHACL guide
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
Condense the CLI section to the essential install/usage snippet and
command groups, pointing to docs.getsemantica.ai for the full
reference instead of maintaining static terminal mockups in the README.
Consolidate the hero around a single category claim (Context and
Accountability Layer for AI agents), drop the named-competitor
comparison table (LangChain/LlamaIndex/Mem0/Zep/Palantir Foundry),
remove GitHub alert-box tips/notes, cut redundant module/changelog
sections, strip vanity feature counts, and trim em dashes for a
cleaner, more premium read.
- Add SQLITE_VEC_AVAILABLE flag via importlib.util.find_spec so the test
suite's skipif actually reflects whether sqlite-vec is installed; it was
previously undefined, causing all sqlite vector store tests to be
silently skipped regardless of installation state.
- Actually apply PRAGMA synchronous=NORMAL alongside journal_mode=WAL when
use_wal=True, matching the documented behavior; document use_wal as an
opt-in kwarg in the docstring and usage guide.
- Correct _is_safe_identifier error messages (regex never allowed hyphens).
- Batch get() and update() with IN(...)/executemany instead of per-id
round trips, consistent with add()/delete().
- Fix flaky test_update_vectors assertion that relied on list.index()
over dicts containing numpy arrays.
- Reorder sqlite_vec_store import alphabetically in vector_store/__init__.py.
Co-Authored-By: Luffy2208 <209925020+Luffy2208@users.noreply.github.com>
Revert badge rows to flat-square (for-the-badge rendered as
mismatched oversized blocks), convert the subtitle to a native
blockquote for GitHub's built-in muted-grey text styling, and
trim "The" from the tagline.
Update the hero tagline, subtitle, and comparison table to frame
Semantica as Palantir-grade knowledge/decision intelligence that is
open source, self-hostable, and priced for startups through
Fortune 500, not just enterprise budgets.
* docs: improve conflict resolution guide onboarding and workflow guidance
* docs: fix conflict resolution implementation mismatches
* docs: correct credibility-weighted example output values
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
The merging example told readers to use merge_entity_group() for
already-confirmed duplicate groups, but the code right below it still
called merge_duplicates() on group.entities, which re-runs duplicate
detection redundantly. Update the call to match the stated guidance.
- Correct the unsupported-rule-key pitfall: absent keys fail compliance,
present keys (any value) pass — the previous wording had this backwards.
- Remove required_ltv/pd/lgd/dsti/credit_score: True from the mortgage
example. required_* checks equality against the given value, so True
against a real numeric field silently marks compliant decisions as
non-compliant (verified: a fully passing decision still returned False).
The min_/max_ rules already enforce presence of ltv/dsti/credit_score.
* docs: improve export guide onboarding and workflow guidance
* docs: fix export guide implementation mismatches
* docs: revert .content to .text in export guide examples
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
* docs: improve agent memory guide onboarding and usage guidance
* docs: align Agent Memory guide with persistence implementation
* docs: fix misleading index_path persistence claim across guides
VectorStore's index_path kwarg is silently absorbed into FAISSStore's
**config and never read anywhere in faiss_store.py, so it does not make
the FAISS index persist across restarts as several docs implied. Real
persistence requires an explicit VectorStore.save()/.load() call, or
AgentContext.save()/.load() which cascades to it.
- docs/reference/context.md: rewrite the "Persist your vector store"
tip to explain the actual save()/load() mechanism instead of the
dead index_path kwarg.
- docs/guides/graphrag.md, decision-intelligence.md, ingest.md,
semantic-extraction.md: drop the dead index_path=... kwarg from
VectorStore(backend="faiss", ...) constructor calls.
Follow-up to #691, which fixed the same false claim in
docs/guides/agent-memory.md but missed these other files.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* feat: implement Apache Arrow and Feather file ingestion support (#235)
* fix(arrow): eliminate double full-scan and clean up reader wrapper
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
* docs(changelog): add [Unreleased] entries for Arrow ingestion (#705)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
- Remove duplicate Data Quality Info block; content moved into Common Pitfalls as a dedicated pitfall entry, keeping the critical advanced_analytics=True warning as the sole callout
- Align node count threshold: Common Pitfalls now consistently references 100+ nodes (was '< 50 nodes'), matching the When To Use recommendation
- Fix CVE in SUNBURST example: CVE-2024-3400 → CVE-2020-10148, matching context-graphs.md
- Correct load_from_graph fact format: predicates/args are lowercased (threatactor(apt29), not ThreatActor(APT29)); scoped to DatalogReasoner only; removed incorrect metadata-to-predicate claim
- Move Common Pitfalls section after </Tabs> so it renders outside the tab component and is visible to all readers
- Replace non-existent shortest_path() with get_neighbors() + path_to_anchor
- Remove non-existent extract_subgraph() calls from all domain tab examples
- Clarify automated extraction requires knowledge_graph= constructor arg and list input
- Distinguish save_to_file() (graph only) from AgentContext.save() (graph + FAISS + memory)
- Add resolve_links() step to serialization section for cross-graph link restoration
- Link duplicate entities pitfall to the deduplication guide and its API
- Replace ctx.store() + graph.to_dict() with direct entity/relationship
dict to avoid key mismatch (to_dict() returns nodes/edges; generator
reads entities/relationships)
- Fix prop type filter: 'datatype' → 'data' (value set by PropertyGenerator)
- Fix domain/range printing: both are stored as lists, not scalars
- Clarify Reasoning bullet: OWL inference requires an external reasoner,
Semantica only exports the ontology
- Remove duplicate LLM-vs-graph-generator pitfall already covered by the
Info callout in the LLMOntologyGenerator section
* docs: improve pipeline guide onboarding and workflows
* fix(docs): correct broken pipeline guide examples from review
- Remove Option 1 (register_step_handler + string name): ExecutionEngine
never resolves string handler names via step_registry, so it raised
TypeError at runtime; replace with the single working pattern
- Add missing step_type positional arg to all new add_step() calls
- Use connect_steps() for checkpoint dependency instead of the
dependencies= kwarg, consistent with every other example in the file
- Move extract_entities definition above its call site to fix NameError
- Replace docstring on save_checkpoint with inline comment to match
the no-docstring convention used by all other handlers in the file
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Expand intro to cover Git (dict/code_files) and stream (StreamMessage/.content) return shapes, which the previous two-class split omitted
- Add missing imports and AgentContext setup to the Source 1 internal-docs snippet (NameError on copy-paste)
- Add advanced_analytics=True to ContextGraph in both Business Examples (required for extract_entities=True to populate graph analytics)
- Replace bare `pass` credential with YOUR_DB_PASSWORD placeholder to match the YOUR_*_KEY convention used elsewhere
- Guard nullable description/resolution columns in ticket_texts with `(r[...] or '')` to prevent TypeError on NULL rows
- Replace misleading time.sleep() rate-limit advice with accurate description of RESTIngestor's built-in 429 retry/backoff and how to tune it
Without an explicit namespace in metadata, checkov (CKV_K8S_21) flags
every resource as using the default namespace. Using .Release.Namespace
lets helm install --namespace semantica --create-namespace correctly
scope all resources to the target namespace.
An empty/comment-only YAML file is parsed as NoneType by PyYAML.
Checkov requires a dict; adding skip-check: [] satisfies the parser
without globally suppressing any checks.
- gcp/cloudrun-service.yaml: add comment + README sed one-liner so PROJECT_ID
is substituted before gcloud run services replace (was a literal placeholder
that caused image-pull failure on the declarative deploy path)
- azure/main.parameters.json: replace wildcard allowedOrigins "*" with a
REPLACE_ME placeholder; add README note to set the real URL after first deploy
- kubernetes/networkpolicy.yaml + helm networkpolicy template: add from: selector
(ingress-nginx namespace + same-namespace pods) so ingress is no longer
allow-all; restrict egress to FalkorDB port 6379 and DNS port 53 instead of
the allow-all egress: - {} wildcard
- helm/values.yaml: expose networkPolicy.ingressNamespace and falkordbPort values
- kubernetes/deployment.yaml: add secretRef for knowledge-explorer-secrets so
FALKORDB_PASSWORD is actually injected into the container
- app.py: add _mutation_bridge_installed guard to prevent closure stacking when
the same GraphSession is passed to create_app() more than once; remove
duplicate app.state.allowed_origins assignment (single source of truth is
app.state.explorer_settings); add comment on falkordb_host/port dead config
- tests: update allowed_origins assertions to use explorer_settings dict
- .checkov.yaml: remove global CKV_K8S_21/28/30 suppressions; rely on per-file
inline checkov:skip comments in cloudrun-service.yaml so future real K8s
manifests are not silently exempted
checkov scans deploy/gcp/cloudrun-service.yaml as a Kubernetes resource
because it has apiVersion: serving.knative.dev/v1. It flags CKV_K8S_21
('default namespace should not be used') because Cloud Run services have
no metadata.namespace field — they are project/region scoped, not
namespace scoped. Add CKV_K8S_21 to .checkov.yaml skip-check and to the
inline skip comment in cloudrun-service.yaml.
Root cause of 6 consecutive CI failures:
MSDO 0.215.0's guardian.cmd wrapper breaks the build whenever checkov exits
with code 1. Checkov exits 1 on ANY violation, including MEDIUM/LOW findings
that are all 'below minimum severity'. This makes Active results = 0 and
'Found no breaking results', yet Guardian still raises BreakException because
it treats the tool's exit code as a first-class breaking signal. The
.checkov.yaml soft-fail setting was never read because the MSDO runner
bypasses repository config files.
Fix:
- Remove checkov from the MSDO tools list (stops the guardian.cmd crash)
- Add a dedicated 'checkov' job on ubuntu-latest using the official
bridgecrewio/checkov-action@v12, which runs a current checkov release,
runs on Linux, and correctly reads .checkov.yaml and respects soft_fail
- Set soft_fail: true in the action so low/medium findings appear in the
Security tab without ever blocking the build
- MSDO continues to run eslint, templateanalyzer (Bicep/ARM), and terrascan;
these tools all have well-behaved exit codes and produce no active results
after the security fixes applied earlier in this PR
.checkov.yaml:
- Replace soft-fail: true (was a failed workaround for MSDO) with
skip-check: [CKV_K8S_28, CKV_K8S_30] — correct suppression for the
Knative false-positives (Cloud Run enforces seccomp + AppArmor at
platform level without requiring K8s annotations)
checkov scans deploy/gcp/cloudrun-service.yaml as a Kubernetes resource
(it has apiVersion: serving.knative.dev/v1) and raises CKV_K8S_28 /
CKV_K8S_30. Adding those annotations to spec.template.metadata.annotations
caused checkov to crash (exit 1 with no SARIF output) — likely a bug in
checkov's AppArmor check when it tries to match the annotation container
name against containers in a Knative RevisionSpec. Fix:
- Remove the AppArmor / seccomp annotations from the template metadata
- Add checkov:skip comments at the file top so the false-positive checks
are suppressed cleanly (Cloud Run enforces these at platform level)
Also drop the legacy seccomp.security.alpha.kubernetes.io/pod annotation
from deploy/helm/knowledge-explorer/values.yaml: run #186 confirmed that
the modern podSecurityContext.seccompProfile.type: RuntimeDefault field
already satisfies CKV_K8S_28 for the Helm chart without the annotation.
Adding the annotation alongside the modern field was causing the same
crash in checkov's Helm-rendered output.
checkov crashes (exit 1) on two constructs introduced in earlier commits:
1. deploy/gcp/cloudrun-service.yaml: pod-level spec.template.spec.securityContext
is not part of Knative RevisionSpec. checkov's Knative parser panics on
this unknown field. Remove it — CKV_K8S_28 (seccomp) and CKV_K8S_30
(AppArmor) are already satisfied by the legacy annotations in
spec.template.metadata.annotations; the container-level securityContext
that IS valid in Cloud Run Gen 2 is kept.
2. deploy/azure/main.bicep: 'vnetInternal ? { ... } : null' compiles to
ARM null() which crashes checkov's Bicep/ARM parser. Replace the inline
null ternary with two concrete variable objects (vnetConfigInternal and
vnetConfigExternal) so both branches are well-typed objects.
Active results are 0 and 'Found no breaking results' but MSDO still fails
because checkov exits with code 1 whenever it finds any violation
(including MEDIUM/LOW below the minimum severity threshold). MSDO v1.12.0
treats a non-zero tool exit code as a breaking result even when Guardian
reports no active findings.
soft-fail: true makes checkov exit 0 in all cases. MSDO Guardian still
reads the full SARIF output and would surface any HIGH/CRITICAL findings
as active results that break the build, so the security posture is
unchanged.
AZR-000363 (Azure.ContainerApp.PublicAccess) — line 29 managedEnvironment:
- Add vnetConfiguration.internal: true (default) so the environment uses
an internal load balancer instead of a public IP
- Parameterize with vnetInternal (bool, default true) and
infrastructureSubnetId so operators can provide their subnet on deploy
AZR-000361 (Azure.ContainerApp.ManagedIdentity) — line 40 containerApp:
- Add identity.type = SystemAssigned so the Container App can
authenticate to Azure services without storing credentials
Also update main.parameters.json and README with the new parameters.
The 2 active checkov HIGH results (CKV_K8S_28 + CKV_K8S_30) were coming
from deploy/gcp/cloudrun-service.yaml — checkov scans it as a Kubernetes
resource (apiVersion: serving.knative.dev/v1) and flagged missing AppArmor
and seccomp on that file, regardless of the fixes made to the k8s/ and
helm/ manifests.
deploy/gcp/cloudrun-service.yaml:
- Add container name (explorer) so AppArmor annotation key matches
- Add AppArmor annotation to pod template metadata (CKV_K8S_30)
- Add legacy seccomp annotation (AC_K8S_0080 / CKV_K8S_28)
- Add pod-level seccompProfile: RuntimeDefault (CKV_K8S_28)
- Add container securityContext (runAsNonRoot, allowPrivilegeEscalation)
Cloud Run Gen 2 supports all of these fields
deploy/kubernetes/deployment.yaml:
- Pin image tag from ':latest' to ':0.5.0' (AC_K8S_0068 / AC_K8S_0069)
- Add legacy seccomp pod annotation alongside existing seccompProfile field
deploy/helm/knowledge-explorer/values.yaml:
- Add legacy seccomp annotation to podAnnotations so it renders into
the Helm-generated pod template alongside the modern seccompProfile
checkov HIGH (2 breaking results, CKV_K8S_30):
- Add AppArmor annotation to k8s deployment pod template
(container.apparmor.security.beta.kubernetes.io/explorer: runtime/default)
- Add AppArmor annotation via Helm values.yaml podAnnotations so it
renders into the Helm-generated pod template
Terrascan warnings (AC_K8S_0087 / AC_K8S_0080 / AC_K8S_0073):
- Add runAsNonRoot: true and seccompProfile: RuntimeDefault at container
securityContext level in both k8s deployment and Helm values (these
were only at pod spec level before)
Terrascan AC_K8S_0002 (noHttps):
- Add nginx ssl-redirect annotation to k8s ingress so HTTPS enforcement
is explicit at the ingress controller layer
Terrascan AC_K8S_0013 (noOwnerLabel):
- Add owner label to k8s namespace.yaml
Terrascan AC_K8S_0068 (imageWithLatestTag):
- Change Helm values.yaml image.tag from 'latest' to '' (falls back to
.Chart.AppVersion at render time)
- Pin values.prod.yaml to explicit release tag 0.5.0
- GCP: remove --allow-unauthenticated, restrict ingress to
internal-and-cloud-load-balancing, replace wildcard ALLOWED_ORIGINS=*
with a substitution variable (_ALLOWED_ORIGINS) so operators supply a
real URL at deploy time; same fix in cloudrun-service.yaml
- Fly.io: replace hardcoded FALKORDB_HOST=localhost with the correct
.internal private-network hostname pattern; update README accordingly
- docker-compose.dev.yml: add missing top-level networks: block so the
frontend service can join the semantica network without --file layering
- K8s/Helm: add readOnlyRootFilesystem: true + runAsUser: 1000 to
container securityContext; mount an emptyDir /tmp so uvicorn can write
temp files
- app.py: fix _read_explorer_settings() or-chain, use in os.environ
checks so an explicit ALLOWED_ORIGINS="" produces an empty allow-list
instead of silently falling through to localhost defaults; remove dead
app.state.falkordb_host/port attributes
- docs: update four locations that still documented {"status":"healthy"}
to reflect the new {"status":"ok"} health response
- tests: update test assertion to read falkordb settings from
app.state.explorer_settings instead of removed top-level attributes
* feat(export): implement Neo4j Bulk CSV Exporter and update registry docs (#261)
* fix(export): address review bugs in Neo4j CSV exporter
- _write_csv: filter **options to known csv.writer dialect params only,
preventing TypeError when callers pass kwargs like delimiter= or encoding=
that would reach csv.writer twice or as unknown arguments
- export_neo4j_csv: split kwargs into constructor-level init_params vs
per-call call_kwargs before forwarding, eliminating the double-pass that
caused dialect params to collide inside _write_csv
- _prepare_export: remove dead node_id_lookup dict that was built but never
consumed by any caller
- export_knowledge_graph dispatch: drop the ambiguous "neo4j" format alias
(kept "neo4j_csv" and "neo4j-csv"); "neo4j" conflicts with the codebase's
established meaning of the live Bolt/Cypher store backend; add inline
comment clarifying that file_path is treated as an output directory for
this format
- export_usage.md: fix all three wrong API examples — constructor params
node_label_sep/strict_validation corrected to label_separator/strict,
non-existent nodes_path/rels_path kwargs removed, convenience-method
example updated to show the correct positional output_dir argument
Co-Authored-By: KaifAhmad1 <kaif2208@gmail.com>
* docs(changelog): add Neo4j Bulk CSV Export entry for PR #665
Documents the new Neo4jCSVExporter feature contributed by @Luffy2208
and the five follow-up bug fixes (TypeError on dialect kwargs,
double-pass kwargs split, dead node_id_lookup removal, ambiguous
format="neo4j" alias removal, and wrong API examples in docs).
Co-Authored-By: KaifAhmad1 <kaif2208@gmail.com>
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: KaifAhmad1 <kaif2208@gmail.com>
Replace uniform cursor-bar hover effects with a differentiated,
light animation layer per element type. Adds global polish:
smooth scroll, custom scrollbar, brand-colored text selection,
page fade-in entrance, emerald focus rings, H1 gradient underline
accent, styled blockquotes, gradient HR dividers, uppercase table
headers, and CTA button glow — all tuned to the #080C10 dark
background and #10B981 emerald brand color.
* docs: add Changelog tab and clean up overview page
- Remove v0.5.0 release banner and stats grid from docs/index.md
- Add Changelog navigation tab to docs/docs.json after FAQ
- Create docs/changelog.md sourced from CHANGELOG.md with full Mintlify
formatting: one accordion per release (Unreleased → v0.0.1), icons,
pip install snippets, Added/Fixed/Security sub-sections, and a change
type legend
* docs: fix broken index#whats-new link in quickstart — point to changelog
* docs: fix onboarding examples for GraphBuilder and temporal queries
* docs: fix provenance import, hollow example, and query comment (#656 follow-up)
- Fix wrong import: ProvenanceTracker lives in semantica.kg, not semantica.provenance
- Replace hollow provenance accordion with actual track_entity/get_all_sources example
- Annotate query="" in TemporalGraphQuery.query_at_time as reserved for future use
* docs: use ProvenanceManager from semantica.provenance in W3C PROV-O example
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* docs: add choose-your-module onboarding guide
* fix(docs): correct export code examples against actual API signatures
- export_to_rdf() returns a string; use export() for file output
- format="json-ld" is invalid; correct value is "jsonld"
- ParquetExporter/LPGExporter/ArangoAQLExporter take file_path as a
required positional arg, not output= / output_dir= kwargs
- ArangoAQLExporter().export(graph) was missing file_path entirely,
which would raise TypeError at runtime
- Remove misleading 'with provenance embedded' comment (no such param)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Fix What's new → link in Info banner (now a proper <a> tag, always clickable)
- Replace 4-stat CardGroup on index with inline premium stats row
- Convert every <CardGroup>/<Card> block site-wide to markdown bullet lists:
content sections → bold-title bullets with sub-bullets, nav cards → [Title](href) — description
- Add cursor-animated list item hover effects to custom.css:
green inset left border, subtle background tint, marker color change on hover
- Affects index, getting-started, quickstart, concepts, modules, faq, architecture,
installation, cookbook, glossary, learning-more, explorer-setup, cli-setup,
community, contributing-guide, governance, citation, project-license,
all integrations pages, and all 20+ reference module pages
Removes all notebooks, data files, and exports under cookbook/use_cases/
(advanced_rag, biomedical, blockchain, capability_gap_defense, cybersecurity,
finance, intelligence, renewable_energy, supply_chain) and the corresponding
docs/use-cases.md page.
Cleans up all references in docs/cookbook.md, docs/docs.json,
docs/concepts.md, docs/modules.md, and docs/learning-more.md.
- Move Discord, GitHub, PyPI, and Follow on X links from sidebar anchors to top-right navbar
- Lock dark mode as default via appearance.strict and hide theme toggle
- Add custom.css with hover highlighting for tables, code blocks, cards, callouts, and inline code
- Move all Tips and Common Pitfalls sections inline next to their relevant content across all 25 reference docs
- Polish context.md: remove duplicates, condense callouts, upgrade Cookbooks to CardGroup
- Convert Troubleshooting and Performance Optimization sections in installation.md, cli-setup.md, explorer-setup.md, learning-more.md, and faq.md from plain headers to AccordionGroup
- Change navigation-hint Tip callouts to Info in concepts.md, faq.md, glossary.md, and modules.md
* fix: replace invalid Mintlify theme 'venus' with 'mint'
* docs: replace em dashes with colons across all docs files
* fix: strip UTF-8 BOM from all docs files (broke frontmatter detection)
- llms.md: use showcase models (llama-3.3-70b-versatile, gpt-4o) in
provider examples and use-case tables; clarify defaults vs recommended
in Defaults and Reproducibility section
- split.md: document that chunk_size is in characters with migration note
- ingest.md: add Note that glob patterns are not supported by ingest()
- explorer-setup.md: remove hardcoded "1.5 seconds" timing claim
- cli-setup.md: expand semantica-worker description with concrete usage
- mcp_server.md: clarify turtle/ttl are aliases for the same RDF format
- semantic_extract.md: remove emoji from code comments
Adds an explicit top-level `permissions` block to the GitHub Actions CI workflow.
This change sets the `GITHUB_TOKEN` permission scope to the minimum required level (`contents: read`), following the principle of least privilege and addressing the CodeQL alert `actions/missing-workflow-permissions`.
The workflow only requires read access to repository contents for checkout and CI tasks, so no additional permissions are needed.
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(explorer): resolve blank dashboard UI and ship frontend bundle in wheel
Fixes#631 — the Explorer server started successfully but the browser showed
a blank page because semantica/static/ was gitignored and never present after
a fresh install or clone.
Changes:
- ci.yml / release.yml: add Node 20 setup + npm ci && npm run build before
python -m build so every wheel contains a CI-built frontend bundle
- pyproject.toml: add package-data patterns (static/*, static/assets/*) so
setuptools includes the bundle in the wheel; add MANIFEST.in for sdist coverage
- app.py: replace silent empty-HTML fallback with a 200 page that clearly
explains the missing bundle and links to /docs; fix CORS allow_credentials
to default false, gated behind EXPLORER_CORS_CREDENTIALS env var to prevent
credentialed cross-origin requests on unauthenticated endpoints
- __init__.py: warn at startup when --host is non-loopback (unauthenticated
network exposure)
- explorer/README.md: full rewrite covering pip-install mode (primary path,
no Node required) and dev-server mode (contributors), CLI flags, env vars,
workspace table, troubleshooting for the blank-page symptom
- README.md: update Knowledge Explorer section with correct command and link
to the new setup guide
* fix(explorer): set build.target esnext to fix esbuild CI failure
esbuild >=0.28 (forced via npm overrides) conflicts with Vite 6 defaults on
Linux CI — it tries to lower destructuring syntax for the implicit browser
target list but errors out. Explicit target: 'esnext' tells esbuild to emit
native syntax unchanged, bypassing the transpilation error entirely. Safe for
a developer tool that runs in modern browsers.
* test(explorer): verify packaged frontend bundle
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Consolidate dual import (module alias + from-import) to a single
`import ... as progress_module` alias and qualify all references.
Replace bare `BaseException` catch with `Exception` in the thread
runner helper.
Co-Authored-By: Zohaib Hassnain <zohaib179949@gmail.com>
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
Updates CHANGELOG [Unreleased] to record that hardcoded GROQ_API_KEY
fallback values were stripped from advanced_rag/01, advanced_rag/02,
blockchain/01_DeFi_Protocol_Intelligence, and biomedical/01 — covering
secret scanning alerts #1–#6 (gsk_SLLE0, gsk_S4dBVJ, gsk_SLOv6,
gsk_lR6Qcj, gsk_ToJis6, gsk_LmbQBr). Keys already removed from HEAD
in a5da533; all 6 keys must be revoked in the Groq console.
* security: fix 9 Dependabot/CodeQL alerts — DOMPurify, vite, uuid, workflow permissions
- Add explicit permissions block to defender-for-devops.yml (CodeQL #25)
- Upgrade vite 5.4.x → 6.4.3; bundled esbuild 0.21.5 → 0.25.12 (Dependabot #2, #7)
- Force dompurify ^3.4.0 via npm overrides; resolves 6 DOMPurify XSS alerts (#4–#6, #8–#11)
- Force uuid ^13.0.1 via npm overrides; fixes buffer bounds check (Dependabot #12)
* fix(ci): exclude bandit from MSDO scan on windows-latest
bandit_runner.exe builds a per-file command line; on a large Python repo
the total command string exceeds the Windows CreateProcess limit and the
process fails to start (Win32 ERROR_FILENAME_EXCED_RANGE 206).
Exclude bandit via the tools param and retain checkov, eslint,
templateanalyzer, terrascan, and binskim.
* fix(ci): drop binskim (no binaries), enable Neptune audit logging
- Remove binskim from MSDO tools: repo has no compiled binaries so
BinSkim raises AnalyzeArgumentNoValuesException and breaks the run
- Add EnableCloudwatchLogsExports: [audit] to NeptuneCluster to fix
Checkov CKV_AWS_101 (the one error-level result breaking the build)
Remove all horizontal rule dividers for a cleaner premium look.
Replace all Hawksight-AI references with semantica-agi org URLs and update footer attribution from Hawksight AI to Semantica.
Deep-copy **options in ingest_examples and batch_public_apis so that
mutable values (e.g. params dicts) are not shared across iterations.
Add rate_limit_delay to the config_only_key strip list in ingest_public_api
so it is not forwarded twice when passed via kwargs.
Replaces the bare GIF with a structured "See Semantica in Action"
section featuring a clickable YouTube thumbnail for the Knowledge
Explorer Tour (https://youtu.be/QfnNZg4-dZA) above the original GIF,
with named subsections and a feature-list subtitle.
* test(benchmarks): add git-lfs infrastructure validation checks
* fix(benchmarks): add assertions and skip markers to LFS validation tests
Three tests had no assertions and always passed vacuously. Replace with
real assertions gated by pytest.mark.skip so logic is reviewed now and
enforcement is enabled later by removing the decorator. Also fix fragile
CWD-relative paths to use Path(__file__)-anchored roots, drop unused os
import and dead expected_patterns list, replace os.walk with Path.rglob,
and add missing newline at EOF.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Replaces the bare GIF with a structured "See Semantica in Action"
section featuring a clickable YouTube thumbnail for the Knowledge
Explorer Tour (https://youtu.be/QfnNZg4-dZA) above the original GIF,
with named subsections and a feature-list subtitle.
* test(benchmarks): add git-lfs infrastructure validation checks
* fix(benchmarks): add assertions and skip markers to LFS validation tests
Three tests had no assertions and always passed vacuously. Replace with
real assertions gated by pytest.mark.skip so logic is reviewed now and
enforcement is enabled later by removing the decorator. Also fix fragile
CWD-relative paths to use Path(__file__)-anchored roots, drop unused os
import and dead expected_patterns list, replace os.walk with Path.rglob,
and add missing newline at EOF.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Extract _discover_modules() to scan benchmarks/ at runtime so the
--module choices list stays accurate as directories are added or
removed; eliminates the stale context_graph_effectiveness entry and
the missing infrastructure entry from the original implementation
- Add an existence guard before passing the resolved path to pytest so
a valid-looking choice that maps to a missing directory fails fast
with a clear error instead of silently collecting 0 tests and exiting 0
- Print the active module filter to the console so users can confirm
the filtered scope in runner output
CLI section:
- Intro updated to mention startup dashboard and Rich polish
- Data In: added semantica watch examples; removed --watch flag from ingest
(watch is now its own command)
- Developer Tools: new subsection covering init, doctor, changelog, shell,
info with representative examples
What's New in v0.5.0:
- Added Modern CLI Experience subsection listing all 11 improvements:
startup dashboard, grouped help, doctor, init, watch, changelog, shell,
progress bars, elapsed timing, error cards, Windows UTF-8 fix
In TOML, declaring [project.urls] inside the [project] block causes all
subsequent key-value pairs (including dependencies = [...]) to be parsed
as project.urls.* keys, producing:
ValueError: invalid pyproject.toml config: project.urls.dependencies
must be string
Fix: move [project.urls] to after the dependencies array closes and before
[project.optional-dependencies], which is the correct TOML position for a
sub-table of [project].
Elapsed timing
- CLIContext._start records time.perf_counter() at context creation
- _ok() appends elapsed seconds to every success message automatically
Structured error cards
- _show_error_card() renders a red-bordered Rich Panel with title, detail,
and an actionable hint line
- _ERROR_HINTS maps common exception types to fix suggestions
- _run_with_error_handling() now routes all errors through the card renderer
instead of raising plain click.ClickException
Rich progress bars
- kg build: per-source Progress bar (SpinnerColumn + BarColumn +
MofNCompleteColumn + TimeElapsedColumn) when multiple --source flags given;
single-source path keeps the spinner
- ingest: spinner added (was missing entirely); shows filename and recursive flag
semantica changelog
- Hits GitHub releases API via stdlib urllib; compares latest tag against
__version__; renders release notes in a rounded Panel; --json supported
semantica doctor
- Checks: Python version, semantica/rich versions, graph store reachability,
vector store importability, LLM provider env vars, config file, log dir
- Rich table with ✓/⚠/✗ per check; summary error/warning count at bottom
semantica init
- Interactive wizard: graph backend, vector backend, optional LLM key
- Writes ~/.semantica/config.yaml via yaml.dump; --force to overwrite
semantica watch
- Wraps watchdog Observer; matches configurable glob patterns; auto-ingests
on created/modified events; graceful Ctrl+C shutdown
- Guards ImportError with pip install semantica[watch] hint
_HELP_SECTIONS updated to surface init, doctor, changelog, watch
- description: rewritten to lead with the accountability/provenance
angle and name concrete capabilities; drops emoji which render
inconsistently across PyPI clients
- keywords: expanded from 8 to 23 terms covering modern search queries
(ai-agents, llm, graph-rag, decision-intelligence, provenance, etc.)
- classifiers: added Information Analysis, Text Processing::Linguistic,
Database Engines/Servers, Information Technology audience
- [project.urls]: new section with Homepage, Documentation, Repository,
Changelog, Bug Tracker, Discord — shown prominently on the PyPI page
and drive clicks to GitHub/docs
- optional-dependencies: added watch = [watchdog>=3.0.0]; bundled into all
2026-06-04 17:37:11 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Prevents UnicodeEncodeError on the default cp1252 code page when Rich
renders box-drawing characters and emoji in the startup banner and panels.
Placed before all other imports so Click and Rich capture the already-
reconfigured streams. Uses reconfigure() (Python 3.7+) which modifies the
existing TextIOWrapper in-place rather than replacing sys.stdout.
- _BANNER: ASCII art shown when `semantica` is run with no subcommand
- _show_startup: dashboard panel with Graph Store / Vector Store / Profile
status cards; suppressed under --quiet and --json
- RichGroup: click.Group subclass that renders --help with grouped sections
(Data Ingestion, Intelligence, Knowledge Graph, Analytics, Export & Viz,
Infrastructure, Services, Tools) plus a Quick Start block
- main decorator: cls=RichGroup + invoke_without_command=True to wire both
- `semantica shell`: interactive REPL that dispatches subcommands while
sharing the parent CLIContext; supports readline on Unix for line editing
- Guard parse_cmd spinner with `fmt == "json"` (default format) to prevent
Rich status output from polluting machine-readable stdout in piped usage
- Remove unused `Rule` import from cli.py
- Remove unused `_orig_print` variable in verify_rich_cli.py
- Unify semantica.cli import style in verify_rich_cli.py; use cli_mod.main
The docs validation workflow runs python docs_check.py with no pip
install step, so rich is not available. Wrap the rich import in a
try/except ModuleNotFoundError and fall back to plain print() calls
so the script works in both environments:
- With rich installed: coloured pass/FAIL output
- Without rich (CI): plain text pass/FAIL output, same exit codes
## Summary
Overhaul the CLI and all library modules to produce polished, modern
terminal output comparable to tools like uv, gh, and cargo. Rich was
already a declared dependency but barely used — this commit wires it
throughout every layer.
## Changes by layer
### semantica/cli.py — visual overhaul
- Add imports: `box`, `Panel`, `Rule`, `Syntax`, `Text` from Rich
- Add 7 style constants (`_BRAND`, `_KEY`, `_VAL`, `_DIM`, `_SUCCESS`,
`_WARN_STY`, `_TABLE_BOX`) for a consistent colour palette
- `_ok()` now prefixes output with a green ✓ checkmark
- New `_info()` helper (neutral · bullet, respects --quiet)
- New `_warn()` helper (yellow ⚠ prefix, never suppressed)
- New `_pprint()` helper: renders dicts/lists as syntax-highlighted JSON
(Rich Syntax, monokai theme) instead of raw Python repr; strings
pass through unchanged; respects --quiet
- `info` command: banner replaced with a rounded Rich Panel showing
version + tagline; component table uses SIMPLE_HEAD box
- All 7 table sites updated: `box=SIMPLE_HEAD`, `show_edge=False`,
consistent `_KEY`/`_VAL` column styles (KG Stats, Reasoning Engines,
Recent Decisions, Configured Backends, Backup Info, MCP Tools)
- `_run_build()`: `console.status(spinner="dots")` wraps the blocking
build call; skipped under --quiet / --json
- `parse`, `extract`, `embed generate`, `reason run`, `reason explain`,
`deduplicate`: each wraps its long-running operation in a status
spinner, guarded by --quiet / --json
- All 30+ `console.print(result)` calls replaced with `_pprint()`
- All raw `[yellow]Warning:[/yellow]` and "not running" patterns
replaced with the new `_warn()` / `_WARN_STY` style
### semantica/explorer/__init__.py
- Error messages use `Console(stderr=True)` with `[bold red]Error:[/bold red]`
- Graph loading wrapped in `console.status()` spinner
- Startup info replaced with a cyan-bordered Rich Panel showing URL,
API docs, and health endpoint
### Library internals — replace print() with structured logger calls
All modules below had active `print()` calls that bypassed the logging
framework, corrupted spinners, and polluted stdout in piped/programmatic
use. All replaced with appropriate `self.logger.*` calls:
- `semantica/kg/graph_builder.py` — 23 calls: entity resolution
progress, graph structure steps, GraphStore persistence timing, and
the two `='*60` completion banners → `self.logger.info/debug()`
- `semantica/semantic_extract/methods.py` — 4 verbose-mode debug
prints → `logger.debug()`
- `semantica/semantic_extract/relation_extractor.py` — progress +
error prints → `self.logger.debug/warning()` with `exc_info`
- `semantica/semantic_extract/triplet_extractor.py` — same pattern
- `semantica/semantic_extract/semantic_network_extractor.py` — batch
error prints → `self.logger.warning/error()`
- `semantica/semantic_extract/coreference_resolver.py` — error print
→ `self.logger.error()`
- `semantica/semantic_extract/providers.py` — debug print →
`self.logger.debug()`
### Tooling
- `benchmarks/benchmarks_runner.py`: Rule banner, ✓/✗/⚠ status lines,
Rule separators around regression alert
- `benchmarks/infrastructure/compare.py`: removed manual ANSI escape
codes; comparison output is now a Rich Table with SIMPLE_HEAD;
summary uses coloured Rule + styled SUCCESS/FAILURE messages
- `cookbook/advanced/snowflake_ingestion_examples.py`: `_section()`
helper using Rule; tabular data rendered as Rich Table; result lines
use ✓/✗/⚠ prefixes; logger.error already present, retained
- `docs_check.py`: `pass`/`FAIL` lines use `[bold green]` /
`[bold red]`; summary uses styled output
## Tests
- `tests/test_cli_commands.py`: fix 3 pre-existing mock mismatches
- `test_kg_stats_json_with_mock`: mock now uses `compute_metrics()`
(the method the code actually calls) instead of `get_statistics()`
- `test_dry_run_not_needed_extract_is_read_only` and
`test_stdin_input`: mock now provides `NERExtractor`,
`RelationExtractor`, `TripletExtractor`, `EventDetector`
(the classes the code imports) instead of `SemanticAnalyzer`
Result: 230/230 tests pass (was 227/230)
- `tests/verify_rich_cli.py`: new verification script; exercises all
14 command groups (92 --help checks, table rendering, dry-run
formatting, --json mode, _pprint helper); 111 pass, 0 fail
- Wire --confidence, --model, --temporal flags to extractors via a flat
extractor_config dict (min_confidence, llm_model, include_temporal)
instead of the unused kwargs dict and sectioned to_dict() spread
- Pass confidence_threshold=confidence directly to RelationExtractor
which exposes it as a named parameter alongside **config
- Remove dead SemanticAnalyzer import and unreachable else branch from
extract; unsupported modes now consistently raise ClickException
- Add _serialize_extract_result() to convert dataclass/list results to
plain dicts so JSON and YAML output is machine-readable, not str()
- Fix kg_stats: remove graph={} arg from compute_metrics() so it uses
the analyzer's loaded graph instead of always computing on empty data
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Covers all 22 command groups introduced in issue #568:
global flags, data in, processing, KG, intelligence (reason/decision/temporal),
provenance, validation, ontology, export, visualize, orchestration
(pipeline/store/backup), services (server/explorer/mcp), and shell completion.
Each section shows real invocation examples rather than flag tables.
P1 — runtime-breaking API mismatches:
- decision record/list/query/trace/similar/impact/check: all six decision
commands now call decision_methods / decision_query using a GraphStore
from _get_graph_store(cli_ctx) instead of passing config= kwargs that
don't exist on the underlying API signatures.
- embed index: load vectors from the Parquet/JSON file into List[np.ndarray]
before calling create_index(), which expects vectors not a file path string.
P2 — stub implementations replaced with real logic:
- backup sync: now collects local data sources via _collect_backup_sources
and performs an incremental copy (skips files whose dst mtime >= src mtime).
- backup restore: detects .enc / tar.gz / .tar / directory, decrypts SEM1
format when --enc, extracts tar archives with leading prefix stripped, or
copies directory trees back to cwd.
P3 — correctness bugs:
- backup create: archive now includes actual config/ontology/store data files
via _collect_backup_sources; manifest records the file list.
- extract: --output now works for all formats (table/rdf/yaml), not only JSON.
- backup create: empty keyfile now raises a clear error instead of silently
producing an unencrypted archive.
- normalize: use Path.is_file() instead of Path.exists() to avoid accidentally
reading a directory that matches the input text.
- visualize: without --output, emit to stdout; do not silently write kg.html.
Minor:
- _setup_cli_logging: replace opaque _ = (quiet, json_output, exc) tuple
with del to suppress unused-variable lint.
- reason list: try to source engines from the reasoning module registry;
fall back to the hardcoded list.
- deduplicate --action report: use method="pairwise" to produce individual
pair objects with similarity scores, distinct from --action detect.
- tests: remove mixed import (from semantica.cli import main) — all 192
runner.invoke calls now use cli_module.main as CodeQL flagged.
- tests: add two focused embed-index regression tests that verify vectors
are loaded from the file before create_index is called.
- embed search: embed query text before calling search_vectors (was passing
raw string to query_vector positional arg, causing TypeError on every call)
- ontology version: import OntologyVersionManager not OntologyVersioning
(symbol never existed; command always failed even with package installed)
- ingest --watch: forward watch flag into _ingest() kwargs (was accepted
but silently dropped, so --watch had no effect)
- store migrate: replace fake success stub with honest ClickException pointing
to the export+embed-index workaround (no bulk-dump API exists in vector store layer)
Fix the export runtime mismatch where get_export_method expected the existing (task, name) registry contract but the CLI passed only the format argument.
Expands semantica/cli.py from a 2-command stub into a complete terminal
interface covering every capability described in issue #568, and ships
253 tests covering all new commands, flags, and error paths.
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
- Remove incorrect # pragma: no cover from _run_with_error_handling
generic Exception branch (test_runtime_errors_are_click_safe already
covers it via the monkeypatched RuntimeError path)
- Add _require_ctx() guard: converts None ctx.obj into a clean
ClickException instead of an AttributeError (protects standalone_mode=False
/ library-use callers); apply to info, kg_build, build_alias commands
- Rename serve group -> services to avoid collision with the future
`semantica server` flat command specified in issue #568; update docstring
to document planned subcommand layout
- Fix command-level config logging: re-call setup_logging() with the
command-level config logging section when -c is used (setup_logging
clears handlers before adding, so no accumulation risk)
- Fix missing log_level_override in command_ctx: global --log-level was
silently dropped when a per-command -c config was present, breaking
the override chain for any nested _build_runtime_config calls
- Add return-shape docstring on _run_build documenting the expected
build_knowledge_base() return dict structure
- Add type annotation to runner fixture (-> CliRunner) so Pylance
correctly types runner.invoke() -> Result across all test functions
- Expand test suite: 25 -> 32 tests
* test_info_command_shows_framework_components
* test_info_command_shows_config_path_when_supplied
* test_log_level_global_override_stores_in_context
* test_command_config_preserves_global_log_level_override
* test_build_result_with_stats_shows_source_count
* test_build_result_without_stats_shows_generic_success
* test_build_result_none_shows_generic_success
* test_require_ctx_raises_click_exception_on_none
* test_require_ctx_returns_ctx_unchanged
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- keep command-level config from overriding logging unless --log-level is set
- validate YAML/JSON config roots and surface parse failures as Click errors
- tighten CLI tests around isolation and cleanup
- add CLI runtime context, global config/log-level handling, and click-safe error wrapping
- implement kg build as a thin wrapper over existing orchestrator build flow
- keep hidden legacy build alias and route both build handlers through shared internal path
- add focused CLI tests for help UX, config flag compatibility, alias parity, and clean error output
- keep tests lightweight by mocking heavy build execution paths
- Rename logo PNG to semantica-logo.png (lowercase, hyphenated)
- Update docs.json logo (light/dark) and favicon to reference new PNG
- Replace legacy purple favicon with new teal brain neural network icon
- llms.md: replace non-exported Anthropic/Ollama imports with LiteLLM provider-prefix pattern; replace ReasoningEngine with Reasoner; replace create_provider with LiteLLM in YAML config example and tip
- concepts.md: replace ReasoningEngine with Reasoner/ReteEngine/GraphReasoner; fix DatalogReasoner.reason() to evaluate()/query(); replace TemporalKnowledgeGraph with TemporalGraphQuery; replace DistanceCalculator with SimilarityCalculator; replace EntityDeduplicator with DuplicateDetector/EntityMerger
- kg.md: replace non-exported build_knowledge_graph with method_registry.execute()
- semantic_extract.md: replace Anthropic import with LiteLLM
- index.md: replace Anthropic/Ollama imports with LiteLLM
- modules.md: fix TemporalKnowledgeGraph, DistanceCalculator, OntologyManager, ReasoningEngine, DatalogEngine, start_explorer, create_provider across code examples and module index table
- triplet_store.md: replace non-exported NamespacePrefixManager with semantica.ontology.NamespaceManager
Replace plain markdown in every docs/reference/ file and docs/concepts.md with
rich Mintlify JSX components — CardGroup, Steps, Tabs, AccordionGroup, Tip,
Warning, Note, and CodeGroup — for a consistent, navigable, production-grade
developer experience.
Frontmatter, intro, heading, Tabs, and Module Map all said "three-layer"
while the architecture-overview.svg and its alt text showed four layers.
Adds Layer 3 (Intelligence: KG, vector store, ontology, triplet store,
embeddings) and renumbers the former Layer 3 Application to Layer 4.
Diagrams (docs/assets/img/diagrams/):
- architecture-overview.svg: 4-column layered architecture
- pipeline-flow.svg: 8-step numbered pipeline flow
- kg-structure.svg: entity/relation graph with typed nodes and labeled edges
- graphrag-flow.svg: dual-path retrieval (vector + graph) to LLM to grounded answer
- extraction-pipeline.svg: NER/Relation/Coreference fan-out to Triplet Generator
- agent-context-flow.svg: AgentContext hub with VectorStore and ContextGraph
- reasoning-chain.svg: forward-chaining inference with explanation path
Wordmark logo (light + dark SVG variants):
- Green rounded-square S icon + Semantica text in green
- docs.json updated to use wordmark SVGs for light and dark modes
Pages updated with diagrams:
- index.md, architecture.md, quickstart.md, concepts.md
- reference/kg.md, reference/pipeline.md, reference/semantic_extract.md
- reference/context.md, reference/reasoning.md
- installation.md: revert card title back to "Getting Started" to match
the Tip text that already links to it by that name
- getting-started.md: restore pip install semantica[all] code block that
was removed in the original PR; users need the copy-paste snippet even
when the Installation guide is the canonical reference; also standardize
link text to "Installation" (was "Installation guide")
- index.md: add Installation card as first entry in "Start Here" CardGroup
so the prose ("install first, then open Quickstart") is backed by an
actual card to click
- quickstart.md: standardize link text to "Installation" (was "Installation guide")
mint export fails with 'file does not exist' for pages named 'contributing'
and 'license' — these are reserved by Mintlify's GitHub integration layer.
Renamed to contributing-guide.md and project-license.md and updated all
nav entries and cross-links throughout the docs.
Also adds .gitattributes LF rules to prevent CRLF issues from Windows devs.
- Validate docs structure with docs_check.py (Python)
- Validate Mintlify build with mint validate (Node 20 LTS)
- Export static site with mint export, deploy to GitHub Pages
- Deploy job skipped on PRs (validate-only for branches)
- docs.yml: replace mkdocs build/deploy with python docs_check.py;
Mintlify deployment is handled by its own GitHub App
- ci.yml: remove dead paths-ignore refs to deleted mkdocs.yml and
requirements-docs.txt
* Add XML file ingestion support
* fix(xml-ingestor): add ingest_string test and document ingest() return keys
- Add test_xml_ingestor_ingests_string to cover the public ingest_string()
method which had no test coverage
- Document all source_type return keys in the ingest() docstring so callers
know to use result["xml"] rather than result["data"] for XML sources
* docs(changelog): add unreleased entry for XML ingestion support (#560)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* docs(readme): redesign for better traction and narrative clarity
- Reorder sections: Problem → Solution → Quick Start → What's New → Integrations
- Add website and docs badges to the top badge strip
- Improve hero tagline and narrative blockquote
- Restore v0.4.0 (Temporal, SKOS, SHACL) and v0.3.0 release sections
- Remove duplicate modules list; consolidate into single table
- Fix broken emoji characters in Enterprise section
- Add blank lines around all headings and list blocks
* docs(readme): concise rewrite with accurate v0.5.0 features and compact layout
- OntologyManager: remove red error banner on HTTP 500; always fall back
to empty state silently (error banners reserved for user actions only)
- AlignmentsTab: remove offline-backend warning when both registry and
alignments requests fail; show empty form silently
- ShaclStudio: fix Monarch tokenizer crash — [@] character class prevents
Monaco from misinterpreting @prefix/@base as language-property refs;
wrap beforeMount in try/catch so any Monaco setup failure cannot crash
the React tree
Decision workspace:
- Add AbortController per loadChain() call; abort previous request when a
new decision is selected, preventing stale out-of-order chain responses
- Guard all setState calls with signal.aborted so unmounted component
state updates are skipped; cancel in-flight request on unmount via a
dedicated cleanup effect
SPARQL workspace:
- Guard results table on both result.rows && result.columns to prevent
runtime crash when backend omits columns field
- Use (result.columns ?? []) inside rows.map() to satisfy TypeScript
narrowing inside the closure
- Add .catch() to clipboard.writeText() — silently swallows permission
errors (query remains visible in the editor as fallback)
- Fix CSV export anchor: append to body before click, remove after, to
ensure cross-browser compatibility
Import/Export workspace:
- Fix download anchor: append to document.body before a.click() and
remove afterwards, matching the standard compatible pattern
Lineage workspace:
- Replace 🔗 emoji empty-state icon with lucide-react Link2 for
consistent theming and sizing
Diff & Merge workspace:
- Add "Sample preview" banner above the mock diff table so users know
the displayed fields are illustrative until the backend is connected
OntologyManager:
- Restore non-blocking warning (flash message) when HTTP response is
non-OK and not a 404; network errors (backend down) stay silent
AlignmentsTab:
- When both registry and alignments promises reject, surface a soft
error banner so users know data is missing rather than just empty
Four issues raised in code review:
- Mode.JSON retry now strips response_format from create_kwargs before
calling json_client.chat.completions.create, preventing incompatible
kwargs from being forwarded to a client configured for a different mode.
- Add exc_info=True to the generate_structured fallback warning in the
manual repair loop so the gateway rejection traceback is visible in
production logs, consistent with the other warnings added in this PR.
- Remove the duplicate is_available definition in GroqProvider. Python
silently kept only the second definition; the first (with diagnostic
branching) was dead code and could cause confusion on future edits.
- Validate base_url scheme in OpenAIProvider._init_client. Non-HTTP(S)
schemes (file://, ftp://, javascript:, etc.) are now rejected with a
ValueError at init time, preventing SSRF if base_url originates from
configuration rather than hardcoded values.
Add 3 new tests: SSRF scheme rejection, valid-URL acceptance, and
exc_info presence on the generate_structured fallback warning (20/20 pass).
Update CHANGELOG.md with full description of all fixes under [Unreleased].
Three bugs caused NERExtractor to silently return pattern-based entities
even when method="llm" was configured:
1. exc_info=True missing on method-failure warning in NERExtractor —
the root exception was swallowed, making the gateway error invisible
in logs even with DEBUG enabled.
2. OpenAIProvider.generate_structured always sent response_format=json_object
to the API. Custom/enterprise gateways (Qwen, LLaMA proxies, internal
gateways) often reject this parameter, causing both the instructor path
and the manual repair loop to fail with the same error on every retry.
3. generate_typed manual repair loop had no fallback when generate_structured
itself raised — it retried the same failing call up to max_retries times,
then propagated the error, triggering _extract_fallback (pattern extraction).
Fixes:
- Add exc_info=True to the method-failure warning so the full traceback
appears in logs and users can diagnose the root cause.
- Skip response_format=json_object in OpenAIProvider.generate_structured
when base_url is set (custom endpoint), since standard OpenAI gateways
don't require it and third-party ones reject it.
- In the generate_typed manual repair loop, catch generate_structured
failures and immediately retry via plain generate() + _parse_json,
breaking the retry-the-same-failing-call loop for custom gateways.
Also adds 17 targeted regression tests covering all three bug paths,
including the exact gateway configuration reported in the issue.
- Add multilingual README links section (30 languages via readme-i18n.com)
- Pin version badge to 0.5.0 with correct release tag link
- Add "What's New in v0.5.0" section covering Distance Intelligence,
Ontology Hub Suite, Parquet ingestion, indexed search, and security fixes
* Added Parquet ingest support (#234)
* docs: Add Parquet ingestion support to CHANGELOG
- Add comprehensive changelog entry for PR #548
- Document ParquetIngestor class and key features
- Include author credit (@Luffy2208) and PR reference
- Follow existing changelog format and structure
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Convert mcp_server.py to package structure (semantica/mcp_server/)
- Add __init__.py and __main__.py for python -m support
- Add semantica-mcp console script entry point in pyproject.toml
- Fix API method calls (extract -> extract_entities/relations/triplets)
- Remove non-existent _result_cache imports
- Update documentation with both usage methods
Resolves pipx installation issue where semantica.mcp_server was not available.
Provides two ways to run: 'semantica-mcp' command or 'python -m semantica.mcp_server'.
- bug_001: top_k_per_entity now uses OR semantics — keep a candidate if
EITHER entity is under quota, preventing high-quality candidates being
silently dropped when a popular counterpart saturates its quota
- bug_002: validate max_results and top_k_per_entity at construction;
negative or non-int values raise ValueError instead of silent empty output
- bug_003: validate min_similarity in [0.0, 1.0] at construction;
out-of-range values raise ValueError
- bug_004: harden ConflictDetector method='relationship' normalization —
always produces List[Dict] before calling detect_relationship_conflicts
- quality_001: update detect_duplicates + incremental_detect docstrings to
reflect configurable sort_by field (not hardcoded 'confidence')
- quality_002: add _normalize_entity_id helper (always str) used in both
_apply_result_limits and _build_duplicate_groups for consistent ID handling
Backward compatible: callers not using new params see no behavior change.
58 tests pass (0 failures)
Fixes#534
- New __init__ params: max_results, top_k_per_entity, min_similarity, sort_by
- _apply_result_limits: drop below min_similarity, sort by sort_by field,
enforce top_k_per_entity per entity, cap at max_results globally
- Wired into detect_duplicates() and incremental_detect()
- 30 new tests in TestResultLimiting; full suite 42/42 passed
Fixes#533
- Removes duplicate `detect_conflicts` definition that was silently overridden,
causing AttributeError for callers passing `method=` or `property_name=` kwargs
- Merges dispatcher logic into the surviving method with `method="all"` default
supporting: "all", "value", "property", "type", "relationship", "temporal",
"logical", "entity"
- Fixes `method="relationship"` incorrectly defaulting `relationships` to the
entities list; now defaults to `[]` with dict normalization
- Removes unreachable dead code block after try/except raise in
`detect_entity_conflicts`
* fix(deps): remove gpu extra from [all] to fix Windows installation failure
faiss-gpu has no Windows builds, so semantica[all] failed with
'No matching distribution found for faiss-gpu>=1.7.0' on Windows.
Removed gpu from both [all] lines — semantica[gpu] remains available
as an explicit opt-in for Linux GPU environments.
Closes#532
* docs(changelog): record faiss-gpu Windows installation failure fix (#532)
Closes#531
- Replace 5 direct sys.stdout.write() calls in ConsoleProgressDisplay.update()
with self._safe_write() so emoji/block characters are encoded safely on
Windows cp1252 consoles
- Add TestProgressTrackerEncoding regression tests (3 cases) covering
_safe_write, pipeline header, and auto emoji-disable on cp1252
test_retry_logic.py injected sys.modules["openai"] = MagicMock() at module
level so providers.py could be imported without the real openai package.
Those mocks were never restored, leaving openai (and spacy, instructor etc.)
as MagicMock objects for the entire test session. This caused
test_pr482_deepseek_openai tests to receive a MagicMock when importing
openai.OpenAI, making MagicMock(spec=OpenAI) raise InvalidSpecError.
Fix: save original sys.modules entries before injection and restore them
immediately after the semantica imports that needed the mocks complete.
The mock objects remain bound inside the already-imported provider module,
so test_retry_logic tests are unaffected; other test modules now see the
real packages again.
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
subprocess.CompletedProcess[str] as a return annotation is not subscriptable
at runtime on Python 3.8, causing test collection to abort before any tests
run. Adding PEP 563 deferred evaluation makes all annotations strings at
import time, restoring 3.8 compatibility without changing behaviour on 3.9+.
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-05 13:40:11 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* fix(ingest): lazy-load optional ingestion backends
* fix(ingest): address qodo review — use ModuleNotFoundError and guard ConfigurationError
Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.
Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
* docs(changelog): record lazy ingest backends fix and qodo review fixes (#535)
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
* fix(tests): set exc.name on blocker's ModuleNotFoundError to match Python import machinery
OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.
Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.
Co-authored-by: Zohaib Hassan (@ZohaibHassan16) <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Zohaib Hassan (@ZohaibHassan16) <zohaib179949@gmail.com>
OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.
Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.
Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Backend (semantica/explorer/routes/ontology.py):
- suggest-alignments: add TF-IDF character-ngram embeddings via sklearn
(SimilarityCalculator-compatible cosine scoring) so embedding_similarity
is populated in results; combined score = 0.4*label + 0.6*embedding when
available, falling back to label-only when sklearn is absent
- suggest-alignments: add token-overlap prefilter before SequenceMatcher so
zero-Jaccard pairs are skipped without computing full similarity; add
_MAX_ENTITIES_PER_SIDE=500 per-ontology cap on top of the existing
_MAX_ANALYSIS_NODES global cap
- suggest-alignments: remove dead try/except OntologyEngine.create_alignment
block that always failed silently (no TripletStore configured); replace
with a comment explaining the intentional ephemeral-only storage model
- health: replace O(alignments x entities) any() scans for alignment coverage
with O(1) set membership checks via assessed_ids
- shacl/validate: run rdflib.Graph().parse(format='turtle') syntax check on
the submitted Turtle before returning; invalid syntax now raises 422 instead
of returning a misleading unavailable/success response
Frontend:
- AlignmentsTab: add pairwise alignment matrix section that groups recorded
alignments by (source_ontology, target_ontology) pair; each cell shows
color-coded relation badges per RELATION_COLORS; clicking a badge populates
the create/edit form for quick editing; matrix is shown when at least two
ontologies are loaded
- ShaclStudio: add selectedShapeId state and fullShacl ref; each shape row in
the library is now a clickable button that extracts its Turtle block from
the full SHACL and pre-populates the Monaco editor; a "View all" toggle
restores the full SHACL; selected shape ID is shown in the editor header
- GraphWorkspace: fix viewMode race in external focus effect — call
setSelectedNodeId directly instead of going through focusNode(), which
captured a stale viewMode in its closure; remove focusNode from the
dependency array since it is no longer called
Tests (14 passing, was 11):
- Add test_suggest_alignments_returns_embedding_similarity: asserts
embedding_similarity is non-null when sklearn is available
- Add test_shacl_validate_rejects_invalid_turtle_syntax: asserts 422 on
syntactically invalid Turtle
- Add test_health_alignment_coverage_uses_set_lookup: asserts alignment
dimension score is non-zero after recording an alignment, verifying the
O(1) set lookup path works correctly end-to-end
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Backend:
- Replace false conforms=True SHACL stub with status=unavailable always;
live validation cannot be wired until OntologyEngine.validate_graph is
connected to a data graph — a stub that returns conforms=True misleads
users editing shapes
- Cap node/edge fetches in health, suggest-alignments, and SHACL generation
at _MAX_ANALYSIS_NODES (5 000) with a logger.warning when the graph
exceeds the limit; unbounded limit=999_999 fetches cause OOM on large graphs
- Set SHACL health dimension score to 0.0 (was 70.0) when status=unavailable;
exclude unavailable dimensions from the total_score average so they neither
inflate nor deflate the result
- Allow alignments to reference external/unloaded URIs (e.g. schema.org)
without raising 404; label falls back to URI fragment or caller-supplied
source_label/target_label fields added to OntologyAlignmentRequest
- Fix _alignment_id to use uuid.NAMESPACE_OID instead of NAMESPACE_URL;
the composite key is not a URL
- Fix _summarize_shapes to normalise \r\n before splitting on .\n so shape
parsing works correctly on Windows line endings
Frontend:
- Wrap handleSave/handleSuggest/handleRemove/handleAcceptSuggestion in
useCallback in AlignmentsTab for consistency with sibling components
- Add ephemeral-storage banner in AlignmentsTab warning that alignments are
session-memory-only and not persisted across restarts
- Fix exportReport in HealthTab to append/remove anchor from document before
clicking and defer URL.revokeObjectURL to avoid Blob URL leak in some browsers
- Derive health dimension grid column count from health.dimensions.length
instead of the hardcoded repeat(5, ...) that breaks if the backend adds
or removes a dimension
- Add minimal Monarch tokenizer for the Monaco turtle language registration
in ShaclStudio so prefix declarations, IRIs, SHACL properties, comments,
and string literals are syntax-highlighted; previously the editor rendered
as plain text despite theme rules being defined
Tests (11 passing, was 5):
- Rename test_shacl_validate_has_stable_contract to
test_shacl_validate_returns_unavailable and assert status == unavailable
- Add test_shacl_validate_rejects_empty_turtle (expects 422)
- Add test_health_returns_404_for_unknown_ontology
- Add test_health_shacl_dimension_is_zero_when_unavailable with total_score check
- Add test_delete_unknown_alignment_returns_404
- Add test_alignment_upsert_is_idempotent (verifies ID stability and created_at
preservation across updates)
- Add test_alignment_accepts_external_uri (verifies no 404 for schema.org URIs)
- Relax test_alignment_suggestions_are_ranked label assertions to substring
checks so the test survives similarity algorithm changes
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
- Fix domain_uri/range_uri always being truthy strings
- Only create rdfs:domain/rdfs:range edges when domain/range are non-empty strings
- Add proper validation with .strip() to handle whitespace-only values
- Apply fix to both 'data' and 'text' mode ontology creation
- Prevents pollution of graph with invalid edges to namespace root
Fixes issue where empty domain/range values like '' or None would still create
edges pointing to namespace root (e.g., 'https://ex/#/') instead of being
properly omitted.
The pattern `<[^>]+>\s+<[^>]+>` in _detect_format() was flagged by CodeQL
(py/polynomial-redos, CWE-1333/730/400) as a polynomial regular expression
on uncontrolled user data.
The `<...>` branch was already unreachable — strings starting with '<' return
'xml' two lines above — but CodeQL does not track that control flow path.
Fix: replace the entire re.match() call with plain startswith / 'in' checks:
- N-Triples with URI subjects are already handled by the XML branch.
- Only blank-node-subject N-Triples (_:word <uri> ...) need detection here,
which is correctly expressed as startswith('_:') and ' <' in stripped.
- Removed the now-unused `import re`.
Closes security advisory #23.
2026-05-01 15:26:17 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Bug 4 — Upload format misdetected:
- Added xml→'xml' and json→'json-ld' to the extension→format map so
.xml and .json files are no longer misidentified as turtle.
- Changed the fallback from '|| "turtle"' to '?? ""' (empty string for
unknown extensions) so the backend _detect_format() runs instead of
blindly assuming turtle for any unrecognised extension.
- Omit the format key entirely from the load request body when no format
was detected, letting the backend auto-detect from content heuristics.
- Added .n3 to the file picker accept list and dropzone hint text.
Bug 1 — Broken registry filters:
fetchRegistry no longer sends format/kind values (owl/skos/internal/external)
as the status query param; those filters are applied client-side via
filteredEntries which already had the correct logic. Only the text search
param q is delegated to the backend.
Bug 2 — Toggle/refresh URI corruption:
Removed removesuffix('/toggle') and removesuffix('/refresh') from
toggle_ontology and refresh_ontology. Starlette's route regex already
strips the literal suffix from the captured path param; the removesuffix
call was a no-op for normal URIs but corrupted any ontology URI that
legitimately ends with /toggle or /refresh.
Bug 3 — SSRF in URL fetch:
Added _validate_fetch_url() which rejects non-http/https schemes and
resolves the hostname to block private, loopback, link-local, reserved,
and multicast addresses before requests.get() is called. Applied to all
three fetch sites: preview, load, and refresh.
Bug 5 — Inconsistent XML hardening:
_parse_rdf_sync now calls _safe_parse_rdf() from
semantica/explorer/utils/rdf_parser.py instead of g.parse() directly,
applying the existing defusedxml-based XXE protection for RDF/XML inputs.
Bug 6 — Search scans whole graph:
search_entities now calls session.search(q, limit*6) which hits the
GraphSearchIndex instead of fetching up to 999,999 nodes and doing a
linear Python substring scan. Results are post-filtered by _SEARCHABLE_TYPES
and entity_type before being returned up to the requested limit.
- Replace invalid inset-left with inset: 0 0 0 72px on ::before at <=680px
- Add matching mobile inset fix to ::after (was still at 88px)
- Merge duplicate .landing-capability-band CSS rule blocks into one
- Fix non-standard font-weight: 850 -> 800 on .landing-launcher-item-title
- Remove unused eyebrow field from LandingAction type and all data entries
- Extract static 42-dot SVG preview array to module-level PREVIEW_DOTS constant
Co-Authored-By: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
- Align _coerce_embedding_vector inner dict-probe key list with
_extract_node_embeddings outer key list (add 'embeddings', reorder to
generic-first) so nested embedding dicts resolve consistently.
- Add TODO comment on _extract_node_embeddings to cache per-session
graph revision and avoid O(N) re-scan on every semantic request.
- Add deprecation docstrings to legacy path-segment routes
(/node/{id}/path and /node/{id}/semantic-neighborhood) documenting
the known slash-in-ID limitation and pointing to the query-param
alternatives.
- Extract _FakeSimilarity to module level so it is shared without
duplication across test classes.
- Rewrite test_legacy_semantic_neighborhood_still_works_for_simple_ids
as a fully isolated TestClient session instead of mutating the
shared module-scoped 'client' fixture, preventing cross-test
state pollution.
- Extract _make_slash_node_session helper to reduce boilerplate in the
slash-safe route tests.
Co-Authored-By: ZohaibHassan16 <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <98801504+KaifAhmad1@users.noreply.github.com>
- Fix dead `if (anchorNodeId)` conditional in buildHeatmapRenderSnapshot
(anchor is always truthy past the early-return guard on line 263)
- Replace O(n) array .includes() with WeakMap-cached Set.has() in
resolveDistanceNodeStyle heatmap path — prevents per-node O(n) scan
during every Sigma reducer pass on large graphs
- Rename GraphDistanceBucketCounts.threeHop → threeHopPlus across
types.ts, graphSceneState.ts, and GraphWorkspace.tsx so the field
name reflects that it accumulates distance ≥ 3, not exactly 3;
update status-strip labels to "3+ hop" accordingly
- Restore hasMetrics guard in PathDistanceIntelPanel to suppress the
empty metric grid <div> when a path result carries no optional metrics
Co-Authored-By: ZohaibHassan16 <zohaib@hawksight.ai>
Co-Authored-By: KaifAhmad1 <mohammadk78600@gmail.com>
- Extract ENTITY_SHAPE_ALIASES and classifyEntityShape into a shared
graphEntityShape.ts utility — resolveEntityShape was duplicated with
divergent signatures in useLoadGraph.ts and graphSceneState.ts; both
now import from one place so aliases can never drift
- graphSceneState.resolveEntityShape falls back to classifyEntityShape
for nodes created programmatically that bypass useLoadGraph
- Fix graphTheme.ts indentation around fullGraphStructure,
fullGraphStructureLayer, and interaction — closing braces were at
wrong indent levels making the nesting visually misleading
- Add comment on fullGraphStructureLayer.mode explaining it is
intentionally "off" as a staged-rollout gate (flip to "auto" to enable
cross-community canvas curve rendering)
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Merge origin/main (Distance Intelligence #502) into feat/explorer-visual-refresh.
Conflict was in the viewModeItems useMemo: the PR's new cluster-based toolbar
structure diverged from main's coreToolbarGroups additions.
Resolution:
- Keep PR's viewModeItems as a clean 3-item segmented control (Full/Grouped/Focused)
- Port Distance Intelligence controls (ego mode, heatmap, structural/semantic overlay)
into a new distanceToolbarItems useMemo that slots into the cluster toolbar as a
"Distance" cluster, visible only when a node is selected
- Wire distanceToolbarItems into toolbarClusters between "local-structure" and
"analysis" clusters
- All other Distance Intelligence additions (state vars, BFS helpers, useEffects,
ego depth slider, GraphInspectorPanel onFocusNode prop) merged cleanly
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Merge blockers (ZohaibHassan16):
- fix: distance-matrix raises HTTP 503 when metric=semantic but no
similarity backend is available, instead of silently returning hop
distances labeled as semantic
- fix: distance-enriched export now requires node_subset (HTTP 422 if
omitted), preventing unbounded all-pairs O(n^2) export over full graph
- fix: DistanceExportRequest default include corrected from ["hops",
"distance_band"] to ["source_id", "target_id", "hop_count",
"distance_band"] so default exports are unambiguous and use the correct
column name
- fix: confidence decay edge weight index now reads graph_dict.get("edges")
or graph_dict.get("relationships") to handle both graph dict shapes,
fixing always-1.0 decay when session returns relationships key
Bot findings (github-code-quality / chatgpt-codex):
- fix: remove unused Iterable import from distance_exporter.py
- fix: remove unused Response import from graph.py
- fix: move logger init before optional KG import; replace empty
except ImportError: pass with logger.debug in distance_exporter.py
- fix: replace two bare except Exception: pass in temporal.distance_history
with logger.warning including source, target, metric, and timestamp context
- fix: remove mixed import style in test_qual003 — use only module import
and reference CausalChainAnalyzer through it
- Fix `import.meta.env.DEV` crash in graphSceneState.ts that broke the
entire test:graph-workspace suite (module load fails in Node.js/tsx)
- Export `resolveGroupedDisplayNodeId` from graphSceneState.ts and
remove the identical copy in GraphWorkspace.tsx
- Add `checkGroupedViewAvailability` helper (Louvain only, no centrality)
so grouped view availability can be checked cheaply on every graph change
- Gate full community graph build (`groupedDisplayCandidate`) on
`viewMode === 'grouped'` to avoid running Louvain + centrality on every
graph version tick when the user is not in grouped view
- Remove dead ternary in `focusNode` where both branches returned `nodeId`
- Add 7 new tests covering resolveGroupedDisplayNodeId,
resolveGroupedDisplayStateSnapshot, and checkGroupedViewAvailability
Co-Authored-By: ZohaibHassan16 <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <mohammadk78600@gmail.com>
Merge conflict resolution:
- Kept fix/graph-motion's conditional layout-stop (only in focused mode)
to preserve live layout motion for derived graphs — the core intent of
this PR.
Must-fix items resolved:
1. plugin.json — removed "hooks": "./hooks/hooks.json" (re-added by this
branch, already removed in PR #489 on main as it is auto-loaded).
Kept "agents": "./agents".
2. Double Louvain per render — added groupedViewAvailable useMemo in
GraphWorkspace (deps: [graphVersion]) that runs community detection
once. Passed result into resolveDisplayGraph and resolveDisplayStateSnapshot
via new groupedViewAvailable option; both functions skip their internal
computeGraphAnalyticsBase call when the value is pre-supplied.
3. graphVersion in displayState deps — removed graphVersion from the
displayState memo dep array. displayState now depends on the stable
boolean groupedViewAvailable, not on every ADD_NODE/ADD_EDGE tick,
so Louvain no longer re-fires on every WebSocket update.
4. hideLabelsOnMove / hideEdgesOnMove flipped to true — intentional:
suppressing labels and edges during pan reduces visual noise and is
part of the flicker-reduction fix described in the PR.
Co-authored-by: ZohaibHassan16 <zohaibhassan1696@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
1. Prevent active-but-disabled Focused button by only disabling when
viewMode is not already "focused" (viewMode !== "focused" && !canActivateFocusedMode).
2. Generalize inspector fallback copy — stale/invalid node IDs are not
necessarily grouped items, so remove the misleading "Activate Focused
mode" hint.
3. Move pluginRuntimeRef.current read out of render by converting
canActivateFocusedMode from useMemo to useState + useEffect, resolving
two ESLint "cannot access refs during render" errors and the missing
toolbar-memo dependency warning.
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Claude Code auto-loads hooks/hooks.json. Declaring it explicitly in
manifest.hooks causes: 'Duplicate hooks file detected ... already-loaded'.
Same pattern as agents: manifest should only reference *additional* hook
files beyond the default.
Two separate schema issues blocked `/plugin marketplace add ./plugins`
followed by `/plugin install semantica@semantica-local`:
1. `marketplace.json` was missing the required top-level `owner` object.
Claude Code rejects with: `owner: Invalid input: expected object,
received undefined`.
2. `plugin.json` declared `"agents": "./agents"` (string), but Claude
Code's manifest schema rejects non-array `agents` with:
`Validation errors: agents: Invalid input`. Auto-discovery from
the default `agents/` directory works when the field is omitted,
provided agents are flat `<name>.md` files with frontmatter (Claude
Code's subagent convention) rather than `<name>/AGENT.md`
subdirectories.
Changes:
- add `owner` object to `marketplace.json`
- drop `agents` field from `plugin.json` (falls back to auto-discovery)
- rename `agents/<name>/AGENT.md` -> `agents/<name>.md` (frontmatter
content is unchanged, just the path)
After this, the documented local-install flow succeeds end-to-end.
- Replace deepseek.Client with openai.OpenAI(base_url="https://api.deepseek.com/v1")
in DeepSeekProvider._init_client(); the deepseek PyPI package has no Client class
- Add self.base_url = "https://api.deepseek.com/v1" to DeepSeekProvider.__init__()
(missing from original PR; caused AttributeError on every instantiation)
- Fix verbose_mode NameError in BaseProvider.generate_typed() instructor path
- Update pyproject.toml: llm-deepseek extra now declares openai>=1.0.0
- Update _init_client warning message to reference openai library
- Add 19 tests in tests/semantic_extract/test_pr482_deepseek_openai.py
- Update CHANGELOG.md
Co-authored-by: liling <liling@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
- Replace list.sort() on every upsert with bisect.insort() — O(log n) per
insert instead of O(n log n); bulk rebuild still sorts once at the end
- Replace list.remove() in remove() with bisect.bisect_left + pop() — O(log n)
find instead of O(n) scan
- Wrap handle_graph_mutation() index mutations in self._lock — mutation bridge
fires from a background thread and was racing concurrent search/rebuild calls
- Drop source/target upserts in add_edge() — edges don't change node text so
the index documents are identical; removes unnecessary cache invalidation
- Sort tag values in _cache_key() — ["a","b"] and ["b","a"] now share a cache
entry since _passes_filters() uses set intersection (order-independent)
- Restore @app.get("/") root handler missing from this branch vs main
Co-authored-by: ZohaibHassan16 <zohaibhassan16@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
- Resolve all merge conflict markers in provenance.py, app.py, .gitignore
- Revert broken session.get_nodes()/get_edges() to session.graph.nodes/edges
- Keep undirected=True ego_graph fix for upstream ancestor traversal
- Add direction field to ProvenanceEdge (upstream/downstream/lateral)
- Group lineage edges in _render_markdown by direction section
- Move ProvenanceNode/ProvenanceEdge/ProvenanceResponse to schemas.py
- Restore complete router import set in app.py (sparql, vocabulary, etc.)
Co-authored-by: Sameer6305 <sameer6305@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
- Add _ttl_block() helper to accumulate all predicate-object pairs before
writing, producing a single valid Turtle subject block terminated by one
period — eliminates the bug where rdfs:subClassOf / domain / range were
appended after a closed '.' block
- Add missing data_properties loop to _export_owl_turtle so
owl:DatatypeProperty declarations are no longer silently dropped
- Add _escape_ttl_str() to escape quotes, backslashes, newlines, carriage
returns, and tabs inside Turtle string literals (rdfs:label, rdfs:comment,
owl:versionInfo)
- Unify optional-field null checks to consistent x = prop.get(); if x: pattern
- Add 43 tests in tests/export/test_owl_exporter.py covering syntax validity,
data properties, string escaping, null handling, and header output
- Update CHANGELOG.md with [Unreleased] entry
Closes#478
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend PathResponse with hop_count (len(path)-1) and distance_band
("direct"|"near"|"mid-range"|"distant") as first-class API fields
- Add classify_path_distance() to semantica/utils/helpers.py as the
single source of truth for hop-count thresholds; both the route and
the visualizer import from it, eliminating duplicate threshold logic
- Populate hop_count and distance_band in find_path route via
classify_path_distance(); remove local _classify_distance() copy
- Add highlight_path: list[str] param to KGVisualizer.visualize_network;
path edges rendered as a distance-aware orange trace (opacity and
stroke width scale from direct→distant: 1.0/4px to 0.35/1.5px)
- Fix bidirectional edge lookup: only forward pairs (A→B) along the
path are added to path_edge_set; reverse back-edges in directed
graphs are no longer incorrectly highlighted
- Add logger.warning when highlight_path contains node IDs absent from
the layout position map, surfacing silent no-op mismatches
- Extend frontend PathResponse type in GraphInspectorPanel.tsx and
GraphWorkspaceShell.tsx with hop_count: number and distance_band
literal union to match the updated API contract
- Add 10 new tests: 2 API-level and 8 unit tests covering all four
band boundaries (0, 1, 2, 3, 4, 6, 7, 20 hops); 104 explorer tests
pass, 0 failures introduced
- Update CHANGELOG.md
- PathFinder.bfs_shortest_path() and dijkstra_shortest_path() gain a
directed: bool = True parameter. When False, a temporary undirected
view (graph.to_undirected()) is used for traversal only; the original
directed edges are preserved and returned in the response.
- _make_undirected_view() helper added to PathFinder; falls back safely
for non-NetworkX graph types.
- GET /api/graph/node/{id}/path exposes ?directed=false query param.
- PathResponse gains a directed: bool field echoing the mode used.
- Route now returns 404 on empty path (previously returned 200 with
path: []).
- 21 new tests: 12 unit (TestBidirectionalPathFinding) + 9 API-level
(TestBidirectionalPathRoute). All 120 tests pass.
- CHANGELOG updated under [Unreleased].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add semantica/kg/knowledge_graph.py with KnowledgeGraph dataclass
(entities, relationships, metadata) plus __len__ and __bool__ helpers
- Export KnowledgeGraph from semantica/kg/__init__.py
- Add KGVisualizer._convert_knowledge_graph() for explicit, non-mutating
conversion from KnowledgeGraph to internal dict format
- Route isinstance(graph, KnowledgeGraph) through _convert_knowledge_graph
inside _normalize_graph so all five visualize_* entry points accept
KnowledgeGraph directly without any manual conversion
- Add TestFormalKnowledgeGraphType (15 tests)
Closes#471
- routes/graph.py: raise HTTPException(404) for missing nodes; wrap path_finder call in try/except → HTTPException(404)
- routes/decisions.py: raise HTTPException(404) on chain, compliance, precedents sub-routes
- routes/annotations.py: raise HTTPException(404) on create (missing node) and delete (missing annotation)
- routes/enrich.py: raise HTTPException(404/503/422) for missing nodes, unavailable services, and extraction errors
- routes/temporal.py: fix TemporalPatternDetector method name detect_patterns → detect_temporal_patterns
- explorer/app.py: root / now serves built index.html when available, falls back to minimal HTML shell; add HTMLResponse import
- tests/explorer/test_explorer_api.py: test_extract accepts 503 (spacy/transformers not installed is a valid service state)
All 45 explorer API integration tests pass.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Clarify plugin README install and usage steps
* feat(explorer): add welcome message to root endpoint and bump version to 0.4.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(plugins): update all plugin READMEs for v0.4.0 with full platform list
- Rewrite main community guide with platform table (8 plugins), skills/agents
inventory, Knowledge Explorer section, and per-platform install steps
- Add v0.4.0 badge and Knowledge Explorer section to VS Code, Cline,
Continue, Windsurf, and OpenClaw READMEs
- Fix inconsistent tool count (12 → 17) across all READMEs
- Bump Python requirement from 3.8+ to 3.10+ across all plugins
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs: add PR description for utils → main
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: remove PR_DESCRIPTION.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove duplicate integrations table from bottom of README
- Move Agentic Frameworks section to top alongside AI tools table
- Show only Agno as supported; list LangChain, LangGraph, CrewAI, LlamaIndex, AutoGen, OpenAI Agents SDK, Google ADK as coming soon
- Add logo icons for all agentic frameworks matching existing plugin style
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add integrations/openclaw/ with OpenClawKGTool (REST) and
OpenClawMCPConfig (mcporter.json generator)
- Add plugins/.openclaw-plugin/ bundle (plugin.json, marketplace.json,
README) with MCP + native tool support
- Add OpenClaw badge to README header
- Reorganize "Works With Every AI Tool" table into labeled groups:
Native Plugin Bundle, MCP Server + Plugin, MCP Server, REST API
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 12:27:25 +05:30
518 changed files with 88736 additions and 302923 deletions
> Note: `Docs` and `Cookbook` are external resources maintained outside this file and may change over time. If a link is unavailable, refer to the repository `README.md` and in-repo documentation as canonical fallbacks.
## 🎉 Major Release: Distance Intelligence & Ontology Hub Complete
> First stable, full public release of Semantica. Covers everything shipped across three release stages: 0.3.0-alpha (2026-02-19), 0.3.0-beta (2026-03-07), and 0.3.0 stable (2026-03-10).
- **12 Critical Vulnerabilities Fixed**: Eval injection, XXE, SQL injection, and more
- **SSRF Protection**: Comprehensive URL validation and hostname resolution
- **Input Validation**: Enhanced file upload restrictions and format detection
- **CORS & Headers**: Proper security headers and WebSocket protection
---
## v0.3.0 — Stable (2026-03-10)
## 📊 **BY THE NUMBERS**
### Context Graph Feature Completeness
**Temporal Validity Windows** (by @KaifAhmad1)
Nodes and edges now carry first-class `valid_from` / `valid_until` ISO datetime fields. These are stored directly on `ContextNode` and `ContextEdge` dataclasses — not in metadata — and survive full serialisation round-trips through `save_to_file()` / `load_from_file()` and `to_dict()` / `from_dict()`.
-`ContextNode.is_active(at_time=None)` and `ContextEdge.is_active(at_time=None)` — returns `True` if the node/edge is live at the given time (defaults to now). Handles both tz-aware and tz-naive datetime inputs correctly.
-`ContextGraph.find_active_nodes(node_type=None, at_time=None)` — filters the entire graph and returns only nodes within their validity window.
-`add_node(valid_from=..., valid_until=...)` and `add_edge(valid_from=..., valid_until=...)` — pass validity fields directly in the call signature.
- Bug fix: `is_active()` previously crashed with `TypeError` when passed a tz-aware `datetime` (e.g. `datetime.now(timezone.utc)`). Fixed by normalising all inputs to tz-naive UTC via a new `_parse_iso_dt()` helper.
- Bug fix: validity fields were silently lost in `add_nodes()`, `add_edges()`, `to_dict()`, and `from_dict()`. All four paths now correctly preserve and restore them.
**Weighted Multi-Hop BFS** (by @KaifAhmad1)
`ContextGraph.get_neighbors(node_id, hops=1, relationship_types=None, min_weight=0.0)` now accepts a `min_weight` threshold. Any edge with weight below the threshold is skipped during BFS traversal, allowing callers to confine multi-hop queries to high-confidence causal links. Default `0.0` is fully backward-compatible.
**Cross-Graph Navigation** (by @KaifAhmad1)
Separate `ContextGraph` instances can now be linked and navigated between — hierarchically, like separate knowledge domains that reference each other.
-`link_graph(other_graph, source_node_id, target_node_id, link_type="CROSS_GRAPH") -> str` — creates a navigable bridge and returns a `link_id`. Records a dedicated `"cross_graph_link"` typed marker node internally (not a phantom `"entity"`) and a marker edge.
-`navigate_to(link_id) -> (other_graph, target_node_id)` — jumps to the target graph and entry node for a given link.
-`graph_id` field — each `ContextGraph` now carries a stable UUID so instances can identify each other across save/load.
-`save_to_file()` — now writes a `links` section alongside nodes and edges, containing `link_id`, `source_node_id`, `target_node_id`, and `other_graph_id` for every cross-graph link.
-`load_from_file()` — restores `graph_id` and populates `_unresolved_links` from the `links` section.
-`resolve_links(registry: Dict[str, ContextGraph]) -> int` — reconnects unresolved links post-load. Pass `{graph_id: graph_instance}` for each linked graph; returns the count of successfully resolved links. `navigate_to()` raises a clear `KeyError` with a `resolve_links()` hint if called before resolution.
- Bug fix: the previous implementation auto-created the synthetic marker target as an `"entity"` node (phantom pollution). Fixed by explicitly pre-creating a `"cross_graph_link"` typed `ContextNode` before the marker edge.
- 14 new tests in `tests/context/test_cross_graph_navigation.py` covering all scenarios including full save/load round-trips with partial registry resolution.
**Other Fixes** (by @KaifAhmad1)
-`PipelineBuilder.add_step()` return type annotation corrected from `"PipelineBuilder"` to `"PipelineStep"` — the implementation was already correct; only the annotation and docstring were stale.
-`test_hybrid_search_performance` timing computation fixed — now accumulates a true `search_times` list instead of reusing the last loop iteration's `start_time`; threshold relaxed to `< 5.0s` for real `sentence-transformers` (384-dim) latency on development machines.
-`_parse_relation_result` in `methods.py` — unmatched subjects/objects now produce a synthetic `UNKNOWN` entity instead of silently dropping the relation. All co-founders returned by the LLM are preserved in the output.
-Duplicate relation fix — an orphaned legacy block that appended every relation twice has been removed.
-`extraction_method` parameter added — typed extraction paths now correctly record `"llm_typed"` in relation metadata instead of `"llm"`.
-`_match_pattern` in `reasoner.py` rewritten — splits patterns on `?var` placeholders first, then escapes only literal segments. Pre-bound variables resolve to exact literals, repeated variables use backreferences, non-greedy `.+?`prevents over-consumption of separators.
-Added `tests/reasoning/test_reasoner.py` (4 tests) and `tests/semantic_extract/test_relation_extractor.py` (6 tests).
-`RDFExporter` now accepts `"ttl"`, `"nt"`, `"xml"`, `"rdf"`, and `"json-ld"` as format aliases in `export_to_rdf()`. Aliases resolve before format validation — zero public API changes.
@@ -24,7 +24,7 @@ Security vulnerabilities should be reported privately to prevent potential explo
### 2. Report Security Issue
Create a [GitHub Security Advisory](https://github.com/Hawksight-AI/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix.
Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/semantica-agi/semantica/issues) with "[SECURITY]" prefix.
Include the following information:
@@ -156,8 +156,8 @@ We appreciate responsible disclosure. Security researchers who help us improve t
For security-related questions or concerns:
- **GitHub Issues**: [Create an issue](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix
This document outlines the architecture, directory structure, and usage of the performance benchmarking suite for the Semantica Agentic RAG framework.
## Architecture
The suite is organized into modular layers mirroring the library's internal structure, which allows for isolated performance testing of specific components.
### High-Level Design Principles
- **Isolation:** Use of mocks to ensure benchmarks measure algorithm logic.
- **Virtualization:** A custom `conftest.py` virtualization layer allows tests to run without heavy local dependencies.
- **Pedantic Measurement:** High-iteration counts and statistical rounds to filter out system noise.
## Directory Structure
Based on the current production environment, the suite is organized as follows:
| core_processing/ | Throughput tests for NER, extraction, and graph building. |
| export/ | Serialization benchmarks for JSON, CSV, RDF, and GraphML. |
| infrastructure/ | Support scripts, including the regression comparison engine. |
| input_layer/ | Ingestion, parsing, and splitting performance. |
| normalize/ | Text cleaning, encoding handling, and date normalization. |
| ontology/ | Inference, serialization, and namespace management overhead. |
| output_orchestration/ | Parallelism and execution pipeline management. |
| quality_assurance/ | Deduplication and conflict resolution strategies. |
| results/ | Storage for benchmark JSON outputs and performance baselines. |
| storage/ | Latency tests for Vector stores (FAISS) and Triplet stores (Jena). |
| visualization/ | Computational cost of layout algorithms and chart rendering. |
## Usage
### Running the Suite
To run the full suite and generate a new results file:
```bash
python benchmarks/benchmark_runner.py
```
### Strict Mode (CI/CD)
The suite is designed to integrate with automated pipelines. Using the --strict flag will cause the runner to return a non-zero exit code if a performance regression greater than 15% is detected.
```bash
python benchmarks/benchmark_runner.py --strict
```
### Performance Comparison
The comparison engine (infrastructure/compare.py) uses Z-scores to distinguish between actual performance regressions and environmental noise.
- Regression: Change > 15% AND Z-score > 2.0.
- Noise: Change > 15% but Z-score < 2.0.
### Updating Baseline
When a performance change is intentional (e.g., a more complex but necessary algorithm is added), update the "gold standard" baseline:
Graph Retrieval-Augmented Generation (GraphRAG): A New Era for Intelligent Search
GraphRAG is an advanced technique that combines the retrieval capabilities of vector databases with the structural reasoning of knowledge graphs. Unlike traditional RAG, which relies solely on vector similarity, GraphRAG leverages the relationships between entities to provide more contextually accurate and comprehensive answers.
Key Components:
1. Knowledge Graph: A structured representation of data where nodes represent entities and edges represent relationships.
2. Vector Search: Finds semantically similar text chunks.
3. Graph Traversal: Navigates the knowledge graph to find related entities that might not be semantically similar but are structurally relevant.
Benefits:
- Improved Context: By following relationships, the system can understand the broader context of a query.
- Multi-hop Reasoning: Can answer complex questions that require connecting multiple pieces of information.
- Reduced Hallucinations: Grounding answers in a verified knowledge structure reduces the likelihood of generating false information.
{"entities":[{"id":"python_org","name":"Python Software Foundation","type":"Organization"},{"id":"guido_van_rossum","name":"Guido van Rossum","type":"Person"}],"relationships":[{"source":"guido_van_rossum","target":"python_org","type":"FOUNDED"}]}
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)\n",
"\n",
"# Drug Discovery Pipeline - Vector Similarity Search\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates a **complete drug discovery pipeline** using Semantica's modular architecture. We'll use individual modules directly to build a comprehensive system for drug-target interaction prediction using vector similarity search and knowledge graphs.\n",
"# Map to biomedical categories based on context\n",
"drugs = [e for e in all_entities if e.label == \"PRODUCT\" or (e.label == \"ORG\" and any(kw in e.text.lower() for kw in [\"drug\", \"pharma\", \"medication\"]))]\n",
"proteins = [e for e in all_entities if e.label == \"ORG\" or (e.label == \"PRODUCT\" and any(kw in e.text.lower() for kw in [\"protein\", \"enzyme\", \"receptor\", \"kinase\", \"target\"]))]\n",
"\n",
"print(f\"Extracted {len(drugs)} drugs and {len(proteins)} proteins\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Extracting Drug-Target Relationships\n",
"\n",
"Extract relationships between drugs and proteins to understand drug-target interactions.\n"
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)\n",
"This notebook demonstrates **genomic variant analysis** using Semantica's modular architecture with focus on **graph analytics**, **pathway analysis**, and **temporal knowledge graphs**. The pipeline analyzes genomic data to extract variant entities, build temporal genomic knowledge graphs, and analyze disease associations through reasoning.\n",
Penicillium chrysogenum originated chloro-diydropyridyl-oxopropanimidic acid derivative as a potent EPSP synthase-targeted bioherbicide against invasive weed species
Synthesis, spectral, thermal, and biological characterization of Se(IV) nanocomplexes derived from vitamin E and amino acid mixed ligands as a metal-drug model
BT-11 targets the LANCL2 pathway to attenuate cognitive deficits and hippocampal pathology in Alzheimer’s transgenic rats
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
KaifAhmad1
