* refactor(export): centralize graph-payload key normalization
Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.
Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.
Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.
Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.
CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.
* fix(export): reject non-mapping input to the YAML exporters
export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.
Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.
Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.
* fix(export): address YAML exporter review findings
- semantica/export/yaml_exporter.py — import Sequence from typing
instead of collections.abc. `Sequence[str]` in _require_mapping's
annotation is evaluated at function-definition time; collections.abc.Sequence
only became subscriptable in Python 3.9, so on the 3.8 this project
declares support for, importing this module raised TypeError.
typing.Sequence has supported subscripting since 3.5.3. Mapping stays
imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
and read exported YAML through a context manager instead of an
unclosed yaml.safe_load(open(...)).
* fix(export): reject YAML export payloads with no recognized key
Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.
- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
collections through normalize_graph_payload(), which raises rather than
returning empty collections for an unrecognized mapping. Adopting the
shared resolver rather than repeating the check locally also brings the
'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
from this library's own graph type to YAML, used in
examples/capability_gap_context_graphs_example.py — exports its records
instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
defaulted lookups and had the same defect; it goes through the resolver
too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
its own key set. Schemas are a separate vocabulary with no aliasing, so
_require_recognized_keys lives in this module rather than in the shared
graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
An export_json envelope carries one, so accepting it would readmit the
case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
output directory, so a rejected export leaves nothing behind.
The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.
Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.
Closes#953.
* fix(export): reject payloads whose records resolve to nothing
Addresses the Qodo findings on #958.
Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.
- utils/helpers.py — split the check in two. _require_recognized_keys keeps
the presence rule; _require_nothing_dropped runs after resolution and
refuses a payload that resolved to nothing while an unread key still holds
records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
always carries a populated 'statistics' dict, and an empty graph must stay
exportable, so 'metadata', 'statistics' and 'count' are named as context
rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
both checks through the shared helpers rather than its own copy, so the
two vocabularies cannot drift apart in what counts as a silent-empty
export.
Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.
- export/yaml_exporter.py — the serialization span now says it serialized,
not that it exported, and export() opens its own span around the
filesystem work that stops as failed on error. Nothing reports a completed
export until the bytes are on disk.
Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.
Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.
* fix(export): validate collection values and make Neo4j mappings strict
Two gaps at the boundary the shared normalizer is supposed to own.
_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.
Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.
Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.
* fix(export): close YAML schema and record validation gaps
Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived. Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.
Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects. Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary. Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.
Tests: 101 tests pass across
tests/utils/test_normalize_graph_payload.py
tests/export/test_yaml_exporter_key_recognition.py
tests/export/test_yaml_exporter_input_validation.py
tests/export/test_neo4j_csv_exporter.py
* fix(export): close exception-type and record-shape gaps in normalize_graph_payload
LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).
Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).
normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.
Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.
* docs(changelog): add entry for #958 YAML export input hardening
Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes#956, #952, #953.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Fix What's new → link in Info banner (now a proper <a> tag, always clickable)
- Replace 4-stat CardGroup on index with inline premium stats row
- Convert every <CardGroup>/<Card> block site-wide to markdown bullet lists:
content sections → bold-title bullets with sub-bullets, nav cards → [Title](href) — description
- Add cursor-animated list item hover effects to custom.css:
green inset left border, subtle background tint, marker color change on hover
- Affects index, getting-started, quickstart, concepts, modules, faq, architecture,
installation, cookbook, glossary, learning-more, explorer-setup, cli-setup,
community, contributing-guide, governance, citation, project-license,
all integrations pages, and all 20+ reference module pages
- Move Discord, GitHub, PyPI, and Follow on X links from sidebar anchors to top-right navbar
- Lock dark mode as default via appearance.strict and hide theme toggle
- Add custom.css with hover highlighting for tables, code blocks, cards, callouts, and inline code
- Move all Tips and Common Pitfalls sections inline next to their relevant content across all 25 reference docs
- Polish context.md: remove duplicates, condense callouts, upgrade Cookbooks to CardGroup
- Convert Troubleshooting and Performance Optimization sections in installation.md, cli-setup.md, explorer-setup.md, learning-more.md, and faq.md from plain headers to AccordionGroup
- Change navigation-hint Tip callouts to Info in concepts.md, faq.md, glossary.md, and modules.md
* fix: replace invalid Mintlify theme 'venus' with 'mint'
* docs: replace em dashes with colons across all docs files
* fix: strip UTF-8 BOM from all docs files (broke frontmatter detection)
Replace plain markdown in every docs/reference/ file and docs/concepts.md with
rich Mintlify JSX components — CardGroup, Steps, Tabs, AccordionGroup, Tip,
Warning, Note, and CodeGroup — for a consistent, navigable, production-grade
developer experience.
- Reduced code examples in all guide pages (getting-started, quickstart, concepts, modules, examples, use-cases, learning-more)
- Added comprehensive cookbook links with descriptions (topics, difficulty, time, use cases)
- Improved structure and organization across all guide pages
- Updated use-cases.md to only include use cases with corresponding cookbooks
- Removed 'Last Updated: 2024' from all documentation files
- Enhanced navigation with better 'Next Steps' sections
- Enhanced introduction/15_Export.ipynb with complete module architecture documentation
- Enhanced advanced/05_Multi_Format_Export.ipynb with all export formats and classes
- Removed HTMLExporter references from intelligence notebooks (class doesn't exist)
- Fixed OWLExporter usage in healthcare notebook (removed invalid export_knowledge_graph call)
- Updated all notebooks to use only class imports, no convenience functions
- Added comprehensive documentation for all exporter classes and methods
- Improved markdown structure and learning objectives in both notebooks
- Added grid‑card layouts, icons, and admonitions to all reference pages
- Removed all mkdocstrings ::: directives that caused alias‑resolution errors
- Fixed export module alias issue (RDFExporter) by cleaning the docs
- Updated README‑style sections for better readability
- Added a PowerShell helper script (remove_mkdocstrings.ps1) for future clean‑ups
