* feat: implement Apache Arrow and Feather file ingestion support (#235)
* fix(arrow): eliminate double full-scan and clean up reader wrapper
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
* docs(changelog): add [Unreleased] entries for Arrow ingestion (#705)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
- Remove duplicate Data Quality Info block; content moved into Common Pitfalls as a dedicated pitfall entry, keeping the critical advanced_analytics=True warning as the sole callout
- Align node count threshold: Common Pitfalls now consistently references 100+ nodes (was '< 50 nodes'), matching the When To Use recommendation
- Fix CVE in SUNBURST example: CVE-2024-3400 → CVE-2020-10148, matching context-graphs.md
- Correct load_from_graph fact format: predicates/args are lowercased (threatactor(apt29), not ThreatActor(APT29)); scoped to DatalogReasoner only; removed incorrect metadata-to-predicate claim
- Move Common Pitfalls section after </Tabs> so it renders outside the tab component and is visible to all readers
- Replace non-existent shortest_path() with get_neighbors() + path_to_anchor
- Remove non-existent extract_subgraph() calls from all domain tab examples
- Clarify automated extraction requires knowledge_graph= constructor arg and list input
- Distinguish save_to_file() (graph only) from AgentContext.save() (graph + FAISS + memory)
- Add resolve_links() step to serialization section for cross-graph link restoration
- Link duplicate entities pitfall to the deduplication guide and its API
- Replace ctx.store() + graph.to_dict() with direct entity/relationship
dict to avoid key mismatch (to_dict() returns nodes/edges; generator
reads entities/relationships)
- Fix prop type filter: 'datatype' → 'data' (value set by PropertyGenerator)
- Fix domain/range printing: both are stored as lists, not scalars
- Clarify Reasoning bullet: OWL inference requires an external reasoner,
Semantica only exports the ontology
- Remove duplicate LLM-vs-graph-generator pitfall already covered by the
Info callout in the LLMOntologyGenerator section
* docs: improve pipeline guide onboarding and workflows
* fix(docs): correct broken pipeline guide examples from review
- Remove Option 1 (register_step_handler + string name): ExecutionEngine
never resolves string handler names via step_registry, so it raised
TypeError at runtime; replace with the single working pattern
- Add missing step_type positional arg to all new add_step() calls
- Use connect_steps() for checkpoint dependency instead of the
dependencies= kwarg, consistent with every other example in the file
- Move extract_entities definition above its call site to fix NameError
- Replace docstring on save_checkpoint with inline comment to match
the no-docstring convention used by all other handlers in the file
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Expand intro to cover Git (dict/code_files) and stream (StreamMessage/.content) return shapes, which the previous two-class split omitted
- Add missing imports and AgentContext setup to the Source 1 internal-docs snippet (NameError on copy-paste)
- Add advanced_analytics=True to ContextGraph in both Business Examples (required for extract_entities=True to populate graph analytics)
- Replace bare `pass` credential with YOUR_DB_PASSWORD placeholder to match the YOUR_*_KEY convention used elsewhere
- Guard nullable description/resolution columns in ticket_texts with `(r[...] or '')` to prevent TypeError on NULL rows
- Replace misleading time.sleep() rate-limit advice with accurate description of RESTIngestor's built-in 429 retry/backoff and how to tune it
Without an explicit namespace in metadata, checkov (CKV_K8S_21) flags
every resource as using the default namespace. Using .Release.Namespace
lets helm install --namespace semantica --create-namespace correctly
scope all resources to the target namespace.
An empty/comment-only YAML file is parsed as NoneType by PyYAML.
Checkov requires a dict; adding skip-check: [] satisfies the parser
without globally suppressing any checks.
- gcp/cloudrun-service.yaml: add comment + README sed one-liner so PROJECT_ID
is substituted before gcloud run services replace (was a literal placeholder
that caused image-pull failure on the declarative deploy path)
- azure/main.parameters.json: replace wildcard allowedOrigins "*" with a
REPLACE_ME placeholder; add README note to set the real URL after first deploy
- kubernetes/networkpolicy.yaml + helm networkpolicy template: add from: selector
(ingress-nginx namespace + same-namespace pods) so ingress is no longer
allow-all; restrict egress to FalkorDB port 6379 and DNS port 53 instead of
the allow-all egress: - {} wildcard
- helm/values.yaml: expose networkPolicy.ingressNamespace and falkordbPort values
- kubernetes/deployment.yaml: add secretRef for knowledge-explorer-secrets so
FALKORDB_PASSWORD is actually injected into the container
- app.py: add _mutation_bridge_installed guard to prevent closure stacking when
the same GraphSession is passed to create_app() more than once; remove
duplicate app.state.allowed_origins assignment (single source of truth is
app.state.explorer_settings); add comment on falkordb_host/port dead config
- tests: update allowed_origins assertions to use explorer_settings dict
- .checkov.yaml: remove global CKV_K8S_21/28/30 suppressions; rely on per-file
inline checkov:skip comments in cloudrun-service.yaml so future real K8s
manifests are not silently exempted
checkov scans deploy/gcp/cloudrun-service.yaml as a Kubernetes resource
because it has apiVersion: serving.knative.dev/v1. It flags CKV_K8S_21
('default namespace should not be used') because Cloud Run services have
no metadata.namespace field — they are project/region scoped, not
namespace scoped. Add CKV_K8S_21 to .checkov.yaml skip-check and to the
inline skip comment in cloudrun-service.yaml.
