mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-07 04:02:40 +00:00
Wei Tao
c25b88c07e
`GET /api/ontology/graph` used to fetch every node and edge for each relevant type with `limit=2**63-1`, normalize all of it, and only then check the result against `_MAX_ANALYSIS_NODES`. So the cap limited the response size, but not the amount of work or memory needed to build it. Using the existing `paginate_nodes`/`paginate_edges` APIs wouldn't help much either. Each call still loads and normalizes the full matching set before slicing out a page, so paging through the results would repeat that cost for every page. This adds `GraphSession.iter_nodes` and `iter_edges`, which yield matches one at a time instead of materializing everything up front. The graph endpoint now applies the ownership filter while scanning, so nodes from other ontologies are dropped before they're normalized or kept in memory. It also stops as soon as the requested ontology's own nodes or selected edges exceed the cap. The ownership check intentionally happens before the cap check. That way, a graph containing a large number of nodes from unrelated ontologies can't make the requested ontology appear too large to open. The old `candidates_by_id` map is gone as well. We now only retain nodes actually owned by the requested ontology instead of building payloads for every schema-typed node in the graph. I also split the endpoint into `_known_ontology_uris`, `_collect_core_nodes`, and `_select_structure_edges` so the main flow is a little easier to follow. External edge endpoints were previously hydrated one at a time with an `await` for each node. With a few thousand selected edges, that meant a few thousand sequential thread dispatches. They're now hydrated concurrently with `asyncio.gather` over `to_thread` calls, using the default thread pool. The final node and edge lists are sorted before building the response, so completion order doesn't affect the output. This isn't fully lazy all the way down. `iter_edges` still gets the full result for an edge type from `ContextGraph.find_edges` before it starts yielding, and `iter_nodes` still snapshots and sorts all node IDs for a type first. A very large type such as `rdf:type` can therefore still do work proportional to its graph-wide size before the cap gets a chance to stop the scan. What this change avoids is the more expensive part: normalizing every schema node in the graph and keeping the full candidate payload map in memory. Making `ContextGraph.find_edges` lazy would address the remaining issue, but that's a lower-level API used elsewhere and needs its own locking design, so that's better handled separately. The old final cap check is removed too. The streaming collectors now raise as soon as their running count exceeds `_MAX_ANALYSIS_NODES`, so by the time collection finishes, `core_node_ids` and `selected_edges` are already guaranteed to be within the limit. There's a comment at that boundary documenting the invariant instead of keeping a redundant check around.