mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
cxzg007and江俊杰
cce5ea177c
PipelineBuilder.set_parallelism() validated and stored a level in
pipeline config, but ExecutionEngine._execute_steps() had no parallel
code path and no code ever read it back, so steps always ran strictly
sequentially regardless of the configured value. parallelism was also
lost across a serialize/deserialize round trip, since the nested
config key was never promoted to the top-level dict build_pipeline()
reads.
Steps are now grouped into dependency layers (declaration order
preserved within each layer). A layer runs concurrently, bounded by
ThreadPoolExecutor(max_workers=min(configured parallelism, engine
max_workers)), only when every one of the following holds: more than
one step in the layer, the shared input is a dict, every step is
opted in via the new PipelineStep.parallel_safe flag, and no step is
in delta_mode. Any layer that doesn't meet all four falls back to the
existing sequential path unchanged.
Each step's input is deep-copied before any handler in the layer
starts, so concurrent steps never share mutable state. Layer results
are merged back in declaration order, not completion order; keys
whose value is unchanged from the shared input are treated as an
echo rather than a write, so two handlers both returning {**data, ...}
don't spuriously conflict on keys neither of them actually touched.
Genuinely conflicting values for the same key raise ProcessingError
naming both the key and the two steps involved. Retry policy, step
status, result/error tracking, and progress reporting are shared
between the sequential and parallel paths so behavior stays identical
either way. On step failure, not yet started futures in the same
layer are cancelled and the error propagates, so no downstream layer
ever runs.
parallel_safe is opt-in per step because handlers that share mutable
state or depend on strict ordering are not safe to run concurrently.
ParallelismManager.execute_pipeline_steps_parallel() is deliberately
not reused here; the engine implements its own bounded layer
scheduler so retry/status semantics stay identical between the
sequential and parallel code paths instead of diverging.
fix(pipeline): address qodo review findings on PR #1226
- detect circular/unknown dependencies in parallel grouping
(ValidationError instead of RecursionError/KeyError)
- skip unchanged echoed keys in parallel result merging to
avoid false conflicts
- fail before COMPLETED status when a parallel step returns a
non-dict; never retry such contract violations
- require strict bool parallel_safe in builder and engine gate
- make per-step progress tracking IDs unique across same-type
parallel steps
---------
Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>