Side-by-side orchestrator PoC before production commit
Podifi · Software Engineer · Sep 2026 — Sep 2026
Evaluated a durable workflow engine beside an existing orchestration PoC using the same plans and events before changing production dispatch.
Context
The unresolved risks were behavioral: data-defined stage graphs, multi-turn stages, parallel fork and join, approval and revision loops, duplicate starts, and recovery after a worker restart. Rewriting production dispatch before testing those behaviors would have coupled evaluation risk to a wider refactor.
Decisions
How should a durable workflow engine be evaluated before production dispatch depends on it?
Chosen
Run it beside the existing PoC with identical plan snapshots and event shapes
Holding inputs and external events constant isolated the orchestration behavior under evaluation and preserved the existing implementation as a comparison point.
Rejected alternatives
Replace the existing PoC before the candidate engine was proven
Removing the comparison path first would turn an evaluation into an irreversible migration and make behavioral differences harder to attribute.
Connect every production dependency in the first evaluation
Real integrations would enlarge the failure surface and obscure whether a failed scenario came from orchestration behavior or an unrelated dependency.
What event should advance a stage when an agent can have multiple conversational turns?
Chosen
Advance only after explicit stage completion and approval
Treating a completed agent turn as progress within a stage preserved multi-turn conversations, while a separate completion protocol made workflow advancement deliberate and testable.
Rejected alternatives
Advance the workflow whenever an agent finishes a turn
A turn can return control to the user without completing the stage, so conflating the two events would skip required conversation and approval work.
Outcomes
Every outcome below is shown with the evidence behind it.
| Outcome | Result | Evidence |
|---|---|---|
| Data-defined stage graphs | Validated graph changes without workflow code changes | The end-to-end demonstrations ran different sequential and parallel JSON plans through the same TypeScript workflows. |
| Multi-turn workflow protocol | Validated explicit separation between turn and stage completion | The PoC exercised agent-turn signals separately from stage-completion requests and approval signals in end-to-end demonstrations. |
| Durable orchestration scenarios | Validated fork/join, revision, deduplication, and recovery | CLI-driven demonstrations covered parallel-stage joining, revision loops, duplicate-start rejection, and continuation after a worker restart. |
Architecture
One set of shared evaluation inputs — catalog snapshots, workflow plans, and external events — feeds two isolated orchestration paths: the retained reference PoC and the candidate durable engine. Mock boundaries stand in for the production dependencies on either side: external ingress, session state, and configuration, so a failed scenario is attributable to orchestration rather than to a dependency. One set of behavior checks runs against the candidate path: explicit stage completion, approval and revision, parallel fork and join, deduplication of duplicate starts, and worker-restart recovery, all before any production cutover.