Orchestrator-Workers and Evaluator-Optimizer Patterns
On this page
Learning Objective
Implement the two dynamic patterns that sit between workflows and full agents: an orchestrator that plans subtasks at run time and delegates them to workers, and an evaluator-optimizer loop in which one model critiques and another revises until a quality bar is met.
Concept Explanation
The workflows from Day 6 assume you know the subtasks in advance. Many real tasks do not allow that. Updating an API across a codebase touches a number of files you only discover by looking. Researching a market question needs a set of searches that depends on the question. The orchestrator-workers pattern handles this: an orchestrator model reads the task, decides what subtasks are needed, sends each to a worker, and synthesizes the results. The control flow is still mostly yours, but the plan is produced by the model at run time.
Orchestrator-workers differs from parallel sectioning in one important way: the subtasks are not predefined. That flexibility comes with new failure modes. The orchestrator can produce too many subtasks, overlapping ones, or vague ones that workers interpret differently. Good orchestrator prompts therefore require each subtask to have a clear objective, an expected output format, the tools or sources to use and the boundaries of the work. Anthropic's write-up of its multi-agent Research feature found that early versions spawned fifty subagents for simple queries and duplicated searches, and that teaching the lead agent to scale effort to the question's complexity fixed it.
Workers usually run in fresh contexts. That is the point: each gets only its brief, does focused work, and returns a compact result, so the orchestrator's context stays small. It also means workers cannot see each other's progress, so the orchestrator must pass any shared facts explicitly and merge conflicting results.
The evaluator-optimizer pattern adds a feedback loop. A generator produces a draft. An evaluator, with its own prompt and often clear criteria, scores the draft and explains what is wrong. The generator revises using that feedback, and the loop repeats until the evaluator passes it or a maximum number of rounds is reached. This works when two conditions hold: there are articulable criteria for "good", and a model can reliably judge them. Translation with tone requirements, code that must pass tests, and search results that must cover every part of a question are good fits.
The evaluator is most reliable when it checks objective signals. Running the code's tests, validating JSON against a schema or checking that every required section exists is better than asking a model whether the output "seems good". When only subjective judgment is available, give the evaluator a rubric with specific criteria and ask for a verdict on each, not one overall score.
| Pattern | Who plans | Loop | Good fit |
|---|---|---|---|
| Orchestrator-workers | Orchestrator model, at run time | Plan, delegate, synthesize | Multi-file changes, research with unknown scope |
| Evaluator-optimizer | Fixed: generate then evaluate | Revise until pass or round limit | Translation, code with tests, reports with a rubric |
| Combined | Orchestrator plus per-worker evaluation | Nested loops | Research reports where each section is verified |
Worked Example: A Code Generator with a Test-Running Evaluator
The evaluator here is objective: it runs the tests. The model only sees failures it must fix, and the loop stops on success or after three rounds.
import subprocess, tempfile, pathlib
import anthropic
client = anthropic.Anthropic()
def generate(spec: str, feedback: str | None, previous: str | None) -> str:
prompt = f"Write a Python module implementing:\n{spec}\nReturn only code."
if feedback:
prompt += f"\n\nYour previous attempt:\n{previous}\n\nIt failed these tests:\n{feedback}\nFix it."
msg = client.messages.create(model="claude-opus-5-5", max_tokens=8000,
messages=[{"role": "user", "content": prompt}])
text = "".join(b.text for b in msg.content if b.type == "text")
return text.removeprefix("```python").removesuffix("```").strip()
def evaluate(code: str, tests: str) -> tuple[bool, str]:
with tempfile.TemporaryDirectory() as d:
pathlib.Path(d, "solution.py").write_text(code)
pathlib.Path(d, "test_solution.py").write_text(tests)
run = subprocess.run(["python", "-m", "pytest", "-q", "--tb=short"], cwd=d,
capture_output=True, text=True, timeout=60)
return run.returncode == 0, run.stdout[-3000:] # only the tail goes back
def evaluator_optimizer(spec: str, tests: str, max_rounds: int = 3) -> str:
code, feedback = None, None
for round_no in range(1, max_rounds + 1):
code = generate(spec, feedback, code)
passed, feedback = evaluate(code, tests)
print(f"round {round_no}: {'pass' if passed else 'fail'}")
if passed:
return code
raise RuntimeError(f"Still failing after {max_rounds} rounds:\n{feedback}")The tests are written by a person or in a separate step, never by the generator in the same loop, otherwise the generator can make them pass trivially. In production this loop runs in a sandbox, because it executes code the model wrote.
Real-World Examples
- Anthropic's June 2025 engineering post on its multi-agent Research system describes an orchestrator-worker design in which a lead agent plans and spawns subagents that search in parallel, and reports that the multi-agent setup outperformed a single agent by about 90 percent on its internal research evaluation while using several times more tokens.
- AlphaCode and later code generation systems used test execution to filter and rank candidate programs, the same idea as an evaluator that runs tests instead of asking a model for an opinion.
Review Questions & Answers
In sectioning the subtasks are fixed in your code. In orchestrator-workers a model decides at run time which subtasks are needed, based on the input, and then synthesizes the workers' results.
A clear objective, the expected output format, which tools or sources to use, and the boundaries of the work, so that workers do not duplicate each other or interpret a vague task differently.
When there are clear criteria for a good result and the evaluator can judge them reliably, ideally with objective checks such as running tests or validating a schema rather than a vague overall quality opinion.
The generator could write tests that its own code trivially passes, so the loop would report success without the code actually meeting the specification. Tests must come from an independent source.