The benchmark that caught a 31× slowdown

A benchmark is an instrument, not a showcase

The LLM decides what to do; the tools do the work. In between sits the ToolExecutor — parallel-safe vs sequential classification, path-conflict detection, cascading cancellation, and the serde machinery around every ToolCall. This layer runs on every step of every agent, so its characteristics directly define how long a research run takes.

That is why the harness is built into the product. Fixtures are created automatically in a temp directory and removed after; each scenario repeats 5–10 times with a warm-up call, and the numbers below are one run on macOS, 10 cores, release build. No network, no LLM — the memory scenario uses an offline TF-IDF embedder, so the whole suite can run in CI.

31× from a regex cache

The first version of extract_symbols compiled two regular expressions (regex::Regex::new) per file. For code_symbols with its limit that was tolerable — the loop stops after the first ~25 files. For repo_map, which honestly walks the whole tree, it was a catastrophe: DFA compilation dominated the runtime.

The fix moved the regexes into OnceLock statics (compiled once per process) and parallelized file reads in repo_map via tokio::spawn + join_all, preserving order. An A/B run over 240 synthetic Rust files:

That is a 31× improvement on a debug build; in release the same tree takes 6.6 ms. The live reference point: before the fix, repo_map over this very project's repository (103 files) took 106 ms in release. After the fix — single-digit milliseconds.

The silent serialization

The second catch was subtler. ToolExecutor splits tools into parallel-safe and sequential; a tool missing from the classification goes sequential "just in case". When web_crawl, web_feed, code_symbols and repo_map landed, nobody added them to the classification — and any batch of several such calls silently ran one at a time, with no errors and no warnings.

The harness showed it in numbers: a spawn batch of 8 × web_feed produced 1.00× — exactly what sequential execution produces — while parse_html under the same conditions produced ~3×. After adding the tools to parallel_safe (and a classification test to keep them there):

Every new read-only tool must ship with a classification test — otherwise parallelism disappears without a trace.— Parallel Research, benchmark notes

The rest of the scoreboard

The same run measures the machinery around the tools. Serializing a ToolCall's arguments costs ~750 ns; the executor's overhead over raw registry dispatch is ~2.5 ms for a single-call batch and amortizes to 753 µs per call in an 8-call batch. The execution layer disappears against the work the tools do.

For CPU-bound batches the difference between execution modes is dramatic: 8 × parse_html of a ~1 MB table runs 3.78× faster under execute_batch_spawn (tokio tasks spread across cores) than sequentially, while join_all — which polls futures on one thread — only helps for I/O waits. HTML selector throughput peaks at ~531k rows/s on small documents (~350k rows/s on large ones), and the quick-xml RSS parser sustains ~1.1M items/s.

A realistic mixed turn — four reads, three writes and a grep in one batch — is partitioned automatically: five calls run concurrently, three serialize, and the result vector still matches the original call order. Total wall time: 70.9 ms.

Run it in CI

Everything above reproduces with one command — parallel-research bench --scenario all — with fixtures that create and clean themselves. The scenarios: dispatch, parallel-io, parallel-cpu, mixed, parse-scale, extract-json, feed-parse, code-map and memory. Because nothing touches the network and nothing calls an LLM, the suite runs in CI on every build — which is exactly how the next silent regression gets caught.

And because synthetic numbers are only half the story, parallel-research stats reads the SQLite tracing of a real session and reports per-tool p50/p95 durations and the batching coefficient from production runs.