Benchmarks
Measured, not claimed.
Release build on macOS arm64 · 10 cores · 16 × ~2047 KB files. All scenarios are offline — no network, no LLM — so they run in CI.
Tool dispatch overhead
How much machinery sits between the LLM's tool call and the actual work.
| Measurement | Iterations | Per-call |
|---|---|---|
| registry.execute (raw dispatch) | 300 | 5140 µs |
| execute_batch, 1 call | 300 | 7614 µs |
| execute_batch, 8 calls (amortized) | 320 | 5893 µs |
| ToolCall args serde round-trip | 100000 | 752 ns |
Executor overhead over raw dispatch: 2473 µs per single-call batch; amortized overhead drops to 753 µs per call in an 8-call batch.
Parallel vs sequential — I/O-bound
16 × file_read of distinct ~2047 KB files.
16/16 calls succeeded (join_all), 16/16 (spawn); all classified parallel-safe.
Parallel vs sequential — CPU-bound
8 × parse_html of a ~1 MB table (3000 rows), selector `tr.item`, texts mode.
join_all shares one thread — spawn spreads CPU work across cores.
Mixed batch — automatic partitioning
A realistic agent turn: reads (parallel-safe) + writes (sequential) in one batch. 5 ran concurrently, 3 serialized; total 70.9 ms, 8 succeeded. Order preserved.
| Tool | Phase | Success | Duration |
|---|---|---|---|
| file_read (r0) | parallel | ✅ | 16 ms |
| file_read (r1) | parallel | ✅ | 20 ms |
| file_read (r2) | parallel | ✅ | 25 ms |
| file_read (r3) | parallel | ✅ | 29 ms |
| file_write (w0) | sequential | ✅ | 2 ms |
| file_write (w1) | sequential | ✅ | 0 ms |
| file_write (w2) | sequential | ✅ | 0 ms |
| grep (g0) | parallel | ✅ | 65 ms |
parse_html scaling with document size
Same selector (`tr.item`, texts mode), increasing document sizes. Up to 531k rows/s.
| Document | Rows | Avg parse | Throughput |
|---|---|---|---|
| 6 KB | 100 | 0.19 ms | 531,915 rows/s |
| 63 KB | 1000 | 4.08 ms | 245,339 rows/s |
| 191 KB | 3000 | 9.55 ms | 314,268 rows/s |
| 773 KB | 12000 | 34.22 ms | 350,723 rows/s |
extract_json throughput
~4 MB JSON document with 20,000 objects. Stateless & parallel-safe (re-parsed per call).
| Query | Iterations | Avg |
|---|---|---|
| wildcard scan `items[*].value` (limit 500) | 10 | 71.24 ms |
| deep single key `items.12345.meta.score` | 10 | 67.78 ms |
| top-level key `total` | 10 | 43.47 ms |
web_feed (quick-xml) — scaling & parallelism
Local RSS fixture; tolerance to feed size and CPU-bound speed under parallelism. Up to 1.1M items/s.
| Feed items | Avg parse | Items/s |
|---|---|---|
| 1000 | 0.93 ms | 1,077,586 |
| 5000 | 4.78 ms | 1,045,588 |
| 12000 | 10.81 ms | 1,110,391 |
| 20000 | 27.10 ms | 738,089 |
8 × web_feed of a ~5 MB feed; 8/8 succeeded both modes.
code_symbols / repo_map — symbol extraction
240 Rust files (~14 KB each, 40 fns + 40 structs + impls per file).
| Tool | Mode | Wall time | Items |
|---|---|---|---|
| code_symbols | single | 7.2 ms | 7007 lines |
| repo_map | single | 34.2 ms | 4330 lines |
8/8 symbol scans succeeded (join_all), 8/8 (spawn).
Semantic memory — absorb / search / digest
Offline TF-IDF embedder (no network, no LLM); in-memory SQLite.
| Batch | Absorbed | Wall time | Per fact |
|---|---|---|---|
| 10 | 10 | 10.2 ms | 1020 µs |
| 100 | 89 | 9.4 ms | 94 µs |
| 500 | 335 | 82.6 ms | 165 µs |
Re-absorbing 100 known facts: 93 skipped, 0 created in 5.1 ms (dedup fast path).
| Hybrid search (vector + BM25) | Matches | Median |
|---|---|---|
| Acme revenue units filing | 5 | 2.30 ms |
| Globex churn rate report | 5 | 1.62 ms |
| headcount in Kazan during 2022 | 5 | 2.23 ms |
| Initech margin units | 5 | 1.95 ms |
| support load Dubai filing | 5 | 1.66 ms |
| Store size | Fill time | Search (median) |
|---|---|---|
| 1000 memories | 53 ms | 5.28 ms |
| 5000 memories | 926 ms | 23.52 ms |
| 10000 memories | 828 ms | 47.47 ms |
Digest build (relevant + TODOs + recent): 4 relevant memories in 4.76 ms.
Reproduce
cargo build --release
./target/release/parallel-research bench --scenario all > bench.md
# individual scenarios: dispatch | parallel-io | parallel-cpu | mixed |
# parse-scale | extract-json | feed-parse | code-map | memory
./target/release/parallel-research stats -o <output-dir> # p50/p95 per toolBenchmarks use no network and no LLM (offline TF-IDF embedder) — runnable in CI. Numbers above are a snapshot on macOS arm64 (10 cores), release build.
Field notes: 31× on repo_map
Benchmarks are how we catch regressions — and how we find wins. Profiling repo_map revealed a silently-serialized regex pass; moving it to a OnceLock-cached compiler took it from 902 ms to 29.5 ms (31×). The same audit unlocked 3.12× / 5.16× on web_feed and code_symbols under execute_batch_spawn.