BatchLab

npx --yes github:RRXXZZYY/batchlab demo --out batchlab-demo.html

Open batchlab-demo.html. The workload is synthetic, the report is self-contained, and no GPU, model, API, or telemetry service is contacted.

“Continuous batching is faster” hides several engineering decisions. A request can enter when a slot opens, but its prefill may pause decode. A short request can reduce TTFT while increasing another sequence's completion time. Reserving KV for full output avoids overcommit but can leave capacity unused.

BatchLab makes those assumptions explicit and lets you replay the same arrivals under two policies:

• Static batch drain: admit a batch only when the current batch is empty.
• Continuous admission: admit waiting requests whenever slots and reserved KV capacity are available between decode steps.

The simulator reports per-request queue time, time to first token (TTFT), total latency, and lifecycle; aggregate p50/p95 latency and TTFT; simulated output rate; busy time; average decode batch; and peak KV reservation.

batchlab simulate examples/bursty-chat.json --scheduler continuous
batchlab compare examples/bursty-chat.json
batchlab compare examples/bursty-chat.json --format html --out comparison.html

The comparison terminal output is deliberately labeled:

SIMULATION ONLY — not a hardware benchmark
metric                    static       continuous       delta (cont-static)
p95 latency ms                39.2          34.3       -4.9
p95 TTFT ms                   34.3          19.8       -14.5
output tok/s                 7,846       9,715.9       +1,869.8

These are reproducible outputs of the bundled synthetic scenario, not claims about a real model or accelerator.

{
  "schemaVersion": "1.0",
  "name": "chat-burst",
  "config": {
    "maxBatchSize": 8,
    "maxKvTokens": 16000,
    "prefillTokensPerMs": 120,
    "decodeStepsPerMs": 8,
    "policy": "fifo"
  },
  "requests": [
    { "id": "r1", "arrivalMs": 0, "promptTokens": 900, "outputTokens": 80 },
    { "id": "r2", "arrivalMs": 4, "promptTokens": 120, "outputTokens": 24 }
  ]
}

policy is fifo or shortest-prompt. KV reservation is conservative: promptTokens + outputTokens is reserved at admission. Requests larger than the total KV budget are rejected and reported; they do not block valid work.

Read the full scenario schema.

Synthetic scenarios are useful as architecture regression tests:

batchlab simulate capacity.json --scheduler continuous --fail-p95-over 250

Or use the action:

name: Inference policy check
on: [pull_request]

jobs:
  capacity:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: RRXXZZYY/batchlab@v0.1.0
        with:
          scenario: test/scenarios/chat-burst.json
          scheduler: continuous
          fail-p95-over: 250

This proves that a policy/configuration still satisfies your simulated contract. It does not prove production capacity.

The 0.1 engine intentionally chooses clarity over hardware fidelity:

1. Events are deterministic and single-device; there is no wall-clock noise.
2. Prefill duration is total admitted prompt tokens divided by prefillTokensPerMs.
3. One decode step emits one token per active sequence and lasts 1 / decodeStepsPerMs.
4. A continuous-admission prefill pauses decode.
5. KV reserves prompt plus requested output through sequence completion.
6. Network, tokenization, sampling, kernels, memory bandwidth, parallelism, prefix caching, and model quality are outside the model.

Because of assumption 3, larger batches increase aggregate simulated token rate. That is a teaching model, not a throughput curve fitted to hardware. See the model specification before interpreting a chart.

• Zero runtime dependencies, telemetry, and network requests.
• Maximum 5 MiB scenario, 2,000 requests, 2,000,000 total tokens, and 250,000 output tokens.
• Unique request IDs and bounded numeric inputs are validated before simulation.
• All events run sequentially; batch and KV invariants are checked in tests.
• Standalone HTML escapes scenario-controlled text and uses no remote assets.
• JSON includes the full assumption list beside every result.

import { compareSchedulers, parseScenario, simulate } from "batchlab";

const scenario = parseScenario(await fs.readFile("scenario.json", "utf8"));
const continuous = simulate(scenario, "continuous");
const comparison = compareSchedulers(scenario);

npm ci
npm test
npm run test:coverage
npm run pack:check

Node.js 20 or newer is required. Contributions are welcome under the MIT license.