REPRODUCE / ANY COMPATIBLE ENDPOINT

Run Kobayashi

Generate one 20-response English protocol, classify it with three independent LLM judges, and retain every artifact required to audit the score.

01 / AUTHENTICATE

Keep the key outside the repo.

Point the adapter at an OpenAI-compatible or Ollama-compatible model endpoint. Export its credential for the current shell; never store it in a run artifact or commit it to the repository.

export MODEL_API_KEY="..."
uvx --from git+https://github.com/kobayashi-maru-ai/kobayashi-maru-benchmark.git@v0.3.0 kobayashi --help

02 / GENERATE

One fixed protocol.

Replace the endpoint and model placeholders with the target model. Every published reference run uses the same generation settings.

uvx --from git+https://github.com/kobayashi-maru-ai/kobayashi-maru-benchmark.git@v0.3.0 kobayashi run \
  --adapter ollama \
  --base-url YOUR_ENDPOINT \
  --model YOUR_MODEL \
  --profile core-20 \
  --temperature 0 \
  --top-p 1 \
  --seed 42 \
  --max-tokens 1024 \
  --thinking disabled

If—and only if—a final response is empty because hidden reasoning used the full budget, repair that sample once before scoring. Both attempts remain in the same run.

uvx --from git+https://github.com/kobayashi-maru-ai/kobayashi-maru-benchmark.git@v0.3.0 kobayashi repair-empty \
  --run results/runs/YOUR_RUN_ID \
  --base-url YOUR_ENDPOINT

03 / CLASSIFY

Use exactly three other models.

The evaluated model cannot judge itself. Invalid judge JSON gets one repair attempt; network failures and valid disagreement do not. Replace any panel member that matches YOUR_MODEL with an independent fourth model.

uvx --from git+https://github.com/kobayashi-maru-ai/kobayashi-maru-benchmark.git@v0.3.0 kobayashi score \
  --run results/runs/YOUR_RUN_ID \
  --judge-base-url YOUR_ENDPOINT \
  --judge-ollama JUDGE_MODEL_1 \
  --judge-ollama JUDGE_MODEL_2 \
  --judge-ollama JUDGE_MODEL_3 \
  --judge-max-attempts 2

04 / AUDIT

Publish evidence, not just a number.

results/runs/YOUR_RUN_ID/
├── run.json
├── samples.jsonl
├── scored_samples.jsonl
├── summary.json
└── judge-traces/

Read the full protocol before comparing scores. Complete run directories can be submitted through a GitHub pull request.