omp-pareto-models

Pick the best bang-for-buck LLM on OpenRouter, grounded in live leaderboard data instead of vibes — and wire the winner into OMP (modelRoles, prewalk, advisor) so the choice actually costs less.

Install

Requirements: Python 3.8+ (stdlib only — no third-party packages), no API key.

git clone ssh://git@git.mooivancrooy.nl:2222/crooy/omp-pareto-models.git
cd omp-pareto-models
./install.sh

This installs two symlinks (the repo stays the source of truth, so edits here take effect immediately):

• ~/.local/bin/openrouter-pareto → openrouter_pareto.py (on your PATH)
• ~/.omp/agent/skills/openrouter-pareto → skills/openrouter-pareto (OMP skill)

Manual install

ln -sfn "$PWD/openrouter_pareto.py" ~/.local/bin/openrouter-pareto
ln -sfn "$PWD/skills/openrouter-pareto" ~/.omp/agent/skills/openrouter-pareto

Quickstart

./openrouter_pareto.py                                  # GPQA Diamond, blended token price (75/25)
./openrouter_pareto.py -b hle --cost-metric task        # HLE, benchmark-verified $/question
./openrouter_pareto.py --min-context 1m                 # drop sub-1M models before computing the frontier
./openrouter_pareto.py --json                           # machine-readable

Benchmarks: gpqa-diamond, hle, browsecomp, deepsearchqa, tau2-bench-airline, widesearch. No API key needed.

Reading the output

Sort every model by price, keep only the ones that raise the score ceiling (the Pareto frontier), then look at the gap between neighbors: if the next model costs 10x more for 1% more intelligence, it's a pointless upgrade — the table prints that as Marginal $/+1%, and the ⭐ BEST ROI row is where the curve bends (the knee).

Column	Meaning
Config	On search-suite benchmarks: the model's search engine / reasoning-effort configuration (- on model leaderboards)
Ctx	Context window from the OpenRouter models API (1.0M, 262k); --min-context 1m filters on it
Score	Benchmark score (higher = smarter)
Price ($)	Chosen cost basis — blended $/1M tokens, or $/question with --cost-metric task
Δ Score / Δ Price	Gain over the previous frontier row
Marginal $/+1%	Price of the next intelligence point — the pointless-upgrade detector
Sweet Spot	⭐ BEST ROI marks exactly one row: the Kneedle knee

A real run (./openrouter_pareto.py -b deepsearchqa --cost-metric task):

=====================================================================================================================================
 OpenRouter Intelligence vs Cost Pareto Analysis: DEEPSEARCHQA
 Metric: TASK BASIS | Prompt: 75% / Completion: 25%
Model Name                     Config                     Ctx    Score    Price ($)    Δ Score    Δ Price   Marginal $/+1% Sweet Spot
-------------------------------------------------------------------------------------------------------------------------------------
DeepSeek: DeepSeek V4 Flash 0  parallel/high             1.3M    51.5%    $0.0226/q          -          -         Baseline
DeepSeek: DeepSeek V4 Flash 0  perplexity/high           1.3M    69.0%    $0.0635/q     +17.5%    +$0.041           $0.002 ★ EFFICIENT
OpenAI: GPT-5.6 Luna           perplexity/xhigh          1.1M    73.0%    $0.1030/q      +4.0%    +$0.040           $0.010 ⭐ BEST ROI
OpenAI: GPT-5.6 Sol            perplexity/high           1.1M    73.6%    $0.5691/q      +0.6%    +$0.466           $0.839
-------------------------------------------------------------------------------------------------------------------------------------
👉 Recommended Sweet Spot: OpenAI: GPT-5.6 Luna (73.0% score @ $0.103)
   Max score gain per dollar before steep diminishing returns kick in.

Example: the wiring the skill hands you

The run above is only half the job — the point is to turn the knee into OMP role settings. Below is the cutover the skill produced on the author's machine on 2026-09-11, from live GPQA-Diamond, tau2-bench-airline and search-suite data. It is illustrative, not a recommendation: slugs and prices go stale within weeks, so re-run the tool and re-apply the mapping in docs/omp-model-roles.md before reusing any row.

Three rules decided the picks:

1. Two axes plus a real cost basis — and the role decides which axis ranks. A capability pick must clear the knowledge column (GPQA) and the agentic/tool-use column (tau2), compared per task, not per token. GPQA alone promotes benchmark-overfit models that collapse once they have to use tools; $/1M alone hides thinking-token burn — a small model can win the price column 10x and lose on $/task. Agency ranks only where the role executes (slow, task); for plan and advisor — whose tools are read/grep/glob, not multi-turn orchestration — intelligence ranks and tau2 is a floor. Rank on marginal $ per point, and compare -pro/:high variants before ranking across families: gpt-5.6-luna and -pro cost the same, with -pro strictly better on both axes.
2. Role intent first. slow must out-rank default (it fixes what the other roles are stuck on) and plan may be the same model as slow — pay for capability above default, not for the frontier price tier; advisor must be a different model family than default, or the review inherits the author's blind spots.
3. A ~1M context floor, except smol. Role picks need 1.0M+ tokens (--min-context 1m). At the top this is free — the frontier models here are 1.05M (gemini-3.7-flash) and 1.31M (glm-5.3-flash) — but it does exclude real mid-tier candidates: grok-4.6 (500k), kimi-k2.7-code (262k), glm-5 (204k), pre-4.6 Anthropic (claude-sonnet-4 and claude-opus-4.5 at 200k — the 4.6+ generation is 1.0M), and the cheap band (gpt-oss-20b 131k, gemma-4-26b 262k). smol is exempt: prewalk handoff and background work buy on price, and 128k is fine there.

Role	Model	Live evidence	Rationale
default	zai/glm-5.3-flash :floor	75.3% tau2, $0.0057/task at openrouter prices — $0 marginal under the z.ai subscription	z.ai subscription covers direct zai; fallback chain zai/glm-5.3-flash :auto → openrouter deepseek-v4-flash :auto
slow	gemini-3.7-flash :high	94.3% GPQA, 80.6% tau2 (best of 123), $1.50/1M	escalation headroom; must out-rank default
plan	gemini-3.7-flash :high	94.3% GPQA, 80.6% tau2, $1.50/1M	same model as slow: gemini-3.1-pro-preview cost 3x for +0.1% GPQA and sat outside the tau2 frontier
task	zai/glm-5.3-flash :floor	75.3% tau2, $0.0057/task — subagents burn the most tokens	z.ai subscription covers direct zai; fallback chain zai/glm-5.3-flash :auto → openrouter qwen3-coder :auto
commit	deepseek-v4-flash :high	75.1% tau2, $0.085/1M	trivial spend, real quality delta
vision	deepseek-v4-flash-vision-exp :auto	88.2% GPQA	actual image input; a text-only pick breaks vision silently
tiny	gemma-4-26b-a4b-it :auto	73.9% GPQA, 68.3% tau2, $0.087/1M	replaces qwen3-30b-a3b-instruct-2507 (63.9% / 37.5%, $0.14/1M): cheaper per token, a wash per task, higher on both axes
smol	zai/glm-5.3-flash :floor	same model as default: 75.3% tau2, $0.0057/task	z.ai subscription covers direct zai; fallback chain zai/glm-5.3-flash :auto → openrouter gpt-oss-20b :auto
advisor	glm-5.3-flash :high	73.3% tau2, $0.0048/task	different family from default → uncorrelated review; cheapest model in the set per real task

modelRoles is one settings record, so omp config set modelRoles.<role> is rejected ("Unknown setting"). Set roles in the /models Roles view or paste the block into ~/.omp/agent/config.yml (selectors are provider-prefixed) — the block below is the 2026-09-11 record, so re-derive it before pasting:

modelRoles:
  default: zai/glm-5.3-flash:floor
  slow: openrouter/z-ai/glm-5.3-flash:auto
  plan: openrouter/z-ai/glm-5.3-flash:floor
  task: zai/glm-5.3-flash:floor
  commit: openrouter/deepseek/deepseek-v4-flash:floor
  vision: openrouter/google/gemma-4-26b-a4b-it:floor
  tiny: openrouter/openai/gpt-oss-20b:floor
  smol: zai/glm-5.3-flash:floor
  advisor: openrouter/openai/gpt-oss-120b:floor

Docs

Path	What
openrouter_pareto.py	The whole tool: scrapes OpenRouter benchmark leaderboards + pricing API, computes the Pareto frontier, marginal $ per +1% score, and the Kneedle knee.
skills/openrouter-pareto/SKILL.md	Agent skill wrapping the tool: how to run it, read it, and map the result onto OMP model-role wiring.
docs/methodology.md	The math: Pareto dominance, marginal cost per intelligence point, knee detection, cost bases, and where the data comes from.
docs/omp-model-roles.md	OMP wiring: the 9 modelRoles, prewalk handoff, advisor config, plus an example cutover record.

Checks

python3 -m py_compile openrouter_pareto.py
python3 test_openrouter_pareto.py      # offline: context-size parsing/formatting
python3 openrouter_pareto.py --help
./openrouter_pareto.py --json          # live run against openrouter.ai

License

MIT — see LICENSE.