Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

← Home

claude-fable-5.1

Running claude-fable-5.1 inside cursor.

Workflow

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

Every run uses the same steps in the same order. Full workflow definition →

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result

Runs

Speed
Reasoning
Run 5 2026-09-02 · standard · high $4.37 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.08 1m 50s 12 7,092 29
reviewPlan succeeded $0.69 1m 30s 12 6,291 26
execute succeeded $1.27 2m 19s 18 8,749 30
reviewAndCorrect succeeded $0.72 1m 33s 16 4,371 23
openPullRequest succeeded $0.60 2m 03s 30 4,560 13
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 11s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.37
Agent time
9m 15s
Input tokens
88
Output tokens
31,063
Cache-read tokens
1,761,425
Tool calls
121

Tool call mix

read
80
shell
25
edit
12
createPlan
2
glob
2
Run 4 2026-09-01 · standard · high $4.38 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.07 1m 49s 12 7,122 29
reviewPlan succeeded $0.71 1m 28s 12 6,417 26
execute succeeded $1.28 2m 20s 18 8,813 30
reviewAndCorrect succeeded $0.72 1m 32s 16 4,284 23
openPullRequest succeeded $0.60 2m 02s 30 4,630 13
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 11s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.38
Agent time
9m 11s
Input tokens
88
Output tokens
31,266
Cache-read tokens
1,743,644
Tool calls
121

Tool call mix

read
80
shell
25
edit
13
glob
2
createPlan
1
Run 3 2026-09-01 · standard · high $4.37 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.07 1m 48s 12 7,093 29
reviewPlan succeeded $0.70 1m 31s 12 6,365 26
execute succeeded $1.29 2m 18s 18 8,817 30
reviewAndCorrect succeeded $0.72 1m 34s 16 4,328 23
openPullRequest succeeded $0.59 2m 01s 31 4,647 13
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 12s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.37
Agent time
9m 12s
Input tokens
89
Output tokens
31,250
Cache-read tokens
1,732,690
Tool calls
121

Tool call mix

read
80
shell
25
edit
12
createPlan
2
glob
2
Run 2 2026-09-01 · standard · high $4.37 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.09 1m 53s 12 7,119 29
reviewPlan succeeded $0.69 1m 30s 12 6,300 26
execute succeeded $1.27 2m 19s 18 8,576 30
reviewAndCorrect succeeded $0.73 1m 32s 16 4,406 23
openPullRequest succeeded $0.60 2m 04s 31 4,587 13
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 10s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.37
Agent time
9m 17s
Input tokens
89
Output tokens
30,988
Cache-read tokens
1,752,088
Tool calls
121

Tool call mix

read
78
shell
29
edit
10
createPlan
2
glob
2
Run 1 2026-09-01 · standard · high $4.38 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $1.07 1m 52s 12 7,023 29
reviewPlan succeeded $0.69 1m 30s 12 6,194 26
execute succeeded $1.30 2m 21s 18 8,905 30
reviewAndCorrect succeeded $0.73 1m 31s 16 4,417 23
openPullRequest succeeded $0.59 2m 01s 29 4,577 13
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 10s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$4.38
Agent time
9m 15s
Input tokens
87
Output tokens
31,116
Cache-read tokens
1,754,304
Tool calls
121

Tool call mix

read
80
shell
23
edit
14
glob
3
createPlan
1