Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

gpt-5.6-sol

Running gpt-5.6-sol inside cursor.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-09-01 · standard · medium $2.03 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.30 1m 50s 11 3,110 27
reviewPlan succeeded $0.26 1m 21s 7 3,053 10
execute succeeded $0.60 2m 06s 65 5,405 38
reviewAndCorrect succeeded $0.51 1m 51s 53 3,890 41
openPullRequest succeeded $0.36 1m 11s 37 2,723 23
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 24s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.03
Agent time
8m 19s
Input tokens
173
Output tokens
18,181
Cache-read tokens
1,726,106
Tool calls
139

Tool call mix

read
59
shell
46
glob
17
edit
9
task
6
createPlan
2
Run 5 2026-09-01 · standard · high $2.13 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.32 2m 02s 11 3,743 26
reviewPlan succeeded $0.31 1m 40s 13 4,000 21
execute succeeded $0.62 2m 05s 62 7,193 42
reviewAndCorrect succeeded $0.48 3m 07s 43 4,674 40
openPullRequest succeeded $0.40 1m 28s 45 3,292 21
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 25s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.13
Agent time
10m 21s
Input tokens
174
Output tokens
22,902
Cache-read tokens
1,687,446
Tool calls
150

Tool call mix

read
72
shell
42
glob
11
task
9
edit
7
updateTodos
5
createPlan
3
getMcpTools
1
Run 4 2026-08-31 · standard · medium $1.96 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.30 1m 30s 13 3,195 29
reviewPlan succeeded $0.24 1m 07s 8 2,741 12
execute succeeded $0.59 1m 42s 61 5,123 39
reviewAndCorrect succeeded $0.49 1m 38s 56 3,842 43
openPullRequest succeeded $0.34 1m 04s 35 2,614 23
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 25s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.96
Agent time
7m 01s
Input tokens
173
Output tokens
17,515
Cache-read tokens
1,686,179
Tool calls
146

Tool call mix

read
62
shell
47
glob
18
edit
9
task
8
createPlan
2
Run 4 2026-08-31 · standard · high $2.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.31 1m 43s 12 3,680 27
reviewPlan succeeded $0.29 1m 26s 12 3,886 23
execute succeeded $0.60 1m 47s 61 6,997 45
reviewAndCorrect succeeded $0.46 2m 46s 43 4,376 42
openPullRequest succeeded $0.38 1m 17s 43 3,167 21
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 23s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.04
Agent time
9m 00s
Input tokens
171
Output tokens
22,106
Cache-read tokens
1,605,628
Tool calls
158

Tool call mix

read
73
shell
42
glob
14
task
10
edit
7
updateTodos
7
createPlan
3
getMcpTools
2
Run 3 2026-08-31 · standard · medium $1.85 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.28 1m 46s 10 2,789 25
reviewPlan succeeded $0.23 1m 16s 8 2,721 9
execute succeeded $0.54 1m 54s 65 4,773 36
reviewAndCorrect succeeded $0.48 1m 43s 54 3,764 39
openPullRequest succeeded $0.32 1m 12s 35 2,374 20
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.85
Agent time
7m 51s
Input tokens
172
Output tokens
16,421
Cache-read tokens
1,582,278
Tool calls
129

Tool call mix

read
57
shell
43
glob
13
edit
9
task
5
createPlan
2
Run 3 2026-08-31 · standard · high $1.91 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.29 1m 59s 10 3,423 26
reviewPlan succeeded $0.27 1m 37s 13 3,449 22
execute succeeded $0.55 1m 56s 64 6,330 38
reviewAndCorrect succeeded $0.43 3m 06s 44 4,305 38
openPullRequest succeeded $0.36 1m 24s 44 2,887 21
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 23s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.91
Agent time
10m 01s
Input tokens
175
Output tokens
20,394
Cache-read tokens
1,502,934
Tool calls
145

Tool call mix

read
70
shell
40
glob
12
task
9
edit
5
updateTodos
5
createPlan
3
getMcpTools
1
Run 2 2026-08-31 · standard · medium $1.73 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.27 1m 26s 11 2,730 22
reviewPlan succeeded $0.23 1m 07s 12 2,588 9
execute succeeded $0.50 1m 37s 64 4,588 36
reviewAndCorrect succeeded $0.43 1m 28s 52 3,312 35
openPullRequest succeeded $0.30 1m 01s 37 2,198 20
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 20s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.73
Agent time
6m 39s
Input tokens
176
Output tokens
15,416
Cache-read tokens
1,452,542
Tool calls
122

Tool call mix

read
57
shell
42
glob
10
edit
7
createPlan
3
task
3
Run 2 2026-08-31 · standard · high $1.81 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.28 1m 35s 14 3,447 24
reviewPlan succeeded $0.26 1m 24s 13 3,355 19
execute succeeded $0.52 1m 40s 65 5,942 33
reviewAndCorrect succeeded $0.41 2m 40s 45 3,952 33
openPullRequest succeeded $0.34 1m 11s 45 2,843 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 19s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.81
Agent time
8m 29s
Input tokens
182
Output tokens
19,539
Cache-read tokens
1,425,996
Tool calls
128

Tool call mix

read
67
shell
37
glob
7
task
6
edit
5
updateTodos
3
createPlan
2
getMcpTools
1
Run 1 2026-08-31 · standard · medium $1.86 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.28 1m 37s 12 2,949 25
reviewPlan succeeded $0.23 1m 11s 9 2,686 10
execute succeeded $0.55 1m 50s 63 4,939 36
reviewAndCorrect succeeded $0.47 1m 37s 54 3,639 38
openPullRequest succeeded $0.32 1m 07s 36 2,441 20
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.86
Agent time
7m 22s
Input tokens
174
Output tokens
16,654
Cache-read tokens
1,573,801
Tool calls
129

Tool call mix

read
58
shell
43
glob
13
edit
8
task
5
createPlan
2
Run 1 2026-08-31 · standard · high $1.93 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.29 1m 49s 12 3,562 25
reviewPlan succeeded $0.28 1m 29s 12 3,639 20
execute succeeded $0.56 1m 49s 63 6,455 40
reviewAndCorrect succeeded $0.43 2m 56s 42 4,169 36
openPullRequest succeeded $0.37 1m 21s 42 3,070 20
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.93
Agent time
9m 23s
Input tokens
171
Output tokens
20,895
Cache-read tokens
1,521,452
Tool calls
141

Tool call mix

read
70
shell
40
glob
9
task
8
edit
6
updateTodos
5
createPlan
2
getMcpTools
1