Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

← Home

gpt-5.6-sol

Running gpt-5.6-sol inside codex.

Workflow

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

Every run uses the same steps in the same order. Full workflow definition →

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result

Runs

Speed
Reasoning
Run 5 2026-09-01 · standard · medium $2.01 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.26 1m 14s 36,794 2,793 6
reviewPlan succeeded $0.13 0m 52s 15,569 2,322 5
execute succeeded $0.47 3m 49s 35,513 8,577 15
reviewAndCorrect succeeded $0.57 2m 28s 81,247 5,589 11
openPullRequest succeeded $0.58 2m 48s 63,044 5,848 21
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 14s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.01
Agent time
11m 10s
Input tokens
232,167
Output tokens
25,129
Cache-read tokens
1,435,994
Tool calls
58

Tool call mix

command_execution
47
file_change
6
mcp_tool_call
4
web_search
1
Run 5 2026-08-31 · standard · high $2.27 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.28 1m 31s 34,025 2,371 8
reviewPlan succeeded $0.36 1m 41s 57,646 4,713 5
execute succeeded $0.82 6m 07s 72,134 13,983 27
reviewAndCorrect succeeded $0.41 3m 30s 39,189 8,348 39
openPullRequest succeeded $0.40 2m 47s 32,868 5,582 20
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 13s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.27
Agent time
15m 36s
Input tokens
235,862
Output tokens
34,997
Cache-read tokens
1,557,597
Tool calls
99

Tool call mix

command_execution
90
file_change
5
mcp_tool_call
4
Run 4 2026-08-31 · standard · medium $1.93 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.25 1m 02s 35,856 2,651 7
reviewPlan succeeded $0.12 0m 46s 14,737 2,296 3
execute succeeded $0.45 3m 20s 34,550 8,256 17
reviewAndCorrect succeeded $0.54 2m 11s 77,478 5,382 12
openPullRequest succeeded $0.56 2m 19s 61,454 5,766 21
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 14s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.93
Agent time
9m 37s
Input tokens
224,075
Output tokens
24,351
Cache-read tokens
1,356,099
Tool calls
60

Tool call mix

command_execution
45
file_change
9
mcp_tool_call
4
web_search
2
Run 4 2026-08-31 · standard · high $2.17 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.26 1m 19s 32,177 2,263 9
reviewPlan succeeded $0.36 1m 32s 57,966 4,447 6
execute succeeded $0.78 5m 31s 72,352 13,324 28
reviewAndCorrect succeeded $0.40 3m 07s 39,031 7,890 38
openPullRequest succeeded $0.37 2m 22s 30,449 5,165 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 12s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.17
Agent time
13m 52s
Input tokens
231,975
Output tokens
33,089
Cache-read tokens
1,451,259
Tool calls
100

Tool call mix

command_execution
89
mcp_tool_call
6
file_change
5
Run 3 2026-08-31 · standard · medium $1.81 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.23 1m 11s 32,273 2,437 4
reviewPlan succeeded $0.11 0m 53s 14,054 2,096 4
execute succeeded $0.42 3m 36s 31,879 7,871 15
reviewAndCorrect succeeded $0.49 2m 30s 69,799 5,111 11
openPullRequest succeeded $0.55 2m 39s 61,175 5,324 19
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 13s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.81
Agent time
10m 50s
Input tokens
209,180
Output tokens
22,839
Cache-read tokens
1,280,056
Tool calls
53

Tool call mix

command_execution
40
file_change
8
mcp_tool_call
4
web_search
1
Run 3 2026-08-31 · standard · high $2.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.25 1m 26s 30,715 2,179 3
reviewPlan succeeded $0.34 1m 44s 54,829 4,249 4
execute succeeded $0.74 6m 03s 68,843 12,678 25
reviewAndCorrect succeeded $0.37 3m 31s 36,011 7,564 37
openPullRequest succeeded $0.37 2m 37s 29,402 5,122 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 12s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.07
Agent time
15m 21s
Input tokens
219,800
Output tokens
31,792
Cache-read tokens
1,381,700
Tool calls
88

Tool call mix

command_execution
83
file_change
3
mcp_tool_call
2
Run 2 2026-08-31 · standard · medium $1.72 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.22 0m 59s 31,241 2,245 4
reviewPlan succeeded $0.11 0m 43s 13,035 2,058 3
execute succeeded $0.40 3m 09s 28,894 7,677 12
reviewAndCorrect succeeded $0.48 2m 03s 70,135 4,597 10
openPullRequest succeeded $0.51 2m 14s 55,420 5,325 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 10s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.72
Agent time
9m 09s
Input tokens
198,725
Output tokens
21,902
Cache-read tokens
1,220,901
Tool calls
48

Tool call mix

command_execution
39
file_change
5
mcp_tool_call
3
web_search
1
Run 2 2026-08-31 · standard · high $1.91 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.23 1m 09s 27,886 2,056 4
reviewPlan succeeded $0.32 1m 25s 52,446 4,032 2
execute succeeded $0.67 5m 19s 61,190 11,737 23
reviewAndCorrect succeeded $0.35 2m 57s 34,088 7,178 37
openPullRequest succeeded $0.33 2m 06s 26,643 4,506 18
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 10s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.91
Agent time
12m 56s
Input tokens
202,253
Output tokens
29,509
Cache-read tokens
1,266,452
Tool calls
84

Tool call mix

command_execution
79
file_change
3
mcp_tool_call
2
Run 1 2026-08-31 · standard · medium $1.84 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.23 1m 04s 33,496 2,456 4
reviewPlan succeeded $0.12 0m 48s 13,924 2,195 3
execute succeeded $0.43 3m 26s 31,719 8,075 14
reviewAndCorrect succeeded $0.51 2m 15s 72,857 5,034 10
openPullRequest succeeded $0.55 2m 27s 59,924 5,532 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 12s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$1.84
Agent time
10m 00s
Input tokens
211,920
Output tokens
23,292
Cache-read tokens
1,309,952
Tool calls
50

Tool call mix

command_execution
40
file_change
6
mcp_tool_call
3
web_search
1
Run 1 2026-08-31 · standard · high $2.06 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.25 1m 20s 30,065 2,164 5
reviewPlan succeeded $0.34 1m 34s 54,387 4,199 3
execute succeeded $0.74 5m 41s 68,171 12,519 24
reviewAndCorrect succeeded $0.37 3m 15s 36,711 7,445 38
openPullRequest succeeded $0.36 2m 26s 29,743 5,006 18
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 11s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$2.06
Agent time
14m 16s
Input tokens
219,077
Output tokens
31,333
Cache-read tokens
1,388,288
Tool calls
88

Tool call mix

command_execution
82
file_change
3
mcp_tool_call
3