Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

gpt-5.6-terra

Running gpt-5.6-terra inside codex.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-09-01 · standard · high $0.80 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.05 0m 53s 7,478 2,247 3
reviewPlan succeeded $0.06 0m 42s 15,108 2,045 2
execute succeeded $0.35 4m 23s 60,358 12,081 13
reviewAndCorrect succeeded $0.16 1m 46s 29,654 4,267 19
openPullRequest succeeded $0.18 1m 43s 38,826 3,940 14
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 36s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.80
Agent time
9m 27s
Input tokens
151,424
Output tokens
24,580
Cache-read tokens
996,724
Tool calls
51

Tool call mix

command_execution
47
file_change
4
Run 5 2026-09-01 · standard · medium $0.67 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.04 0m 35s 7,312 1,245 3
reviewPlan succeeded $0.05 0m 34s 10,576 1,642 2
execute succeeded $0.20 2m 30s 28,698 6,531 15
reviewAndCorrect succeeded $0.25 2m 17s 64,997 5,936 26
openPullRequest succeeded $0.14 1m 35s 22,799 3,324 12
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 25s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.67
Agent time
7m 32s
Input tokens
134,382
Output tokens
18,678
Cache-read tokens
887,844
Tool calls
58

Tool call mix

command_execution
50
file_change
8
Run 4 2026-08-31 · standard · high $0.77 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.05 0m 45s 7,296 2,057 5
reviewPlan succeeded $0.06 0m 39s 13,991 1,980 2
execute succeeded $0.34 3m 43s 60,018 11,390 13
reviewAndCorrect succeeded $0.15 1m 34s 28,888 4,055 20
openPullRequest succeeded $0.18 1m 27s 39,742 3,809 13
verifyPullRequest succeeded $0.00 0m 00s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 34s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.77
Agent time
8m 09s
Input tokens
149,935
Output tokens
23,291
Cache-read tokens
955,404
Tool calls
53

Tool call mix

command_execution
49
file_change
4
Run 4 2026-08-31 · standard · medium $0.64 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.04 0m 31s 7,164 1,219 4
reviewPlan succeeded $0.04 0m 30s 9,718 1,552 1
execute succeeded $0.19 2m 19s 27,813 6,345 13
reviewAndCorrect succeeded $0.23 1m 55s 59,285 5,665 26
openPullRequest succeeded $0.13 1m 24s 21,289 3,337 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 25s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.64
Agent time
6m 39s
Input tokens
125,269
Output tokens
18,118
Cache-read tokens
850,280
Tool calls
55

Tool call mix

command_execution
50
file_change
5
Run 3 2026-08-31 · standard · high $0.74 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.05 0m 53s 6,850 1,985 3
reviewPlan succeeded $0.05 0m 42s 12,946 1,820 1
execute succeeded $0.33 4m 04s 58,844 11,121 11
reviewAndCorrect succeeded $0.15 1m 40s 28,293 3,946 18
openPullRequest succeeded $0.17 1m 33s 37,835 3,749 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 31s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.74
Agent time
8m 52s
Input tokens
144,768
Output tokens
22,621
Cache-read tokens
906,456
Tool calls
44

Tool call mix

command_execution
40
file_change
4
Run 3 2026-08-31 · standard · medium $0.60 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.04 0m 32s 6,534 1,142 2
reviewPlan succeeded $0.04 0m 33s 8,880 1,509 1
execute succeeded $0.18 2m 27s 25,245 5,789 13
reviewAndCorrect succeeded $0.22 2m 05s 56,731 5,238 25
openPullRequest succeeded $0.12 1m 33s 20,315 2,929 12
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.60
Agent time
7m 10s
Input tokens
117,705
Output tokens
16,607
Cache-read tokens
812,078
Tool calls
53

Tool call mix

command_execution
48
file_change
5
Run 2 2026-08-31 · standard · high $0.68 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.04 0m 43s 6,306 1,866 1
reviewPlan succeeded $0.05 0m 35s 12,404 1,778 1
execute succeeded $0.30 3m 33s 53,932 10,311 10
reviewAndCorrect succeeded $0.13 1m 24s 25,138 3,658 18
openPullRequest succeeded $0.16 1m 23s 34,521 3,395 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 27s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.68
Agent time
7m 37s
Input tokens
132,301
Output tokens
21,008
Cache-read tokens
817,606
Tool calls
41

Tool call mix

command_execution
39
file_change
2
Run 2 2026-08-31 · standard · medium $0.56 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.03 0m 27s 5,940 1,057 1
reviewPlan succeeded $0.04 0m 30s 8,841 1,381 1
execute succeeded $0.17 2m 06s 24,494 5,679 10
reviewAndCorrect succeeded $0.20 1m 48s 52,445 4,656 25
openPullRequest succeeded $0.11 1m 21s 19,488 2,747 10
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 18s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.56
Agent time
6m 12s
Input tokens
111,208
Output tokens
15,520
Cache-read tokens
756,441
Tool calls
47

Tool call mix

command_execution
43
file_change
4
Run 1 2026-08-31 · standard · high $0.73 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.05 0m 47s 6,756 1,979 3
reviewPlan succeeded $0.05 0m 39s 13,405 1,911 1
execute succeeded $0.32 3m 50s 57,035 10,745 11
reviewAndCorrect succeeded $0.14 1m 36s 27,774 3,926 18
openPullRequest succeeded $0.17 1m 30s 36,505 3,705 12
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 32s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.73
Agent time
8m 22s
Input tokens
141,475
Output tokens
22,266
Cache-read tokens
896,256
Tool calls
45

Tool call mix

command_execution
42
file_change
3
Run 1 2026-08-31 · standard · medium $0.61 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.04 0m 31s 6,661 1,128 3
reviewPlan succeeded $0.04 0m 32s 9,302 1,466 1
execute succeeded $0.18 2m 19s 26,358 5,971 13
reviewAndCorrect succeeded $0.22 1m 59s 57,633 5,210 25
openPullRequest succeeded $0.12 1m 25s 20,345 3,076 11
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.61
Agent time
6m 46s
Input tokens
120,299
Output tokens
16,851
Cache-read tokens
814,336
Tool calls
53

Tool call mix

command_execution
48
file_change
5