Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

gpt-6-luna

Running gpt-6-luna inside codex.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements the landing page; may run repeated preview captures; no harness step timeout
5 reviewAndCorrect agent Visual review against the brief and reference render; fixes layout and copy; no harness step timeout
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-10-08 · standard · high $0.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.00 0m 27s 272 2,131 3
reviewPlan succeeded $0.00 0m 35s 204 3,085 2
execute succeeded $0.02 2m 12s 1,020 13,874 22
reviewAndCorrect succeeded $0.01 1m 07s 612 6,431 17
openPullRequest succeeded $0.01 0m 51s 952 4,204 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.04
Agent time
5m 13s
Input tokens
3,060
Output tokens
29,725
Cache-read tokens
880,072
Tool calls
63

Tool call mix

command_execution
60
file_change
3
Run 4 2026-10-07 · standard · high $0.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.00 0m 28s 269 2,102 3
reviewPlan succeeded $0.00 0m 35s 200 3,025 2
execute succeeded $0.02 2m 13s 1,035 13,711 22
reviewAndCorrect succeeded $0.01 1m 09s 612 6,550 17
openPullRequest succeeded $0.01 0m 50s 941 4,274 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.04
Agent time
5m 14s
Input tokens
3,057
Output tokens
29,662
Cache-read tokens
883,632
Tool calls
63

Tool call mix

command_execution
61
file_change
2
Run 3 2026-10-07 · standard · high $0.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.00 0m 28s 267 2,079 3
reviewPlan succeeded $0.00 0m 34s 204 3,093 2
execute succeeded $0.02 2m 11s 1,000 14,012 22
reviewAndCorrect succeeded $0.01 1m 08s 603 6,352 17
openPullRequest succeeded $0.01 0m 50s 961 4,185 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 23s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.04
Agent time
5m 11s
Input tokens
3,035
Output tokens
29,721
Cache-read tokens
868,115
Tool calls
63

Tool call mix

command_execution
59
file_change
4
Run 2 2026-10-07 · standard · high $0.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.00 0m 27s 275 2,119 3
reviewPlan succeeded $0.00 0m 35s 202 3,018 2
execute succeeded $0.02 2m 10s 1,027 13,946 22
reviewAndCorrect succeeded $0.01 1m 06s 624 6,429 17
openPullRequest succeeded $0.01 0m 51s 963 4,278 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 19s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.04
Agent time
5m 09s
Input tokens
3,091
Output tokens
29,790
Cache-read tokens
885,762
Tool calls
63

Tool call mix

command_execution
59
file_change
4
Run 1 2026-10-07 · standard · high $0.04 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 — 0 0 0
plan succeeded $0.00 0m 28s 271 2,120 3
reviewPlan succeeded $0.00 0m 35s 203 3,114 2
execute succeeded $0.02 2m 14s 1,028 13,773 22
reviewAndCorrect succeeded $0.01 1m 08s 624 6,284 17
openPullRequest succeeded $0.01 0m 52s 968 4,308 19
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 24s 0 0 0
summarizeUsage succeeded $0.00 — 0 0 0
score succeeded $0.00 — 0 0 0

Run totals

Cost
$0.04
Agent time
5m 17s
Input tokens
3,094
Output tokens
29,599
Cache-read tokens
885,350
Tool calls
63

Tool call mix

command_execution
61
file_change
2