Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

gpt-5.6-luna

Running gpt-5.6-luna inside codex.

FastAPI: Add POST /api/v1/projects/{project_id}/tasks endpoint Plan → Review plan → Execute → Review & fix → Open PR → Verify PR → Verify e2e

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements code and tests from the reviewed plan
5 reviewAndCorrect agent Runs checks, reviews against the plan, fixes, re-runs checks
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result
Speed
Reasoning
Run 5 2026-08-21 · standard · high $0.09 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 57s 27,929 2,631 4
reviewPlan succeeded $0.01 0m 49s 15,868 2,465 2
execute succeeded $0.02 2m 04s 37,269 5,284 19
reviewAndCorrect succeeded $0.02 1m 55s 61,063 4,269 19
openPullRequest succeeded $0.03 1m 34s 67,518 3,616 17
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 11s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.09
Input tokens
209,647
Output tokens
18,265
Cache-read tokens
1,101,056
Tool calls
61

Tool call mix

command_execution
55
file_change
5
mcp_tool_call
1
Run 4 2026-08-20 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 1m 01s 16,400 2,899 3
reviewPlan succeeded $0.00 0m 39s 5,698 1,887 1
execute succeeded $0.02 2m 57s 41,844 8,013 44
reviewAndCorrect succeeded $0.01 1m 18s 24,052 3,447 14
openPullRequest succeeded $0.03 1m 36s 55,954 3,713 12
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.07
Input tokens
143,948
Output tokens
19,959
Cache-read tokens
1,111,552
Tool calls
74

Tool call mix

command_execution
70
file_change
4
Run 3 2026-08-20 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 54s 17,578 2,527 3
reviewPlan succeeded $0.00 0m 33s 9,096 1,661 1
execute succeeded $0.02 2m 18s 39,135 6,255 17
reviewAndCorrect succeeded $0.02 1m 45s 59,875 3,930 26
openPullRequest succeeded $0.02 1m 28s 37,291 3,494 25
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 11s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.07
Input tokens
162,975
Output tokens
17,867
Cache-read tokens
781,568
Tool calls
72

Tool call mix

command_execution
65
file_change
7
Run 2 2026-08-20 · standard · high $0.07 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 48s 27,964 2,134 3
reviewPlan succeeded $0.00 0m 42s 6,756 1,923 3
execute succeeded $0.02 2m 14s 35,537 5,752 19
reviewAndCorrect succeeded $0.02 1m 42s 36,043 4,484 17
openPullRequest succeeded $0.02 1m 52s 33,909 4,614 21
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 22s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.07
Input tokens
140,209
Output tokens
18,907
Cache-read tokens
1,143,296
Tool calls
63

Tool call mix

command_execution
58
file_change
5
Run 1 2026-08-19 · standard · high $0.06 PASS

Steps

Swipe to view more

Agent steps start a fresh session. Host steps are included in the run cost.
Step Status Cost Time In tok Out tok Tools
pullIssue succeeded $0.00 0 0 0
plan succeeded $0.01 0m 46s 24,219 2,196 2
reviewPlan succeeded $0.00 0m 38s 8,921 1,897 1
execute succeeded $0.02 2m 00s 26,924 5,320 17
reviewAndCorrect succeeded $0.01 1m 22s 27,139 3,359 19
openPullRequest succeeded $0.02 1m 39s 42,159 3,769 15
verifyPullRequest succeeded $0.00 0m 01s 0 0 0
verifyPullRequestCheck succeeded $0.00 0m 21s 0 0 0
summarizeUsage succeeded $0.00 0 0 0
score succeeded $0.00 0 0 0

Run totals

Cost
$0.06
Input tokens
129,362
Output tokens
16,541
Cache-read tokens
961,024
Tool calls
54

Tool call mix

command_execution
50
file_change
4