Benchmark AI_TRUECOST_

real AI cost, not vanity metrics

Follow on X updates & details

Add POST /api/v1/projects/{project_id}/tasks endpoint

The target repo is a FastAPI service that stores projects and the tasks on them. Clients can already create a project and list its tasks.

This ticket adds POST /api/v1/projects/{project_id}/tasks so a client can create a task on an existing project.

Stack it runs on

Language
Python 3.13
API
FastAPI + Uvicorn
Validation
Pydantic v2
ORM / DB
SQLAlchemy 2.0 + SQLite
Package / runtime
uv
Lint / types / tests
Ruff, ty, pytest
CI
GitHub Actions

All steps run as one workflow, inside a single Docker container.

Swipe to view more

5 of the 10 steps are agent sessions, each started fresh: only the repository working tree and the previous step's output carry over. Host steps orchestrate and verify, and cost nothing.
# Step Kind What happens
1 pullIssue host Fetches the GitHub issue and creates a branch
2 plan agent Drafts a plan from the ticket, including tests
3 reviewPlan agent Always runs; returns the full final plan
4 execute agent Implements code and tests from the reviewed plan
5 reviewAndCorrect agent Runs checks, reviews against the plan, fixes, re-runs checks
6 openPullRequest agent Commits, pushes, opens a pull request into main
7 verifyPullRequest host Checks that the exact pull request is open
8 verifyPullRequestCheck host Waits for the required end-to-end check to pass
9 summarizeUsage host Totals tokens and cost
10 score host Records the final result

How a run is scored

A run passes when the workflow succeeded, the pull request is open, and the required end-to-end check passed on CI.

What this ticket requires

  • 404 for an unknown project, 409 for a duplicate external_id, 422 for an invalid body
  • The created task survives a restart
  • Unit and integration tests for the new endpoint
  • No edits to the e2e acceptance suite or the CI workflows