# MAGI: A Risk-Reduction System Inside an Agentic Loop (pt.2) ### Table of Contents - Introduction - Context - The platform - Setup of the run - How the run unfolded - The pipeline as executed - Timeline - Live log - Effort by agent - The MAGI board in detail - Independent analyses - The three proposals - Cross-review with authors masked - Preventive review by QA, SRE, Security and the Tech Writer - Claim verification and selection - Implementation: the code written by the developers - DEV1, the writer - DEV2, the pair - Verification and reviews of the implemented change - QA: acceptance scenarios - SRE - Security - Tech Writer - Final review by the three architects - Functional requirements: from request to evidence - Traceability - Requirement-by-requirement outcome - Findings overall - Discussion - What the architecture changed in this run - Threats to validity - Cost of robustness - Second run: the same request on Amazon Bedrock AgentCore - What AgentCore changes - The deployment in the AWS console - The run: a second design round - What it took to get a SUCCESS - What the gates did not see - Independent check outside the platform - Two runs side by side - Conclusion - Appendix: mandatory gates Well, here we are again. After the [autopsy of the agentic loop](/posts/autopsy-of-an-agentic-loop/) I wanted to see the engine on something bigger than a Python upgrade: a long, demanding change request, written by a human, in Italian, for a service that already exists. Not a toy, not a fix. A feature. ## Introduction This is a single end-to-end run of **magi-system**, an autonomous software-change platform. Three independent "architect" agents (Melchior, Balthasar, Casper) analyze a change request, propose plans, review each other's proposals with the authors masked, and have every factual claim checked against the repository by tools before code computes which plan is admissible. Two developer agents (writer and pair) implement the selected plan. QA, SRE and Security verify it, a Tech Writer documents it, the three architects review the final candidate in a fresh context, and a deterministic gate decides the outcome. The request asked for a three-tool "Dev/SRE Toolbox" inside an existing FastAPI service (`pytbak`). The run ended with **SUCCESS** after **19 min 19 s** and **US$ 1.05**: - all ten mandatory gates passed on the final commit - a 21-file change (+2,340 / -51 lines) with 67 new tests - no regression in the 79 existing ones - no push, no merge, no deployment In this post I follow the run as it happened: what each agent produced, how each requirement went from the request to code and evidence, and where the safeguards changed the course of the run. Including the places where the platform got it wrong, because there are a few. ## Context ### The platform magi-system is a LangGraph application. The graph state is JSON-native and checkpointed in SQLite, agents call an LLM through a provider adapter, and every command that touches the target runs in a container with no network. Three properties matter here: 1. **Facts are checked, not voted on.** Architects state claims in a typed form (file exists, symbol exists, text present, package version, policy key, command succeeds). A verifier checks them against the baseline commit. A plan that relies on a refuted critical claim, an unresolved critical unknown or a confirmed blocker is inadmissible, however well it scores. 2. **Evidence is bound to a commit.** Each check result carries the commit it ran on, and the gate reads only the final candidate's results. Any new commit invalidates earlier evidence. 3. **No human in the loop, explicit failure.** Every loop has a budget, and every run ends in either a verified candidate or a typed failure state. ### Setup of the run | Item | Value | | ------------- | ------------------------------------------------------------------------------------------------------------------------ | | Target | `pytbak` 2.0.0 (FastAPI 0.143.1, Pydantic 2.14.1, Python 3.11 or later), 139 tracked files, baseline `e3920b5` | | Target config | `.magi/config.yaml`: build `python -m compileall -q app`, test `python -m pytest -q`, lint `ruff --select E9,F63,F7,F82` | | Runner | Docker, image with the target's dependencies preinstalled, `--network=none` | | Model | `deepseek/deepseek-v4.1-flash` for all nine agents, temperature 0, 32,000 output tokens per call | | Routing | OpenRouter `provider: {sort: throughput, require_parameters: true}`, schema-constrained output | | Budget | 3,000 tool calls, 600 LLM calls, 30 M tokens, US$ 5, 3,600 s | | Request | 230 lines in Italian | This is not the first attempt. The configuration is the result of eleven previous runs on the same target, and each one exposed a problem that became a fix. This is run twelve. ## How the run unfolded ### The pipeline as executed {{< mermaid >}} flowchart TD A[Intake: snapshot e3920b5, 139 files, config from .magi/config.yaml] --> B subgraph MAGI[MAGI board, round 0] B[3 independent analyses, 135 requirements traced to quotes] --> C[3 proposals: A Casper, B Melchior, C Balthasar] C --> D[Cross-review with authors masked, plus QA SRE Security Tech Writer preventive review] D --> E[Claim verification by tools: 290 claims, 240 supported, 42 refuted, 8 unverifiable] E --> F{Code computes admissibility and rank} end F -->|B admissible, rank 1, score 4.59| G[DEV1 writer: 20 files, commit be60190] F -.->|A admissible, rank 2, tie on score| X1[not selected] F -.->|C inadmissible| X2[rejected] G --> H[Initial checks: build PASS, lint PASS, test FAIL 1 of 146] H --> I[DEV2 pair: 11 concerns, fixes the wrong test expectation, commit d93229e] I --> J[All checks: build, test, lint, 11 QA scenarios, integrity, scope PASS] J --> K[Specialist review on the code: QA, SRE, Security PASS] K --> L[Tech Writer: README and docs/14-dev-sre-toolbox.md, commit 5cce890] L --> M[Evidence invalidated: every check re-run on 5cce890, PASS] M --> N[Final review by the 3 architects, fresh context, PASS] N --> O[Deterministic gate: 10 of 10 mandatory gates PASS] O --> P([SUCCESS: patch and local branch, not pushed]) {{< /mermaid >}} Read it top to bottom and notice where models talk and where code decides. The models write analyses, proposals and code. Admissibility, rank, gates and the final verdict are computed. ### Timeline {{< mermaid >}} gantt title run-20261015-205501-e0a70f, wall clock, 15 Oct 2026 dateFormat HH:mm:ss axisFormat %H:%M section MAGI board Analysis (3 in parallel) :a1, 20:55:01, 124s Proposals (3 in parallel) :a2, 20:57:10, 97s Cross-review and preventive review :a3, 20:58:47, 87s Claim verification and selection :a4, 21:00:14, 11s section Implementation DEV1 writer :b1, 21:00:25, 178s Initial checks (container) :b2, 21:03:23, 7s DEV2 pair (assess and fix) :b3, 21:03:30, 166s section Verification Checks and QA scenarios :c1, 21:06:16, 34s Specialist review :c2, 21:06:50, 13s Tech Writer :c3, 21:07:03, 116s Re-verification on docs commit :c4, 21:09:00, 81s Final review (3 architects) :c5, 21:10:21, 239s Gate :milestone, 21:14:20, 0s {{< /mermaid >}} The MAGI board, the phase that decides *what* to build, took 5 min 24 s (28% of the run). Implementation took 5 min 51 s. Verification, documentation and the final review took the remaining 8 minutes. The final review was the single longest step. One architect's output was truncated by the 32,000-token cap twice (a temperature-0 repetition loop) and needed a technical retry. More on that in the discussion. ### Live log I followed the run live through its event stream (`events.jsonl`). The commentary below is what I was told during the run, translated and lightly condensed. It shows what an operator sees and has to judge at each step. > **20:55:01, run_start.** Run 12 started: commit `e3920b5`, environment `container:docker:...pytbak-magi:local`, policy `default`. The three architects start their analysis in parallel, each with read-only tools and without seeing the others' work. > **20:57:05, analysis collected.** 135 requirements accepted (111 explicit, 24 derived), every explicit one carrying a verbatim **Italian** quote of the request. 5 rejected: 3 paraphrased quotes not found in the request, 2 derived "requirements" with no traceable source (for example "protect the endpoints with Basic Auth": an architect's interpretation, correctly kept out of the requirements). > **21:00:25, selection.** Plan selected in round 0: **Proposal B, Melchior**. Balthasar's was inadmissible: a critical claim was refuted ("the platform policy lists `.github/workflows/**` ... as protected paths") and a critical unknown about write scopes was unresolved. Note for later: that claim is *true*, but it was written as a `policy_exists` claim on a path, which the verifier looks up as a policy key. It did not block the run, since a plan was selected anyway. > **21:03:23, handoff dev.writer.** DEV1 wrote the code in 3 min, 4 escalations. Melchior's structure follows the repo's own convention: models in `app/models/toolbox.py`. > **21:03:29, check project.test FAIL** on `be60190`. Checked on a copy: 145 passed, 1 failed (`test_empty_documents_are_skipped`), the same test that failed in run 9 with a different plan and author. A recurring trap with empty YAML documents. > **21:06:16, handoff dev.pair.** DEV2 changed the **test**, not the code. That could be a legitimate correction or a weakened test, so I checked the diff: only the expected value changes (`documents == 3` to `== 2`), with a comment explaining PyYAML's behavior, and the actual assertion (`resources_inspected == 1`) stays. Verified independently: PyYAML returns 2 documents (`[None, dict]`) for `---\n---\n`. DEV2 was right, DEV1's expectation was wrong, and the code was correct. > **21:07:03, specialists on `d93229e`.** `qa.acceptance` PASS (11 scenarios), `sre.review` PASS (5 non-blocking findings), `security.review` PASS (0 findings). > **21:09:04, check on `5cce890`.** The Tech Writer has finished, producing a new commit `5cce890`. All checks are rerunning on it, and tests are already green. Next come the specialists and then the final review. > **21:10:21, specialists on `5cce890`.** The specialists pass again on the final candidate: QA 11 scenarios, SRE 5 non-blocking findings, Security 8 non-blocking findings. Next is the final review by the three architects, then the gate. > **21:12:56, agent_failure magi.balthasar.** Balthasar's final review was truncated twice at 32k, as in run 10 (a repetition loop). A technical retry is running for Balthasar only; Melchior's and Casper's reviews are kept. > **21:14:20, magi.final_review PASS.** Three final reviews completed, 7 findings, none confirmed blocking. **run_end: SUCCESS**, `change_level: verified`, final commit `5cce890`. ### Effort by agent {{< mermaid >}} pie showData title Input tokens by agent, total 4.74 M "Melchior" : 600897 "Balthasar" : 739261 "Casper" : 830805 "DEV1 writer" : 543558 "DEV2 pair" : 765383 "Tech Writer" : 861686 "QA" : 132193 "SRE" : 132193 "Security" : 132190 {{< /mermaid >}} | Agent | LLM calls | Input tokens | Output tokens | Tool calls | | ------------------- | --------- | ------------- | ------------------------ | ---------- | | Melchior | 17 | 600,897 | 106,201 | 79 | | Balthasar | 20 | 739,261 | 137,386 | 90 | | Casper | 21 | 830,805 | 79,983 | 94 | | DEV1 writer | 12 | 543,558 | 62,295 | 61 | | DEV2 pair | 13 | 765,383 | 43,752 | 38 | | Tech Writer | 13 | 861,686 | 44,376 | 31 | | QA / SRE / Security | 3 each | 132,193 each | 41,871 / 18,028 / 17,843 | 0 | | **Total** | **105** | **4,738,166** | **551,735** | **393** | Input dominates (90% of tokens) because each tool-loop turn resends the transcript. The three architects account for 46% of input tokens. The analysis is the part most exposed to hallucination, and it is where the platform spends most. I think that is the right place. ## The MAGI board in detail ### Independent analyses The three architects started from the same snapshot, each with read-only tools (`list_files`, `read_file`, `grep`) and without seeing the others' work. Each analysis returns an understanding of the mandate, the requirements with their source quotes, typed claims about the repository, assumptions, open questions and risks. | | Melchior (evolution, minimal change) | Balthasar (failure modes) | Casper (challenge) | | --------------------------- | ------------------------------------ | ------------------------- | ------------------ | | Requirements extracted | 79 | 64 | 27 | | Claims about the repository | 30 | 34 | 18 | | Assumptions | 8 | 10 | 7 | | Open questions | 7 | 10 | 6 | | Risks | 8 | 10 | 8 | | Duration | 62 s | 63 s | 123 s | The requirement sets overlap and are merged by id, so the union is the 135 requirements of the run. Strange... the three analyses converged on the same open questions without seeing each other: the URL prefix (`/api/v1/toolbox` as written, or `/api/toolbox` following the repository), whether to reuse the Basic Auth of `/api/debug/*`, and whether to gate the feature behind a flag like `DEBUG_ENDPOINTS_ENABLED`. Their risks show the different perspectives: - **Melchior** looked at fit with the existing app: the app-wide 400 handler strips error detail, so limit errors must not use 400. The error-injection middleware (`?inject_error=`) and the global slowapi rate limit (100/min) apply to the new endpoints and could interfere with tests. - **Balthasar** looked at what breaks: new routes change `GET /api/mgmt/mappings` and make the README's "31 routes registered" stale without any test noticing. `yaml.safe_load_all` still expands anchors and aliases, so a hostile manifest can explode in memory. Offloading to the default thread pool shares it with the existing CPU-bound endpoints. - **Casper** challenged the scope: custom Prometheus metrics could introduce high-cardinality labels, and limits must be enforced before or during parsing, not after. ### The three proposals All three proposals adapted the prefix to `/api/toolbox` (the repository has a single `/api` prefix and no versioning, and the request explicitly allows adapting it), reused `verify_credentials`, offloaded parsing with `loop.run_in_executor` after the existing `/api/fib/{x}` precedent, pinned PyYAML and kept analysis code free of FastAPI. They differed in structure and in a few contracts: | | **B, Melchior** (selected) | **A, Casper** | **C, Balthasar** | | ---------------------- | --------------------------------------- | ----------------------------------- | -------------------------------------------------------------- | | Analysis code | `app/toolbox/` (base, registry, tools/) | `app/toolbox/` + `runner.py` | `app/services/toolbox/` + `runner.py` | | Models | `app/models/toolbox.py` | `app/models/toolbox.py` | `app/models/toolbox.py` | | Planned file changes | **21** | 23 | 20 | | Assumptions | 12 | 12 | 8 | | Unknown tool | 404 via the app-wide handler | 422 with a readable reason | undecided (200 + `TOOL_UNKNOWN` finding, *or* 404) | | Over-limit input | 413 | 413 | 413 for size; 200 + limit finding for documents/depth/findings | | Findings cap | analysis outcome, never an HTTP error | `outcome=error` + dedicated finding | 200 + limit finding | | Kill switch | `TOOLBOX_ENABLED` | none | none | | Rollback | revert, or `TOOLBOX_ENABLED=false` | revert | revert | | Regression test module | yes | yes | no (existing suites re-run) | Each proposal also listed its trade-offs explicitly. Melchior's, for example: > *Limit rejections use 413 rather than 400: the app-wide 400 handler would erase the specific reason; 413 keeps the message but is a slightly unusual code for "too many YAML documents".* And: *Findings cap is an analysis outcome rather than an HTTP error: preserves "a finding is not an HTTP error" and "no silent truncation", at the cost of two different limit-handling paths that must both be documented.* ### Cross-review with authors masked Each architect scored the other two proposals, never its own, on the six rubric criteria of the platform policy, without knowing who wrote them (labels A, B, C). | Reviewer, proposal | Requirement fit (x3) | Simplicity (x2) | Risk (x2) | Operability (x1) | Testability (x2) | Maintainability (x1) | Weighted | | ----------------------------- | -------------------- | --------------- | --------- | ---------------- | ---------------- | -------------------- | -------- | | Balthasar on **B** (Melchior) | 5 | 4 | 4 | 5 | 5 | 4 | 4.55 | | Casper on **B** (Melchior) | 5 | 4 | 4 | 5 | 5 | 5 | 4.64 | | Melchior on **A** (Casper) | 5 | 4 | 4 | 4 | 5 | 4 | 4.45 | | Balthasar on **A** (Casper) | 5 | 5 | 4 | 4 | 5 | 5 | 4.73 | | Melchior on **C** (Balthasar) | 4 | 4 | 3 | 4 | 4 | 4 | 3.82 | | Casper on **C** (Balthasar) | 4 | 4 | 3 | 4 | 4 | 4 | 3.82 | {{< mermaid >}} xychart-beta title "Peer score per proposal, weighted rubric, mean of two reviewers" x-axis ["B Melchior", "A Casper", "C Balthasar"] y-axis "Score (1-5)" 3 --> 5 bar [4.59, 4.59, 3.82] {{< /mermaid >}} Both reviewers of C gave it the same scores and the same reason: the unknown-tool contract is left undecided in the plan, and it carries an unresolved assumption about whether the runtime image installs PyYAML from `requirements.txt` or `docker/requirements.txt`. Casper's note on C: *"regression coverage is delegated to pre-existing tests rather than newly implemented"*. ### Preventive review by QA, SRE, Security and the Tech Writer Before any code was written, the four specialists reviewed all three proposals: 49 findings in total, plus 16 from the architects' cross-review. {{< mermaid >}} xychart-beta title "Findings raised on the proposals, by reviewer" x-axis ["SRE", "Tech Writer", "QA", "Security", "Casper", "Balthasar", "Melchior"] y-axis "Findings" 0 --> 18 bar [17, 12, 10, 10, 7, 5, 4] {{< /mermaid >}} - **SRE** raised the same three *major* findings against every proposal: *no per-run timeout or concurrency bound on tool execution*; *`run_id` is not propagated to tracing, so response/log/trace correlation is not achieved*; *a new hard dependency imported at application-import time, so a missing install takes the whole app down*. On C it also flagged the undecided unknown-tool contract. - **Security** confirmed on all three proposals that *CPU-bound tool execution shares the process-wide default thread pool with other endpoints*. On C, that *PyYAML availability in the runtime image is an unresolved critical assumption* (major). On B, that the 404 body for an unknown tool is masked by the app-wide 404 handler. Whether `json.validate` should echo the parsed value back in the response stayed unresolved. - **QA** wrote the 12 acceptance scenarios and raised findings on testability: OpenAPI presence only in the verification plan, not in a test; the no-network guarantee asserted only by static import inspection. Two of its objections were **refuted** by verification, for example *"allowPrivilegeEscalation absent is mapped to severity error"* on A, when the plan maps it to a warning. An objection without support is not a veto. - **The Tech Writer** required that *curl examples must carry the Basic Auth header to be "working"* (the request asks for working examples), and that the new settings be documented in `.env.example`. Findings are not votes. Each one is resolved by verifying its supporting claims: it becomes `CONFIRMED`, `REFUTED` or `UNRESOLVED`, and only a confirmed blocker excludes a proposal. ### Claim verification and selection The verifier checked 290 claims against `e3920b5` in 11 seconds: 240 supported, 42 refuted, 8 unverifiable (operational guarantees and business assumptions, which cannot be checked mechanically and block only when critical). Typical supported claims: `verify_credentials` defined in `app/auth.py`; `loop.run_in_executor` used in `app/routers/api.py`; `APIRouter(prefix="/api"` in the existing routers; `DEBUG_ENDPOINTS_ENABLED` handled in `tests/conftest.py`. Code then computed admissibility and rank: | Proposal | Admissible | Score | Rank | Reason | | ------------ | ---------- | ------ | ----- | -------------------------------------------------------------------------------------------- | | B, Melchior | yes | 4.5909 | **1** | none | | A, Casper | yes | 4.5909 | 2 | none | | C, Balthasar | no | n/a | n/a | refuted critical claim about protected paths; unresolved critical unknown about write scopes | A and B tied exactly. The policy's tie-break (`fewer_planned_changes`, `fewer_assumptions`, `magi_order`) chose B: 21 planned changes against 23. No vote was taken at any point. ## Implementation: the code written by the developers ### DEV1, the writer DEV1 received the selected plan, the requirements and the write scopes, and implemented it in 12 LLM calls and 61 tool calls (3 min). Its handoff is explicit about what is not done: > *The work is NOT fully green: the full suite reports 1 failed / 145 passed / 25 skipped, and the README.md update command failed, so README.md is unchanged. Both are reported below as escalations with evidence; I could not fix them in this turn.* I like that. A writer that says "not green" in its own handoff is a writer I can work with. The code follows the plan closely. The common contract, in `app/toolbox/base.py`: ```python class Tool(Protocol): """Common contract every toolbox tool implements.""" name: str version: str description: str input_model: type[BaseModel] def execute(self, params: BaseModel, limits: ToolboxLimits) -> ToolResult: ... ``` The explicit registry rejects empty and duplicate identifiers and returns tools in a deterministic order: ```python def register(tool: Tool) -> None: """Register a tool. Raises ValueError on an empty or duplicate identifier.""" if not tool.name: raise ValueError("tool name must not be empty") if tool.name in _REGISTRY: raise ValueError(f"duplicate tool name: {tool.name}") _REGISTRY[tool.name] = tool ``` The router keeps the four error channels the request asked to keep apart: 422 for invalid tool input, 404 for an unknown tool, 413 for limits, a sanitized 500 for anything unexpected. Findings never change the HTTP status. Parsing runs in the thread pool, and the log line carries only `run_id`, tool, status, outcome and duration: ```python try: result = await loop.run_in_executor(None, tool.execute, params, limits) except ToolLimitExceeded as exc: ... raise HTTPException(status_code=413, detail=str(exc)) from exc except ToolInputError as exc: raise HTTPException(status_code=422, detail=str(exc)) from exc except Exception: logger.exception( "toolbox run failed | run_id={} tool={} status={} duration_ms={}", run_id, tool.name, ExecutionStatus.FAILED.value, duration_ms, ) raise HTTPException(status_code=500, detail="Internal server error") from None ``` Balthasar's risk about alias expansion was handled in the YAML loader itself: ```python class _NoAliasSafeLoader(yaml.SafeLoader): """SafeLoader that refuses YAML aliases. Aliases make it possible to build a small document that expands to a huge object graph (a "billion laughs" bomb). The toolbox never needs them, so they are rejected instead of being expanded. """ ``` The request's rule "init containers: no probe rules" is a single explicit branch, while security checks still apply to init containers: ```python # --- probes: regular containers only, never init containers --- if not is_init: if not container.get("readinessProbe"): add("K8S_MISSING_READINESS_PROBE", Severity.WARNING, ...) ``` The distinction between *missing* and *explicitly risky* configuration is encoded as two codes with different severities: `allowPrivilegeEscalation: true` gives `K8S_ALLOW_PRIVILEGE_ESCALATION_ENABLED` (error), while a missing field gives `K8S_ALLOW_PRIVILEGE_ESCALATION_NOT_DISABLED` (warning). DEV1 wrote 67 tests in five files: | File | Tests | Covers | | ------------------------------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `test_toolbox_k8s_manifest.py` | 29 | clean and dirty manifests, multi-document, empty documents, unsupported kind, missing kind, unexpected structure, non-mapping document, requests/limits, probes, untagged / `latest` / digest images, privileged, privilege escalation true vs absent, non-root at pod level, init containers (no probes, security still checked), StatefulSet and DaemonSet, unsafe YAML tag, aliases, invalid YAML as negative outcome, document/depth/findings/size limits, determinism | | `test_toolbox_api.py` | 13 | auth on catalog and runs, wrong credentials, uniform schema, finding not an HTTP error, unknown tool 404, malformed 422, invalid tool input 422, over-limit 413, sanitized internal error, OpenAPI exposure, determinism | | `test_toolbox_json_validate.py` | 10 | valid object and scalar, error position, empty and whitespace input, URL-like and path-like strings treated as text, oversized input, determinism | | `test_toolbox_text_diff.py` | 9 | identical, changed, empty inputs, context 0, context above limit, oversized left/right, determinism | | `test_toolbox_regression.py` | 6 | contexts, fibonacci, health, debug auth, unchanged 400 envelope, route mappings | The sanitized-error test, for example, injects a tool that raises `RuntimeError("secret internal detail")` and asserts that the response contains neither the message nor a traceback. ### DEV2, the pair DEV2 first assessed DEV1's candidate independently, *without seeing DEV1's handoff*, from the mandate, the plan, the diff and the check results. It raised 11 concerns, among them: > *BLOCKING: project.test FAILS. `test_empty_documents_are_skipped` expects `output['documents'] == 3` but the tool returns 2 for `'---\n---\n' + GOOD_DEPLOYMENT`.* And: *The custom loader rejects all YAML aliases/anchors, so a syntactically valid manifest that uses anchors is reported as `K8S_YAML_INVALID`. This is a defensible hardening choice but is a false-negative risk that should be documented.* And: *The findings cap keeps `max_findings-1` findings plus a notice; it is not silent, but confirm the dropped count is reported.* Then it read the whole package, ran a direct `yaml.safe_load_all` probe in the container, and concluded that the tool was right and the test wrong: ```diff def test_empty_documents_are_skipped(): + # PyYAML yields one document per '---' marker: the leading '---\n---\n' + # produces a single empty (None) document followed by the Deployment, so + # the stream holds 2 documents and only 1 is inspected. manifest = "---\n---\n" + GOOD_DEPLOYMENT result = _run(manifest) - assert result.output["documents"] == 3 + assert result.output["documents"] == 2 assert result.output["resources_inspected"] == 1 ``` DEV2 also fixed the order of the docs index and escalated what it could not do: the README update, blocked because a helper script it tried to write at the repository root was outside the write scopes. The Tech Writer later updated the README. ## Verification and reviews of the implemented change ### QA: acceptance scenarios QA had defined 12 acceptance scenarios during the preventive review, each linked to requirement ids and to a command. Eleven had a command and were run in the container on `d93229e` and again on `5cce890`. The twelfth ("no superfluous infrastructure is introduced") has no mechanical check and was assessed in the reviews. | Scenario | Command | Verdict | | ---------------------------------------------------------- | --------------------------------------------- | ------------------ | | acc-01 the app still compiles and starts | `python -m compileall -q app` | PASS | | acc-02 new and existing tests pass | `python -m pytest -q -p no:cacheprovider` | PASS | | acc-03 lint gate passes | `ruff check --select E9,F63,F7,F82 app tests` | PASS | | acc-04 three tools in the catalog and runnable | `python -m pytest -q -k toolbox` | PASS | | acc-05 typed requests and responses in OpenAPI | `python -m pytest -q -k toolbox` | PASS | | acc-06 uniform result schema | `python -m pytest -q -k toolbox` | PASS | | acc-07 no network, cluster or credentials | `python -m pytest -q -k toolbox` | PASS | | acc-08 hostile or over-limit inputs give controlled errors | `python -m pytest -q -k toolbox` | PASS | | acc-09 json.validate contract | `python -m pytest -q -k toolbox` | PASS | | acc-10 text.diff contract | `python -m pytest -q -k toolbox` | PASS | | acc-11 k8s.manifest.inspect contract | `python -m pytest -q -k toolbox` | PASS | | acc-12 no superfluous infrastructure | none | assessed in review | Eight scenarios share the same command, which runs every toolbox test. They are weaker than they look: each one passes if the whole toolbox suite passes, not because a test specific to that scenario passed. ### SRE On the code, SRE confirmed what was solved and what was not. Solved: *"Findings cap behaviour matches the no-silent-truncation requirement"*, *"No new infrastructure, persistence or retrieval endpoint introduced"*. Open, non-blocking: *"Observability log line is claimed but not asserted by any test"*; *"New toolbox limits are not surfaced in `.env.example`"*; an edge case where `max_findings=0` still returns one finding (the notice); and an unresolved doubt that the request-size cap relies on `Content-Length` and could be bypassed with chunked transfer encoding, with the per-field input limit as the backstop. Of the three major concerns raised before implementation, the hard-import concern remained a design choice (the dependency is pinned), and no per-run timeout or tracing propagation was added. ### Security Security confirmed by tests that the endpoints reuse Basic Auth (401 without and with wrong credentials), that unsafe YAML tags and aliases are rejected, and that internal errors are sanitized. It raised two non-blocking findings of substance: *"`json.loads` recursion depth is not visibly guarded"* (a deeply nested JSON document can raise `RecursionError`; the request asks for depth limits, and only YAML has one), and the `max_findings=0` edge case. ### Tech Writer The Tech Writer read the implemented code, not the plan, and wrote `docs/14-dev-sre-toolbox.md` (360 lines: feature description, curl examples with Basic Auth, response examples, configuration and limits, how to add a tool, limitations of the Kubernetes checks, architecture note). It also updated the README, the docs index and the CHANGELOG. Its commit `5cce890` invalidated all previous evidence, and every check was re-run. Yes, for a Markdown-only commit. That is on purpose. ### Final review by the three architects The three architects reviewed the final diff in a fresh context. Melchior and Casper completed in 93 s and 16 s. Balthasar's review was truncated twice by a repetition loop and needed a technical retry. The gate recorded *"three final reviews completed; 7 finding(s), none confirmed blocking"*. Verification confirmed Balthasar's *"json.validate has no recursion/depth guard"*, matching Security's finding independently. It refuted *"Pinned PyYAML version differs between plan and diff"* as a defect (the plan said 6.0.2, the code pins 6.0.3: a documentation mismatch, not a bug). And one refutation is **wrong**. Balthasar reported *"Image reference with an empty tag is not flagged"*, and verification refuted it. Reading the code shows the finding was correct: for `nginx:` the last path segment contains `:`, so the "no tag" branch is skipped, and the empty tag is not `latest`, so nothing is reported: ```python if ":" not in last_segment: add("K8S_IMAGE_TAG_MISSING", Severity.WARNING, ...) elif last_segment.rsplit(":", 1)[1] == "latest": add("K8S_IMAGE_TAG_LATEST", ...) ``` The finding was minor and non-blocking, so the outcome would not have changed. But it is a real defect that the run shipped and that the platform marked as refuted. ## Functional requirements: from request to evidence ### Traceability The request was decomposed into 135 requirements. An explicit requirement is accepted only if its `source_quote` occurs verbatim (whitespace- and case-normalized) in the request. That is what prevents an architect from inventing a requirement. QA then turned the requirements into 11 acceptance scenarios, each bound to a command run in the container on the candidate commit. {{< mermaid >}} flowchart LR R[Request, Italian, 230 lines] --> Q[Verbatim quotes] Q --> REQ[135 requirements: 111 explicit and 24 derived] REQ --> COV[Proposal coverage: every mandatory requirement covered] REQ --> SC[11 acceptance scenarios from QA preventive review] SC --> CMD[Command in container on the candidate commit] CMD --> EV[Evidence bound to commit 5cce890] EV --> G{Gate qa.acceptance} {{< /mermaid >}} ### Requirement-by-requirement outcome | Functional requirement | Implementation in the final commit | Verified by | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Modular feature inside the existing app, no new service, no rewrite | Package `app/toolbox/` + router `app/routers/toolbox.py`, mounted in `create_app()` behind `TOOLBOX_ENABLED` (default `true`); `app/main.py` +8 lines | Scenario "application compiles/starts without regressions"; regression tests; 79 pre-existing tests pass | | Reuse conventions and dependencies | Prefix adapted to the repo: `/api/toolbox/...` (the repo has no `/api/v1`); settings in the existing Pydantic `Settings`; `loop.run_in_executor` pattern copied from `/api/fib/{x}`; PyYAML pinned in `requirements.txt` | Claims verified at analysis (`loop.run_in_executor` present in `app/routers/api.py`, `verify_credentials` in `app/auth.py`) | | Common tool contract and explicit registry | `Tool` protocol (`name`, `version`, input model, `execute(params, limits) -> ToolResult`) in `app/toolbox/base.py`; `app/toolbox/registry.py` with a dict keyed by unique id, duplicate registration rejected, deterministic order | Catalog scenario; unit tests | | `json.validate`: JSON text, parsed value or error with position, no URL/path | `app/toolbox/tools/json_validate.py`; invalid JSON is a negative outcome (HTTP 200, `outcome: fail`, error finding with line/column); URL-like strings are treated as plain text, never fetched | `tests/test_toolbox_json_validate.py`; smoke test: HTTP 200 + `outcome: fail` | | `text.diff`: two strings, unified diff, bounded context lines | `app/toolbox/tools/text_diff.py` (`difflib.unified_diff`), fields `left`/`right`, context lines bounded by `TOOLBOX_MAX_DIFF_CONTEXT_LINES` | `tests/test_toolbox_text_diff.py`; smoke test: wrong field names give 422 with a structured error | | `k8s.manifest.inspect`: multi-document, Deployment/StatefulSet/DaemonSet, static only | `app/toolbox/tools/k8s_manifest_inspect.py` (473 lines); YAML loaded with a `SafeLoader` subclass that also refuses aliases (no "billion laughs"); depth and document-count limits; unsupported kinds and non-mapping documents reported as findings | `tests/test_toolbox_k8s_manifest.py` (349 lines) | | Report missing requests/limits, probes, `latest`/untagged images, privileged, `allowPrivilegeEscalation`, non-root | 18 finding codes, including `K8S_CONTAINER_MISSING_REQUESTS/LIMITS`, `K8S_MISSING_READINESS/LIVENESS_PROBE`, `K8S_IMAGE_TAG_LATEST`, `K8S_IMAGE_TAG_MISSING`, `K8S_PRIVILEGED_CONTAINER`, `K8S_RUN_AS_NON_ROOT_NOT_SET` | Smoke test on a bad Deployment: all 8 expected codes, 1 error + 7 warnings | | Distinguish *missing* from *explicitly risky* | Separate codes: `K8S_ALLOW_PRIVILEGE_ESCALATION_NOT_DISABLED` (missing, warning) vs `K8S_ALLOW_PRIVILEGE_ESCALATION_ENABLED` (explicit `true`, error) | Unit tests; doc table | | Init containers handled separately, no probe rules | Containers and `initContainers` walked separately (`is_init`); probe checks skipped for init containers | Unit tests "init containers" | | `GET /tools`: id, description, version, input schema | `ToolListResponse`, input schema generated from each tool's Pydantic model | Catalog and OpenAPI scenarios | | `POST /runs`: synchronous, HTTP 200 with result and `run_id`, no queue | `run_tool` awaits `run_in_executor(tool.execute)`; UUID `run_id` per call; no persistence, no `GET /runs/{id}` | API tests; smoke test | | Uniform typed result; status vs outcome vs validation vs internal error kept distinct | `ToolRunResponse` (`run_id, tool, tool_version, status, outcome, summary, findings, output, duration_ms`); 422 for malformed requests and invalid tool input, 404 unknown tool, 413 over limit, 500 sanitized (`"Internal server error"`, no stack trace) | Scenario "uniform schema"; tests "malformed", "sanitized internal error" | | Reuse auth; no public bypass | `APIRouter(..., dependencies=[Depends(verify_credentials)])` | Smoke test: 401 without credentials; auth tests | | Configurable limits, controlled rejection, no silent truncation | `TOOLBOX_ENABLED` plus six `TOOLBOX_MAX_*` limits (request bytes, input bytes, YAML documents, YAML depth, findings, diff context lines); a finding-cap notice when findings are capped | Scenario "hostile or over-limit inputs"; final-review findings discuss the cap edge cases | | Logs without payloads; reuse observability | `logger.info` with `run_id`, tool, status, duration only; no new observability stack | SRE review (5 non-blocking findings) | | Tests: units per tool, API, Kubernetes cases, regression | 5 new test files, 67 new tests | `project.test`: 146 passed, 25 skipped | | Documentation: feature, curl, responses, limits, how to add a tool, k8s limitations, ADR note | `docs/14-dev-sre-toolbox.md` (360 lines), README section, `docs/README.md` index, `CHANGELOG.md` | `docs.updated` gate; Tech Writer preventive review asked for Basic Auth in every curl example | ### Findings overall The 101 findings raised during the run show where the specialists added value beyond the tests. Before code was written, SRE raised the same three *major* concerns against all three proposals (no per-run timeout or concurrency bound, `run_id` not propagated to tracing, a hard import-time dependency on PyYAML). The Tech Writer required every curl example to carry the Basic Auth header to be "working", as the request demands. During implementation, DEV1 raised a *blocker* against its own work (the failing test) and a *major* finding (README not updated after a failed edit command), both handled downstream. On the final candidate, 24 findings were raised and none was confirmed blocking. Examples: `json.loads` recursion depth not visibly guarded; new `TOOLBOX_*` settings not added to `.env.example`; an edge case where `max_findings=0` still returns one (notice) finding; observability guarantees documented but not asserted by any executed check. Several final-review findings were **refuted** by verification, one of them wrongly (the empty image tag). The confirmed ones are recorded in the report as residual risks. They did not stop the run because the policy blocks only on confirmed blockers. {{< mermaid >}} pie showData title Findings by status, 101 in total "Confirmed" : 66 "Refuted by verification" : 25 "Unresolved" : 10 {{< /mermaid >}} ## Discussion ### What the architecture changed in this run - **Plan choice by facts.** Two proposals tied on the rubric score (4.59). The tie was broken by the deterministic rule in the policy, not by a vote. The third proposal was excluded before any code existed. - **Pair review as a correctness check, not a formality.** The only failing test came from a wrong expectation in a test written by DEV1. DEV2 fixed the test with a justification that an operator could verify in one command, and no assertion was weakened. - **Evidence invalidation.** The documentation commit changed only Markdown, yet every check was re-run on `5cce890`. The gate never accepts results from an earlier commit. If it isn't there, it can't lie. - **Fresh-context final review.** The three architects re-read the final change. In this run they found nothing blocking. In the previous run (run 10) the same step found a real bug that all 12 QA scenarios had missed (summary counts computed before findings truncation). ### Threats to validity I'd rather list them myself than have you find them: - **One run, one target, one model.** The same request produced different plans in runs 9, 10 and 12 (Balthasar's, Casper's and Melchior's). The result is an existence proof, not a success rate. - **Self-consistency risk.** All nine agents used the same model, so common-mode errors are possible. Tool-based verification and container checks reduce this risk, they do not remove it. - **Verifier coverage.** 42 claims were refuted. Most were real model errors, but at least one class is a verifier limitation: a claim that a path is protected, expressed as `policy_exists` on the path, is refuted although true. It contributed to one proposal's inadmissibility. - **Lint scope.** The configured lint gate checks only syntax-level errors (`E9,F63,F7,F82`), because the target does not pass full ruff at baseline. Type checking is not configured in the target and was not run. - **Reviewers saw a truncated diff.** The specialists and the final reviewers receive the last 60,000 characters of the diff (`diff[-60000:]` in `engine/nodes_verify.py`). This change is about 99,000 characters, so the router, models, base, registry, settings and `main.py` were cut off, and several reviewers said so in their findings ("Provided diff is truncated"). Their conclusions on those files rest on tests and claims, not on reading the code. This is a platform limitation, and it matters for large changes. - **False refutations still exist.** Besides the protected-paths claim, a correct final-review finding (`nginx:` with an empty tag is not flagged) was refuted by verification. A refuted finding drops out of the decision. When the refutation is wrong, a real defect is silently accepted. - **Coarse acceptance scenarios.** Eight of the eleven executed scenarios share the command `pytest -k toolbox`, so they do not discriminate between requirements. - **Operator judgment.** The test change by DEV2 was checked by a human observer (me) during the run. The platform itself would have accepted it on the evidence alone. ### Cost of robustness Six model outputs were rejected because they hit the 32,000-token cap. The excerpts of earlier runs show these are repetition loops at temperature 0 (for example "No X. No Y. No Z. ..." for about 150,000 characters). The cap plus a retry with feedback recovered every case, at the price of about four minutes of the final review and a share of the output tokens. ## Second run: the same request on Amazon Bedrock AgentCore A single run is an existence proof, so I did the obvious thing: same request, same kind of target (a public FastAPI repository, `fastapi-agentcore`), but the whole pipeline moved from my laptop to **Amazon Bedrock AgentCore**. Same three-tool Dev/SRE Toolbox, same ten gates, same model family through OpenRouter. What changes is where everything runs. It ended **SUCCESS** in **25 min 44 s** and **US$ 1.35** of model cost, plus about US$ 0.12 of infrastructure. Final commit `067510f`, 21 files (+2,329 / -49), 54 new passing tests (133 passed, 25 skipped, against 79 before), ten of ten gates, no push, no merge, no deployment. ### What AgentCore changes One run is one AgentCore Runtime session. The graph is the same, the surroundings are not: | Concern | Local run | AgentCore run | | ------------------------- | --------------------------- | ----------------------------------------------------------------------------------- | | Orchestrator | local process | one Runtime microVM per run (2 vCPU, 8 GB), kept alive with `/ping` = `HealthyBusy` | | Workspace and artifacts | local disk | Amazon EFS, `/magi//...` | | Run state and checkpoints | SQLite | DynamoDB, plus S3 for large checkpoints | | Target commands | Docker, `--network=none` | AgentCore Code Interpreter in VPC mode, no internet, per-run EFS access point | | Toolchain in the sandbox | image with the dependencies | wheels installed by the Runtime (which has NAT) into `.git/magi-toolchain` | | Code deployment | `uv run` | image with the locked dependencies, code bundle in S3 loaded at session start | | Observability | files | CloudWatch Logs, 15 metric filters, a dashboard filtered by run id | The sandbox is the part I care about. It has no internet, so the Runtime installs the toolchain for it, and the platform's rule stays the same: a gate is PASS only if a command was executed and succeeded. ### The deployment in the AWS console Screenshots taken after the run, account id blacked out, region `eu-north-1`. **Network.** One VPC, six subnets over two availability zones (public, isolated and private in each), an internet gateway, a NAT gateway and the S3 and DynamoDB gateway endpoints. The Runtime and the sandbox sit in the private subnets, and the NAT gateway is the only way out. ![VPC resource map of the lab VPC: six subnets over two availability zones, route tables, internet gateway, NAT gateway and the S3 and DynamoDB endpoints](/images/magi-a-risk-reduction-system-inside-an-agentic-loop/agentcore-vpc-resource-map.webp) **Runtime.** Version 8 of the `magi_runtime` is ready, with the `DEFAULT` endpoint, started from the image `magi-agentcore:base-1`, an IAM service role of its own and IAM inbound authentication. ![AgentCore Runtime, version 8, endpoint DEFAULT, status Ready](/images/magi-a-risk-reduction-system-inside-an-agentic-loop/agentcore-runtime-version.webp) **Image and code are separate.** The base image (178 MB: Python, git, uv and the locked dependencies, no magi-system code) is pushed once. The code lives in the S3 `code/` prefix, a zip and a small JSON pointer per version, so shipping a fix does not mean rebuilding the image. This run used bundle `1ff47e1`, the first one with the preflight. ![ECR repository with the base image magi-agentcore:base-1](/images/magi-a-risk-reduction-system-inside-an-agentic-loop/agentcore-ecr-image.webp) ![S3 code bucket, code/ prefix, one zip and one pointer per bundle version](/images/magi-a-risk-reduction-system-inside-an-agentic-loop/agentcore-s3-code-bundles.webp) **Observability.** The `magi-runs` dashboard, filtered on this run: model cost against infrastructure cost, tokens, calls and latency, gate checks by verdict, tables per run and per agent. The numbers agree with the platform's own report: 116 calls, 5,474,396 input and 663,977 output tokens, US$ 1.3517 (computed and provider-reported are identical), 1,636 active seconds. ![CloudWatch dashboard magi-runs for the run: cost, tokens, calls, latency and gate checks](/images/magi-a-risk-reduction-system-inside-an-agentic-loop/agentcore-cloudwatch-dashboard.webp) The "Run outcomes" panel shows no data in my capture: the end event fell outside the selected time window. I kept the screenshot as it is. ### The run: a second design round The first run selected a plan in round 0. This one did not. **None of the three round-0 proposals was admissible**, the board went into a revision round, and the plan was chosen in round 1. That cost about 4 more minutes and US$ 0.19 of architect calls, on top of US$ 0.42 for round 0. The architects alone are US$ 0.60 of the 1.35. Why they were rejected is the interesting part. Verification of the 491 claims of the run: 359 supported, 77 refuted, 55 unverifiable. In round 0: | Proposal | Why inadmissible | | --------- | ---------------------------------------------------------------------------------------------------- | | Melchior | 22 critical claims refuted, 2 critical unknowns unresolved (PyYAML not installed, the 400 handler) | | Balthasar | 1 mandatory requirement not covered, 1 critical claim refuted (the protected `.github/workflows/**`) | | Casper | 8 critical claims refuted, 2 critical claims unresolved (no network needed, no payload logged) | Two different causes hide here. The first is real: an uncovered requirement, unresolved unknowns. The second is a defect of the platform, not of the agents. Nearly all the refuted critical claims of Melchior and Casper have the form "the router module exists after the change", or "the documentation page exists". The architects wrote claims in the future tense, the verifier checks the **baseline**, and on the baseline those files do not exist yet. In round 0, `file_exists` had 6 supported and 20 refuted, `text_present` 29 and 11. In round 1 the same kinds were all supported, because the architects restated them in a form the baseline can confirm. The platform did the safe thing, and the policy allowed one revision round, so the run recovered. But a claim about a file the plan creates should be a plan item, not a baseline fact. That is on my list. Can I blame something I changed between the runs? I looked. The MAGI phase code, the prompts, the selection logic and the verification tools are unchanged, and the preflight, the S3 bundle and the toolchain all act before or after the board. The likeliest explanation is model variance on an existing weakness. Not proven: one run per case, no controlled comparison. The selected plan (Balthasar's, round 1, rank score 4.68 against 4.41) mounts the router at the flat `/api/toolbox` prefix behind `verify_credentials`, parses YAML through a custom `SafeLoader` with a depth budget and an alias-expansion budget enforced during composition, dispatches CPU-bound work with `run_in_executor` wrapped in `asyncio.wait_for`, and answers over-limit input with 413 naming the limit. Casper, as usual, asked something the others did not: the original Italian text was not visible to it, only an English translation. ### What it took to get a SUCCESS This was the tenth run on AgentCore and the first SUCCESS. The earlier nine failed for platform reasons, not model ones: | Defect | Fix | | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The sandbox had no toolchain (no pytest, no git) | the Runtime installs aarch64 wheels into `.git/magi-toolchain`, which survives `git clean -fdx` and is not scanned by `flake8 .` | | Heredocs and tools waited on stdin | commands run with stdin closed | | QA scenarios that searched source text or guessed `-k` selectors gave false negatives | text scans and empty selections are advisory, QA can correct a failed scenario once | | Claims written as `app/auth.py` instead of `app/auth.py::verify_credentials` were refuted | rejected at the schema, so the agent retries | | Warm containers kept serving old code after a new bundle was published | the Runtime reports `commit@bundle`, the launcher compares it with the published pointer and stops a stale start before it spends (4 of 5 starts in this run were stale) | | The previous run (US$ 1.44) passed every executed gate and ended FAILED_VERIFICATION: the target had no build or lint command | **preflight at intake**: a mandatory project gate that cannot run stops the run at cost zero, and the target got a `.magi/config.yaml` | The last row is the one I like. A pipeline that spends a dollar and a half to discover that the target has no lint command is not a risk-reduction system. QA also corrected one of its own scenarios. `acc-toolbox-routes-registered` first looked for "toolbox" in `create_app().routes`, which fails because the app mounts sub-applications. QA rewrote it to check `app.openapi()['paths']`, and it passed. The verdict is still the result of an executed command, what changed is that a scenario testing an implementation detail was replaced by one testing behaviour. ### What the gates did not see Ten of ten gates passed, and the patch still has things I would not merge as they are: - **The generated documentation has no code fences.** `docs/14-dev-sre-toolbox.md` has five bare `bash` lines where a fence should open. The `docs.updated` gate checks that documentation changed, not how it renders. QA and a couple of architects noticed it as a minor finding, nobody was required to fix it. - **A failing test was rewritten.** `dev.writer` rewrote `test_document_budget` after it failed with `DID NOT RAISE ToolLimitExceeded`, and reported it itself as a major finding. Reviewers saw the final test and the guard is now exercised, but the platform has no rule that tells "the test was wrong" from "the test was weakened to pass". In the first run the same situation ended well, with a human checking the diff. - **Reviewers saw truncated diffs again.** The pair, Security and all three final reviewers say the diff did not contain the router, `main.py`, models and settings. - **Known minors shipped.** `text.diff` advertises `context_lines` up to 50 while the effective default is 10, so a client following the schema gets a 413. `max_findings` is checked after all findings are materialised. The 504 path has no test and cannot cancel the worker thread. And the empty tag `nginx:` is treated as clean, **the same defect as in the first run**. No gate failed, so no remediation cycle ran. Also, of the 64 mandatory explicit requirements, 29 are demonstrated by an executable scenario. The other 35 are process requirements ("read the repository instructions", "report what could not be verified"), satisfied by what the agents did and reported, not by a command. The platform does not claim otherwise. ### Independent check outside the platform Same idea as before: do not trust the report, run the thing. The patch was applied to a fresh clone of the target, the 133 tests passed, the Docker image built, the container answered `/api/mgmt/ready` with 200, and the three tools answered over HTTP with Basic auth. The k8s tool reports 7 warnings on an `nginx:latest` Deployment as findings, not as an HTTP error, which was the requirement. ![curl call to POST /api/toolbox/runs with k8s.manifest.inspect on a privileged nginx:latest Deployment](/images/magi-a-risk-reduction-system-inside-an-agentic-loop/agentcore-toolbox-k8s-curl.webp) This check exercised the tools with sample inputs, not the edge cases above. ### Two runs side by side | | Local run | AgentCore run | | --------------- | --------------------------- | -------------------------------------- | | Outcome | SUCCESS | SUCCESS | | Wall clock | 19 min 19 s | 25 min 44 s | | Model cost | US$ 1.05 | US$ 1.35 (+ about 0.12 infrastructure) | | Design rounds | 1 | 2 | | Model calls | 105 | 116 | | Claims checked | 290 (42 refuted) | 491 (77 refuted) | | Change | 21 files, +2,340 / -51 | 21 files, +2,329 / -49 | | Tests | 67 new, 146 passed | 54 new, 133 passed | | Mandatory gates | 10 of 10 | 10 of 10 | | Shared defect | empty image tag not flagged | empty image tag not flagged | Two runs, one pass each, a different plan chosen each time. It is still not a success rate. What it does say is that the move to a managed runtime did not change the verdicts: the gates stay executed commands, the evidence stays bound to a commit, and the same blind spots show up in the same places. ## Conclusion On a real FastAPI service and a demanding 230-line change request, magi-system delivered a verified change autonomously: requirements traced to the request, a plan selected by checked facts, code written and corrected by two developer agents, eleven acceptance scenarios and three specialist reviews passed, documentation updated, all evidence re-run on the final commit, and a final review by three architects. About one US dollar, 19 minutes. The change was not integrated. The patch and the local branch are artifacts for a human to review and merge, and that is by design. What I take from the run is not the dollar or the minutes. It is that the two real defects (the empty image tag and the truncated diff) were found by reading the evidence, not by the pipeline. The pipeline is good at not lying about what it checked. It is not yet good at knowing what it did not check. The second run on AgentCore says the same thing from another angle. Different infrastructure, a different plan, a second design round, and the same blind spots: the empty image tag, the truncated diff, a test rewritten to pass. ## Appendix: mandatory gates All ten, on the final commit `5cce890`: | Gate | Verdict | Detail | | ------------------------- | ------- | ----------------------------------------------------------------- | | project.build | PASS | `python -m compileall -q app`, exit 0 | | project.test | PASS | `python -m pytest -q -p no:cacheprovider`, 146 passed, 25 skipped | | project.lint | PASS | `ruff check --select E9,F63,F7,F82 app tests` | | qa.acceptance | PASS | 11 scenarios, no confirmed blocker | | security.review | PASS | 8 findings, none blocking | | sre.review | PASS | 5 findings, none blocking | | docs.updated | PASS | `README.md`, `docs/14-dev-sre-toolbox.md` | | magi.final_review | PASS | 3 reviews, 7 findings, none blocking | | integrity.protected_paths | PASS | no protected change | | scope.write_scopes | PASS | all changes inside authorized scopes | Machine-specific paths in the run artifacts were replaced by placeholders before writing this post. If you want the engine's design first, start from [Autopsy of an Agentic Loop](/posts/autopsy-of-an-agentic-loop/). The gh-aw workflows that started the whole story are in [Git on AI Steroids](/posts/github-agentic-workflows-git-on-ai-steroids/).