MAGI: A Risk-Reduction System Inside an Agentic Loop (pt.2)

Table of Contents

• Introduction
• Context
  • The platform
  • Setup of the run
• How the run unfolded
  • The pipeline as executed
  • Timeline
  • Live log
  • Effort by agent
• The MAGI board in detail
  • Independent analyses
  • The three proposals
  • Cross-review with authors masked
  • Preventive review by QA, SRE, Security and the Tech Writer
  • Claim verification and selection
• Implementation: the code written by the developers
  • DEV1, the writer
  • DEV2, the pair
• Verification and reviews of the implemented change
  • QA: acceptance scenarios
  • SRE
  • Security
  • Tech Writer
  • Final review by the three architects
• Functional requirements: from request to evidence
  • Traceability
  • Requirement-by-requirement outcome
  • Findings overall
• Discussion
  • What the architecture changed in this run
  • Threats to validity
  • Cost of robustness
• Second run: the same request on Amazon Bedrock AgentCore
  • What AgentCore changes
  • The deployment in the AWS console
  • The run: a second design round
  • What it took to get a SUCCESS
  • What the gates did not see
  • Independent check outside the platform
  • Two runs side by side
• Conclusion
• Appendix: mandatory gates

Well, here we are again. After the autopsy of the agentic loop I wanted to see the engine on something bigger than a Python upgrade: a long, demanding change request, written by a human, in Italian, for a service that already exists. Not a toy, not a fix. A feature.

Introduction

This is a single end-to-end run of magi-system, an autonomous software-change platform. Three independent “architect” agents (Melchior, Balthasar, Casper) analyze a change request, propose plans, review each other’s proposals with the authors masked, and have every factual claim checked against the repository by tools before code computes which plan is admissible.

Two developer agents (writer and pair) implement the selected plan. QA, SRE and Security verify it, a Tech Writer documents it, the three architects review the final candidate in a fresh context, and a deterministic gate decides the outcome.

The request asked for a three-tool “Dev/SRE Toolbox” inside an existing FastAPI service (pytbak). The run ended with SUCCESS after 19 min 19 s and US$ 1.05:

• all ten mandatory gates passed on the final commit
• a 21-file change (+2,340 / -51 lines) with 67 new tests
• no regression in the 79 existing ones
• no push, no merge, no deployment

In this post I follow the run as it happened: what each agent produced, how each requirement went from the request to code and evidence, and where the safeguards changed the course of the run. Including the places where the platform got it wrong, because there are a few.

Context

The platform

magi-system is a LangGraph application. The graph state is JSON-native and checkpointed in SQLite, agents call an LLM through a provider adapter, and every command that touches the target runs in a container with no network. Three properties matter here:

1. Facts are checked, not voted on. Architects state claims in a typed form (file exists, symbol exists, text present, package version, policy key, command succeeds). A verifier checks them against the baseline commit. A plan that relies on a refuted critical claim, an unresolved critical unknown or a confirmed blocker is inadmissible, however well it scores.
2. Evidence is bound to a commit. Each check result carries the commit it ran on, and the gate reads only the final candidate’s results. Any new commit invalidates earlier evidence.
3. No human in the loop, explicit failure. Every loop has a budget, and every run ends in either a verified candidate or a typed failure state.

Setup of the run

This is not the first attempt. The configuration is the result of eleven previous runs on the same target, and each one exposed a problem that became a fix. This is run twelve.

How the run unfolded

The pipeline as executed

Read it top to bottom and notice where models talk and where code decides. The models write analyses, proposals and code. Admissibility, rank, gates and the final verdict are computed.

Timeline

The MAGI board, the phase that decides what to build, took 5 min 24 s (28% of the run). Implementation took 5 min 51 s. Verification, documentation and the final review took the remaining 8 minutes.

The final review was the single longest step. One architect’s output was truncated by the 32,000-token cap twice (a temperature-0 repetition loop) and needed a technical retry. More on that in the discussion.

Live log

I followed the run live through its event stream (events.jsonl). The commentary below is what I was told during the run, translated and lightly condensed. It shows what an operator sees and has to judge at each step.

20:55:01, run_start. Run 12 started: commit e3920b5, environment container:docker:...pytbak-magi:local, policy default. The three architects start their analysis in parallel, each with read-only tools and without seeing the others’ work.

20:57:05, analysis collected. 135 requirements accepted (111 explicit, 24 derived), every explicit one carrying a verbatim Italian quote of the request. 5 rejected: 3 paraphrased quotes not found in the request, 2 derived “requirements” with no traceable source (for example “protect the endpoints with Basic Auth”: an architect’s interpretation, correctly kept out of the requirements).

21:00:25, selection. Plan selected in round 0: Proposal B, Melchior. Balthasar’s was inadmissible: a critical claim was refuted (“the platform policy lists .github/workflows/** … as protected paths”) and a critical unknown about write scopes was unresolved. Note for later: that claim is true, but it was written as a policy_exists claim on a path, which the verifier looks up as a policy key. It did not block the run, since a plan was selected anyway.

21:03:23, handoff dev.writer. DEV1 wrote the code in 3 min, 4 escalations. Melchior’s structure follows the repo’s own convention: models in app/models/toolbox.py.

21:03:29, check project.test FAIL on be60190. Checked on a copy: 145 passed, 1 failed (test_empty_documents_are_skipped), the same test that failed in run 9 with a different plan and author. A recurring trap with empty YAML documents.

21:06:16, handoff dev.pair. DEV2 changed the test, not the code. That could be a legitimate correction or a weakened test, so I checked the diff: only the expected value changes (documents == 3 to == 2), with a comment explaining PyYAML’s behavior, and the actual assertion (resources_inspected == 1) stays. Verified independently: PyYAML returns 2 documents ([None, dict]) for ---\n---\n<doc>. DEV2 was right, DEV1’s expectation was wrong, and the code was correct.

21:07:03, specialists on d93229e. qa.acceptance PASS (11 scenarios), sre.review PASS (5 non-blocking findings), security.review PASS (0 findings).

21:09:04, check on 5cce890. The Tech Writer has finished, producing a new commit 5cce890. All checks are rerunning on it, and tests are already green. Next come the specialists and then the final review.

21:10:21, specialists on 5cce890. The specialists pass again on the final candidate: QA 11 scenarios, SRE 5 non-blocking findings, Security 8 non-blocking findings. Next is the final review by the three architects, then the gate.

21:12:56, agent_failure magi.balthasar. Balthasar’s final review was truncated twice at 32k, as in run 10 (a repetition loop). A technical retry is running for Balthasar only; Melchior’s and Casper’s reviews are kept.

21:14:20, magi.final_review PASS. Three final reviews completed, 7 findings, none confirmed blocking. run_end: SUCCESS, change_level: verified, final commit 5cce890.

Effort by agent

Input dominates (90% of tokens) because each tool-loop turn resends the transcript. The three architects account for 46% of input tokens. The analysis is the part most exposed to hallucination, and it is where the platform spends most. I think that is the right place.

The MAGI board in detail

Independent analyses

The three architects started from the same snapshot, each with read-only tools (list_files, read_file, grep) and without seeing the others’ work. Each analysis returns an understanding of the mandate, the requirements with their source quotes, typed claims about the repository, assumptions, open questions and risks.

The requirement sets overlap and are merged by id, so the union is the 135 requirements of the run.

Strange… the three analyses converged on the same open questions without seeing each other: the URL prefix (/api/v1/toolbox as written, or /api/toolbox following the repository), whether to reuse the Basic Auth of /api/debug/*, and whether to gate the feature behind a flag like DEBUG_ENDPOINTS_ENABLED.

Their risks show the different perspectives:

• Melchior looked at fit with the existing app: the app-wide 400 handler strips error detail, so limit errors must not use 400. The error-injection middleware (?inject_error=) and the global slowapi rate limit (100/min) apply to the new endpoints and could interfere with tests.
• Balthasar looked at what breaks: new routes change GET /api/mgmt/mappings and make the README’s “31 routes registered” stale without any test noticing. yaml.safe_load_all still expands anchors and aliases, so a hostile manifest can explode in memory. Offloading to the default thread pool shares it with the existing CPU-bound endpoints.
• Casper challenged the scope: custom Prometheus metrics could introduce high-cardinality labels, and limits must be enforced before or during parsing, not after.

The three proposals

All three proposals adapted the prefix to /api/toolbox (the repository has a single /api prefix and no versioning, and the request explicitly allows adapting it), reused verify_credentials, offloaded parsing with loop.run_in_executor after the existing /api/fib/{x} precedent, pinned PyYAML and kept analysis code free of FastAPI. They differed in structure and in a few contracts:

Each proposal also listed its trade-offs explicitly. Melchior’s, for example:

Limit rejections use 413 rather than 400: the app-wide 400 handler would erase the specific reason; 413 keeps the message but is a slightly unusual code for “too many YAML documents”. And: Findings cap is an analysis outcome rather than an HTTP error: preserves “a finding is not an HTTP error” and “no silent truncation”, at the cost of two different limit-handling paths that must both be documented.

Cross-review with authors masked

Each architect scored the other two proposals, never its own, on the six rubric criteria of the platform policy, without knowing who wrote them (labels A, B, C).

Both reviewers of C gave it the same scores and the same reason: the unknown-tool contract is left undecided in the plan, and it carries an unresolved assumption about whether the runtime image installs PyYAML from requirements.txt or docker/requirements.txt. Casper’s note on C: “regression coverage is delegated to pre-existing tests rather than newly implemented”.

Preventive review by QA, SRE, Security and the Tech Writer

Before any code was written, the four specialists reviewed all three proposals: 49 findings in total, plus 16 from the architects’ cross-review.

• SRE raised the same three major findings against every proposal: no per-run timeout or concurrency bound on tool execution; run_id is not propagated to tracing, so response/log/trace correlation is not achieved; a new hard dependency imported at application-import time, so a missing install takes the whole app down. On C it also flagged the undecided unknown-tool contract.
• Security confirmed on all three proposals that CPU-bound tool execution shares the process-wide default thread pool with other endpoints. On C, that PyYAML availability in the runtime image is an unresolved critical assumption (major). On B, that the 404 body for an unknown tool is masked by the app-wide 404 handler. Whether json.validate should echo the parsed value back in the response stayed unresolved.
• QA wrote the 12 acceptance scenarios and raised findings on testability: OpenAPI presence only in the verification plan, not in a test; the no-network guarantee asserted only by static import inspection. Two of its objections were refuted by verification, for example “allowPrivilegeEscalation absent is mapped to severity error” on A, when the plan maps it to a warning. An objection without support is not a veto.
• The Tech Writer required that curl examples must carry the Basic Auth header to be “working” (the request asks for working examples), and that the new settings be documented in .env.example.

Findings are not votes. Each one is resolved by verifying its supporting claims: it becomes CONFIRMED, REFUTED or UNRESOLVED, and only a confirmed blocker excludes a proposal.

Claim verification and selection

The verifier checked 290 claims against e3920b5 in 11 seconds: 240 supported, 42 refuted, 8 unverifiable (operational guarantees and business assumptions, which cannot be checked mechanically and block only when critical). Typical supported claims: verify_credentials defined in app/auth.py; loop.run_in_executor used in app/routers/api.py; APIRouter(prefix="/api" in the existing routers; DEBUG_ENDPOINTS_ENABLED handled in tests/conftest.py.

Code then computed admissibility and rank:

A and B tied exactly. The policy’s tie-break (fewer_planned_changes, fewer_assumptions, magi_order) chose B: 21 planned changes against 23. No vote was taken at any point.

Implementation: the code written by the developers

DEV1, the writer

DEV1 received the selected plan, the requirements and the write scopes, and implemented it in 12 LLM calls and 61 tool calls (3 min). Its handoff is explicit about what is not done:

The work is NOT fully green: the full suite reports 1 failed / 145 passed / 25 skipped, and the README.md update command failed, so README.md is unchanged. Both are reported below as escalations with evidence; I could not fix them in this turn.

I like that. A writer that says “not green” in its own handoff is a writer I can work with.

The code follows the plan closely. The common contract, in app/toolbox/base.py:

class Tool(Protocol):
    """Common contract every toolbox tool implements."""

    name: str
    version: str
    description: str
    input_model: type[BaseModel]

    def execute(self, params: BaseModel, limits: ToolboxLimits) -> ToolResult:
        ...

The explicit registry rejects empty and duplicate identifiers and returns tools in a deterministic order:

def register(tool: Tool) -> None:
    """Register a tool. Raises ValueError on an empty or duplicate identifier."""
    if not tool.name:
        raise ValueError("tool name must not be empty")
    if tool.name in _REGISTRY:
        raise ValueError(f"duplicate tool name: {tool.name}")
    _REGISTRY[tool.name] = tool

The router keeps the four error channels the request asked to keep apart: 422 for invalid tool input, 404 for an unknown tool, 413 for limits, a sanitized 500 for anything unexpected. Findings never change the HTTP status. Parsing runs in the thread pool, and the log line carries only run_id, tool, status, outcome and duration:

    try:
        result = await loop.run_in_executor(None, tool.execute, params, limits)
    except ToolLimitExceeded as exc:
        ...
        raise HTTPException(status_code=413, detail=str(exc)) from exc
    except ToolInputError as exc:
        raise HTTPException(status_code=422, detail=str(exc)) from exc
    except Exception:
        logger.exception(
            "toolbox run failed | run_id={} tool={} status={} duration_ms={}",
            run_id, tool.name, ExecutionStatus.FAILED.value, duration_ms,
        )
        raise HTTPException(status_code=500, detail="Internal server error") from None

Balthasar’s risk about alias expansion was handled in the YAML loader itself:

class _NoAliasSafeLoader(yaml.SafeLoader):
    """SafeLoader that refuses YAML aliases.

    Aliases make it possible to build a small document that expands to a huge
    object graph (a "billion laughs" bomb). The toolbox never needs them, so
    they are rejected instead of being expanded.
    """

The request’s rule “init containers: no probe rules” is a single explicit branch, while security checks still apply to init containers:

        # --- probes: regular containers only, never init containers ---
        if not is_init:
            if not container.get("readinessProbe"):
                add("K8S_MISSING_READINESS_PROBE", Severity.WARNING, ...)

The distinction between missing and explicitly risky configuration is encoded as two codes with different severities: allowPrivilegeEscalation: true gives K8S_ALLOW_PRIVILEGE_ESCALATION_ENABLED (error), while a missing field gives K8S_ALLOW_PRIVILEGE_ESCALATION_NOT_DISABLED (warning).

DEV1 wrote 67 tests in five files:

The sanitized-error test, for example, injects a tool that raises RuntimeError("secret internal detail") and asserts that the response contains neither the message nor a traceback.

DEV2, the pair

DEV2 first assessed DEV1’s candidate independently, without seeing DEV1’s handoff, from the mandate, the plan, the diff and the check results. It raised 11 concerns, among them:

BLOCKING: project.test FAILS. test_empty_documents_are_skipped expects output['documents'] == 3 but the tool returns 2 for '---\n---\n' + GOOD_DEPLOYMENT. And: The custom loader rejects all YAML aliases/anchors, so a syntactically valid manifest that uses anchors is reported as K8S_YAML_INVALID. This is a defensible hardening choice but is a false-negative risk that should be documented. And: The findings cap keeps max_findings-1 findings plus a notice; it is not silent, but confirm the dropped count is reported.

Then it read the whole package, ran a direct yaml.safe_load_all probe in the container, and concluded that the tool was right and the test wrong:

 def test_empty_documents_are_skipped():
+    # PyYAML yields one document per '---' marker: the leading '---\n---\n'
+    # produces a single empty (None) document followed by the Deployment, so
+    # the stream holds 2 documents and only 1 is inspected.
     manifest = "---\n---\n" + GOOD_DEPLOYMENT
     result = _run(manifest)
-    assert result.output["documents"] == 3
+    assert result.output["documents"] == 2
     assert result.output["resources_inspected"] == 1

DEV2 also fixed the order of the docs index and escalated what it could not do: the README update, blocked because a helper script it tried to write at the repository root was outside the write scopes. The Tech Writer later updated the README.

Verification and reviews of the implemented change

QA: acceptance scenarios

QA had defined 12 acceptance scenarios during the preventive review, each linked to requirement ids and to a command. Eleven had a command and were run in the container on d93229e and again on 5cce890. The twelfth (“no superfluous infrastructure is introduced”) has no mechanical check and was assessed in the reviews.

Eight scenarios share the same command, which runs every toolbox test. They are weaker than they look: each one passes if the whole toolbox suite passes, not because a test specific to that scenario passed.

SRE

On the code, SRE confirmed what was solved and what was not. Solved: “Findings cap behaviour matches the no-silent-truncation requirement”, “No new infrastructure, persistence or retrieval endpoint introduced”.

Open, non-blocking: “Observability log line is claimed but not asserted by any test”; “New toolbox limits are not surfaced in .env.example”; an edge case where max_findings=0 still returns one finding (the notice); and an unresolved doubt that the request-size cap relies on Content-Length and could be bypassed with chunked transfer encoding, with the per-field input limit as the backstop.

Of the three major concerns raised before implementation, the hard-import concern remained a design choice (the dependency is pinned), and no per-run timeout or tracing propagation was added.

Security

Security confirmed by tests that the endpoints reuse Basic Auth (401 without and with wrong credentials), that unsafe YAML tags and aliases are rejected, and that internal errors are sanitized.

It raised two non-blocking findings of substance: "json.loads recursion depth is not visibly guarded" (a deeply nested JSON document can raise RecursionError; the request asks for depth limits, and only YAML has one), and the max_findings=0 edge case.

Tech Writer

The Tech Writer read the implemented code, not the plan, and wrote docs/14-dev-sre-toolbox.md (360 lines: feature description, curl examples with Basic Auth, response examples, configuration and limits, how to add a tool, limitations of the Kubernetes checks, architecture note). It also updated the README, the docs index and the CHANGELOG.

Its commit 5cce890 invalidated all previous evidence, and every check was re-run. Yes, for a Markdown-only commit. That is on purpose.

Final review by the three architects

The three architects reviewed the final diff in a fresh context. Melchior and Casper completed in 93 s and 16 s. Balthasar’s review was truncated twice by a repetition loop and needed a technical retry.

The gate recorded “three final reviews completed; 7 finding(s), none confirmed blocking”. Verification confirmed Balthasar’s “json.validate has no recursion/depth guard”, matching Security’s finding independently. It refuted “Pinned PyYAML version differs between plan and diff” as a defect (the plan said 6.0.2, the code pins 6.0.3: a documentation mismatch, not a bug).

And one refutation is wrong. Balthasar reported “Image reference with an empty tag is not flagged”, and verification refuted it. Reading the code shows the finding was correct: for nginx: the last path segment contains :, so the “no tag” branch is skipped, and the empty tag is not latest, so nothing is reported:

            if ":" not in last_segment:
                add("K8S_IMAGE_TAG_MISSING", Severity.WARNING, ...)
            elif last_segment.rsplit(":", 1)[1] == "latest":
                add("K8S_IMAGE_TAG_LATEST", ...)

The finding was minor and non-blocking, so the outcome would not have changed. But it is a real defect that the run shipped and that the platform marked as refuted.

Functional requirements: from request to evidence

Traceability

The request was decomposed into 135 requirements. An explicit requirement is accepted only if its source_quote occurs verbatim (whitespace- and case-normalized) in the request. That is what prevents an architect from inventing a requirement. QA then turned the requirements into 11 acceptance scenarios, each bound to a command run in the container on the candidate commit.

Requirement-by-requirement outcome

Findings overall

The 101 findings raised during the run show where the specialists added value beyond the tests. Before code was written, SRE raised the same three major concerns against all three proposals (no per-run timeout or concurrency bound, run_id not propagated to tracing, a hard import-time dependency on PyYAML). The Tech Writer required every curl example to carry the Basic Auth header to be “working”, as the request demands.

During implementation, DEV1 raised a blocker against its own work (the failing test) and a major finding (README not updated after a failed edit command), both handled downstream.

On the final candidate, 24 findings were raised and none was confirmed blocking. Examples: json.loads recursion depth not visibly guarded; new TOOLBOX_* settings not added to .env.example; an edge case where max_findings=0 still returns one (notice) finding; observability guarantees documented but not asserted by any executed check.

Several final-review findings were refuted by verification, one of them wrongly (the empty image tag). The confirmed ones are recorded in the report as residual risks. They did not stop the run because the policy blocks only on confirmed blockers.

Discussion

What the architecture changed in this run

• Plan choice by facts. Two proposals tied on the rubric score (4.59). The tie was broken by the deterministic rule in the policy, not by a vote. The third proposal was excluded before any code existed.
• Pair review as a correctness check, not a formality. The only failing test came from a wrong expectation in a test written by DEV1. DEV2 fixed the test with a justification that an operator could verify in one command, and no assertion was weakened.
• Evidence invalidation. The documentation commit changed only Markdown, yet every check was re-run on 5cce890. The gate never accepts results from an earlier commit. If it isn’t there, it can’t lie.
• Fresh-context final review. The three architects re-read the final change. In this run they found nothing blocking. In the previous run (run 10) the same step found a real bug that all 12 QA scenarios had missed (summary counts computed before findings truncation).

Threats to validity

I’d rather list them myself than have you find them:

• One run, one target, one model. The same request produced different plans in runs 9, 10 and 12 (Balthasar’s, Casper’s and Melchior’s). The result is an existence proof, not a success rate.
• Self-consistency risk. All nine agents used the same model, so common-mode errors are possible. Tool-based verification and container checks reduce this risk, they do not remove it.
• Verifier coverage. 42 claims were refuted. Most were real model errors, but at least one class is a verifier limitation: a claim that a path is protected, expressed as policy_exists on the path, is refuted although true. It contributed to one proposal’s inadmissibility.
• Lint scope. The configured lint gate checks only syntax-level errors (E9,F63,F7,F82), because the target does not pass full ruff at baseline. Type checking is not configured in the target and was not run.
• Reviewers saw a truncated diff. The specialists and the final reviewers receive the last 60,000 characters of the diff (diff[-60000:] in engine/nodes_verify.py). This change is about 99,000 characters, so the router, models, base, registry, settings and main.py were cut off, and several reviewers said so in their findings (“Provided diff is truncated”). Their conclusions on those files rest on tests and claims, not on reading the code. This is a platform limitation, and it matters for large changes.
• False refutations still exist. Besides the protected-paths claim, a correct final-review finding (nginx: with an empty tag is not flagged) was refuted by verification. A refuted finding drops out of the decision. When the refutation is wrong, a real defect is silently accepted.
• Coarse acceptance scenarios. Eight of the eleven executed scenarios share the command pytest -k toolbox, so they do not discriminate between requirements.
• Operator judgment. The test change by DEV2 was checked by a human observer (me) during the run. The platform itself would have accepted it on the evidence alone.

Cost of robustness

Six model outputs were rejected because they hit the 32,000-token cap. The excerpts of earlier runs show these are repetition loops at temperature 0 (for example “No X. No Y. No Z. …” for about 150,000 characters). The cap plus a retry with feedback recovered every case, at the price of about four minutes of the final review and a share of the output tokens.

Second run: the same request on Amazon Bedrock AgentCore

A single run is an existence proof, so I did the obvious thing: same request, same kind of target (a public FastAPI repository, fastapi-agentcore), but the whole pipeline moved from my laptop to Amazon Bedrock AgentCore. Same three-tool Dev/SRE Toolbox, same ten gates, same model family through OpenRouter. What changes is where everything runs.

It ended SUCCESS in 25 min 44 s and US$ 1.35 of model cost, plus about US$ 0.12 of infrastructure. Final commit 067510f, 21 files (+2,329 / -49), 54 new passing tests (133 passed, 25 skipped, against 79 before), ten of ten gates, no push, no merge, no deployment.

What AgentCore changes

One run is one AgentCore Runtime session. The graph is the same, the surroundings are not:

The sandbox is the part I care about. It has no internet, so the Runtime installs the toolchain for it, and the platform’s rule stays the same: a gate is PASS only if a command was executed and succeeded.

The deployment in the AWS console

Screenshots taken after the run, account id blacked out, region eu-north-1.

Network. One VPC, six subnets over two availability zones (public, isolated and private in each), an internet gateway, a NAT gateway and the S3 and DynamoDB gateway endpoints. The Runtime and the sandbox sit in the private subnets, and the NAT gateway is the only way out.

Runtime. Version 8 of the magi_runtime is ready, with the DEFAULT endpoint, started from the image magi-agentcore:base-1, an IAM service role of its own and IAM inbound authentication.

Image and code are separate. The base image (178 MB: Python, git, uv and the locked dependencies, no magi-system code) is pushed once. The code lives in the S3 code/ prefix, a zip and a small JSON pointer per version, so shipping a fix does not mean rebuilding the image. This run used bundle 1ff47e1, the first one with the preflight.

Observability. The magi-runs dashboard, filtered on this run: model cost against infrastructure cost, tokens, calls and latency, gate checks by verdict, tables per run and per agent. The numbers agree with the platform’s own report: 116 calls, 5,474,396 input and 663,977 output tokens, US$ 1.3517 (computed and provider-reported are identical), 1,636 active seconds.

The “Run outcomes” panel shows no data in my capture: the end event fell outside the selected time window. I kept the screenshot as it is.

The run: a second design round

The first run selected a plan in round 0. This one did not. None of the three round-0 proposals was admissible, the board went into a revision round, and the plan was chosen in round 1. That cost about 4 more minutes and US$ 0.19 of architect calls, on top of US$ 0.42 for round 0. The architects alone are US$ 0.60 of the 1.35.

Why they were rejected is the interesting part. Verification of the 491 claims of the run: 359 supported, 77 refuted, 55 unverifiable. In round 0:

Two different causes hide here. The first is real: an uncovered requirement, unresolved unknowns. The second is a defect of the platform, not of the agents. Nearly all the refuted critical claims of Melchior and Casper have the form “the router module exists after the change”, or “the documentation page exists”. The architects wrote claims in the future tense, the verifier checks the baseline, and on the baseline those files do not exist yet. In round 0, file_exists had 6 supported and 20 refuted, text_present 29 and 11. In round 1 the same kinds were all supported, because the architects restated them in a form the baseline can confirm.

The platform did the safe thing, and the policy allowed one revision round, so the run recovered. But a claim about a file the plan creates should be a plan item, not a baseline fact. That is on my list.

Can I blame something I changed between the runs? I looked. The MAGI phase code, the prompts, the selection logic and the verification tools are unchanged, and the preflight, the S3 bundle and the toolchain all act before or after the board. The likeliest explanation is model variance on an existing weakness. Not proven: one run per case, no controlled comparison.

The selected plan (Balthasar’s, round 1, rank score 4.68 against 4.41) mounts the router at the flat /api/toolbox prefix behind verify_credentials, parses YAML through a custom SafeLoader with a depth budget and an alias-expansion budget enforced during composition, dispatches CPU-bound work with run_in_executor wrapped in asyncio.wait_for, and answers over-limit input with 413 naming the limit.

Casper, as usual, asked something the others did not: the original Italian text was not visible to it, only an English translation.

What it took to get a SUCCESS

This was the tenth run on AgentCore and the first SUCCESS. The earlier nine failed for platform reasons, not model ones:

The last row is the one I like. A pipeline that spends a dollar and a half to discover that the target has no lint command is not a risk-reduction system.

QA also corrected one of its own scenarios. acc-toolbox-routes-registered first looked for “toolbox” in create_app().routes, which fails because the app mounts sub-applications. QA rewrote it to check app.openapi()['paths'], and it passed. The verdict is still the result of an executed command, what changed is that a scenario testing an implementation detail was replaced by one testing behaviour.

What the gates did not see

Ten of ten gates passed, and the patch still has things I would not merge as they are:

• The generated documentation has no code fences. docs/14-dev-sre-toolbox.md has five bare bash lines where a fence should open. The docs.updated gate checks that documentation changed, not how it renders. QA and a couple of architects noticed it as a minor finding, nobody was required to fix it.
• A failing test was rewritten. dev.writer rewrote test_document_budget after it failed with DID NOT RAISE ToolLimitExceeded, and reported it itself as a major finding. Reviewers saw the final test and the guard is now exercised, but the platform has no rule that tells “the test was wrong” from “the test was weakened to pass”. In the first run the same situation ended well, with a human checking the diff.
• Reviewers saw truncated diffs again. The pair, Security and all three final reviewers say the diff did not contain the router, main.py, models and settings.
• Known minors shipped. text.diff advertises context_lines up to 50 while the effective default is 10, so a client following the schema gets a 413. max_findings is checked after all findings are materialised. The 504 path has no test and cannot cancel the worker thread. And the empty tag nginx: is treated as clean, the same defect as in the first run. No gate failed, so no remediation cycle ran.

Also, of the 64 mandatory explicit requirements, 29 are demonstrated by an executable scenario. The other 35 are process requirements (“read the repository instructions”, “report what could not be verified”), satisfied by what the agents did and reported, not by a command. The platform does not claim otherwise.

Independent check outside the platform

Same idea as before: do not trust the report, run the thing. The patch was applied to a fresh clone of the target, the 133 tests passed, the Docker image built, the container answered /api/mgmt/ready with 200, and the three tools answered over HTTP with Basic auth. The k8s tool reports 7 warnings on an nginx:latest Deployment as findings, not as an HTTP error, which was the requirement.

This check exercised the tools with sample inputs, not the edge cases above.

Two runs side by side

Two runs, one pass each, a different plan chosen each time. It is still not a success rate. What it does say is that the move to a managed runtime did not change the verdicts: the gates stay executed commands, the evidence stays bound to a commit, and the same blind spots show up in the same places.

Conclusion

On a real FastAPI service and a demanding 230-line change request, magi-system delivered a verified change autonomously: requirements traced to the request, a plan selected by checked facts, code written and corrected by two developer agents, eleven acceptance scenarios and three specialist reviews passed, documentation updated, all evidence re-run on the final commit, and a final review by three architects. About one US dollar, 19 minutes.

The change was not integrated. The patch and the local branch are artifacts for a human to review and merge, and that is by design. What I take from the run is not the dollar or the minutes. It is that the two real defects (the empty image tag and the truncated diff) were found by reading the evidence, not by the pipeline. The pipeline is good at not lying about what it checked. It is not yet good at knowing what it did not check.

The second run on AgentCore says the same thing from another angle. Different infrastructure, a different plan, a second design round, and the same blind spots: the empty image tag, the truncated diff, a test rewritten to pass.

Appendix: mandatory gates

All ten, on the final commit 5cce890:

Machine-specific paths in the run artifacts were replaced by placeholders before writing this post.

If you want the engine’s design first, start from Autopsy of an Agentic Loop. The gh-aw workflows that started the whole story are in Git on AI Steroids.