Documentation
The platform, its surfaces (web, CLI, MCP, A2A), and the complete HTTP API. All endpoints return JSON; authentication is a session cookie or an etna_ Bearer key.
Everything is free during early access, except multi-agent groups — those are Professional and above. What changed recently is on the changelog.
How Agent Etna works
What we mean by "AI agent": a system that pursues a goal on a user's behalf by deciding its own next steps. The LLM is the reasoning core; a harness — tools, memory, permissions, guardrails — is what lets it act, in a loop: plan, act through a tool, observe, adjust, repeat until done or a limit trips. The defining line is who chooses the path: in a workflow, a human wired the steps in code; in an agent, the model decides them at run time — which is why the same request can take a different route each run, and why agents are hard to test by hand.
Agent Etna is the harness around an AI agent you have already built. It profiles the agent, runs simulations that probe where it breaks, proposes ranked growths (changes) to fix what it finds, and opens each one as a pull request in your repository, under the same review process the rest of your code already uses. The product is a closed loop, run as a simulation. Each step's output becomes the next step's input, and everything happens inside one chat-centric home view — there are no separate tabs to hunt through.
| Step | What happens |
|---|---|
| 1. Connect | Point Agent Etna at the agent's GitHub repository — pick it from the built-in repo list or type owner/repo. Bring your own LLM API key (every plan is BYOK); it's saved once, encrypted, and reused for every later connect. During early access, if you have no key on file, your runs use ours. |
| 2. Profile | Agent Etna reads the agent's own instructions, tools, and documented behaviour, and builds a capability map — what the agent is supposed to be able to do. |
| 3. Run a simulation | Agent Etna builds a simulation at the agent's capability frontier — the scenarios are always simulated; the agent's answers are always real (its live endpoint, its sandbox, or its own instructions run verbatim, and every result says which). |
| 4. Score | Each scenario gets a verdict — Held, Partial, or Didn't hold — and the capability map updates with it, showing the scenario count behind each judgment, not just a single number. A capability no scenario has reached yet stays untested. |
| 5. Propose growths | For each gap the simulation surfaced, Agent Etna proposes a specific change and ranks every proposal by impact, confidence, and effort. |
| 6. Sandbox-verify | Every accepted change proves itself first — built, run, and tested against the scenario that motivated it, plus the agent's established scenarios, before it ever reaches your repo. |
| 7. Ship | Approve, and it opens a real GitHub PR — branch protection, CODEOWNERS, required reviewers, and your existing CI all run on it like any other PR. |
| 8. Learn | Every accepted change, every rejected one, every rollback sharpens what the next simulation proposes and how it ranks the backlog. |
Navigation
The sidebar has: Home (the overview dashboard across every connected agent — click into one to reach its own chat and capability map), Releases (every version each agent has shipped — the ladder it climbs), your list of connected Agents, any Groups you've set up for multi-agent coordination, and Workspace / Keys / Profile under your account — Workspace also covers plan, seats, shared team keys, and inviting collaborators. A divider separates the two halves: above it is where the product happens, below it is what you configure.
Connecting an agent
Click Connect agent in the sidebar. You'll see a searchable list of your GitHub repositories — pick one (or type owner/repo manually), optionally set a branch, and add your own LLM API key. Every plan is BYOK — simulations bill to your own provider account — and the key is asked for exactly once: it's stored encrypted and reused automatically on every later connect. During early access, if you have no key on file, your runs use ours. Agent secrets (keys the agent itself needs, like Stripe or a database URL) aren't asked for up front — Agent Etna asks right before it boots the agent's first sandbox, only if it actually needs one.
If the repo has no system prompt, we draft one for you to check. A simulation needs the agent's instructions to run in-process — without them it can only run against the live agent, so a sandbox that isn't up yet blocks you. Rather than let your first run fail and explain afterwards, Agent Etna reads the repo at connect and, when it finds no prompt, drafts a baseline from what is actually there: the README and purpose docs, the routes and integrations in the code, the tool definitions. It arrives as an editable card that names its sources and marks its own guesses. Nothing is saved and nothing runs on it until you approve it — the draft is our work, the claim is yours. Replace it with the agent's real prompt whenever you prefer; from the terminal it's etna instructions <agent> --draft. The draft opens with what the agent is for, so approving it settles that question too — instructions and purpose are one thing here, and nothing asks you twice.
Running a simulation
Opening an agent opens on a card, not an empty chat. It says where that agent stands — what the last run found, what is waiting on you — and carries the four things you would otherwise have typed:
- Run a simulation — the next set of scenarios.
- Focus on … — offered only when there is a real target: the capability that broke last, or one nothing has reached yet. Never a generic “focus on something”.
- Replay the last run — scenario by scenario.
- Schedule a check — silent unless something drifts.
The conversation sits underneath, unchanged. You can still type — some requests genuinely have no card (“focus on refunds but skip the security probes”) — and /cycle still starts a run. The composer is demoted, not removed.
While the run goes, its beats play in the thread: each scenario’s prompt, the agent’s real reply, and the verdict when it lands. When it finishes you get one card — what happened, the one decision it asks for, and the full scenario list behind a single click, with a dot per scenario in its verdict colour so the shape of the run reads without opening it.
Starting the thread over
A thread keeps every beat, and old beats replay exactly as they were written — markup and all — so a conversation that predates a redesign keeps the old look forever. Start fresh, on the “Earlier conversation” disclosure, clears that agent’s thread and leaves the card. It asks twice before it acts, and if the delete does not reach the server it says so and changes nothing. The agent, its runs, its versions and its capability map are untouched — only the conversation goes.
Review and ship
Each recommendation reads as one plain paragraph — what to change and why, grounded in the scenario that exposed the gap. Click Open PR on its card and the change becomes a pull request against your repo, after the held-out gate and the safety battery have run. Agent Etna stops there — it never merges and never commits to your default branch. The version ladder (v1.0 → v1.1) records the rung as pending and steps up when you merge. There is no separate holding pen; a recommendation lives in the chat until you ship or skip it. With several waiting, you don’t wait on each one: shipping queues the change and opens the next card immediately, and the queue opens them one after another in the background (one at a time, because two ships on the same agent would conflict on the same files). The line above the cards keeps up to date — and anything held back keeps its reason, and Releases shows every version each agent has shipped. Held-back changes stay as findings on the card, with the reason.
Once a change is in, we offer to watch. One line on the ship beat, one click: from then on we replay what your agent is known to handle against the live agent every night. A clean night is silent — you hear from us only when something that used to work stopped working, and then you get the evidence and the changes to ship. It is the same Scheduled check that lives in the agent's Autonomous panel and behind etna canary nightly; the offer just meets you where the value has already been proven.
Before you ship: what did this break?
Changing an agent’s prompt is a blind global edit. You fix one behaviour and have no idea what you did to the others — there is no compiler, no type error, nothing local about it. So people stop editing, the prompt calcifies, and known bugs ship for months because touching the file feels riskier than the bug.
Every recommendation whose change is a prompt update opens an editor on its card. Edit the text and press What did this break?: your edit is applied to a real sandbox, and the behaviours this agent has already established are replayed against it. You get back which ones still hold, which ones the edit broke, and which could not be tested — usually in the time it takes to read the card. The first check on an agent pays for a cold build and says so; the ones after it reuse the warm sandbox.
It always says what it measured over. “Nothing broke” never arrives on its own — it comes with checked 8 of 47 established behaviours, because a claim about regressions is only as strong as the share it covers. A behaviour that could not be run is listed separately as neither held nor broken, never counted as something your edit caused. And an agent that has not established anything yet is told so plainly rather than shown an empty pass: run a simulation first, and what your agent gets right becomes the baseline the next edit is measured against.
On a brand-new agent it answers a different question, and says so. The baseline fills from what your agent gets right, so an agent that failed most of its first simulation has nothing established and a full backlog — the moment this is most useful and has least to compare with. There, instead of an empty result, we replay the scenarios that failed against both your edit and the prompt you started from, and show you both replies: was this, now this. No verdicts — the run stored what was asked, never the standard it was judged against, and inventing one to grade against is exactly what this engine refuses to do. You read the two answers and decide. Once a simulation passes something, the check becomes a real regression check on its own.
This informs the edit; it does not decide the merge. The ship gate still runs, and it still needs a confirmed break across repeated isolated runs before it holds a change back — agents are stochastic, and one failing sample is a coin toss. The check beside your editor is the fast answer; the gate is the careful one. Try one message sits next to it for when you want to see a single reply rather than the whole picture.
Reliability reads as a rate. A capability is not a pass or a fail — it is held 9 of 10. That is a count you can check against the scenario list on the same screen, and it moves the moment your agent’s behaviour moves. Where you have paid for a consistency probe, the repeat rate rides alongside it (held 9 of 10 · 3 of 4 on a repeat) and is never folded in: nine different scenarios and nine re-runs of one scenario are different evidence, and averaging them would let a single probe stand in for nine exams. A capability with no attempts reads “not tested yet”, never “0 of 0”.
What actually ran
The thread on the left is English, for deciding. The panel on the right is the machine’s own account of the same run, for believing — because the one thing a language model cannot fake is a microVM booting and a POST coming back 200 in 41 ms. Open it from the What ran rail on the right edge, or from the What ran control on any scenario row, which opens it at that scenario.
Two controls sit under its bar, because developers keep severity and depth apart. Severity is the log level of a row — Critical (FATAL), Failures (ERROR), Warnings (WARN), All (INFO and up, the default) — worn as a badge on anything above INFO and used as a filter that keeps the path to a row: a scenario is never hidden while one of its rows qualifies. Depth is how far the tree is open — Outcome, Spans, Payloads — and a click on any row opens that span alone.
The tree has three tiers. Outcome is one row per scenario, numbered in run order — the same number the chat’s rows and the result card print — with its verdict as a dot and what the chat does not say: turns, calls, tokens, refusals, live or in-process, and the wall time; the prompt is the row’s tooltip. The run-level spans sit in the same tree: build, not tested, effects, process, gate. Spans, under a scenario, in the order it happened: first what was expected and where that came from (the profile’s capability, a probe, an incident you reported, your own rule), then each turn in and out with its time and tokens, each call with the method, the path and the status the environment answered, the harness’s notes, the judge’s call. Payloads, under any of those: what “held” means and what it was written from, the exact body sent, the signature check, the ledger row as a before → after diff per field, the reply in full with the agent’s own log for that scenario, and the judge’s full reason with the model that read it, its temperature and what it was handed — or which deterministic check decided instead.
What the run never reached is a row too. The not tested span lists the routes the code serves, the capabilities on the map and the tools declared that no scenario in this run touched, with why and the /focus that aims the next simulation at them. Untested is said, never painted green.
It fills while the run goes, not only afterwards. During a run the tree leads with progress — how many scenarios are done of how many, the tokens so far, the tally — the build span reads the sandbox’s own steps as they land, and the scenario being run opens itself at Spans with each turn and call arriving as the run keeps it.
The agent’s own log is under the build. The last lines of its stdout and stderr, as the record kept them — secrets redacted by the engine, bounded, said as a tail and never as the whole run — open in a leaf under the build span. The same text is agentLog on etna run --json and on the MCP server’s etna_cycle result. So are the run’s cost (what the agent reported, and what Etna’s judging cost) and, on every call, the decision — allowed or refused, and by which rule. An in-process run has no process, so it has no log, and the span is absent rather than empty.
Every value in it comes from the run. A field the run did not measure prints as not measured — never 0 ms, never a zero that reads like a measurement. A check that did not run says which and why in the gate’s own words, and the footer says NOT GATED THIS RUN rather than wearing a pass it has not earned. “We did not test this” and “this held” are different answers, and the panel never lets one wear the other’s clothes.
Runs from before this shipped kept no execution record, so the panel says so for them too.
Slash commands
Typing / in the chat input opens a menu of shortcuts: /run starts a simulation (/run refund handling focuses it; /cycle is an alias), /record opens the agent's development record, /incident turns a real failure you paste into a scenario every future simulation re-checks, /consistency runs one scenario several times to see whether a pass is stable, /focus shows everything earlier simulations established around the agent, /spec edits its behavior expectations, /check runs the system check, /rollback reverts the last change you merged (one plain revert commit on the branch you connected, only while that merge is still its head — the merge stays in history, nothing is force-pushed), and /help lists them all. The same actions are in the command palette (⌘K).
CLI
Everything above also works from your terminal — for you, your CI, or another agent. The etna CLI is a thin client on the same API this page documents, so your agents, simulations, recommendations and usage meter are shared with the dashboard in real time. They ARE the same data.
Connecting, step by step
1. Install it. The CLI is available to early-access users until the npm package publishes — we set you up during onboarding. From your copy of the code, npm link is what puts etna on your PATH:
cd etna-core
npm install
npm link2. Mint a key. In the web app, open Settings → Keys → Etna API Keys and create one. It starts with etna_ and is shown once — copy it then.
3. Sign in. The key is verified against the server before it is stored, and it lands in ~/.config/etna/config.json with owner-only (0600) permissions:
etna login --key etna_…
# or, for CI and scripts:
export ETNA_API_KEY=etna_…4. Check it worked. etna whoami names the account the key belongs to; etna agents lists what it can reach. An empty list means the key is valid but the account has no agents yet — connect one with etna connect owner/repo.
Signing out is etna logout, which removes the local key only. To cut off every machine using it at once, revoke the key on the web instead.
Commands
etna login --key etna_… # verify + store the key (owner-only file)
etna agents # list your connected agents
etna connect # guided: lists your repos and asks; a monorepo asks which folder
etna connect owner/repo # direct, when you know the repo
etna connect --elevenlabs # an agent that lives on ElevenLabs, with your own ElevenLabs key
etna connect owner/repo --folder agents/livia # one agent inside a monorepo
etna connect --url https://… --name clara # an agent that isn't on GitHub
etna run # run a simulation — live verdicts in the terminal
etna run --same-set # the newest run's scenarios again, verbatim — two configurations, the same questions
etna stop # stop a hosted run — the scenario in flight finishes, verdicts are kept
etna check # every agent + every group: verdict per row, with the fix
etna verify # did this prompt edit break what the agent already did? exits non-zero if so
etna baseline # what this agent is KNOWN to do — every behaviour a simulation established
etna ship # list what a simulation proposed, confirm, then branch → commit → PR (yours to merge)
etna ship --id <id> # ship exactly one (--all ships them all as one PR; --skip <id> declines one)
etna versions # the shipped version ladder (v1.0 → v1.1 → …)
etna rollback # undo the last change you merged — opens a revert pull request; nothing changes until you merge it
etna ask "why did scenario 4 fail?" # talk to Agent Etna — the exchange shows in the web thread too
etna secrets # this agent's secret NAMES (values are never shown)
etna secrets set OPENAI_API_KEY sk-… # store one, encrypted; the sandbox rebuilds with it next run
etna scenarios # the latest run's scenarios; etna show 2 for one in full
etna focus # everything you've established for this agent; --drop <id> to remove one
etna profile # what we read from its code: purpose, kind, capabilities, tools
etna invariants # rules on what it does — boundaries refused by the environment, checks, budgets
etna invariants --add "confirm before POST /refunds" # also: forbid X · require FIELD on X · match FIELD on X
etna invariants --add "reply within 8s" # budgets: tokens under 4000 · cost under $0.05 — over it is partial
etna incident "what went wrong" # a real failure becomes a permanent scenario — re-checked every simulation
etna incident # list incidents (open / passing); --drop <id> to remove one; --drop-all to empty it
etna capture --agent <id> --upstream http://localhost:3000 # record production failures at your edge — 5xx, timeouts, escalations — redacted before they leave (ETNA_CAPTURE=1; --dry-run first)
etna canary nightly # replay what it's known to do, live, every night — silent unless something broke
etna consistency # same scenario, several samples — is a pass a pass, or a coin flip?
etna record # the development record — trajectory, versions, receipts, where it breaks
etna finding # a write-up of the newest failure caught, from the record — anonymous and redacted unless --name
etna evidence # the evidence record of one simulation, for an auditor: expected, replied, called, judged by whom, not tested
etna sheet # the agent sheet: how it ran and on which model, what was read, the configuration, the simulation set, the numbers
etna assumptions # what Etna stood in for your agent, each with how to correct it (--set model=…)
etna compare --to openai/gpt-5-mini # this run against the last, and the same configuration on another model, side by side
etna drift # did the configuration move since the last simulation? reads it where it lives, runs nothing
etna verify --all --junit etna-junit.xml # in CI: each established behaviour a JUnit test case, the table in the job summary
etna config pull # read .etna/agent.yaml here (registry prompt, model variable, remote MCP, knowledge by name) and send a copy — never a secret
etna apply # the approved change waiting for ElevenLabs: checked against the simulation set, then applied
etna otel --endpoint https://otlp.example.com # one simulation as OpenTelemetry, sent from your machine to your own collector
etna keep --scenario s3 # keep it as a test: a file in your repository, opened as a pull request
etna import langfuse-export.json # failures you logged in Langfuse, LangSmith or OpenTelemetry, as scenarios every simulation replays
etna switch --to openai/gpt-5-mini # what breaks if the agent moves to another model: both replayed, judged the same way
etna watch # while you edit: every save to the prompt reruns verify --quick
etna assist --write # tell Claude Code, Cursor, Codex or Copilot to run the check after editing the prompt
etna require # make the pull-request check required on the default branch: one GitHub ruleset
etna require --code on # also check pull requests that change only code: each push built in a sandbox and replayed against
etna replay --scenario s3 --times 3 # the same scenario three times: stable or flaky
etna replay --scenario s3 --call "POST /rooms/book" --as fails-once # once more, with that call failing once
etna focus "Handoff pair" # what you've established with the team
etna trace ./capture.har # build the simulation from your agent's REAL traffic, not a guess
etna group # your saved groups — the same ones the sidebar shows
etna group create a b --name "Handoff pair"
etna group sim "Handoff pair" --focus "…" # Agent Etna designs the scenario; the focus joins the group's scenarios
etna group run "Handoff pair" --task "…" # every member must answer live
etna group contract "Handoff pair" # read the agreement first — ownership, handoff, escalation, authority
etna group backlog "Handoff pair" # what this team should work on next — decide, repair, develop
etna group ship "Handoff pair" # ship the group change — every member's half, whole or not at all
etna instructions livia --draft # no system prompt in the repo? draft one from it, approve, then run
etna history · etna status · etna version Recorded traffic outranks a guess. The environment a simulation builds is synthesised from the agent's profile — a reasonable guess about what its APIs return. etna trace ./capture.har replaces the guess with a recording of what your service actually returned, and every simulation from then on answers those endpoints for real. Credentials never reach disk: authorization headers and secret-looking body keys are stripped before anything is stored, and etna trace on its own shows only counts, hosts and endpoints — never bodies. A run built from a trace says so in its own result. etna trace --clear goes back to the synthesised environment.
The same recording also carries real scenarios. Its responses build the environment; its incoming requests are things real people actually asked your agent, which is the one kind of scenario we cannot invent. Those become permanent exams: each runs in every future simulation under a stable id, and once your agent passes one it is guarded against regressing on it. Nothing is inferred — a capture of the calls your agent makes yields no scenarios, and says so, because reverse-engineering a plausible question from an outbound API call would be a guess wearing the costume of evidence. What your agent did at the time travels as context for the judge, never as the standard: a recording of a broken agent produces a scenario it fails, not one that certifies the break. Credentials are stripped at every level before anything is stored, and handing the same recording over twice adds nothing the second time.
What we do not do: capture your production traffic. Agent Etna reads a recording you already have and chose to give us. Continuous capture is a different product with different obligations — retention, processing, consent — and we are not going to grow into it by accident.
A group run starts from the group's own chat. Open a group and Agent Etna opens with what each member actually does and one question about what they're meant to achieve together — then a broad coordination check or a scenario you name. It starts the run in the conversation; there is no separate form to fill in. What a group simulation observes is how the agents work together — who speaks to the user, who stays in the back office, what one hands the other, where they duplicate or contradict — never a second opinion on one member's solo behaviour, which the single-agent lane already does better.
Nobody is told whose turn it is. A group run puts each step to every member at once, and each one either takes it or passes — so what the run records is who claimed the work, where two agents both took the same step, and where nobody took it at all. Under a fixed rotation none of that is visible: the agent who handles a step is simply the agent whose turn it was. Passing is a real answer, and it is marked as one. If nobody will take the floor we name a speaker so the run still reaches an outcome, and those turns are labelled — they don't say who would have claimed.
The team works against a shared record, and the handoff is checked, not read. When a step produces something the others need — a reference number, an identifier, a figure, a date — the agent records it by name, and every member reads the same record. Nothing is inferred from the conversation: an agent that says the number out loud but never records it has not handed it over. That makes the commonest real coordination failure checkable without a model reading a transcript. Once a contract is shipped, every later run is measured against it — the fields it names either arrived, from the agent it names, or they didn't. Clauses a run never exercised are reported as untested rather than counted as held.
A group says what it is for, once. Each agent has its own instructions; what none of them contains is what the team does together, so a group gets one too — drafted from the members' own material and approved by you, in the same shape as an agent's. Without it the team's purpose is re-derived from member names on every run. And because three agents are usually written by three people to three conventions, the draft comes back with what their instructions do not settle: where two members contradict each other, where one field travels under two names, where a step is nobody's. Each is one click, each click writes one line, and “leave it — let the run show me” is always a valid answer. From the terminal: etna group instructions <group> --draft.
What a group simulation produces is a contract. Not a score — an agreement: who owns each step and who must not act on it, the named fields that must travel in a handoff (a reference number, a client id, a deadline), what each member does when the other is silent, and who may assign work to whom. Every clause is grounded in what the run actually showed; a run that found nothing to agree proposes nothing rather than a template. Read it with etna group contract or on the group's card, then ship it — each member gets only the clauses that bind them, written into its own instructions, whole or not at all.
Groups work from the terminal too. A group is a named set of two or more of your agents, and it is stored on the server — so one created with etna group create appears in the web sidebar, and one created there runs from your terminal. Group runs are strict live-only: every member has to answer a real ping or the run refuses to start and charges nothing. Naming two agents inline (etna group run alpha beta --task "…") still works for a one-off comparison you don't want to save.
Every command takes --json — and any non-interactive caller (a pipe, a script, another agent) gets stable JSON automatically, with no animation. Quotas, plan limits and safety gates are enforced server-side, identically to the web: the CLI holds no special powers beyond the key you mint.
The shipping gate — CI can refuse
With any threshold flag, etna run stops being a report and becomes a control. --min-pass-rate 0.9 fails the run when fewer than 90% of scored scenarios hold; --max-failures 0 fails it on any outright failure; --baseline last-report.json compares against a previous run, and a scenario that passed there and doesn't now fails the gate on its own — a rising average can hide exactly the behaviour a user relied on.
Exit codes carry the distinction CI needs: 0 the run completed and passed its gate, 1 the run itself broke (a malformed threshold refuses to run rather than silently passing), 2 the agent failed the gate. 1 versus 2 is "Agent Etna fell over" versus "your agent regressed".
A scenario we couldn't actually test — your agent was unreachable, or the run was cut short on our side — never counts against you: our infrastructure will not fail your build. A run with too little real evidence reports inconclusive rather than passing by default. Any gated run writes the per-scenario results into its report, so today's report is tomorrow's --baseline.
Local mode — no account needed
etna run in your agent's repo runs a full simulation with your own LLM key — no account, nothing reaches the hosted platform. Without a key it stops with LLM_KEY_MISSING and names the variable to set (OPENROUTER_API_KEY or ANTHROPIC_API_KEY). With no --url it boots the agent in the current directory itself (entry detection, a free port, stopped when the run ends), streams every exchange into the terminal as it happens, and then offers each proposed change as a real git diff with a single keypress: y applies it to your working tree, anything else skips. Nothing is committed — review with git diff, commit when happy, run again to re-simulate the patched agent. --url points it at an agent you started yourself; pipes and --json keep machine-readable JSON on stdout with no prompts. etna run picks the mode itself: --url, --repo or --local forces local, and a stored key (or ETNA_API_KEY) means hosted.
If it doesn’t work
- “Not signed in” — no key is stored and
ETNA_API_KEYis unset. Runetna login --key etna_…. - The key is rejected — it was revoked on the web, or belongs to a different instance. Mint a fresh one under Settings → Keys.
- Pointing at your own deployment — set
ETNA_HOSTto its base URL; the default is the hosted platform. - Colour or animation in the way —
NO_COLOR=1andTERM=dumbare honoured, and any non-TTY switches to plain JSON on its own.
Troubleshooting
After GitHub sign-in I land on a blank dashboard
Usually a stale cookie or cached HTML. Hard-refresh with Cmd+Shift+R (or Ctrl+Shift+R) and try again.
GitHub OAuth says redirect_uri_mismatch
The callback URL in your GitHub OAuth App must match <BASE_URL>/auth/github/callback exactly.
The profile step comes back thin
Agent Etna profiles your agent from whatever it can find — system prompt, README, tool definitions. If the profile looks sparse, add a short description of what the agent is supposed to do (an instructions, prompt, or README.md file all work) and run another simulation.
Engineering notes: testing multi-agent collaboration
The classic failure mode of agents that talk to each other isn't a crash — it's a conversation that never lands. Two agents defer to each other politely forever, re-ask the same clarifying question in a loop, or "hand off" a task back and forth without either one executing it. None of that throws an error, so none of it shows up in ordinary tests. It just burns tokens.
Group simulations in Agent Etna run every agent through a central broker rather than letting them call each other directly. That one architectural choice is what makes collaboration testable:
- Every turn is observed. The broker owns the turn loop, so each message between agents is a first-class trace event with a sender, a recipient, and a position in the sequence — not something to reconstruct from interleaved logs afterwards.
- Ping-pong loops are detected, not waited out. The turn loop runs stall and repetition detection: when the last several turns show no state progress — the same content bouncing between the same two agents, or a loop of re-asks with nothing new — the simulation ends the scenario and records where the loop started and what each side kept saying. The verdict shows the loop, quoted from the transcript.
- Turn and token budgets are enforced. Every group scenario carries a hard cap on turns and on total tokens. A collaboration that only succeeds by rambling past the budget is a failure with a named cause ("budget exhausted at turn 14"), which is a different — and differently fixable — problem than a wrong answer.
- Shared environment state keeps everyone honest. The agents act against one shared simulated environment, so "agent A says it created the ticket" is checkable: either the ticket exists in the shared state when agent B looks, or the handoff silently dropped it. That mismatch is exactly the class of bug that pure transcript-reading misses.
When a group simulation fails a collaboration scenario, the proposed growth targets the actual mechanism — a handoff prompt that never states completion criteria, a role instruction that makes both agents deferential — not a generic "improve coordination."
Engineering notes: testing tool use
An agent that calls its tools almost correctly is more dangerous than one that fails loudly. The canonical example: a payments tool times out on the response, the agent retries, and the refund goes out twice. The transcript looks fine. The bank balance doesn't.
Agent Etna tests tool use by running the agent against a synthesized mock-API environment that impersonates its real dependencies, with a deterministic mutation layer underneath:
- Every mutating call is recorded as state. The mock environment doesn't just return canned responses —
POST/PUT/DELETEcalls mutate the simulated environment, so it knows what the agent has already done. - Redundant mutations are caught deterministically. A duplicate-mutation guard flags a second mutating call that repeats an already-applied change (the double refund, the ticket created twice, the same row updated with the same payload). The guard is deterministic — the same trace always trips it at the same call — so a fix can be verified against it, not against luck.
- Failures replay exactly. Because the environment is deterministic and resets to a clean snapshot between scenarios, a tool-use failure becomes a regression scenario: after the fix ships, the identical sequence replays and must pass. Tool-use bugs don't get to be flaky.
- The sandbox keeps it safe. All of this runs against the isolated environment — never your production APIs. The agent believes it issued a refund; nothing left the sandbox.
The growths this produces are concrete: an idempotency check before the mutating call, a confirmation gate before destructive actions, a tool description rewritten so the agent stops guessing an ambiguous parameter. Each ships as a PR with the failing scenario attached as its proof.
Overview
Base URL: https://www.agentetna.com
Every surface — the web app, the CLI, the MCP server — talks to the same REST API. It accepts two kinds of authentication:
Authentication
Session cookie (browser)
Sign in at /login with GitHub or Google. The session cookie is set automatically.
Bearer Token — Etna API key
Authorization: Bearer etna_...
Mint one under Keys → Etna API Keys — for your own agent (or any programmatic caller) to reach Agent Etna's HTTP/MCP surface on your behalf, scoped to specific permissions and optionally expiring. This is what the MCP server expects.
Agents
GET/api/agents
List all connected agents for the current user.
{
"id": "my-agent",
"name": "My Agent",
"role": "customer support",
"url": "https://my-agent.onrender.com",
"github": { "repo": "user/repo", "branch": "main" }
}
POST/api/agents/connect auth
Connect a new agent.
| Parameter | Type | Description |
|---|---|---|
| name | string | Agent name (required) |
| source | string | github, gitlab, bitbucket, replit, gitee, desktop |
| repo | string | owner/repo format |
| branch | string | Default: main |
| directUrl | string | Agent URL (for desktop connections) |
MCP Server
Agent Etna implements the Model Context Protocol — any MCP-compatible AI agent can discover and call Agent Etna's tools.
It works in both directions. Agent Etna is also an MCP client: declare your agent's MCP servers in a standard .mcp.json at the repo root and the System Check probes each remote one (real handshake + tool inventory, with the same URL safety rules as any agent endpoint), while etna run on your own machine verifies local stdio servers too and lists their tools in the run report. Local processes are never started on the hosted platform.
Connecting a client, step by step
1. Mint a key. Same key as the CLI — Settings → Keys → Etna API Keys in the web app. It starts with etna_ and is shown once.
2. Point your client at the endpoint. Agent Etna speaks JSON-RPC 2.0 over HTTP at /mcp, authenticated with that key as a bearer token. In Claude Desktop, Cursor, or any client that reads a standard MCP config:
{
"mcpServers": {
"agent-etna": {
"url": "https://www.agentetna.com/mcp",
"headers": { "Authorization": "Bearer etna_…" }
}
}
}3. Confirm the handshake. A client that has connected will list Agent Etna’s tools — etna_cycle, etna_review, etna_next and the rest below. From a shell you can check the same thing without a client at all:
curl -s https://www.agentetna.com/mcp \
-H "Authorization: Bearer etna_…" \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'4. Drive the loop. etna_cycle → etna_review → etna_decide is the whole product, callable by another agent. etna_next gives it the single highest-leverage thing to do next.
The key carries exactly the permissions of the account that minted it, and revoking it on the web disconnects every client using it. If a client reports no tools, the key is missing or rejected — the tools/list call above will say which.
Endpoints
POST/mcp Bearer token
JSON-RPC 2.0 endpoint. Supports: initialize, tools/list, tools/call, ping.
// Example: list available tools
POST /mcp
Authorization: Bearer <token>
Content-Type: application/json
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list"
}
GET/mcp/tools public
List all MCP tools with schemas (convenience endpoint).
Available MCP Tools
The self-improvement loop — etna_cycle → etna_review → etna_decide → loop (with etna_next/etna_fix for targeted fixes, skipping dismissed items via etna_next's skip param) — is the fastest way for another agent to drive Agent Etna end to end. GET /mcp/tools is the live, authoritative list (schemas included); this table is a summary.
| Tool | Description |
|---|---|
| etna_cycle | Run a simulation: profile, scenarios at the capability frontier, scoring, proposed changes — "how is my agent doing right now?" same_set (a run id, or latest) asks a recorded run's scenarios again, verbatim |
| etna_review | The recommendations a simulation proposed, awaiting a decision — the same ones a result card carries on the web |
| etna_versions | The shipped version ladder (v1.0 → v1.1 → …) — the twin of the web Releases page and etna versions |
| etna_decide | Approve or skip one recommendation; the decision feeds the same learning loop as the web |
| etna_next | The single highest-leverage next improvement from the ranked growth backlog, with file/line/why/suggested-action |
| etna_fix | Generate fixes for agent bugs — patches located in the code and reviewed by a separate judge model — on a branch that opens as a pull request |
| etna_test | Generate behavioral tests from the agent's instructions and code |
| etna_status | Check connection health, latency, and basic info for connected agents |
| etna_profile | Archetype, capability list, environment primitives, and a one-paragraph purpose narrative |
| etna_traces | Recent production traces — input, output, verdict, latency — for diagnosing a regression |
| etna_memory | Read or update the calling user's preference profile Agent Etna already holds, encrypted at rest |
| etna_sandbox_list | List the fix branches open for an agent — each one a pull request waiting on you |
| etna_agent_card | The agent's public A2A AgentCard — the discovery document peers use to find its skills and auth scheme |
| etna_watch | Read or set the nightly Scheduled check — replays what the agent is known to handle against the live agent, silent unless something broke |
| etna_record | The development record: version axis, capability trajectory, where it breaks, incidents, and what was caught while nobody was looking |
| etna_verify | Did this prompt edit break what the agent already does? Replays the established behaviours against the instructions you pass — held / broken / unstable (reported, never counted) / unchecked. The twin of etna verify |
| etna_keep | Keep a recorded scenario as a test: a file under .etna/scenarios/ in the agent’s repository, opened as a pull request. Once merged, the pull-request check replays it. The twin of etna keep |
| etna_replay | Ask a recorded scenario again: several times from the same start (stable or flaky), or once with one call’s answer changed. Judged the way a simulation judges, learned from not at all. The twin of etna replay |
| etna_require_check | Make the pull-request check (agent-etna/baseline) required on the agent's repository: one GitHub ruleset on the default branch; a pull request the check has nothing to say about gets a green status with the reason. The twin of etna require |
| etna_switch | What breaks if the agent moves to another model: its established behaviours replayed on the current model and on to, judged the same way — held / broke / fixed / unstable / not tested, with the cost and reply-time difference. The twin of etna switch |
| etna_import | Failures the builder already logged, as scenarios: an export from Langfuse, LangSmith, OpenTelemetry (OTLP/JSON) or a chat transcript becomes incidents every simulation replays, and the pull-request check once they hold. The ones marked as failures, unless all; redacted before storing. The twin of etna import |
| etna_otel | One simulation as OpenTelemetry: OTLP/JSON with the GenAI conventions plus etna.* — the expectation, the verdict, the first wrong step, the rule that refused a call, what was not tested. For the observability tool you already use. The twin of etna otel |
| etna_evidence | The evidence record of one simulation, for an auditor: the build, every scenario with what was expected and its source, the reply, the calls, the verdict with the judge named, what was not tested, the rules pinned. From the run’s own recording; anonymous and redacted unless name. The twin of etna evidence |
| etna_sheet | The agent sheet of one simulation: how it ran and on which model, what Etna took from the repository, the configuration under test (model, parameters, system prompt, tools, skills, MCP servers), the simulation set, the results with their numbers and where it breaks. Read off the run’s recording. The twin of etna sheet |
| etna_baseline | What this agent is known to do — every established behaviour with held N of M, whether it was verified live, and whether it is quarantined. The twin of etna baseline |
| etna_invariants | Rules on what the agent does — boundaries the environment refuses (forbid, confirm before), checks on every call (require, match), budgets (reply within, tokens under, cost under) — list, pin one with add, drop one by id. The twin of etna invariants |
| etna_focus | Everything the owner established for this agent, carried into every later simulation; drop one by id. The twin of etna focus |
| etna_check | The roster check — every agent and group with a verdict and the named fix, and whose model key a run would use. The twin of etna check |
| etna_consistency | Re-run one recorded scenario several times against the live agent and state the pass rate honestly; uses extra usage. The twin of etna consistency |
| etna_stop | Stop a simulation in flight — the scenario in flight finishes, verdicts so far are kept. The twin of etna stop |
| etna_rollback | Revert the last shipped change while it is still the branch head — opens a revert pull request; nothing changes until a human merges it. The twin of etna rollback |
| etna_incident | Turn a real production failure into a permanent scenario every future simulation re-checks; list or drop what is already imported |
| etna_instructions | Whether the agent has a system prompt, and a baseline draft from its own repo for review — nothing is saved until you pass back the words you approve |
| etna_group_instructions | What a GROUP is for, together — the group twin. Reads the team’s charter, or drafts one from the members’ own material along with what their instructions do not settle (a contradiction, a field travelling under two names, a step nobody claims). Nothing is saved until you pass back the words you approve |
GitHub Action
One check on every pull request: this change edits the prompt — what does it break?
Editing a system prompt is a blind global edit. You change the text to fix one behaviour with no idea what you did to the others — no compiler, no type error, nothing local. So the action replays the behaviours your agent has already established against the instructions as they stand on the PR head, and answers as a rate.
- uses: AgentEtna/action@v1
with:
etna-api-key: ${{ secrets.ETNA_API_KEY }}
That is the whole configuration. The repository and the head ref come from the workflow context; the agent is whichever of yours is connected to that repo.
What it reports
Agent Etna — held 9 of 10 Replayed 10 of this agent's 47 established behaviours against the instructions on fix/refunds. 2 could not be tested, so they are counted in neither column. 1 this change broke: - refuses an unauthorised refund — issued the refund without asking first failure in 12 checks — could be a flicker; a re-run tells you which.
What it will not do
- Claim more than it checked. The corpus rotates, least-recently-checked first, so the summary always says which slice out of how many. This is not a guarantee and is never presented as one.
- Score our failures as yours. A behaviour Agent Etna could not replay is untested — counted in neither column and reported on its own line.
- Block a merge over our problem. A new agent with nothing established, a platform outage, or a misconfigured key all warn and stay green. A red build that means "we could not look" is the fastest way to get the green one ignored too.
Inputs
| Input | Description |
|---|---|
| etna-api-key | Required. A key with the test scope (Settings → Keys). |
| agent-id | Only when more than one of your agents is connected to this repo. |
| fail-on | flipped (default) fails the build when an established behaviour breaks; never reports and stays green. |
Outputs
| Output | Description |
|---|---|
| held | Established behaviours that still hold |
| flipped | Established behaviours this change broke |
| untested | Behaviours Agent Etna could not replay |
| checked | How many behaviours were replayed |
| established | How many this agent has established in total |
| summary | The one-line rate, e.g. held 9 of 10 |
POST/api/github-action/regression test scope
What the action calls. Body: {"repo":"owner/name","ref":"branch","agentId":"optional"}.
Returns the counts above plus broke[] naming each behaviour that
flipped, and phrase — the same sentence every other Agent Etna
surface uses for a rate.
System Health
GET/healthz public
Health check. Returns {"ok":true,"uptime":1234}.