Machine actions reference
A machine action is a Fly.io Machines API call that an investigation proposes as part of its mitigation and that stromboli runs only after a named human approves it. The code is in internal/action.
Actions
| Action | Fly Machines API | Waits for started |
Undo, as reported in the result |
|---|---|---|---|
restart |
restart the machine | yes | none needed |
start |
start a stopped machine | yes | fly machine stop <id> -a <app> |
stop |
stop the machine | no | fly machine start <id> -a <app> |
cordon |
stop routing traffic to it; it keeps running | no | fly machine uncordon <id> -a <app> |
uncordon |
route traffic to it again | no | fly machine cordon <id> -a <app> |
clone |
create a copy, in region or the source’s region |
yes (the new machine) | fly machine destroy <new-id> -a <app> --force |
Allowed by default (empty fly.allow): restart, clone, cordon, uncordon. stop and start must be listed in fly.allow.
Statuses
proposed ──approve──▶ running ──▶ succeeded
│ └──▶ failed
├──reject──▶ rejected
├──TTL──▶ expired
└──revised mitigation──▶ superseded
| Status | |
|---|---|
proposed |
waiting for a human, until expires_at (STROMBOLI_ACTION_TTL after creation) |
running |
approved and executing |
succeeded |
every machine was handled |
failed |
a call failed or a machine did not come up; result lists what was done before |
rejected |
a human rejected it |
expired |
nobody decided before expires_at. Applied when actions are read |
superseded |
the investigation revised its mitigation before anyone decided |
Actions that are running when the server restarts are marked failed.
Guardrails
Checked when the action is proposed and again when it is approved:
STROMBOLI_ACTIONS_ENABLED=true(which requiresSTROMBOLI_API_TOKEN).- The service runs on Fly.io (
runtime: fly.ioor aflyblock). - The service’s
fly.actionsnames a connector, and it is afly_machinesconnector. - The action is one of the six above and is in the service’s allow-list.
- 1 to 10 distinct machine IDs (
serviceis required when the investigation covers several services). - Every machine belongs to the service’s Fly app, as listed by the Machines API.
clonenever targets a machine with a volume: the clone would start with an empty volume.- The app has not changed in the catalog since the proposal, and the service still uses the same connector.
Then, at approval:
- an approver name is required;
- the action must still be
proposedand not expired; - it is claimed atomically, so it runs at most once even if two people approve at the same time.
Execution
Machines are handled one at a time, in order. restart, start and clone wait up to 60 seconds for the machine to reach started. Execution stops at the first failure and reports how many machines were left untouched, so a rolling restart never takes down more than one machine that did not come back. A whole action is bounded to 10 minutes.
The result is one line per machine, for example:
restarted 148e123b (started)
cloned 148e123b → 3d8d9a1c in ams (started) · undo: fly machine destroy 3d8d9a1c -a checker --force
Records and notifications
Each action stores who decided (decided_by), when (decided_at), their note (or rejection reason), the result, the error and finished_at. With Slack, the outcome (or rejection) is posted in the investigation’s thread; with buttons, the message shows the action running and then replaces the buttons with the outcome.
Ways to decide:
| Where | Approver recorded |
|---|---|
stromboli actions approve / reject |
--as or STROMBOLI_USER, as given |
ActionService/ApproveAction / RejectAction |
approver, as given |
| Slack Apply / Reject | @<name> (Slack <user id>); limited to SLACK_APPROVERS when set |
MCP clients can list and read actions but cannot decide them.