Agent reference

An investigation is a deterministic brief, then one LLM conversation with read-only tools, in two phases: triage until submit_mitigation succeeds, then investigate until submit_report. The code is in internal/agent.

The brief

Built without the LLM, for every service of the investigation, and also by stromboli brief, GetBrief and the get_brief MCP tool. Each section degrades to a note when its source is missing or fails.

Window. For an investigation, from 30 minutes before the alert’s time until now, bounded to the last 24 hours. For brief --since <d>, the last <d>. The baseline is the window of the same length right before it.

Section Content
Service the catalog entry, as YAML
Dependencies depends on (downstream), called by (upstream), blast radius
Logs total and error-like line counts against the baseline, and the top 10 error patterns with incident and baseline counts, NEW when absent from the baseline, first-seen time and a redacted sample
Metrics presets marked brief of the service’s metrics connector, against the baseline
Recent changes deploys (or commits) of the service and its direct dependencies, from 2 hours before the window
Fly.io for Fly services: the app’s machines (ID, region, state, volumes), whether stromboli can run machine actions, and the Fly command with its undo for each mitigation
Runbooks the service’s runbooks
Similar past investigations the 3 most recent finished investigations of the service, with their reported cause

Conversation

The first message holds the investigation title, source, services, current time, the alert payload as JSON, and the briefs. The system prompt and tool definitions are static, so providers that cache prompts reuse them across investigations.

Rule Value
Max output tokens per turn 16 000
Tool calls before the agent is told to submit a mitigation now STROMBOLI_MITIGATION_TOOL_CALLS (6)
Tool calls before the agent is told to stop and submit STROMBOLI_MAX_TOOL_CALLS (30)
Tool calls after which the investigation fails STROMBOLI_MAX_TOOL_CALLS + 10
Wall-clock limit STROMBOLI_INVESTIGATION_TIMEOUT (8m)
Timeout per tool call 60 s
Max tool result 24 KiB, then truncated with … (truncated: narrow the query)

Tools called in the same turn run in parallel. A submit_report before any accepted mitigation is rejected and the agent is sent back to submit_mitigation. If the model answers twice in prose without calling a tool, its text becomes the report summary (cause unknown, confidence low). A refusal or hitting the output token limit fails the investigation.

Investigation tools

All read-only. since_minutes and until_minutes count back from the investigation’s start.

Tool Arguments Returns
list_services every service with its dependencies, tier, and whether it has logs and metrics
get_service service the catalog entry, upstream, downstream and blast radius
log_patterns service, filter (RE2; default error-like, . for all), since_minutes (30, max 1440), until_minutes (0) log patterns in the window compared with the baseline (top 15)
query_logs service, filter, since_minutes (30, max 1440), until_minutes (0), limit (30, max 100) raw lines, newest first, each truncated to 500 characters and redacted
log_timeline service, filter (default error-like), since_minutes (60, max 1440), buckets (12, max 24) matching line counts per time bucket
get_metrics service, metric (a preset) or query (PromQL with $selector), since_minutes (60, max 1440), until_minutes (0) series with min, max, avg, last, baseline average and 12 points, at most 10 series
list_metrics service the connector’s presets and the raw metric names with series for the service in the last hour
recent_deploys service, since_minutes (240, max 10080) deploys, or commits when no workflow or environment is configured
get_commit service, sha message, author and changed files with truncated diffs
read_runbook path the runbook’s Markdown
search_past_investigations service, query (in the title or report) up to 5 finished investigations, newest first, with their cause

Log filters are case-insensitive. Tool output (log lines, patterns, commit diffs) is redacted before the model sees it: see Redaction.

Submit tools

submit_mitigation

Phase 1. Can be called again to revise the mitigation; each call republishes it (and supersedes a pending machine action). Fields, all required except machine_action:

Field
kind rollback, restart, scale_up, drain, disable_feature, block_traffic, failover, degrade, none
target service, deploy SHA, region, instance, flag or dependency
action the exact command(s) a human runs
rationale which symptoms point to it
verify how to tell within minutes that it worked
undo how to revert it
risk safe, reversible, destructive
confidence low, medium, high
impact current user impact
machine_action {service, action, machine_ids, region}: only when the brief says stromboli can run actions for the service. See Machine actions

When machine_action is missing but action holds a fly machine restart|clone|cordon|uncordon|stop|start command stromboli could run, it is filled in from the command. A machine_action that fails the guardrails is dropped: the mitigation is published as a suggestion and the agent is told why.

On success: the mitigation is stored, mitigated_at is set (on the first one), it is posted to Slack, and the investigate model takes over.

submit_report

Phase 2. Ends the investigation. Fields, all required:

Field
summary what is happening, for someone joining the incident
likely_cause one or two sentences, or unknown
confidence low, medium, high
evidence at most 4 observations
ruled_out at most 4 rejected hypotheses, with the reason
suggested_mitigation confirmed, what to do instead and why, or none
fix the long-term fix, or unknown
next_steps at most 3 things a human should check

Mitigation kinds

Kind When the agent picks it
rollback errors started right after a deploy of this service
restart stuck, leaking or wedged processes: memory climbing, OOM kills; a full root filesystem
scale_up saturation under load: CPU near 100%, throttling, concurrency at its limit; legitimate memory use; a full volume (extend it)
drain one region, zone or instance bad while others are healthy
disable_feature a flag-guarded change
block_traffic abusive or malformed traffic
failover a failing dependency with a standby
degrade shed non-essential work to keep the core working
none nothing is safe, or no evidence of user impact

Redaction

Applied before data reaches the model or is stored: raw log lines, log patterns and their samples, deploy titles, commit messages and diffs, and metric query errors.

Matches Replaced by
Bearer …, Basic … credentials Bearer [REDACTED]
JWTs [REDACTED_JWT]
AWS access key IDs (AKIA…, ASIA…) [REDACTED_AWS_KEY]
GitHub tokens (ghp_…, github_pat_…) [REDACTED_GITHUB_TOKEN]
Stripe keys (sk_live_…) [REDACTED_STRIPE_KEY]
sk-… API keys [REDACTED_API_KEY]
Slack tokens (xox…) [REDACTED_SLACK_TOKEN]
values of keys named password, secret, token, api_key, access_key, private_key, client_secret, authorization, cookie… [REDACTED]
credentials in URLs (scheme://user:pass@) scheme://[REDACTED]@
email addresses [REDACTED_EMAIL]

Audit trail

Every brief, tool call (input, output, duration, error), model turn (task, model, stop reason, input, cache read, cache write and output tokens), mitigation, report and error is stored as a step, in order. Token totals are added to the investigation after each turn.

Edit this page on GitHub ↗