# Praxen - Full Content
> Detects production issues and fixes them by opening pull requests.
Praxen is built by Praxen Inc. in San Francisco, founded by Boris Tane (built Baselime, acquired by Cloudflare; led observability at Cloudflare). Signing up is one command: `curl -fsSL https://getpraxen.app/setup | bash` installs the CLI, connects the coding agents on the machine, and creates the account. What the script does: https://getpraxen.app/setup. See https://getpraxen.app/llms.txt for the index version of this document, including the API endpoints agents can call.
## The loop
Praxen runs a closed loop over production software, in five steps: **Context** (agents build the context graph live: every resource and dependency across your clouds, observability providers and codebase), **Remediation** (agents fix the issue rather than the symptom: Praxen never changes production on its own, so the fix is investigated first, written on a branch and validated, then arrives as a pull request with the full evidence trail; your review and CI still gate the merge), **Review** (agents review every pull request against the production it deploys to, whoever opened it, walking the graph from the repository to the resources behind it and leaving one comment updated in place on every push, with a pass the default and a blocking check available), **On-call** (agents answer the questions on-call gets, in the thread where they are asked, with every claim linking back to the query, log line or change record behind it, in shareable transcripts), and **Coding agents** (your coding agent gets the same production context over one MCP endpoint, the context graph, telemetry queries, deploys and code search, reads by default with writes behind a default-deny policy).
The homepage walks the five steps as five reconstructions of the product at work, in the console, on a pull request, in Slack, and in a terminal, across three services. Context is the live context graph around a Cloudflare Worker named checkout-edge, with one incident's blast radius on it: checkout-edge, hd-prod, payments-db on PlanetScale, cart-svc, edge-gateway. Remediation is the pull request screen: the diff that restores the Hyperdrive pool, the executor that wrote it, CI green, and a Merge button waiting on a person, because Praxen never merges on its own. Review is a production-impact review on a GitHub pull request in orders-api: a migration adds CREATE INDEX without CONCURRENTLY, which holds a write lock on the orders table for the whole build while checkout sustains about 38 writes a second on it, so the comment names the file and line, attaches the write-rate chart with the projected lock window, and says to build the index concurrently instead. On-call is a Slack thread: asked whether checkout is slow because of them or payments-api, Praxen answers that checkout-edge Wall Time P99 is 18x its own baseline with requests queuing for a Hyperdrive connection while payments-api is flat at 180ms, and names deploy 9f3c2a1 for shrinking the hd-prod pool from 50 connections to 5; asked how long it has been queuing, it attaches the pool's queue-depth series for the last 30 minutes and says it is already working on the pull request that fixes it. Coding agents is a coding agent in a terminal, asked to make region a required field on the checkout request schema: over Praxen's MCP endpoint it searches the callers of POST /checkout and queries request shapes, finds that cart-svc and edge-gateway still send region-less requests (41,200 in the last 24 hours), and proposes defaulting the field and migrating both callers before requiring it.
## Product capabilities (full)
### Context graph
Nobody remembers how production is wired. Praxen keeps your code and cloud in one live graph, so your coding agent stops guessing too.
### Issue detection
No thresholds to tune. Praxen reads the charts and speaks up only when something is actually wrong.
### Alert intelligence
Forward your alerts. Praxen reads the query behind each one, tells you which are real, and keeps digging into those.
### Impact intelligence
Praxen reviews every pull request against the infrastructure it deploys to. The diff that would cause tonight's incident gets flagged before it merges.
### Investigations
Praxen works out the cause one step at a time. Every step shows its evidence, or admits it doesn't know.
### Instrumentation
You can't debug what you didn't log. Praxen finds the blind spots in your code and closes them.
### Slack agent
Ask about production in any channel and get an answer grounded in your real infrastructure.
### Autofix
Praxen writes the fix only once it has proven the cause, and checks it against the signal that raised the issue before you see it.
## Use cases (full)
The same loop underneath every scenario: it learns your system first (The context graph connects every resource and dependency across your clouds, repos and observability providers, so Praxen reasons over your real system instead of guessing.) · detection without thresholds (Built-in checks for every provider, plus checks built from your own saved queries and dashboards. Statistics and an agent decide together, and a metric getting better never raises an issue.) · investigations that cite their sources (Every claim links back to the query, log line or deploy behind it. Without evidence, the verdict is inconclusive.) · writes are earned, never assumed (Accounts connect read-only, and write calls are refused until you turn read-only off. Code changes go through your normal review.) · it gets sharper every week (It remembers what it confirms, keeps daily notes, and re-checks its monitoring queries against real data, so every incident makes the next one faster.)
### Praxen for impact analysis
URL: https://getpraxen.app/use-cases/impact-analysis/
Catch issues before they break production. Praxen analyses the potential impact of every pull request on production. It checks each change against your live infrastructure and tells you what could go wrong before you merge.
- Every pull request analysed against live production
- A comment that explains the risk, shows the evidence and suggests the fix
- A check you can make required, so the risky merge is blocked
**Praxen analyses every pull request for its impact on production** It reads the change against the infrastructure it deploys to. If production would break, it comments with the cause, the evidence and a fix, and the check fails. Make the check required and the merge is blocked.
**It adds missing observability to your code** When a route can't emit one useful event per request, it adds the instrumentation using the setup you already have, as a small change of its own that can merge first.
**It keeps watching after the merge and finds what broke** Praxen watches production after every deploy. When something regresses, the issue arrives with the change that landed just before it.
- **How does production impact analysis work?** A GitHub app analyses each pull request against the live infrastructure it deploys to. The result lands as a comment plus a 'Praxen production impact' check.
- **Will it block my merges?** Only if you want it to. Require the 'Praxen production impact' check through branch protection and risky merges are blocked. Leave it optional and the verdict is advisory.
- **Is this for AI-generated code?** It's for all code, but it matters most when nobody read the change closely. AI-written or hand-written, the same analysis catches the mistake.
- **How is this different from CI?** CI runs your tests against the code. Praxen analyses the change against production: the config, capacity and dependencies it lands on, and what they emit right now.
- **Will it ever merge or deploy on its own?** Not by default. Every change goes through your review, and anything that merges or touches production on its own is something you switch on.
- **How often does it fail a pull request?** Rarely. Pass is the default, and a fail has to name how a line in the diff breaks production. “Might” does not count.
- **Does it comment on every push?** One comment, updated in place. Documentation and test-only changes pass straight through without one.
- **Does the instrumentation it adds bring new dependencies?** No. It writes with your logger, your OpenTelemetry setup and your conventions, and it never logs a secret.
---
### Praxen for DevOps
URL: https://getpraxen.app/use-cases/devops/
See everything you run in one context graph. Praxen connects your clouds, repos and observability tools into one context graph. It records every change and watches the crons and queues that never send an alert.
- Every resource across your providers in one live graph
- Every change on the record, with when it landed and what it touched
- Queues, crons, and workers watched like user-facing services
**Praxen connects all your providers into one graph** AWS, Cloudflare, Vercel, Render, Fly.io, Kubernetes, PlanetScale, Supabase and Modal connect through hostnames, DNS and IPs. No tags, no diagrams.
**It watches your crons and queues and tells you when they stop** Queues, scheduled jobs and background workers fail quietly. Praxen watches them like user-facing services and raises an issue when one stops.
- **Which providers does Praxen support today?** Praxen supports clouds, observability tools, source control and coding agents. The integrations page has the full list, including what's coming next.
- **How does it link resources across clouds?** The way traffic actually flows: hostnames, DNS records, and IP addresses matched across providers, environment variables that reveal dependencies, traces where you have them, and infrastructure-as-code that declares what deploys where.
- **Do I need to tag resources or draw the topology?** No. Connect each account read-only and the graph builds and maintains itself. You can correct or annotate anything.
- **How much access does each cloud need?** The minimum, read-only by default: Scoped roles and tokens on every provider. Write access is a separate, per-account decision.
- **Which background systems does it watch?** Whatever your providers run. If it's in a connected account, it's in the graph. Each integration's page at getpraxen.app/integrations lists the resources it syncs.
- **How is this different from a CMDB?** Nobody has to maintain it. The graph syncs from your providers and re-ranks what matters every day, so it stays current.
- **Can my coding agent read the graph?** Yes, over the MCP server or the CLI. Claude Code, Cursor, Codex and the rest query the same graph Praxen investigates with.
---
### Praxen as your AI SRE
URL: https://getpraxen.app/use-cases/ai-sre/
An AI SRE that fixes what it finds. Praxen does an SRE's work: it learns your system, reviews changes, handles incidents, fixes bugs and keeps CI green.
- Learns your system and watches production on its own
- Reviews changes, handles incidents, fixes bugs and keeps CI green
- Shows the evidence behind every conclusion
**It learns your system before anything breaks** Praxen connects your clouds, repos and observability tools into one context graph, so it knows what depends on what before something goes wrong.
**It reviews changes before it ships** Every pull request is checked against the production it deploys to. If it would break something, Praxen says why before you merge.
**It handles incidents while you sleep** Every alert is checked, repeats are grouped, and real incidents are investigated with the evidence attached.
**It turns bug reports into fixes** Hand it a bug report and it proves the cause before it writes the fix, or passes the work to Devin, Cursor or Factory.
**It keeps CI green when main breaks** When a build fails on main, Praxen finds the commit that broke it and writes the fix.
- **What is an AI SRE?** An AI SRE does the operational work of keeping production healthy: it watches your systems, triages alerts, investigates issues and fixes them. Praxen does all four and shows the evidence for each conclusion.
- **Does an AI SRE replace my on-call rotation?** It takes the first pass at everything, so people step in only when something needs a decision. Praxen isn't a pager: it tells you by email, Slack and the console, and each issue arrives already investigated.
- **Is it safe to give an AI SRE access to production?** Accounts connect read-only, and write calls are refused until you turn read-only off for that account. Code changes go through your normal review.
- **How is an AI SRE different from an observability tool?** Observability tools store your telemetry and show it to you. Praxen reads it, decides what matters, finds the cause and writes the fix.
- **Does it only diagnose, or does it fix too?** It fixes too. Once the cause is proven, Praxen writes the fix and tests it, ready for your review.
- **Do I have to set thresholds?** No. Praxen judges each signal against how that resource normally behaves at that hour on that day, and assumes nothing is wrong until the data says otherwise.
- **Will it wake me for a blip?** No. A change has to be big, sustained and heading the wrong way, and repeats fold into one issue.
- **What happens to an alert it dismisses?** It's kept with its reasoning, and one click sends it back for investigation. An alert that fires again gets triaged again instead of dismissed twice.
---
### Praxen for Continuous Integration
URL: https://getpraxen.app/use-cases/continuous-integration/
Fix broken builds before they block your team. When the build breaks on main, Praxen finds the commit that broke it and writes the fix, so your CI stays green without anyone digging through logs.
- Every failed build on main investigated
- The commit that broke it, found from the logs
- A tested fix, ready for you to review
**It watches main for every failed build** A failed or timed-out run on main or in the merge queue becomes an issue the moment it happens. GitHub Actions, Depot and any CI that reports to GitHub.
**It reads the logs and finds the culprit** Praxen reads the failing job, the commit that broke the build and everything else that changed, and shows how they connect.
**It writes the fix and tests it first** The fix runs through your build and tests before you see it. When main passes again, the issue closes itself.
- **Which CI providers does Praxen support?** GitHub Actions, Depot, and any other CI that reports checks to GitHub. GitLab CI and CircleCI are coming. The current list is at getpraxen.app/integrations.
- **Does it fix CI failures on my pull requests?** No. Praxen acts on your default branch and merge queue, where a red build blocks everyone. On your own branches the red check already tells you. When CI fails on a fix Praxen wrote, it fixes that too.
- **What does Praxen read when a build fails?** The failed run, the failing commit and the job logs, along with your code, recent changes and the rest of the context graph.
- **Can I approve fixes before Praxen starts?** Yes. Fixes start on their own by default. Switch autofix to approval and each one waits until someone presses Run.
- **Will it merge the fix?** Not by default. The fix goes through your normal review.
---
### Praxen for ticket resolution
URL: https://getpraxen.app/use-cases/ticket-resolution/
Turn bug reports into fixes ready to review. Praxen investigates each bug report using your production context. It proves the cause before it writes the fix.
- Bugs investigated with production context, not guesses
- A causal chain where every link cites its evidence
- The fix comes back ready to review, or goes to Devin, Cursor, Factory or Conductor
**Report the bug right from Slack** Praxen already has your infrastructure, telemetry, recent changes and code, so there's nothing to explain. It turns the report into an issue and starts investigating.
**It finds the cause and proves it** Each step cites the query, log line or deploy behind it, or says what is still unknown. No proven cause, no fix.
**It writes the fix and explains it** Praxen writes the change on a branch, validates it and explains its reasoning. Prefer your own agent? Connect Devin, Cursor, Factory or Conductor.
- **How do I hand Praxen a ticket?** Mention @Praxen in Slack, start a thread in the console, or use the CLI or your editor over MCP. Praxen already has the graph, the telemetry, the recent changes and the code.
- **What if it can't find the cause?** It says so. An investigation ends resolved, diagnosed or inconclusive, and anything that needs a human decision becomes a single escalation.
- **Does it learn from past tickets?** Yes. What it confirms on one ticket is remembered and used on the next, and daily notes keep a record of what happened.
- **Who writes the fix?** Praxen's own executor by default, or connect your own coding agent and every autofix routes through it. Either way, you review it before it merges. Supported agents are at getpraxen.app/integrations.
- **Which repositories does it work with?** Any repository you connect through source control. The fix lands on a branch in the repo behind the affected service, with the investigation linked from the change. Supported providers are at getpraxen.app/integrations.
- **Can I ask for a fix from Slack?** Yes. Mention @Praxen in any channel or DM. Updates come back to the same thread: failed CI, the merge, the close. Reply there to steer it.
- **Does it test several theories at once?** Only when you ask. By default one agent does the work; ask for a swarm and several agents test the open questions in parallel.
- **What if there is nothing to fix?** Praxen says so. It won't invent a diff to look busy.
- **What happens to a fix nobody reviews?** One reminder after a week, then Praxen closes it and says why. The branch stays, so reopening brings the change back unchanged.
## Blog posts (full content)
### How to build a memory system for proactive agents
URL: https://getpraxen.app/blog/how-to-build-a-memory-system-for-proactive-agents/
Date: Oct 22, 2026
Author: Boris Tane
Description: Proactive agents run without a prompt, so each run has to start from what earlier runs learned. How we built memory for Praxen's agents on git: one repository per team, a nightly clean-up, and lessons from merged pull requests.
import DemoVideo from "../../components/DemoVideo.astro";
import MemoryRepoExplorer from "../../components/blog/MemoryRepoExplorer.astro";
import { REPO_FILES } from "../../data/blog/git-memory";
What happens when an agent investigates the same failure twice? It does all the work again. It reads the same logs, tests the same hypotheses, and ultimately, it lands on the same answer. This is not only slow, but also not token efficient, investigation agents need to remember old incidents, resolutions, and team preferences.
At [Praxen](/), our agents keep that memory in a git repository, one per team, hosted on [Cloudflare Artifacts](https://developers.cloudflare.com/artifacts/). They edit markdown files in it while they work, and every night, an agent does what we call dreaming: it verifies what it's in the memory against ground truth, deletes what is no longer true and re-organises the repo to facilitate navigating it by other agents.
Figure 1
How Praxen's memory works
```mermaid alt="Agent threads write to a git repository, a nightly memory update rewrites it, and new threads start from its index"
graph TB
T["Agent threads"] -->|"save what they learn"| R["Memory repository"]
S["The day's threads and PRs"] --> D["Nightly memory update"]
D -->|"notes, fixes, index"| R
R -->|"index and daily notes"| B["Context of the next thread"]
B --> T
style R fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
## What Praxen needs to remember
Every agent run learns something the next one could use:
- **Root causes it already found.** If an investigation confirmed that checkout-api times out because the payment vendor's sandbox can't handle parallel load, the next one shouldn't have to find out again.
- **Known noise.** The alert that fires every night during the batch job and that nobody acts on.
- **How the team changes our code.** If developers keep rewriting tests in the PRs Praxen opens to use their fixtures before merging, the next pull request should just use the fixtures.
- **How the team wants to work.** How to behave in a Slack channel, etc.
- **What happened recently.** The deploys, incidents and patterns of the last few days.
- **Tribal knowledge.** Things you can't read from telemetry, like "this alert is expected" or "that service is being retired".
On top of that, the memory has to be shared by the whole team, and it has to be easy to correct. A fact that was true last month can be wrong today.
## Why git
We started with vector embeddings, like most agent memory does. We quickly realised it was not the right abstraction. A wrong fact was hard to find and harder to fix, there was no history of who wrote what, and the agent only ever saw whatever the similarity search decided to return.
Once we had that list, the requirements looked a lot like source code. We wanted history, to see how a fact changed and why. We wanted diffs, so a thread can incrementally get changes. We wanted lots of writers at the same time, as there could be multiple concurent investigations.
That's git, and agents are already very good at git. Give the agent a shell and it can grep, read files, edit them and look at the log, etc.
It also means we don't need a search index. A root file `MEMORY.md` can link to the files in the memory repository, and it's the entry point for the agent when exploring the memory.
## What's in the repository
Each workspace on Praxen gets one repository with a common structure: a main `MEMORY.md` file that outlines all other files, daily notes which summarizes the agent activity daily, individual files about specific topics, and preferences files for how people like to interact with Praxen in Slack or on Github for example.
Preferences live in the same repository as facts: instructions for how the agents should behave in a Slack channel, or what to look for when they assess the impact of a pull request in a repository. These are just files that agent can read at any point in time. Engineers can edit those files in the web console or via MCP, which builds a real git commit and pushes it, so a change to a channel's instructions has the same history, diff and revert as any other memory. When a user then starts a thread from a Slack channel, the preferences for this channel are loaded into the prompt dynamically.
Everything else is a topic file, and the agents follow a few simple rules for those: one fact per file, a first line that says what the file is about, and the name of the resource and a date on every entry. Here's what one looks like after a couple of investigations:
```md
checkout-api: 504s caused by the payment vendor's sandbox
- 2026-10-14: checkout-api integration tests fail intermittently because the
payment vendor's sandbox returns 504 under parallel load. Confirmed from
pipeline logs. [thread_7c2e, 2026-10-14]
- 2026-10-21: the same 504s show up in production traffic after deploys that
warm the connection pool. [thread_91ab, 2026-10-21]
```
That's it. No schema, no embeddings. Anything that can run `cat` can read it, and so can you.
## Every thread shares the same memory
Praxen runs a lot of threads at the same time: an investigation into an alert, a review of a pull request, someone asking a question in Slack. They all work on the same repository.
The first time a thread uses its sandbox, we clone the memory repository into it at `/workspace/memory`. From there the agent uses the same tools it uses on any codebase. It greps for the service it's looking at, reads what earlier threads found, and edits markdown when it confirms something new.
What it can't do is push. There are no git credentials in the sandbox, on purpose. An agent that spent the day reading logs and web pages should not be writing straight to a team's shared memory. So the agent only changes files, and everything that touches the remote happens in code, after its turn ends.
## Every night, Praxen dreams
Agents write to memory all day, and left alone that turns into a mess: duplicates, facts that stopped being true, and an index pretty much impossible to read. So once a day, every workspace gets a memory update, we call it dreaming.
Dreaming starts from what happened during the day. After every turn, a small model summarises each thread in two parts: what's worth remembering for weeks, and what this thread found that will probably be stale soon.
Dreaming consists of fetching the last 24 hours of summaries and the pull requests people merged or closed from the ones Praxen opened. Then it works through the same set of checks and tasks every night:
- **Daily note.** It writes a short note about the day: only what someone would still find useful a week later.
- **Outcomes.** It turns how Praxen's pull requests ended into lessons for that repository.
- **Corrections.** It checks a handful of files against production and fixes what's wrong.
- **Tidy.** It merges files that say the same thing and repairs broken links.
- **Index.** It rewrites `MEMORY.md` so it stays short and points at everything else.
- **State.** It records what it checked, so tomorrow's run knows where to start.
Each task is its own commit. If one of them goes wrong, you revert that commit and keep the rest.
Most of that is housekeeping. Two tasks are where the memory actually gets better: checking it against production, and learning from how work ended.
### Checking memory against production
A fact that was true last week can be wrong today. The service moved region, someone raised the connection limit, the cron job was deleted. Our memory system needs to accomodate this, otherwise, it slowly fills up with things that used to be true.
So every night, the dreaming process picks files and checks every claim in them against the real system: the cloud APIs, monitoring queries, the code, etc. True claims remain, and wrong ones get fixed, with the evidence in the commit message. If a claim can't be fixed or verified, it gets deleted.
```mermaid alt="Each claim the nightly update checks is kept, fixed, deleted within the cap, or kept and marked unverified"
graph TB
C["A claim to check"] -->|"checkable"| R["Checked"]
C -->|"not checkable"| U["Keep, mark unverified"]
R -->|"true"| K["Keep, record date"]
R -->|"wrong"| W["Can it be fixed?"]
W -->|"yes"| X["Fix it"]
W -->|"no"| D["Delete, within cap"]
style X fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style D fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
Which files get checked is decided in code, not by the agent. First the files that changed since the last run, then the ones never checked, then the ones checked the longest ago, so over a few weeks everything gets looked at again. There's also a cap on how much one night can delete. An agent that decides half the memory is wrong is more likely to be wrong itself.
```ts
// Tonight's files: changed since the last run, then never checked, then stalest.
function selectFilesToVerify(files: string[], changed: string[], checked: Record, today: string) {
const topic = files.filter(isTopicFile);
const recent = topic.filter((path) => changed.includes(path));
const rest = topic.filter((path) => !recent.includes(path) && checked[path] !== today);
const never = rest.filter((path) => checked[path] === undefined);
const stale = rest.filter((path) => checked[path] !== undefined).sort((a, b) => checked[a].localeCompare(checked[b]));
return [...recent, ...never, ...stale].slice(0, 25);
}
// The most files one night may delete: 15% of the topic files, at least three.
const deletionCap = (topicFiles: number) => Math.max(3, Math.floor(topicFiles * 0.15));
```
The dreaming thread keeps track of what it checked in `.dream/state.json`. It's the only file in the repository that nobody else can write, not even engineers:
```json
{
"lastRun": "2026-10-22T03:12:09Z",
"lastCommit": "9f2c41e",
"checked": {
"services/checkout-api.md": "2026-10-22",
"services/orders-db.md": "2026-10-22"
},
"unverified": ["queues/invoice-events.md"]
}
```
Every entry the dreaming thread writes also carries its source, like `[thread_abc, acme/web#41, 2026-10-17]`. This helps Praxen verify the origin of memories when things are ambigous.
### Learning from merged pull requests
When someone merges a [pull request Praxen opened](/use-cases/ticket-resolution/), what they changed before merging says more than any review comment. Nobody writes "please use our test fixtures". They just fix it and merge.
So every night, we look at what people did on top of our pull requests: the commits they pushed, their comments and their reviews. A pull request merged untouched means the fix was fine as it was.
```mermaid alt="Merges and closes by a person teach the nightly update; a lesson needs a repeated cause or a stated reason"
graph TB
P["An agent's pull request"] --> M["Merged"]
P --> C["Closed"]
M -->|"untouched"| A["Accepted"]
M -->|"edited"| L["Lesson if it repeats"]
C -->|"by a person"| L
C -->|"by the system"| N["Nothing learned"]
style L fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style N fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
The lesson goes into a file for that repository, and the next pull request there starts from it.
## Memory is shared and portable
Memory belongs to the workspace. When your teammate's investigation figures out why checkout-api returns 504s after a deploy, your next question about checkout-api goes to an agent that already knows.
```mermaid alt="People and agents write to one workspace memory, and every teammate's next thread reads it"
graph TB
P["People: thread, Slack, web app"] --> R["Workspace memory"]
A["Agents: threads, MCP"] --> R
R --> T["Every teammate's next thread"]
style R fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
And Praxen isn't the only one writing. People teach it how they work:
- **Ask in a thread.** "save a note: the nightly batch alert is expected, nobody acts on it"
- **Tell it in Slack.** Ask it to behave differently in a channel, and it updates that channel's preferences file.
- **Edit the files.** Open the memory in the web console and change what you want.
- **Let your agent do it.** Our [MCP](https://modelcontextprotocol.io/) server has `readMemory`, `writeMemory` and `deleteMemory`, so your coding agent can read and update the same memory.
Because it's git, you can always see who taught it what. Agent commits point to the thread they came from, and a person's edit carries their name. When a fact is wrong, `git log` tells you where it came from, and a revert takes it out.
## Memory and the prompt cache
Every thread starts with `MEMORY.md` and the last three daily notes in its context. Anything else, the agent greps for.
The catch is that memory changes all day. If every turn put the latest version in the prompt, the start of the prompt would keep changing, and the model provider's prompt cache would miss on almost every turn. A thread can run for days, and that means paying full price for the same tokens over and over.
So the prompt doesn't get the latest version. It gets a snapshot, and only the nightly update, or someone editing the index by hand, publishes a new one. Each thread pins the snapshot it started with and keeps it for its whole life. The start of the prompt never changes, and the cache keeps working.
```ts
async function getMemorySnapshot(threadId: string) {
const versions = await storage.get("memoryPromptSnapshots");
const latest = versions.at(-1);
const pin = await storage.get(`memoryPin:${threadId}`);
if (!pin) {
// First turn: pin whatever is newest now.
await storage.put(`memoryPin:${threadId}`, { version: latest.version, pinnedAt: Date.now() });
return latest.snapshot;
}
// Every later turn: the same version, so the prompt prefix never changes.
return (versions.find((entry) => entry.version === pin.version) ?? latest).snapshot;
}
```
That doesn't mean the agent works from stale memory. The repository on disk is always current, and the prompt tells the agent how to see what changed since its snapshot: `git -C /workspace/memory log --since={time} --name-only`. We also tell it to treat every entry as something that was true when it was written, not as a verified fact.
## Start with files
If you're building proactive agents, start with a git repository of markdown files. Agents already know how to use it, and git gives you history and reverts for free. Then spend your effort on keeping it true: check memory against production every night, and keep the agent away from the push.
---
### Nobody reads pull request descriptions, so we made videos
URL: https://getpraxen.app/blog/nobody-reads-pull-request-descriptions-so-we-made-videos/
Date: Oct 14, 2026
Author: Vi Tran,Boris Tane
Description: Almost none of the pull requests Praxen opens were asked for by an engineer, and a longer description does not make them easier to review. We are experimenting with a narrated video for every fix. Why, what it looks like, and what happened to merge rates.
import BlogImage from "../../components/blog/BlogImage.astro";
import DemoVideo from "../../components/DemoVideo.astro";
import BlogTable from "../../components/blog/BlogTable.astro";
import ChapterDescent from "../../components/blog/ChapterDescent.astro";
import CheckBeforeAfter from "../../components/blog/CheckBeforeAfter.astro";
import ComparisonBarChart from "../../components/blog/ComparisonBarChart.astro";
import RankedBars from "../../components/blog/RankedBars.astro";
import ShareDonut from "../../components/blog/ShareDonut.astro";
import { CHECK_RUNS_WITH_WARNINGS, CHECK_VS_RENDER, CHECK_WARNINGS, OUTCOMES, PUBLISH_OUTCOMES } from "../../data/blog/autofix-video";
export const mergeRate = (o) => (100 * o.merged) / (o.merged + o.closed);
What happens when you get a pull request you did not prompt into existence? You still have to understand what it's trying to improve, how it's doing it, and if it follows the existing patterns of your codebase.
We recently faced a clear problem. [Praxen](/use-cases/ticket-resolution/) is a pro-active agent, it watches production, investigates what it finds, and automatically opens pull requests with fixes. But engineers were having a very hard time groking exactly what problem Praxen is fixing, and how it's doing it.
```mermaid alt="An alert leads Praxen to investigate and open a pull request that an engineer must review without the investigation"
graph TB
A["Alert, failed CI, or a finding"] --> B["Praxen investigates"]
B --> C["Praxen opens a pull request"]
C --> D["An engineer decides if it ships"]
style D fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
This is what self-improving software looks like. A growing share of a codebase is written by an agent triggered by things such as an alert, a failed CI run, or just an agent noticing something is wrong in prod. Nobody triggers these changes, but today, they still needs a person to decide whether it ships.
The obvious anwer is "read the agent thread", but nobody has time for that. We are experimenting with novel ways to explain a pull request to an engineer. This post is about the first one: a narrated video.
## Context is always the problem
A pull request from a teammate arrives with context, there was a ticket, you discusssed it at standup or it was mentionned in Slack. You know who wrote it and why, and if unsure you can ask them.
A pull request from a pro-active agent is different. It arrives with as much context as the agent is willing to give you. It is the conclusion of an investigation you never saw, from the triggering alert to the logs read, the hypotheses investigated, etc. You essentially have to rebuild that story from a diff and wall of text the agent gives you.
So you're not really assessing if the code is correct yet, the main question is "why does this change exist, and should I believe it?".
## The wall of text
Our first answer was to write more. Every autofix pull request carried a description with the evidence, a diagram, why this fix, what is out of scope, the full causal chain and the files changed. Here is one from our own repository (I don't expect you to read it, it's for illustration purposes). The actual code diff is 58 lines, but the description is 565 words.
The main issue is how difficult it is to build a mental model of the change from this wall of text. It might actually be easier to just read the diff. A longer pull request description moves the work from the agent to the reviewer.
## Video walkthrough
Now, every autofix pull request now gets a narrated video walkthrough of two to three minutes, at the top of its description. We find it's much easier to build a mental model of a system or an issue from visuals. For the same pull request above, here's the generated walkthrough.
Figure 2
After: the same fix in 2 minutes 24 seconds
Generated by an agent with no human edits, and attached to the pull request.
The video tells the story as you need it to review a pull request: what we fixed and why, what production showed, where the code broke, why it broke, and what the change does. Every number on screen comes from the issue, and every line of code comes from the diff. The video has multiple chapters, and those let you jump through sections easily.
## Did it help?
We compared the merge rate of autofix pull requests submitted by Praxen with and without a walkthrough, in the same workspaces over the same weeks.
`${Math.round(n)}%` },
]}
/>
Pull requests with video walkthroughs had a 85% merge rate, when pull requests without a video walkthough only had 74%. In other words, when reviewers can clearly see why a change exists and what it does, they are more likely to ship it.
## How we built this
Every video is made by an agent in a sandbox. It reads the incident and the diff, writes the narration, writes the animation using the [Manim Python library](https://www.manim.community/), checks its own work and publishes.
```mermaid alt="The pipeline from an opened pull request to a video in its description"
graph TB
A["Autofix pull request opened"] --> B["Agent collects the evidence"]
B --> C["Narration and Manim scene"]
C --> D["Check, then render once"]
D --> E["Video in the PR description"]
style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
When Praxen opens an autofix pull request, we trigger a dedicated agent thread for the video. The first message holds the pull request description, the causal chain from the investigation, and the incident it fixes. It also includes relevant metrics, logs and traces coming from the investigation.
### Storyboarding
The agent writes the narration before it writes any animation, because the animation is timed to the speech. It splits the story into beats: one idea and one visual change each, in plain technical English. A `narrate` tool sends every beat to [Google's Gemini text-to-speech model](https://ai.google.dev/gemini-api/docs/speech-generation) and saves one clip per beat in the sandbox. Each clip is sped up by 25% and measured, and the tool returns the durations, so the agent can size every animation before the first render.
The speech model returns audio but no word timestamps, and a reveal needs to land on the word that names it. So each clip goes through speech recognition, and we match the recognised words to the script.
```mermaid alt="How a narration beat becomes a timed cue in the scene"
graph TB
A["Beat text"] --> B["Gemini speech clip"]
B --> C["Speech recognition"]
C --> D["Scene cue on a spoken phrase"]
```
### Animation with Manim
[Manim](https://www.manim.community/) is the community fork of the animation engine behind the [3Blue1Brown](https://www.3blue1brown.com/) videos. You describe a scene in Python, and it draws every frame. Our agent writes one scene file per video.
Inside a beat, a cue holds the animation until the narrator says the exact words:
```python
with self.beat("b3") as tr:
self.chapter("Where it broke", "One request, three components")
self.reveal(arch, run_time=min(2.4, self.until(tr, "container") - 0.1))
self.at_phrase(tr, "never answers")
self.play(arch.nodes["container"].card.animate.set_stroke(RED, width=3))
```
On top of Manim, we wrote a small library for the needs of pull request walkthroughs:
- architecture diagrams with real logos for cloud components
- code cards with highlighted diffs
- bar charts, lines charts, etc.
Those are the pieces the agent picks from, which gives consistency across videos. Every pull request walkthrough tells the story of the pull request in the same order, from the widest view down to a single line. One critical design requirement for us is consistency. We want engineers to know what to expect everytime they click on a pull request walkthrough.
## We need to keep experimenting
This is just the first experiment, as agents produce and operate more of our software, it's necessary to iterate on how we present all this information to engineers who are ultimately responsible for the application once they ship.
---
### Turn your app into a context graph
URL: https://getpraxen.app/blog/turn-your-app-into-a-context-graph/
Date: Oct 5, 2026
Author: Boris Tane
Description: Agents that run software need a context graph of the app: every resource, what it connects to, the repository that deploys it and the team that owns it. How we built one on Durable Objects, keep it fresh across every provider without polling AWS, decide what counts as a change, and delete from it safely.
import GraphGalaxy from "../../components/blog/GraphGalaxy.astro";
import MeasureTable from "../../components/blog/MeasureTable.astro";
import RankedBars from "../../components/blog/RankedBars.astro";
import ResourceGraph from "../../components/blog/ResourceGraph.astro";
import ShareDonut from "../../components/blog/ShareDonut.astro";
import SyncRanges from "../../components/blog/SyncRanges.astro";
import ButterflyBars from "../../components/blog/ButterflyBars.astro";
import Treemap from "../../components/blog/Treemap.astro";
import {
GRAPH_TOOL_CALLS,
GRAPH_SIZES,
SYNC_TIMES,
TYPE_COUNTS,
PROVIDERS,
providerName,
providerBars,
typeName,
typeProvider,
typeLogo,
changesPerSync,
toolShare,
toolMedianMs,
largestGraph,
netNodes,
totalCreated,
totalDeleted,
fmtInt,
} from "../../data/blog/context-graph-measurements";
We're getting closer to self-improving software, where agents continuously read the signals your app produces and improve it automatically. To do that, an agent first has to understand how your app fits together. It needs to know which services talk to which databases, what each queue triggers, and which repository deploys what.
So, before an agent reads a single log line, it's necessary to give it a graph of everything your app uses to run: clouds, repos, observability providers, etc.
```mermaid alt="Clouds, repositories and telemetry feed the context graph used for detection, investigation and autofix"
graph TB
C["Cloud providers"] --> G["Context graph"]
R["Repositories and IaC"] --> G
T["Telemetry vendors"] --> G
G --> D["Detection"]
G --> I["Investigation"]
G --> F["Autofix"]
style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
This is the [context graph](/use-cases/devops/). Every resource in every connected account is a node, every relationship between them is an edge, and the graph is continuously updated, as you deploy new things, delete old ones, and refactor your codebase. The context graph also encompasses all the data that lives alongside the nodes, in vector databases, in key-value stores, etc. This additional data is the difference between an agent that can traverse a graph and one that actually understands your app.
This post walks through how we turn an app into a context graph: what goes into it, how we keep it in sync with every provider, and how we connect resources across clouds without guessing.
## What's in the context graph?
The context graph is made of nodes and edges. A node is a resource: a Lambda function, a Worker, a Kubernetes deployment, a repository, a team. An edge is a typed relationship between two of them, read in one direction: a queue `triggers` a function, a function `assumes` a role and `connects_to` a table, a repository `deploys_to` the function, a team `owns` the repository.
We use [Cloudflare Durable Objects](/blog/how-we-fixed-our-cloudflare-durable-objects-memory-exceeded-errors/) for the graph itself. Each workspace on Praxen gets its own instance, with both compute and SQLite storage, so there's no cluster to run, no connection pool to manage, and no shared tables to keep tenants apart. The graph only holds what a resource is and how it connects to other nodes.
The majority of workspaces in Praxen have fewer than 100 nodes, but a subset of customers have more than 10,000 nodes. Our solution must scale to those apps too.
({ label: b.bucket, value: b.workspaces }))}
valueName="Workspaces"
share
label="Figure 3"
title="Share of workspaces by the number of resources in their graph"
caption={`One Durable Object per workspace, whatever the size.`}
/>
The graph itself is two tables, nodes and edges. A node's identity is five fields, `provider`, `account`, `region`, `type` and `id`, and the database derives the key from them itself, so no caller can ever build it differently:
```sql
CREATE TABLE nodes (
provider TEXT NOT NULL,
account TEXT NOT NULL,
region TEXT NOT NULL,
type TEXT NOT NULL,
id TEXT NOT NULL,
composite_id TEXT GENERATED ALWAYS AS ('provider#' || provider || '#account#' || account || '#region#' || region || '#type#' || type || '#id#' || id) VIRTUAL NOT NULL UNIQUE,
data TEXT,
created INTEGER NOT NULL,
updated INTEGER NOT NULL,
-- ...
);
```
The `data` field holds the properties of the node, as JSON. For a Lambda function that's all the data returned by the AWS API when calling `GetFunction`. Environment variable values are stripped before anything is stored, only the keys are kept:
```json
{
"FunctionName": "checkout-api",
"Runtime": "nodejs22.x",
"Handler": "dist/index.handler",
"MemorySize": 1024,
"Timeout": 30,
"Architectures": ["arm64"],
"Role": "arn:aws:iam::123456789012:role/checkout-api-role",
"VpcConfig": { "SubnetIds": ["subnet-0a1b2c3d"], "SecurityGroupIds": ["sg-0e4f5a6b"] },
"EventSourceMappings": [{ "EventSourceArn": "arn:aws:sqs:eu-west-1:123456789012:orders-events", "BatchSize": 10, "State": "Enabled" }],
"LastModified": "2026-10-04T18:22:41.000+0000",
"RevisionId": "4c1e9a0b-7d2f-4e61-9a3c-2b8d5f0e1a77",
"State": "Active",
"envVarCount": 3,
"envVarKeys": ["ORDERS_TABLE", "STRIPE_SECRET_KEY", "LOG_LEVEL"],
"packageType": "Zip",
"imageUri": null
}
```
Edges point from one node's key to another's, and a pair of nodes can have at most one edge of each relationship:
```sql
CREATE TABLE edges (
source TEXT, -- composite_id of the node the edge starts from
target TEXT, -- composite_id of the node it points to
relationship TEXT, -- connects_to, triggers, deploys_to, owns, ...
properties TEXT, -- JSON: the evidence the edge was drawn from
discovered_via TEXT, -- infrastructure_sweep, environment_variables, repository, ...
created_by TEXT NOT NULL,
last_edited_by TEXT NOT NULL,
created INTEGER NOT NULL,
updated INTEGER NOT NULL,
UNIQUE(source, target, relationship)
);
```
`discovered_via` records how we know the relationship between the nodes. It could be through an environment variable, an integration, or manually added by a user. The `properties` field keeps the evidence of the connection, like the DNS record an edge was drawn from, or the image reference and what it was matched on, so an agent can check why two resources are connected.
## What lives next to the graph
The context graph is not just the graph database. We also enrich each node with a wide range of additional data that the agent can access alongside it:
- **A catalog** that finds a resource from the name an engineer would use for it
- **A resource object** that remembers how each resource normally behaves, so the agent can tell healthy from broken without querying a provider
- **The repository** that deploys each resource, so the agent goes straight to the code that's running on it
- **Key queries**, the questions worth asking about each resource when something goes wrong, prepared ahead of time
```mermaid alt="Catalog search leads an agent to a graph node, its resource object, repository, key queries and telemetry"
graph TB
A["Agent asks about checkout-api"] --> S["Catalog full-text and vector search"]
S -->|"finds the node"| G["Context graph node, edges, neighbours"]
G --> R["Resource object metrics, baselines, tier"]
G --> C["Repository the code that deploys it"]
G --> K["Key queries what to watch"]
G --> T["Telemetry providers logs, metrics, traces"]
style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style S fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
style R fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
style C fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
style K fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
```
### The catalog: finding the node
An agent rarely knows a node's ID. It knows "the checkout database". In order to use the graph and figure out its dependencies, it's necessary to go from "the checkout database" to its ID in the cloud provider. We implemented a second Durable Object per workspace which holds a full-text search index over every node's name and alternate names, alongside embeddings in a vector database. The agent can use this to find the ID of the database based on its human-readable name.
```mermaid alt="Full-text and vector search resolve the checkout database name to prod-orders-pg-01"
graph TB
Q["the checkout database"] --> F["Full-text index"]
Q --> V["Vector database"]
F --> N["prod-orders-pg-01"]
V --> N
style N fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
### The resource object: is it healthy?
For the workspace with thousands of cloud resources, the agent should not need to call the observability provider to know basic metrics about the resource every time it's accessing it, it's actually impossible given all providers have pretty stringent rate-limits.
For every resource we keep its baseline of metrics, alongside recent timeseries and common logs. This data lets an agent quickly and cheaply know if a resource is healthy or misbehaving:
```mermaid alt="The checkout-api resource object stores metrics history, multi-horizon baselines and log templates"
graph TB
N["checkout-api node"] --> R["Resource object"]
R --> M["Metrics history"]
R --> B["Baselines over several horizons"]
R --> L["Log templates"]
style R fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
```
The baselines cover several horizons, so a Monday morning peak doesn't read as an issue.
### The repository: what's running on it
An organisation can have hundreds of repositories, and an agent that has to guess which one is behind a failing service wastes its time in the wrong code. Every resource in the graph knows the repository that deploys it, and often the exact file that defines it: the Terraform module, the `wrangler.toml`, the task definition. When the agent needs to read code, it starts in the right place.
```mermaid alt="A checkout-api node links to its deploying repository, defining file and running code"
graph TB
N["checkout-api"] -->|"deployed by"| R["acme/checkout"]
R --> F["The file that defines it"]
F --> S["The code running on it"]
style R fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
```
### Key queries: what to watch
To further improve the context we give to the agents, for each node, we compute "key queries". Key queries are the questions an engineer would ask about every piece of their architecture when they get paged. For example, for an auth service, they will want to know:
- How many logins are failing right now, and for what reason?
- How long does token verification take at the 95th percentile?
- Is the identity provider rate limiting us?
We compute these ahead of time by looking at metrics, logs but also custom spans a developer added to their codebase, such that we can answer the type of questions impossible to answer with just metrics.
```mermaid alt="An error log becomes a key question, a provider query and a recurring check"
graph TB
L["An error log in the code"] --> Q["Key question"]
Q --> P["Provider query"]
P --> C["Check, every few minutes"]
style C fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
```
The graph is the index that joins all of this together. An agent investigating `checkout-api` starts from its node, walks to the database it connects to, checks that database's baselines in its resource object, searches the repository that deploys it, and queries its logs, without ever being told where any of it lives.
## How we keep the graph up to date
An app continuously evolves, and keeping the context graph in sync with the reality of the app is a complex challenge. We need to react to changes in the application and surgically create, update and delete nodes as they evolve in the application. Each cloud provider is different with different APIs and mechanisms, so we have to adapt to each:
```mermaid alt="Regional syncs collect nodes and edges, prune missing resources and link providers, repositories and owners"
graph TB
S["Start the sync"] --> RA["eu-west-1"]
S --> RB["us-east-1"]
S --> RC["global"]
RA --> NA["Nodes, then edges"]
RB --> NB["Nodes, then edges"]
RC --> NC["Nodes, then edges"]
NA --> P["Prune what disappeared"]
NB --> P
NC --> P
P --> X["Link across providers"]
X --> C["Link repositories and owners"]
C --> D["Complete"]
style P fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style X fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style C fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
For example, for AWS we use CloudTrail, for Kubernetes we use a small Helm agent that dials out through a Cloudflare Tunnel, for Vercel we use webhooks, and for Cloudflare, we rely on audit logs.
The key constraints of syncing are how long it takes and how much of the provider's rate limit budget it consumes. Every call a sync makes is a call the customer's own tooling can't, so a sync has to finish quickly and touch as little as possible. Most of them find nothing to change: a PlanetScale or Supabase sync almost never adds or removes a resource, while an AWS sync touches about 20 resources and a Kubernetes one about 12. The chart below shows how long syncs take for each provider, from the fastest tenth to the slowest, and how much each one changes.
({ label: providerName(t.provider), logo: PROVIDERS[t.provider]?.logo, logoLight: PROVIDERS[t.provider]?.logoLight, p10: t.p10, p50: t.p50, p90: t.p90, extra: changesPerSync(t.provider) >= 1 ? changesPerSync(t.provider).toFixed(1) : changesPerSync(t.provider) >= 0.01 ? changesPerSync(t.provider).toFixed(2) : "<0.01" }))}
extraName="Resources added or removed per sync"
label="Figure 5"
title="How long a sync takes, and how much it changes"
caption="Completed account syncs in the last 30 days, from the sync starting to the account being marked ready. The right-hand column is every resource added or removed over the same window, divided by the number of syncs."
/>
## Connecting resources across clouds
Most applications run across multiple providers, for instance AWS for long-running containers, Cloudflare for CDN and edge-computing and Vercel for sandboxes. Syncing each cloud individually is great, but the interesting edges cross providers, because each provider only knows its own half.
After the provider work, every sync runs one shared linker over the whole workspace. A DNS record pointing at `.elb.amazonaws.com` connects to that load balancer, one pointing at `.vercel.app` to that Vercel project, `.cfargotunnel.com` to that tunnel. An environment variable holding an RDS endpoint connects the function to the database.
Put together, the syncs and the linkers give every customer a graph that looks like their stack rather than like any one provider's console.
({ label: typeName(t.label), value: t.value, group: typeProvider(t.label), logo: typeLogo(t.label) })),
]}
wholeName="the twelve most common types"
groups={[
{ key: "aws", label: "AWS", color: "#f59e0b" },
{ key: "kubernetes", label: "Kubernetes", color: "#3b82f6" },
{ key: "cloudflare", label: "Cloudflare", color: "#f97316" },
{ key: "code", label: "GitHub", color: "#84cc16" },
{ key: "railway", label: "Railway", color: "#8b5cf6" },
{ key: "vercel", label: "Vercel", color: "#64748b" },
]}
label="Figure 7"
title="What's in a context graph"
caption={`The twelve most common resource types, each sized and labelled by its share of those twelve. Together they make up ${Math.round((100 * TYPE_COUNTS.slice(0, 12).reduce((n, t) => n + t.value, 0)) / netNodes())}% of every resource in every customer graph.`}
/>
## How agents use the graph
The Praxen agents get a narrow tools over the graph:
- find nodes by any field in the provider's JSON
- read a node and its neighbours
- walk the graph in either direction, and
- compute a blast radius.
({ label: t.tool, value: t.calls }))}
mono
label="Figure 8"
title="Which graph tools agents call"
caption="Each tool's share of every graph tool call agents made in the last 30 days, across every customer workspace."
/>
Agents really like `findNodes` as they use it to filter nodes across any dimensions. For example they can find all the AWS Lambda function with Go runtime in a single call.
[t.tool, toolShare(t.tool) < 0.1 ? "<0.1%" : `${toolShare(t.tool) < 10 ? toolShare(t.tool).toFixed(1) : Math.round(toolShare(t.tool))}%`, `${fmtInt(t.p50Ms)} ms`, `${fmtInt(t.p95Ms)} ms`])}
numeric={[1, 2, 3]}
code={[0]}
label="Table 1"
title="Graph tool latency"
caption="End to end, from the agent's tool call to its result, over the last 30 days."
/>
The blast radius tool is used when the agent wants to know the downstream services of a specific node. It's useful when it identifies a node as the potential cause of an incident and it wants to know grade the severity of the incident. For example, if a database fails, the services that `connects_to` it fail too, so the fault travels against the edge. If a queue fails, the functions it `triggers` stop, so the fault travels with it.
Agents are only as good as the context you give them. We built a system to create a rich context graph for any application, and it powers everything Praxen does.
---
### We swapped our LLMs for Jev. It's 39% cheaper.
URL: https://getpraxen.app/blog/we-swapped-our-llms-for-jev/
Date: Sep 25, 2026
Author: Vi Tran,Boris Tane
Description: Our agent makes thousands of small decisions a day. We moved them from LLMs to Jev, a model that doesn't write text, and cut the cost by 39%.
import DecisionComparisonChart from "../../components/blog/DecisionComparisonChart.astro";
import BenchmarkScatter from "../../components/blog/BenchmarkScatter.astro";
import DecisionTakeaways from "../../components/blog/DecisionTakeaways.astro";
import ComparisonBarChart from "../../components/blog/ComparisonBarChart.astro";
import { latencyCostMetrics, JEV_ROUTING_MEASUREMENT, JEV_INCIDENT_CONNECTION_MEASUREMENT, JEV_CLOSE_REASON_MEASUREMENT, JEV_AUTOFIX_SEVERITY_MEASUREMENT, JEV_RESOURCE_TIER_MEASUREMENT } from "../../data/blog/jev-purpose-measurements";
import { JEV_CALL_OVERALL, JEV_MEASUREMENT_SCOPE, JEV_MODEL_BENCHMARK } from "../../data/blog/jev-measurements";
We're building an always-on on-call agent, which constantly makes decisions, for example:
- how severe is this issue?
- should we be pro-active on this slack thread?
- have we seen this incident before?
Up until last week, we've been using LLMs for these tasks. As soon as we got access to [Jev](https://docs.typesafe.ai/models) this week we started experimenting with it.
## What is Jev?
Jev is a decision model from [TypeSafe AI](https://typesafe.ai/blog/introducing-system-one-models-and-jev). It doesn't generate text: it answers questions about your data with typed values and calibrated probabilities. TypeSafe pitches it for "smart if-statements", the classify, route and score steps where hand-written logic is too brittle, and quotes 70 to 500 ms per response with free output tokens. Our agents make those decisions all day, so we put it in production.
### How it works
A request has two parts: the `state`, which is the text or JSON you want a decision about, and the `questions` you want answered. Each question has one of three types:
```mermaid alt="Jev takes state and questions, returning yes-or-no probabilities, choices or rubric scores"
graph TB
S["State: a message, thread, or evaluation case"] --> J["Jev"]
Q["Questions"] --> J
J --> N["Noul yes/no"]
J --> C["Choice pick an option"]
J --> R["Score rate against a rubric"]
style J fill:#29e047,stroke:#00801e,color:#15151a
style N stroke:#3b7dd8,color:#2b5fa8
style C stroke:#3b7dd8,color:#2b5fa8
style R stroke:#3b7dd8,color:#2b5fa8
```
- **Noul** returns the probability of `yes`.
- **Choice** selects one of the provided options.
- **Score** returns a probability-weighted value across an ordered rubric.
## Use-case 1: Routing, or when should the agent respond?
Our agents are pro-active on Slack and Github. They will respond to Slack messages or Github comments if they have a meaningful insight to provide to the user. The agents need to act when a user requests it, without overreacting to every event. This is a perfect use-case for Jev's `Noul` questions.
```mermaid alt="Jev decides whether a PR comment or Slack message should wake an agent or need no action"
flowchart LR
E["New event"] --> M["PR comment Slack message"]
M --> J["Jev Noul Triage?"]
J -->|Yes| W["Wake agent to follow up"]
J -->|No| N["No action"]
style J fill:#29e047,stroke:#00801e,color:#15151a
style W stroke:#29e047,color:#00801e
style N fill:#e5e5e8,stroke:#d1d2d6,color:#47484d
```
- **Comments on PRs our agents submitted:** a reviewer asking for a change wakes the agent up. A CI status update or a "thanks" doesn't.
- **Slack channels:** the agent jumps in uninvited only when it can clearly help, like a direct infrastructure question. It stays out of humans coordinating with each other.
- **Slack threads the agent is in:** it answers messages meant for it, ignores humans talking to each other, and leaves when someone asks it to.
Here is an example of a Jev request to determine if an agent should follow up on a Slack message.
```json
{
"model": "jev-1.13.0",
"state": {
"slack_channel": "#Deployment",
"user_message": "Watch this PR until fully deployed"
},
"questions": {
"respond": {
"type": "noul",
"instructions": {
"question": "Should the agent follow up on this user message?"
}
}
}
}
```
We ran Jev for about a week and compared with the LLM we previously used for this task.
Routing got 3x faster at P90, from 1.5 seconds to under 500 ms, and the cost dropped by almost half.
## Use case 2: Classification, or what does the evidence mean?
We also run a few tasks where the agent needs to classify things into different buckets. For example, based on the provided evidence, should the agent start investigating the issue, should it fold it as related to an existing issue, or is it a duplicate of a previously resolved issue?
A Jev Choice question turns that evidence into one label from criteria we define, and the next step is decided by the label. It looks like this:
```mermaid alt="Jev classifies incident evidence as the same root cause, related or independent"
flowchart LR
I["New incident"] --> E["Evidence from tool calls"]
E --> J["Jev Choice"]
J -->|Same root cause| D["Defer to the original"]
J -->|Related| L["Link both incidents"]
J -->|Independent| N["Investigate on its own"]
style J fill:#29e047,stroke:#00801e,color:#15151a
style D stroke:#29e047,color:#00801e
style L stroke:#29e047,color:#00801e
style N fill:#e5e5e8,stroke:#d1d2d6,color:#47484d
```
We run three classifications this way. Each one uses a Choice question with explicit criteria, and each one replaced a different LLM, so we measured them separately.
### Are these incidents related?
When a new incident opens, the agent compares it with existing incidents: do they share one root cause, are they related, or are they independent? This illustrative example shows one candidate; a production call compares several against the same new incident.
```json
{
"model": "jev-1.13.0",
"state": {
"incident": "Checkout cannot authenticate to the database.",
"candidate_0": "Billing cannot authenticate to the same database.",
"evidence": "Both services use a credential revoked at 14:00."
},
"questions": {
"candidate_0": {
"type": "choice",
"instructions": "How is candidate_0 connected to the new incident?",
"criteria": {
"duplicate_same_root_cause": "One underlying problem explains both",
"related": "Distinct problems share a trigger or blast radius",
"independent": "No evidenced connection"
}
}
}
}
```
By switching to Jev for this use-case, we made it almost 8x faster at P90, from 2.9 seconds to under 400 ms, and 27% cheaper.
### Why did this PR we submitted get closed?
Our agents submit pull requests to developers, and one of our key success metrics is the merge rate. We need to understand why pull requests get closed such that we can improve the product.
When one of our PRs is closed without merging, the agent reads the reviews, the discussion, and references to other work, then picks the reason: the fix was wrong, a human fixed it another way, the issue was a false positive, the PR went stale, or the behaviour was intended.
By switching to Jev for this use-case, we improved latency to 6x faster at P90, but only 17% cheaper.
### How pressing is this pull request?
Before submitting a pull request to developers, our agents need to rank it such that higher severity things rise to the top. For this classification, the agent grades the underlying problem on a ladder from critical to info. It grades current impact, not hypothetical risk.
Switching to Jev here substantially reduced cost: 59% cheaper than DeepSeek V4.1 Flash, and nearly 5x faster at P90.
Jev is faster on every classification. Where it replaced GPT-OSS 120B, the gain is mostly latency. Where it replaced DeepSeek V4.1 Flash, it also cut the bill by more than half.
## Use case 3: Ranking, or how important is this cloud resource?
To get the best out of Praxen, teams connect their cloud accounts. We create a [context graph](/use-cases/devops/) of all the cloud resources, such that the agents can quickly understand the relationship between compute nodes, databases, queues, etc.
We have teams on the platform with extremely busy cloud accounts, with 10s of thousands of nodes. Each server, sandbox, database, and queue is a node in our context graph. It's necessary to rank each of these nodes such that agents know what is critical to your application, and what is essentially "fine" to fail.
We assign one of four priority tiers to each resource: Critical, Standard, Low, or Minimal.
```mermaid alt="Jev uses each context-graph node and its metrics to assign Critical, Standard, Low or Minimal priority"
flowchart LR
C["Cloud account"] --> G["Context graph"]
G -->|Each node + metrics| J["Jev Choice"]
J --> T1["Critical"]
J --> T2["Standard"]
J --> T3["Low"]
J --> T4["Minimal: fine to fail"]
style J fill:#29e047,stroke:#00801e,color:#15151a
style T1 stroke:#29e047,color:#00801e
style T2 stroke:#29e047,color:#00801e
style T3 stroke:#29e047,color:#00801e
style T4 fill:#e5e5e8,stroke:#d1d2d6,color:#47484d
```
Jev evaluates a Choice question for each resource, with context based on configuration, environment, recent metrics, and dependencies.
```json
{
"model": "jev-1.13.0",
"state": {
"instructions": "Assign importance relative to the other resources in this cohort.",
"cohort": [
{
"id": "database-a",
"environment": "production",
"daily_queries": 80000,
"dependents": 6
},
{
"id": "database-b",
"environment": "preview",
"daily_queries": 0,
"dependents": 0
}
]
},
"questions": {
"resource_0": {
"type": "choice",
"instructions": "Assign the importance tier for database-a.",
"criteria": {
"1": "Critical: substantial production traffic or blast radius",
"2": "Standard: active and operationally relevant",
"3": "Low: limited activity or importance",
"4": "Minimal: idle or disposable, without meaningful dependents"
}
}
}
}
```
Ranking is where Jev shines: more than 10x faster at P90, from 5.4 seconds to about half a second, and 34% cheaper. It's also our highest-volume decision, so it drives most of the overall savings.
## Summary
Overall, Jev delivered substantial overall reductions in latency and estimated cost per 1,000 calls:
- P90 latency reduction: 4,752 ms --> 508 ms.
- Cost per 1,000 calls reduction: $0.76199 --> $0.46369.
Per model, Jev is both the fastest and the cheapest: slightly cheaper than DeepSeek V4.1 Flash, and well below both GPT-OSS models.
Wherever our agents pick from a fixed set of answers, Jev is now the default: it is faster and cheaper on every decision we moved.
---
### How we prevent slop from hitting prod
URL: https://getpraxen.app/blog/how-we-prevent-slop-from-hitting-prod/
Date: Sep 21, 2026
Author: Vi Tran,Boris Tane
Description: Nothing in your pipeline asks whether a change is safe for prod. This is the check we built that does, and how we got it to finish before your tests do.
import BigNumber from "../../components/blog/BigNumber.astro";
import PrVerdictComment from "../../components/blog/PrVerdictComment.astro";
import DailyChart from "../../components/blog/DailyChart.astro";
import ForecastChart from "../../components/blog/ForecastChart.astro";
import Waterfall from "../../components/blog/Waterfall.astro";
import { REREVIEW_DAYS } from "../../data/blog/pr-review-delta";
import { REVIEW_WEEK, avoidedPerWorkspace } from "../../data/blog/pr-review-week";
You're probably either building a software factory or leasing one from a provider. You have a system that takes you from a prompt to a pull request. It handles tests, linting, formatting, and automated code reviews.
But you still don't have an answer to the most important question: **is this change okay to go to prod?**.
```mermaid alt="Tests, linting and code review leave a production-safety question unanswered before deploy"
graph TB
W["Agent writes the code"] --> T["Types and tests"]
T --> L["Linter and formatter"]
L --> R["Code review"]
R --> Q(["Is this okay for prod?"])
Q --> D["Deploy"]
style Q fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
All of your current checks look at the diff, but nothing in your factory knows anything about the production system the diff is about to land on. Nothing can really prevent slop from hitting prod.
We built this capability inside of [Praxen](https://getpraxen.app) and this post is about the technical details of how we implemented it.
## How it works
The one questions this system should answer is:
> Would this change, once merged and deployed, have a negative impact on production?
We do not really care about the typical things a code-review agent would check such as style, naming, test coverage, etc. And we decided to answer this question at the pull-request stage, along side all your existing tests.
Ultimately, Praxen comments on the pull request with a simple "go" / "no-go" message with the receipts from its investigation.
The flow is fairly simple:
- Is this Pull request touching files that may affect production?
- Which cloud resources are potentially afftected?
- Gather context about the current state of production for those resources
- Evaluate multiple potential failure modes this change may introduce
- Forecast how production might change with these changes deployed
- Alert developers of the likely potential failure modes
This all relies on the [context graph](/use-cases/devops/) we continuously build connecting all the cloud resources in your various cloud accounts.
```mermaid alt="A PR check finds affected resources, gathers context and forecasts failure trajectories before its verdict"
graph TB
P["Open pull request"] --> Q(["Meaningful code change?"])
Q -->|"docs, tests, comments"| C["Conclude the check"]
Q -->|"yes"| R["Find affected resources in the context graph"]
R -->|"none"| C
R -->|"yes"| E["Gather context"]
E --> X["Build failure trajectories"]
X --> F["Forecast the affected series"]
F --> V["Verdict"]
V --> C
style X fill:#fef3c7,stroke:#fcd34d,color:#78350f
style F fill:#fef3c7,stroke:#fcd34d,color:#78350f
style V fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
## Assembling the context
Our context graph is key to making this work, it builds a registry of all your cloud resources, all your repositories, teams, etc. For instance, a compute node such as Lambda function is connected to the database it reads from and the queue that triggers it. We also add repositories to the graph. This is done by looking at the typical manifest files in the repository, for example terraform files, Cloudformation files or Wrangler files. This enables the connection between repositories and cloud resources.
```mermaid alt="A repository deploys a Lambda function triggered by SQS and connected to RDS"
graph LR
Q["SQS queue orders-events"] -->|"triggers"| A["Lambda function checkout-api"]
R["Repository checkout-edge"] -->|"deploys_to"| A
A -->|"connects_to"| D["RDS instance orders-db"]
style R fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
When a pull request is submitted to the repository, we follow the paths on the context graph to collect all the cloud resources potentially affected by the change. We use a small models to filter through the resources as there could be a large number of resources deployed from the same repository. We pass this context alongside the diff, the PR description and the commits in the PR to the agent.
```mermaid alt="A small model filters candidate resources and passes them with the diff to the review agent"
graph LR
R["Repository"] -->|"deploys_to"| C["Candidate resources"]
C --> F(["Small model filters"])
F --> A["Agent"]
D["Diff, description, commits"] --> A
style A fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
We also pass the diff through a set of deterministic heuristics to quickly guide the agent's attention towards things that are usually likely to have a negative impact on production:
- a migration that has to be applied by hand, or in a particular order relative to the deploy
- `CREATE INDEX` without `CONCURRENTLY`, or `ADD COLUMN ... NOT NULL` with no default, both of which take a lock on the table for the duration
- an endpoint being removed while the currently deployed version still reads it
- code that starts reading an environment variable, secret or binding that nothing in the diff provisions
These are recommendations, streering the agent to probably deployment risks.
## Failure trajectories
With the context provided above, the model comes up with various failure modes the new diff may introduce in production and goes and investigates each.
Its output is a ledger of failure trajectories. A trajectory is one causal chain from a trigger, through the changed code, to an observable degradation on a specific metric, and every link in it carries a citation: a file and line, a log template with its count, a metric read, a config key, a graph edge.
The agent tries to both validate and invalidate each trajectory before coming to a verdict. Each ends in one of three states:
- `confirmed` when the trajectory has been confirmed against production.
- `plausible` when the chain is concrete but one or multiple links could only be "guessed" with no telemetry data to confirm it.
- `refuted` when production telemetry data provided enough evidence that this trajectory is unlikely to happen in production.
We take a conservative approach and any `confirmed` trajectory leads to a failed assessment.
```ts
export function deriveTrajectoryVerdict(trajectories: Trajectory[]): "pass" | "fail" {
return trajectories.some((entry) => entry.status === "confirmed") ? "fail" : "pass";
}
```
```mermaid alt="Three failure trajectories are confirmed, plausible or refuted; the confirmed risk produces a no-go verdict"
graph LR
D["The diff"] --> T1["Dropped index still used by checkout"]
D --> T2["Retry change amplifies load on orders-db"]
D --> T3["Removed binding breaks the worker"]
T1 --> C1["confirmed"]
T2 --> C2["plausible"]
T3 --> C3["refuted"]
C1 --> V["No-go"]
C2 --> V
C3 --> V
style C1 fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style C2 fill:#dbeafe,stroke:#93c5fd,color:#1e3a8a
style C3 fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style V fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
## Forecasting the impact
We give the agent a tool to forecast timeseries based on historical data with injecting potential external factors.
Say a trajectory claims some change halves the timeout on a path that retries. Whether that matters depends on the traffic, and traffic isn't really a single number, it has a shape. A queue sitting at 60% depth and staying there is fine. The same queue at 60% but climbing every week is a different situation, and reading the last hour of metrics won't tell you which one you're looking at.
So before the agent drafts the verdict of a trajectory, it pulls the historical series for the affected resources and forecasts them together. We currently self-host the [Toto-2.0-22m model](https://huggingface.co/Datadog/Toto-2.0-22m).
```mermaid alt="The agent queries provider telemetry and sends aligned series to a model returning p10, p50 and p90 forecasts"
graph LR
A["Agent"] -->|"telemetry query"| P["Provider"]
P -->|"aligned series"| A
A -->|"up to 16 series"| M["Forecasting model"]
M -->|"p10, p50, p90"| A
style M fill:#fef3c7,stroke:#fcd34d,color:#78350f
```
The reason for a dedicated multivariate model rather than another prompt is that the series aren't independent. Request rate, error rate, latency and queue depth on one resource move together, and forecasting each on its own throws away the correlation that makes the forecast worth having. Every series in a call shares one attention group.
This is the most experimental capability we recently added, and still measuring its impact. But it already passes the vibes-eval.
## The latency constraint
Running the production impact assessment in the pull request flow means it must to be fast. Nobody wants a step that adds 15 minutes to their CI pipeline. For instance, on our own repositories this assesment is a required CI step, if it's slow, our entire SDLC gets bogged down.
We essentially have a budget of 2 to 3 minutes to produce an accurate impact assesment. Anything beyond that is not acceptable in a CI/CD pipeline.
Our first prototype was completely failing this constraint. Our median was nearly 7 minutes and it was common to see runs take up to 20 minutes.
The goal became figuring out how to reduce the number of model steps to get the entire flow under 3 minutes. We typically ran 30 to 40 sequential model steps, and in the worst case scenarios up to nearly 600 steps, each consuming 17 seconds of our budget, and the vast majority of their output was reasoning tokens.
We made quite a few changes over the past 3 weeks, with various impact on performance. Here are the most significant ones.
### Setting a step budget and communicating it to the agent
We introduced a step budget, capped at 30 steps by agent turn and we clearly communicate to the model how many steps it has consumed on every step. Putting the count in the prompt lets the model plan around it, which turns out to matter more than the number itself.
```mermaid alt="A review agent tracks its 30-step budget and records a verdict when one step remains"
graph TB
A["Turn starts, 30 steps"] --> B["Model step"]
B --> T["Tool call"]
T --> C(["Steps left?"])
C -->|"1"| E["Record the verdict now"]
C -->|"more"| R["Next step, carrying the count in the prompt"]
R --> B
style R fill:#fef3c7,stroke:#fcd34d,color:#78350f
style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
### Starting new reviews instead of folding into the existing one
A typical pull request keeps getting new commits after it's opened. In the first prototype, we would fold the entire new diff in the same review thread as a steering user message. This steer would greatly confuse the model and lead to reading files it had already examined and re-running telemetry queries it had already completed.
Now, a new commit cancels the previous assesment run and starts a brand new thread. It seems counter-intuitive, but it ultimately brought down the latency for complete reviews.
### Re-using previous work
When a new commit lands in the pull request after a review was already completed, we used to naively start a new impact assessment from scratch.
We introduced the ability to re-use the previous assessment, and trigger a new assesment on the smaller diff between the two consecutive commits.
We deployed these changes throughout September and have gradually improved latency and the latency comfortably sits within budget.
({ day: d.day, value: d.medianSeconds, n: d.reviews, partial: d.partial }))}
unit="seconds"
ticks={[
{ v: 0, label: "0 s" },
{ v: 150, label: "150 s" },
{ v: 300, label: "300 s" },
{ v: 450, label: "450 s" },
{ v: 600, label: "600 s" },
]}
valueName="Median impact assessment turn"
nName="re-reviews"
label="Figure 4"
title="Median latency for an impact assessment turn"
caption=""
/>
## Does this even matter?
What's the point of burning all these tokens if we don't see any meaningful results? Our success metric is the number of incidents we prevent per week for each of our customers. A prevented incident is a pull request where:
- we flag a potential risk to production
- an engineer pushes one or multiple commits
- a new assessment concludes the potential risk is mitigated
- the pull request gets merged
This is already pretty significant and we expect this to keep growing. We are continously iterating and running evals to improve the performance and quality of our agent.
---
### Sub-agents are just wrong
URL: https://getpraxen.app/blog/sub-agents-are-just-wrong/
Date: Sep 14, 2026
Author: Aleksandr Diamond,Boris Tane
Description: Praxen's autofix used to be a workflow of multiple agents, each responsible for a single task, with an orchestrator managing state between them. We moved to a single agent which does the entire job, from evidence gathering, hypothesis testing to pull request.
import BlogImage from "../../components/blog/BlogImage.astro";
import BlogTable from "../../components/blog/BlogTable.astro";
import DailyChart from "../../components/blog/DailyChart.astro";
import DataTable from "../../components/blog/DataTable.astro";
import { dayLabel, hours, pct, usd } from "../../components/blog/format";
import { FIX_RUN_CUTOVER, FIX_RUN_DAYS } from "../../data/blog/fix-run-delta";
Praxen's [autofix agent](/use-cases/ticket-resolution/) was originally built as a workflow of multiple sub-agents, each responsible for a single task:
- a triage to sort through thousands of alerts and signals
- a coordinator to operate sub-agents
- up to fifteen sub-agents to investigate all possible hypotheses for the issue
- and a coding agent to ultimately submite a pull request
```mermaid alt="Triage hands an issue to a coordinator, up to 15 hypothesis agents and a coding agent"
graph TB
S["Alerts and signals"] --> T["Triage agent"]
T -->|"confirmed issue"| C["Coordinator agent"]
C -->|"hypotheses"| H["Up to 15 hypothesis sub-agents"]
H -->|"verdicts"| C
C -->|"plan"| A["Coding agent"]
A --> PR["Pull request"]
style C fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style H fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
A confirmed issue could lead to up to 18 agents and sub-agents for the investigation and the fix. Today the same work is done by a single agent.
The workflow was a sound design based on what we knew at the time: small prompts, small tool sets, an orchestrator to carry results between specialists. It was however extremely expensive and difficult to operate and reason about.
## What is an issue?
Praxen scans logs, metrics and traces from every cloud resource (Lambda function, Cloudflare Worker, Vercel project, etc.) on every connected provider. For each resource we store baselines across multiple horizons, such that we can capture seasonality in the data. We then assess the current state of the cloud resource and compare it with historical data. Values that break the baseline are recorded as issues.
Issues can also be created by alerts we receive from observability and error tracking solutions.
On AWS, the identifier tells the investigator which evidence to gather. [EC2ThrottledException](/learn/troubleshooting/EC2ThrottledException/) points to Lambda initialization and EC2 API throttling; [AsyncEventsDropped](/learn/troubleshooting/AsyncEventsDropped/) counts unsuccessful asynchronous events. [ChangeMessageVisibility](/learn/troubleshooting/ChangeMessageVisibility/) is an SQS operation, so its failures call for receipt-handle and timeout evidence rather than Lambda async retry settings.
Missing telemetry is a separate hypothesis: check the installed [LambdaInsightsExtension](/learn/troubleshooting/LambdaInsightsExtension/) collector for system metrics, or the application's [AWS_EMF_ENVIRONMENT](/learn/troubleshooting/AWS_EMF_ENVIRONMENT/) configuration for custom EMF metrics. Keeping the raw response and the collection path in the same investigation prevents a handoff from turning those different problems into one generic "Lambda error."
```mermaid alt="Resource telemetry outside its baseline or a provider alert creates an issue"
graph TB
R["Cloud resource"] -->|"telemetry"| B["Compare with its baselines"]
A["Alert from an observability or error tracking provider"] --> I
B -->|"breaks the baseline"| I["Issue"]
B -->|"within the baseline"| N["No issue"]
style I fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
## How do we resolve issues?
To resolve an issue, we need to:
1. **Triage:** Gather data to confirm or reject the issue.
2. **Investigate:** Assess multiple hypotheses and determine the root cause.
3. **Open a pull request or escalate.** Either write a pull request to resolve the issue, escalate to an engineer if the fix is not a code change, or write a report highlighting the findings and stop.
The vast majority of runs terminate before a pull request or an escalation, when Praxen concludes the issue is either a false positive or benign.
```mermaid alt="Triage and investigation end in a report, a fix PR or escalation to an engineer"
graph TB
I["Triage the issue"] --> V["Investigate the root cause"]
V -->|"dismiss, fold, or reopen"| E["Report and stop"]
V -->|"confirm"| X["Open a pull request"]
V -->|"confirm, no code change"| Z["Escalate to an engineer"]
style V fill:#fef3c7,stroke:#fcd34d,color:#78350f
style X fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
## Sub-agents architecture
Our initial architecture was based on multiple sub-agents.
```mermaid alt="An orchestrator fans out hypotheses, aggregates agent verdicts and hands a plan to a coding agent"
graph TB
F["Finding"] --> T["Triage agent"]
T -->|"confirm"| C["Orchestrator agent"]
C -->|"hypotheses"| W["Fan-out workflow"]
W --> H1["Hypothesis agent 1"]
W --> H2["Hypothesis agent 2"]
W --> H3["Hypothesis agent ..."]
W --> H15["Hypothesis agent 15"]
H1 --> AG["Vote and summarize"]
H2 --> AG
H3 --> AG
H15 --> AG
AG -->|"summary as a message"| C
C -->|"confirmed hypothesis"| AF["Coding agent"]
AF --> PR["Pull request"]
style C fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style AG fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
We used an orchestrator agent to cordinate work between multiple sub-agents. Sub-agents were instanciated to investigate hypotheses, trying to prove or disprove each hypothesis with different starting points (for example starting by looking at the logs, or starting with the codebase, etc.).
The findings of the sub-agents were voted with simple arithmetic and summarised by another agent before being rolled up to the coordinator agent.
Each hypothesis's verdict was a confidence-weighted vote across its passes.
```ts
const CONFIDENCE_RANK = { definitive: 4, strong: 3, moderate: 2, weak: 1, speculative: 0 };
function aggregatePassVerdicts(passes: PassResult[]): Verdict {
const weights: Record = {};
const counts: Record = {};
for (const p of passes) {
weights[p.verdict] = (weights[p.verdict] ?? 0) + CONFIDENCE_RANK[p.confidence];
counts[p.verdict] = (counts[p.verdict] ?? 0) + 1;
}
const totalWeight = Object.values(weights).reduce((a, b) => a + b, 0);
const [winner, winnerWeight] = Object.entries(weights).sort((a, b) => b[1] - a[1])[0];
const countMajority = counts[winner] > passes.length / 2;
const weightMajority = winnerWeight > totalWeight / 2;
if (!countMajority && !weightMajority) {
return { verdict: "inconclusive", confidence: "weak", summary: `No consensus across ${passes.length} passes.` };
}
const majority = passes.filter((p) => p.verdict === winner);
const confidence = majority.reduce((min, p) => (CONFIDENCE_RANK[p.confidence] < CONFIDENCE_RANK[min] ? p.confidence : min), majority[0].confidence);
const lead = majority.reduce((best, p) => (CONFIDENCE_RANK[p.confidence] > CONFIDENCE_RANK[best.confidence] ? p : best));
return { verdict: winner, confidence, summary: lead.summary };
}
```
Once all hypotheses were investigated, if a root cause was identified, the coordinator would either escalate to an engineer or write a plan for the fix. This plan would be handed to a coding sub-agent to implement the fix and submit the pull request.
The main issue in this architecture is the context loss between sub-agents. Each agent was passing up or down just fractions of their context, through summarization, and both upstream and downstream agents regularly had to redo the same work.
```mermaid alt="Sub-agent handoffs lose evidence as briefs become summaries and then a coding plan"
graph TB
I["Issue"] --> O["Orchestrator splits the issue into briefs"]
O -->|"brief A"| A["Sub-agent A investigates brief A"]
O -->|"brief B"| B["Sub-agent B investigates brief B"]
A -.->|"summary A"| M["Orchestrator, later writes the plan from the summaries"]
B -.->|"summary B"| M
M -.->|"plan"| F["Coding agent writes the fix"]
F --> PR["Pull request"]
style I fill:none,stroke:none
style PR fill:none,stroke:none
style O stroke:#4C8C57,color:#3d7046
style A stroke:#3b7dd8,color:#2b5fa8
style B stroke:#7c5cd6,color:#5b3fb0
style M stroke:#a07d1c,color:#7a5f14
style F stroke:#6b7280,color:#4b5563
%% aside O right #4C8C57 Issue, alerts and history
%% aside A left #3b7dd8 Brief A and its evidence
%% aside B right #7c5cd6 Brief B and its evidence
%% aside M right #a07d1c History and two summaries, no evidence
%% aside F right #6b7280 The plan, nothing else
```
Moreover, the agent writing the fix only had the plan, lacking context on the original issue or the evidence collected in the investigation. This led to sub-par pull requests which often focused on symptoms rather than the root cause.
We chose this architecture primarily because when we first designed it in March 2026, frontier models were not always capable of running an investigation end-to-end, including writing the fix, and also plainly, our harness was not that good.
It became increasingly harder to build on this foundation because:
- **Debugging took many traces.** Reasoning around an end-to-end investigation required looking through multiple, sometimes dozens of traces
- **Evaluation was per stage.** Each sub-agent could be evaluated individually and pass, while the end result was poor, because the failures lived in the handoffs.
We iterated on this architecture for months, improving each agent individually and tweaking the prompts and the handoffs, but the results never matched the effort. It was expensive, hard to debug, and the pull requests were not good enough.
## Single agent
On Sept 3, we replaced the sub-agents workflow with a single agent doing everything from triage to pull request. The same agent collects the evidence, investigates all the hypotheses, and submits the pull request.
```mermaid alt="One agent handles triage, investigation and a fix PR or escalation without handing off context"
graph TB
F["Issue"] --> A["Single agent"]
A --> I["Triage"]
I --> B["Investigate"]
B --> V["One verdict"]
V -->|"confirm"| X["Clone, edit, validate in the sandbox"]
X --> PR["Open the pull request"]
V -->|"dismiss, fold, or reopen"| E["Report and stop"]
V -->|"confirm, no code change"| Z["Escalate to an engineer"]
style A fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style PR fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
We deleted the triage agent, the coordinator, the hypothesis agents, and the autofix agent, along with the workflows that carried results between them and the gates that sat in front of the pull request. The operational benefits were immediate: one trace to evaluate, and no summaries between sub-agents.
More importantly, the quality of the pull requests improved, and the time between detecting an issue and opening the pull request that fixes it collapsed. Over the month around the cutover, the median went from 2.2 hours to 35 minutes and the p90 from nine days to under two hours. Under the pipeline a finding typically sat for multiple hours in a chain of workflows, each waiting on the last. Under the single agent, the pull request opens just minutes after the issue is detected.
({ day: d.day, value: d.medianHours, band: d.p90Hours, n: d.withPr, partial: d.partial }))}
unit="hours"
scale="ticks"
ticks={[
{ v: 0.25, label: "15 min", span: 1.25 },
{ v: 1, label: "1 h", span: 1.25 },
{ v: 4, label: "4 h", span: 2.5 },
{ v: 24, label: "1 d" },
{ v: 96, label: "4 d" },
{ v: 336, label: "2 wk" },
]}
cutover={FIX_RUN_CUTOVER}
cutoverLabel="3 Sep: one agent"
valueName="Daily median"
bandName="p90"
height={330}
label="Figure 1"
title="Hours from detecting an issue to opening its pull request"
caption=""
/>
We now also open far more pull requests. When we were using sub-agents, 0.6% of detected issues ended in a pull request. Under the single agent it is 4.2%, and rising. The difference is failure modes. A pipeline with multiple sub-agents has to survive every handoff: triage has to promote the finding, the coordinator has to produce hypotheses, the fan-out has to reach a verdict, etc. Each handoff is a potential failure mode.
({ day: d.day, value: d.prShare, partial: d.partial }))}
unit="percent"
ticks={[
{ v: 0, label: "0%" },
{ v: 2, label: "2%" },
{ v: 4, label: "4%" },
{ v: 6, label: "6%" },
{ v: 8, label: "8%" },
{ v: 10, label: "10%" },
]}
cutover={FIX_RUN_CUTOVER}
cutoverLabel="3 Sep: one agent"
valueName="Share with a pull request"
label="Figure 2"
title="Share of detected issues that ended in a pull request"
caption=""
/>
Cost per pull request fell as well. Every run now starts on the stronger model, so a dismissed finding costs more than it did under the pipeline. But, the average cost per pull request went from $111 to about $18 in the first nine days of the single agent. This drop is not exclusively the result of the change in architecture, as we're actively iterating on every aspect of Praxen. Some of it is the ordinary drift of a system under constant iteration.
({ day: d.day, value: d.spendPerPr, partial: d.partial }))}
unit="usd"
ticks={[
{ v: 1, label: "$1" },
{ v: 10, label: "$10" },
{ v: 100, label: "$100" },
{ v: 1000, label: "$1,000" },
]}
cutover={FIX_RUN_CUTOVER}
cutoverLabel="3 Sep: one agent"
valueName="Spend per pull request"
label="Figure 3"
title="Model spend per pull request opened"
caption=""
/>
d.day === FIX_RUN_CUTOVER}
columns={[
{ header: "Day", cell: (d) => dayLabel(d.day), note: (d) => (d.day === FIX_RUN_CUTOVER ? "one agent" : d.partial ? "to 16:00" : "") },
{ header: "Detected issues with a pull request", cell: (d) => pct(d.prShare) },
{ header: "Median", cell: (d) => hours(d.medianHours) },
{ header: "p90", cell: (d) => hours(d.p90Hours) },
{ header: "Spend per pull request", cell: (d) => usd(d.spendPerPr) },
]}
/>
## Don't build sub-agents
- **Handoffs lose more than they save.** Every summary passed between agents is context the next agent will never have. The agent that gathers the evidence should be the agent that acts on it.
- **Evaluate the run, not the agents.** Per-agent evaluations pass while the system fails, because the failures live between the agents.
- **Keep one trace per run.** A wrong decision spread across a dozen traces takes an afternoon to explain. In one trace it takes a scroll.
---
### How We Fixed Our Cloudflare Durable Objects Memory Exceeded Errors
URL: https://getpraxen.app/blog/how-we-fixed-our-cloudflare-durable-objects-memory-exceeded-errors/
Date: Aug 26, 2026
Author: Aleksandr Diamond,Boris Tane
Description: Cloudflare Durable Objects run in V8 isolates with a hard 128 MB limit. Ours idled at ~140 MB and was reset ~300 times a day. 130 MB of zod schemas were built at module load, most of them by code that was never called. How we profiled the production bundle, the two fixes, and the numbers, 218 MB to 82 MB and resets to zero.
import BlogImage from "../../components/blog/BlogImage.astro";
import BlogTable from "../../components/blog/BlogTable.astro";
Every agent thread in Praxen runs inside its own Cloudflare Durable Object, all of them instances of one class. In August 2026, the Durable Objects of our threads were reset about 300 times a day for exceeding the memory limit. We heard the same story from multiple teams running their agentic flows on Cloudflare Durable objects.
If you've hit `exceededMemory` on a Durable Object, the usual advice is to look at what your requests allocate: a payload that is too big, a chat history that grows without bound, a cache that never evicts. None of that applied to us. The isolate was over the limit before it served a single request, so the weight had to be in the code we shipped rather than the data we served.
This post is about what was actually in the heap, how we found it on a platform that doesn't let you attach a profiler, and the two changes that took module-scope memory from 218 MB to 82 MB and the resets to zero.
## Cloudflare Durable Objects in one minute
If you haven't used them: a Durable Object is a small, stateful, single-threaded server that Cloudflare guarantees is unique for a given ID. Call `idFromName()` with that ID from a Worker and every request for it, from anywhere in the world, lands on the same instance, with its own SQLite database, in-memory state and alarms. It hibernates when idle and wakes up where it left off.
In Praxen, we create a Durable Object instance for each thread. Every conversation with the agent, whether a person started it or an alert did, gets its own Durable Object. The object's SQLite holds the thread's messages and tool results, the agent loop runs inside it, and its tools call out to whichever providers the thread needs: Datadog, Sentry, Honeycomb, GitHub, Cloudflare and the rest. When the thread goes quiet the object hibernates, and when the next message arrives it picks up exactly where it stopped.
```mermaid alt="A Worker routes each thread to a Durable Object with SQLite storage and an agent loop calling provider tools"
graph TB
U["New message or alert"] --> W["Worker"]
W -->|"idFromName(threadId)"| DO["One Durable Object per thread"]
DO --> S["SQLite: messages, tool results"]
DO --> L["Agent loop"]
L --> T["Tools"]
T --> P1["Datadog"]
T --> P2["Sentry"]
T --> P3["Honeycomb"]
T --> P4["GitHub, Cloudflare, ..."]
style DO fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
## How Durable Objects memory works
- **Isolates and the 128 MB cap.** Durable Objects run inside [V8 isolates](https://developers.cloudflare.com/workers/reference/how-workers-works/#isolates), and an isolate has a [hard memory limit of 128 MB](https://developers.cloudflare.com/workers/platform/limits/#memory) that is the same on every plan and not configurable.
- **Co-location.** [A single isolate hosts many Durable Objects of the same class](https://developers.cloudflare.com/durable-objects/observability/metrics-and-analytics/#memory-usage), along with the Worker code around them, and they all share that isolate's memory. The limit is per isolate rather than per object.
- **Noisy neighbours.** Because of co-location, the object that gets reset when the isolate [runs out of memory](https://developers.cloudflare.com/workers/platform/limits/#memory) is often not the object that used the memory. [Every memory sample Cloudflare reports is the whole isolate's](https://developers.cloudflare.com/durable-objects/observability/metrics-and-analytics/#memory-usage), and so is every reset.
```mermaid alt="Durable Object instances share module-scope memory and a 128 MB isolate cap; exceeding it resets an instance"
graph TB
subgraph ISO["One V8 isolate, 128 MB cap"]
M["Module scope, loaded once"]
D1["Instance A"]
D2["Instance B"]
D3["Instance C"]
end
M --- D1
M --- D2
M --- D3
D3 -->|"heap crosses 128 MB"| R["exceededMemory, C is reset"]
style M fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style R fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
There is no way to attach a memory profiler to a production Durable Object: the [heap snapshot tooling Cloudflare documents](https://developers.cloudflare.com/workers/observability/dev-tools/memory-usage/) runs against a local dev session. In Kubernetes you'd turn profiling on and attach until you found the problem. In workerd, `process.memoryUsage()` is an [unenv polyfill that returns zeros](https://github.com/unjs/unenv/blob/main/src/runtime/node/internal/process/process.ts), and the baseline memory and CPU figures aren't visible to the code running inside. What you get is Cloudflare's [GraphQL analytics API](https://developers.cloudflare.com/durable-objects/observability/metrics-and-analytics/#query-via-the-graphql-api): [memory percentiles](https://developers.cloudflare.com/durable-objects/observability/metrics-and-analytics/#memory-usage) and crash counts per namespace, after the fact.
## What the memory graph told us
Cloudflare's analytics API reports memory for a Durable Object namespace as percentiles across the isolates running it, in fifteen-minute buckets, alongside the count of exceeded-memory errors in each bucket. Every graph in this post comes from that API, filtered to the thread namespace and compared across deploy markers once we started shipping fixes.
The median isolate, the blue line in figure 1, sat at about 140 MB for the entire week, and the 99th percentile sat near 190 MB. Both lines are above the dashed 128 MB limit, which means the typical isolate in the namespace was already past the point where Cloudflare is entitled to reset it, and was only spared until the next allocation tipped it over. What the graph did not show was any relationship to traffic. The line was as flat during the quiet hours as during the busy ones, and none of the roughly 300 resets a day came with a stack trace, because it was never our code that threw.
We looked where you would expect to look first. We read the payload sizes, we checked how chat history was truncated, and we searched the memory curve for a request pattern, and found nothing that moved the line. That flatness turned out to be the important clue. An isolate's memory is one of two things: it is **data**, meaning request payloads, chat history, tool outputs and anything else allocated while serving traffic, or it is **baseline**, meaning the objects the code itself creates when a module loads and keeps alive for the isolate's lifetime, such as imports, functions and schemas. Data rises and falls with requests, while baseline is there before the first request and never goes away, so a graph that is high and flat at idle is describing baseline. The problem had to be in what we were shipping rather than in what we were serving.
```mermaid alt="Isolate heap splits into module-load baseline allocations and per-request data"
graph TB
H["Isolate heap"] --> B["Baseline: built at module load"]
H --> D["Data: allocated per request"]
B --> B1["imports"]
B --> B2["tool schemas"]
B --> B3["package barrels"]
D --> D1["payloads"]
D --> D2["chat history"]
D --> D3["tool outputs"]
style B fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
## Why baseline memory is so easy to miss
Baseline memory hides in two places you don't normally look.
The first is the bundler. We assume tree shaking removes the code we don't use, and mostly it does, but esbuild has rules about when it can prune and they aren't the rules you'd guess. A package without `"sideEffects": false` in its `package.json` is never pruned at all. An `export *` inside a lazily evaluated module is never pruned either. A dynamic `import()` of a package root marks every export as used. We'll come back to each of these in the fixes, because they explain half of the 130 MB.
```ts
// Three ways to keep every zod schema in a package alive in the isolate,
// none of which look like a mistake.
// The package has no "sideEffects": false, so nothing in it is pruned.
import { getWorkspaceStub } from "@scope/durable-workspaces";
// Inside a lazily evaluated module, export * is never pruned.
export * from "./zod";
// A namespace object marks every export as used, schemas included.
const pkg = await import("@scope/durable-workspaces");
```
The second is the schema library. Types feel free because they vanish at compile time, but a zod schema is not a type, it is a tree of closures built the moment its module evaluates. One mid-sized zod 4 object schema, a dozen fields with descriptions and refinements, costs ~134 KB of heap. A bare `z.string()` costs ~12 KB. The equivalent plain JSON schema object costs a few hundred bytes. None of that is in the README.
```ts
// What each of these costs the moment its module evaluates.
const Params = z.object({ // ~134 KB for a dozen fields like these
owner: z.string().describe("Repository owner"),
repo: z.string().describe("Repository name"),
pullNumber: z.number().int().describe("PR number"),
});
const Name = z.string(); // ~12 KB
const params = { // a few hundred bytes
type: "object",
properties: { owner: { type: "string" }, repo: { type: "string" } },
} as const;
```
## We had done nothing exotic
Every one of our problems came from a default. Every agent tool declared its input with `z.object()` at module scope, because that is how the docs do it. Every internal package had a barrel `index.ts` with `export * from "./zod"`, because that is tidy. A few paths used `await import("@scope/package")` because lazy loading is supposed to be cheaper. Each one is a reasonable choice on its own. Together they were 130 MB, in an isolate with 128 to spend.
```ts
// packages/tools/src/github/get-pull-request.ts, and 250 more like it
export const parameters = z.object({
owner: z.string().describe("Repository owner"),
repo: z.string().describe("Repository name"),
pullNumber: z.number().int().describe("PR number"),
});
// packages/thread-core/src/index.ts, and every other internal package
export * from "./zod";
export * from "./thread";
// packages/durable-threads/src/agent.ts
const { getWorkspaceStub } = await import("@scope/durable-workspaces");
```
## What it was costing us
Our agent has over 250 tools, which it uses through code mode and dynamic workers. Their zod definitions alone cost 78 MB of heap at module load, more than half the isolate's budget, spent on descriptions of arguments before any of them was called. Every tool we added cost another ~134 KB, whether or not that tool ever ran.
```text
250+ tools × ~134 KB of zod each ≈ 78 MB evaluated before the first request
128 MB isolate cap − 78 MB = 50 MB left for every thread's actual data
```
The resets weren't free either. Every `exceededMemory` throws away an in-flight agent turn: a retry, a second model call, a user watching a spinner. At ~300 a day that stops being an incident and becomes a standing tax, and because of co-location it was paid by whichever object happened to be in the isolate at the time.
```text
Error: Durable Object's isolate exceeded its memory limit and was reset.
```
## How we found it: a heap profiler for the production bundle
We couldn't profile production, so we built a small profiler that runs locally on the exact bundle production runs. It measures the V8 heap cost of evaluating every module and prints a ranked table. We ran it in a loop: profile, remove the top of the ranking, profile again. The final check for every fix was a production deploy compared against Cloudflare's memory metrics on either side of the deploy marker.
```mermaid alt="A dry-run production bundle is instrumented and run under Node to measure exclusive heap per module"
graph TB
A["wrangler deploy --dry-run"] --> B["production bundle + metafile"]
B --> C["instrument the __esm helper"]
C --> D["run under Node with stubs"]
D --> E["exclusive heap per module"]
style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
Three ideas make it work:
- **Same bundle.** `wrangler deploy --dry-run --outdir --metafile` emits the exact esbuild bundle a deploy would upload, plus its module graph. Everything is measured on that bundle.
- **Module attribution.** esbuild wraps lazily evaluated modules in `__esm(...)` initialiser closures. We rewrite that single helper in the built bundle so every module init records `v8.getHeapStatistics().used_heap_size` before and after itself, with an init stack separating a module's own cost (exclusive) from its dependencies' (inclusive). The result is a flamegraph-shaped dataset.
- **Controlled runs.** The instrumented bundle runs under plain Node with a loader shim that resolves `cloudflare:*` imports to stubs.
The complete recipe, ready to paste into a coding agent, is at the end of this post. The zod unit costs quoted above come from a micro-benchmark run through the same harness.
### What the first profile showed
| Source | Exclusive heap |
| --- | --- |
| `packages/tools` (250+ agent tool definitions, zod) | 78.0 MB |
| Package barrels re-exporting zod schema modules (rows below) | ~66 MB |
| `durable-workspaces` (workspace state and schedules) | 15.6 MB |
| `thread-core` (thread data layer) | 10.1 MB |
| `durable-automations` (automation definitions) | 9.6 MB |
| `durable-threads` (thread list and live updates) | 7.3 MB |
| `durable-automation` (one automation run) | 6.8 MB |
| `db` (D1 client and models) | 6.3 MB |
| `durable-skills` (skill definitions) | 5.7 MB |
| `durable-autofixes` (autofix branches and merges) | 4.9 MB |
| 12 smaller packages | ~9 MB |
The second row is the surprising one. Those are schemas that reach the bundle only through `export * from "./zod"` in package barrels. Our code never used them, the tree shaker could not remove them, and they cost a third of the memory limit, all of it in modules that nothing ever called.
### Fix A: tool definitions as data, not code
Every tool definition declared its input schema in zod and converted it to JSON schema at runtime, because JSON schema is what the model gets sent anyway. We were building ~134 KB of closures per tool to produce a few hundred bytes of data, so we wrote the data directly.
```mermaid alt="Direct JSON schemas replace 134 KB of runtime Zod allocations per tool with 0.3 KB of data"
graph TB
Z["zod schema, ~134 KB per tool"] --> J["toJSONSchema() at runtime"]
J --> M["JSON schema sent to the model"]
D["JSON schema, ~0.3 KB per tool"] --> M
style Z fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style D fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
```ts
// before: ~134 KB of closures per tool, built at module load
export const parameters = z.object({
owner: z.string().describe("Repository owner"),
repo: z.string().describe("Repository name"),
pullNumber: z.number().int().describe("PR number"),
});
// after: a few hundred bytes of data, and the type comes from the schema
export const parameters = defineParameters({
type: "object",
properties: {
owner: { type: "string", description: "Repository owner" },
repo: { type: "string", description: "Repository name" },
pullNumber: { type: "number", description: "PR number" },
},
required: ["owner", "repo", "pullNumber"],
} as const);
export type Input = ParametersInput;
```
`defineParameters` is a thin wrapper over the AI SDK's `jsonSchema()`. `ParametersInput` is `json-schema-to-ts` doing the type inference zod used to do. Runtime validation moved to a ~300-line validator that reproduces the properties the agent loop depends on: unknown keys stripped, defaults filled, unions resolved by discriminator, and error messages worded so the model can repair its own tool call on retry.
Module-evaluation heap went from 218.6 to 154.0 MB locally, and `packages/tools` from 78 MB to 0.7 MB. In production, resets fell from 40-110 an hour to 0-6 an hour at the deploy marker, and the median isolate's memory from ~140 MB to ~120 MB.
### Fix B: barrels that can actually be tree-shaken
The remaining ~66 MB was schemas the isolate never used. Three bundler behaviours explain it:
1. Without `"sideEffects": false` in a package's `package.json`, esbuild prunes nothing from it.
2. Even with the flag, `export * from "./zod"` is never pruned inside a lazily evaluated module, and anything reachable through a dynamic `import()` is lazily evaluated. Named re-exports (`export { zFoo } from "./zod"`) are pruned fine.
3. `await import("@scope/package")` materialises the package's entire namespace object, marking every export as used: schemas, classes, everything.
We confirmed each of these with a five-file fixture built with the esbuild version wrangler embeds: an entry, a package with an `index.ts` barrel, a `zod.ts` holding one schema whose constructor announces when it runs, a `do.ts` class importing that schema, and a lazy intermediary between them. The table records whether the schema's constructor ran at evaluation for each combination.
| Entry imports barrel via | Barrel re-export shape | `sideEffects: false` | Schema included |
| --- | --- | --- | --- |
| Static import | `export *` | Yes | No |
| Static import | `export *` | No | Yes |
| Dynamic `import()` of package | Any | Yes | Yes |
| Static import from a lazily loaded module | `export *` | Yes | **Yes** |
| Static import from a lazily loaded module | Named list | Yes | No |
| Any | Named list | No | Yes |
Rows 3 and 4 are the two that surprise people, and together they accounted for the ~66 MB. Here is what one barrel was doing to us:
```mermaid alt="Importing a small helper through a package barrel also loads 15.6 MB of unused Zod schemas"
graph TB
A["do.ts imports one function"] --> B["durable-workspaces barrel"]
B -->|"export * from './zod'"| C["zod/*.ts, 15.6 MB of schemas"]
B -->|"used"| D["getWorkspaceStub(), ~1 KB"]
style C fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style D fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
The fix is mechanical: `"sideEffects": false` in all 110+ workspace packages, rewrite the barrels' `export *` to named re-export lists, and replace `await import("@scope/package")` of package roots with static named imports.
An ablation on the real production bundle shows each piece is necessary.
| Configuration | Heap after module evaluation | Schema modules in bundle |
| --- | --- | --- |
| Baseline (after fix A) | 154.0 MB | 77 |
| `sideEffects: false` alone | 143.6 MB | 77 |
| + Static imports, barrels back to `export *` (control) | 98.6 MB | 29 |
| + Named re-export lists (full fix) | **82.1 MB** | **0** |
The control row is the interesting one: even a fully static import graph keeps 29 schema modules, so the named lists are not optional.
### The end state
| | Module-scope heap (local probe) | Production median memory | Production resets |
| --- | --- | --- | --- |
| Before | 218.6 MB | ~140 MB | ~300/day |
| After A | 154.0 MB | ~120 MB | ~10/day |
| After A+B | 82.1 MB | ~70 MB | 0 |
A Durable Object that idled above the platform's memory limit now idles at barely half of it, and the ~300 daily resets are gone.
## This is not a config flag
Rewriting more than 250 tool definitions from zod to raw JSON schema, and writing a 300-line validator to replace what zod did for us, took a few days, even with coding agents. Adding `sideEffects: false` to over 100 packages and turning every `export *` into a generated named list is unglamorous work, and you typecheck the whole repo afterwards and fix what breaks. The easy alternative is to add a retry and live with the resets, and plenty of teams do. If your Durable Object is anywhere near the limit at idle, I'd argue the week is worth it, because the limit doesn't move and your tool count only goes up.
The three things I'd want to have known from day one:
- **High and flat at idle means baseline.** Profile what you ship, not what you serve.
- **Tree shaking has rules.** `sideEffects: false`, named re-exports, static imports. Miss one and the whole package rides along.
- **Schemas are code, not types.** If the consumer wants JSON schema, write JSON schema.
## Do this today to fix your Durable Object's memory
The profiler is ~100 lines with no dependencies beyond Node and wrangler. Paste the recipe below into your coding agent at the root of any repo that deploys with wrangler, read the top ten rows of the table it prints, and see what's there. Then work down the list: plain JSON schema for anything the model receives as JSON schema anyway, `"sideEffects": false` and named re-exports for the barrels, static imports for the package roots.
````markdown [skills/profile-module-scope-heap.md]
Build a per-module heap profiler for my worker's production bundle.
1. Emit the exact production bundle and metafile:
cd
npx wrangler deploy --dry-run --env --config wrangler.jsonc \
--outdir /tmp/heap-probe --metafile /tmp/heap-probe/meta.json
If the worker's import graph is fully static, esbuild emits no lazy
`__esm` wrappers and per-module attribution is impossible. In that case
build from a probe-only entry that reaches the real entry through a
dynamic import (and satisfies wrangler's Durable Object export check
with a placeholder class):
// probe-entry.ts
export default { fetch: () => new Response("probe") };
export class {}
export const probeLoad = () => import("");
npx wrangler deploy --dry-run --env --config wrangler.jsonc \
--outdir /tmp/heap-probe --metafile /tmp/heap-probe/meta.json probe-entry.ts
2. Instrument the bundle. Write instrument.mjs and run
`node instrument.mjs /tmp/heap-probe`:
import { readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
const outDir = process.argv[2];
const bundlePath = join(outDir, "probe-entry.js"); // or index.js
const source = readFileSync(bundlePath, "utf-8");
const esmHelperPattern = /var __esm = \(fn, res(?:, \w+)?\) => function __init\(\) \{[\s\S]*?\n\};\n/;
if (!esmHelperPattern.test(source)) throw new Error("__esm helper not found; esbuild output shape changed");
const instrumentedHelper = `var __probeInitStack = [];
var __esm = (fn, res, err2) => function __init() {
if (err2) throw err2[0];
if (!fn) return res;
const probe = globalThis.__moduleHeapProbe;
if (!probe) {
try { return (res = (0, fn[__getOwnPropNames(fn)[0]])(fn = 0)), res; }
catch (e) { throw ((err2 = [e]), e); }
}
const moduleName = __getOwnPropNames(fn)[0];
const frame = { child: 0 };
const before = probe.heap();
__probeInitStack.push(frame);
try { return (res = (0, fn[moduleName])(fn = 0)), res; }
catch (e) { throw ((err2 = [e]), e); }
finally {
const total = probe.heap() - before;
__probeInitStack.pop();
if (__probeInitStack.length > 0) __probeInitStack[__probeInitStack.length - 1].child += total;
probe.record(moduleName, total - frame.child, total);
}
};
`;
writeFileSync(join(outDir, "instrumented.mjs"), source.replace(esmHelperPattern, instrumentedHelper));
3. Write a Node loader shim so worker-targeted code loads under Node.
register.mjs resolves `cloudflare:*` to a stub module (an empty module
exporting throwing placeholders for DurableObject, WorkerEntrypoint,
env, etc.) via module.registerHooks; add a `load` hook for any
non-JS rules in your wrangler config (for us: `.sql` files become
text default exports, mirroring wrangler's Text rule).
4. Write probe.mjs and run it:
node --expose-gc --import ./register.mjs probe.mjs /tmp/heap-probe
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import v8 from "node:v8";
const outDir = process.argv[2];
const records = [];
globalThis.__moduleHeapProbe = {
heap: () => v8.getHeapStatistics().used_heap_size,
record: (moduleName, exclusive, total) => records.push({ moduleName, exclusive, total }),
};
const settle = async () => { await new Promise((r) => setTimeout(r, 0)); globalThis.gc(); globalThis.gc(); };
const mb = (b) => (b / 1048576).toFixed(2);
await settle();
const bundle = await import(pathToFileURL(join(outDir, "instrumented.mjs")).href);
if (bundle.probeLoad) await bundle.probeLoad();
await settle();
console.log(`heap after module evaluation: ${mb(v8.getHeapStatistics().used_heap_size)} MB`);
for (const r of records.sort((a, b) => b.exclusive - a.exclusive).slice(0, 40))
console.log(`${mb(r.exclusive).padStart(8)} MB ${r.moduleName}`);
const byPackage = new Map();
for (const r of records) {
const m = r.moduleName.match(/node_modules\/((?:@[^/]+\/)?[^/]+)|(packages\/[^/]+)/);
const key = m ? (m[1] ? `npm:${m[1]}` : m[2]) : "other";
byPackage.set(key, (byPackage.get(key) ?? 0) + r.exclusive);
}
for (const [k, v] of [...byPackage.entries()].sort((a, b) => b[1] - a[1]))
if (v > 131072) console.log(`${mb(v).padStart(8)} MB ${k}`);
5. Gotchas that will otherwise burn an afternoon:
- The bundle's unenv polyfill replaces globalThis.process at init and its
memoryUsage() reports zeros. Read v8.getHeapStatistics() instead.
- Compare deltas between runs of this probe, never absolutes against
production: Node's heap baseline differs from workerd's.
- Modules evaluated eagerly at the top level (not wrapped in __esm) are
invisible to attribution; the probe-entry trick in step 1 fixes that.
````
If you want the whole audit done for you, here is a skill for your coding agent:
````markdown
Audit this repo's worker bundles for module-scope schema weight, and fix what you find. Work in this order and show me numbers at every step.
1. Baseline. Using the "profile module-scope heap" recipe, build the
production bundle of our most memory-sensitive worker and produce the
per-module and per-package exclusive-heap ranking. Report heap after
module evaluation.
2. Identify schema weight. From the ranking and the esbuild metafile, list
every module matching your schema conventions (zod/valibot/etc. modules,
e.g. packages/*/zod*) that survived into the bundle, with bytes. For each,
compute one import chain from the entry using the metafile's `imports`
graph (BFS), so we know *why* it is in the bundle.
3. Classify each surviving schema module:
a. Actually used at runtime by this worker: leave it, or move the boundary.
b. Reached through `export *` in a package barrel: candidate for named lists.
c. Reached through `await import("")`: candidate for a static
named import.
d. Reached because the package lacks `"sideEffects": false`: candidate flag.
4. Apply, in this order, re-profiling after each:
a. Add `"sideEffects": false` to every internal package that has no
import-time side effects. Audit first: grep package sources for
top-level globalThis mutations, addEventListener, polyfill assignment.
Any true side-effect file gets `"sideEffects": ["./that-file.ts"]`.
b. Rewrite `export * from "./"` in package barrels to explicit
`export { ... }` / `export type { ... }` lists. Generate the lists with
the TypeScript compiler API (walk ExportDeclarations recursively,
classify value vs type), never by regex. Typecheck the repo after.
c. Replace every value-position `await import("@scope/pkg")` of a bare
package root with a static named import of the symbols actually used.
Check the site is not lazy for a *different* reason first (circular
imports, Node-only test loading, genuine cold-path npm dependency).
5. If tool/LLM definitions build schema-library objects at module scope,
propose converting them to plain JSON schema with types via
json-schema-to-ts, and estimate the saving from step 1's ranking before
doing it.
6. Verify: re-profile (report the delta), typecheck, run the affected
packages' tests, and dry-run build every worker. Then add a CI assertion
that reads the metafile of the memory-sensitive worker and fails if any
schema module survives tree shaking into it, printing the import chain.
7. After deploy, compare the platform memory metrics across the deploy
marker and report before/after median and 99th percentile memory, and reset
counts.
````
We have also added this skill to [Praxen itself](/for/cloudflare/), so every Praxen user gets this deep investigation into their Durable Object memory out of the box, with nothing to paste.
---
### I'm Betting My Company on Proactive Agents
URL: https://getpraxen.app/blog/proactive-agents/
Date: Jul 5, 2026
Author: Boris Tane
Description: Agents can do almost anything you ask. That's the problem, you still have to ask. The next evolution is agents that figure out the job to be done. I started with operations, because nobody should be on-call in 2026.
import PromptBox from "../../components/blog/PromptBox.astro";
import CliBox from "../../components/blog/CliBox.astro";
import AutomationCard from "../../components/blog/AutomationCard.astro";
import PullRequestCard from "../../components/blog/PullRequestCard.astro";
Agents can do almost anything you ask them to, and that's the problem: **you still have to ask**.
I keep hearing people call agents "digital coworkers". That framing is wrong. A coworker who sits in silence until you hand them a perfectly-scoped task, completes it, then goes back to waiting, is not a coworker. Every agent you've ever used works exactly like this. The models got smarter, the harnesses got better, the runs got longer, but the interface never changed: you bring the work, the agent brings the labour.
And most people haven't noticed, because the prompt box has quietly become what AI *is* in our heads.
## Every interface is still a prompt box
The input box is how this whole era started. ChatGPT put one on top of a model and became the fastest-growing product in history, and we all copied it. Every AI product since has been a variation on the same interaction: human types, machine responds, machine waits.
Claude Code was the next evolution. The agent moved into your terminal, picked up your files, your shell, and your git history, and started doing real work instead of talking about it. It changed what agents could do, but not how they start: you type, it works, it stops, and it waits for you to type again.
Then agents moved to the cloud. Codex, Devin, Claude Code on the web. They run for hours instead of minutes, spin up sub-agents to parallelise the work, and don't die when you close your laptop. You hand one a task before lunch and come back to a pull request.
```mermaid alt="A human task fans out to cloud sub-agents that produce a pull request"
graph TD
A[You write the task] --> B[Cloud agent]
B --> C[Sub-agent]
B --> D[Sub-agent]
B --> E[Sub-agent]
C --> F[Pull request]
D --> F
E --> F
style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
The prompt even stopped being a keyboard-only thing. Background agents can be kicked off by an alert or a webhook, and most agent platforms now offer automations: when this event fires, or this cron ticks, run an agent with these instructions.
But what is an automation? It's **a prompt you wrote in advance**. You predicted the failure mode, picked the event, and wrote down what to do about it. The trigger fires the agent, but the judgement inside it is yours, frozen at setup time. An automation catches exactly what you anticipated, and nothing else.
## You are the scheduler
Strip away the tooling and the division of labour hasn't moved in three years. The agent does the work. Deciding what the work is remains your job.
You read the dashboards, you listen to the users, you figure out what matters, and you compress everything you've learned into a prompt, either live at the keyboard or ahead of time in a trigger. The agent executes brilliantly, but every piece of judgement in the system originates with you.
```mermaid alt="Dashboards, alerts and complaints still need a human to prompt or configure an agent"
graph TD
A[Dashboards] --> D[You]
B[Alerts] --> D
C[User complaints] --> D
D --> E[The prompt you type today]
D --> F[The automation you configured last month]
E --> G[Agent]
F --> G
style D fill:#fee2e2,stroke:#fca5a5,color:#991b1b
```
## The next evolution: agents that find the work
I'm betting the next evolution is agents figuring out the job to be done. No prompt, no trigger to configure, no instructions written in advance. You connect your stack, and the agent finds the work by itself: it watches the same signals you watch, notices what's off, decides whether it matters, and starts working on it autonomously.
This is easy to say, and brutally hard to build, because proactivity is three problems stacked on top of each other, and skipping any one of them gives you something worse than the agent behind an input box.
**Context.** Obviously, the agent needs a live model of the world it operates in, not a snapshot you pasted into the context window at prompt time. A reactive agent with bad context gives you a bad answer. A proactive agent with bad context drops your prod database because it thought it was staging.
**Judgement.** This is what has been giving me the hardest time. At any given moment, thousands of things in a production system are slightly off. An agent that flags all of them is a noise machine, noise machines get muted, and muted agents are dead agents. The entire value of proactivity lives in the gap between "something changed" and "something matters":
```json [triage-verdicts.json]
[
{
"signal": "memory up 3% on checkout-edge",
"verdict": "no anomaly",
"reasoning": "within the seasonal range for this hour on this worker"
},
{
"signal": "new error pattern, 2 minutes after deploy 9f3c2a1",
"verdict": "incident",
"reasoning": "error class never seen on this worker, tightly correlated with a deploy"
}
]
```
**Action.** Noticing without acting is just a smarter alert, and alerts are the thing I'm trying to kill. The agent has to finish the job, inside boundaries that make autonomy safe: reversible actions, receipts for everything, and a hard gate before anything irreversible happens.
None of these three involve the agent deciding **what good looks like**. It decides what's off, whether it matters, and what to do about it, and that's the whole list, because in production nobody has to define good: errors at zero, latency at baseline, queues drained, certificates valid. The desired state comes with the territory.
Which means the real shift isn't from "you prompt" to "the agent prompts itself". It's from **imperative operations to declarative operations**. An automation is imperative: you enumerate the failure modes in advance and script a response to each. A proactive agent is a reconciliation loop: it holds the system you have up against the system you should have, and works to close the gap. Kubernetes did this for infrastructure a decade ago, you write down three replicas and a controller does whatever it takes to keep three replicas alive. Nobody has done it for the operation of software itself. And here there isn't even YAML to write, because the desired state is already known. It works out of the box.
## I started with operations
Ask a proactive agent to pick your product roadmap and you get an opinionated intern, because product direction is a matter of taste. Production isn't. It's the one domain where all three problems are solvable today.
The work announces itself: error rates climb, a deploy goes sideways, a certificate expires, a queue backs up. The work is already sitting in the telemetry, waiting for someone to notice it. And unlike taste-driven domains, ground truth exists: the error rate either spiked or it didn't, the rollback either restored the baseline or it didn't, so the agent's judgement gets scored by the system itself, continuously, with no room for vibes.
Most of all, we already staff this job with humans. We call it *on-call*: a person sleeping next to a phone, waiting for a machine to say that another machine is unhappy. I spent years in observability, founded an observability company and Cloudflare acquired it, and wrote a whole [manifesto](https://loggingsucks.com) about structuring telemetry so the answer is one query away. Its thesis is the reason this company exists: **nobody should be on-call in 2026**.
Because the uncomfortable truth about the last decade of observability is that we made systems easier for humans to interrogate at 3am, then declared victory while humans were still the ones being woken up. The dashboards got prettier and the pager stayed on the nightstand. Observability without action is just expensive storage.
## What it looks like
This is what [Praxen](https://getpraxen.app) does. Here is a concrete Tuesday:
```mermaid alt="An overnight latency spike leads to investigation, rollback and a fix PR before the engineer wakes up"
graph TD
A["02:14 — checkout p99 jumps from 180ms to 2.1s"] --> B["02:15 — agent flags it: new error pattern, right after the 01:52 deploy"]
B --> C["02:16 — incident opens, 3 hypotheses investigated in parallel"]
C --> D["02:31 — verdict: connection pool exhausted by a new N+1 query"]
D --> R["02:33 — the 01:52 deploy is rolled back, p99 back to 180ms. The incident is over."]
R --> E["02:38 — PR opened with the fix and the evidence attached"]
E --> F["08:30 — you wake up, read the investigation, merge"]
style A fill:#fee2e2,stroke:#fca5a5,color:#991b1b
style R fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
Nobody configured a check for this failure mode. There are no rules to write and no thresholds to tune: agents evaluate every connected resource on a continuous cadence, and your existing dashboards and saved queries become their checklist. Judgement is where I am strictest. The default verdict is *no anomaly*, because it's much better to miss a borderline issue than to wake someone for noise. "A deploy could introduce a bug" is true of every deploy and never grounds to page anyone.
When something is real, [the investigation](/use-cases/ai-sre/) runs several competing hypotheses in parallel, and each agent's job is to disprove its hypothesis rather than confirm it, so correlation never gets to masquerade as cause. When the confirmed root cause is a code change, the fix arrives as a pull request with the investigation attached:
## Proactive is not unsupervised
The autonomy is in the noticing, the triage, the 3am archaeology, and the mitigation. Anything that restores a known-good state, the agent does on its own: roll back the bad deploy, flip the flag back off. Those actions are reversible by construction, and they are the ones that actually silence the pager. What stays gated is anything that creates a new state: a code change ships through your review and your CI, never around them.
The line isn't human versus agent. It's **reversible versus not**. And it's why you're not secretly still on-call: the rollback ended the incident at 02:33, six hours before you merged the pull request. The PR was never what stopped the bleeding. It's what stops it from happening again, and that can wait for coffee.
And the gate is yours to delegate. Code review agents already read every pull request in your repo. There is a world, not far from this one, where your review agent reads the fix at 02:41, checks it against the investigation, approves it, and your CI deploys to production before you wake up. Nothing about the loop changes except who holds the approve button. That is where this ends: **software that fixes itself**, with you writing the policy instead of clicking merge.
```mermaid alt="Detection and investigation lead to autonomous rollback or a reviewed PR, with memory feeding the next detection"
graph TD
A[Signals] --> B[Detection]
B --> C[Investigation]
C --> R[Rollback, on its own]
C --> D[Pull request]
D --> E[You or your agent review, it merges]
R --> F[Memory]
E --> F
F --> B
style R fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style D fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style F fill:#dbeafe,stroke:#93c5fd,color:#1e40af
```
Notice what's missing from that loop:
The prompt box was a great way to learn to trust these systems, and it's a terrible way to run production. The agent has seen every deploy, every log line, and every metric across every service at once. Keeping it behind a prompt means the best-informed member of your team only speaks when spoken to.
Nobody should be on-call in 2026.
## Guides (full content)
### How to add MCP servers to Cursor
> How to add MCP servers to Cursor: project and global mcp.json files, remote and local servers, OAuth sign-in, secrets from env vars and tool approvals.
Part 2 of [Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 3 min read
Canonical: https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/
To add an MCP server to Cursor, create .cursor/mcp.json in your project, or ~/.cursor/mcp.json for every project, and add an entry under mcpServers. A remote server needs a url; a local one needs a command and args. Cursor signs in to servers that use OAuth and asks before it runs each tool.
To add an MCP server to Cursor, **create `.cursor/mcp.json` in your project, or `~/.cursor/mcp.json` for every project, and add the server under `mcpServers`.** A remote server needs a `url`. A local server needs the `command` that starts it. Cursor signs in to servers that use OAuth and asks before it runs each tool.
MCP, the Model Context Protocol, is how Cursor's agent calls tools outside your editor: a database, an issue tracker, your production logs. The steps below follow [Cursor's MCP documentation](https://cursor.com/docs/context/mcp).
#### 1. Pick project or global config
Cursor reads two files:
| File | Use it for |
| --- | --- |
| `.cursor/mcp.json` in the repository | Tools for this project. Commit it so the team gets the same servers. |
| `~/.cursor/mcp.json` in your home directory | Tools you want in every project, such as your own notes or a personal account. |
Start with the project file for anything the whole team needs. Keep personal accounts in the global file, so a teammate's checkout never tries to sign in as you.
#### 2. Add a remote server
A remote server runs somewhere else and speaks Server-Sent Events or Streamable HTTP. It needs only a name and a `url`:
```json
{
"mcpServers": {
"praxen": {
"url": "https://mcp.getpraxen.app/mcp"
}
}
}
```
If the server uses OAuth, Cursor opens the sign-in flow the first time the agent needs it, and you approve access in the browser. Servers that take a static key instead accept `headers`:
```json
{
"mcpServers": {
"internal-api": {
"url": "https://mcp.example.com/mcp",
"headers": {
"Authorization": "Bearer ${env:INTERNAL_API_TOKEN}"
}
}
}
}
```
#### 3. Add a local server
A local server is a program Cursor starts on your machine and talks to over stdio. Give it the `command`, its `args` and any `env` it needs:
```json
{
"mcpServers": {
"context7": {
"command": "npx",
"args": ["-y", "@upstash/context7-mcp"]
}
}
}
```
Everything in the file runs with your permissions, so only add servers you trust, from sources you can read.
#### 4. Keep secrets out of the file
Cursor resolves variables in the `command`, `args`, `env`, `url` and `headers` fields. Use `${env:NAME}` for anything secret, as in the remote example above, and the file you commit holds only the variable's name. The same works in a local server's `env` block. `${workspaceFolder}` and `${userHome}` are also available, which helps when a local server needs a path inside the repository.
#### 5. Check it works, and decide what runs without asking
Open **Customize** in the sidebar to see each server, whether it connected, and the tools it offers. You can turn a server off there without removing it from `mcp.json`.
By default Cursor asks before it uses any MCP tool and shows the arguments it is about to send. Read them the first few times: it is the quickest way to learn what a server does. Once you trust a read-only tool, allowlist it so it runs without a prompt in Auto-review mode, and keep anything that writes behind the prompt.
#### What to connect first
The servers that pay off soonest give the agent context it cannot get from the code: the issue it is fixing, the database schema, and what the code is doing in production. Production is usually the gap. Cursor can read the function that failed, but not the logs, traces and deploys that show how it failed. [Praxen brings that into Cursor](/for/cursor/) through the single remote server shown in step 2, covering every cloud and observability tool you connect behind one sign-in.
Using Cursor? See [how to give Cursor production context with Praxen](https://getpraxen.app/for/cursor/).
#### Common questions
**Where is Cursor's MCP config file?**
Cursor reads .cursor/mcp.json in the project for project tools and ~/.cursor/mcp.json in your home directory for tools you want everywhere.
**Which MCP transports does Cursor support?**
Cursor supports three: stdio for local servers it starts itself, and Server-Sent Events and Streamable HTTP for remote servers.
**How do I keep API keys out of mcp.json?**
Reference an environment variable with ${env:NAME}. Cursor resolves variables in the command, args, env, url and headers fields, so the file you commit holds the name and your shell holds the secret.
**Does Cursor run MCP tools without asking?**
Not by default. Cursor asks for approval before it uses an MCP tool, and shows the arguments it is about to send. Allowlisted tools can run without a prompt in Auto-review mode.
**How do I turn a server off without deleting it?**
Open Customize in the sidebar and disable it there. The entry stays in mcp.json, so you can turn it back on later.
#### Sources
- [Model Context Protocol (Cursor docs)](https://cursor.com/docs/context/mcp)
- [Model Context Protocol specification: transports](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
Previous: [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
Next: [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
#### Related
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [Workers Issues: route Cloudflare errors to coding agents](https://getpraxen.app/learn/incident-response/cloudflare-workers-issues-error-monitoring-agents/): Cloudflare's new Workers Issues groups exceptions and 5xx errors and sends them to Claude Code, Cursor, Devin or a webhook. Setup, automations and limits.
---
### How to add MCP servers to GitHub Copilot in VS Code
> How to add MCP servers to GitHub Copilot in VS Code: workspace and user mcp.json, HTTP and stdio servers, MCP: Add Server and confirming tool calls.
Part 5 of [Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 3 min read
Canonical: https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/
To add an MCP server to GitHub Copilot, run MCP: Add Server in VS Code from the Command Palette, or edit .vscode/mcp.json in your workspace and add an entry under servers. An HTTP server needs a type and url; a stdio server needs a command and args. Use Configure Tools in the Copilot chat input to turn its tools on or off.
To add an MCP server to GitHub Copilot, **run MCP: Add Server from VS Code's Command Palette, or add the server under `servers` in `.vscode/mcp.json`.** An HTTP server needs a `type` and a `url`; a local server needs the `command` that starts it. Then use **Configure Tools** in the Copilot chat input to choose which of its tools Copilot can use.
MCP, the Model Context Protocol, is how GitHub Copilot's chat and agent mode in VS Code call tools outside your editor. Copilot reads its servers from VS Code's MCP config. The steps below follow [VS Code's MCP documentation](https://code.visualstudio.com/docs/copilot/customization/mcp-servers).
#### 1. Pick workspace or user config
| File | Use it for |
| --- | --- |
| `.vscode/mcp.json` in the workspace | Servers for this project. Commit it so the team gets the same tools. |
| `mcp.json` in your user profile | Servers you want in every workspace. Open it with **MCP: Open User Configuration**. |
VS Code also reads a portable `.mcp.json` at the workspace root, which uses `mcpServers` in place of `servers`.
#### 2. Add a server with the guided flow
Open the Command Palette and run **MCP: Add Server**. Choose **Command (stdio)** for a program VS Code starts on your machine, or **HTTP** for a remote server, then give it an ID and choose where to save it. The flow writes the same JSON you would write by hand.
#### 3. Or write the JSON yourself
A remote server needs `type` and `url`; a local one needs `type`, `command` and `args`:
```json
{
"servers": {
"praxen": {
"type": "http",
"url": "https://mcp.getpraxen.app/mcp"
},
"playwright": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@playwright/mcp"]
}
}
}
```
Avoid hardcoding API keys in the file. VS Code supports input variables and environment files for secrets, so the version you commit holds a reference and the value stays on your machine.
#### 4. Trust and confirmation
Workspace servers inherit Workspace Trust, so in a trusted workspace they start without a separate prompt. For servers that don't come from your trusted workspace, VS Code asks when a server first starts or its configuration changes. Decline and the server doesn't start; Copilot carries on without its tools.
You might be asked to confirm each tool invocation. Read the first few: it is the quickest way to learn what a server actually does.
#### 5. Choose which tools Copilot can use
Select **Configure Tools** in the Copilot chat input to see every server's tools and turn them on or off. Fewer tools means less for the model to choose between, so turn off the ones a project doesn't need.
The first server in the example is Praxen's. [It lets GitHub Copilot debug production](/for/copilot/): logs, traces and deploys from every provider you connect, through one HTTP server.
Using GitHub Copilot? See [how to give GitHub Copilot production context with Praxen](https://getpraxen.app/for/copilot/).
#### Common questions
**Where does GitHub Copilot read MCP servers from?**
From VS Code's MCP config: .vscode/mcp.json for the workspace, under a top-level servers object, and in the mcp.json of your user profile, which MCP: Open User Configuration opens.
**How do I add an MCP server without editing JSON?**
Run MCP: Add Server from the Command Palette. It walks you through choosing Command (stdio) or HTTP, a server ID and where to save it.
**How do I keep API keys out of mcp.json?**
Don't hardcode them. VS Code's docs recommend input variables or environment files, so the file you commit holds a reference and the secret stays on your machine.
**Will Copilot run MCP tools without asking?**
You might be asked to confirm each tool invocation. Configure Tools in the chat input shows each server's tools and lets you turn them on or off.
**Why didn't my MCP server start?**
For a server that doesn't come from your trusted workspace, VS Code asks the first time it starts or when its config changes. If you decline, it doesn't start and chat continues without its tools.
#### Sources
- [Add and manage MCP servers in VS Code (VS Code docs)](https://code.visualstudio.com/docs/copilot/customization/mcp-servers)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
Previous: [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
Next: [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
---
### How to use Claude Code: a practical guide
> How to use Claude Code: install it, write a CLAUDE.md, pick a permission mode, steer it mid-task, script it with -p and connect it to your tools with MCP.
Part 1 of [Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 6 min read
Canonical: https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/
To use Claude Code, install it with the native installer, run claude inside your project and sign in. Run /init to create a CLAUDE.md with your build and test commands, describe each task specifically, and press Shift+Tab to switch into plan mode for bigger changes. Add MCP servers with claude mcp add to give it your other tools.
To use Claude Code, **install it, run `claude` in your project, give it a CLAUDE.md, and describe each task as specifically as you would to a colleague.** Use plan mode for anything bigger than a small fix, interrupt it the moment it heads the wrong way, and connect your other tools with MCP so it can look things up instead of guessing.
Claude Code is Anthropic's coding agent. It reads your files, runs commands and edits code from a session in your terminal, and asks before it does anything you haven't allowed. The steps below follow [Anthropic's own documentation](https://code.claude.com/docs/en/quickstart), in the order you will need them.
#### 1. Install Claude Code and sign in
The native installer is the recommended route on macOS, Linux and WSL:
```bash
curl -fsSL https://claude.ai/install.sh | bash
```
On Windows PowerShell, run `irm https://claude.ai/install.ps1 | iex`. Homebrew (`brew install --cask claude-code`) and WinGet (`winget install Anthropic.ClaudeCode`) work too, but only the native install updates itself in the background.
Open a new terminal and check it worked:
```bash
claude --version
```
Then start it inside a project:
```bash
cd ~/code/my-app
claude
```
The first run asks you to sign in. A Claude Pro, Max, Team or Enterprise subscription works, as does a Claude Console account with API credits, or Amazon Bedrock, Google Cloud or Microsoft Foundry for teams that buy models through their cloud.
#### 2. Ask before you change anything
Claude Code reads files as it needs them, so you don't have to paste code into the prompt. Start with questions that make it read the project:
```text
what does this project do, and where is the main entry point?
```
```text
how does a request get from the router to the database here?
```
Then give it a task, and be specific. "Fix the bug" leaves it guessing. "Fix the login bug where users see a blank screen after a wrong password" tells it where to look and what done means. For larger work, list the steps in order, and say how to check the result: which test to run, which page to load.
#### 3. Write a CLAUDE.md
Every session starts with an empty context. A [CLAUDE.md](https://code.claude.com/docs/en/memory) file is how you give every session the same starting knowledge. Run this inside a session:
```text
/init
```
`/init` reads the codebase and writes a CLAUDE.md with the build commands, test instructions and conventions it finds. Edit it down to what matters: the commands to build, test and lint, the rules a new engineer would get wrong, and the directories to stay out of.
Claude Code loads these files at launch:
| File | Use it for | Who sees it |
| --- | --- | --- |
| `./CLAUDE.md` or `./.claude/CLAUDE.md` | Team instructions for the project | Everyone, committed to Git |
| `./CLAUDE.local.md` | Your own notes for this project | Only you; add it to `.gitignore` |
| `~/.claude/CLAUDE.md` | Your preferences for every project | Only you |
A CLAUDE.md can pull in other files with `@path/to/file`, so a long architecture document can live in `docs/` and still load. Keep the file short and concrete: the docs note that the more specific and concise your instructions are, the more consistently they are followed. CLAUDE.md guides what Claude does; to block an action outright, use a hook.
#### 4. Choose a permission mode
Claude Code asks before it edits files or runs commands. Press `Shift+Tab` to cycle the [permission mode](https://code.claude.com/docs/en/permission-modes) of the current session:
- **Manual** (`default`): it asks before edits and commands.
- **Accept edits** (`acceptEdits`): it edits files without asking, and still asks before other actions.
- **Plan** (`plan`): it reads and plans, and changes nothing until you approve the plan.
Plan mode is the one to reach for on anything that touches more than a few files. Read the plan, correct it, then let it work. You can start a session in it directly:
```bash
claude --permission-mode plan
```
#### 5. Steer it while it works
You will get better results from steering early than from fixing late. The controls worth learning first:
- **`Esc`** stops the current response or tool call. Claude keeps the work done so far, and you can redirect it.
- **`Esc` `Esc`** on an empty prompt opens the rewind menu, which restores the session to an earlier point.
- **`@`** autocompletes a file path, so you can point it at the exact file.
- **`!`** at the start of the prompt runs a shell command and adds its output to the session.
- **`/clear`** wipes the conversation. Use it between unrelated tasks so old context stops steering new work.
To pick up where you left off, `claude -c` continues the most recent conversation in the directory and `claude -r` opens a picker of earlier sessions.
#### 6. Run it from scripts
`claude -p` runs a single prompt and exits, which makes Claude Code usable in scripts and CI:
```bash
claude -p "explain what changed in the last commit"
cat build.log | claude -p "why did this build fail?"
claude -p "list the TODOs in src/" --output-format json
```
In print mode, `--max-turns` caps how many steps it takes and `--allowedTools` lists the tools it may use without asking, so an unattended run cannot wander.
#### 7. Connect your other tools with MCP
Claude Code can call tools outside your repository through the [Model Context Protocol](https://code.claude.com/docs/en/mcp). Add a remote server with one command:
```bash
claude mcp add --transport http
```
Servers are added for you in the current project by default. Add `--scope project` to write the server to a `.mcp.json` at the repository root, which you commit so the whole team gets it, or `--scope user` to have it in every project. Run `/mcp` inside a session to check each server's status and to sign in to servers that use OAuth.
Production is the context Claude Code usually lacks. It can read the code that broke, but not the logs, traces and deploys that show how it broke. [Praxen gives Claude Code that context](/for/claude-code/) through one MCP server that covers every provider you connect:
```bash
claude mcp add --transport http praxen https://mcp.getpraxen.app/mcp
```
Then ask it what you would otherwise open five dashboards for, such as why checkout started returning 502s this morning, and it answers from the real data.
Using Claude Code? See [how to give Claude Code production context with Praxen](https://getpraxen.app/for/claude-code/).
#### Common questions
**Do I need a paid plan to use Claude Code?**
You need a Claude Pro, Max, Team or Enterprise subscription, a Claude Console account with API credits, or access through a supported cloud provider. Claude Code asks you to sign in the first time you run it.
**How do I undo something Claude Code changed?**
Press Esc twice with an empty prompt to open the rewind menu and restore an earlier point in the session. For anything you have committed, use Git as usual.
**Where should CLAUDE.md go?**
Put team instructions in ./CLAUDE.md or ./.claude/CLAUDE.md and commit it. Personal preferences for every project go in ~/.claude/CLAUDE.md, and private notes for one project go in ./CLAUDE.local.md, which you add to .gitignore.
**Can I use Claude Code outside the terminal?**
Yes. Claude Code also runs on the web, as a desktop app, in VS Code and JetBrains IDEs, in Slack, and in CI through GitHub Actions and GitLab.
**How do I run Claude Code from a script?**
Use claude -p with a prompt. It runs once and exits, reads piped input, and takes --output-format json when another program reads the result.
**How do I give Claude Code access to production logs?**
Add an MCP server that exposes them. One HTTP server such as Praxen's covers logs, metrics, traces and deploys across every provider you connect, so Claude Code can answer questions about production without leaving the terminal.
#### Sources
- [Quickstart (Claude Code docs)](https://code.claude.com/docs/en/quickstart)
- [CLI reference (Claude Code docs)](https://code.claude.com/docs/en/cli-reference)
- [Interactive mode (Claude Code docs)](https://code.claude.com/docs/en/interactive-mode)
- [How Claude remembers your project (Claude Code docs)](https://code.claude.com/docs/en/memory)
- [Connect Claude Code to tools via MCP (Claude Code docs)](https://code.claude.com/docs/en/mcp)
- [Permission modes (Claude Code docs)](https://code.claude.com/docs/en/permission-modes)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
Next: [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
#### Related
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [Workers Issues: route Cloudflare errors to coding agents](https://getpraxen.app/learn/incident-response/cloudflare-workers-issues-error-monitoring-agents/): Cloudflare's new Workers Issues groups exceptions and 5xx errors and sends them to Claude Code, Cursor, Devin or a webhook. Setup, automations and limits.
---
### How to use Codex CLI: install, AGENTS.md, MCP and exec
> How to use Codex CLI: install it, sign in, write an AGENTS.md, set permissions, resume sessions, script it with codex exec and add MCP servers.
Part 3 of [Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 3 min read
Canonical: https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/
To use Codex CLI, install it with OpenAI's installer, run codex in your project and sign in with your ChatGPT account. Run /init to create an AGENTS.md, use /permissions to choose what Codex may do, and codex resume to reopen a chat. For scripts and CI, use codex exec, and add MCP servers with codex mcp add.
To use Codex CLI, **install it, run `codex` in your project, sign in with ChatGPT, and run `/init` to give it an AGENTS.md.** Choose what it may do with `/permissions`, reopen work with `codex resume`, and use `codex exec` when a script or CI job runs it instead of you.
Codex CLI is OpenAI's coding agent for the terminal. It reads your repository, edits files and runs commands from a session in your shell. The steps below follow [OpenAI's Codex CLI documentation](https://learn.chatgpt.com/docs/codex/cli).
#### 1. Install Codex CLI and sign in
On macOS and Linux, the standalone installer is one line:
```bash
curl -fsSL https://chatgpt.com/codex/install.sh | sh
```
Then start it from a project:
```bash
cd ~/code/my-app
codex
```
The first run asks you to sign in. Choose **Sign in with ChatGPT** to use your ChatGPT plan.
#### 2. Give it an AGENTS.md
Codex reads project instructions from an AGENTS.md file. Create one from inside a session:
```text
/init
```
`/init` writes an AGENTS.md with instructions for Codex based on your repository. Edit it down to what a new engineer would need: how to build, how to run the tests, the conventions that are easy to get wrong. Commit it, so every session and every teammate starts from the same rules.
#### 3. Choose what Codex may do
Codex asks before it acts outside what you have allowed. Use the `/permissions` slash command to choose what Codex is allowed to do in the session. Start strict on a repository you care about, and loosen it once you trust how it works there.
The other slash commands worth learning first:
| Command | What it does |
| --- | --- |
| `/model` | Choose the model and its reasoning effort |
| `/status` | Show the current session configuration |
| `/review` | Review your changes and find issues |
| `/init` | Create an AGENTS.md |
#### 4. Pick up where you left off
```bash
codex resume
```
`codex resume` reopens a recent chat from the current repository, with its history, so you don't have to explain the task again.
#### 5. Run it from scripts with codex exec
`codex exec` runs Codex without the interactive UI, which is how you use it from repeatable workflows and pipelines:
```bash
codex exec "update the changelog for the commits since the last tag"
```
Keep scripted tasks narrow, with a clear definition of done, and review what they change before it merges.
#### 6. Add MCP servers
Codex calls tools outside your repository through the Model Context Protocol. Each server is a `[mcp_servers.]` table in `~/.codex/config.toml`, or in a trusted project's `.codex/config.toml`.
A local server needs the `command` that starts it:
```toml
[mcp_servers.context7]
command = "npx"
args = ["-y", "@upstash/context7-mcp"]
```
A remote server needs a `url`. If it takes a bearer token, name the environment variable that holds it with `bearer_token_env_var`:
```toml
[mcp_servers.praxen]
url = "https://mcp.getpraxen.app/mcp"
```
You can also add servers from the command line with `codex mcp add`. For a server that uses OAuth, finish the sign-in with:
```bash
codex mcp login praxen
```
Production is the context Codex usually lacks: it can read the code that failed, but not the logs, traces and deploys that show how. [Praxen gives Codex that context](/for/codex/) through the one remote server above, covering every provider you connect.
Using Codex? See [how to give Codex production context with Praxen](https://getpraxen.app/for/codex/).
#### Common questions
**How do I sign in to Codex CLI?**
Run codex in a project directory. The first time, it asks you to sign in, and you can choose Sign in with ChatGPT.
**Where does Codex read project instructions from?**
From AGENTS.md. Run /init inside Codex to create one with instructions for your repository, then edit it and commit it.
**How do I run Codex without the interactive UI?**
Use codex exec. It is the entry point for repeatable workflows and pipelines, such as a CI job.
**Where is Codex's config file?**
~/.codex/config.toml by default. A trusted project can also have its own .codex/config.toml.
**How do I connect Codex to an MCP server that uses OAuth?**
Add the server with its URL, then run codex mcp login with the server's name to complete the sign-in.
#### Sources
- [Codex CLI (OpenAI docs)](https://learn.chatgpt.com/docs/codex/cli)
- [Model Context Protocol in Codex (OpenAI docs)](https://learn.chatgpt.com/docs/extend/mcp?surface=cli)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
Previous: [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
Next: [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
---
### How to use OpenCode: install, plan and build, MCP
> How to use OpenCode: install it, connect a provider, write an AGENTS.md with /init, switch between Plan and Build, undo changes and add MCP servers.
Part 4 of [Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 3 min read
Canonical: https://getpraxen.app/learn/coding-agents/how-to-use-opencode/
To use OpenCode, install it with the install script, run opencode in your project and use /connect to add a model provider. Run /init to create an AGENTS.md, press Tab to switch between the Plan and Build agents, and use /undo to revert a change. Add MCP servers under mcp in opencode.json.
To use OpenCode, **install it, connect a model provider with `/connect`, run `/init` to create an AGENTS.md, and switch between the Plan and Build agents with Tab.** Undo anything you don't like with `/undo`, and add MCP servers in `opencode.json` to give it tools beyond your repository.
OpenCode is an open-source coding agent for the terminal. The steps below follow [OpenCode's documentation](https://opencode.ai/docs/).
#### 1. Install OpenCode
The install script is the quickest route:
```bash
curl -fsSL https://opencode.ai/install | bash
```
It is also on npm (`npm install -g opencode-ai`) and Homebrew (`brew install anomalyco/tap/opencode`), with Chocolatey, Scoop and Docker options for Windows.
#### 2. Connect a model provider
Start OpenCode in your project:
```bash
cd ~/code/my-app
opencode
```
Then run `/connect` in the interface, pick the provider, sign in and paste the API key it gives you.
#### 3. Create an AGENTS.md
```text
/init
```
`/init` analyses your project and creates an AGENTS.md in the project root. Read it, cut anything vague, add the commands to build and test, and commit it: OpenCode's docs recommend checking it into Git so everyone on the project shares it.
#### 4. Plan first, then build
OpenCode has two primary agents, and Tab switches between them:
- **Plan** blocks changes and suggests how to implement what you asked.
- **Build** makes the changes.
For anything bigger than a small fix, start in Plan, read and correct the plan, then press Tab and let Build do it.
#### 5. Undo and redo
`/undo` reverts the most recent changes and restores the prompt that caused them, so you can rephrase and try again. Run it several times to step further back. `/redo` puts an undone change back.
#### 6. Add MCP servers
OpenCode reads MCP servers from the `mcp` object in `opencode.json`. A remote server needs its `url`:
```json
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"praxen": {
"type": "remote",
"url": "https://mcp.getpraxen.app/mcp",
"enabled": true
}
}
}
```
A local server gives the `command` as an array, with any `environment` it needs:
```json
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"context7": {
"type": "local",
"command": ["npx", "-y", "@upstash/context7-mcp"],
"enabled": true
}
}
}
```
For a remote server that needs sign-in, OpenCode starts the OAuth flow when the server answers with a 401, using dynamic client registration when the server supports it. `opencode mcp list` shows your servers, and `opencode mcp auth ` runs the sign-in by hand.
The remote server in the first example is Praxen's. [It brings production into OpenCode](/for/opencode/): logs, traces and deploys from every provider you connect, behind one sign-in.
Using OpenCode? See [how to give OpenCode production context with Praxen](https://getpraxen.app/for/opencode/).
#### Common questions
**How do I install OpenCode?**
Run curl -fsSL https://opencode.ai/install | bash, or install it with npm (npm install -g opencode-ai) or Homebrew (brew install anomalyco/tap/opencode).
**What is the difference between Plan and Build in OpenCode?**
Plan blocks changes and suggests how to implement what you asked. Build makes the changes. Press Tab to switch between them.
**How do I undo a change OpenCode made?**
Run /undo. It reverts the most recent changes and restores your original prompt, and you can run it more than once. /redo puts an undone change back.
**Should I commit AGENTS.md?**
Yes. OpenCode's docs recommend committing the AGENTS.md that /init creates, so everyone on the project starts from the same instructions.
**How does OpenCode sign in to a remote MCP server?**
For a server that needs it, OpenCode detects the 401 response and starts the OAuth flow on its own, using dynamic client registration when the server supports it. opencode mcp auth with the server's name runs it by hand.
#### Sources
- [Intro (OpenCode docs)](https://opencode.ai/docs/)
- [MCP servers (OpenCode docs)](https://opencode.ai/docs/mcp-servers/)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
Previous: [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
Next: [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
---
### How to use the Pi coding agent
> How to use the Pi coding agent: install it, sign in with /login, pick a model, branch sessions with /tree and /fork, compact context and script it with -p.
Part 6 of [Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 3 min read
Canonical: https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/
To use Pi, install it with the installer script or npm, run pi in your project and use /login to connect a subscription or API key. Choose a model with /model, branch a session with /tree, /fork or /clone to try another approach, and use /compact when the history grows. For scripts, pi -p runs once and prints the reply.
To use Pi, **install it, run `pi` in your project, connect a model with `/login`, and learn the session commands: `/tree`, `/fork` and `/clone` to try another approach, `/compact` when the history gets long.** For scripts, `pi -p` runs once and prints the answer.
Pi is a minimal, extensible coding agent harness for the terminal. It ships with strong defaults and deliberately leaves out features such as sub-agents and plan mode: you add what you need, or install it as a package. The steps below follow [Pi's documentation](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent).
#### 1. Install Pi
Pi needs Node.js 22.19 or newer. On macOS and Linux:
```bash
curl -fsSL https://pi.dev/install.sh | sh
```
Or with npm:
```bash
npm install -g --ignore-scripts @earendil-works/pi-coding-agent
```
#### 2. Start it and sign in
```bash
cd ~/code/my-app
pi
```
Run `/login` inside Pi to connect a subscription or an API key for a built-in provider. The startup header lists the instructions and resources Pi loaded, which is the quickest way to check it picked up what you expected.
#### 3. Pick a model and how hard it thinks
| Command | What it does |
| --- | --- |
| `/model` | Select a model |
| `/thinking` | Select how much reasoning the current model uses |
| `/settings` | Change common preferences |
| `/logout` | Remove a provider's access |
From the command line, `--model` selects a model by exact or fuzzy name, and `--provider` limits the lookup to one provider.
#### 4. Work in sessions
Pi saves every session. The commands that make that useful:
- **`/new`** starts a fresh session for a new task.
- **`/name`** gives the current session a name you will recognise later.
- **`/resume`** opens another saved session; `pi --continue` from the same folder resumes the most recent one.
- **`/tree`, `/fork` and `/clone`** let you explore another approach without losing the work you have.
- **`/compact`** reduces the conversation history sent to the model when a long session starts to slow down.
- **`/session`** shows the session's file, ID, message count, token usage and cost.
Branching is Pi's answer to a wrong turn: fork before a risky change, and if it goes badly, go back to the branch that worked.
#### 5. Script it
```bash
pi -p "summarise the changes on this branch"
pi --mode json "list the failing tests"
```
`-p` runs the prompt once, prints the final reply and exits. `--mode json` streams JSONL events instead, for a program that wants to follow each step.
#### Give Pi production context
Pi works from your repository. It can read the code that failed, but not the logs, traces and deploys that show how it failed. [Praxen gives Pi that context](/for/pi/): it connects to the platforms you already run on, so Pi can see when something breaks there.
Using Pi? See [how to give Pi production context with Praxen](https://getpraxen.app/for/pi/).
#### Common questions
**What is Pi?**
A minimal, extensible coding agent harness for the terminal. It ships with strong defaults and leaves out features such as sub-agents and plan mode, which you can add yourself or install as packages.
**What does Pi need to run?**
Node.js 22.19 or newer. Install it with the installer script, or with npm as @earendil-works/pi-coding-agent.
**How do I resume a Pi session?**
Run pi --continue in the same folder to resume the most recent session, or /resume inside Pi to open another saved one.
**How do I try a different approach without losing my work?**
Use /tree, /fork or /clone. Each lets you explore another direction while the existing session stays intact.
**Can I run Pi from a script?**
Yes. pi -p runs the prompt once, prints the final reply and exits, and --mode json streams JSONL events instead.
#### Sources
- [Pi coding agent (README)](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent)
- [Using Pi (Pi docs)](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/usage.md)
- [Pi CLI reference (Pi docs)](https://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/docs/cli.md)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Using coding agents: Claude Code, Cursor, Codex and more](https://getpraxen.app/series/using-coding-agents/)
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/)
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/)
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/)
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/)
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/)
Previous: [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/)
---
### How to fix a broken build on main
> How to fix a broken build on main: read the failing job, decide between rerun, revert and fix forward, find the breaking commit and stop the next one.
Part 1 of [Shipping safely: deploys, config and change risk](https://getpraxen.app/series/shipping-safely/)
By Boris Tane, Founder of Praxen · Published October 17, 2026 · 6 min read
Canonical: https://getpraxen.app/learn/continuous-integration/how-to-fix-a-broken-build-on-main/
To fix a broken build on main, read the failing job's log first, then pick one of three moves: rerun only if the failure is unrelated to the code, revert the breaking commit if the cause is not obvious within minutes, or push a small fix if it is. Confirm the next run on main passes, then add the protection that would have caught it.
To fix a broken build on main, **read the failing job's log, then rerun, revert or fix forward depending on what it shows.** Find the commit that broke the build, confirm the next run on main is green, and add the branch protection that would have stopped it.
A red main blocks everyone. Nobody can trust a merge, the next deploy waits, and every change merged on top makes the breaking one harder to isolate. Martin Fowler's description of continuous integration puts it plainly in a practice called [Fix Broken Builds Immediately](https://martinfowler.com/articles/continuousIntegration.html): getting main green again comes before new work. The steps below use GitHub Actions and the GitHub CLI, but the order works on any CI.
#### 1. Make one person own it
Say in the team channel that main is red and who is on it. Two people fixing the same failure in parallel produce two conflicting fixes, and nobody fixing it because everyone assumed someone else was is the more common failure.
Until main is green, hold merges that are not the fix. Every extra commit widens the range you have to search and can hide a second failure behind the first.
#### 2. Read the failing job's log
The red cross only tells you that a check failed. Find the run and read only the failed steps:
```bash
gh run list --branch main --status failure --limit 5
gh run view --log-failed
```
[`gh run view --log-failed`](https://cli.github.com/manual/gh_run_view) prints the log for the failed steps only, which is usually the twenty lines that matter out of thousands. Start from the first error: later errors are often consequences of it.
Sort what you find into two kinds:
- **The code is wrong.** A compile error, a failing assertion, a type error, a lint rule, a missing import. These fail again on every run.
- **The run was unlucky.** A runner that lost its network, a package registry that timed out, a job that was cancelled or ran out of disk. These are about the machine.
#### 3. Choose: rerun, revert or fix forward
**Rerun only when the failure is clearly outside your code.** [GitHub re-runs use the same commit SHA](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs), so a real bug fails again and a rerun just costs time. Re-run only the failed jobs, with debug logging if the cause is unclear:
```bash
gh run rerun --failed --debug
```
If a test passes on rerun, note it. A test that fails and passes on the same commit is flaky, and a flaky test on main teaches the team to ignore red builds.
**Revert when the cause is not obvious.** If you cannot name the cause within a few minutes, revert the breaking commit. A revert returns main to code that already passed, which is the fastest safe way back to green, and the author can fix the change on a branch without the whole team waiting. Fowler's article recommends the same: when the cause is not obvious, take main back to the last known good build.
```bash
git revert
```
**Fix forward when the cause is clear and the fix is small.** A missing import, a renamed function one caller still uses, a snapshot that needs updating: push the fix through a normal pull request so CI runs on it before it reaches main.
#### 4. Find the commit that broke it
The breaking commit sits between the last green run on main and the first red one. List the runs with their commits:
```bash
gh run list --branch main --limit 20 --json headSha,conclusion,createdAt
git log --oneline ..
```
If that range holds one commit, you have it. If it holds several, let [`git bisect`](https://git-scm.com/docs/git-bisect) test them for you with the command that fails:
```bash
git bisect start
git bisect run npm test
git bisect reset
```
`git bisect run` treats exit code 0 as good, 125 as "skip this commit", and any other code from 1 to 127 as bad. Use the narrowest command that reproduces the failure: running one test file is faster and more reliable than the whole suite.
If every commit in the range passes on its own, the build broke because two changes met. That is the case step 6 prevents.
#### 5. Confirm main is green
Watch the run on main after your fix or revert lands, and treat the job as done only when it passes:
```bash
gh run watch --exit-status
```
Then tell the team main is green and merges can resume. If you reverted, leave a note on the original pull request with the failing log so the author starts from the evidence.
#### 6. Stop the next one
Most broken builds on main come from changes that were tested against an older main. Three settings close that gap.
- **Required status checks.** In [branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches), require the CI checks that matter, so nothing merges on red.
- **Up-to-date branches.** Requiring branches to be up to date before merging makes each pull request run against the current main. It is safe but slow on a busy repository, because every merge sends the others back to rebase.
- **A merge queue.** A [merge queue](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue) tests each pull request together with the ones ahead of it before anything reaches main. If a queued pull request fails its checks, it is removed from the queue and the rest continue. Workflows must listen for the `merge_group` event or the queue waits forever:
```yaml
on:
pull_request:
merge_group:
```
With a merge queue in place, two pull requests that are green alone but red together fail in the queue instead of on main.
#### Where Praxen fits
[Praxen detects issues and fixes them by opening pull requests](/use-cases/continuous-integration/). When the build on main breaks, it works out why, and the fix arrives ready for your team's normal review.
See [Praxen for Continuous Integration](https://getpraxen.app/use-cases/continuous-integration/).
#### Common questions
**Should I rerun a failed build on main?**
Only when the log shows a failure outside your code: a runner that lost its network, a registry that timed out, a cancelled job. A rerun uses the same commit, so a real bug fails again. Rerunning until it passes hides a flaky test instead of fixing it.
**Is it better to revert or fix forward?**
Revert when the cause is not obvious or the fix is more than a few lines. A revert returns main to a state that already passed, which is the fastest safe green. Fix forward when the cause is clear and the change is small enough to review in minutes.
**How do I find which commit broke the build?**
Compare the last green run on main with the first red one: the breaking commit is in that range. If the range holds several commits, run git bisect with the failing command to test each one automatically.
**Why did main break when every pull request passed?**
Each pull request was tested against an older main. Two changes can pass alone and fail together. Requiring branches to be up to date before merging, or using a merge queue, tests each change against what main will actually become.
**How long can I rerun a GitHub Actions workflow?**
GitHub lets you re-run a workflow up to 30 days after its initial run. The re-run uses the same commit SHA as the original, so it tests exactly the same code.
**Can a tool fix the build on main for me?**
Praxen can. When the build on your main branch breaks, it works out why and opens a pull request with the fix, ready for your team's normal review.
#### Sources
- [Continuous Integration, Fix Broken Builds Immediately (Martin Fowler)](https://martinfowler.com/articles/continuousIntegration.html)
- [Re-running workflows and jobs (GitHub Docs)](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs)
- [gh run view (GitHub CLI manual)](https://cli.github.com/manual/gh_run_view)
- [gh run list (GitHub CLI manual)](https://cli.github.com/manual/gh_run_list)
- [git-bisect (Git documentation)](https://git-scm.com/docs/git-bisect)
- [About protected branches (GitHub Docs)](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)
- [Managing a merge queue (GitHub Docs)](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to correlate incidents with deploys using evidence](https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/): Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
- [What is change intelligence? A production safety guide](https://getpraxen.app/learn/deployment-safety/what-is-change-intelligence/): What is change intelligence? Learn how deploys, config, flags, infrastructure and dependency changes connect to incidents, using tools you already have.
- [Nobody reads pull request descriptions, so we made videos](https://getpraxen.app/blog/nobody-reads-pull-request-descriptions-so-we-made-videos/): Almost none of the pull requests Praxen opens were asked for by an engineer, and a longer description does not make them easier to review. We are experimenting with a narrated video for every fix. Why, what it looks like, and what happened to merge rates.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [How we prevent slop from hitting prod](https://getpraxen.app/blog/how-we-prevent-slop-from-hitting-prod/): Nothing in your pipeline asks whether a change is safe for prod. This is the check we built that does, and how we got it to finish before your tests do.
---
### How to keep AI coding agents from breaking production
> Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
Part 2 of [AI x Production: coding agents, MCP and autofix](https://getpraxen.app/series/ai-x-production/)
By Boris Tane, Founder of Praxen · Published September 29, 2026 · Updated October 6, 2026 · 14 min read
Canonical: https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/
An AI coding agent reaches production in two ways: through the code it merges, and through the actions it takes with its own credentials. Close the second path with a separate, read-only identity and human approval for every write. Gate the first with branch protection the agent can't bypass, required checks that really block, and a review of each change against the live system it deploys to.
To keep an AI coding agent from breaking production, control the two routes it has there. One is the code it merges. The other is the commands and tool calls it runs with its own credentials. Give the agent its own identity with read-only access to anything live. Make every change land as a pull request. Let branch protection, required checks and a person decide the merge. Then check each change against the production system it lands on, and make a bad deploy quick to spot and quick to undo.
You don't need a new tool to start. GitHub branch protection, a CODEOWNERS file, a scoped credential and a written rules file for the agent cover most of the risk today. This page walks through each control, the settings that matter, and the gaps agents slip through when a control is only half set up.
#### What it means for an AI coding agent to break production
An AI coding agent is a language model with tools. It reads your repository, runs shell commands, edits files and opens pull requests. Through MCP servers and other integrations it can also call APIs, query databases and change infrastructure. Breaking production means one of two things:
1. **Through the merge.** The agent writes a change that passes review and CI, deploys, and then hurts live traffic. Typical examples: a migration that locks a busy table, code that reads an environment variable nothing provisions, an endpoint removed while callers still use it, or a dependency whose package name the model made up.
2. **Through its own actions.** The agent runs something against a live system directly: a write to a production database, a cloud config change, a push straight to `main`, a deleted resource.
The bugs agents write look like the bugs people write. The difference is how many changes arrive and how fast. An agent keeps producing diffs for as long as it runs, so reviewers get more to read, and each diff comes with less of the author's reasoning behind it. So put your controls where speed can't skip them: in credentials the agent doesn't hold, and in merge rules enforced on the server.
#### Close the direct path: the agent's own credentials
Start here. A merge gate is useless if the agent can reach the database without going through it.
1. **Give the agent its own identity.** Use a bot account or GitHub App and a dedicated cloud role. Never hand it your personal token. Audit logs then show which actions came from the agent, and branch protection can treat it as a normal, non-admin contributor.
2. **Make that identity read-only on anything live.** Reading logs and metrics and describing resources is enough for most agent work. Grant no deploy rights and no write access to production data stores.
3. **Keep production secrets out of the agent's environment.** Don't load the production `.env` into the sandbox or workspace where the agent runs. If a credential appears in its context, it can end up in generated code or commit history.
4. **Require a person's approval for every irreversible action.** That covers deletes, migrations run by hand, infrastructure changes and anything that touches customer data.
5. **Scope each MCP server like an IAM policy.** Expose only the tools the task needs, require authentication on external connections, and log every tool call. The MCP config file is a permission grant, so review it the way you review IAM policies.
6. **Treat everything the agent reads from outside as untrusted.** Issue text, web pages, READMEs in third-party packages and MCP responses can all carry instructions. Prompt injection tops the OWASP Top 10 for LLM applications. Give any agent that reads untrusted content read-only, least-privilege access, and keep writes behind approval.
A local hook stops the most common slip, where the agent pushes its work straight to `main`:
```bash
#!/bin/sh
# .git/hooks/pre-push
protected="refs/heads/main"
while read local_ref local_sha remote_ref remote_sha; do
if [ "$remote_ref" = "$protected" ]; then
echo "Direct pushes to main are blocked. Open a pull request." >&2
exit 1
fi
done
```
This hook only helps. Anything that can run `git push --no-verify` walks straight past it. The real control lives on the server, and that's the next section.
#### Gate the merge with branch protection the agent cannot bypass
Branch protection rules set what has to happen before anything reaches a branch. By default a rule already blocks force pushes and branch deletion. For a branch that deploys to production, turn on the settings below. In the repository, open **Settings**, then **Branches**, then **Add classic branch protection rule**, and enter `main` as the branch name pattern.
1. Select **Require a pull request before merging**, then **Require approvals**.
2. Select **Dismiss stale pull request approvals when new commits are pushed**. An approval then covers only the diff that was approved.
3. Select **Require approval of the most recent reviewable push**. Someone other than the last pusher must approve. When the agent pushed last, a person has to sign off.
4. Select **Require review from Code Owners** and add a CODEOWNERS file for the paths where agent changes hurt most.
5. Select **Require status checks to pass before merging** and **Require branches to be up to date before merging**. Search for your checks and pick each one's expected source app.
6. Select **Require deployments to succeed before merging** and choose your staging environment.
7. Select **Do not allow bypassing the above settings**. Without it, admins and roles with the bypass permission skip every rule.
8. In organisation repositories, select **Restrict who can push to matching branches**.
9. Click **Create**.
A CODEOWNERS file sends the risky paths to the people who own them:
```text
# .github/CODEOWNERS
/migrations/ @acme/db-owners
/infra/ @acme/platform
/.github/workflows/ @acme/platform
/tests/ @acme/platform
/AGENTS.md @acme/platform
```
If a path lists several owners, an approval from any one of them is enough. Once the rule is live, a direct push that skips a failing check is refused on the server:
```shell
remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: error: Required status check "ci-build" is failing
```
Only one classic branch protection rule applies to a branch at a time. If several patterns overlap, it can be hard to tell which one is in force. Rulesets don't have that limit, so consider converting once you have more than one rule.
#### Make required checks block the merge they are meant to block
A required check protects you only if a failure really shows up as a failure. With GitHub Actions, the common trap is job dependencies. When a job fails, the jobs that depend on it are skipped, and a skipped job doesn't report a failure. A pull request that requires the dependent job may merge anyway. The fix is one gate job that runs `always()`, reads the results of the jobs it needs, and is the only check you mark as required:
```yaml
name: ci
on:
pull_request:
merge_group:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- run: npm ci
- run: npm test
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- run: npm ci
- run: npm audit --audit-level=high
gate:
if: always()
needs: [test, audit]
runs-on: ubuntu-latest
steps:
- name: Fail if any needed job failed or was cancelled
if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
run: exit 1
```
Three more details matter once an agent is pushing commits:
- **Pin the source of each required check.** Anyone with write permission can set the state of any status check in the repository, and an agent's token has write permission. Pick the expected GitHub App for each check. A status set by anyone else then fails with a message that the check was not set by the expected app.
- **Add the `merge_group` trigger if you use a merge queue.** Without it, required checks never run for queued pull requests and the merge fails.
- **Keep job names unique across workflows.** Two workflows with the same job name give ambiguous results for a required check.
A workflow skipped by a path filter, a branch filter or a commit message instruction such as `skip-checks: true` leaves its checks pending. A pull request that requires those checks stays blocked, which is what you want. It also means anything you care about must be a required check. An agent can quietly skip checks that are merely optional.
#### Check the change against production before it merges
Tests check the code against itself. Many production failures depend on what is live: how big a table is, what older callers still send, which secrets exist in which environment. Say an agent adds an index to the orders table. Every test passes, because the test database is empty. In production, a plain `CREATE INDEX` holds a write lock on the table for the whole build, while checkout keeps trying to write to it.
Add review steps, in CI or in your reviewers' checklist, for the changes that tests can't judge:
- Migrations that must be applied by hand, or in a set order relative to the deploy.
- `CREATE INDEX` without `CONCURRENTLY`, and `ADD COLUMN ... NOT NULL` with no default. Both lock the table while they run.
- An endpoint or field removed while the deployed version or another service still reads it.
- Code that reads an environment variable, secret or binding that nothing in the diff provisions.
- New dependencies. Confirm the package exists, check who maintains it, pin it in the lockfile and run software composition analysis. Models suggest package names that don't exist, and attackers can register those names.
- A raised timeout or limit presented as a fix. It usually hides the cause.
Some of these are easy to catch with a script. This one fails CI when a migration builds an index without `CONCURRENTLY`:
```bash
#!/bin/sh
# scripts/check-migrations.sh
git diff --name-only origin/main...HEAD -- 'migrations/*.sql' | while read -r f; do
if grep -Eiq 'create (unique )?index' "$f" && ! grep -Eiq 'index concurrently' "$f"; then
echo "$f: CREATE INDEX without CONCURRENTLY blocks writes on the table" >&2
exit 1
fi
done
```
A grep won't find the rest, because they need to know what runs in production and what depends on it. Those need a person who knows the system, or a review step that can query your live telemetry and topology.
#### Write the rules down where the agent reads them
Agents read an instruction file such as `AGENTS.md` at the start of a task. Put your production rules there, in plain words:
```markdown
# AGENTS.md
## Production rules
- Never push to main. Open a pull request and stop.
- Never run commands against production. Use read-only tools.
- Build indexes with CREATE INDEX CONCURRENTLY. New NOT NULL columns need a default.
- Ship schema changes in a separate pull request before the code that uses them.
- Name every new dependency in the pull request description with its registry link.
- Do not edit .github/workflows, infra/ or tests/ unless the task asks for it.
- Never disable, skip or delete a failing test. Report it in the pull request.
- Read secrets from the secrets manager. Never write a key into code or config.
## Scope
Change only what the task needs. List anything else you noticed in the pull request description.
```
A rules file guides the agent, and the model can still ignore it. Pair every rule with a check that enforces it: the pre-push hook and branch protection for pushes, CODEOWNERS for workflow and test edits, the migration script for indexes. Keep the file under version control and give it an owner in CODEOWNERS. A poisoned input that rewrites the rules should then show up in review.
A written specification does the same job for scope. Say what must change and what must stay as it is. Then compare the diff with it. A bug fix that also reorganises imports and deletes an unreliable test has gone out of scope.
#### Catch the change that gets through anyway
After a change ships, [change intelligence](/learn/deployment-safety/what-is-change-intelligence/) connects deploys, configuration and flag changes to the system's behavior. Follow the practical guide to [correlating incidents with deploys](/learn/deployment-safety/how-to-correlate-incidents-with-deploys/) to turn release markers into a hypothesis backed by request evidence.
Some bad changes will pass every gate. Plan for that:
1. **Deploy to staging before production**, and use the branch protection setting that requires the staging deployment to succeed. A preview environment per pull request catches integration problems even earlier.
2. **Record every deploy with its commit SHA.** I built and led the Workers observability team at Cloudflare, and this is still the first thing I'd set up. When a graph moves, you want the change that landed just before it on the same timeline.
3. **Watch a few signals after each deploy:** error rate, p95 latency, and the depth of any queue the change touches. Compare each one with its value before the deploy.
4. **Make rollback one step.** Know the command before you need it:
```bash
git revert --no-edit -m 1
git push origin HEAD:revert-
gh pr create --base main --fill
```
If your platform can redeploy the previous release in one action, use that first. Put the revert through review afterwards.
#### What goes wrong: the gaps agents slip through
These are the ways a setup that looks safe still lets an agent's change through, with the fix for each.
- **The agent runs with an admin token.** By default, branch protection doesn't apply to admins or roles with the bypass permission. Fix: give the agent a non-admin identity and turn on **Do not allow bypassing the above settings**.
- **The agent marks its own check green.** Any writer can set any status. Fix: pick the expected source app for every required check.
- **A failed job hides behind a skipped one.** Dependent jobs are skipped and report nothing. Fix: require a single gate job that uses `always()` and `needs`.
- **The agent pushes after a person approved.** The approval still counts unless you say otherwise. Fix: dismiss stale approvals and require approval of the most recent reviewable push.
- **The agent fixes CI by editing CI.** It disables a test, loosens a lint rule or edits a workflow file. Fix: put workflows, tests and lint config under CODEOWNERS and require code owner review.
- **Local hooks are the only guard.** `--no-verify` skips them. Fix: enforce the same rule on the server with branch protection.
- **The merge queue never runs the checks.** The workflow lacks the `merge_group` trigger. Fix: add it next to `pull_request`.
- **Reviewers approve by reflex.** Send every change to a person and people stop reading. Fix: keep human review for high-risk paths such as migrations, auth, infra and payments, and let deterministic checks clear the rest.
- **The agent follows instructions hidden in an issue.** Fix: keep agents that read untrusted text on read-only credentials, and require approval for every write.
- **A new check can't be selected as required.** GitHub lists only checks that completed successfully in the repository in the past seven days. Fix: run the workflow once on a branch, then add it to the rule.
#### A checklist to finish this week
- The agent has its own identity, with no admin rights and read-only cloud access.
- There are no production secrets anywhere in the agent's environment.
- MCP servers are scoped, authenticated and logged, and their config goes through review.
- `main` requires a pull request, approval of the latest push by someone else, code owner review and passing required checks, with bypassing turned off.
- Required checks come from a pinned source app, and a single gate job decides pass or fail.
- CODEOWNERS covers migrations, infra, workflows, tests and the agent rules file.
- `AGENTS.md` states your production rules, and each rule has a check behind it.
- Deploys go through staging, carry a commit SHA and can be rolled back in one step.
#### Catching what agents break in production with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to the platforms your team already runs on, so when an agent's change breaks something there it works out why, and the fix arrives ready for your team's normal review.
See [Praxen for impact analysis](https://getpraxen.app/use-cases/impact-analysis/).
#### Common questions
**Should I let an AI coding agent merge its own pull requests?**
Not on branches that deploy to production. Turn on Require approval of the most recent reviewable push, so someone other than the last pusher has to approve. When the agent pushed last, that someone is a person. For paths that never ship to users, such as docs, auto merge after required checks pass can be fine.
**Can an AI coding agent bypass GitHub branch protection?**
Yes, if it holds an admin token or a role with the bypass branch protections permission, because the rules skip those actors by default. It can also set a status check to green, since any writer can set any status. Fix both: give it a non-admin identity, turn on Do not allow bypassing the above settings, and pin the expected source app for every required check.
**Do protected branches work on private repositories?**
Protected branches work in public repositories on GitHub Free and GitHub Free for organisations. For private repositories you need GitHub Pro, Team, Enterprise Cloud or Enterprise Server. Restricting who can push to a branch also needs an organisation-owned repository.
**Why can't I add my new CI check as a required status check?**
GitHub only lets you require checks that completed successfully in the repository during the past seven days. Push a branch that triggers the workflow, let it pass once, then pick the check in the branch protection rule. Give the job a name that no other workflow uses.
**What should go in AGENTS.md to protect production?**
The rules you would tell a new hire: never push to main, never run commands against production, build indexes concurrently, name every new dependency, don't touch workflows or tests unless asked, and never disable a failing test. Keep it short and specific. Back each rule with a server-side check, and put the file under CODEOWNERS so edits get reviewed.
**How do I stop an agent adding a malicious or made-up package?**
Treat every new dependency as a review item. Confirm the package exists, check who maintains it and its release history, pin it in the lockfile, and run software composition analysis in CI. For more control, resolve packages from a scoped internal registry or an allow-list of approved packages.
**Do I still need staging if every pull request is reviewed?**
Yes. Review and preview environments check one change at a time. Staging is where merged changes, migrations and shared libraries first run together. Branch protection can require a successful deployment to your staging environment before a pull request merges.
**How do I protect against prompt injection in coding agents?**
Assume any text from outside your team can carry instructions: issues, web pages, package READMEs and tool responses. Give agents that read that content read-only, least-privilege credentials. Require a person's approval for irreversible actions, and keep the agent's rules file under version control so tampering shows up in review.
#### Sources
- [Balancing speed and safety: A control framework for AI coding agents (AWS Security Blog)](https://aws.amazon.com/blogs/security/balancing-speed-and-safety-a-control-framework-for-ai-coding-agents/)
- [About protected branches (GitHub Docs)](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)
- [Managing a branch protection rule (GitHub Docs)](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule)
- [Status checks (GitHub Docs)](https://docs.github.com/articles/about-status-checks)
- [Troubleshooting required status checks (GitHub Docs)](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/troubleshooting-required-status-checks)
- [Slopsquatting (Wikipedia)](https://en.wikipedia.org/wiki/Slopsquatting)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content, including How we prevent slop from hitting prod](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[AI x Production: coding agents, MCP and autofix](https://getpraxen.app/series/ai-x-production/)
1. [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/)
2. [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/)
Previous: [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/)
#### Related
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [Nobody reads pull request descriptions, so we made videos](https://getpraxen.app/blog/nobody-reads-pull-request-descriptions-so-we-made-videos/): Almost none of the pull requests Praxen opens were asked for by an engineer, and a longer description does not make them easier to review. We are experimenting with a narrated video for every fix. Why, what it looks like, and what happened to merge rates.
- [How we prevent slop from hitting prod](https://getpraxen.app/blog/how-we-prevent-slop-from-hitting-prod/): Nothing in your pipeline asks whether a change is safe for prod. This is the check we built that does, and how we got it to finish before your tests do.
- [How to fix a broken build on main](https://getpraxen.app/learn/continuous-integration/how-to-fix-a-broken-build-on-main/): How to fix a broken build on main: read the failing job, decide between rerun, revert and fix forward, find the breaking commit and stop the next one.
- [How to correlate incidents with deploys using evidence](https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/): Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
---
### How to correlate incidents with deploys using evidence
> Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 8 min read
Canonical: https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/
To learn how to correlate incidents with deploys, put the time a release began serving traffic on the same timeline as request failures, latency and logs. Match service, environment and version, then compare changed and unchanged traffic and inspect the diff for a failure mechanism. A deploy near an alert is a candidate cause, not a verdict.
To understand how to correlate incidents with deploys, **line up the release's traffic exposure with the first failing requests, then compare affected and unaffected versions.** Put deploy markers beside errors, latency and logs, using the same service and environment identity. Inspect the changed code or configuration for a mechanism before calling the deploy a cause.
This is one part of [change intelligence](/learn/deployment-safety/what-is-change-intelligence/): configuration, flags, infrastructure and dependency changes also belong in the investigation. The process below starts with deployment records you can retrieve today and builds toward a timeline another responder can verify.
#### 1. Define the incident window from telemetry
Start with the user-visible symptom, not the notification time. Find the earliest affected requests and a healthy window before them. Record the service, environment, route, region if available, and whether every request or only a cohort is failing.
Use the [four golden signals from Google's SRE book](https://sre.google/sre-book/monitoring-distributed-systems/): latency, traffic, errors and saturation. For a web service, useful views include request counts, failure rates, latency distributions and the resources limiting its work. Google's guidance also distinguishes successful-request latency from failed-request latency; a fast failure can make an aggregate latency graph look deceptively healthy.
As a starting investigation window, try 15 minutes before the first failure through the present, then widen it if the behavior depends on delayed jobs or accumulated state. That is a working choice, not a provider requirement or universal alert threshold. Check the graph's aggregation window: a five-minute bucket cannot establish which of two events seconds apart happened first.
Keep a request or trace link for representative failures. Record the query and time bounds, not just a screenshot. An alert may arrive after the failure began, so the last deploy before the page is not necessarily the last deploy before impact.
#### 2. Collect deployment events with the right meaning
##### GitHub: requests and externally reported statuses
GitHub's [`deployment` event](https://docs.github.com/en/rest/deployments/deployments) represents a request to deploy a ref. External tooling performs the deployment and reports statuses. Subscribe to [`deployment_status`](https://docs.github.com/en/webhooks/webhook-events-and-payloads#deployment_status) for progress; the webhook includes both `deployment` and `deployment_status`. GitHub does not emit that webhook for an `inactive` status.
For a historical investigation, use these documented read endpoints, substituting your owner, repository and deployment ID:
```text
GET /repos/{owner}/{repo}/deployments?environment=production
GET /repos/{owner}/{repo}/deployments/{deployment_id}/statuses
```
The [deployment response](https://docs.github.com/en/rest/deployments/deployments#list-deployments) includes `id`, `sha`, `ref`, `environment`, `created_at` and `statuses_url`. The [status response](https://docs.github.com/en/rest/deployments/statuses#list-deployment-statuses) includes `state`, `created_at`, `environment`, `environment_url` and `log_url`. Follow pagination for the incident window. Keep both records: deployment creation time and successful status time are not interchangeable.
A `success` state is evidence of what the integration reported. Confirm whether your deploy job reports it after upload, readiness checks or traffic cutover. Do not treat a commit push or a green build as a production marker. The [GitHub integration page](/integrations/github/) provides context on the repository side of an investigation; the serving platform remains the source for runtime exposure.
##### Vercel: distinguish readiness from promotion
The [Vercel webhook reference](https://vercel.com/docs/webhooks/webhooks-api) documents `deployment.created`, `deployment.ready`, `deployment.error` and `deployment.promoted`. For production correlation, the last one is especially useful: Vercel says it fires after a production deployment is promoted to start serving production traffic, whether promotion follows a successful build automatically or happens explicitly.
Save the top-level `id`, `type` and `createdAt`, plus `payload.deployment.id`, the project ID and deployment links. Treat `deployment.ready` as readiness, not evidence that the production domain has switched. Keep the original payload because field placement differs between event types; for example, the ready event documents `payload.target`, while the promoted event has no target field in its listed payload.
Also preserve `deployment.rollback`. Vercel documents it as acceptance of an instant rollback request, possibly before traffic finishes switching. Verify recovery from request telemetry rather than placing “recovered” at that webhook timestamp. For platform context, see [monitoring Vercel in production](/for/vercel/).
##### Cloudflare Workers: versions and traffic assignments
[Cloudflare distinguishes a version from a deployment](https://developers.cloudflare.com/workers/configuration/versions-and-deployments/). A version captures Worker state; a deployment assigns traffic to one version or splits it between two. Record deployment creation time, version IDs and traffic percentages. An uploaded version may never have served production.
From the Worker's project directory, use the [documented Wrangler commands](https://developers.cloudflare.com/workers/wrangler/commands/workers/):
```bash
npx wrangler deployments list
npx wrangler versions list
npx wrangler tail my-worker --status error --version-id
```
The list commands expose recent deployment/version history, and the tail filters live invocation logs by status and version. The tail is for current reproduction, not retrieval of old requests. It can sample on busy Workers, so absence of an exception in the stream is not proof of absence.
Compare errors on each serving version and retain the exact exception. Cloudflare [defines error 1101 as a Worker throwing a JavaScript exception](https://developers.cloudflare.com/workers/observability/errors/); use the [error 1101 guide](/learn/troubleshooting/cloudflare-workers-error-1101/) to move from the platform error to the code path. Keep changes to storage resources separately: Cloudflare says their state is not tracked by Worker versions.
##### Kubernetes: the rollout is an interval
For a Deployment called `checkout`, use the [Kubernetes deployment guide's inspection commands](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), substituting your namespace and name:
```bash
kubectl -n production rollout history deployment/checkout
kubectl -n production rollout status deployment/checkout
kubectl -n production describe deployment checkout
kubectl -n production get rs
```
History shows revisions; describe shows conditions and events such as `ScalingReplicaSet`; ReplicaSets help identify the old and new Pod templates. Save these observations during the incident and link them to the image and source revision recorded by your delivery system. A ReplicaSet's `pod-template-hash` identifies a Pod template, not a Git SHA.
Only Pod-template changes trigger a deployment rollout. Scaling does not create a revision. Record scaling and configuration edits as separate changes rather than assuming rollout history is exhaustive. Kubernetes reports stalled progress with `ProgressDeadlineExceeded`; its docs explicitly say it takes no action beyond reporting that condition. A successful rollout likewise needs application-level health checks before you conclude the service recovered.
#### 3. Normalize the records into one change timeline
Use a small common record across sources. This example is an application-defined schema, not a Vercel or GitHub payload:
```json
{
"source": "vercel",
"source_event_id": "example-promotion-42",
"service": "checkout",
"environment": "production",
"kind": "deployment_promoted",
"version": "example-release-b",
"occurred_at": "2026-10-06T14:04:00Z",
"observed_at": "2026-10-06T14:04:03Z"
}
```
Keep source links, commit identity when available and the raw record alongside it. Normalize timestamps to UTC and preserve whether they represent requested, ready, serving or completed states. Deduplicate with source event IDs and retain arrival time so a late webhook does not appear to be a late deploy.
Put markers on the dashboard you already use. [Grafana annotations](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/annotate-visualizations/) can display descriptions, tags and links on supported panels, and tag filters can select events for the current service. Represent a rolling update as a region annotation when you know its start and end. Filter production events apart from previews.
#### 4. Join deploy markers to metrics and logs
Make your own application logs carry service, environment, release/version, route, outcome and request/trace identity where available. These are recommended fields, not platform-provided guarantees. The release value must describe the code handling that request, rather than the latest release your CI happened to upload.
Compare failure **rates** with request denominators by version or cohort. A candidate receiving less traffic can have fewer total errors while failing a larger share of requests. Check comparable routes, regions and flag exposure. Use logs and traces to establish whether failures follow the changed path.
The [SRE canary chapter](https://sre.google/workbook/canarying-releases/) explains why per-population breakdowns matter and why before/after evaluation can be misleading. Shared backends can contaminate both candidate and control. If both versions slow down, investigate a common dependency and concurrent changes before concluding the release is irrelevant.
#### 5. Work the hypothesis: a checkout example
The following timeline and measurements are fictional, to illustrate the method. They are not platform benchmarks or alert defaults.
| UTC | Observation | Interpretation |
| --- | --- | --- |
| 14:00 | Release B finishes building | Artifact exists; no evidence of production exposure |
| 14:04 | Production promotion is recorded | B becomes a candidate for failures after this point |
| 14:05 | Checkout failures begin | Symptom onset, before the page |
| 14:06 | A flag enables a new response parser | Too late to explain the first failures; might affect later ones |
| 14:09 | Incident notification arrives | Notification time is not onset time |
| 14:12 | Rollback request is accepted | Mitigation requested; verify actual traffic and recovery |
| 14:14 | Requests run release A; failures return to baseline | Recovery evidence supports the release hypothesis |
Suppose a comparable five-minute window contains 60 failures among 2,000 checkout requests on B, versus 2 among 2,000 on A. Those fictional counts yield 3% and 0.1% respectively. The comparison strengthens the hypothesis, but it still needs a mechanism.
The failing requests contain a parser exception in a code path changed by B. The diff assumes an upstream field is always present; the saved failing response lacks it. The 14:06 flag cannot explain failures beginning at 14:05, so investigate the release independently and check whether the flag worsened the later period.
After rollback, confirm which release serves the requests and whether the failure rate returns to its normal range. Save the diff, failing example, comparison query and recovery window. If traffic volume also collapsed or another team changed the upstream, recovery has competing explanations that belong in the incident notes.
#### 6. Keep the result reproducible
Finish with a short conclusion: candidate change, exposed population, mechanism, supporting evidence, contradictions and remaining unknowns. Use “correlated” until the mechanism is supported. Do not infer health from missing logs or label every nearby deploy as contributing.
Automate the records that were hard to retrieve and test collection with the next ordinary release. Retain config and flag events alongside deploys. For an exception investigation on another hosting platform, see [Vercel FUNCTION_INVOCATION_FAILED](/learn/troubleshooting/vercel-function-invocation-failed/). For prevention before shipping, see [production controls for AI coding agents](/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/). The goal is a timeline that narrows the investigation while leaving room for evidence to disprove the first guess.
Running on Vercel? See [how Praxen monitors Vercel in production](https://getpraxen.app/for/vercel/).
#### Common questions
**Which deploy timestamp should I use?**
Use the time the release began serving affected traffic, and preserve build, upload and completion times separately. On a gradual rollout, record the exposure stages. If the source only reports readiness or a requested deployment, label that limitation rather than inventing an exact traffic-switch time.
**Which Vercel webhook marks production promotion?**
Vercel documents deployment.promoted as firing after a production deployment is promoted to serve production traffic. deployment.ready only says a deployment is ready. deployment.rollback confirms that an instant rollback request was accepted and can arrive before traffic finishes switching.
**Does GitHub know exactly when my servers switched versions?**
No. GitHub stores deployment requests and statuses reported by external tooling. Use deployment_status with the deployment's SHA and environment, but confirm what the reporting integration means by success and how it relates to real traffic exposure.
**Does Kubernetes rollout success prove the application is healthy?**
No. It confirms the rollout completed according to Deployment availability conditions. Check application errors, latency and request logs too. A stalled rollout reports ProgressDeadlineExceeded; Kubernetes does not automatically roll it back on that condition.
**Can I retrieve historical failures with wrangler tail?**
No. Wrangler tail streams live console and exception logs, with optional status and version filters. Use retained logs for historical requests. On busy Workers, a live tail can enter sampling mode, so a quiet stream alone does not establish that there were no failures.
#### Sources
- [Monitoring Distributed Systems (Google SRE Book)](https://sre.google/sre-book/monitoring-distributed-systems/)
- [Canarying Releases (Google SRE Workbook)](https://sre.google/workbook/canarying-releases/)
- [Annotate visualizations (Grafana docs)](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/annotate-visualizations/)
- [REST API endpoints for deployments (GitHub Docs)](https://docs.github.com/en/rest/deployments/deployments)
- [REST API endpoints for deployment statuses (GitHub Docs)](https://docs.github.com/en/rest/deployments/statuses)
- [Webhook events and payloads (GitHub Docs)](https://docs.github.com/en/webhooks/webhook-events-and-payloads#deployment_status)
- [Webhooks API Reference (Vercel docs)](https://vercel.com/docs/webhooks/webhooks-api)
- [Versions and deployments (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/configuration/versions-and-deployments/)
- [Workers commands (Wrangler docs)](https://developers.cloudflare.com/workers/wrangler/commands/workers/)
- [Errors and exceptions (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/errors/)
- [Deployments (Kubernetes docs)](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [What is change intelligence? A production safety guide](https://getpraxen.app/learn/deployment-safety/what-is-change-intelligence/): What is change intelligence? Learn how deploys, config, flags, infrastructure and dependency changes connect to incidents, using tools you already have.
- [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/): Error 1101 means your Cloudflare Worker threw an uncaught JavaScript exception. Find the exception in logs, match it to its cause, fix it and roll back fast.
- [Fix Vercel “FUNCTION_INVOCATION_FAILED” 500 errors](https://getpraxen.app/learn/troubleshooting/vercel-function-invocation-failed/): Why a Vercel Function returns 500 FUNCTION_INVOCATION_FAILED, how to find the crash in runtime logs, and how to fix throws, missing env vars, files and memory.
- [Nobody reads pull request descriptions, so we made videos](https://getpraxen.app/blog/nobody-reads-pull-request-descriptions-so-we-made-videos/): Almost none of the pull requests Praxen opens were asked for by an engineer, and a longer description does not make them easier to review. We are experimenting with a narrated video for every fix. Why, what it looks like, and what happened to merge rates.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
---
### What is change intelligence? A production safety guide
> What is change intelligence? Learn how deploys, config, flags, infrastructure and dependency changes connect to incidents, using tools you already have.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 7 min read
Canonical: https://getpraxen.app/learn/deployment-safety/what-is-change-intelligence/
What is change intelligence? It is the practice of connecting production changes to the behavior of the systems they affect, so responders can explain what changed, where it took effect, and whether it contributed to an incident. It covers deploys, configuration, feature flags, infrastructure and dependencies; a nearby timestamp is a lead to investigate, not proof of causation.
What is change intelligence? **Change intelligence is the practice of connecting production changes to system behavior, so you can identify what changed, where it took effect, and whether it contributed to an incident.** It covers code deploys, configuration edits, feature flags, infrastructure changes and dependency updates. The useful output is an evidence-backed explanation, not a list of commits near an alert.
Consider a checkout failure. The application release might be unchanged, but a flag could have enabled a different payment path, a timeout could have been reduced, or a dependency could have changed its response. A deploy-only history cannot answer all three questions. A change timeline should show those possibilities next to the failing requests and the resources they touched.
#### Why look for changes when an incident starts?
There is published evidence for starting here. In [Google's analysis of thousands of postmortems from 2010–2017](https://sre.google/workbook/postmortem-analysis/), binary pushes accounted for **37%** and configuration pushes for **31%** of the listed outage triggers. Those are findings from Google's historical sample, not a measured rate for your application or the whole industry. The appendix also separates triggers from contributing root causes: the change that exposed a defect is not necessarily the only thing that needs fixing.
The [Google SRE Workbook's release chapter](https://sre.google/workbook/canarying-releases/) explains the mechanism. Tests and test environments do not cover every production input or state. New code, dependency behavior and configuration can expose defects only when real traffic arrives. Feature flags also separate feature launches from binary releases: deploying code and enabling its behavior are distinct changes.
This makes recent changes a useful first hypothesis. It does not justify ignoring traffic shifts, resource exhaustion or failures elsewhere. A timeline should help reject irrelevant changes as readily as identify a plausible one.
#### What a useful change record tracks
The following is a recommended record design, not a universal provider schema. Give each event enough context that an on-call engineer can test a hypothesis without asking its author to reconstruct the release.
| Change type | Record | Question it helps answer |
| --- | --- | --- |
| Code deploy | Service, environment, artifact/version, source SHA, rollout stages | Which requests ran the changed code? |
| Configuration | Resource, setting name, old/new non-secret values, applied revision | Did the runtime actually receive the edit? |
| Feature flag | Flag, environment, targeting rule, rollout percentage, effective time | Which users entered the changed path? |
| Infrastructure | Resource identity, applied diff, capacity or routing change | Did the failing service depend on that resource? |
| Dependency | Locked package version, image digest or external API change | Did behavior change below the application? |
Record the source event ID, actor when available, source URL, event time and collection time too. Keep unknown fields unknown. Record secret rotation identifiers rather than secret values. The person or automation that made a change is useful for finding context, not assigning blame.
Distinguish an edit from its application. Google's [configuration guidance](https://sre.google/workbook/configuration-design/) recommends logging both configuration changes and their application because a committed configuration may not yet be applied. An approved pull request, uploaded artifact and serving release are three different observations.
Platform boundaries matter. [Cloudflare Workers versions](https://developers.cloudflare.com/workers/configuration/versions-and-deployments/) capture code, assets, bindings and compatibility settings, while deployments determine the versions serving traffic. Associated storage state is not tracked by those versions. In [Kubernetes](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment), changing a Deployment's Pod template triggers a rollout; scaling it does not. Both scaling and rollout can belong on an operational timeline even though only one creates a new deployment revision.
#### Change intelligence versus neighboring practices
These are working distinctions for organizing your response process. Teams and vendors use the terminology differently; judge a setup by the questions it can answer.
**Change management** asks whether a change should proceed and under what controls. Its records include review, authorization and scheduling. Change intelligence asks what happened after those decisions. An approved change can still cause a failure, and an emergency dashboard edit may never have passed through the normal review workflow.
**Observability** supplies the evidence about system behavior. Google's [monitoring chapter](https://sre.google/sre-book/monitoring-distributed-systems/) distinguishes symptoms from causes and describes retrospective analysis as asking what else happened when latency rose. Change intelligence brings the change records into that analysis. Without telemetry, a change history cannot tell you whether users were affected. Without change context, a graph alone cannot identify the responsible edit.
**Deployment tracking** records the progress of releases. It is a necessary input, but a successful deploy status says little about later flag edits, routing changes or an external API. Change intelligence connects deployment tracking to those other changes and to incident evidence.
**Delivery performance measurement** summarizes outcomes over time. [DORA defines change fail rate](https://dora.dev/guides/dora-metrics/) as the ratio of deployments requiring immediate intervention, and failed deployment recovery time as the time to recover from such a deployment. Neither is the percentage of all incidents caused by changes. Individual incident attribution can inform those measurements, but proximity alone should not label a deploy as failed.
#### Correlation is the beginning of the investigation
A good change hypothesis passes several checks:
1. **Order:** the change took effect before the relevant failures began. Use exposure time, not just commit time.
2. **Scope:** the failing service, region or cohort actually received the change.
3. **Mechanism:** the diff or setting explains the observed exception, delay or rejected request.
4. **Comparison:** an unchanged version or unexposed cohort behaves differently under comparable conditions.
5. **Recovery:** stopping or reversing the suspected change removes the symptom, when that action is appropriate and reversible.
These checks are a practical investigation framework. The [SRE canary chapter](https://sre.google/workbook/canarying-releases/) provides the underlying comparison method: compare a changed population with a control, and account for shared dependencies. It also warns that before/after comparisons are vulnerable to time-dependent factors. If both populations fail because they share a backend, the new release is not exonerated or convicted by their similarity alone.
For example, suppose only requests with a new flag enabled fail. The binary was deployed earlier without errors. A flag event, per-request flag exposure and a log showing failure in the newly enabled branch make a stronger case than the binary's deploy timestamp. If turning the flag off restores the route, retain that observation alongside the code defect it exposed.
#### Start with tools you already operate
Pick one user-facing service rather than ingesting every audit stream at once. Agree on one service name and environment name across its deployment records, logs and dashboard. Save the artifact identity and the time it began serving traffic for the next release.
Then put that marker on the service's health charts. [Grafana annotations](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/annotate-visualizations/) support event descriptions, tags and links, and can be created manually, via the HTTP API or from annotation queries. A manual marker with a release link is a reasonable first step. Make the next step automation from the deploy system so the record does not depend on someone remembering during an incident.
Add configuration and flag changes next. Test one harmless change through the entire path: source record, stored event, dashboard marker and log filter. Write down any blind spot, such as an external provider with no event feed. “No changes found” means little unless you know which sources were checked.
The companion guide on [how to correlate incidents with deploys](/learn/deployment-safety/how-to-correlate-incidents-with-deploys/) walks through platform events and a worked investigation. For a Workers application, compare the [Cloudflare monitoring overview](/for/cloudflare/) with the [Cloudflare integration's available context](/integrations/cloudflare/). When evaluating a [change intelligence tool](/use-cases/impact-analysis/), ask whether its conclusions link back to the original change records and request evidence.
#### What a mature setup adds
A mature setup should preserve a reconstructable history, not just decorate a dashboard. These are useful acceptance criteria:
- **Coverage:** code, flags, configuration, infrastructure and dependencies have named sources and documented gaps.
- **Identity:** each change maps to an environment and resource; dependency relationships narrow which changes could affect an incident.
- **Exposure:** staged rollouts record their boundaries, so a responder can distinguish uploaded, available and serving versions.
- **Reliable collection:** source IDs support deduplication, delayed arrivals retain original timestamps, and collection failures are visible.
- **Evidence:** a hypothesis links the change diff to representative failing requests and comparable healthy traffic.
- **Retention:** records and incident evidence remain available long enough for your postmortem process.
- **Learning:** confirmed contributing changes and rejected hypotheses feed back into tests, review and rollout practices.
Use real incidents to judge completeness. Can a teammate reproduce your conclusion from the saved records? Could they discover that a flag changed without a deploy? Can they see when collection was unavailable? These questions are more useful than counting integrations.
For the controls before a change reaches production, see [keeping AI coding agents from breaking production](/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/). Change intelligence completes that loop by making the production outcome inspectable.
Running on Cloudflare? See [how Praxen monitors Cloudflare in production](https://getpraxen.app/for/cloudflare/).
#### Common questions
**What is change intelligence?**
Change intelligence connects production changes to system behavior and incident evidence. It records what changed, where and when it became effective, then checks whether the affected requests or resources explain the failure.
**How does change intelligence differ from change management?**
Change management governs how changes are proposed, reviewed and authorized. Change intelligence investigates what actually reached production and how it relates to system health. The two practices can share records, but approval is not evidence that a change was harmless.
**Does a deploy immediately before an incident prove the cause?**
No. Check exposure, compare affected and unaffected versions or cohorts, read the relevant diff, and look for a failure mechanism in logs or traces. A rollback followed by recovery adds evidence, but concurrent changes can still confuse attribution.
**Can I start without buying another tool?**
Yes. Combine your deployment history, configuration audit records, structured application logs and dashboard annotations. Start with one service and one shared service/environment identity, then add more change sources as incidents reveal gaps.
**Does deployment history include every production change?**
No. Feature flags, infrastructure settings and dependency behavior may change independently of application releases. Cloudflare Worker versions do not track associated storage state, and Kubernetes scaling changes do not create Deployment revisions. Track those separately.
#### Sources
- [Results of Postmortem Analysis (Google SRE Workbook)](https://sre.google/workbook/postmortem-analysis/)
- [Canarying Releases (Google SRE Workbook)](https://sre.google/workbook/canarying-releases/)
- [Configuration Design and Best Practices (Google SRE Workbook)](https://sre.google/workbook/configuration-design/)
- [DORA's software delivery performance metrics](https://dora.dev/guides/dora-metrics/)
- [Monitoring Distributed Systems (Google SRE Book)](https://sre.google/sre-book/monitoring-distributed-systems/)
- [Versions and deployments (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/configuration/versions-and-deployments/)
- [Deployments (Kubernetes docs)](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
- [Annotate visualizations (Grafana docs)](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/annotate-visualizations/)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to correlate incidents with deploys using evidence](https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/): Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [Nobody reads pull request descriptions, so we made videos](https://getpraxen.app/blog/nobody-reads-pull-request-descriptions-so-we-made-videos/): Almost none of the pull requests Praxen opens were asked for by an engineer, and a longer description does not make them easier to review. We are experimenting with a narrated video for every fix. Why, what it looks like, and what happened to merge rates.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
- [How to build a memory system for proactive agents](https://getpraxen.app/blog/how-to-build-a-memory-system-for-proactive-agents/): Proactive agents run without a prompt, so each run has to start from what earlier runs learned. How we built memory for Praxen's agents on git: one repository per team, a nightly clean-up, and lessons from merged pull requests.
---
### Supabase vs Convex in production: reliability and cost
> Supabase vs Convex in production: compare data models, realtime, consistency, scaling, migrations, monitoring, backups and costs before choosing your backend.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 8 min read
Canonical: https://getpraxen.app/learn/reliability/supabase-vs-convex-in-production/
Supabase vs Convex in production comes down to which operating model fits your app: SQL and a dedicated Postgres instance, or reactive queries and transactional TypeScript functions. Choose Supabase when Postgres access and SQL tooling matter most; choose Convex when coordinated realtime updates are central to the product. Test your busiest reads, concurrent writes and restore procedure before committing to either.
The practical Supabase vs Convex in production decision starts with the data access boundary. [Supabase gives each project a full Postgres database](https://supabase.com/docs/guides/database/overview); [Convex reads documents through query and mutation functions](https://docs.convex.dev/database/reading-data). Your choice determines how you express transactions, keep the UI current, investigate slow reads and recover after a bad write.
This comparison covers the hosted products and documentation checked on October 6, 2026. The recommendations below are operating advice, not benchmark results. Before choosing, take one real user workflow through a load test, a backwards-compatible schema change and a restore rehearsal.
#### Production comparison at a glance
| Concern | Supabase | Convex |
| --- | --- | --- |
| Data model | Full Postgres, SQL and extensions ([docs](https://supabase.com/docs/guides/database/overview)) | Documents, explicit indexed reads and JavaScript joins ([docs](https://docs.convex.dev/database/reading-data)) |
| Realtime | Subscribe to published table changes; authorization and throughput need planning ([docs](https://supabase.com/docs/guides/realtime/postgres-changes)) | Query dependencies drive subscriptions and automatic caching ([docs](https://docs.convex.dev/realtime)) |
| Transactions | Postgres defaults to Read Committed; stronger isolation is available ([docs](https://www.postgresql.org/docs/current/transaction-iso.html)) | Serializable transactions with optimistic concurrency control ([docs](https://docs.convex.dev/database/advanced/occ)) |
| Capacity | Project compute, disk performance and connection limits ([docs](https://supabase.com/docs/guides/platform/compute-and-disk)) | Deployment-class concurrency and per-function transaction limits ([docs](https://docs.convex.dev/production/state/limits)) |
| Schema changes | Versioned SQL migration files ([docs](https://supabase.com/docs/guides/deployment/database-migrations)) | Deploy schema and functions while retaining compatibility with existing data and clients ([docs](https://docs.convex.dev/production)) |
| Recovery | Paid-plan daily database backups; optional PITR; storage objects excluded ([docs](https://supabase.com/docs/guides/platform/backups)) | Consistent snapshots, optional files; code and pending schedules excluded ([docs](https://docs.convex.dev/database/backup-restore)) |
Neither operating model removes production work. The useful question is which kind of work your team can make routine.
#### Data models and consistency: compare the transaction you need
Supabase's database is Postgres rather than an abstraction over it. Auth, Storage, Realtime and Edge Functions sit on that foundation, and you can use SQL or standard database tooling. That is a strong fit when the team already thinks in relational schemas, needs extensions, or has another service that must access the same database. [Supabase database overview](https://supabase.com/docs/guides/database/overview).
Convex exposes document reads through its function API. You define indexes in the schema and explicitly select them with `withIndex()`. More complex joins and aggregations are expressed in JavaScript, within transaction limits. This is useful when backend logic and data access belong together in TypeScript, but it changes how you design reporting and large scans. [Convex reading data](https://docs.convex.dev/database/reading-data).
Do not reduce the consistency comparison to “SQL is strong, documents are eventual.” Convex documents true serializability and atomic mutations. Its deterministic transactions can be retried after an optimistic concurrency conflict. Network calls belong in actions, which are outside those deterministic query/mutation restrictions. [Convex OCC](https://docs.convex.dev/database/advanced/occ) and [runtimes](https://docs.convex.dev/functions/runtimes).
Postgres supports serializable isolation too, but its default Read Committed level gives each statement a new snapshot. Applications using serializable transactions must handle serialization failures by retrying the entire transaction. [PostgreSQL isolation reference](https://www.postgresql.org/docs/current/transaction-iso.html).
For a checkout test, write down the invariant: inventory cannot go negative, and the order and inventory update must commit together. Then test concurrent purchases and a failure after the database commit but before the payment response. My recommendation is to evaluate the database invariant separately from external side effects; neither product's transaction guarantee proves that your payment workflow is correct.
#### Realtime: change events versus reactive query results
Convex tracks the dependencies of subscribed query functions and triggers updates when their data changes. It caches query results automatically, updates the cache when dependencies change, and describes subscriptions as updating to the same database snapshot. Cached reads do not incur database bandwidth charges. [Convex realtime](https://docs.convex.dev/realtime).
Supabase Postgres Changes requires the tables to be in the `supabase_realtime` publication, with the appropriate database privileges and Row Level Security policies. Its documentation warns that every change must be checked against each subscribed user's access, and changes are processed on a single thread to preserve ordering. Upgrading compute does not automatically solve that particular throughput bottleneck. [Supabase Postgres Changes](https://supabase.com/docs/guides/realtime/postgres-changes).
Choose a test that exposes fan-out: many clients watching a frequently updated object. Measure write latency, time until the screen changes, reconnect behavior and the amount of repeated work. For Supabase, also test the application's handling of change events. For Convex, test how much data each subscribed query reads when invalidated. “Supports realtime” is too broad a requirement to choose a backend from.
#### Scaling limits and cold starts
Each Supabase project has a dedicated Postgres instance, but smaller compute sizes share CPU. Capacity includes database connections, pooler clients, disk throughput and IOPS. Micro's documented defaults are 60 database connections and 200 pooler clients. Compute changes incur downtime, usually less than two minutes but potentially longer. [Supabase compute and disk](https://supabase.com/docs/guides/platform/compute-and-disk).
Convex's current limits distinguish serverless and dedicated deployment classes. Free and Starter use S16; Professional uses S256. Those classes allow 16 and 256 concurrent queries respectively. Across classes, a document is limited to 1 MiB, and each query or mutation can read 16 MiB and scan 32,000 documents. Filtered-out documents still count as scanned. [Convex limits](https://docs.convex.dev/production/state/limits).
That makes the migration from a small dataset to a large one worth testing early. An unbounded aggregation is a different problem from insufficient concurrency. Inspect the query shape before paying for a higher tier. Our [Postgres connection exhaustion guide](/learn/troubleshooting/postgres-remaining-connection-slots-are-reserved/) covers the corresponding SQL-side failure.
Cold starts need a narrower comparison. Convex documents no cold starts for its default runtime; Node.js is a separate, action-only runtime. Supabase documents possible Edge Function cold starts, while its database runs on a dedicated instance. Free Supabase projects can also pause for inactivity, whereas paid projects do not. [Convex runtimes](https://docs.convex.dev/functions/runtimes), [Supabase functions](https://supabase.com/docs/guides/functions) and [pricing](https://supabase.com/pricing).
Measure the path you actually ship: browser, application server or function, then database. A database choice cannot explain away latency in another layer.
#### Migrations and old clients
Supabase's recommended workflow captures SQL changes in migration files, tests them locally and deploys with `supabase db push`. Direct remote Dashboard edits bypass migration history and can cause sync errors. [Supabase migrations](https://supabase.com/docs/guides/deployment/database-migrations).
Convex checks a deployed schema against existing data unless schema enforcement is disabled. Its production guide recommends adding an optional field, populating it, then making it required. Function arguments must remain compatible with old clients, and scheduled functions execute the currently deployed version with previously scheduled arguments. [Convex production workflow](https://docs.convex.dev/production).
For either backend, rehearse adding a field while an old browser tab remains open. Keep the backfill bounded and define the point at which old clients may stop working. I would reject a migration plan that only tests a fresh client on an empty development database.
#### Observability and the failures to investigate first
Supabase publishes detection checks for API/Auth errors, connection pressure, blockers, query regressions and capacity growth. Its connection check uses 80% of `max_connections` as a starting pressure threshold, while noting that reserved slots, role limits and pooler limits can constrain clients sooner. Query comparisons require saved snapshots; missing history is “unable to assess,” not healthy. [Supabase detection checks](https://supabase.com/docs/guides/observability/detecting).
Convex log streams expose function duration, status, usage, write-conflict information, successful mutation retry counts and scheduler lag. Streams require a Pro plan and provide best-effort delivery: events can be dropped or duplicated. [Convex log streams](https://docs.convex.dev/production/integrations/log-streams/).
My starter incident checks would be:
- **Supabase request failures:** separate API/Auth errors from database connection failures; inspect blockers and recent migration changes before raising pool limits.
- **Convex latency with successful mutations:** inspect retries and conflicting document IDs, not just the final success status.
- **Either backend:** verify that evidence is arriving and preserve the incident window before changing anything.
Use the detailed [Supabase monitoring playbook](/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/) and [Convex monitoring playbook](/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/) to implement those checks. The [Supabase platform page](/for/supabase/) and [integration setup](/integrations/supabase/), plus the [Convex platform page](/for/convex/) and [integration setup](/integrations/convex/), show the corresponding Praxen connections.
#### Backups: recover data and then recover the app
Supabase takes daily backups for Pro, Team and Enterprise, with documented retention of 7, 14 and up to 30 days respectively. PITR is an add-on requiring at least Small compute; its documented worst-case RPO is two minutes. Restoring makes the project inaccessible during the operation. Database backups contain Storage metadata, not the stored objects. [Supabase backups](https://supabase.com/docs/guides/platform/backups).
Convex backups are consistent table snapshots that can include file storage. Manual backups are kept for seven days; scheduled daily and weekly backups require Pro and are documented as retained for seven and fourteen days. Restores replace existing table data. Backups exclude code, environment variables and pending scheduled functions. Dedicated classes also have physical backups with support-assisted restores. [Convex backup and restore](https://docs.convex.dev/database/backup-restore).
The recovery test should include the compatible code version, credentials, files and jobs, plus a decision about writes accepted after the backup. Time the whole procedure until a user can complete a workflow. Backup frequency alone does not tell you how long recovery will take.
#### Pricing at scale and choose-if guidance
Supabase bills an organization subscription plus compute per project and usage beyond quotas. Pro starts at $25/month and includes $10 in compute credits; a second project adds compute rather than sharing the first project's instance. Realtime, egress, disk, log drains and PITR have their own billing dimensions. [Supabase pricing](https://supabase.com/pricing).
Convex's usage dimensions include function calls, database I/O, database and index storage, action compute, file storage and egress. Subscription updates count as calls. Its limits page says each index is priced as another copy of the table and distinguishes dedicated compute pricing from serverless query/mutation compute. [Convex limits and pricing dimensions](https://docs.convex.dev/production/state/limits).
Budget from a representative workload: active sessions, subscription updates, documents read per update, retained data, background jobs and recovery copies. Include monitoring and staging. Recheck the official pricing at purchase time rather than extrapolating a free-tier comparison.
**Choose Supabase if** SQL, Postgres tooling, extensions and shared access from existing services are central, and your team can own connection budgets, query plans and migrations.
**Choose Convex if** reactive query results and TypeScript transactional logic match the product, and your team can keep reads bounded, handle external effects deliberately and monitor retries and function usage.
If both fit, decide from the load test and recovery drill. The better production backend is the one whose failure you can diagnose and recover from with your actual team.
Running on Supabase? See [how Praxen monitors Supabase in production](https://getpraxen.app/for/supabase/).
#### Common questions
**Can Supabase and Convex both run production apps?**
Both document production workflows. Supabase exposes a full Postgres database with platform services; Convex combines a document database with query, mutation and action functions. Evaluate the limits, monitoring and restore process for the workload you intend to ship.
**Is Convex eventually consistent?**
Convex documents serializable transactions using optimistic concurrency control and client subscriptions updated to a consistent database snapshot. That guarantee does not make a third-party API call inside an action part of the same database transaction.
**Does Supabase have cold starts?**
Each project has its own dedicated Postgres instance. Supabase separately documents possible cold starts for Edge Functions, and Free projects can be paused for inactivity. Treat database availability and function startup as separate measurements.
**Which is cheaper at scale?**
There is no workload-independent winner. Supabase combines organization subscription, project compute and usage charges. Convex charges for calls, database I/O, storage and action compute according to plan and deployment class; subscription updates count as calls.
**Do backups recover the whole application?**
No. Supabase database backups exclude Storage API objects. Convex backups can include files but exclude code, environment variables and pending scheduled functions. Keep the rest of the recovery procedure alongside the data restore.
#### Sources
- [Supabase database overview](https://supabase.com/docs/guides/database/overview)
- [Convex reading data](https://docs.convex.dev/database/reading-data)
- [Convex realtime](https://docs.convex.dev/realtime)
- [Convex OCC and atomicity](https://docs.convex.dev/database/advanced/occ)
- [PostgreSQL transaction isolation](https://www.postgresql.org/docs/current/transaction-iso.html)
- [Supabase Postgres Changes](https://supabase.com/docs/guides/realtime/postgres-changes)
- [Supabase compute and disk](https://supabase.com/docs/guides/platform/compute-and-disk)
- [Convex limits and pricing dimensions](https://docs.convex.dev/production/state/limits)
- [Supabase Edge Functions](https://supabase.com/docs/guides/functions)
- [Convex runtimes](https://docs.convex.dev/functions/runtimes)
- [Supabase migrations](https://supabase.com/docs/guides/deployment/database-migrations)
- [Convex production workflow](https://docs.convex.dev/production)
- [Supabase detection checks](https://supabase.com/docs/guides/observability/detecting)
- [Convex log streams](https://docs.convex.dev/production/integrations/log-streams/)
- [Supabase backups](https://supabase.com/docs/guides/platform/backups)
- [Convex backup and restore](https://docs.convex.dev/database/backup-restore)
- [Supabase pricing](https://supabase.com/pricing)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/): Monitor a Supabase app in production: scrape the Metrics API into Prometheus, drain logs, run SQL health checks, trace requests and alert on what matters.
- [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/): Monitor a Convex app in production: use the Health page and CLI, stream logs to Axiom or Datadog, report exceptions to Sentry, and alert on failures.
- [Fix “remaining connection slots are reserved” in PostgreSQL](https://getpraxen.app/learn/troubleshooting/postgres-remaining-connection-slots-are-reserved/): Why Postgres refuses new logins with “remaining connection slots are reserved”, how to find who holds the connections, and how to pool and cap them for good.
- [Fix Prisma “P1001: Can't reach database server”](https://getpraxen.app/learn/troubleshooting/prisma-p1001-cant-reach-database-server/): Why Prisma prints “P1001: Can't reach database server”, how to test the host and port it names, and how to fix private hosts, IPv6, paused databases and URLs.
- [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/): Fix Cloudflare Hyperdrive connection errors: config codes 2008 to 2016, pool exhaustion, connection_refused and stale clients reused across Worker requests.
---
### AsyncEventsDropped: diagnose Lambda async event loss
> AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 6 min read
Canonical: https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/
AsyncEventsDropped counts Lambda asynchronous events discarded without successful function execution. Events can exceed maximum age, exhaust function-error retries, or be diverted immediately when reserved concurrency is zero. Compare event age, errors and throttles, inspect the exact invocation target's configuration, and verify failure capture.
`AsyncEventsDropped` counts Lambda asynchronous events discarded without successful function execution. The [official metric definition](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html#async-invocation-metrics) names maximum event age, exhausted retries and reserved concurrency zero as reasons. Compare event age, errors and throttles, then inspect failure capture for the exact function, version or alias the producer invokes.
A retained failure can still count as dropped. The metric describes failed original processing; the destination or DLQ tells you whether a recoverable payload survived.
#### Meaning: acceptance is not completion
An asynchronous [Invoke](https://docs.aws.amazon.com/lambda/latest/api/API_Invoke.html) uses `InvocationType=Event`. HTTP 202 acknowledges acceptance into Lambda's internal queue, not the handler's successful result. Later execution can fail, retry, wait for capacity or expire.
`AsyncEventsReceived` counts queued events, whereas `Invocations` counts function executions, including retries. Do not subtract them to estimate loss over a short interval. One event can cause several executions, queued work can cross the interval boundary, and Lambda's eventually consistent queue can deliver duplicates even without an error.
Dropped also does not mean never attempted. A handler might complete a side effect and then fail. Check the original business ID in the downstream system before replaying; the metric alone cannot establish whether that side effect happened.
#### Causes: age, retries and unavailable capacity
AWS documents separate retry paths in [asynchronous error handling](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-error-handling.html):
- **Function errors**, including timeouts: by default two retries after the initial execution, waiting one minute and then two minutes.
- **Throttling and system errors**: Lambda returns the event to its queue and retries for up to six hours by default. Intervals increase from one second to a maximum of five minutes, and a large queue can further slow retries and reads.
The [configuration API](https://docs.aws.amazon.com/lambda/latest/api/API_PutFunctionEventInvokeConfig.html) accepts `MaximumEventAgeInSeconds` from 60 to 21,600 and `MaximumRetryAttempts` from zero to two. Retry attempts govern function errors; they do not turn the throttling path into a two-retry policy. A short maximum age can expire an event before capacity or a dependency recovers.
Reserved concurrency zero is a special case. AWS [states](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-retain-records.html) that new async events begin going directly to the configured DLQ or on-failure destination without retries. Restoring concurrency does not automatically replay records already sent there. Check whether zero was intentional and identify the resulting recovery backlog.
Do not invent a most-common cause from the metric name. Use errors, throttles, age and configuration changes to determine which documented path occurred for your function.
#### Confirm the metric and related signals
In **CloudWatch → Metrics → All metrics → Lambda**, select the affected function and graph `AsyncEventsDropped` with **Sum** at a one-minute period. Align the same window with these signals from the [metric guide](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html):
| Metric | Statistic | Diagnostic question |
| --- | --- | --- |
| `AsyncEventsReceived` | Sum | Are producers successfully enqueueing? |
| `AsyncEventAge` | Maximum and useful percentiles | How long did events wait before invocation? |
| `Errors` | Sum | Are handlers failing or timing out? |
| `Throttles` | Sum | Is execution capacity unavailable? |
| `ConcurrentExecutions` | Maximum | Is concurrency reaching a configured limit? |
| `DestinationDeliveryFailures` | Sum | Did invocation-record delivery fail? |
| `DeadLetterErrors` | Sum | Did DLQ delivery fail? |
`AsyncEventAge` is observed when Lambda invokes the function; it is not a complete inventory of every event still waiting in the queue. Rising age is useful evidence of delayed processing but not a payload-level queue inspector.
Use a precise UTC incident range for a terminal query:
```bash
export AWS_REGION=us-east-1
FUNCTION=orders-worker
aws cloudwatch get-metric-statistics --region "$AWS_REGION" \
--namespace AWS/Lambda --metric-name AsyncEventsDropped \
--dimensions Name=FunctionName,Value="$FUNCTION" \
--start-time 2026-10-06T10:00:00Z --end-time 2026-10-06T11:00:00Z \
--period 60 --statistics Sum
```
Replace the sample times. Sort returned datapoints by timestamp before aligning them with logs. Missing points are not proof of successful processing: confirm producer traffic, account, Region and the resource dimension being queried.
#### Inspect the producer's actual target
If the producer uses `orders-worker:live`, inspect the alias and configuration rather than assuming `$LATEST` settings apply:
```bash
aws lambda get-alias --function-name "$FUNCTION" --name live
aws lambda get-function-event-invoke-config \
--function-name "$FUNCTION" --qualifier live
aws lambda list-function-event-invoke-configs --function-name "$FUNCTION"
aws lambda get-function-concurrency --function-name "$FUNCTION"
aws lambda get-function-configuration --function-name "$FUNCTION" \
--query '{Timeout:Timeout,DLQ:DeadLetterConfig,Role:Role}'
```
In the console, inspect **Configuration → Asynchronous invocation** and **Configuration → Concurrency**, with the relevant version or alias selected. Save the maximum age, retries, failure destination, DLQ, execution role and any concurrency change during the incident.
A missing explicit configuration does not mean "no retries." Compare default behavior and all configured target settings before deciding which policy governed the call. Match the producer's ARN to what you inspected.
#### Fix the cause and retain future failures
For rising `Errors`, read timeout and exception logs and repair the handler or its dependency. For rising `Throttles` and age, inspect reserved and account concurrency and other workloads consuming capacity. Raising concurrency helps only if the downstream service can absorb the resulting load. Initialization failures such as [EC2ThrottledException](/learn/troubleshooting/EC2ThrottledException/) need their own preparation/API investigation.
Increase maximum age only if delayed processing still has business value. A late report may be acceptable; a stale time-limited authorization may not be. Retain idempotency even when the immediate failure is fixed, because retries and duplicates remain part of the invocation model.
Configure failure capture before the next incident. This example targets an existing standard SQS queue; replace its ARN and grant the function execution role `sqs:SendMessage` on it first:
```bash
aws lambda update-function-event-invoke-config \
--function-name "$FUNCTION" --qualifier live \
--maximum-event-age-in-seconds 21600 \
--maximum-retry-attempts 2 \
--destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:123456789012:orders-async-failures"}}'
```
The [configuration guide](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-configuring.html) recommends `update` for selected changes; `put` overwrites configuration and removes omitted settings. Re-read both failure and any existing success settings after updating. If there is no explicit configuration yet, check the API result and configure the initial policy intentionally rather than treating a missing resource response as success.
AWS [supports](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-retain-records.html) standard SQS, standard SNS, Lambda and EventBridge destinations, plus S3 for failures only. Invocation records include request payload, response context and the failure condition. A Lambda DLQ instead carries the original event with error attributes and supports standard SQS or SNS. Destination settings can be per function, version or alias; DLQ settings are function-level.
Check permissions, encryption policies, resource existence and payload size. Unsupported FIFO destinations fail delivery. SNS's 256 KB maximum can be exceeded by a large async payload plus record metadata. Alarm on delivery-failure metrics separately: a configured target ARN does not guarantee successful retention.
#### Verify capture and replay without duplicating effects
Use a test alias or disposable function to exercise a known failure. Confirm that the record reached the destination, inspect its failure condition and original payload, and check delivery-failure metrics. Then send a known successful event through the repaired path and verify its business result.
Replay only retained operations that still need completion. A destination wraps the original input in `requestPayload`; sending the entire record changes the handler's input schema. A DLQ body has a different shape. Extract the intended payload, retain the original idempotency key and rate-limit replay so it does not recreate the backlog.
Without successful capture, the metric cannot reconstruct the event. Check an upstream retained source or audit trail for replay options instead of promising recovery from CloudWatch counts. Restoring capacity is not the same as completing all discarded work.
SQS-triggered Lambda uses polling and synchronous batch invocation. Its recovery policy belongs on the source queue. Follow [ChangeMessageVisibility](/learn/troubleshooting/ChangeMessageVisibility/) for visibility and Lambda mapping settings rather than applying this async-queue policy to SQS messages.
#### Prevention
Alert on dropped events where every accepted event matters, and earlier on event age relative to your processing deadline. Group age, errors and throttles into one incident with the likely cause and impact. Monitor failure-target backlog and age too: captured work that nobody reprocesses is still unfinished.
Praxen's [AWS integration](/integrations/aws/) connects it to AWS, so it can see when a function starts dropping events and work out why. The [AWS overview](/for/aws/) has more. Retention, idempotency and replay still need to be configured in AWS and the application.
Running on AWS? See [how Praxen monitors AWS in production](https://getpraxen.app/for/aws/).
#### Common questions
**Does AsyncEventsDropped mean the function never ran?**
Not necessarily. An event can be discarded after function-error retries were exhausted, so the handler may have run and failed several times. Others expire before execution or are diverted with reserved concurrency zero.
**Does a DLQ keep AsyncEventsDropped at zero?**
No. Capturing a failed event does not make original processing successful. Check the failure target and DeadLetterErrors or DestinationDeliveryFailures as well as dropped-event counts.
**Does MaximumRetryAttempts control throttling retries?**
It controls function-error retries, from zero to two. Throttling and system errors follow the queue retry path and are bounded by maximum event age, rather than the same two-retry limit.
**Does this metric describe SQS-triggered Lambda processing?**
No. SQS event source mappings poll the queue and invoke synchronous batches. Inspect the SQS visibility timeout, redrive policy and mapping instead of Lambda's internal asynchronous queue.
**Can CloudWatch recover the dropped payload?**
No. The metric is a count, not event storage. Recover from a successfully delivered failure destination, a DLQ or an upstream replay source. Without retained payloads the count cannot reconstruct an event.
#### Sources
- [Lambda metric definitions](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html)
- [Asynchronous errors and retries](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-error-handling.html)
- [Retaining asynchronous invocation records](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-retain-records.html)
- [Asynchronous configuration API](https://docs.aws.amazon.com/lambda/latest/api/API_PutFunctionEventInvokeConfig.html)
- [Configuring asynchronous error handling](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-configuring.html)
- [Praxen AWS integration](https://docs.getpraxen.app/integrations/aws)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [ChangeMessageVisibility: troubleshoot SQS message timeout errors](https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/): ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
- [EC2ThrottledException: fix Lambda EC2 API throttling](https://getpraxen.app/learn/troubleshooting/EC2ThrottledException/): EC2ThrottledException means EC2 throttled Lambda initialization. Check VPC state and API activity, distinguish concurrency limits, and verify a safe recovery.
- [AWS_EMF_ENVIRONMENT: valid values and missing EMF fixes](https://getpraxen.app/learn/troubleshooting/AWS_EMF_ENVIRONMENT/): AWS_EMF_ENVIRONMENT overrides EMF library detection. Check Node, Python and Java values separately, trace stdout or agent delivery, and fix missing metrics.
- [LambdaInsightsExtension: setup and missing metrics fixes](https://getpraxen.app/learn/troubleshooting/LambdaInsightsExtension/): LambdaInsightsExtension is the AWS Lambda Insights layer name. Choose a verified regional ARN, grant collector log permissions, and diagnose missing metrics.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
---
### AWS_EMF_ENVIRONMENT: valid values and missing EMF fixes
> AWS_EMF_ENVIRONMENT overrides EMF library detection. Check Node, Python and Java values separately, trace stdout or agent delivery, and fix missing metrics.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 7 min read
Canonical: https://getpraxen.app/learn/troubleshooting/AWS_EMF_ENVIRONMENT/
AWS_EMF_ENVIRONMENT overrides environment detection in AWS embedded-metrics client libraries. Node.js and Java document Local, Lambda, Agent, EC2 and ECS; Python's current resolver recognizes Local, Lambda, EC2 and Default, case-insensitively. Choose the value supported by your installed library and verify the full route from emitted EMF JSON to CloudWatch Logs and extracted metrics.
`AWS_EMF_ENVIRONMENT` overrides the environment selected by AWS's embedded-metrics client libraries. It controls how the library decorates and delivers metric log events; it is not the name of a CloudWatch namespace or a production/staging label. Check the value against your language library before investigating missing metrics, because the supported overrides differ.
The verified [Node.js configuration](https://github.com/awslabs/aws-embedded-metrics-node#configuration) and [Java configuration](https://github.com/awslabs/aws-embedded-metrics-java#configuration) list `Local`, `Lambda`, `Agent`, `EC2` and `ECS`. Python's [current resolver](https://github.com/awslabs/aws-embedded-metrics-python/blob/master/aws_embedded_metrics/environment/environment_detector.py) instead recognizes `Local`, `Lambda`, `EC2` and `Default`, case-insensitively. Follow the installed release, not an assumed universal AWS value list.
#### What the variable actually changes
Embedded metric format, or EMF, is JSON that carries both log context and a `_aws.CloudWatchMetrics` directive. When a valid event reaches CloudWatch Logs, CloudWatch extracts the declared metrics. The [EMF specification](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) defines that extraction contract; the library's environment setting decides which local transport and metadata to use before the event reaches the service.
That gives you three separate checks:
1. Did the application emit and flush a valid metric event?
2. Did stdout collection or an agent deliver that event to CloudWatch Logs?
3. Did CloudWatch extract the expected metric identity and timestamp?
Setting the variable alone does not install the CloudWatch Agent, grant logging permissions or route a container's stdout. Nor does it change native Lambda metrics such as `Errors` or [AsyncEventsDropped](/learn/troubleshooting/AsyncEventsDropped/).
#### Valid values, verified separately by library
The following reflects the AWS-owned library documentation and resolver source checked on 6 October 2026. Package releases can differ from the default branch, so inspect the installed release when debugging an older service.
| Library | Supported override spellings | Important behavior |
| --- | --- | --- |
| Node.js `aws-embedded-metrics` | `Local`, `Lambda`, `Agent`, `EC2`, `ECS` | Local/Lambda write stdout; Agent/EC2 use agent transport; ECS supports its documented FireLens path |
| Python `aws-embedded-metrics` | `Local`, `Lambda`, `EC2`, `Default` | Resolver lowercases input; Default selects the default agent environment; unknown values fall through to detection |
| Java `software.amazon.cloudwatchlogs:aws-embedded-metrics` | `Local`, `Lambda`, `Agent`, `EC2`, `ECS` | Documented stdout for Local/Lambda and TCP for agent environments; `AWS_EMF_WRITE_TO_STDOUT=true` can override agent output |
Node's [environment detector](https://github.com/awslabs/aws-embedded-metrics-node/blob/master/src/environment/EnvironmentDetector.ts) tests Lambda, then ECS, then EC2 when no valid override is supplied, and falls back to its default environment. It caches resolution and starts resolving at module load. Configure the process environment before importing the library rather than trying to switch modes halfway through a request.
Python's [resolver](https://github.com/awslabs/aws-embedded-metrics-python/blob/master/aws_embedded_metrics/environment/environment_detector.py) tests Lambda and then EC2, with a default-environment fallback. Its [configuration provider](https://github.com/awslabs/aws-embedded-metrics-python/blob/master/aws_embedded_metrics/config/environment_configuration_provider.py) reads `AWS_EMF_ENVIRONMENT` from the process environment. Python's README does not provide Node's override list; the resolver is the evidence for its distinct accepted values.
Use documented casing for Node and Java; do not assume Python's lowercase normalization exists everywhere. Do not invent values such as `EKS`, `Fargate`, `Production` or `CloudWatch`. EKS or an on-premises host can use an agent-based delivery path without a same-named environment override.
#### Confirm the actual package and process configuration
Run these in the application's environment, not only on your development laptop:
```bash
# Node.js project
npm ls aws-embedded-metrics
# Python process environment
python -m pip show aws-embedded-metrics
python -c 'import os; print(os.getenv("AWS_EMF_ENVIRONMENT", ""))'
# Maven project
mvn dependency:tree -Dincludes=software.amazon.cloudwatchlogs:aws-embedded-metrics
```
For Lambda, inspect only relevant keys rather than dumping every environment variable, which may include secrets:
```bash
aws lambda get-function-configuration --function-name orders-worker \
--query '{Runtime:Runtime,EMF:Environment.Variables.AWS_EMF_ENVIRONMENT,LogFormat:LoggingConfig.LogFormat}'
```
Check the version actually reached by the alias too. A changed `$LATEST` environment does not update an existing published version. For containers, inspect the running task/pod configuration and its log driver or collector, not just the Dockerfile used to build it.
In Python, you can inspect the installed resolver directly from an interactive session with `inspect.getsource`. That lets you establish which override values your wheel accepts without assuming the current repository matches the installed code.
#### Produce one known-good metric before changing the pipeline
For Node.js, install a current compatible `aws-embedded-metrics` release. The [README](https://github.com/awslabs/aws-embedded-metrics-node) specifically requires versions 4.1.1+, 3.0.2+ or 2.0.7+ for Lambda JSON log format, warning of metric loss with earlier releases in that configuration.
A minimal Lambda handler using the library's scope wrapper is:
```javascript
const { metricScope, Unit } = require("aws-embedded-metrics");
exports.handler = metricScope(metrics => async () => {
metrics.setNamespace("Orders/Diagnostics");
metrics.setDimensions({ Service: "orders-worker" });
metrics.putMetric("Probe", 1, Unit.Count);
metrics.setProperty("Purpose", "emf-delivery-check");
return { ok: true };
});
```
The wrapper handles flushing at the end of the scoped invocation. If you create a logger manually, `await metrics.flush()` before the Lambda handler returns. Do not launch the flush as unobserved background work.
For a local Python smoke check, set the environment before launching the interpreter:
```bash
AWS_EMF_ENVIRONMENT=Local python -c '
from aws_embedded_metrics import metric_scope
@metric_scope
def probe(metrics):
metrics.set_namespace("Orders/Diagnostics")
metrics.set_dimensions({"Service": "orders-worker"})
metrics.put_metric("Probe", 1, "Count")
probe()
'
```
This proves local EMF serialization, not CloudWatch ingestion. In a real Lambda environment, use autodetection or the library's `Lambda` override and its supported logging path. For Python running with an agent, the accepted override is `Default`; for Node or Java it is `Agent`. Configure the agent endpoint and log group separately according to that library's configuration documentation.
#### Missing EMF: trace the first broken step
##### Nothing leaves the application
Confirm the metric-producing code actually ran, then inspect raw stdout or the agent-facing output. Check that the logger is flushed and errors from flush are observed. In Python, also check `AWS_EMF_DISABLE_METRIC_EXTRACTION`: the [library documentation](https://github.com/awslabs/aws-embedded-metrics-python#configuration) says it can deliberately omit EMF metadata. Plain log properties without that metadata are not extraction-ready metrics.
An invalid override can fall back to detection rather than failing deployment. That is why a successful startup is insufficient verification. Confirm the emitted event and transport selected by the installed package.
##### EMF appears locally but not in CloudWatch Logs
`Local` writes stdout; it does not call the CloudWatch service directly. Follow the actual logging pipeline. On Lambda, check execution-role log permissions and the function's configured log group. In an agent-based environment, check the configured endpoint, agent listener, connectivity and the agent's own logs.
If you intentionally route stdout from an agent-based Java environment, the [Java README](https://github.com/awslabs/aws-embedded-metrics-java#configuration) documents `AWS_EMF_WRITE_TO_STDOUT=true`. That is a Java setting, not a portable instruction for all three clients. Non-Lambda Java agent output can also be asynchronously buffered; follow the README's graceful-shutdown sink procedure so a process exit does not abandon buffered events.
##### Logs arrive but no metric appears
Open the raw CloudWatch log event, not just a rendered summary. The [specification](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) requires a JSON object with no prefixed or suffixed text. `_aws` and metric/dimension targets must have the correct case and root-level placement. A timestamp prefix, stringified JSON nested inside `message`, or a multiline formatting pipeline can alter the extraction contract.
Check the declared `Namespace`, metric name, dimension sets, numeric values and timestamp. The EMF timestamp is epoch milliseconds, no more than 14 days old or two hours in the future. A seconds timestamp is not interchangeable. Inspect parsing/validation failure metrics in `AWS/Logs` as AWS recommends, and do not troubleshoot serialization by changing the transport override alone.
For the probe above, locate the metric identity explicitly:
```bash
aws cloudwatch list-metrics --namespace Orders/Diagnostics \
--metric-name Probe \
--dimensions Name=Service,Value=orders-worker
```
The namespace and dimensions are case-sensitive identities. A metric with additional default dimensions is distinct from the single-dimension series in this probe. Allow ingestion and discovery time, then query a time range containing the event's timestamp. Old or incorrectly timestamped events may exist in logs while the current chart remains empty.
#### Verification and prevention
Success means you can find all three pieces: a valid emitted JSON event, the same event in CloudWatch Logs, and a datapoint with the exact declared metric identity. Run this check in the real target environment after library, runtime, log-format or collector changes. Keep package versions pinned and document why an override is necessary; leave autodetection enabled when it already selects the correct route.
Keep high-cardinality request IDs as log properties rather than metric dimensions. The [EMF specification](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) explains that each unique dimension combination creates a custom metric, with corresponding cost. A successful extraction test should not generate an unbounded set of series.
For the separate AWS system collector, use the [LambdaInsightsExtension guide](/learn/troubleshooting/LambdaInsightsExtension/). For an automated investigator, the [AWS read-access guidance](/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/) helps separate telemetry reads from the credentials that configure and deliver telemetry.
Praxen's [AWS integration](/integrations/aws/) connects it to AWS, so it can see when something breaks there and work out why. The [AWS overview](/for/aws/) has more. When EMF delivery breaks, the useful first result is evidence of which pipeline step failed; missing custom metrics are not evidence that the application is healthy.
Running on AWS? See [how Praxen monitors AWS in production](https://getpraxen.app/for/aws/).
#### Common questions
**When should I set the override?**
Use it when the installed library's automatic detection selects the wrong metadata or delivery path, or when a controlled local test needs stdout. Set it before library initialization and verify the emitted event and actual destination; leave detection enabled when it already works.
**Is AWS_EMF_ENVIRONMENT the deployment stage?**
No. It selects a library environment and delivery behavior, such as Lambda stdout or an agent transport. Use an explicit low-cardinality dimension or property for production, staging or a service name.
**Can I set AWS_EMF_ENVIRONMENT=Agent in Python?**
The current Python environment resolver recognizes Default for its default agent environment, not Agent. It lowercases the supplied value and falls back to detection for unknown values. Check the resolver in your installed release rather than copying Node's value list.
**Does Local publish metrics directly to CloudWatch?**
No. Local writes EMF to stdout without cloud-environment decoration. Unless a supported pipeline delivers those JSON events into CloudWatch Logs for extraction, the metrics remain local output.
**Does the Lambda Insights layer use my application's EMF override?**
AWS_EMF_ENVIRONMENT is a setting for the embedded-metrics client library. Lambda Insights is a separate collector with its own installation and log delivery requirements. Do not assume changing the application's variable configures that extension.
#### Sources
- [AWS embedded-metrics Node.js configuration](https://github.com/awslabs/aws-embedded-metrics-node#configuration)
- [Node.js environment detector](https://github.com/awslabs/aws-embedded-metrics-node/blob/master/src/environment/EnvironmentDetector.ts)
- [Python environment detector](https://github.com/awslabs/aws-embedded-metrics-python/blob/master/aws_embedded_metrics/environment/environment_detector.py)
- [Python environment configuration provider](https://github.com/awslabs/aws-embedded-metrics-python/blob/master/aws_embedded_metrics/config/environment_configuration_provider.py)
- [AWS embedded-metrics Java configuration](https://github.com/awslabs/aws-embedded-metrics-java#configuration)
- [CloudWatch EMF specification](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html)
- [Praxen AWS integration](https://docs.getpraxen.app/integrations/aws)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [LambdaInsightsExtension: setup and missing metrics fixes](https://getpraxen.app/learn/troubleshooting/LambdaInsightsExtension/): LambdaInsightsExtension is the AWS Lambda Insights layer name. Choose a verified regional ARN, grant collector log permissions, and diagnose missing metrics.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
- [ChangeMessageVisibility: troubleshoot SQS message timeout errors](https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/): ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
- [EC2ThrottledException: fix Lambda EC2 API throttling](https://getpraxen.app/learn/troubleshooting/EC2ThrottledException/): EC2ThrottledException means EC2 throttled Lambda initialization. Check VPC state and API activity, distinguish concurrency limits, and verify a safe recovery.
---
### ChangeMessageVisibility: troubleshoot SQS message timeout errors
> ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 7 min read
Canonical: https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/
ChangeMessageVisibility sets a new timeout for an in-flight SQS message using its receipt handle. The countdown starts when the change is made, replacing the old deadline; it does not reset the 12-hour maximum from ReceiveMessage. Use the latest handle, extend before expiry, and inspect per-entry failures for batch changes.
`ChangeMessageVisibility` changes how long an in-flight Amazon SQS message stays hidden from other consumers. The [API reference](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html) requires a queue URL, the receipt handle from `ReceiveMessage`, and a timeout in seconds. Extend active work before visibility expires, or set zero to release it; neither operation deletes the message.
When a change fails, inspect the error and receipt-handle lifecycle before increasing the timeout. For Lambda consumers, first inspect the source queue and event source mapping rather than importing a standalone worker's heartbeat strategy.
#### Meaning: a new countdown, not added time
A received SQS message is hidden for the queue's visibility timeout or the timeout supplied to `ReceiveMessage`. The queue default is 30 seconds. It stays in flight until deletion or the end of visibility.
`ChangeMessageVisibility` starts a new countdown immediately. AWS's [example](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html) receives a message with a 60-second timeout, waits 15 seconds, then changes visibility to ten seconds. It becomes eligible for visibility at about second 25, not second 70. A shorter change can therefore release work earlier than the old deadline.
The valid duration range is 0 to 43,200 seconds. A per-message change is temporary; the next receive uses the queue's original setting unless that receive supplies an override. It does not permanently rewrite queue configuration or the message's default timeout.
There is also a [12-hour maximum from the ReceiveMessage request](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/best-practices-processing-messages-timely-manner.html). Renewals do not reset it. Asking for the full 43,200 seconds after the response arrives may already exceed remaining time because the receive consumed part of that budget. SQS fails the request rather than clamping it to what remains.
Visibility reduces concurrent delivery; it is not an exactly-once processing guarantee. Keep business operations idempotent even when every renewal returns successfully.
#### Causes to test when a change fails
These are diagnostic possibilities, not prevalence claims:
- The worker supplied a message ID instead of a receipt handle, or reused a handle from an older receive.
- Visibility expired or another path deleted the message before the change reached SQS.
- The requested timeout exceeds remaining time in the 12-hour budget.
- The URL points to the wrong queue, account or Region, or the queue was deleted.
- Frequent heartbeat requests reached a request-rate limit.
- Authorization or request signing is wrong.
- A batch returned failed entries that the worker ignored because HTTP status was 200.
Capture queue ARN, message ID, receive time, requested duration, AWS request ID and exact error code. Avoid publishing receipt handles and message bodies in broad logs. They are not useful high-cardinality metric dimensions and can expose application data.
#### Confirm queue configuration and handle ownership
Start with read operations in the intended account and Region:
```bash
export AWS_REGION=us-east-1
QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/orders
aws sts get-caller-identity
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
--attribute-names QueueArn VisibilityTimeout RedrivePolicy \
ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible
```
In **SQS → Queues → your queue → Configuration**, inspect visibility timeout and redrive policy. In **Monitoring**, look at visible and in-flight backlog and message age. Those are queue-level indicators, not proof that a particular handle's renewal succeeded.
Use a disposable queue or known test message for reproduction. Receiving a live production message changes its visibility and can compete with real workers:
```bash
aws sqs receive-message --queue-url "$QUEUE_URL" \
--max-number-of-messages 1 --wait-time-seconds 10 \
--visibility-timeout 120
# Set this to ReceiptHandle from the response, not MessageId.
aws sqs change-message-visibility --queue-url "$QUEUE_URL" \
--receipt-handle "$RECEIPT_HANDLE" --visibility-timeout 120
```
The successful response is HTTP 200 with an empty body, so the CLI may print nothing. It does not return a new receipt handle. Continue tracking the handle for that receive; a subsequent receive has its own handle.
#### Exact errors and the next check
The [operation-specific error table](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html#API_ChangeMessageVisibility_Errors) lists these errors with HTTP 400:
| Error | Meaning and confirmation |
| --- | --- |
| `MessageNotInflight` | Message is not in flight; inspect expiry, deletion and active ownership. |
| `ReceiptHandleIsInvalid` | Handle is invalid; check the latest receive and corresponding queue. |
| `QueueDoesNotExist` | Inspect exact URL, Region, account and deletion history. |
| `RequestThrottled` | Request rate exceeded permitted limits; inspect heartbeat rate and batching. |
| `InvalidSecurity` | Request lacked HTTPS or SigV4 signing; use the SDK/CLI signing path. |
| `InvalidAddress` | Specified address/ID is invalid; check the target. |
| `UnsupportedOperation` | Unsupported operation; inspect request and service response. |
Common authorization errors can occur as well. Preserve the actual SDK error instead of reducing every failure to "timeout." The reference says exceeding remaining visibility time returns an error but does not establish one universal named exception for every such request. Diagnose that case with elapsed time, requested duration and returned message.
Do not retry a permanently invalid or expired handle forever. Once the worker cannot establish a successful renewal, concurrent processing is possible. Stop or checkpoint where the application supports it, and use idempotency to prevent duplicated irreversible effects.
#### Fix standalone consumers with bounded heartbeats
AWS's [timely-processing guidance](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/best-practices-processing-messages-timely-manner.html) suggests an initial two-minute visibility timeout and extensions every minute for work of uncertain length. It is an example, not a universal rate. Leave margin for latency and retries, measure elapsed time monotonically from receiving, and stop extending abandoned or completed jobs.
Boto3 uses these real renewal and deletion operations:
```python
import boto3
from botocore.config import Config
sqs = boto3.client(
"sqs",
config=Config(retries={"mode": "standard", "total_max_attempts": 4}),
)
def extend_message(queue_url, receipt_handle, seconds):
# Caller checks remaining receive budget and keeps a safety margin.
return sqs.change_message_visibility(
QueueUrl=queue_url, ReceiptHandle=receipt_handle,
VisibilityTimeout=seconds,
)
def delete_completed_message(queue_url, receipt_handle):
return sqs.delete_message(
QueueUrl=queue_url, ReceiptHandle=receipt_handle,
)
```
These helpers do not implement a complete scheduler. The worker must observe heartbeat failures, serialize renewal completion with shutdown/deletion and enforce an overall processing deadline. A background task that fails silently has not renewed anything.
Delete after the business operation completes durably. Repeatedly renewing a stuck job delays retry and holds in-flight capacity. For work longer than 12 hours, split/checkpoint the job or use another workflow such as Step Functions, as AWS recommends.
#### Batch extensions: inspect every entry
[ChangeMessageVisibilityBatch](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibilityBatch.html) accepts at most ten entries, each with a distinct batch ID, current receipt handle and timeout. HTTP 200 can include both successful and unsuccessful entries:
```python
result = sqs.change_message_visibility_batch(
QueueUrl=queue_url,
Entries=[
{"Id": "job-1", "ReceiptHandle": handle_one, "VisibilityTimeout": 120},
{"Id": "job-2", "ReceiptHandle": handle_two, "VisibilityTimeout": 120},
],
)
for failure in result.get("Failed", []):
print(failure["Id"], failure["Code"], failure["SenderFault"])
# Retry or stop this entry according to its error and remaining budget.
```
Batch request errors include `BatchEntryIdsNotDistinct`, `EmptyBatchRequest`, `InvalidBatchEntryId` and `TooManyEntriesInBatchRequest`. Fix malformed input instead of retrying it unchanged. For transient failures, retry only appropriate failed entries and recompute remaining time. Do not reset a failed entry's local deadline because another entry succeeded.
#### Lambda interplay: source queue settings first
Lambda polls SQS and invokes synchronous batches. Its [configuration guide](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html) recommends queue visibility of at least six times function timeout plus `MaximumBatchingWindowInSeconds` when a batch window is used. Lambda validates that function timeout is no greater than queue visibility when creating or updating a mapping.
A 30-second function timeout and five-second batch window give a recommended starting visibility of 185 seconds:
```bash
aws sqs set-queue-attributes --queue-url "$QUEUE_URL" \
--attributes VisibilityTimeout=185
aws lambda list-event-source-mappings --function-name orders-worker
```
Calculate from the actual function and window. Inspect mapping state, batch size, concurrency and `FunctionResponseTypes`. Partial batch responses let Lambda retry only failed records instead of successful records from the same batch. Configure the DLQ on the SQS source queue; [AsyncEventsDropped](/learn/troubleshooting/AsyncEventsDropped/) belongs to Lambda's separate internal async invocation queue.
A handler that calls this API needs explicit `sqs:ChangeMessageVisibility` permission. The call does not extend Lambda execution time. Smaller work units and bounded downstream calls are better than preserving a message for a handler that cannot finish it.
#### Verify recovery and prevent repeats
In a test queue, renew a received message, verify it remains unavailable during the intended window, then allow expiry and confirm later redelivery. Delete a completed test message and check the durable business result. Exercise a deliberately invalid batch entry so your failure handling is verified, not merely the happy path.
Track processing duration, renewal failures, receive counts and DLQ backlog. The [LambdaInsightsExtension guide](/learn/troubleshooting/LambdaInsightsExtension/) helps investigate function CPU and memory pressure. Keep [investigation reads separate from operational writes](/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/) such as receives and renewals.
Praxen's [AWS integration](/integrations/aws/) connects it to AWS, so it can see when a queue backs up and work out why. The [AWS overview](/for/aws/) has more. Visibility deadlines and duplicate-handling guarantees still belong to the consumer design.
Running on AWS? See [how Praxen monitors AWS in production](https://getpraxen.app/for/aws/).
#### Common questions
**Does ChangeMessageVisibility add time to the existing deadline?**
No. The requested duration starts from the change and replaces the current countdown. Setting 120 seconds means about 120 seconds from the change, not 120 seconds after the old deadline.
**Can I extend a message beyond 12 hours?**
No. The maximum is measured from when SQS receives the ReceiveMessage request. Extensions do not reset it, and a request beyond the remaining time fails rather than being automatically shortened.
**Does HTTP 200 mean every batch entry succeeded?**
No. ChangeMessageVisibilityBatch can return HTTP 200 with a non-empty Failed array. Inspect each entry's Id, Code, Message and SenderFault before treating that message as extended.
**Should a Lambda SQS handler extend messages automatically?**
Start with queue visibility and event source mapping configuration. AWS recommends at least six times the function timeout plus the batch window. An extension cannot make the handler run past its function timeout.
**Does setting visibility to zero delete the message?**
No. It ends current visibility so the message can be received again. DeleteMessage is a separate operation used after processing completes durably.
#### Sources
- [ChangeMessageVisibility API reference](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html)
- [ChangeMessageVisibilityBatch API reference](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibilityBatch.html)
- [Timely processing and the visibility limit](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/best-practices-processing-messages-timely-manner.html)
- [Lambda SQS mapping configuration](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html)
- [Praxen AWS integration](https://docs.getpraxen.app/integrations/aws)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [LambdaInsightsExtension: setup and missing metrics fixes](https://getpraxen.app/learn/troubleshooting/LambdaInsightsExtension/): LambdaInsightsExtension is the AWS Lambda Insights layer name. Choose a verified regional ARN, grant collector log permissions, and diagnose missing metrics.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
- [EC2ThrottledException: fix Lambda EC2 API throttling](https://getpraxen.app/learn/troubleshooting/EC2ThrottledException/): EC2ThrottledException means EC2 throttled Lambda initialization. Check VPC state and API activity, distinguish concurrency limits, and verify a safe recovery.
- [AWS_EMF_ENVIRONMENT: valid values and missing EMF fixes](https://getpraxen.app/learn/troubleshooting/AWS_EMF_ENVIRONMENT/): AWS_EMF_ENVIRONMENT overrides EMF library detection. Check Node, Python and Java values separately, trace stdout or agent delivery, and fix missing metrics.
---
### EC2ThrottledException: fix Lambda EC2 API throttling
> EC2ThrottledException means EC2 throttled Lambda initialization. Check VPC state and API activity, distinguish concurrency limits, and verify a safe recovery.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 6 min read
Canonical: https://getpraxen.app/learn/troubleshooting/EC2ThrottledException/
EC2ThrottledException means Amazon EC2 throttled Lambda during function initialization using the function's execution role. Confirm the Invoke error, inspect VPC preparation and update state, and reduce competing EC2 API requests before retrying with backoff. This is distinct from Lambda invocation concurrency throttling.
`EC2ThrottledException` means Amazon EC2 throttled Lambda during function initialization using the execution role supplied for the function. The [Lambda Invoke reference](https://docs.aws.amazon.com/lambda/latest/api/API_Invoke.html#API_Invoke_Errors) assigns it HTTP 502. Confirm that response, inspect VPC preparation and EC2 API activity in the same account and Region, then retry only after addressing the request pressure.
Raising Lambda reserved concurrency does not raise an EC2 API request limit. Spreading out network configuration work or reducing a competing EC2 poller can help, but first identify which API and lifecycle operation failed.
#### Meaning: distinguish three different failures
| Evidence | Meaning | Where to investigate |
| --- | --- | --- |
| `EC2ThrottledException` on Invoke | EC2 throttled initialization | VPC preparation and EC2 API requests |
| `TooManyRequestsException` on Invoke | Lambda throughput or concurrency limit | Concurrency settings and `Throttles` |
| `ENILimitReachedException` or `SubnetIPAddressLimitReachedException` | Interface quota or subnet address exhaustion | VPC resource capacity |
These names come from the [Invoke error table](https://docs.aws.amazon.com/lambda/latest/api/API_Invoke.html). A handler's own EC2 SDK call can instead fail with EC2's `RequestLimitExceeded`. That is application execution, not necessarily the Lambda service's initialization exception. Preserve the service name, operation and error code in caller logs so those paths stay distinguishable.
Lambda's [metric definitions](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html) say rejected invocation requests do not count as `Invocations` or function `Errors`. An empty handler log is therefore compatible with a pre-execution failure. Do not use a quiet error chart as proof that every invocation request worked.
#### Modern Hyperplane ENIs change the diagnosis
The current [VPC guide](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#configuration-vpc-enis) describes shared Hyperplane ENIs, associated with subnet and security-group combinations. Functions in the same account can reuse those interfaces when their combinations match. Each interface supports up to 65,000 connections or ports, and Lambda adds interfaces when network traffic and concurrency require them.
This is not the historical model of creating an interface for every execution environment. A traffic burst does not imply one new ENI per invocation. New network combinations, lifecycle recreation and competing requests to the same EC2 API are hypotheses to verify rather than inevitable consequences of invocation volume.
For a new function, interface preparation can take several minutes and the function stays `Pending` until ready. For an existing function, additional operations such as code updates or version creation can be blocked while a new interface is prepared, but previous versions can continue serving invocations. An update problem is therefore not equivalent to an outage of every deployed version.
AWS also documents reclamation of unused interfaces after 14 idle days. An affected function becomes `Inactive`; its next invocation attempt fails and preparation restarts. Read the returned error and state reason before calling that lifecycle an EC2 throttling incident. Reclamation alone does not prove a rate bucket was exhausted.
#### Confirm the operation, target and state
Use AWS CLI v2 with the affected account and explicit Region:
```bash
export AWS_PROFILE=production-reader
export AWS_REGION=us-east-1
FUNCTION=orders-worker
aws sts get-caller-identity
aws lambda get-function-configuration \
--function-name "$FUNCTION" --region "$AWS_REGION" \
--query '{State:State,Reason:StateReason,ReasonCode:StateReasonCode,Update:LastUpdateStatus,UpdateReason:LastUpdateStatusReason,UpdateCode:LastUpdateStatusReasonCode,VPC:VpcConfig,Role:Role}'
```
In **Lambda → Functions → your function → Configuration → VPC**, inspect subnets and security groups. If the caller invokes an alias, inspect the alias and its published version too. A healthy `$LATEST` does not prove an alias points to it.
Save the failed request's UTC timestamp, Region, function ARN, qualifier, request ID and exact error from the caller's SDK or CLI output. Separate `UpdateFunctionConfiguration` failures from `Invoke` failures: state reason codes describe preparation and update work, while an Invoke exception describes an invocation request. These observations can correlate without being interchangeable.
For reproduction, use a harmless event or a dedicated test function. Invoke executes real code; replaying a payment event merely to capture a response can repeat side effects. A read-only configuration check is a better first operation.
#### Find competing EC2 API requests
EC2 [throttles by account, Region and API action](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html). Token buckets provide burst capacity and refill at a sustained rate. Some actions have their own buckets, and some unfiltered, unpaginated describe requests use smaller buckets. A `RunInstances` limit is not a `DescribeNetworkInterfaces` limit.
Open **CloudTrail → Event history** in the incident Region, filter **Event source** to `ec2.amazonaws.com`, and inspect recorded operations in the failure window. The CLI starting point is:
```bash
aws cloudtrail lookup-events --region "$AWS_REGION" \
--lookup-attributes AttributeKey=EventSource,AttributeValue=ec2.amazonaws.com \
--start-time 2026-10-06T10:00:00Z \
--end-time 2026-10-06T11:00:00Z \
--query 'Events[].{Time:EventTime,Name:EventName,Event:CloudTrailEvent}'
```
Replace those times with the incident. Inspect caller identity, parameters and errors for network-interface activity. Event history is not a complete request-rate counter; an absent event does not rule out throttling. Correlate caller logs and the EC2 API usage/throttle monitoring described in the guide.
In **Service Quotas → Amazon EC2**, search the implicated action's request bucket maximum capacity and refill rate. Read applied quotas rather than assuming published defaults equal your account's effective limit. Gather available EC2 request IDs and the workload responsible before requesting an increase.
#### Fix the evidence-backed cause
**Reduce bursts and polling.** Stagger infrastructure jobs that target the same APIs and pause unnecessary inventory tasks. Filter and paginate describe requests. A tight readiness loop can add API pressure while the resource is still preparing; use bounded polling with an increasing interval.
**Reuse compatible network configurations.** AWS recommends the same subnet/security-group combinations where appropriate so Lambda can reuse Hyperplane ENIs. Keep required network isolation: merging security groups solely to reduce interface creation changes access and requires an independent design review.
**Wait for an update before issuing the next one.** After configuration changes, use the CLI waiter instead of immediately publishing or updating again:
```bash
aws lambda wait function-updated-v2 \
--function-name "$FUNCTION" --region "$AWS_REGION"
aws lambda get-function-configuration \
--function-name "$FUNCTION" --region "$AWS_REGION" \
--query '[State,LastUpdateStatus,StateReasonCode,LastUpdateStatusReasonCode]'
```
A waiter failure should lead back to the state reasons, not an infinite retry loop. New-function activation and existing-function update completion are related but different checks.
**Retry transient requests with limits.** EC2 recommends exponential backoff, jitter, a maximum delay and maximum attempts. Use [SDK retry configuration](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html) instead of layering an unbounded application loop over automatic retries. For AWS CLI:
```bash
export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=5
```
Those settings enable the CLI retry policy; they do not guarantee every error is retried or that capacity recovers. Keep an overall deadline and preserve the final error. For side-effecting invocations, use an application idempotency key before resending.
**Request the specific rate increase when needed.** Follow the EC2 guide's support process with the account, Region, action and incident evidence. An ENI-count quota increase does not increase an API bucket, and Lambda concurrency does not either. Access-denied failures require permissions work rather than rate-limit work.
#### Verification and prevention
Confirm `State=Active` and `LastUpdateStatus=Successful`, then send a safe event through the actual production alias or version. Verify the service response and expected business result. Track caller-side initialization errors alongside native Lambda metrics so rejected requests remain visible.
For asynchronous producers, HTTP 202 means queued, not completed. Check [AsyncEventsDropped](/learn/troubleshooting/AsyncEventsDropped/) and event age for backlog or discarded work. For system-level evidence once code runs, use the [LambdaInsightsExtension checks](/learn/troubleshooting/LambdaInsightsExtension/).
Record EC2 callers, polling frequencies and network combinations created by infrastructure jobs. Alert on the specific failed API and inspect state reasons before changing capacity. An automated investigator should receive the [read permissions needed for evidence](/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/) separately from identities that change networking.
#### Praxen and Lambda start-up failures
Praxen's [AWS integration](/integrations/aws/) connects it to AWS, so it can see when a function fails to start and work out why. The [AWS overview](/for/aws/) has more. EC2 request quotas and retry behaviour still require the corresponding AWS and application fixes.
Running on AWS? See [how Praxen monitors AWS in production](https://getpraxen.app/for/aws/).
#### Common questions
**Is EC2ThrottledException a Lambda concurrency error?**
No. Lambda documents it as EC2 throttling during initialization, with HTTP 502. Lambda concurrency throttling uses TooManyRequestsException, with HTTP 429. Check the actual service error before changing reserved concurrency.
**Does every Lambda invocation create an ENI?**
No. Lambda uses shared Hyperplane ENIs for subnet and security-group combinations. Each supports up to 65,000 connections or ports, and Lambda can add interfaces as network demand grows.
**Will adding EC2 permissions fix throttling?**
Permissions fix access denial, not an exhausted API rate bucket. Verify VPC permissions for access-denied errors; do not broaden IAM just because EC2ThrottledException appeared.
**Can previous versions run while VPC configuration updates?**
AWS says previously published versions can still be invoked while Lambda prepares the new Hyperplane ENI. New functions remain Pending until the interface is ready, and further update operations can be blocked during preparation.
**Why might there be no handler logs?**
The request can fail before code execution. Save the caller's request ID and service error, then inspect function state. Native Errors and Invocations do not count every rejected invocation request.
#### Sources
- [Lambda Invoke API errors](https://docs.aws.amazon.com/lambda/latest/api/API_Invoke.html)
- [Lambda VPC configuration and Hyperplane ENIs](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html)
- [EC2 API request throttling](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html)
- [Lambda metric definitions](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html)
- [Praxen AWS integration](https://docs.getpraxen.app/integrations/aws)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [LambdaInsightsExtension: setup and missing metrics fixes](https://getpraxen.app/learn/troubleshooting/LambdaInsightsExtension/): LambdaInsightsExtension is the AWS Lambda Insights layer name. Choose a verified regional ARN, grant collector log permissions, and diagnose missing metrics.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
- [ChangeMessageVisibility: troubleshoot SQS message timeout errors](https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/): ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
- [AWS_EMF_ENVIRONMENT: valid values and missing EMF fixes](https://getpraxen.app/learn/troubleshooting/AWS_EMF_ENVIRONMENT/): AWS_EMF_ENVIRONMENT overrides EMF library detection. Check Node, Python and Java values separately, trace stdout or agent delivery, and fix missing metrics.
---
### LambdaInsightsExtension: setup and missing metrics fixes
> LambdaInsightsExtension is the AWS Lambda Insights layer name. Choose a verified regional ARN, grant collector log permissions, and diagnose missing metrics.
By Boris Tane, Founder of Praxen · Published October 6, 2026 · 6 min read
Canonical: https://getpraxen.app/learn/troubleshooting/LambdaInsightsExtension/
LambdaInsightsExtension is the official x86-64 layer name for CloudWatch Lambda Insights; ARM64 uses LambdaInsightsExtension-Arm64. Attach the ARN from AWS's current table for the function's Region and architecture, grant collector logging permissions, and verify performance logs and extracted metrics. It is not an error code or a metric name.
`LambdaInsightsExtension` is the official x86-64 layer name for CloudWatch Lambda Insights, not an exception or a metric. AWS's [x86-64 table](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsx86-64.html) uses that name, and its [ARM64 table](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsARM.html) uses `LambdaInsightsExtension-Arm64`. For missing data, verify the collector on the invoked version, its logging permissions, delivery path and EMF performance events.
The extension adds execution-environment evidence that ordinary invocation counters do not provide. It does not automatically instrument every application span or establish whether a business transaction succeeded.
#### Meaning: system collector, layer name and namespace
The [Lambda Insights overview](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights.html) describes collection of CPU, memory, disk and network metrics, plus cold-start and worker-shutdown diagnostic information. For standard functions it emits a performance log event per invocation, with CloudWatch extracting metrics through embedded metric format.
The [standard-function definitions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-metrics-lambda-functions.html) distinguish `LambdaInsights` time series from additional fields in performance logs:
| Signal | Where to inspect | Meaning |
| --- | --- | --- |
| `cpu_total_time` | Metrics and logs | CPU time in milliseconds, not wall-clock duration |
| `init_duration` | Metrics | Time in the execution environment's init phase |
| `memory_utilization`, `used_memory_max` | Metrics and logs | Memory pressure and measured environment memory |
| `tmp_used`, `tmp_free` | Metrics | Temporary-storage use and free space |
| `rx_bytes`, `tx_bytes`, `total_network` | Metrics and logs | Network bytes, including runtime activity |
| `agent_version`, `agent_memory_max`, `fd_use` | Performance logs | Collector version, memory and descriptor evidence |
Not every log field is a standalone time series. Searching for `agent_version` in the metrics browser is not a reliable installation check. Native invocation metrics remain in the separate `AWS/Lambda` namespace.
For Lambda Managed Instances, use AWS's [separate metrics table](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-metrics-managed-instances.html). The extension version notes describe one-minute EMF emission for that environment. Do not apply standard per-invocation expectations to its collection path.
#### Pick regional and architecture ARNs from current tables
An ARN includes Region, publisher account, layer name and a numeric layer version. The final number is not the software release number and can vary for the same release across Regions. Neither swapping a Region string nor guessing an increment gives a verified artifact.
The official tables, checked on 6 October 2026, provide these examples:
| Architecture and Region | Software release | Full ARN |
| --- | --- | --- |
| x86-64, `us-east-1` | `1.0.776.0` | `arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension:66` |
| x86-64, `us-east-2` | `1.0.776.0` | `arn:aws:lambda:us-east-2:580247275435:layer:LambdaInsightsExtension:71` |
| ARM64, `us-east-1` | `1.0.660.0` | `arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension-Arm64:31` |
| ARM64, `us-east-2` | `1.0.660.0` | `arn:aws:lambda:us-east-2:580247275435:layer:LambdaInsightsExtension-Arm64:33` |
These are dated examples from the [x86-64](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsx86-64.html) and [ARM64](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsARM.html) tables, not a permanent latest alias. Recheck the applicable table before installation. Availability and releases do not necessarily match across architectures.
The x86-64 `1.0.776.0` notes describe an RPM packaging fix. That can explain a collector that fails to start in an affected image, but it does not prove every telemetry gap has that cause. Correlate release notes with installed package versions and actual startup errors.
#### Confirm the real function and invoked version
Inspect configuration before attaching anything:
```bash
export AWS_REGION=us-east-1
FUNCTION=orders-worker
aws lambda get-function-configuration --function-name "$FUNCTION" \
--query '{Architecture:Architectures,Runtime:Runtime,Package:PackageType,Role:Role,Layers:Layers,VPC:VpcConfig,State:State}'
aws lambda get-alias --function-name "$FUNCTION" --name live
```
For alias traffic, inspect its resolved version too. Updating `$LATEST` does not change an existing published version. In the console check **Layers**, **Runtime settings**, **Configuration → Permissions**, and the selected version or alias.
The [overview](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights.html) supports agent runtimes based on Amazon Linux 2 and Amazon Linux 2023. For ZIP functions, attach the architecture-matched layer. For image functions, follow the [container setup procedure](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-Getting-Started-docker.html) to install the extension package in the image; attaching a ZIP layer is not equivalent.
#### Install while preserving existing layers
AWS's [CLI setup](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-Getting-Started-cli.html) requires the collector's execution-role policy:
```bash
aws iam attach-role-policy --role-name orders-worker-role \
--policy-arn arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy
```
This example is for the commercial partition. Use the role actually attached to the function. The [policy reference](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchLambdaInsightsExecutionRolePolicy.html) grants log-group creation and stream/event writes for `/aws/lambda-insights`; the collector uses EMF extraction rather than requiring a handler `PutMetricData` call.
`--layers` replaces the whole layer list. Keep other layers and replace an existing Insights collector rather than adding a second one. This Boto3 example deliberately targets an x86-64 ZIP function in `us-east-1`; substitute a verified ARN for other targets:
```python
import boto3
client = boto3.client("lambda", region_name="us-east-1")
name = "orders-worker"
insights_arn = "arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension:66"
cfg = client.get_function_configuration(FunctionName=name)
assert cfg["PackageType"] == "Zip"
assert cfg["Architectures"] == ["x86_64"]
other_layers = [
layer["Arn"] for layer in cfg.get("Layers", [])
if layer["Arn"].split(":layer:")[1].split(":")[0]
not in {"LambdaInsightsExtension", "LambdaInsightsExtension-Arm64"}
]
client.update_function_configuration(
FunctionName=name, RevisionId=cfg["RevisionId"],
Layers=other_layers + [insights_arn],
)
client.get_waiter("function_updated_v2").wait(FunctionName=name)
```
Check the five-layer and total unzipped package-size limits first. `RevisionId` avoids silently overwriting a concurrent configuration update. Put the equivalent change in infrastructure code for repeatability, then publish and route traffic according to your normal version workflow.
#### Missing metrics: identify the first failed stage
Run a harmless test invocation through the target. Inspect **CloudWatch → Logs → Log groups → /aws/lambda-insights** in the same Region and account. Filter by function and read current timestamps and `agent_version`. A terminal starting point is:
```bash
aws logs tail /aws/lambda-insights --since 10m --format short
aws cloudwatch list-metrics --namespace LambdaInsights \
--dimensions Name=function_name,Value=orders-worker
```
The dimension is `function_name`, optionally accompanied by `version`; it is not native Lambda's `FunctionName`. Incorrect dimensions can produce an empty chart even when data exists.
In Logs Insights, select `/aws/lambda-insights` and run a function-scoped query over the test's time window:
```text
fields @timestamp, function_name, version, agent_version, cpu_total_time, used_memory_max
| filter function_name = "orders-worker"
| sort @timestamp desc
| limit 20
```
Compare the version in these events with the version resolved by the alias. If only an older version appears, confirm which version received the test traffic before changing collector permissions. Save one representative performance event and its timestamp as installation evidence, then query the matching metric interval rather than an arbitrary current chart window.
**No performance logs:** confirm the invoked version contains the collector, its architecture matches, and startup logs show no extension failure. Check execution-role log permissions. For private subnets without internet access, the CLI setup requires a CloudWatch Logs interface VPC endpoint if one is absent. Inspect endpoint policy, private DNS and security-group connectivity; IAM cannot fix a blocked delivery route.
**Logs exist but no extracted metric:** inspect EMF metadata and the metric's namespace, dimensions and timestamp. Choose a field actually listed as aggregated, use an invocation-covering time range and allow ingestion time. The [AWS_EMF_ENVIRONMENT guide](/learn/troubleshooting/AWS_EMF_ENVIRONMENT/) explains the extraction contract, but that application-library variable is not a documented collector configuration switch.
**Native metrics exist but Insights does not:** they have independent production paths. `Invocations` does not establish layer installation or performance-log delivery. For a request rejected during [EC2ThrottledException initialization](/learn/troubleshooting/EC2ThrottledException/), use caller and state evidence rather than expecting a collector sample for every rejected invocation.
#### Cost, cold starts and verification
AWS's [pricing section](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights.html) charges for metrics/logs and consumed extension execution time in one-millisecond increments. Consult current CloudWatch and Lambda prices for the Region; there is no universal cost per installed layer.
Extensions [share CPU, memory and storage](https://docs.aws.amazon.com/lambda/latest/dg/using-extensions.html) and must complete initialization before code runs. Compare equivalent enabled and disabled versions under the same workload. Inspect cold-start latency, `init_duration`, memory, billed duration and `PostRuntimeExtensionsDuration`. The last measures extension work after the runtime, not all initialization overhead.
Verify current performance logs, expected time-series datapoints, a successful business result and acceptable resource/latency impact. Pin the ARN in infrastructure code, review release notes before upgrades and repeat after runtime, architecture, role or VPC changes. Define a separate [read-access identity](/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/) for investigations.
Praxen connects to [AWS](/integrations/aws/), so it can see when a function breaks there and work out why. The collector still needs to be installed and delivering data; its layer name alone cannot reveal missing CPU or memory samples.
Running on AWS? See [how Praxen monitors AWS in production](https://getpraxen.app/for/aws/).
#### Common questions
**Is LambdaInsightsExtension an error?**
No. It is the official x86-64 layer name in AWS's published ARNs. ARM64 uses LambdaInsightsExtension-Arm64. The metrics namespace is LambdaInsights, not the layer name.
**Can one layer ARN work in every Region?**
No. Copy the full ARN from the current AWS table for the function's Region and architecture. Publisher IDs and layer version numbers vary; software release numbers are not numeric layer versions.
**Does the collector require cloudwatch:PutMetricData?**
Its documented execution-role policy grants CloudWatch Logs operations. The extension writes EMF performance events and CloudWatch extracts metrics from those logs.
**Is Lambda Insights free or overhead-free?**
No. Logs, metrics and consumed extension execution time are billed. Extensions share CPU, memory and storage and must initialize before invocation, so compare enabled and disabled versions under an equivalent workload.
**Why do native metrics appear while Insights is empty?**
They use separate paths. Native metrics do not prove the invoked version has the collector installed or that performance logs arrived. Check the layer, execution role, networking and /aws/lambda-insights log group.
#### Sources
- [Lambda Insights overview](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights.html)
- [Official x86-64 versions and ARNs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsx86-64.html)
- [Official ARM64 versions and ARNs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsARM.html)
- [Lambda Insights CLI setup](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-Getting-Started-cli.html)
- [Collector execution-role policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchLambdaInsightsExecutionRolePolicy.html)
- [Standard-function metric definitions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-metrics-lambda-functions.html)
- [Extension lifecycle and resource impact](https://docs.aws.amazon.com/lambda/latest/dg/using-extensions.html)
- [Container-image installation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-Getting-Started-docker.html)
- [Praxen AWS integration](https://docs.getpraxen.app/integrations/aws)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [AWS_EMF_ENVIRONMENT: valid values and missing EMF fixes](https://getpraxen.app/learn/troubleshooting/AWS_EMF_ENVIRONMENT/): AWS_EMF_ENVIRONMENT overrides EMF library detection. Check Node, Python and Java values separately, trace stdout or agent delivery, and fix missing metrics.
- [EC2ThrottledException: fix Lambda EC2 API throttling](https://getpraxen.app/learn/troubleshooting/EC2ThrottledException/): EC2ThrottledException means EC2 throttled Lambda initialization. Check VPC state and API activity, distinguish concurrency limits, and verify a safe recovery.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [ChangeMessageVisibility: troubleshoot SQS message timeout errors](https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/): ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
---
### Fix Cloudflare “Error 1102: Worker exceeded resource limits”
> Why a Cloudflare Worker returns Error 1102, how to tell a CPU time overrun from a 128 MB memory overrun, and the code and Wrangler changes that fix each.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 14 min read
Canonical: https://getpraxen.app/learn/troubleshooting/cloudflare-error-1102-worker-exceeded-resource-limits/
```text
Error 1102: Worker exceeded resource limits
```
Error 1102 means your Worker went over its CPU time limit or its 128 MB memory limit, so Cloudflare stopped it and served an error page. Check Metrics > Errors > Invocation Statuses to see whether it was Exceeded CPU Time Limits or Exceeded Memory. Fix CPU overruns by profiling the hot code or raising limits.cpu_ms on the Paid plan, and fix memory overruns by streaming bodies and keeping large data out of the isolate.
Your users see a Cloudflare error page with code 1102 and the line `Worker exceeded resource limits`. The Worker didn't throw. It ran out of room. The Workers runtime stopped it because it used more CPU time or more memory than its limits allow, and it had no response to send.
The message is the same for both limits, so the first job is to find out which one you hit. The Worker's metrics tell you in one click. After that the fix is specific: profile and trim the code, or raise the CPU limit, for a CPU overrun; stream bodies and move large data out of the isolate for a memory overrun.
#### What “Worker exceeded resource limits” means
Cloudflare's [Error 1102 page](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1102/) defines it as a Worker that has exceeded its CPU time limit or its memory limit. The [Workers limits page](https://developers.cloudflare.com/workers/platform/limits/) gives the numbers behind both:
- **CPU time.** 10 ms per HTTP request on the Workers Free plan. On the Workers Paid plan the default is 30 seconds, and you can raise it to 5 minutes. CPU time is the time spent executing your code, such as loops and JSON parsing. Waiting on `fetch()`, KV reads or database queries does not count.
- **Memory.** 128 MB per isolate on both plans, including the JavaScript heap and WebAssembly allocations. The limit applies to the isolate, and one isolate can handle many requests at the same time.
Each limit leaves its own trace. The client gets the same 1102 page either way, but the dashboard, analytics and Logpush name them differently:
- CPU: `Exceeded CPU Time Limits` under **Metrics** > **Errors** > **Invocation Statuses**, outcome `exceededCpu`.
- Memory: `Exceeded Memory` in the same chart, outcome `exceededMemory`. You may also see the runtime error `Memory limit would be exceeded before EOF` when the Worker tries to buffer a body that would cross the limit.
The [errors reference](https://developers.cloudflare.com/workers/observability/errors/) still describes 1102 in its table as a CPU time overrun, so don't let that convince you memory is ruled out. The support page and the limits page both map memory overruns to 1102 too.
```mermaid alt="A Worker request exceeding CPU time or 128 MB of memory ends in Cloudflare Error 1102"
graph TB
A[Request reaches the Worker] --> B[Code runs in an isolate]
B --> C{CPU time over the limit?}
C -->|yes| D[Outcome exceededCpu]
C -->|no| E{Isolate over 128 MB?}
E -->|yes| F[Outcome exceededMemory]
E -->|no| G[Response returned]
D --> H[Client gets Error 1102]
F --> H
style H fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
At Cloudflare I built and led the Workers observability team, and the invocation status was always the first thing to look at for a 1102. It splits the problem in two before you open any code.
#### Telling a CPU overrun from a memory overrun
Start in the dashboard:
1. Go to **Workers & Pages** and select the Worker.
2. Under **Metrics**, select **Errors** > **Invocation Statuses**.
3. Read the two rows: **Exceeded CPU Time Limits** and **Exceeded Memory**. Note when they started and whether the start lines up with a deploy.
Then find the requests behind them in [Workers Logs](https://developers.cloudflare.com/workers/observability/logs/workers-logs/). Logs only exist if observability is on for the Worker. Check your Wrangler file:
```jsonc
{
"observability": {
"enabled": true,
"head_sampling_rate": 1
}
}
```
```toml
[observability]
enabled = true
head_sampling_rate = 1
```
`head_sampling_rate` defaults to 1, which logs every request. A low value can hide the few requests that blow the limit.
In the Workers Logs query builder, the errors reference suggests `$metadata.error EXISTS` to find every log with an error, and filtering on `$workers.outcome` to find requests by result. The limits page names the outcomes `exceededCpu` and `exceededMemory` for analytics and Logpush; it doesn't say Workers Logs uses the same strings, so check the outcome values your logs actually show before you save the filter:
```text
$workers.outcome = "exceededCpu"
$workers.outcome = "exceededMemory"
```
Each Worker invocation writes one invocation log, and CPU time and wall time appear in it. Group the failing requests by path. If one route or one kind of payload owns most of them, you have found where to look.
#### Cause 1: the Worker uses more CPU time than its plan allows
Start here. CPU time is the only one of the two limits that differs by plan, and 10 ms on the Free plan is easy to cross. The limits page says the average Worker uses about 2.2 ms of CPU per request, while heavier work such as authentication, server-side rendering or parsing large payloads typically takes 10 to 20 ms. A Free plan Worker doing any of those sits right at its limit.
**How to tell it's yours:** the **Exceeded CPU Time Limits** row is the one climbing, the logs show `exceededCpu`, and the invocation logs for those requests show CPU time at or near your limit.
The limits page also notes that each isolate has some built-in flexibility for a Worker that only goes over now and then. Once it hits the limit consistently, the runtime enforces it. So a route that was fine at low traffic or with small payloads can start failing when either grows.
##### Find the expensive code with a CPU profile
Measuring CPU time inside a production Worker is hard, because Workers only advance timers on I/O. Cloudflare's [CPU profiling guide](https://developers.cloudflare.com/workers/observability/dev-tools/cpu-usage/) does it locally with DevTools:
1. Start the Worker locally:
```bash
npx wrangler dev
```
2. Press `D` in the terminal to open DevTools.
3. Select the **Profiler** tab and select **Start**.
4. Send requests that look like production. Hit the same route, with payloads of the same size. Remote bindings let you use production-like data.
5. Select **Stop** and read the chart. Each box is a function, and its width is the CPU time it used. Switch to **Heavy (Bottom Up)** to list functions by total cost.
A large share of garbage collection in that list is a hint that you are also allocating too much, which leads to cause 2.
##### Cut the work per request
The 1102 support page lists the usual fixes:
- Reduce the number of iterations in loops.
- Optimise JSON parsing. Parse only the payloads you need, and avoid parsing the same body twice.
- Cache computed values. A value that is the same for every request can be computed once per isolate.
- Break large operations into smaller chunks, spread across multiple requests.
The limits page adds one more: move expensive computation to a [Durable Object](https://developers.cloudflare.com/durable-objects/), or process the data in smaller chunks across multiple requests.
##### Raise the CPU limit on the Paid plan
If the work really needs the CPU, raise the limit. This only applies on Workers Paid, where you can go from the 30 second default up to 5 minutes. On the Free plan the limit stays at 10 ms.
1. Set `limits.cpu_ms` in your Wrangler file. In `wrangler.jsonc`:
```jsonc
{
"limits": {
"cpu_ms": 300000 // default is 30000 (30 seconds)
}
}
```
Or in `wrangler.toml`:
```toml
[limits]
cpu_ms = 300_000
```
2. Deploy:
```bash
npx wrangler deploy
```
3. Or change it in the dashboard: **Workers & Pages** > your Worker > **Settings**, then adjust the CPU time limit.
Pick a value with headroom over what your slowest real request uses. Cron Triggers have their own ceiling: 30 seconds for schedules more often than once an hour, and 15 minutes for hourly or less often.
#### Cause 2: one request buffers a large body into memory
The memory limit is 128 MB on every plan. The quickest way to cross it is to read a whole request or response body into a string, an `ArrayBuffer` or a parsed object. The 1102 page names this pattern first: buffering a body that could be large.
**How to tell it's yours:** **Exceeded Memory** is the row climbing, the logs show `exceededMemory`, and you may see `Memory limit would be exceeded before EOF` as a runtime error. The failing requests carry large uploads or call an upstream that returns large responses. Search your code for `await request.text()`, `await response.arrayBuffer()` or `await response.json()` on anything that isn't small.
##### Stream the body through a TransformStream
The fix is to process the body as it flows. This version holds the whole upstream response in memory:
```ts
export default {
async fetch(request, env, ctx): Promise {
const upstream = await fetch("https://example.com/export.ndjson");
const text = await upstream.text(); // the whole body lands in the isolate
return new Response(text, { headers: upstream.headers });
},
} satisfies ExportedHandler;
```
This version passes it through a [`TransformStream`](https://developers.cloudflare.com/workers/runtime-apis/streams/transformstream/), so only the chunk in flight is in memory:
```ts
export default {
async fetch(request, env, ctx): Promise {
const upstream = await fetch("https://example.com/export.ndjson");
let bytes = 0;
const { readable, writable } = new TransformStream({
transform(chunk, controller) {
bytes += chunk.byteLength; // do per-chunk work here
controller.enqueue(chunk);
},
flush() {
console.log({ message: "export streamed", bytes });
},
});
ctx.waitUntil(upstream.body!.pipeTo(writable));
return new Response(readable, { headers: upstream.headers });
},
} satisfies ExportedHandler;
```
Steps:
1. Find each place you read a body with `.text()`, `.json()` or `.arrayBuffer()`.
2. For bodies whose size you don't control, move the work into a `TransformStream`, or use [`node:stream`](https://developers.cloudflare.com/workers/runtime-apis/nodejs/streams/) if your code is already written against Node streams.
3. When you call `fetch()` and don't need the response body, cancel it to free the memory, as the limits page shows:
```ts
const response = await fetch(url);
if (response.status > 299) {
response.body?.cancel();
}
```
4. Be careful with code that grows a string or array on every chunk. The 1102 page warns against appending to strings or arrays repeatedly, and that undoes the streaming.
#### Cause 3: the isolate's 128 MB fills up across requests
The limit is per isolate. The limits page is explicit that a single isolate handles many requests concurrently, and every one of them draws on the same 128 MB. So the request that fails is often a normal request that arrived while memory was already full.
**How to tell it's yours:** `exceededMemory` shows up across many routes, including small ones. It gets worse with traffic, and requests that fail in production pass when you replay them one at a time. Look in your code for data held at module scope: a cache in a `Map` that never evicts, a large JSON file imported into the bundle, an array that grows on each request.
When an isolate crosses 128 MB, the runtime lets in-flight requests complete and creates a new isolate for the requests that follow. During extremely high load it may cancel some incoming requests to stay stable. A leak shows as a sawtooth: memory climbs, the isolate is replaced, and it climbs again.
```mermaid alt="Diagnosing Worker memory overruns by body size, traffic growth and startup allocations"
graph TB
A[Exceeded Memory in metrics] --> B{Fails on large bodies only?}
B -->|yes| C[Stream the body]
B -->|no| D{Grows with traffic?}
D -->|yes| E[Snapshot module scope]
E --> F[Move data to KV, R2 or D1]
D -->|no| G[Check startup allocations]
style A fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style C fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
##### Take a memory snapshot
The [memory profiling guide](https://developers.cloudflare.com/workers/observability/dev-tools/memory-usage/) uses the same DevTools session:
1. Run `npx wrangler dev` and press `D`.
2. Select the **Memory** tab.
3. Send a large volume of requests, with `curl` in a loop or a load tool.
4. Select **Take snapshot**, then choose **Statistics** to see which types hold the memory.
5. Send more traffic and take a second snapshot. Whatever keeps growing between the two is your leak. The **Summary** view lists the objects, and clicking through takes you to the code that keeps them alive.
##### Keep large data out of the isolate
The limits page recommends storing large data in KV, R2 or D1 and keeping it out of Worker memory. Here a lookup table is bundled and parsed into every isolate:
```ts
import prices from "./prices.json"; // parsed into every isolate at startup
export default {
async fetch(request, env, ctx): Promise {
const sku = new URL(request.url).searchParams.get("sku") ?? "";
return Response.json(prices[sku] ?? null);
},
} satisfies ExportedHandler;
```
Store each entry under its own key in KV and read only the one you need:
```ts
export default {
async fetch(request, env, ctx): Promise {
const sku = new URL(request.url).searchParams.get("sku") ?? "";
const price = await env.PRICES.get(`sku:${sku}`, "json");
return Response.json(price);
},
} satisfies ExportedHandler;
```
```jsonc
{
"kv_namespaces": [{ "binding": "PRICES", "id": "" }]
}
```
For an in-memory cache you want to keep, give it a hard size cap and evict old entries.
One dependency is worth checking. The limits page says that if your Worker uses Zod, you should use version 4.5.0 or later, because earlier versions use substantially more memory per schema:
```bash
npm ls zod
npm install zod@^4.5.0
```
We hit this ourselves in Durable Objects, where an isolate filled with schemas built at module load before it served a request. The write-up is in [how we fixed our Durable Objects memory exceeded errors](/blog/how-we-fixed-our-cloudflare-durable-objects-memory-exceeded-errors/).
#### Cause 4: startup work fails the deploy with error 10021
A related error shows up when you deploy. A Worker must parse and run its global scope, everything outside the handlers, within 1 second. If it can't, the upload fails validation with `Script startup exceeded CPU time limit`, error code `10021`. The [errors reference](https://developers.cloudflare.com/workers/observability/errors/#validation-errors-10021) lists a memory twin as well: `Script startup exceeded memory limit`, for top-level code that allocates more than 128 MB.
**How to tell it's yours:** `wrangler deploy` or `wrangler versions upload` fails with one of those messages. Users don't see a 1102, because the new version never goes live. Heavy startup code also eats into the memory every request has left, which can feed cause 3.
Steps:
1. Profile the startup phase locally:
```bash
npx wrangler check startup
```
It prints a summary and writes a `.cpuprofile` file you can open in Chrome DevTools or VS Code. When a deploy fails with a startup time error, Wrangler generates this profile for you.
2. Check the bundle size, since larger bundles start slower:
```bash
npx wrangler deploy --outdir bundled/ --dry-run
```
3. Move expensive setup out of global scope, into the handler or to build time. The limits page names generating or consuming a large schema at the top level as a common cause. Build it on first use:
```ts
let schemas: ReturnType | undefined;
function getSchemas() {
schemas ??= buildSchemas();
return schemas;
}
export default {
async fetch(request, env, ctx): Promise {
const body = getSchemas().order.parse(await request.json());
return Response.json({ ok: true, id: body.id });
},
} satisfies ExportedHandler;
```
4. Deploy with `npx wrangler@latest deploy` and read `startup_time_ms` in the output.
#### Confirming the 1102s have stopped
1. Deploy the fix and note the time.
2. Watch **Metrics** > **Errors** > **Invocation Statuses**. The **Exceeded CPU Time Limits** or **Exceeded Memory** row should fall to zero from the deploy onward.
3. In Workers Logs, run your outcome filter again for the period after the deploy. It should return nothing.
4. For a CPU fix, open a few invocation logs on the route that used to fail and compare the CPU time with your limit. A fix that only brings CPU time down to just under the limit will fail again when payloads grow. You want comfortable headroom.
5. For a memory fix, repeat the load test against `wrangler dev` with two snapshots. The growth between them should be gone.
#### Keeping Workers under their CPU and memory limits
- **Alert on the outcomes.** Send Workers trace events to your log tool with Logpush and alert on `exceededCpu` and `exceededMemory`. CPU time and wall time appear at the top level of the Workers Trace Events object.
- **Run `npx wrangler check startup` in CI,** so a dependency that makes startup heavier fails the build before it fails a deploy.
- **Stream by default.** Treat `.text()` and `.arrayBuffer()` on a body you don't control as a review comment.
- **Cap every module-scope cache.** A cache with no size limit grows until the isolate is replaced.
- **Keep sampling high enough to catch the rare request.** A `head_sampling_rate` of 0.01 logs one request in a hundred, and the one that blew the limit may not be in it.
- **Set `cpu_ms` on purpose** on the Paid plan, so the limit reflects the work the Worker does.
#### Tracing a Worker's 1102s back to the code with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Cloudflare, so when a Worker starts failing on resource limits it works out why, and the fix arrives ready for your team's normal review.
Running on Cloudflare? See [how Praxen monitors Cloudflare in production](https://getpraxen.app/for/cloudflare/).
#### Common questions
**Is Error 1102 about CPU or memory?**
It can be either. Cloudflare returns the same page and the same message, Worker exceeded resource limits, for both. Open the Worker in the dashboard and go to Metrics > Errors > Invocation Statuses: CPU overruns show as Exceeded CPU Time Limits and memory overruns as Exceeded Memory. In analytics and Logpush the outcomes are exceededCpu and exceededMemory.
**What is the CPU time limit for a Cloudflare Worker?**
On the Workers Free plan it is 10 ms per HTTP request. On the Workers Paid plan the default is 30 seconds, and you can raise it to 5 minutes (300,000 ms) with limits.cpu_ms in your Wrangler file or in the Worker's Settings in the dashboard. Cron Triggers on Paid get 30 seconds for intervals under an hour and 15 minutes for an hour or more.
**Can I raise the 128 MB memory limit?**
There is no Wrangler setting for it. The limits page lists 128 MB per isolate on both the Free and Paid plans, and it covers the JavaScript heap and WebAssembly allocations. Cloudflare's limits page links a Limit Increase Request Form for limits in general, and Cloudflare contacts you if the limit can be increased.
**Does time spent waiting on fetch count toward CPU time?**
No. CPU time only counts the time the CPU spends executing your code, such as loops and JSON parsing. Waiting on fetch() calls, KV reads or database queries does not count. A Worker that is slow because of a slow upstream will not hit 1102 for that reason.
**Why does Error 1102 only happen on some requests?**
Two reasons. Each isolate has some built-in flexibility for a Worker that only goes over the CPU limit now and then, and the limit is enforced once it happens consistently. Memory is shared by every request an isolate handles at once, so a request that is fine on its own can push the isolate over 128 MB when it lands next to a few heavy ones.
**What is the difference between Error 1101 and Error 1102?**
1101 means the Worker threw a JavaScript exception it never caught. 1102 means it went over a resource limit, CPU time or memory. The dashboard counts them separately: Uncaught Exception for 1101, and Exceeded CPU Time Limits or Exceeded Memory for 1102.
**What does Script startup exceeded CPU time limit mean?**
It is the deploy-time version of this problem. Cloudflare runs your Worker's global scope when you deploy, and if that takes more than the 1 second startup limit the deploy fails with validation error 10021. Run npx wrangler check startup to get a CPU profile of the startup phase, then move the expensive work into your handler or to build time.
**Can Error 1102 show up as a 503?**
Yes. Cloudflare's 1102 troubleshooting page lists Error 503 as related, because Workers CPU or memory limits can also cause it. If you see 503s on a Worker route, check the same Invocation Statuses chart before you look anywhere else.
#### Sources
- [Error 1102 (Cloudflare Support docs)](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1102/)
- [Limits (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/platform/limits/)
- [Errors and exceptions (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/errors/)
- [Profiling CPU usage (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/dev-tools/cpu-usage/)
- [Profiling Memory (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/dev-tools/memory-usage/)
- [Workers Logs (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/logs/workers-logs/)
- [Wrangler Workers commands (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/wrangler/commands/workers/)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/): Error 1101 means your Cloudflare Worker threw an uncaught JavaScript exception. Find the exception in logs, match it to its cause, fix it and roll back fast.
- [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/): Debug Cloudflare Workers errors step by step: read 1101 and 1102 codes, enable Workers Logs and source maps, use wrangler tail, DevTools and local traces.
- [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/): Fix Cloudflare Hyperdrive connection errors: config codes 2008 to 2016, pool exhaustion, connection_refused and stale clients reused across Worker requests.
- [How to build a memory system for proactive agents](https://getpraxen.app/blog/how-to-build-a-memory-system-for-proactive-agents/): Proactive agents run without a prompt, so each run has to start from what earlier runs learned. How we built memory for Praxen's agents on git: one repository per team, a nightly clean-up, and lessons from merged pull requests.
- [Nobody reads pull request descriptions, so we made videos](https://getpraxen.app/blog/nobody-reads-pull-request-descriptions-so-we-made-videos/): Almost none of the pull requests Praxen opens were asked for by an engineer, and a longer description does not make them easier to review. We are experimenting with a narrated video for every fix. Why, what it looks like, and what happened to merge rates.
---
### Kubernetes CrashLoopBackOff: causes and fixes
> What CrashLoopBackOff means in Kubernetes, how to read the exit code and events behind it, and the fix for each common cause, from bad config to failing probes.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 11 min read
Canonical: https://getpraxen.app/learn/troubleshooting/kubernetes-crashloopbackoff/
```text
CrashLoopBackOff
```
CrashLoopBackOff means a container in the pod keeps exiting and the kubelet is waiting before it restarts it again, with a delay that starts at 10 seconds and doubles up to 5 minutes. It is a symptom: the cause is in the container's last exit. Run kubectl logs --previous and kubectl describe pod, read the Last State reason and exit code, then fix the app error, config, memory limit, probe or command it points to.
You run `kubectl get pods` and one row says `CrashLoopBackOff`. READY is `0/1`, the RESTARTS count keeps climbing, and the pod never serves traffic. If a rollout started it, the rollout stalls while the old pods keep serving.
CrashLoopBackOff is a waiting state. It tells you the container has exited several times and the kubelet is holding off before the next restart. It doesn't tell you why the container exits. That answer sits in the previous container's logs, its exit code and the pod's events, and this page shows you how to read each one.
#### What CrashLoopBackOff means
The kubelet on the node prints it. When a container exits and the pod's `restartPolicy` allows a restart, the kubelet restarts it in the same pod on the same node. If it exits again, the kubelet applies an exponential back-off before each new attempt. The [Pod Lifecycle docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-restarts) give the defaults: 10s, 20s, 40s and so on, capped at 300 seconds (5 minutes). Once a container has run for 10 minutes without problems, the kubelet resets the timer.
While the timer runs, the container sits in the Waiting state with the reason `CrashLoopBackOff`, and that reason is what kubectl shows in STATUS. The pod's phase stays `Running`. The docs call STATUS "a kubectl display field for user intuition" and warn you not to confuse it with the phase.
You'll see the same state in three places. In the kubelet source, [`doBackOff`](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_manager.go) records a Warning event with the reason `BackOff`:
```text
Warning BackOff 12s (x8 over 2m) kubelet Back-off restarting failed container api in pod api-7d9c_default(3f1b...)
```
It also sets the container's waiting message to the current delay, which reads like this once the cap is reached:
```text
back-off 5m0s restarting failed container=api pod=api-7d9c_default(3f1b...)
```
And it returns the error `CrashLoopBackOff`, which is the reason you see in `kubectl describe pod` under `State: Waiting`.
```mermaid alt="Repeated container exits trigger Kubernetes restart back-off, reset after ten minutes of running"
graph TB
A[Container exits] --> B[Kubelet restarts it]
B --> C{Exits again?}
C -->|yes| D[Wait 10s, doubling to 300s]
D --> E[STATUS shows CrashLoopBackOff]
E --> B
C -->|runs 10 minutes| F[Back-off timer resets]
style E fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
The delay explains why a crash loop feels slow to debug. After a few failures, each new attempt is up to 5 minutes away, so a fix you apply by hand can look like it did nothing for a while.
##### Changing the back-off on newer clusters
Two kubelet features change these numbers. Neither fixes a crash, but they change how long you wait between attempts.
- **KubeletCrashLoopBackOffMax**, beta and enabled by default since v1.35. You set `maxContainerRestartPeriod` under `crashLoopBackOff` in the kubelet configuration, between `"1s"` and `"300s"`, per node. Delays still start at 10s and double, but stop at your cap. A cap under 10s becomes the starting delay too.
- **ReduceDefaultCrashLoopBackOffDecay**, alpha since v1.33 and off by default. With it on, restarts start at 1s and cap at 60s across the cluster. A node's own `maxContainerRestartPeriod` wins over these defaults.
```yaml
kind: KubeletConfiguration
crashLoopBackOff:
maxContainerRestartPeriod: "100s"
```
#### Reading the last exit before you change anything
Every cause below leaves a different trace. Collect three things first.
1. The logs of the container that crashed. Plain `kubectl logs` reads the current instance, which may be waiting and empty. `--previous` reads the one that died:
```bash
kubectl logs -c --previous
```
2. The container's last state and the pod's events:
```bash
kubectl describe pod
```
Look at `Last State: Terminated`, its `Reason` and `Exit Code`, then the Events table at the bottom.
3. The events on their own, if the describe output is long:
```bash
kubectl events --for pod/
kubectl get events --field-selector involvedObject.name=
```
The API server keeps events for one hour by default: the [kube-apiserver](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) flag `--event-ttl` defaults to `1h0m0s`. The Last State in the container status survives longer, so read it even when the events are gone.
The exit code narrows the search fast. The [Bash manual](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html) says a process killed by signal N reports 128 plus N, a command that isn't found reports 127, and one that isn't executable reports 126. So 137 means SIGKILL (9) and 143 means SIGTERM (15).
```mermaid alt="Pod exit reasons and probe failures narrow a crash loop to memory, application, entrypoint or health checks"
graph TB
A[kubectl describe pod] --> B{Last State reason?}
B -->|OOMKilled, 137| C[Memory limit too low]
B -->|Error, code 1| D[App exits: read previous logs]
B -->|Error, 127 or 126| E[Command or entrypoint wrong]
A --> F{Probe failed in Events?}
F -->|yes| G[Liveness or startup probe]
style C fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style G fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
```
#### Why the container keeps exiting
The Pod Lifecycle docs list the usual causes: application errors that make the container exit, configuration errors such as wrong environment variables or missing files, resource constraints, and liveness or startup probes that fail. Here they are with the trace each one leaves, most common first.
##### Cause 1: the application exits on startup
**How to tell it's yours:** `Last State` shows `Reason: Error` and a small non-zero exit code, often `1`. `kubectl logs --previous` ends with a stack trace, a panic or a message such as a failed database connection. The [Debug Running Pods guide](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#copying-a-pod-while-changing-its-command) shows exactly this shape:
```text
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
```
**Fix:**
1. Read the end of the previous logs. The last lines before the exit usually name the problem.
2. If the app exits before it logs anything, make Kubernetes keep its last output. With `terminationMessagePolicy: FallbackToLogsOnError`, the kubelet copies the last chunk of the log into the termination message when the container exits with an error, limited to 2048 bytes or 80 lines:
```yaml
containers:
- name: api
image: registry.example.com/api:1.4.2
terminationMessagePolicy: FallbackToLogsOnError
```
Then read it from the pod status:
```bash
kubectl get pod -o go-template='{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}'
```
3. If a deploy started the loop, roll back first and debug second:
```bash
kubectl rollout history deployment/api
kubectl rollout undo deployment/api
kubectl rollout status deployment/api
```
To go back to a specific revision, add `--to-revision=2`.
4. Reproduce it outside the cluster. The docs suggest running the same image locally, for example with `docker run --rm registry.example.com/api:1.4.2`, using the same environment variables the pod gets.
##### Cause 2: bad configuration, environment variables or secrets
**How to tell it's yours:** the same image runs fine elsewhere, and the previous logs complain about a missing variable, an unreadable file or a value that doesn't parse. Often the crash started right after a change to a ConfigMap, a Secret or the Deployment's `env`.
**Fix:**
1. Compare what the pod gets with what the app expects:
```bash
kubectl get pod -o jsonpath='{.spec.containers[0].env}'
kubectl get pod -o jsonpath='{.spec.containers[0].envFrom}'
kubectl describe configmap
```
2. If the app has no shell to inspect, start a copy of the pod with the command replaced by one, as the debug guide shows. The `--container` flag replaces that container's command:
```bash
kubectl debug -it --copy-to=-debug --container=api -- sh
```
Inside, run `env`, check the mounted files and start the app by hand to watch it fail.
3. Fix the ConfigMap, Secret or `env` entry, then restart the Deployment so new pods pick it up:
```bash
kubectl rollout restart deployment/api
```
4. Delete the debug copy when you're done: `kubectl delete pod -debug`.
##### Cause 3: the container runs out of memory
**How to tell it's yours:** `Last State` shows `Reason: OOMKilled` and `Exit Code: 137`. The previous logs often stop mid-line, because the kernel killed the process without warning.
The container used more memory than its limit, and the kernel killed it. Each restart climbs back to the same usage and dies again, which turns into a crash loop. The fix is either a higher limit or a smaller footprint: read `kubectl top pod --containers`, size requests and limits, and match Node.js or JVM heap flags to the limit.
##### Cause 4: a failing liveness or startup probe
**How to tell it's yours:** the logs look healthy, then the container stops. The Events show `Unhealthy` warnings followed by `Killing`. The [probes guide](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) shows the pattern:
```text
Warning Unhealthy 10s (x3 over 20s) kubelet Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Normal Killing 10s kubelet Container liveness failed liveness probe, will be restarted
```
When a liveness or startup probe fails, the kubelet kills the container and applies the restart policy. A slow-starting app with a tight liveness probe gets killed during startup, every time, and loops. Probe `timeoutSeconds` defaults to 1 second, which is short for an endpoint that touches a database.
**Fix:**
1. Check what the probe hits and whether the app answers in time:
```bash
kubectl get pod -o jsonpath='{.spec.containers[0].livenessProbe}'
```
2. Add a startup probe that covers the slowest start you expect. Liveness and readiness probes don't run until it succeeds. The docs give this example, which allows up to 30 × 10 = 300 seconds to start:
```yaml
ports:
- name: liveness-port
containerPort: 8080
livenessProbe:
httpGet:
path: /healthz
port: liveness-port
failureThreshold: 1
periodSeconds: 10
startupProbe:
httpGet:
path: /healthz
port: liveness-port
failureThreshold: 30
periodSeconds: 10
```
3. Keep the liveness endpoint cheap. It should answer whether this process can make progress. A check that calls a database or another service fails when that dependency is slow, and the kubelet restarts healthy containers as a result.
4. If the startup probe never succeeds, the container is killed after its window and the loop goes on. In that case the problem is the app, so go back to cause 1.
##### Cause 5: the wrong command or entrypoint
**How to tell it's yours:** the container exits at once with no app logs at all. With a shell entrypoint, the exit code is `127` (command not found) or `126` (found but not executable). If the binary named in `command` doesn't exist in the image, the process never starts: read the Last State message and the Events for the container runtime's error.
Common triggers are a `command` that overrides the image's `ENTRYPOINT` with a wrong path, a script without execute permission, and a multi-stage build that left the binary out of the final image.
**Fix:**
1. Compare the pod's `command` and `args` with the image:
```bash
kubectl get pod -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'
docker image inspect registry.example.com/api:1.4.2 --format '{{.Config.Entrypoint}} {{.Config.Cmd}}'
```
2. Open the image with a shell and look for the file:
```bash
kubectl debug -it --copy-to=-debug --container=api -- sh
ls -l /app
```
3. For an image with no shell, attach an ephemeral debug container. `--target` points it at the process namespace of the named container, and the runtime has to support it:
```bash
kubectl debug -it --image=busybox:1.28 --target=api
```
4. Fix the path or permissions in the manifest or Dockerfile (`RUN chmod +x /app/start.sh`), rebuild and redeploy.
#### Confirming the crash loop is over
A single successful start proves little, because the container might crash again a minute later. Watch it:
```bash
kubectl get pod -w
```
Expected output once it's fixed:
```text
NAME READY STATUS RESTARTS AGE
api-7d9c 1/1 Running 6 (12m ago) 31m
```
Two signs matter. READY shows `1/1`, so the readiness probe passes. And the RESTARTS count stays flat for more than 10 minutes. That is the point where the kubelet resets the back-off timer, so the container has now run long enough to count as healthy. For a Deployment, `kubectl rollout status deployment/api` should report that the rollout finished.
#### Stopping CrashLoopBackOff from coming back
- **Set a rollout deadline.** `.spec.progressDeadlineSeconds` defaults to 600. When a new version crash-loops, the Deployment gets a `Progressing` condition with reason `ProgressDeadlineExceeded`. Kubernetes only reports it, so alert on it or have your deploy tool roll back.
- **Give slow starters a startup probe** sized from a measured worst case, and keep liveness checks local to the process.
- **Validate config at startup and say what's missing.** An app that logs `DATABASE_URL is not set` before it exits turns a 20-minute hunt into a one-line fix.
- **Turn on `FallbackToLogsOnError`** for containers that can die before their logging is up.
- **Alert on restarts.** A container that restarts a few times an hour never shows CrashLoopBackOff for long, yet it is still failing. Alert on the restart count rising over a window, along with the status.
- **Ship events somewhere that keeps them.** With a one-hour default TTL, the evidence for an overnight crash loop is gone by morning.
The habit worth keeping is simple: look at the last exit, the last logs and the events together. With those three side by side, most crash loops take minutes.
#### Tracing a crash loop back to its cause with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Kubernetes, so when a pod starts crash looping it works out why, and the fix arrives ready for your team's normal review.
Running on Kubernetes? See [how Praxen monitors Kubernetes in production](https://getpraxen.app/for/kubernetes/).
#### Common questions
**How long does Kubernetes wait between restarts in CrashLoopBackOff?**
By default the kubelet waits 10 seconds after the first crash, then 20, then 40, doubling each time up to a cap of 300 seconds (5 minutes). Once a container has run for 10 minutes without problems, the kubelet resets the timer and the next crash counts as the first one.
**Can I shorten the CrashLoopBackOff delay?**
Yes, per node. The KubeletCrashLoopBackOffMax feature is beta and on by default since Kubernetes v1.35, so you can set crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration to anything from 1s to 300s. A separate alpha gate, ReduceDefaultCrashLoopBackOffDecay, changes the cluster defaults to a 1 second start and a 60 second cap, and it is off by default.
**Is CrashLoopBackOff a pod phase?**
No. It shows in the STATUS column of kubectl get pods and as the Waiting reason of a container, but the pod's phase is usually still Running. The Kubernetes docs warn against confusing that kubectl display field with the phase in the Pod API.
**Why does kubectl logs show nothing for my crashing pod?**
Plain kubectl logs reads the current container, which may be waiting in back-off and has printed nothing yet. Add --previous (or -p) to read the instance that crashed, and -c with the container name in a multi-container pod. If the app dies before it logs anything, set terminationMessagePolicy to FallbackToLogsOnError so the last 80 lines or 2048 bytes land in the pod status.
**What does exit code 137 mean in CrashLoopBackOff?**
137 is 128 plus 9, the number of SIGKILL. If the Last State reason is OOMKilled, the container went over its memory limit. If the reason is Error and the events show a failed liveness or startup probe, the kubelet killed it after the probe failed.
**My events are gone. How do I find out what happened?**
The API server keeps events for one hour by default (the kube-apiserver --event-ttl flag defaults to 1h0m0s). The container status still holds the last termination reason, exit code and message, so kubectl describe pod and kubectl get pod -o yaml still show Last State after the events expire.
**Should I just delete the pod?**
Deleting it makes the Deployment create a new pod with a fresh back-off timer, but the new container hits the same cause and loops again. Delete only after you've changed something: the image, the config, the limit or the probe.
**How do I roll back a deploy that started the crash loop?**
Run kubectl rollout history deployment/ to see the revisions, then kubectl rollout undo deployment/, or add --to-revision= for a specific one. Watch it with kubectl rollout status deployment/.
#### Sources
- [Pod Lifecycle (Kubernetes docs)](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)
- [kuberuntime_manager.go, doBackOff (Kubernetes source)](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_manager.go)
- [kubelet.go, crash loop back-off defaults (Kubernetes source)](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go)
- [Debug Running Pods (Kubernetes docs)](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/)
- [Configure Liveness, Readiness and Startup Probes (Kubernetes docs)](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
- [Determine the Reason for Pod Failure (Kubernetes docs)](https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/)
- [Deployments (Kubernetes docs)](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
- [kubectl events (Kubernetes docs)](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_events/)
- [kube-apiserver reference (Kubernetes docs)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/)
- [kubectl rollout restart (Kubernetes docs)](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/)
- [signal(7), Linux manual page](https://man7.org/linux/man-pages/man7/signal.7.html)
- [Exit Status (GNU Bash manual)](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html)
- [Kubernetes integration (Praxen docs)](https://docs.getpraxen.app/integrations/kubernetes)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
- [How to correlate incidents with deploys using evidence](https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/): Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
- [What is change intelligence? A production safety guide](https://getpraxen.app/learn/deployment-safety/what-is-change-intelligence/): What is change intelligence? Learn how deploys, config, flags, infrastructure and dependency changes connect to incidents, using tools you already have.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [ChangeMessageVisibility: troubleshoot SQS message timeout errors](https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/): ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
---
### Fix “remaining connection slots are reserved” in PostgreSQL
> Why Postgres refuses new logins with “remaining connection slots are reserved”, how to find who holds the connections, and how to pool and cap them for good.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 11 min read
Canonical: https://getpraxen.app/learn/troubleshooting/postgres-remaining-connection-slots-are-reserved/
```text
FATAL: remaining connection slots are reserved for non-replication superuser connections
```
Postgres has used up every connection slot except the few it keeps back for superusers, so it refuses new logins from ordinary roles with SQLSTATE 53300. Find which app and role hold the connections in pg_stat_activity, close the idle ones, then put a pooler such as Supabase's transaction mode on port 6543 or PgBouncer in front of the database and cap each client's pool. Raise max_connections or the compute size only once the connection count is under control.
This error usually arrives in a burst. A deploy goes out, traffic rises, or a batch of serverless functions wakes up, and suddenly every new request fails at login with `FATAL: remaining connection slots are reserved for non-replication superuser connections`. Requests that already hold a connection keep working. Anything that needs a new one gets turned away.
Postgres is telling you it is nearly full. It still has a handful of connection slots, but it keeps those for superusers so an administrator can always get in to fix things. Your app's role isn't a superuser, so it gets refused. The fix is to find who is holding the other slots, release the ones nobody is using, and stop your clients from asking for more than the database has.
#### What “remaining connection slots are reserved” means
Every client connection to Postgres takes one slot out of a fixed pool sized by `max_connections`. The [PostgreSQL docs](https://www.postgresql.org/docs/current/runtime-config-connection.html) say the default is typically 100, and the setting can only be changed at server start. A second setting, `superuser_reserved_connections`, keeps some of those slots back. Its default is 3. Once the number of active connections reaches `max_connections` minus `superuser_reserved_connections`, only superusers can connect.
The check lives in `InitPostgres()` in `src/backend/utils/init/postinit.c`. On [PostgreSQL 15 and earlier](https://github.com/postgres/postgres/blob/REL_15_STABLE/src/backend/utils/init/postinit.c), a regular backend for a role that isn't a superuser asks whether there are still `ReservedBackends` free slots. If not, it raises `FATAL` with SQLSTATE `53300`, which [errcodes.txt](https://github.com/postgres/postgres/blob/REL_16_STABLE/src/backend/utils/errcodes.txt) names `too_many_connections`.
PostgreSQL 16 split the reserve in two. The [release notes](https://www.postgresql.org/docs/16/release-16.html) add a `reserved_connections` setting for roles with the `pg_use_reserved_connections` role, and the [REL_16_STABLE source](https://github.com/postgres/postgres/blob/REL_16_STABLE/src/backend/utils/init/postinit.c) rewords the message. You now see one of these, still with SQLSTATE 53300:
```text
FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute
FATAL: remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections" role
```
The first appears when fewer free slots remain than `superuser_reserved_connections`. The second appears when the free slots sit inside the `reserved_connections` band and your role lacks `pg_use_reserved_connections`. `reserved_connections` defaults to 0, so on most PostgreSQL 16 and later servers you only ever see the SUPERUSER wording.
When the last slot goes too, everyone gets `sorry, too many clients already`, from the same SQLSTATE. So the ceiling for your app's roles is:
```text
usable slots = max_connections - superuser_reserved_connections - reserved_connections
```
On a default server that is 100 - 3 - 0 = 97.
```mermaid alt="Postgres rejects ordinary connections when only reserved slots remain; pooling and caps prevent exhaustion"
graph TB
A[App opens a new connection] --> B{Free slots above reserve?}
B -->|yes| C[Login succeeds]
B -->|no| D{Role is a superuser?}
D -->|yes| C
D -->|no| E[FATAL with SQLSTATE 53300]
E --> F[Find who holds the slots]
F --> G[Pool and cap client connections]
style E fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
#### Why the slots ran out: who holds the connections
Before you change anything, connect as a superuser (the reserve exists for exactly this) and count the connections by role, application and state. `pg_stat_activity` has one row per server process. The [statistics docs](https://www.postgresql.org/docs/current/monitoring-stats.html) say `backend_type` is `client backend` for ordinary client sessions, and `state` is one of `active`, `idle`, `idle in transaction`, `idle in transaction (aborted)`, `fastpath function call` or `disabled`.
```sql
SELECT usename,
application_name,
client_addr,
state,
count(*) AS connections,
max(now() - state_change) AS longest_in_state
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY usename, application_name, client_addr, state
ORDER BY connections DESC;
```
Then compare the total with the limit:
```sql
SELECT current_setting('max_connections')::int AS max_connections,
current_setting('superuser_reserved_connections')::int AS superuser_reserved,
current_setting('reserved_connections')::int AS reserved, -- PostgreSQL 16+
(SELECT count(*) FROM pg_stat_activity
WHERE backend_type = 'client backend') AS client_connections;
```
On PostgreSQL 15 and earlier, drop the `reserved_connections` line, because the setting doesn't exist there.
On Supabase, the role tells you which service owns a connection. The [connection management guide](https://supabase.com/docs/guides/database/connection-management) maps them: `authenticator` is the Data API (PostgREST), `supabase_auth_admin` is Auth, `supabase_storage_admin` is Storage, `supabase_admin` is Supabase monitoring and Realtime, and `postgres` or your own roles are the Dashboard and external tools such as Prisma, SQLAlchemy and psql.
The result usually points at one of four causes.
#### Serverless functions connecting straight to Postgres
**How to tell it's yours:** a large count for one role, many distinct `client_addr` values, short connection lifetimes, and the error lines up with traffic spikes or a burst of function invocations.
Each function instance opens its own connection, or its own pool, and many instances run at once. Supabase's [connection guide](https://supabase.com/docs/guides/database/connecting-to-postgres) recommends the shared pooler in transaction mode for serverless and edge functions, because they open many short-lived connections. Transaction mode hands a server connection to a client only for the length of a transaction, so many clients share a few Postgres slots.
**Fix: send serverless traffic through a transaction-mode pooler.**
1. Copy the transaction pooler string from the Supabase Dashboard. It uses port 6543 and a pooler host you can't build from the region alone:
```text
postgresql://postgres.[PROJECT-REF]:[YOUR-PASSWORD]@[POOLER-HOST]:6543/postgres
```
2. Create the client once at module scope, outside the request handler, so warm invocations reuse it.
3. Set the pool to 1 connection per function instance.
4. Turn off prepared statements, which transaction mode doesn't support. Supabase lists `prepare: false` for Postgres.js and Drizzle, `pgbouncer=true` for Prisma, `statement_cache_size=0` for asyncpg and `prepareThreshold=0` for JDBC.
5. Set SSL to `require`.
For Postgres.js, that looks like:
```ts
import postgres from "postgres";
// Module scope: one client per function instance.
const sql = postgres(process.env.DATABASE_URL!, {
max: 1,
prepare: false,
ssl: "require",
});
export default async function handler() {
const rows = await sql`select now()`;
return Response.json(rows);
}
```
Keep migrations off the transaction pooler. Supabase says to run them over the direct connection, because they are single sessions using native Postgres commands.
Outside Supabase, PgBouncer does the same job. Its [configuration reference](https://www.pgbouncer.org/config.html) defaults `pool_mode` to `session`, which only returns a server connection to the pool after the client disconnects. Set it to `transaction` for serverless clients:
```ini
[databases]
app = host=10.0.0.5 port=5432 dbname=app
[pgbouncer]
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20
```
`default_pool_size` (default 20) is the most server connections PgBouncer opens per user and database pair. `max_client_conn` (default 100) is how many clients it accepts. Keep the sum of server-side pools below your usable slots.
#### Idle and leaked connections
**How to tell it's yours:** most rows have `state = 'idle'` or `'idle in transaction'`, and `longest_in_state` runs to minutes or hours. Often one `application_name` dominates.
An `idle` session is waiting for its next command. An app that opens connections and never closes them, or a pool with no idle timeout, fills the slots this way. `idle in transaction` is worse. The [client defaults docs](https://www.postgresql.org/docs/current/runtime-config-client.html) say an open transaction also stops vacuum from removing recently dead rows, which leads to table bloat.
**Fix: release them now, then stop them building up.**
1. List the worst offenders:
```sql
SELECT pid, usename, application_name, state, now() - state_change AS for_how_long
FROM pg_stat_activity
WHERE backend_type = 'client backend'
AND state IN ('idle', 'idle in transaction')
ORDER BY state_change
LIMIT 50;
```
2. End the idle sessions of the app that leaks. `pg_terminate_backend` sends SIGTERM to the backend. The [admin functions docs](https://www.postgresql.org/docs/current/functions-admin.html) say you need to be a member of the target's role or have `pg_signal_backend`, and only superusers can end superuser sessions:
```sql
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE backend_type = 'client backend'
AND state = 'idle'
AND application_name = 'my-api'
AND state_change < now() - interval '10 minutes';
```
3. Set `idle_in_transaction_session_timeout` for the app's role. It ends any session that stays idle inside an open transaction for longer than the value, and its default of 0 disables it:
```sql
ALTER ROLE app_user SET idle_in_transaction_session_timeout = '60s';
```
New sessions for that role pick up the setting. Existing ones keep the old value until they reconnect.
4. Fix the leak in code. Release every connection in a `finally` block, or use the pool's query helper, which checks a connection out and back in for you.
`idle_session_timeout` also exists, for sessions idle outside a transaction. The Postgres docs warn against using it on connections that come through a pooler or other middleware, which may not react well when the server closes a connection without warning. Apply it per role for interactive users, if at all.
#### Pool size multiplied by replicas and services
**How to tell it's yours:** the count per `client_addr` is steady and equals your pool size, and the total grew when you scaled out, added a worker, or deployed a new service on the same database.
Every app instance keeps its own pool. The total is pool size times instances, summed across every service. The [Prisma connection pool docs](https://www.prisma.io/docs/orm/v7/prisma-client/setup-and-configuration/databases-connections/connection-pool) give two defaults worth knowing. Prisma ORM 6 used `num_cpus::get_physical() * 2 + 1` connections per instance. The Prisma 7 `pg` driver adapter uses `max: 10`. Ten replicas on Prisma 7 can ask for 100 connections before anything else connects.
**Fix: budget the slots and cap each pool.**
1. Write down every client: app replicas, workers, cron jobs, migrations, BI tools and admin sessions.
2. Divide the usable slots between them, with headroom. On Supabase, the guide says to stay under 40% of the database's max connections for the pooler if you use the Data API heavily, and up to 80% otherwise, to leave room for Auth and other services.
3. Set each pool's maximum explicitly. With the Prisma 7 `pg` adapter:
```ts
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "./generated/prisma/client";
const adapter = new PrismaPg({
connectionString: process.env.DATABASE_URL,
max: 5,
idleTimeoutMillis: 10_000,
});
export const prisma = new PrismaClient({ adapter });
```
4. Recheck the budget whenever autoscaling limits change. The number that matters is the maximum replica count.
#### max_connections set too low for the workload
**How to tell it's yours:** the connections are all legitimate and mostly `active`, pools are already capped, a pooler is already in place, and you still hit the limit at peak.
On Supabase, `max_connections` follows the compute size. The [compute docs](https://supabase.com/docs/guides/platform/compute-and-disk) list 60 for Nano and Micro, 90 for Small, 120 for Medium, 160 for Large and 240 for XL, rising to 500 on the largest sizes. The [Supabase troubleshooting entry](https://supabase.com/docs/guides/troubleshooting/database-error-remaining-connection-slots-are-reserved-for-non-replication-superuser-connections-3V3nIb) for this error says that if you already pool and still hit the limit, upgrade the compute add-on.
**Fix: raise the ceiling, knowing what it costs.**
1. On Supabase, upgrade the compute size in the Dashboard, or set the value directly with the CLI. The [config guide](https://supabase.com/docs/guides/database/custom-postgres-config) marks `max_connections` as CLI only, and changing it restarts the database:
```bash
supabase postgres-config update --project-ref --experimental \
--config max_connections=200
```
2. On self-managed Postgres, edit `postgresql.conf` and restart:
```ini
max_connections = 200
superuser_reserved_connections = 3
```
3. On a standby, set `max_connections` to the same value or higher than on the primary. The Postgres docs say queries aren't allowed on the standby otherwise.
PostgreSQL sizes some resources, including shared memory, directly from `max_connections`. A bigger limit costs memory on every server, so treat it as the last lever.
#### Confirming the database has headroom again
Run the count query again during normal traffic and at peak. The fix holds when:
```sql
SELECT count(*) AS client_connections,
current_setting('max_connections')::int
- current_setting('superuser_reserved_connections')::int
- current_setting('reserved_connections')::int AS usable_slots
FROM pg_stat_activity
WHERE backend_type = 'client backend';
```
`client_connections` stays clearly below `usable_slots`, and your logs show no new `FATAL` lines with SQLSTATE 53300. Redeploy or scale out once on purpose and watch the count. It should rise by your per-instance pool size and no more.
#### Stopping the slots filling up again
- **Alert on connection pressure.** Alert well before client connections reach the usable slots. Supabase's Grafana dashboard has a Client Connections graph for both Supavisor and Postgres, and the Dashboard's Database client connections chart (Teams and Enterprise) breaks them down by Postgres, PostgREST, Reserved, Auth, Storage and other roles.
- **Alert on the log line.** Match `remaining connection slots are reserved` and `too many clients` in your Postgres logs. Either one means users are already seeing errors.
- **Cap every pool in code.** Never rely on a driver's default, which differs between libraries and versions.
- **Put a budget in the deploy checklist.** Max replicas times pool size, summed across services, must fit inside the usable slots.
- **Keep the idle-in-transaction timeout on** for application roles.
- **Leave the superuser reserve alone.** It is what lets you log in and run the queries above during an incident.
#### Tracing a full Supabase connection pool with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Supabase, so when the connection pool fills up it works out why, and the fix arrives ready for your team's normal review.
Running on Supabase? See [how Praxen monitors Supabase in production](https://getpraxen.app/for/supabase/).
#### Common questions
**Why does PostgreSQL 16 show a different message for the same problem?**
PostgreSQL 16 added the reserved_connections setting and the pg_use_reserved_connections role, and reworded the check. When fewer free slots remain than superuser_reserved_connections, it prints “remaining connection slots are reserved for roles with the SUPERUSER attribute”. When the free slots fall inside the reserved_connections band and your role lacks pg_use_reserved_connections, it prints “reserved for roles with privileges of the pg_use_reserved_connections role”. Both carry SQLSTATE 53300.
**What is the difference between this error and “sorry, too many clients already”?**
Both use SQLSTATE 53300 (too_many_connections). “remaining connection slots are reserved” means a few slots are still free but held back for superusers or reserved roles. “sorry, too many clients already” means no slot is left at all, so even a superuser can't log in. Same cause, one step further along.
**How many connections can a non-superuser actually open?**
max_connections minus superuser_reserved_connections minus reserved_connections. With the defaults of 100, 3 and 0, that is 97. On Supabase, max_connections follows the compute size, for example 60 on Nano and Micro, 90 on Small and 120 on Medium.
**Should I raise max_connections to fix it?**
Only after you have found where the connections come from. PostgreSQL sizes some resources, including shared memory, directly from max_connections, and it can only be changed at server start. On Supabase it is set with supabase postgres-config update --config max_connections=200 and needs a restart, so a leak will fill the bigger limit too.
**Which Supabase connection string should a serverless function use?**
The shared pooler in transaction mode, on port 6543. Supabase recommends creating the client once at module scope, setting the pool to 1 connection, turning off prepared statements and setting SSL to require. With Prisma, add pgbouncer=true to that URL.
**Is it safe to terminate idle connections with pg_terminate_backend?**
Terminating a session in the idle state drops a connection that is waiting for its next command, so the client sees a closed connection and its pool should open a new one. Avoid ending sessions that are active or idle in transaction unless you accept that their open transaction is rolled back. You need superuser or pg_signal_backend to end another role's session.
**What pool size should I give each app instance?**
Work back from the limit. Divide the non-superuser slots by the number of instances that can run at once, then leave room for migrations and admin tools. Prisma ORM 6 defaulted to num_physical_cpus * 2 + 1 per instance and the Prisma 7 pg adapter defaults to 10, so ten instances can ask for 100 connections.
#### Sources
- [postinit.c, REL_15_STABLE (PostgreSQL source)](https://github.com/postgres/postgres/blob/REL_15_STABLE/src/backend/utils/init/postinit.c)
- [postinit.c, REL_16_STABLE (PostgreSQL source)](https://github.com/postgres/postgres/blob/REL_16_STABLE/src/backend/utils/init/postinit.c)
- [errcodes.txt (PostgreSQL source)](https://github.com/postgres/postgres/blob/REL_16_STABLE/src/backend/utils/errcodes.txt)
- [Connections and Authentication (PostgreSQL docs)](https://www.postgresql.org/docs/current/runtime-config-connection.html)
- [Client Connection Defaults (PostgreSQL docs)](https://www.postgresql.org/docs/current/runtime-config-client.html)
- [System Administration Functions (PostgreSQL docs)](https://www.postgresql.org/docs/current/functions-admin.html)
- [The Cumulative Statistics System (PostgreSQL docs)](https://www.postgresql.org/docs/current/monitoring-stats.html)
- [PostgreSQL 16 release notes](https://www.postgresql.org/docs/16/release-16.html)
- [Database error: remaining connection slots are reserved (Supabase Docs)](https://supabase.com/docs/guides/troubleshooting/database-error-remaining-connection-slots-are-reserved-for-non-replication-superuser-connections-3V3nIb)
- [Connect to your database (Supabase Docs)](https://supabase.com/docs/guides/database/connecting-to-postgres)
- [Connection management (Supabase Docs)](https://supabase.com/docs/guides/database/connection-management)
- [Compute and Disk (Supabase Docs)](https://supabase.com/docs/guides/platform/compute-and-disk)
- [Customizing Postgres configs (Supabase Docs)](https://supabase.com/docs/guides/database/custom-postgres-config)
- [ALTER ROLE (PostgreSQL docs)](https://www.postgresql.org/docs/current/sql-alterrole.html)
- [PgBouncer configuration](https://www.pgbouncer.org/config.html)
- [Connection pool (Prisma ORM v7 docs)](https://www.prisma.io/docs/orm/v7/prisma-client/setup-and-configuration/databases-connections/connection-pool)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/): Monitor a Supabase app in production: scrape the Metrics API into Prometheus, drain logs, run SQL health checks, trace requests and alert on what matters.
- [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/): Fix Cloudflare Hyperdrive connection errors: config codes 2008 to 2016, pool exhaustion, connection_refused and stale clients reused across Worker requests.
- [Fix Prisma “P1001: Can't reach database server”](https://getpraxen.app/learn/troubleshooting/prisma-p1001-cant-reach-database-server/): Why Prisma prints “P1001: Can't reach database server”, how to test the host and port it names, and how to fix private hosts, IPv6, paused databases and URLs.
- [Supabase vs Convex in production: reliability and cost](https://getpraxen.app/learn/reliability/supabase-vs-convex-in-production/): Supabase vs Convex in production: compare data models, realtime, consistency, scaling, migrations, monitoring, backups and costs before choosing your backend.
- [How to monitor a vibe-coded app in production](https://getpraxen.app/learn/monitoring-coverage/vibe-coding-production-monitoring/): Monitor a vibe-coded app in production: surface swallowed errors, add a health route, tag deploys, watch cron jobs and alert on what users feel first.
---
### Fix Prisma “P1001: Can't reach database server”
> Why Prisma prints “P1001: Can't reach database server”, how to test the host and port it names, and how to fix private hosts, IPv6, paused databases and URLs.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 11 min read
Canonical: https://getpraxen.app/learn/troubleshooting/prisma-p1001-cant-reach-database-server/
```text
Error: P1001: Can't reach database server at `localhost:5432`
```
Prisma tried to open a network connection to the host and port in your connection URL and got no usable answer: the name didn't resolve, the port refused the connection, or the connect timed out. Test the same host and port with psql or nc from the same machine, then fix whatever blocks it, most often a private hostname used outside its network or at build time, an IPv6-only database host, a paused database or a password that isn't percent-encoded.
You usually meet P1001 in one of two places. The first is a Prisma CLI command such as `prisma migrate deploy`, `prisma migrate dev` or `prisma db push`, often inside a CI job or a platform build. The second is your running app, where Prisma Client throws it on the first query after start-up. Either way, the host and port in the message are the ones Prisma read from your connection URL.
The message says nothing about credentials, schemas or migrations. Prisma never got that far. It tried to open a network connection to that address and failed, so the whole job is to find out why that address can't be reached from where the command ran.
#### What “P1001: Can't reach database server” means
P1001 is one of Prisma's common connection errors. Its template lives in [`libs/user-facing-errors/src/common.rs`](https://github.com/prisma/prisma-engines/blob/main/libs/user-facing-errors/src/common.rs) in prisma-engines:
```text
Can't reach database server at `{database_location}`
Please make sure your database server is running at `{database_location}`.
```
The CLI prints it with the code in front, so you see ``Error: P1001: Can't reach database server at `host:port` ``. The exact quoting depends on your Prisma version. Releases from before a [May 2024 change](https://github.com/prisma/prisma-engines/commit/e9771e62de) print `` `host`:`port` ``, with two pairs of backticks. Later releases print `` `host:port` ``. The [v7 error reference](https://www.prisma.io/docs/orm/v7/reference/error-reference) still shows the older form.
In the Rust engine, two kinds of failure become P1001 for PostgreSQL, MySQL and SQL Server: a connection error, and a connect timeout. That second one surprises people. A firewall that silently drops packets produces the same P1001 as a host that doesn't exist. The related P1002 means something different: the server was reached, and then timed out.
#### Which Prisma version printed it
Prisma's behaviour around this error has changed across three major versions, so check which one you run before reading logs.
**Prisma ORM 6 and earlier.** The Rust query engine and the schema engine open the connection. Both raise P1001 for refused connections, unresolved names and connect timeouts. The [Prisma 7 connection pool docs](https://www.prisma.io/docs/orm/v7/prisma-client/setup-and-configuration/databases-connections/connection-pool) give the version 6 default `connect_timeout` as 5 seconds.
**Prisma ORM 7.** Prisma Client now uses driver adapters by default, so the Node.js driver opens the connection. Since a [July 2025 change](https://github.com/prisma/prisma-engines/commit/7982122277), driver adapters can report P1001 too. The [`@prisma/adapter-pg` error mapping](https://github.com/prisma/prisma/blob/7.9.x/packages/adapter-pg/src/errors.ts) on the 7.9.x branch sorts the pg driver's socket errors like this:
- `ENOTFOUND` and `ECONNREFUSED` become P1001, `Can't reach database server at host:port`.
- `ETIMEDOUT` becomes P1008, `Operation has timed out`.
- `ECONNRESET` becomes P1017, `Server has closed the connection`.
- TLS certificate errors become P1011.
- Anything else is thrown as the raw driver error.
So with Prisma 7, a refused port or a bad hostname still says P1001 at runtime. A dropped packet shows up as P1008, or as a raw pg error. The pg adapter's `connectionTimeoutMillis` defaults to 0, which the docs describe as no timeout, so a blocked port can hang for a long time first. Prisma CLI commands in version 7 read the URL from `datasource.url` in `prisma.config.ts` and still print P1001.
**Prisma ORM 8.** Prisma's docs now default to Prisma ORM 8. Its [error reference](https://www.prisma.io/docs/orm/reference/error-reference) uses dotted codes in place of P-codes. A failed connection is `DRIVER.CONNECTION_FAILED`, raised when the database refused the connection, rejected the credentials, or did not answer within a 20 second connect timeout. Prisma ORM 7 remains supported, with its docs under `/orm/v7`. The rest of this page covers P1001 on versions 6 and 7.
#### Testing the host and port Prisma tried
Run every check from the machine, container or CI job that failed. A check from your laptop proves nothing about a build container.
```bash
# 1. What URL is the command actually using? (prints host and port only)
node -e 'const u = new URL(process.env.DATABASE_URL); console.log(u.hostname, u.port || 5432)'
# 2. Does the name resolve, and to which address family?
dig +short A db.example.internal
dig +short AAAA db.example.internal
# 3. Is the port open?
nc -vz db.example.internal 5432
# 4. Can a real Postgres client log in?
psql "$DATABASE_URL" -c 'select 1'
```
If `psql` works from the same place, the network path is fine, and the problem is in how Prisma gets its URL: a different environment variable, a stale `.env`, or the wrong value in `prisma.config.ts`. If `psql` fails too, the output points at one of the causes below.
```mermaid alt="Diagnosing Prisma P1001 through DNS resolution, port reachability and a direct database login"
graph TB
A[P1001 from Prisma] --> B{Host name resolves?}
B -->|no| C[Private or misspelt hostname]
B -->|yes| D{nc reaches the port?}
D -->|refused| E[Database down or wrong port]
D -->|times out| F[Firewall, allowlist or IPv6]
D -->|yes| G{psql logs in?}
G -->|yes| H[Prisma reads a different URL]
G -->|no| I[Check URL encoding]
style A fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style H fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
#### Cause 1: a private hostname used outside its network
**How to tell it's yours:** the host in the message is an internal name such as `postgres.railway.internal`, and the error appears in build logs, in CI, or on your laptop. `dig` returns nothing from there.
Railway gives every service an internal DNS name under `railway.internal`. The [private networking docs](https://docs.railway.com/networking/private-networking/how-it-works) say private networking is only available at runtime. The build phase has no access to it, so build scripts can't reach other services over it, and migrations that need internal connectivity should run as part of the start command. In environments created before October 16, 2025, those names resolve to IPv6 addresses only. Newer environments get both IPv4 and IPv6.
**Fix: run migrations where the private network exists, and use the public URL everywhere else.**
1. Take `prisma migrate deploy` out of the build command.
2. Add it as a Railway pre-deploy command. The [pre-deploy docs](https://docs.railway.com/deployments/pre-deploy-command) say these run between build and deploy, inside your private network, with your service's variables:
```bash
npx prisma migrate deploy
```
If the command fails, it isn't retried and the deployment doesn't go ahead, which is what you want for a migration.
3. Or run it at the start of the start command:
```bash
npx prisma migrate deploy && node dist/server.js
```
4. For connections from outside Railway, such as CI or your laptop, open the database's **Settings, Networking** and add **Public Access**. The [PostgreSQL docs](https://docs.railway.com/databases/postgresql) say this creates a TCP Proxy and fills a `DATABASE_PUBLIC_URL` variable. Traffic through the proxy is billed as network egress.
```bash
DATABASE_URL="$DATABASE_PUBLIC_URL" npx prisma migrate status
```
#### Cause 2: an IPv6-only database host on an IPv4 network
**How to tell it's yours:** the host is `db.[PROJECT-REF].supabase.co`, `dig AAAA` returns an address, `dig A` returns nothing, and the machine you run from has no IPv6 route.
Supabase's [connection guide](https://supabase.com/docs/guides/database/connecting-to-postgres) says the direct connection is IPv6 only on Free and paid plans. Paid plans with the IPv4 add-on get IPv4 only, because the [add-on](https://supabase.com/docs/guides/platform/ipv4-address) swaps the AAAA record for an A record. Both shared pooler modes support IPv4 on all plans.
**Fix: pick the Supabase string that fits the job and the network.**
1. For Prisma Client in your app, use the transaction pooler on port 6543. It's built for many short-lived connections, and it doesn't support prepared statements, so add `pgbouncer=true`:
```bash
DATABASE_URL="postgresql://postgres.[PROJECT-REF]:[PASSWORD]@[POOLER-HOST]:6543/postgres?pgbouncer=true"
```
2. For Prisma CLI commands, Supabase recommends the direct connection. From an IPv4-only network, use the session pooler on port 5432 of the pooler host:
```bash
DIRECT_URL="postgresql://postgres.[PROJECT-REF]:[PASSWORD]@[POOLER-HOST]:5432/postgres"
```
Copy `[POOLER-HOST]` from the Dashboard. Supabase says it can't be built from the region alone.
3. In Prisma 7, point the CLI at that URL in `prisma.config.ts`, as the [PgBouncer guide](https://www.prisma.io/docs/orm/v7/prisma-client/setup-and-configuration/databases-connections/pgbouncer) shows:
```ts
import "dotenv/config";
import { defineConfig, env } from "prisma/config";
export default defineConfig({
schema: "prisma/schema.prisma",
datasource: {
url: env("DIRECT_URL"),
},
});
```
4. If you need the direct host over IPv4, enable the IPv4 add-on.
#### Cause 3: a firewall or IP allowlist dropping the connection
**How to tell it's yours:** the name resolves, but `nc -vz host 5432` hangs and then times out. The same command works from another network.
Managed databases often filter by source IP. Supabase's [network restrictions](https://supabase.com/docs/guides/platform/network-restrictions) control which IP ranges can connect to Postgres and its pooler, and they are enforced before traffic reaches the database. Cloud security groups and corporate firewalls behave the same way. Dropped packets give you no refusal, so Prisma waits for its connect timeout and then reports P1001.
**Fix:**
1. Find the egress IP of the failing machine:
```bash
curl -s https://ifconfig.me
```
2. Add that IP or range to the database's allowlist, or run the command from a network that's already allowed.
3. While you debug, give slow paths more time. On Prisma 6 and earlier, add `connect_timeout` to the URL. The [PostgreSQL connector docs](https://www.prisma.io/docs/orm/v7/core-concepts/supported-databases/postgresql) list its default as 5 seconds, with 0 meaning no timeout:
```bash
DATABASE_URL="postgresql://user:pass@db.example.com:5432/app?connect_timeout=30"
```
On Prisma 7 with the pg adapter, set it on the adapter:
```ts
const adapter = new PrismaPg({
connectionString: process.env.DATABASE_URL,
connectionTimeoutMillis: 30_000,
});
```
A longer timeout only helps a slow path. If the firewall drops everything, a longer wait still ends in P1001.
#### Cause 4: the database is paused, sleeping or down
When evaluating a replacement backend, [Supabase vs Convex in production](/learn/reliability/supabase-vs-convex-in-production/) separates database availability, function cold starts and recovery requirements.
**How to tell it's yours:** the connection worked before, nothing in the URL changed, and `nc` is refused or times out. The provider's dashboard shows the database as paused, stopped or restarting.
Supabase [pauses Free Plan projects](https://supabase.com/docs/guides/platform/free-project-pausing) that show low activity over a 7-day period. It sends a warning email about a week before the pause, and you can restore a paused project for up to 1 year. On Railway, a service with [Serverless](https://docs.railway.com/deployments/serverless) enabled sleeps once it stops sending outbound traffic, between 5 and 10 minutes after the last packet. The first request wakes it, and that request may fail.
**Fix:**
1. On Supabase, open the Dashboard, select the organisation and the paused project, then click **Resume project**. Paid plans aren't paused, so upgrade if the database must stay up.
2. On Railway, turn Serverless off for the database service and redeploy, because the setting only applies to new containers.
3. Self-hosted, check the server and the port it listens on:
```bash
pg_isready -h db.example.com -p 5432
```
If it can't get an answer, check that Postgres is running and that the URL's port matches the one the server listens on.
#### Cause 5: special characters in the password
**How to tell it's yours:** the host in the P1001 message isn't your database host. It looks like part of your password followed by the real host, or the port is wrong.
The [connection URL docs](https://www.prisma.io/docs/orm/v7/reference/connection-urls) say you must percent-encode special characters in any part of a PostgreSQL URL, passwords included. Their example turns `p@$$w0rd` into `p%40%24%24w0rd`. An unencoded `@`, `/`, `:` or `#` moves where the URL parser thinks the host begins.
**Fix:**
1. Encode the password:
```bash
node -e 'console.log(encodeURIComponent(process.argv[1]))' 'p@$$w0rd'
```
```text
p%40%24%24w0rd
```
2. Put the encoded value in the URL and run the test from the start of this page again.
3. Or rotate the password to one made of letters and digits.
#### Confirming Prisma can reach the database
From the same machine, container or CI step that failed:
```bash
npx prisma migrate status
```
It should connect and list your migrations, with no P1001. Then deploy, and search the app and deploy logs for `P1001` and `Can't reach database server` for a while. On Prisma 7, also look for `P1008` and raw `ECONNREFUSED` or `ETIMEDOUT` errors, since some failures surface under those.
#### Stopping P1001 from coming back
- **Keep two URLs.** The app uses the pooled or private URL. Migrations and CI use the direct or public one, set in `prisma.config.ts`.
- **Run migrations at release time.** Use a Railway pre-deploy command or your platform's release step, never the build.
- **Set a connect timeout on purpose.** On the Prisma 7 pg adapter the default is no timeout, so a blocked port hangs until the OS gives up.
- **Check the URL in CI.** Fail the job if `DATABASE_URL` points at a `.railway.internal` host or contains an unencoded `@` in the password.
- **Keep production off the Free Plan pause.** A paused Supabase project takes every query down with it.
#### Spotting P1001 in Railway and Supabase logs with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Railway and Supabase, so when P1001 errors start it works out why, and the fix arrives ready for your team's normal review.
Running on Railway? See [how Praxen monitors Railway in production](https://getpraxen.app/for/railway/).
#### Common questions
**What is the difference between P1001 and P1002?**
P1001 means Prisma couldn't reach the server at all: the name didn't resolve, the connection was refused, or the connect timed out. P1002 means the server was reached but then timed out. A P1001 points at the network path or the address, while a P1002 points at a slow or overloaded database.
**Does Prisma 7 still throw P1001, or a raw ECONNREFUSED?**
Prisma Client 7 with @prisma/adapter-pg maps the pg driver's ENOTFOUND and ECONNREFUSED errors to P1001, with the message Can't reach database server at host:port. ETIMEDOUT becomes P1008, ECONNRESET becomes P1017 and TLS certificate errors become P1011. Any other connection error, such as pg's own connect timeout, is thrown as the raw driver error.
**Why does prisma migrate deploy fail with P1001 during a Railway build?**
Railway's private network, the *.railway.internal hostnames, is only available at runtime, and the build can't reach it. Move prisma migrate deploy out of the build into a pre-deploy command, which runs inside the private network, or into the start command.
**Which Supabase connection string should Prisma migrations use?**
Supabase recommends the direct connection for migrations, but the direct host db.[PROJECT-REF].supabase.co is IPv6 only unless you buy the IPv4 add-on. From an IPv4-only network, use the session pooler on port 5432 of the pooler host, which supports IPv4 on all plans. Keep the transaction pooler on port 6543, with pgbouncer=true, for Prisma Client.
**How do I give Prisma more time to connect?**
On Prisma ORM 6 and earlier, add connect_timeout to the URL, for example ?connect_timeout=30. The default is 5 seconds and 0 means no timeout. On Prisma 7 with the pg adapter, set connectionTimeoutMillis on the adapter, whose default is 0 (no timeout).
**My password has an @ in it. Is that the problem?**
It can be. Prisma's docs say you must percent-encode special characters in any part of a PostgreSQL URL, passwords included. p@$$w0rd becomes p%40%24%24w0rd. An unencoded @ makes the URL parser read part of the password as the host.
**What replaces P1001 in Prisma ORM 8?**
Prisma's docs now default to Prisma ORM 8, which uses dotted codes such as DRIVER.CONNECTION_FAILED in place of P-codes. That code covers a refused connection, rejected credentials or no answer within its 20 second connect timeout. Prisma ORM 7 stays supported, with its docs under /orm/v7 and the CLI installed as prisma@prev.
#### Sources
- [user-facing-errors common.rs (prisma-engines source)](https://github.com/prisma/prisma-engines/blob/main/libs/user-facing-errors/src/common.rs)
- [Add driver adapter error mappings for P1001, P1011, P1017 (prisma-engines #5556)](https://github.com/prisma/prisma-engines/commit/7982122277)
- [Fix highlighting of injected info bits in error messages (prisma-engines #4368)](https://github.com/prisma/prisma-engines/commit/e9771e62de)
- [adapter-pg errors.ts, branch 7.9.x (Prisma source)](https://github.com/prisma/prisma/blob/7.9.x/packages/adapter-pg/src/errors.ts)
- [client-engine-runtime user-facing-error.ts, branch 7.9.x (Prisma source)](https://github.com/prisma/prisma/blob/7.9.x/packages/client-engine-runtime/src/user-facing-error.ts)
- [Error reference, Prisma ORM v7 (Prisma Docs)](https://www.prisma.io/docs/orm/v7/reference/error-reference)
- [Error reference, Prisma ORM 8 (Prisma Docs)](https://www.prisma.io/docs/orm/reference/error-reference)
- [Connection URLs, Prisma ORM v7 (Prisma Docs)](https://www.prisma.io/docs/orm/v7/reference/connection-urls)
- [PostgreSQL, Prisma ORM v7 (Prisma Docs)](https://www.prisma.io/docs/orm/v7/core-concepts/supported-databases/postgresql)
- [Connection pool, Prisma ORM v7 (Prisma Docs)](https://www.prisma.io/docs/orm/v7/prisma-client/setup-and-configuration/databases-connections/connection-pool)
- [Configure Prisma Client with PgBouncer, Prisma ORM v7 (Prisma Docs)](https://www.prisma.io/docs/orm/v7/prisma-client/setup-and-configuration/databases-connections/pgbouncer)
- [How private networking works (Railway Docs)](https://docs.railway.com/networking/private-networking/how-it-works)
- [PostgreSQL (Railway Docs)](https://docs.railway.com/databases/postgresql)
- [Pre-deploy command (Railway Docs)](https://docs.railway.com/deployments/pre-deploy-command)
- [Serverless (Railway Docs)](https://docs.railway.com/deployments/serverless)
- [Connect to your database (Supabase Docs)](https://supabase.com/docs/guides/database/connecting-to-postgres)
- [IPv4 address (Supabase Docs)](https://supabase.com/docs/guides/platform/ipv4-address)
- [Network restrictions (Supabase Docs)](https://supabase.com/docs/guides/platform/network-restrictions)
- [Project pausing (Supabase Docs)](https://supabase.com/docs/guides/platform/free-project-pausing)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [Fix “remaining connection slots are reserved” in PostgreSQL](https://getpraxen.app/learn/troubleshooting/postgres-remaining-connection-slots-are-reserved/): Why Postgres refuses new logins with “remaining connection slots are reserved”, how to find who holds the connections, and how to pool and cap them for good.
- [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/): Monitor a Supabase app in production: scrape the Metrics API into Prometheus, drain logs, run SQL health checks, trace requests and alert on what matters.
- [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/): Set up health checks, JSON logs, OpenTelemetry traces, monitors and crash alerts for a FastAPI service on Railway, with the commands and config to copy.
- [Fix Railway “Application failed to respond” (502)](https://getpraxen.app/learn/troubleshooting/railway-application-failed-to-respond/): Why Railway returns “Application failed to respond” with a 502, how to tell a wrong host, port or target port from a crash or overload, and how to fix each.
- [Supabase vs Convex in production: reliability and cost](https://getpraxen.app/learn/reliability/supabase-vs-convex-in-production/): Supabase vs Convex in production: compare data models, realtime, consistency, scaling, migrations, monitoring, backups and costs before choosing your backend.
---
### Fix Railway “Application failed to respond” (502)
> Why Railway returns “Application failed to respond” with a 502, how to tell a wrong host, port or target port from a crash or overload, and how to fix each.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 12 min read
Canonical: https://getpraxen.app/learn/troubleshooting/railway-application-failed-to-respond/
```text
Application failed to respond
```
Railway's edge proxy shows this page with a 502 Bad Gateway when it can't reach your app. Most often the server isn't listening on host 0.0.0.0 and the port in the PORT variable, or the domain's target port points at a different port. Bind to 0.0.0.0:$PORT, make the target port match, and check the deploy logs and Metrics tab for a crash or a CPU ceiling.
You open your Railway domain and get a page titled “Application failed to respond” where your app should be. The browser's network tab shows a 502. Sometimes it happens on the very first deploy of a new service. Sometimes a service that worked yesterday starts doing it after a change to the start command, a new framework version, or a spike in traffic.
The page comes from Railway, and your app never saw the request. The fix is to work out why Railway couldn't hand the request over: wrong host, wrong port, nothing running, or too busy to answer.
#### What “Application failed to respond” means on Railway
Every public request to a Railway service goes through Railway's edge proxy first. The proxy works out which service and which internal port the domain points at, and passes the request to your app on that port. According to [Railway's troubleshooting page](https://docs.railway.com/networking/troubleshooting/application-failed-to-respond), this error means the edge proxy can't communicate with your application, so the request fails with a 502 Bad Gateway status code.
Railway lists three reasons, in this order of frequency:
1. Your app isn't listening on the correct host or port. This is the most common one.
2. The domain's target port is set to a port your app doesn't listen on.
3. Far less often, the app is under heavy load and can't respond.
There is a fourth case the docs imply: if the process isn't running at all, because it crashed or hasn't started yet, there is nothing for the proxy to talk to either.
```mermaid alt="Railway returns 502 when the target port has no reachable listener or the app fails to answer in time"
graph TB
A[Request hits Railway edge] --> B{Domain has a target port?}
B -->|yes| C[Proxy dials the target port]
B -->|no| D[Proxy dials the app port]
C --> E{App listening on 0.0.0.0 there?}
D --> E
E -->|no| F[502 Application failed to respond]
E -->|yes| G{App answers in time?}
G -->|no| F
G -->|yes| H[Your app returns a response]
style F fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style H fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
#### Find the failing requests first
Before changing code, confirm you're looking at this error and see which requests hit it. Railway records HTTP logs for every request at the edge, and the [logs docs](https://docs.railway.com/observability/logs) list an `@responseDetails` attribute that is only populated when the application fails to respond.
From the CLI, with the project linked:
```bash
railway logs --http --status 502 --lines 50
railway logs --http --status 500..599 --since 1h
```
In the dashboard's Log Explorer, the same filter is:
```text
@httpStatus:502
```
Then read the deploy logs for the active deployment. They capture your app's stdout and stderr, so they show the line where your server says which address and port it bound to:
```bash
railway logs --deployment --lines 100
railway logs --latest
```
`--latest` shows the newest deployment even if it failed or is still building. With the HTTP logs and the listen line side by side, match your case to one of the causes below.
#### Cause 1: the server listens on localhost or a hardcoded port
**How to tell it's yours:** the deploy logs show the server started, but on `localhost`, `127.0.0.1`, or a fixed port such as `3000` that doesn't match `PORT`. Typical lines:
```text
Server listening on http://localhost:3000
Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
```
Railway injects a `PORT` environment variable into each deployment, and its docs say your web server should bind to host `0.0.0.0` and listen on that port. The [vibe-coded app guide](https://docs.railway.com/guides/vibe-coding-deploy) spells out the two traps. AI tools often hardcode the port, as in `app.listen(3000)`. And binding to `localhost` or `127.0.0.1` makes the app unreachable from outside the container. A loopback address only accepts connections from inside the same container, and the edge proxy is outside it.
You can check what Railway set by printing the variable from inside the service:
```bash
railway ssh -- printenv PORT
```
**Fix: read PORT and bind to 0.0.0.0.** The code differs by framework. The idea doesn't.
1. **Node with Express.** Pass the host as the second argument to `listen`:
```javascript
const express = require("express");
const app = express();
const port = process.env.PORT || 3000;
app.listen(port, "0.0.0.0", () => {
console.log(`Listening on 0.0.0.0:${port}`);
});
```
NestJS takes the same two arguments: `await app.listen(port, "0.0.0.0")`.
2. **Next.js.** `next start` needs the port flag, as Railway's page shows:
```bash
next start --port ${PORT-3000}
```
3. **Python with uvicorn.** Railway's page says uvicorn needs extra flags to listen on `0.0.0.0` and `PORT`, so pass both in the start command:
```bash
uvicorn main:app --host 0.0.0.0 --port $PORT
```
4. **Python with gunicorn.** Railway's docs say gunicorn listens on `0.0.0.0` and `PORT` by default, so `gunicorn main:app` works as is. If you pass your own bind, keep it on all interfaces:
```bash
gunicorn main:app --bind 0.0.0.0:$PORT
```
5. **Go with net/http.** Read the variable, fall back when it's empty, and give the full address:
```go
package main
import (
"log"
"net/http"
"os"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
})
port := os.Getenv("PORT")
if port == "" {
port = "3000"
}
log.Printf("listening on 0.0.0.0:%s", port)
log.Fatal(http.ListenAndServe("0.0.0.0:"+port, mux))
}
```
6. If the start command lives in the service settings or in `railway.json`, update it there too. A config file setting overrides the dashboard:
```json
{
"$schema": "https://railway.com/railway.schema.json",
"deploy": {
"startCommand": "uvicorn main:app --host 0.0.0.0 --port $PORT"
}
}
```
7. Redeploy and check that the new listen line shows `0.0.0.0` and the value of `PORT`.
#### Cause 2: the domain's target port doesn't match the app
**How to tell it's yours:** the app binds to `0.0.0.0` on some port, the deploy logs prove it, and the domain still returns 502. Open the service, go to **Settings**, then **Networking**, and look at the port shown next to the domain.
Railway's [domains docs](https://docs.railway.com/networking/domains/working-with-domains) call these target ports. They map one domain to one internal port your app listens on, which is how a single service can expose several HTTP ports through several domains. When you generate a Railway domain and the app listens on a single port, Railway detects it and sets it as the target port. When you add a custom domain, you pick from a list or type a custom port.
That detection runs once, so the setting can go stale. Railway's own example: a domain configured for port `3000` while the app was listening on `8080`. That happens when you change the port in code or in the start command after the domain was created, or when the app was listening on a temporary port at the moment you generated the domain.
**Fix: point the domain at the port the app uses.**
1. Read the real port from the deploy logs, for example `Listening on 0.0.0.0:8080`.
2. In **Settings > Networking**, click the edit icon next to the domain and set that port.
3. Repeat for every domain on the service. Custom domains and the Railway domain each have their own target port.
4. If the app deliberately listens on a port other than `PORT`, also set a `PORT` service variable to that number. The [healthcheck docs](https://docs.railway.com/deployments/healthchecks) say Railway uses `PORT` for health checks, and leaving it unset with target ports can make the health check return `service unavailable`:
```bash
railway variable set PORT=8080
```
The simplest setup is one port everywhere: the app reads `PORT`, and the target port equals that value.
#### Cause 3: the app crashed or hasn't started listening yet
**How to tell it's yours:** the deployment shows as `Crashed` in the service's deployment list, or the deploy logs end in a stack trace, a missing environment variable, or a database connection error before any listen line. Another sign is 502s that cluster in the first seconds or minutes after each deploy, then stop.
The [deployments reference](https://docs.railway.com/deployments/reference) explains why the second pattern happens. Without a healthcheck, Railway marks a deployment `Active` as soon as the container starts. If your app spends time running migrations or warming a cache before it calls `listen`, requests routed to it in that window find nothing on the port. With a healthcheck, Railway waits for the endpoint to return a 2xx before it makes the new deployment active.
A crash is easier to spot. A deployment stays `Active` until it crashes, then becomes `Crashed`. The [restart policy](https://docs.railway.com/deployments/restart-policy) defaults to `On Failure` with a maximum of 10 restarts. Once those are used up, the service stays down, and every request gets the 502 page. A process that exits with code 0 shows as `Completed`, which `On Failure` doesn't restart.
```bash
railway logs --latest --lines 200
railway logs --deployment --filter "@level:error" --lines 50
```
**Fix: make the app start reliably and tell Railway when it's ready.**
1. Fix the error at the end of the deploy logs. On Railway the usual culprits are variables that only exist in your local `.env` and connection strings that point at `localhost`. Railway's guide says databases run as separate services, so reference their connection URL as a variable.
2. Add a health endpoint that returns 200 once the app is ready, and set it as the healthcheck path:
```json
{
"$schema": "https://railway.com/railway.schema.json",
"deploy": {
"healthcheckPath": "/health",
"healthcheckTimeout": 300
}
}
```
The default timeout is 300 seconds. If the app doesn't return a 2xx within it, Railway marks the deploy as failed and keeps the previous one serving traffic.
3. If your app restricts hosts, allow `healthcheck.railway.app`. Railway sends health checks from that hostname, and rejecting it shows up as `failed with service unavailable` or `failed with status 400`.
4. Move slow one-off work such as migrations into a pre-deploy command, so the container that serves traffic starts listening straight away:
```json
{
"$schema": "https://railway.com/railway.schema.json",
"deploy": {
"preDeployCommand": ["npm run db:migrate"],
"startCommand": "node server.js"
}
}
```
5. For a service that must always be up, change the restart policy to `Always` in the service settings. It isn't available on the Free plan or during a trial.
One related case: if Serverless is on, Railway puts the service to sleep after at least 5 minutes with no outbound traffic. The [Serverless docs](https://docs.railway.com/deployments/serverless) say the first request to a slept service may return a 502 Bad Gateway while it wakes. If that matches your pattern, turn Serverless off for that service.
#### Cause 4: the app is too busy to answer
**How to tell it's yours:** host, port and target port are all correct, most requests succeed, and the 502s come in bursts during traffic peaks. Slow responses show up in the HTTP logs before the failures do:
```bash
railway logs --http --filter "@responseTime:>=1000" --lines 50
railway metrics --cpu --memory --since 6h
railway metrics --http --since 6h
```
`railway metrics --http` reports request totals, status-code buckets, error rate and latency percentiles. Railway's troubleshooting page gives one concrete marker for Node.js: vCPU usage peaking at around 1 vCPU in the Metrics tab is a good sign of heavy load, because Node runs your JavaScript on a single thread.
Keep in mind how replicas show up. The [scaling docs](https://docs.railway.com/deployments/scaling) say metrics from all replicas are summed, so two replicas each using 100 MB appear as 200 MB.
**Fix: spread the load.**
1. Add replicas. In the service settings, raise the replica count in the Scale section. Railway distributes public traffic randomly across the replicas in a region, and each replica gets the full resources of your plan. In config:
```json
{
"$schema": "https://railway.com/railway.schema.json",
"deploy": {
"multiRegionConfig": {
"us-west2": { "numReplicas": 2 }
}
}
}
```
2. Use the cores you already have. A single Node process runs your JavaScript on one thread, so run several processes or workers per replica if your server supports it.
3. Find the hot path. Filter the HTTP logs by `@path` and `@responseTime` to see which routes are slow, and fix the query or call behind them.
4. Remember that Railway doesn't support sticky sessions. Any session state has to live in a shared store before you scale out.
#### Confirming the 502 is gone
After the redeploy, check three things:
1. The deploy logs show your server's listen line with `0.0.0.0` and the port Railway gave it:
```text
Listening on 0.0.0.0:8080
```
2. The public domain answers with a 200:
```bash
curl -sS -o /dev/null -w "%{http_code}\n" https://your-app.up.railway.app/
```
Expected output:
```text
200
```
3. The HTTP logs have no new 502s:
```bash
railway logs --http --status 502 --since 15m
```
#### Stopping “Application failed to respond” from coming back
```mermaid alt="A Railway deploy binds to PORT and passes a health check before going active, with crash alerts afterward"
graph TB
A[Deploy starts] --> B[App reads PORT]
B --> C[App binds 0.0.0.0 on PORT]
C --> D{Healthcheck returns 2xx?}
D -->|no| E[Deploy fails, old one serves]
D -->|yes| F[New deploy goes Active]
F --> G[Webhook alerts on Crashed]
style E fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
- **Never hardcode the port or the host.** Read `PORT` in code or in the start command, and keep the domain's target port equal to it.
- **Keep a healthcheck path on every web service.** A deploy that never listens then fails before it takes traffic.
- **Alert on crashes.** Railway's [alerting guide](https://docs.railway.com/guides/alerts-crashes-failed-deploys) shows how a project webhook can post `Failed` and `Crashed` deployment events to Slack or Discord with no middleware. Monitors on CPU and RAM thresholds need the Pro plan.
- **Watch for 502s from outside.** Railway only calls the healthcheck while a deploy goes live, never after. An uptime check on the public domain, or a saved `@httpStatus:502` filter in the Log Explorer, catches a service that falls over later.
- **Check replicas before launches.** If Metrics shows CPU near its ceiling at normal traffic, add replicas before the next spike.
#### Catching Railway 502s with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Railway, so when a service starts returning 502s it works out why, and the fix arrives ready for your team's normal review.
Running on Railway? See [how Praxen monitors Railway in production](https://getpraxen.app/for/railway/).
#### Common questions
**What port does Railway expect my app to listen on?**
Railway injects a PORT environment variable into every deployment, and your server should listen on that value on host 0.0.0.0. Read it at startup with process.env.PORT in Node, $PORT in a uvicorn start command, or os.Getenv("PORT") in Go. Don't hardcode 3000 or 8080 unless the domain's target port is set to the same number.
**Why does Railway say “Application failed to respond” when my app works locally?**
Locally your browser and server share one machine, so a server bound to localhost or 127.0.0.1 answers. On Railway the edge proxy connects from outside the container, so it needs a listener on 0.0.0.0. Railway's docs say binding to localhost makes the app unreachable from outside the container.
**Where do I change the target port of a Railway domain?**
Open the service, go to Settings, then Networking, and click the edit icon next to the domain. Railway sets the target port automatically when it detects a single listening port, and you can change it at any time. It must equal the port the deploy logs show your server listening on.
**How do I find which requests got the 502?**
Run railway logs --http --status 502 from the CLI, or filter the HTTP logs in the dashboard with @httpStatus:502. The @responseDetails attribute is only populated when the application fails to respond, so it marks these requests. Add @path or @host to narrow it to one route or domain.
**Can high CPU cause “Application failed to respond”?**
Yes, though Railway's docs call it far less common than a host or port problem. For a Node.js app, the docs say vCPU usage peaking around 1 vCPU in the Metrics tab is a good sign of heavy load, because Node runs JavaScript on one thread. Add replicas in the service settings to spread requests across more instances.
**Does a healthcheck stop this error after a deploy?**
It stops the window where traffic reaches a deploy that isn't listening yet. With a healthcheck path set, Railway only marks a deployment Active after the endpoint returns a 2xx, and the default timeout is 300 seconds. Railway doesn't call the healthcheck again once the deploy is live, so it won't catch a crash later.
**Why do I get a 502 on the first request after my service was idle?**
If Serverless is enabled, Railway puts the service to sleep after at least 5 minutes with no outbound traffic. Railway's docs say the first request to a slept service may return a 502 Bad Gateway while it wakes up. Turn Serverless off for services that must answer the first request.
#### Sources
- [Application Failed to Respond (Railway Docs)](https://docs.railway.com/networking/troubleshooting/application-failed-to-respond)
- [Working with Domains: target ports (Railway Docs)](https://docs.railway.com/networking/domains/working-with-domains)
- [Healthchecks (Railway Docs)](https://docs.railway.com/deployments/healthchecks)
- [Deploy a Vibe-Coded App on Railway (Railway Docs)](https://docs.railway.com/guides/vibe-coding-deploy)
- [Logs (Railway Docs)](https://docs.railway.com/observability/logs)
- [railway logs (Railway CLI Docs)](https://docs.railway.com/cli/logs)
- [railway metrics (Railway CLI Docs)](https://docs.railway.com/cli/metrics)
- [Scaling (Railway Docs)](https://docs.railway.com/deployments/scaling)
- [Deployments reference (Railway Docs)](https://docs.railway.com/deployments/reference)
- [Restart Policy (Railway Docs)](https://docs.railway.com/deployments/restart-policy)
- [Serverless (Railway Docs)](https://docs.railway.com/deployments/serverless)
- [railway ssh (Railway CLI Docs)](https://docs.railway.com/cli/ssh)
- [Config as code reference (Railway Docs)](https://docs.railway.com/config-as-code/reference)
- [Set Up Alerts for Crashes, Restarts, and Failed Deploys (Railway Docs)](https://docs.railway.com/guides/alerts-crashes-failed-deploys)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/): Set up health checks, JSON logs, OpenTelemetry traces, monitors and crash alerts for a FastAPI service on Railway, with the commands and config to copy.
- [Fix Render “Port scan timeout reached, no open ports detected”](https://getpraxen.app/learn/troubleshooting/render-port-scan-timeout-reached-no-open-ports-detected/): Why a Render deploy fails with “Port scan timeout reached, no open ports detected”, how to spot the cause in the logs, and how to bind to 0.0.0.0 on PORT.
- [Fix Prisma “P1001: Can't reach database server”](https://getpraxen.app/learn/troubleshooting/prisma-p1001-cant-reach-database-server/): Why Prisma prints “P1001: Can't reach database server”, how to test the host and port it names, and how to fix private hosts, IPv6, paused databases and URLs.
- [Fix Vercel “FUNCTION_INVOCATION_FAILED” 500 errors](https://getpraxen.app/learn/troubleshooting/vercel-function-invocation-failed/): Why a Vercel Function returns 500 FUNCTION_INVOCATION_FAILED, how to find the crash in runtime logs, and how to fix throws, missing env vars, files and memory.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
---
### Fix Render “Port scan timeout reached, no open ports detected”
> Why a Render deploy fails with “Port scan timeout reached, no open ports detected”, how to spot the cause in the logs, and how to bind to 0.0.0.0 on PORT.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 11 min read
Canonical: https://getpraxen.app/learn/troubleshooting/render-port-scan-timeout-reached-no-open-ports-detected/
```text
Port scan timeout reached, no open ports detected. Bind your service to at least one port. If you don't need to receive traffic on any port, create a background worker instead.
```
Render's deploy waits for your web service to open a port on host 0.0.0.0, and fails the deploy when it never sees one. Usually the server is bound to localhost or 127.0.0.1, ignores the PORT variable (10000 by default), or crashes or hangs before it listens. Bind to 0.0.0.0:$PORT, or deploy code that takes no traffic as a background worker.
The build log is green, the start command runs, and then the deploy just waits. Lines like `==> No open ports detected, continuing to scan...` repeat in the log. Eventually the deploy gives up with “Port scan timeout reached, no open ports detected” and is marked as failed. If an earlier deploy succeeded, it keeps serving traffic. If this is the first deploy, the service never goes live.
Render is telling you it started your process and never saw it accept connections. The message also suggests a way out: if the service doesn't need incoming traffic, it shouldn't be a web service at all.
#### What “no open ports detected” means on Render
A Render web service is reachable at its `onrender.com` subdomain, but your process isn't exposed to the internet directly. Render's load balancer receives each request and forwards it to your service on one port. The [web services docs](https://render.com/docs/web-services#port-binding) set the contract: every web service must bind to a port on host `0.0.0.0` to serve HTTP requests, and the default expected port is `10000`. If Render fails to detect a bound port, the deploy fails and shows an error in the logs.
Render's [boot and port binding tutorial](https://render.com/tutorials/when-deploys-go-wrong/boot-and-port-binding) describes how a deploy goes live. The build succeeds, Render runs your start command, and a port scanner waits for a TCP listener on `0.0.0.0`. Only when it finds one does Render move on to health checks and route traffic to the new deploy. While it waits, it logs the “continuing to scan” line. When it stops waiting, the deploy fails with the message above.
```mermaid alt="Render scans for a listener on 0.0.0.0 before health checks; a timeout fails the deploy"
graph TB
A[Build succeeds] --> B[Render runs the start command]
B --> C{Listener on 0.0.0.0?}
C -->|not yet| D[No open ports, keep scanning]
D --> C
D -->|scan gives up| E[Port scan timeout, deploy fails]
C -->|yes| F[Detected service running on port]
F --> G[Health checks, then live]
style E fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
The tutorial also explains why a server that works on your laptop can fail here. Render's port scanner runs in a separate network namespace from your process. A server listening only on a loopback address is invisible to it.
#### Read the runtime log before changing anything
Open the failed deploy in the dashboard and read the lines between `==> Running ''` and the timeout. You can also stream them with the [Render CLI](https://render.com/docs/cli-reference):
```bash
render logs --resources srv-abc123 --tail
```
Look for three things:
1. **Does your app print a listen line at all?** No output after the start command points at a crash, a hang or a Docker image with nothing to run.
2. **Which address does it name?** `localhost`, `127.0.0.1` or `[::1]` is a loopback bind.
3. **Which port does it name?** Compare it with `PORT`, which is `10000` unless you've set it yourself.
The tutorial gives a quick way to read the listen line:
- `Server listening on http://localhost:3000`: loopback and a port other than `PORT`, so both are wrong.
- `Server listening on http://0.0.0.0:3000`: right interface, port other than `PORT`.
- `Server listening on http://localhost:10000`: right port, loopback interface.
Then match your case to one of the causes below. They're ordered by how often people hit them.
#### Cause 1: the server binds to localhost or 127.0.0.1
**How to tell it's yours:** your app logs a listen line, and the address in it is `localhost`, `127.0.0.1` or the IPv6 loopback `[::1]`. A typical log:
```text
==> Running 'npm start'
Server listening on http://localhost:3000
==> No open ports detected, continuing to scan...
==> No open ports detected, continuing to scan...
```
When Render can see the port on loopback, the timeout line names it. Deploy logs posted in GitHub issues show this form:
```text
Port scan timeout reached, no open ports detected on 0.0.0.0. Detected open ports on localhost -- did you mean to bind one of these to 0.0.0.0?
```
Render's tutorial names the usual sources: the Rails default, some Node servers in dev mode, and some Go HTTP libraries. It calls out an IPv6 version too: Puma can print `Listening on tcp://[::1]:3000`, which the scanner doesn't see.
**Fix: bind to 0.0.0.0 on PORT.** Use the idiom for your server:
1. **Node with Express.** Read `PORT` and pass the host:
```javascript
const express = require("express");
const app = express();
const port = process.env.PORT || 10000;
app.listen(port, "0.0.0.0", () => {
console.log(`Listening on 0.0.0.0:${port}`);
});
```
2. **Python with gunicorn.** On Render's native Python runtime, the [default environment variables](https://render.com/docs/environment-variables) include `GUNICORN_CMD_ARGS` set to `--preload --access-logfile - --bind=0.0.0.0:10000`. A plain start command such as `gunicorn myapp.wsgi` binds correctly. If you set the bind yourself, in a flag or in `gunicorn.conf.py`, keep it on all interfaces:
```bash
gunicorn myproject.wsgi -b 0.0.0.0:$PORT
```
3. **Python with uvicorn.** Render's [FastAPI guide](https://render.com/docs/deploy-fastapi) uses this start command:
```bash
uvicorn main:app --host 0.0.0.0 --port $PORT
```
4. **Flask's built-in server.** Pass the host and port explicitly:
```python
import os
port = int(os.environ.get("PORT", 10000))
app.run(host="0.0.0.0", port=port)
```
5. **Rails with Puma.** Bind in the start command:
```bash
bundle exec rails server -b 0.0.0.0 -p $PORT
```
6. **Go with net/http.** Give the full address, with the host:
```go
port := os.Getenv("PORT")
if port == "" {
port = "10000"
}
log.Fatal(http.ListenAndServe("0.0.0.0:"+port, nil))
```
Before pushing, run the same start command locally with `PORT=10000` and check the listen line names `0.0.0.0:10000`.
#### Cause 2: the server ignores PORT or picks a reserved port
**How to tell it's yours:** the listen line shows `0.0.0.0`, but the port is hardcoded, or it's one of Render's reserved ports.
The web services docs recommend binding your HTTP server to the port in `PORT`. Its default value is `10000` for all web services, and you can override it in the Render Dashboard. Render says that if you bind to a different port, it is usually able to detect and use it. Usually isn't always, and the tutorial's examples show a hardcoded `3000` among the failures. Binding to `PORT` removes the guesswork.
Three ports are reserved by Render and can't be used: `18012`, `18013` and `19099`.
Watch out for a service that opens several listeners. Render forwards public traffic to only one HTTP port per web service. The tutorial shows the log for that case:
```text
==> Detected service running on port 10000
==> Detected service running on port 8080
```
That isn't a failure on its own, but make sure the public HTTP server is the one on `PORT`.
**Fix: let PORT drive the port.**
1. Remove hardcoded ports from code and start commands. Read `PORT` instead, as in the snippets above.
2. If you need a specific port, set the `PORT` environment variable on the service to that number. Render's environment variables reference describes `PORT` as the port your HTTP server binds to.
3. Move any listener off `18012`, `18013` and `19099`.
4. Keep extra listeners, such as a metrics or admin endpoint, on other ports, and keep the public server on `PORT`.
#### Cause 3: the app crashes or hangs before it listens
**How to tell it's yours:** after `==> Running '...'` the log shows a stack trace, a missing environment variable, or a database error, then no listen line. Or the log shows output from something that never finishes, such as a migration or a console.
The tutorial gives two examples of a start command that never reaches the server:
```text
==> Running 'rails console'
Loading production environment (Rails 7.1.0)
==> No open ports detected, continuing to scan...
```
```text
==> Running 'rails db:migrate && rails server'
==> Migrating database
==> No open ports detected, continuing to scan...
```
In the first, the start command is an interactive console that waits for input forever. In the second, the server only starts after the migration, so a slow migration holds the port closed.
**Fix: start the server, and only the server.**
1. Fix the error at the end of the log. Render's [troubleshooting guide](https://render.com/docs/troubleshooting-deploys) lists the usual ones: missing environment variables, an invalid start command, and missing modules.
2. Move one-off work into the pre-deploy command. The [deploys docs](https://render.com/docs/deploys) say it runs after the build and before the start command, on a separate instance from your running service. In a Blueprint:
```yaml
services:
- type: web
name: api
runtime: ruby
buildCommand: bundle install
preDeployCommand: bundle exec rails db:migrate
startCommand: bundle exec rails server -b 0.0.0.0 -p $PORT
```
The pre-deploy command is available for paid web services, private services and background workers.
3. Make the start command a long-running server. A REPL, a one-off script or `date` will never open a port.
#### Cause 4: a worker is deployed as a web service
**How to tell it's yours:** the code is a queue consumer, a bot, a scheduler or another loop that never serves HTTP. It runs fine, logs its own work, and the deploy still fails the port scan.
The message names this case itself: if you don't need to receive traffic on any port, create a background worker instead. Render's [background workers docs](https://render.com/docs/background-workers) describe them as services that run continuously, like a web service or a private service, but don't receive any incoming network traffic. They usually poll a task queue, with Celery, Sidekiq, BullMQ, Asynq or similar.
The [private services docs](https://render.com/docs/private-services) draw the line between the two internal types:
- If the service binds to at least one port and receives private network traffic, create a private service. It isn't reachable from the public internet and gets no `onrender.com` subdomain.
- Otherwise, create a background worker. Workers can make network requests but can't receive them.
```mermaid alt="Choose a Render web service, private service or background worker based on incoming connections"
graph TB
A[Does it accept connections?] -->|no| B[Background worker]
A -->|yes| C{From the public internet?}
C -->|yes| D[Web service on 0.0.0.0 PORT]
C -->|no| E[Private service]
style B fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style D fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
**Fix: deploy it as the right service type.**
1. Create a new **Background Worker** in the dashboard from the same repo and branch, with the same build and start commands and environment variables.
2. Or declare it in your Blueprint with `type: worker`:
```yaml
services:
- type: worker
name: queue-worker
runtime: python
buildCommand: pip install -r requirements.txt
startCommand: celery -A tasks worker --loglevel=info
```
3. If other services call it over the private network, use `type: pserv` and bind it to at least one port.
4. Once the new service is healthy, delete the old web service so it stops failing deploys.
#### Cause 5: the Docker image has nothing to run
**How to tell it's yours:** the service uses the Docker runtime, the image builds, and then there's no application output at all.
The troubleshooting guide says a Dockerfile must include a `CMD` or `ENTRYPOINT` directive, because Render uses one of them to run your app. Without both, the deploy might appear to hang indefinitely. The [Docker docs](https://render.com/docs/docker) also describe a **Docker Command** field under **Advanced** that replaces the Dockerfile's `CMD`. A stale or wrong value there runs the wrong thing, or nothing that listens.
**Fix: give the container a server to start.**
1. Add a `CMD` that starts the server on `0.0.0.0` and `PORT`. The exec form passes signals to your app:
```dockerfile
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["node", "server.js"]
```
2. Check the **Docker Command** field in the service settings. Leave it empty to use the Dockerfile's `CMD`, or set the full command. For several commands, Render's docs pass them to `/bin/sh -c`:
```bash
/bin/sh -c python manage.py migrate && gunicorn myapp.wsgi:application --bind 0.0.0.0:10000
```
3. In a Blueprint, the same setting is `dockerCommand`, which defaults to the Dockerfile's `CMD`.
#### Confirming Render found the port
Redeploy and read the log again. Render's tutorial shows the lines you want after the fix:
```text
==> Detected service running on port 10000
==> Your service is live 🎉
```
Then call the service from outside:
```bash
curl -sS -o /dev/null -w "%{http_code}\n" https://your-service.onrender.com/
```
A `200` (or whatever your root route returns) means the load balancer can reach your process. If the deploy now finds the port but still fails, the problem has moved to health checks. Render's [health checks docs](https://render.com/docs/health-checks) say a default TCP check passes when the instance accepts the connection within five seconds, and an HTTP check passes on a 2xx or 3xx response within five seconds.
#### Stopping the port scan from failing again
- **Bind to `0.0.0.0:$PORT` everywhere.** Put it in the code or the start command, never in a local-only config file.
- **Test the exact start command locally** with `PORT=10000`, and check the listen line before you push.
- **Keep the start command to one long-running server.** Migrations go in the pre-deploy command.
- **Pick the service type on purpose.** Web services for public HTTP, private services for internal traffic, background workers for everything that only makes outbound calls.
- **Turn on deploy failure notifications.** Render's [notifications](https://render.com/docs/notifications) can email you or post to Slack when a build or deploy fails, so a failed port scan on an auto-deploy doesn't go unnoticed.
#### Finding the failed port scan with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Render, so when a deploy fails its port scan it works out why, and the fix arrives ready for your team's normal review.
Running on Render? See [how Praxen monitors Render in production](https://getpraxen.app/for/render/).
#### Common questions
**What port does Render expect a web service to use?**
Render sets PORT to 10000 for every web service by default, and you can override it as an environment variable in the dashboard. Your server should read PORT and bind to host 0.0.0.0. Render is usually able to detect a different port, but binding to PORT is the documented recommendation.
**Why does my app say it's listening but Render finds no open port?**
It is almost always listening on a loopback address such as localhost, 127.0.0.1 or [::1]. Render's tutorial explains that its port scanner runs in a separate network namespace from your process, so a loopback listener is invisible to it. Change the bind address to 0.0.0.0 and redeploy.
**Which ports can't I use on Render?**
Render reserves ports 18012, 18013 and 19099, and they can't be used. A service can have at most 75 open ports. Render forwards public traffic to only one HTTP port per web service, so bind the public server to PORT and use any other ports for private network traffic.
**My service doesn't serve HTTP. How do I stop this error?**
Deploy it as a different service type. Render's docs say a background worker runs continuously but receives no incoming network traffic, which suits queue consumers such as Celery, Sidekiq or BullMQ. If other services need to call it over the private network, create a private service, which must still bind to at least one port.
**Does gunicorn on Render bind to 0.0.0.0 by default?**
On Render's native Python runtime, Render sets GUNICORN_CMD_ARGS to --preload --access-logfile - --bind=0.0.0.0:10000, so a plain gunicorn start command binds correctly. If you pass your own --bind, make it 0.0.0.0:$PORT. A bind to 127.0.0.1 in a config file or flag leads straight to this error.
**Can a health check cause “no open ports detected”?**
No. The port scan happens before health checks matter, and Render's tutorial shows the scanner waiting for a TCP listener first. Once a port is found, a failing HTTP health check is a different problem. Render counts an HTTP check as passing when it gets a 2xx or 3xx response within five seconds.
**How do I run migrations without blocking the port scan?**
Move them into the pre-deploy command, which runs after the build and before the start command on a separate instance. The start command should then only start the server. The pre-deploy command is available for paid web services, private services and background workers.
#### Sources
- [Web Services: port binding (Render Docs)](https://render.com/docs/web-services)
- [Background Workers (Render Docs)](https://render.com/docs/background-workers)
- [Private Services (Render Docs)](https://render.com/docs/private-services)
- [Private Network: port restrictions (Render Docs)](https://render.com/docs/private-network)
- [Default Environment Variables (Render Docs)](https://render.com/docs/environment-variables)
- [Docker on Render (Render Docs)](https://render.com/docs/docker)
- [Deploying on Render: deploy steps (Render Docs)](https://render.com/docs/deploys)
- [Health Checks (Render Docs)](https://render.com/docs/health-checks)
- [Troubleshooting Your Deploy (Render Docs)](https://render.com/docs/troubleshooting-deploys)
- [Deploy a FastAPI App (Render Docs)](https://render.com/docs/deploy-fastapi)
- [Your First Render Deploy (Render Docs)](https://render.com/docs/your-first-deploy)
- [Render CLI reference (Render Docs)](https://render.com/docs/cli-reference)
- [Blueprint YAML reference (Render Docs)](https://render.com/docs/blueprint-spec)
- [Notifications (Render Docs)](https://render.com/docs/notifications)
- [Boot and port binding (Render Tutorials)](https://render.com/tutorials/when-deploys-go-wrong/boot-and-port-binding)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/): Monitor a Django app on Render: JSON logs, a real health check, failure alerts, log and metrics streams, OpenTelemetry under Gunicorn and Celery checks.
- [Fix Railway “Application failed to respond” (502)](https://getpraxen.app/learn/troubleshooting/railway-application-failed-to-respond/): Why Railway returns “Application failed to respond” with a 502, how to tell a wrong host, port or target port from a crash or overload, and how to fix each.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [ChangeMessageVisibility: troubleshoot SQS message timeout errors](https://getpraxen.app/learn/troubleshooting/ChangeMessageVisibility/): ChangeMessageVisibility resets SQS message visibility. Use current receipt handles, bounded timeout extensions and per-entry batch error checks with Lambda.
- [EC2ThrottledException: fix Lambda EC2 API throttling](https://getpraxen.app/learn/troubleshooting/EC2ThrottledException/): EC2ThrottledException means EC2 throttled Lambda initialization. Check VPC state and API activity, distinguish concurrency limits, and verify a safe recovery.
---
### Fix Vercel “FUNCTION_INVOCATION_FAILED” 500 errors
> Why a Vercel Function returns 500 FUNCTION_INVOCATION_FAILED, how to find the crash in runtime logs, and how to fix throws, missing env vars, files and memory.
By Boris Tane, Founder of Praxen · Published October 5, 2026 · 11 min read
Canonical: https://getpraxen.app/learn/troubleshooting/vercel-function-invocation-failed/
```text
FUNCTION_INVOCATION_FAILED
```
Vercel returns this 500 Internal Server Error when the process running your function crashes, most often because Node.js or Bun hit an uncaught exception or an unhandled promise rejection. Open the request in runtime logs to read the stack trace, then fix the throw: usually a missing environment variable, a file left out of the function bundle, or a function running out of memory.
You see this code on Vercel's 500 page, in the Logs tab next to a red 5xx row, or in a client that got an empty Internal Server Error back from an API route. It can hit one route or every route that imports the same broken module. It often starts right after a deploy that worked fine on your machine.
The code itself says very little. It tells you the function process died before it sent a response. The reason is almost always printed one line above it in your runtime logs, as a stack trace. This page shows how to get to that line fast, and how to fix the four causes that produce it most often.
#### What FUNCTION_INVOCATION_FAILED means
Vercel's [error reference](https://vercel.com/docs/errors/FUNCTION_INVOCATION_FAILED) defines it as a failed function invocation, with status code `500` and the name Internal Server Error. It lists two possible causes:
- The runtime process (Node.js, Bun, Python and so on) crashed.
- Node.js or Bun threw an unhandled rejection or an uncaught exception.
Vercel files it under function errors. The platform did its job: it routed the request to your function and started it. Your code, or the environment it runs in, then failed hard enough to take the process down.
That makes it different from the timeout codes. A function that runs too long fails with `FUNCTION_INVOCATION_TIMEOUT` and a 504. If that's what you're seeing, read [how to debug Vercel function timeouts](/learn/troubleshooting/how-to-debug-vercel-function-timeouts/). This page covers crashes only.
When I worked on observability for Workers at Cloudflare, most "the platform returned a 500" reports turned out to be one uncaught throw at module load. The fix starts with reading the right log line, so do that before you change any code.
#### Finding the crash in Vercel's runtime logs
Runtime logs hold everything your function writes to `stdout` and `stderr`, grouped per request. Open your project, go to **Logs** in the sidebar, and filter by **Status Code** `500`. Since March 2026, the request details panel also shows the specific error code next to the HTTP status, so you can confirm the row is `FUNCTION_INVOCATION_FAILED` and not another 500 ([changelog](https://vercel.com/changelog/view-specific-error-codes-in-runtime-logs)).
The CLI does the same from a terminal:
```bash
# Production 500s from the last hour, full messages
vercel logs --environment production --status-code 500 --since 1h --expand
# Everything logged for one request
vercel logs --request-id --expand
# Only Vercel Function logs, as JSON for jq
vercel logs --source serverless --level error --json | jq '.message'
```
To get the request ID for a request you made yourself, read the `x-vercel-id` response header. Vercel's [custom error page docs](https://vercel.com/docs/custom-error-pages) say the request ID token matches that header's value.
In the request details, look at three things:
1. **Log Messages.** The stack trace or error message your runtime printed just before the crash. This names the cause in most cases.
2. **Function.** The function name, runtime, duration and memory usage for that invocation.
3. **Deployment.** The deployment ID and branch. If the errors start at one deployment, diff that deployment against the previous one.
Act quickly on older failures. The [runtime logs docs](https://vercel.com/docs/logs/runtime) keep logs for 1 hour on Hobby, 1 day on Pro and 3 days on Enterprise, or 30 days with Observability Plus.
```mermaid alt="Vercel request logs distinguish exceptions, missing environment variables, missing files and memory exhaustion"
graph TB
A[Request returns 500] --> B[Open the request in Logs]
B --> C{What does the log show?}
C -->|Stack trace from your code| D[Catch and handle the throw]
C -->|Env var is undefined| E[Add it and redeploy]
C -->|ENOENT or module not found| F[Include the file in the bundle]
C -->|High memory, no trace| G[Raise memory or cut usage]
style A fill:#fee2e2,stroke:#fca5a5,color:#7f1d1d
style D fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style E fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style F fill:#d1fae5,stroke:#6ee7b7,color:#065f46
style G fill:#d1fae5,stroke:#6ee7b7,color:#065f46
```
#### Cause 1: an uncaught exception or unhandled promise rejection
**How to tell it's yours:** the log for the failed request ends with a stack trace from your own code or a library it calls. Typical messages are `TypeError: Cannot read properties of undefined`, a JSON parse error, or a rejected `fetch` or database call.
This is the most common cause and the first one in Vercel's list. An exception that nothing catches ends the process. Promise rejections do the same. The [Node.js CLI docs](https://nodejs.org/api/cli.html#--unhandled-rejectionsmode) say that since Node.js 15 the default `--unhandled-rejections` mode is `throw`: Node emits `unhandledRejection`, and if no handler is set, it raises the rejection as an uncaught exception. So a single `await` without error handling, or a promise you fired and forgot, is enough to crash the function.
**Fix:**
1. Read the top frame of the stack trace that belongs to your code. That's the line to fix.
2. Wrap the handler body in `try/catch`, which is the first fix Vercel's error reference suggests. Log the error with its stack and return a proper error response:
```ts
// app/api/orders/route.ts
export async function POST(request: Request) {
try {
const body = await request.json();
const order = await createOrder(body);
return Response.json(order, { status: 201 });
} catch (err) {
console.error("POST /api/orders failed", err);
return Response.json({ error: "Could not create order" }, { status: 500 });
}
}
```
3. Await every promise you start, or attach a `.catch()` to it. A background call you don't await can reject after the response logic has moved on:
```ts
// Before: a rejection here has no handler
sendWelcomeEmail(user);
// After
sendWelcomeEmail(user).catch((err) => {
console.error("welcome email failed", err);
});
```
4. Add process-level handlers in a module every function imports, so any throw you missed is logged with its stack. Keep them for logging. They don't replace the fixes above:
```ts
// lib/crash-logging.ts
process.on("unhandledRejection", (reason) => {
console.error("unhandledRejection", reason);
});
process.on("uncaughtException", (err) => {
console.error("uncaughtException", err);
});
```
5. Validate input before you use it. A request body that's missing a field is a common source of `undefined` errors on one route only.
Check for throws at module load too. Code at the top level of a file, such as a client constructor that reads config, runs before your handler. If it throws, every request to that function fails, and a `try/catch` inside the handler can't help. Move that setup into a function you call inside the handler, or guard it.
#### Cause 2: an environment variable that's missing at runtime
**How to tell it's yours:** the stack trace mentions a value that is `undefined`, an SDK complains that an API key or connection string is missing, or a database client fails to connect with an empty host. It works locally and in one environment, and fails in another.
Vercel scopes each environment variable to Production, Preview or Development, and optionally to a Git branch. A variable that exists in your local `.env` or only in Preview isn't there in Production. The [environment variables docs](https://vercel.com/docs/environment-variables) also say changes are not applied to previous deployments. They only apply to new ones. So adding the variable fixes nothing until you redeploy.
**Fix:**
1. List what each environment has:
```bash
vercel env ls production
vercel env ls preview
```
Sensitive values are hidden in this list, which is expected. Check that the name is present and spelled the same way your code reads it.
2. Add or update the missing variable for the right environment:
```bash
vercel env add DATABASE_URL production
vercel env update DATABASE_URL production
```
3. Reproduce locally with the exact production set. You can pull it to a file or run a command with it, without writing it to disk:
```bash
vercel env pull .env.production.local --environment=production
vercel env run -e production -- next build
```
4. Redeploy so the function gets the new values:
```bash
vercel --prod
```
5. Fail fast with a clear message. Check required variables when the handler runs, so the log says which one is missing:
```ts
function requireEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing environment variable ${name}`);
return value;
}
```
One more limit to know. Vercel allows 64 KB of environment variables in total per deployment, and no single variable can be larger than that. A very large certificate or JSON blob can push you over it.
#### Cause 3: a file or dependency traced out of the function bundle
**How to tell it's yours:** the log shows `ENOENT: no such file or directory` for a path your code reads, or `Cannot find module` for a package or a native binary. The same code works with `next dev` or `vercel dev`.
At build time, Vercel and Next.js trace which files each function needs and put only those in the bundle. Files read through dynamic paths, templates, fonts, data files and some native binaries can be missed. The [Next.js output docs](https://nextjs.org/docs/app/api-reference/config/next-config-js/output) say tracing can fail to include required files, and give `outputFileTracingIncludes` as the fix.
**Fix for Next.js:**
1. Add the missing files to the trace for the routes that need them. Keys are route globs and values are file globs from the project root:
```js
// next.config.js
module.exports = {
outputFileTracingIncludes: {
"/api/report": ["./templates/**/*"],
"/*": ["./src/config/runtime/**/*.json"],
},
};
```
2. In a monorepo, files outside the Next.js project folder aren't traced by default. Set `outputFileTracingRoot` and include them:
```js
const path = require("path");
module.exports = {
outputFileTracingRoot: path.join(__dirname, "../../"),
outputFileTracingIncludes: {
"/api/report": ["../shared/assets/**/*"],
},
};
```
3. Keep patterns narrow. The Next.js docs warn against `**/*` at the repo root because it makes oversized traces.
**Fix for other frameworks:** use `includeFiles` in the `functions` block of `vercel.json`. The [vercel.json reference](https://vercel.com/docs/project-configuration/vercel-json) notes that it is not supported in Next.js:
```json
{
"$schema": "https://openapi.vercel.sh/vercel.json",
"functions": {
"api/report.js": {
"includeFiles": "templates/**"
}
}
}
```
Then redeploy and check the build output. Vercel caps a function bundle at 250 MB uncompressed, or 500 MB for Python, per the [functions limits](https://vercel.com/docs/functions/limitations). If adding files pushes you near that, use `outputFileTracingExcludes` or `excludeFiles` to drop what the function doesn't use.
#### Cause 4: the function runs out of memory
**How to tell it's yours:** the request has no useful stack trace, or the log stops mid-work. The **Function** section of the request details shows memory usage close to the function's limit. Failures cluster on routes that handle large payloads, images, PDFs or big query results.
Vercel's [knowledge base guide on OOM failures](https://vercel.com/kb/guide/detect-memory-and-oom-failures-in-serverless-functions) says that when a function hits its memory limit, the runtime terminates it and returns a 500 error. That matches the "runtime process crashed" cause in the error reference.
**Fix:**
1. Confirm it. Go to **Observability**, then **Vercel Functions**, pick the route, and read the **Peak memory** chart against the provisioned limit. The guide says a function that regularly uses 80 to 90% of its memory is at risk under traffic spikes.
2. Cut the memory the request needs. Stream large responses and files, paginate big queries, and avoid loading a whole upload into memory at once.
3. If the work really needs more, raise the memory. The [memory docs](https://vercel.com/docs/functions/configuring-functions/memory) say you can't set memory in `vercel.json`, and doing so gives a warning at build time. On Pro and Enterprise:
- Open the project, then **Settings**, then **Functions**.
- Select **Advanced Settings**.
- Under **Function CPU**, switch from Standard (2 GB, 1 vCPU) to Performance (4 GB, 2 vCPUs).
- Create a new deployment. The change applies only to future deployments.
4. On Hobby, memory is fixed at 2 GB with 1 vCPU, and 2 GB is also the plan's maximum. Reducing usage is the only fix there, or moving to Pro, where the maximum is 4 GB.
#### How to confirm FUNCTION_INVOCATION_FAILED is fixed
After the new deployment is live:
1. Call the route that failed and check for a 200 or the status you expect:
```bash
curl -s -o /dev/null -w "%{http_code}\n" https://your-app.vercel.app/api/orders
```
2. Watch for new 500s on the new deployment:
```bash
vercel logs --environment production --status-code 500 --since 30m --expand
```
No new rows with `FUNCTION_INVOCATION_FAILED` in the request details means the crash is gone.
3. Open **Observability**, select **Vercel Functions**, and check the **Error Rate** graph. Sort the routes by error rate. The route you fixed should be back at its usual baseline.
If the 500s continue but the stack trace changed, you've fixed one throw and found the next. Repeat from the logs.
#### Stopping crashes from coming back
- **Keep the logging handlers.** `unhandledRejection` and `uncaughtException` logging means the next crash arrives with a stack trace.
- **Check required environment variables in CI.** Run the build with production variables through `vercel env run -e production -- ` so a missing variable fails before deploy.
- **Test the deployed bundle.** Hit every route on a Preview deployment before promoting it. Missing traced files only show up in the deployed bundle.
- **Watch memory before it runs out.** Review the **Peak memory** chart for heavy routes after each release.
- **Alert on 5xx spikes.** Vercel's knowledge base guide says Vercel raises an alert when function failures cause a spike in errors on a route. Route those alerts to the channel your on-call reads, and keep your logs long enough to read them.
#### Tracing FUNCTION_INVOCATION_FAILED to its cause with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Vercel, so when a function crashes it works out why, and the fix arrives ready for your team's normal review.
Running on Vercel? See [how Praxen monitors Vercel in production](https://getpraxen.app/for/vercel/).
#### Common questions
**What status code does FUNCTION_INVOCATION_FAILED return?**
It returns 500, named Internal Server Error in Vercel's error reference. Vercel classes it as a function error, so the cause sits in your function code or its environment. Filter runtime logs with vercel logs --status-code 500 to find the affected requests.
**Why does my function work locally but crash on Vercel?**
The two usual reasons are environment variables and files. A variable set in your local .env may not exist in the Production environment on Vercel, so check with vercel env ls production. A file your code reads at runtime may not be traced into the function bundle, which you fix with includeFiles in vercel.json or outputFileTracingIncludes in next.config.js.
**I added the missing environment variable. Why do I still get the error?**
Vercel applies environment variable changes only to new deployments. The deployment that is crashing still has the old set of variables. Redeploy after you add or update the variable, then check the new deployment's logs.
**Can I raise function memory in vercel.json?**
No. Vercel's memory docs say memory can't be set in vercel.json, and you get a warning at build time if you try. On Pro and Enterprise, change it under Settings, Functions, Advanced Settings, Function CPU: Standard is 2 GB with 1 vCPU and Performance is 4 GB with 2 vCPUs. Hobby always runs with 2 GB.
**Does an unhandled promise rejection really crash the function?**
Yes, on any current Node.js version. Since Node.js 15 the default --unhandled-rejections mode is throw, which raises an unhandled rejection as an uncaught exception when no unhandledRejection handler is set. Vercel's error reference lists that as one of the two causes of FUNCTION_INVOCATION_FAILED.
**How long do I have to read the runtime logs for a failed request?**
Runtime logs are kept for 1 hour on Hobby, 1 day on Pro and 3 days on Enterprise. Observability Plus on Pro or Enterprise extends that to 30 days. Each request can log up to 256 lines and 1 MB in total, so log the error once with its stack trace.
**Is FUNCTION_INVOCATION_FAILED the same as a timeout?**
No. A function that runs past its maximum duration fails with FUNCTION_INVOCATION_TIMEOUT and a 504 status. FUNCTION_INVOCATION_FAILED is a 500 and means the process crashed or threw before it could send a response.
**Where do I find the request ID for a failed request?**
Every Vercel response carries an x-vercel-id header, and Vercel's custom error page docs say the request ID token matches that header's value. Each row in the Logs tab also shows the request ID. Pass it to vercel logs --request-id to pull every log line for that one request.
#### Sources
- [FUNCTION_INVOCATION_FAILED (Vercel Docs)](https://vercel.com/docs/errors/FUNCTION_INVOCATION_FAILED)
- [FUNCTION_INVOCATION_TIMEOUT (Vercel Docs)](https://vercel.com/docs/errors/FUNCTION_INVOCATION_TIMEOUT)
- [vercel deploy (Vercel CLI Docs)](https://vercel.com/docs/cli/deploy)
- [Runtime Logs (Vercel Docs)](https://vercel.com/docs/logs/runtime)
- [vercel logs (Vercel CLI Docs)](https://vercel.com/docs/cli/logs)
- [View specific error codes in runtime logs (Vercel Changelog)](https://vercel.com/changelog/view-specific-error-codes-in-runtime-logs)
- [vercel env (Vercel CLI Docs)](https://vercel.com/docs/cli/env)
- [Environment Variables (Vercel Docs)](https://vercel.com/docs/environment-variables)
- [Configuring Memory and CPU for Vercel Functions (Vercel Docs)](https://vercel.com/docs/functions/configuring-functions/memory)
- [Vercel Functions Limits (Vercel Docs)](https://vercel.com/docs/functions/limitations)
- [Static Configuration with vercel.json (Vercel Docs)](https://vercel.com/docs/project-configuration/vercel-json)
- [Detect memory and OOM failures in Vercel Functions (Vercel Knowledge Base)](https://vercel.com/kb/guide/detect-memory-and-oom-failures-in-serverless-functions)
- [Observability (Vercel Docs)](https://vercel.com/docs/observability)
- [Custom error pages (Vercel Docs)](https://vercel.com/docs/custom-error-pages)
- [output: outputFileTracingIncludes (Next.js Docs)](https://nextjs.org/docs/app/api-reference/config/next-config-js/output)
- [--unhandled-rejections (Node.js CLI Docs)](https://nodejs.org/api/cli.html#--unhandled-rejectionsmode)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/): Debug Vercel FUNCTION_INVOCATION_TIMEOUT 504s: find the slow call with vercel logs and httpstat, fix hangs, set maxDuration and verify on a preview.
- [Fix Prisma “P1001: Can't reach database server”](https://getpraxen.app/learn/troubleshooting/prisma-p1001-cant-reach-database-server/): Why Prisma prints “P1001: Can't reach database server”, how to test the host and port it names, and how to fix private hosts, IPv6, paused databases and URLs.
- [Fix Railway “Application failed to respond” (502)](https://getpraxen.app/learn/troubleshooting/railway-application-failed-to-respond/): Why Railway returns “Application failed to respond” with a 502, how to tell a wrong host, port or target port from a crash or overload, and how to fix each.
- [How to correlate incidents with deploys using evidence](https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/): Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
- [Fix Cloudflare “Error 1102: Worker exceeded resource limits”](https://getpraxen.app/learn/troubleshooting/cloudflare-error-1102-worker-exceeded-resource-limits/): Why a Cloudflare Worker returns Error 1102, how to tell a CPU time overrun from a 128 MB memory overrun, and the code and Wrangler changes that fix each.
---
### Workers Issues: route Cloudflare errors to coding agents
> Cloudflare's new Workers Issues groups exceptions and 5xx errors and sends them to Claude Code, Cursor, Devin or a webhook. Setup, automations and limits.
By Boris Tane, Founder of Praxen · Published October 1, 2026 · 13 min read
Canonical: https://getpraxen.app/learn/incident-response/cloudflare-workers-issues-error-monitoring-agents/
Cloudflare announced Issues on 30 September 2026. It is built-in error monitoring for Workers, in open beta and free during the beta, that groups uncaught exceptions, failed invocations, 5xx responses and error logs into issues with no SDK to install. Turn it on by setting observability.issues.enabled to true and deploying, add an automation that sends issues to Claude Code, Cursor, Devin or a generic webhook, and connect the Workers Observability MCP server separately if your agent needs to query logs and traces.
Cloudflare launched **Issues** on 30 September 2026. It is error monitoring built into the Workers runtime, and it is in open beta. Once you turn it on, every failure your Worker produces is recorded as an occurrence, and occurrences from the same bug are grouped into one issue. An **automation** then sends that issue, with its stack trace, logs, traces and Worker version, to a coding agent, a webhook, a chat channel or an incident tool. It fires when the issue crosses an occurrence threshold, or when it comes back after a quiet period.
This guide covers what Issues records, how it changes the way errors reach your agent, how to turn it on today, and the beta limits and gotchas stated in Cloudflare's announcement and docs. Issues is free for every Workers account during the beta.
#### What Cloudflare Workers Issues is
Issues is a detection and grouping layer that sits inside the Workers runtime. There is no SDK to install and no wrapper around your handler. You change one line of configuration and deploy. Cloudflare records each failure as an **occurrence**, then groups related occurrences from the same Worker into an **issue**. Every failed request has its own request ID, but if the failures share a cause they become one issue. The issue shows when the error first appeared, how many times it has happened, and whether it is getting more frequent.
An issue has one of three statuses:
- **Active**: unresolved. Every new issue starts here.
- **Resolved**: you consider it fixed. A newer occurrence moves it back to Active.
- **Ignored**: it stays ignored when new occurrences arrive, until you change it.
##### What Issues counts as a failure
According to the Investigate issues docs, Issues detects:
- uncaught exceptions
- failed invocations
- HTTP 5xx responses returned by the Worker
- console logs written at error level, or that contain an `Error` object or a stack trace
The announcement adds that Issues flags runaway alarm conditions and code that writes large volumes of logs inside loops. Detection works without Workers Logs or tracing. You only need tracing to attach your own application context, which is covered below.
#### How Issues changes the path from a failing Worker to your agent
Coding agents can already query telemetry, read a repository, write a fix and open a pull request. Cloudflare's announcement points at the manual part in between. Someone has to notice that a hundred failed requests share one bug, collect the logs and traces, paste them into an agent and check later whether the fix worked. Without a structured handoff, the agent has to search raw telemetry to work out the scope of the failure before it can start.
Issues takes over three of those steps:
1. **Grouping.** Repeated exceptions, 5xx responses and error logs become one issue with a count and a trend.
2. **Packaging.** Each occurrence carries the error, a source-mapped stack trace when available, the logs and traces before and after it, the Worker version, invocation and request details, and any attributes you set on the active span.
3. **Handoff.** An automation sends that package to the destination you choose, once, when its trigger condition is met.
The fourth step stays with you. Cloudflare's docs say the agent can propose code and test changes, and you review and deploy them, then mark the issue resolved. Issues never closes anything automatically.
Cloudflare tested this on Workflows, which runs on the Workers platform. Within a day of turning Issues on, the Workflows team found two bugs hidden in a large volume of traffic: a control plane migration stuck retrying on a SQLite foreign key error, and an instance deletion that could exceed a Workers subrequest limit and never finish. Their automation sent both issues to an agent, which proposed fixes.
#### Turning on Issues for a Worker
You can enable Issues three ways. Pick the one that matches how you deploy, because the deploy tool's config wins.
##### With Wrangler
You need Wrangler 4.134.0 or later.
1. Check your version:
```bash
npx wrangler --version
```
2. Set `observability.issues.enabled` in your Wrangler config:
```jsonc
// wrangler.jsonc
{
"$schema": "./node_modules/wrangler/config-schema.json",
"observability": {
"issues": {
"enabled": true
}
}
}
```
Or in TOML:
```toml
[observability.issues]
enabled = true
```
3. Deploy:
```bash
npx wrangler deploy
```
##### From the dashboard
Open the Worker's **Issues** page and select **Enable issues**. If you deploy with Wrangler, also set the flag in your config file. Otherwise your next `wrangler deploy` turns Issues off again.
##### With the cf CLI
`cf` is Cloudflare's new CLI for the public API and Workers projects, and it is also in beta. Projects use a typed `cloudflare.config.ts`:
```ts
// cloudflare.config.ts
import { defineConfig } from "cf/config";
import * as entrypoint from "./src/index.ts" with { type: "cf-worker" };
export default defineConfig({
worker: {
name: "example-worker",
entrypoint,
compatibilityDate: "2026-09-30",
observability: {
issues: {
enabled: true,
},
},
},
});
```
```bash
npm install --global cf
cf deploy
```
If the project is still configured for Wrangler, run `cf migrate` before `cf deploy`. You can run `cf` resource commands next to a Wrangler project without migrating it.
Issues only processes traffic that arrives after you turn it on. It does not scan past failures. Send production traffic, then open the Issues page.
#### Recording handled errors and adding user and account context
##### Report errors your code catches
An error you catch and turn into a fallback response is invisible to Issues unless you log it. Pass the caught value to `console.error()` before you return:
```ts
try {
return await handleRequest(request);
} catch (error) {
console.error(error);
return new Response("Service unavailable", { status: 503 });
}
```
##### Attach the identifiers your agent needs
Cloudflare captures platform context, but it can't know which user, account or session matters to your app. Turn on tracing for the Worker, then set attributes on the active span as early in the invocation as you can, using the runtime's built-in OpenTelemetry API:
```ts
import { tracing } from "cloudflare:workers";
export default {
async fetch(request: Request): Promise {
const { userId, accountId, sessionId } = await getAuthDetails(request);
const span = tracing.getActiveSpan();
span?.setAttribute("user.id", userId);
span?.setAttribute("account.id", accountId);
span?.setAttribute("session.id", sessionId);
return handleRequest(request);
},
} satisfies ExportedHandler;
```
These attributes appear with each occurrence, so you can see whether failures cluster in one account before you hand the issue off. Having built and led the Workers observability team at Cloudflare, I'd set them on day one: an agent given an account ID can tell a single bad tenant from a broken release. Never put secrets, access tokens or sensitive request content in span attributes, logs or error messages. All of it travels to whatever destination you configure.
#### Reading an issue from the dashboard or the cf CLI
In the dashboard, the Issues overview lists active and resolved issues with occurrence totals. Open an issue to see when it started, how often it happened and its status. Then select an occurrence for the error, the stack trace, the related logs and traces, the Worker version, request details and your span attributes. What you see depends on the failure type and the telemetry captured for that invocation.
From a terminal, or from an agent that runs `cf`, the announcement gives two commands. The first lists the most frequent issue. The second fetches one occurrence by the ID it returns:
```bash
cf observability issues list --order-by count --order desc --per-page 1
cf observability issues occurrences --per-page 1
```
Cloudflare's suggested first prompt asks the agent to enable Issues in `cloudflare.config.ts`, show the diff and ask before deploying. After the deploy it waits a minute and runs those two commands. If there are issues, it diagnoses one, fixes it locally, runs checks and asks again before deploying the fix.
```text
> enable Issues on this Worker, then diagnose the most frequent issue and show me the diff
```
*Keep both approval points: before enabling Issues and before deploying any fix.*
#### Sending issues to Claude Code, Cursor, Devin or a webhook
An automation pairs a trigger with a destination. You set it up in the dashboard or with `cf`.
##### Choosing a trigger
- **Occurrence threshold**: runs once when an issue's occurrence count crosses the value you set. It does not run again for each occurrence after that. The minimum threshold is 1.
- **Recurrence after inactivity**: runs when an existing issue comes back after a set period with no newer occurrence. The period can be anywhere from 1 hour to 365 days, and the issue needs at least one earlier occurrence.
A threshold catches new bugs that matter. A recurrence trigger catches regressions of bugs you thought were fixed.
##### Creating the automation
1. Open the Worker's **Issues** page.
2. Select **Automations**, then **Add automation**.
3. Pick an occurrence threshold or recurrence trigger and enter its value.
4. Pick an existing destination or create one.
5. Turn on **Enabled** and select **Create**.
##### What each coding agent destination needs
Every coding agent destination needs a display name. Beyond that:
- **Claude Code**: a routine ID and an authorisation token, from a routine with an API trigger.
- **Cursor**: an automation webhook URL, with an optional authorisation header.
- **Devin**: an authorisation token and an organisation ID, with optional repositories and a playbook ID.
The agent receives the issue summary and the captured context: exception, source-mapped stack trace, leading and trailing logs and traces, Worker version and your application context. What it does next is set by the agent's own configuration, which can range from triage to opening a pull request.
The other destination types are **Chat**, which posts a summary to a team channel, and **Incident management**, which creates an incident or notifies an on-call workflow.
##### Building your own receiver with a generic webhook
To use your own agent or an internal workflow, create a **Generic Webhook** destination. Give it a name and an HTTPS URL, then either enter a webhook secret or turn on mutual TLS. Generic webhooks use the standard Cloudflare Notifications payload. That means:
- Your secret arrives in the `cf-webhook-auth` header. Reject any request where it is missing or wrong.
- Requests come from Cloudflare's published IP ranges, so allowlist them if a firewall sits in front of the endpoint.
- Delivery only goes to a publicly resolvable address on port 80 or 443. For a private endpoint, put a Worker or a Cloudflare Tunnel in front of it.
A minimal receiver written as a Worker:
```ts
interface Env {
WEBHOOK_SECRET: string;
}
export default {
async fetch(request: Request, env: Env): Promise {
if (request.headers.get("cf-webhook-auth") !== env.WEBHOOK_SECRET) {
return new Response("unauthorised", { status: 401 });
}
const event = await request.json();
// Store the event, then hand it to your agent or a queue.
console.log(JSON.stringify(event));
return new Response("ok");
},
} satisfies ExportedHandler;
```
Cloudflare's Notifications webhook page says webhooks need an account with at least one zone on a Pro plan or above. The Issues docs don't repeat that requirement, so check it in your account before you build on generic webhooks.
##### Testing without waiting for a trigger
Open any issue and pick a destination from its actions to send it straight away. The manual run shows up in the issue's activity history. Each run is pending, succeeded or failed. Cloudflare tries the handoff up to three times before marking it failed. A succeeded run means Cloudflare Notifications accepted the event for delivery, which does not prove your agent acted on it.
#### Giving the agent read access to logs and traces
An automation sends an issue. It does not give the destination any permission to query your Cloudflare account. If you want the agent to pull more logs and traces than the issue carries, configure the Workers Observability MCP server for it separately. Grant only the access the investigation needs, and treat destination URLs, webhook secrets and access tokens as credentials.
Before you pick a destination, review its access controls and data handling. Issue data can include error messages, stack traces, request metadata, logs and the identifiers you attached.
#### Beta limits and retention to plan around
**Workers Issues limits and retention during the open beta**
| Item | Value |
| --- | --- |
| Price during open beta | Free for all Workers accounts |
| Automations per account | 50 |
| Minimum occurrence threshold | 1 |
| Recurrence inactivity period | 1 hour to 365 days |
| Completed automation-run history | 30 days |
| Occurrence detail retention | 7 days |
| Handoff attempts per automation run | Up to 3 |
| Minimum Wrangler version | 4.134.0 |
Issues stay listed after their occurrences expire, but the stack traces and logs are gone after seven days. If your agent or your team works an issue later than that, the evidence has to come from somewhere else.
The 50-automation cap is per account. One automation per Worker per trigger type uses it up fast on a large account, so plan shared destinations early.
#### Where teams trip up with Workers Issues
- **Issues switches itself off after a deploy.** You enabled it in the dashboard and deployed with Wrangler. Fix: put `observability.issues.enabled` in the Wrangler config, or `worker.observability.issues.enabled` in `cloudflare.config.ts`.
- **Your Wrangler version is too old.** Issues needs Wrangler 4.134.0 or later, and older versions are not supported. Fix: upgrade Wrangler.
- **The page is empty.** Issues ignores failures from before you enabled it. Fix: send traffic, or wait for real production traffic, then check again.
- **Caught errors never show up.** Your handler swallows them and returns a fallback. Fix: call `console.error(error)` in the catch block.
- **Span attributes are missing.** Tracing is off. Fix: turn on tracing for the Worker before using `tracing.getActiveSpan()`.
- **The agent only gets one ping for a busy issue.** That is how the threshold trigger behaves. Fix: add a recurrence-after-inactivity automation for regressions.
- **Issues pile up as Active forever.** Nothing resolves on its own. Fix: mark issues Resolved after the fix ships. A new occurrence moves the issue back to Active, which is your regression signal. A delayed occurrence recorded before you resolved it does not reopen it.
- **The agent can't find more context.** The automation carries no account access. Fix: connect the Workers Observability MCP server to the agent.
- **The run says succeeded but nothing happened.** Succeeded only means Notifications accepted the event. Fix: log what your receiver gets, and check the agent's own run history.
- **Secrets leak into an agent's context.** A token ended up in a log line or span attribute. Fix: scrub it at the source, because every destination sees what Issues captured.
- **`cf deploy` fails in a Wrangler project.** Fix: run `cf migrate` first, or keep deploying with Wrangler.
#### A rollout order for your first Worker
1. Upgrade Wrangler to 4.134.0 or later, set the Issues flag in config and deploy.
2. Add `console.error()` to the catch blocks that return fallbacks.
3. Turn on tracing and set user, account and session IDs on the active span.
4. Let a day of traffic arrive, then read the top issues yourself before you automate anything.
5. Create one occurrence-threshold automation to a coding agent, and test it with a manual run.
6. Connect the Workers Observability MCP server if the agent needs to query beyond the issue.
7. Add a recurrence automation for regressions, and a chat or incident destination for anything people must see.
8. Review every proposed change, deploy it yourself and mark the issue Resolved.
To turn Issues off, set the flag to `false` and deploy. Issues stops detecting new failures.
#### Taking Workers issues from alert to pull request with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Cloudflare, so when a Worker starts throwing errors it works out why, and the fix arrives ready for your team's normal review.
Running on Cloudflare? See [how Praxen monitors Cloudflare in production](https://getpraxen.app/for/cloudflare/).
#### Common questions
**When did Cloudflare release Workers Issues, and what does it cost?**
Cloudflare announced Issues on 30 September 2026 as an open beta. The docs say it is available to all Workers accounts and free during the beta period. Cloudflare has not published pricing for after the beta.
**Do I need an SDK, Workers Logs or tracing to use Issues?**
No SDK or application wrapper is needed, because Issues is built into the Workers runtime. Detection also works without Workers Logs or tracing. You only need tracing when you want to attach your own attributes, such as user or account IDs, with tracing.getActiveSpan().
**Which coding agents can Issues send errors to?**
Claude Code, Cursor and Devin are the prebuilt coding agent destinations. Claude Code needs a routine ID and authorisation token, Cursor needs an automation webhook URL, and Devin needs an authorisation token and organisation ID. Any other agent can receive issues through a generic HTTPS webhook.
**Will the automation fire on every new occurrence?**
No. An occurrence-threshold automation runs once, when the count crosses the threshold you set, which can be as low as 1. To hear about a bug that returns, add a recurrence-after-inactivity automation with a quiet period between 1 hour and 365 days.
**Can the agent query my Cloudflare logs after it receives an issue?**
Not through the automation. An automation hands over the issue's context but grants no permission to query your account. Configure the Workers Observability MCP server for the agent separately if it needs related logs and traces.
**How long does Cloudflare keep issue data?**
Occurrence details, including stack traces and logs, are kept for seven days. Issues stay listed after their occurrences expire. Completed automation-run history is kept for 30 days.
**Why did Issues turn off after I deployed?**
You probably enabled it in the dashboard and then deployed with Wrangler. Cloudflare's docs say the next Wrangler deploy turns Issues off unless observability.issues.enabled is true in your Wrangler config. Add the flag and redeploy.
**Does Issues resolve an issue once my fix is deployed?**
No. Issues never resolve automatically, so you mark them Resolved after you deploy a fix. If a newer occurrence arrives, the issue goes back to Active, which tells you the fix did not hold.
#### Sources
- [Detect and send production issues straight to your agent (Cloudflare Blog)](https://blog.cloudflare.com/real-time-issue-detection/)
- [Issues · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/issues/)
- [Investigate issues · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/issues/investigate/)
- [Set up an automation · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/issues/automations/)
- [MCP server · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/mcp-server/)
- [Cloudflare CLI · Cloudflare CLI docs](https://developers.cloudflare.com/cf/)
- [Configure webhooks · Cloudflare Notifications docs](https://developers.cloudflare.com/notifications/get-started/configure-webhooks/)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Cloudflare integration · Praxen Docs](https://docs.getpraxen.app/integrations/cloudflare)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/): Debug Cloudflare Workers errors step by step: read 1101 and 1102 codes, enable Workers Logs and source maps, use wrangler tail, DevTools and local traces.
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [How to build a memory system for proactive agents](https://getpraxen.app/blog/how-to-build-a-memory-system-for-proactive-agents/): Proactive agents run without a prompt, so each run has to start from what earlier runs learned. How we built memory for Praxen's agents on git: one repository per team, a nightly clean-up, and lessons from merged pull requests.
---
### How to debug Vercel function timeouts
> Debug Vercel FUNCTION_INVOCATION_TIMEOUT 504s: find the slow call with vercel logs and httpstat, fix hangs, set maxDuration and verify on a preview.
Part 5 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published October 1, 2026 · 12 min read
Canonical: https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/
A Vercel function timeout is a 504 FUNCTION_INVOCATION_TIMEOUT: the invocation ran past its maximum duration (300 seconds by default with Fluid compute) and Vercel stopped it. Find the failing route with vercel logs, time it with vercel httpstat, and search the logs for timeout and ECONNREFUSED to see whether the cause is an upstream call, a code path with no response, a loop or long work. Fix that cause first, raise maxDuration only for work that needs the time, and check the fix on a preview deployment before you ship it.
A Vercel function timeout means one invocation ran longer than its maximum duration, so Vercel stopped it. The caller gets a 504 with the code `FUNCTION_INVOCATION_TIMEOUT`. With Fluid compute, Node.js, Bun and Python functions get 300 seconds by default on every plan. A function that hits that limit has usually been stuck waiting: on a database or API that never answered, on a code path that never sent a response, or on a loop that never ended.
So debugging a timeout means finding where the time went. Confirm the timeout in the runtime logs, measure the route, check its configuration and time each `await` inside the handler. Then fix the cause, and raise `maxDuration` only when the work needs that long. Every command below is a Vercel CLI command that works on a linked project. Run `vercel link` first if you haven't.
#### What FUNCTION_INVOCATION_TIMEOUT means on Vercel
Max duration is the longest one invocation can run before Vercel stops it. For a request handler, the clock covers handling the request and sending the response, and that includes streamed responses. When time runs out, Vercel returns `504 FUNCTION_INVOCATION_TIMEOUT: Gateway Timeout`.
**Figure 1. Default and maximum duration for Node.js, Bun and Python functions with Fluid compute, from Vercel's limits docs**
| Metric | Default (seconds) | Maximum (seconds) |
| --- | ---: | ---: |
| Hobby (seconds) | 300 | 300 |
| Pro (seconds) | 300 | 800 |
| Enterprise (seconds) | 300 | 800 |
Hobby is capped at 300 seconds, and you can't raise it. Pro and Enterprise can set a function to 800 seconds. A beta raises that to 1,800 seconds (30 minutes) for specific runtime versions. Edge runtime functions follow a different rule: they must begin sending a response within 25 seconds.
The limit protects you as well. Vercel sets a default maximum so a runaway function can't use resources without end. A timeout is the symptom. The cause is whatever the function was doing when the clock ran out.
#### The four things that run a Vercel function out of time
Vercel's error reference names four causes. Each one leaves a different trace:
- **Slow work.** An API or database request takes longer than the function's limit. In the logs you see the request start, maybe some progress, then nothing until the 504.
- **No response.** Every code path must return an HTTP response, even if it's an error. If a branch returns nothing, Vercel waits until the maximum duration has passed and then times out. Vercel calls out unhandled exceptions as a common cause. The log usually shows an error line, then silence until the timeout.
- **An infinite loop.** A loop or recursive call never ends. Vercel's guidance: if the logs show a function running right up to the limit with no output, a loop that never finishes is the likely cause.
- **Upstream errors.** An external API or database that the function calls is failing or refusing connections. You will often find `timeout` or `ECONNREFUSED` in the logs.
Slow functions that stay under the limit come from the same causes, plus cold starts, too little memory and a region far from your data. Fixing those makes a timeout less likely the next time traffic spikes.
#### Finding the cause of your timeout, step by step
1. **Confirm the 504s and find the routes.** The JSON output of `vercel logs` includes each request's `requestPath` and `responseStatusCode`, so you can filter for timeouts:
```bash
vercel logs --environment production --source serverless --since 1h --json | jq 'select(.responseStatusCode == 504) | .requestPath'
```
`--source serverless` limits results to Vercel Functions and leaves out static assets and CDN requests.
2. **Read the full log messages for those requests.**
```bash
vercel logs --environment production --source serverless --since 1h --expand
```
3. **Rank routes by duration.** The JSON output doesn't include duration. Open the Vercel Functions tab in Observability on the dashboard and sort by duration to see which routes run closest to the limit.
4. **Measure one endpoint.** `vercel httpstat` breaks a request into DNS lookup, TCP connection, TLS handshake, server processing and content transfer:
```bash
vercel httpstat /api/slow-endpoint
```
Server processing is the time spent inside your function. If that number is the biggest, the problem is in your code or in what it calls.
5. **Check what the function runs with.**
```bash
vercel inspect DEPLOYMENT_URL
```
Check three things: memory (a function with too little gets CPU-throttled), region (a function far from its database adds latency to every query) and max duration (the limit you are actually hitting).
6. **Search for upstream failures.**
```bash
vercel logs --environment production --query "timeout" --since 1h --expand
vercel logs --environment production --query "ECONNREFUSED" --since 1h --expand
```
Matches here point to a dependency more than to your own code. The External APIs tab in Observability shows latency for every external call your functions make. Sort it by P75 to find the slowest upstream service.
7. **Rule cold starts in or out.** Call the endpoint three times in a row:
```bash
vercel httpstat /api/slow-endpoint
vercel httpstat /api/slow-endpoint
vercel httpstat /api/slow-endpoint
```
If only the first call is much slower, a new instance is starting up. That rarely accounts for a full 300 seconds by itself, but it can push a function that is already slow over the limit.
##### Runtime logs expire fast on Hobby
Runtime logs are kept for 1 hour on Hobby, 1 day on Pro and 3 days on Enterprise. If a timeout happened overnight on Hobby, the evidence may already be gone. Each log line can be up to 256 KB. You can also open the logs at the `/_logs` path on the deployment URL, for example `https://my-deployment-my-username.vercel.app/_logs`. On Pro and Enterprise, set up a Log Drain so the logs reach a tool that keeps them longer.
##### Timing every await inside the handler
The 504 tells you that the function ran out of time. It doesn't tell you which call it was waiting on. I built and led the Workers observability team at Cloudflare, and the same habit applies on Vercel: log how long each outbound call takes, so the last timing line before the silence names the culprit. Vercel Functions support the full console API, including `console.time` and `console.timeEnd`:
```ts
export async function GET(request: Request) {
console.time('db:orders');
const orders = await db.query('select id, total from orders where user_id = $1', [userId]);
console.timeEnd('db:orders');
console.time('api:pricing');
const res = await fetch('https://pricing.example.com/v1/quote');
console.timeEnd('api:pricing');
return Response.json({ orders, prices: await res.json() });
}
```
Redeploy, trigger the route and read the runtime logs. A healthy request prints both lines:
```text
db:orders: 41.207ms
api:pricing: 312.554ms
```
A request that times out prints `db:orders` and then nothing. That tells you the pricing call never returned.
#### Fixing each cause
##### An upstream call that hangs needs a deadline
Vercel's advice for upstream failures is to handle the error and return a response, so you don't wait forever. Give every outbound call a deadline well under the function's limit, and turn a failure into a clear status code:
```ts
export async function GET() {
try {
const res = await fetch('https://pricing.example.com/v1/quote', {
signal: AbortSignal.timeout(8000),
});
if (!res.ok) {
return Response.json({ error: 'pricing upstream error' }, { status: 502 });
}
return Response.json(await res.json());
} catch (err) {
console.error('pricing call failed', err);
return Response.json({ error: 'pricing unavailable' }, { status: 503 });
}
}
```
The function now fails in eight seconds with a log line that names the dependency. Your client can retry a 503, and nothing hangs for five minutes. Do the same for database clients: set the driver's query and connection timeouts, and use connection pooling so each request doesn't open a new connection.
##### Every code path has to send a response
The missing-response bug usually lives in an early return or a `catch` block that logs and stops. This Pages Router handler times out on any request that fails validation:
```ts
export default async function handler(req, res) {
const body = parse(req.body);
if (!body) {
console.error('invalid body');
return; // no response sent: Vercel waits until max duration
}
res.status(200).json(await process(body));
}
```
The fix is to send a response on every branch, including errors:
```ts
export default async function handler(req, res) {
try {
const body = parse(req.body);
if (!body) {
res.status(400).json({ error: 'invalid body' });
return;
}
res.status(200).json(await process(body));
} catch (err) {
console.error('handler failed', err);
res.status(500).json({ error: 'internal error' });
}
}
```
##### A loop that never exits
Look for `while` loops whose exit condition depends on data that may never arrive, pagination that follows a cursor without checking whether the cursor changed, and retries with no maximum count. Give each loop an explicit cap. Log the iteration count so a runaway shows up in the logs before the limit does.
##### Long work that needs more time: raise maxDuration
If the work is long by nature (a report, a batch import, a long model call), give that route more time. On Next.js App Router, export it from the route file:
```ts
// app/api/report/route.ts
export const maxDuration = 800;
```
On the Pages Router and Node.js `/api` routes, use the config object:
```ts
export const config = {
maxDuration: 800,
};
```
For older Next.js versions, and for Rust, Go, Python or Ruby, set it per path in `vercel.json`:
```json
{
"$schema": "https://openapi.vercel.sh/vercel.json",
"functions": {
"api/report.js": { "maxDuration": 800 },
"api/*.js": { "maxDuration": 60 }
}
}
```
For Python framework apps (FastAPI, Flask, Django), the whole app builds into one function. Use the entrypoint file as the key, for example `app/main.py`.
These limits assume Fluid compute, which is on by default for new projects. On an older project, turn it on: open the project, go to **Settings**, select **Functions**, enable the **Fluid Compute** toggle, click **Save** and redeploy.
For durations above 800 seconds, the beta needs per-function configuration on `nodejs20.x`, `nodejs22.x`, `nodejs24.x`, Bun `1.x` or `1.4.x`, or `python3.12` to `python3.14`.
##### Work the caller doesn't need to wait for
If the slow part runs after you already know the answer (sending an email, writing analytics, warming a cache), send the response first and keep working in the background with `waitUntil`, or with `after` in Next.js. For AI responses, stream, so output starts before the full computation finishes. For work longer than any single request should take, Vercel points to Workflows, which can pause, resume and keep state without a duration limit.
##### Placement and resources
If server processing time is high but no single call stands out, look at where and how the function runs. Functions run in `iad1` by default, so move the region next to your database if it lives elsewhere. Raise memory from the 2 GB default if the function does heavy computation, because more memory also gives it more CPU. Run independent calls in parallel with `Promise.all`. To shorten cold starts, cut unused dependencies from the bundle and move expensive setup outside the request handler.
#### Proving the fix on a preview deployment
1. Deploy a preview: `vercel deploy`.
2. Measure the same route against it and compare with your earlier numbers:
```bash
vercel httpstat /api/slow-endpoint --deployment PREVIEW_URL
```
3. Check that it still returns the right response:
```bash
vercel curl /api/slow-endpoint --deployment PREVIEW_URL
```
4. To test a deadline fix, point the preview at a dependency you know is slow, and confirm the route returns its 503 well before the limit.
5. Ship with `vercel deploy --prod`, then watch real traffic:
```bash
vercel logs --environment production --source serverless --since 5m
```
#### Where Vercel timeout fixes go wrong
- **Raising maxDuration over a hang.** A call that never returns will use 800 seconds just as it used 300. You also pay provisioned memory for the instance all that time. Fix: add the deadline first, and raise the limit only for work that finishes.
- **Expecting Hobby to go past 300 seconds.** Hobby's default and maximum are both 300 seconds. Fix: move the work out of the request with `waitUntil` or Workflows, or upgrade to Pro.
- **Setting a project default above 800 seconds.** During the beta, values above 800 seconds only work per function, in code or in `vercel.json`. Fix: set it on the route.
- **Expecting the beta on Secure Compute or Static IPs.** Neither supports durations above 800 seconds during the beta.
- **A vercel.json glob that never matches.** Pattern order matters. In a Next.js project that uses the `src` directory, paths need the `/src/` prefix. Fix: check the limit with `vercel inspect` after you deploy.
- **Streams that idle out on HTTP/1.1.** For long-running handlers on HTTP/2, Vercel sends `PING` frames while the response sits idle. HTTP/1.1 has nothing like that, so clients or proxies may close the connection. Fix: stream progress or heartbeat data while the work runs.
- **Forgetting the Edge rule.** Edge runtime functions must begin a response within 25 seconds, whatever `maxDuration` says. Fix: start streaming early, or move the route to Node.js.
- **Running out of file descriptors.** Each instance has 1,024 file descriptors shared across concurrent executions. With Fluid compute's concurrency, sockets that leak can starve other requests on the same instance. Fix: close connections and reuse a pooled client.
- **Debugging a proxied rewrite as a function.** A request proxied to an external origin has its own 120 second proxied request timeout in Vercel's general limits. Fix: check whether the route is a rewrite before you change function settings.
#### Keeping Vercel timeouts from coming back
Put a deadline on every outbound call, and keep those `console.time` lines on the calls that matter. Check the Observability Functions tab sorted by duration after each release, and the External APIs tab sorted by P75, so you catch a dependency that is slowing down before it reaches the limit. On Pro and Enterprise, send logs through a Log Drain into the tool that pages you, and alert on 504s per route. Then a single slow endpoint shows up within minutes, and you are not relying on log retention.
#### Finding the cause of Vercel timeouts with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Vercel, so when a function starts timing out it works out why, and the fix arrives ready for your team's normal review.
Running on Vercel? See [how Praxen monitors Vercel in production](https://getpraxen.app/for/vercel/).
#### Common questions
**What is the maximum duration of a Vercel Function on the Hobby plan?**
With Fluid compute, Hobby functions on Node.js, Bun and Python have a 300 second default and a 300 second maximum, and you can't raise it. Pro and Enterprise also default to 300 seconds but can go to 800 seconds. A beta raises that to 1,800 seconds for supported runtimes.
**Why does a Vercel timeout show up as a 504?**
When an invocation runs past its maximum duration, Vercel stops it and returns 504 FUNCTION_INVOCATION_TIMEOUT: Gateway Timeout. Filter vercel logs output on responseStatusCode 504 to list the routes that hit it.
**Does streaming time count toward maxDuration?**
Yes. For request handlers, max duration includes handling the request and sending the response, streamed responses included. If clients connect over HTTP/1.1, stream heartbeat data during long idle stretches, because those connections can be closed when nothing is being sent.
**Can I set every function in my project to 30 minutes?**
Not yet. During the beta, durations above 800 seconds must be set per function, in code or in vercel.json, and project-level defaults above 800 seconds aren't supported. The beta covers nodejs20.x, nodejs22.x, nodejs24.x, Bun 1.x and 1.4.x, and python3.12 to python3.14, but not Secure Compute or Static IPs.
**Am I billed while my function waits on a slow API?**
Active CPU billing pauses while the function waits on I/O. Provisioned memory is billed for as long as the instance runs. A call that hangs until the timeout still costs memory time, so a deadline on the call saves money as well as the request.
**Where do I find the logs for a timed-out request?**
Open Logs in the project sidebar on the dashboard, run vercel logs --environment production --source serverless --expand, or visit the /_logs path on the deployment URL. Runtime logs are kept for 1 hour on Hobby, 1 day on Pro and 3 days on Enterprise, so check them quickly or set up a Log Drain.
**Does the Edge runtime have a different timeout?**
Yes. Edge runtime functions must begin sending a response within 25 seconds to keep streaming. If your work takes longer before the first byte, start streaming early or move the route to the Node.js runtime.
**Will more memory stop my function timing out?**
Only when the function is CPU-bound. A function with too little memory gets CPU-throttled, so raising memory from the 2 GB default can shorten heavy computation, up to 4 GB on Pro and Enterprise. It won't help a function waiting on a database or API that doesn't answer.
#### Sources
- [FUNCTION_INVOCATION_TIMEOUT (Vercel Docs)](https://vercel.com/docs/errors/function_invocation_timeout)
- [How to stop Vercel Functions from timing out (Vercel Knowledge Base)](https://vercel.com/kb/guide/what-can-i-do-about-vercel-serverless-functions-timing-out)
- [Debugging slow Vercel Functions (Vercel Docs)](https://vercel.com/docs/functions/debug-slow-functions)
- [Configuring Maximum Duration for Vercel Functions (Vercel Docs)](https://vercel.com/docs/functions/configuring-functions/duration)
- [Vercel Functions Limits (Vercel Docs)](https://vercel.com/docs/functions/limitations)
- [Limits (Vercel Docs)](https://vercel.com/docs/limits)
- [Vercel Function Logs (Vercel Docs)](https://vercel.com/docs/functions/logs)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
Next: [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
#### Related
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [Fix Vercel “FUNCTION_INVOCATION_FAILED” 500 errors](https://getpraxen.app/learn/troubleshooting/vercel-function-invocation-failed/): Why a Vercel Function returns 500 FUNCTION_INVOCATION_FAILED, how to find the crash in runtime logs, and how to fix throws, missing env vars, files and memory.
---
### How to use Claude Code to debug production issues
> Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
Part 1 of [AI x Production: coding agents, MCP and autofix](https://getpraxen.app/series/ai-x-production/)
By Boris Tane, Founder of Praxen · Published September 29, 2026 · Updated September 30, 2026 · 12 min read
Canonical: https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/
Give Claude Code the production evidence it can't see by itself. Pipe in logs and stack traces, let it run read-only CLIs such as gh, aws, gcloud and sentry-cli, or connect an MCP server that queries your telemetry. Start in plan mode and ask for hypotheses with the evidence for each. Check recent commits, then have it write a failing test that reproduces the bug before it fixes the root cause. Keep production credentials read-only, and ship the fix only through a pull request you review.
Claude Code needs two things to debug a production issue well. The first is your code, which it reads from the repository. The second is the production evidence, which it can't see until you give it a way in. You bring that evidence in by piping logs and stack traces into the session, by letting it run read-only command-line tools, or by connecting an MCP server that queries your telemetry.
After that, the workflow is the one a careful engineer follows. Read before editing. List hypotheses and check each one against the evidence and the recent commits. Reproduce the bug in a failing test, fix the root cause, and ship the fix as a pull request. This page walks through each step with real commands and prompts. It also covers the guardrails that stop a debugging session from changing production.
#### What debugging production with Claude Code means
Claude Code is Anthropic's agentic coding environment. It runs in the terminal, and there are also versions for VS Code, JetBrains IDEs, the desktop and the web. It reads files, runs shell commands, edits code and works through a problem in a loop until it judges the work done. To debug production with it, you feed that loop the evidence of a live failure, and it traces the evidence back through your code to a cause.
The boundary matters. Claude Code needs a command, a pasted file, a URL you give it or an MCP server before it can see production data such as logs, metrics, traces or deploy history. When that evidence is missing, it has nothing to check its reasoning against, so ask it to show what each conclusion rests on. I built and led the Workers observability team at Cloudflare, and the lesson carries straight over: an agent can only reason over the telemetry you emit and hand to it.
Claude Code starts when you prompt it, schedule it or trigger it from an event: routines can fire on API calls or GitHub events, GitHub Actions can run it in CI, and channels can push alerts and webhooks into a session from an MCP server. For most teams, it is the tool you reach for once you already know something is wrong. Tools sold as AI SREs aim at a different job, and the related guide on what an AI SRE is covers that category.
#### Three ways to get production evidence into the session
##### Pipe logs and stack traces in directly
This is the fastest route. Filter the logs to the incident window first, then send them in:
```bash
grep "req_8f2c" /var/log/checkout/app.log | claude
git log --oneline -20 | claude -p "summarize these recent commits"
```
`claude -p` runs one query and exits, which suits quick questions and scripts. You can paste a stack trace straight into the prompt. You can also paste a screenshot of a dashboard or an error page with Ctrl+V (Alt+V on Windows and WSL), or give it an image path. Tell Claude whether the failure is intermittent or consistent, and give it the command that reproduces it if you have one.
##### Let it run the CLIs you already use
CLI tools are the most context-efficient way for Claude Code to reach external services. Tell it to use `gh` for pull requests and CI runs, and `aws`, `gcloud` or `sentry-cli` for your cloud and error data. Give those CLIs read-only credentials. Then write down the commands it can't guess in `CLAUDE.md`, the file Claude reads at the start of every session:
```markdown
# Production debugging
- Production logs: ./scripts/prod-logs.sh SERVICE MINUTES (read-only)
- Errors: use sentry-cli; the token in this shell is read-only
- Deploys are tagged with the commit SHA; map a deploy to its changes with git log
- Never run commands that write to production resources
```
Keep the file short. Claude follows a concise `CLAUDE.md` more reliably than a long one. Run `/init` to generate a starter file, and `/context` to confirm it loaded.
##### Connect your telemetry over MCP
If your observability tool or platform runs an MCP server, register it once and Claude can query it mid-conversation:
```bash
claude mcp add --transport http telemetry https://mcp.example.com/mcp
```
For a server the whole team shares, put it in `.mcp.json` at the repository root under the `mcpServers` key. A project-scoped server needs a one-time approval, so run `/mcp` to check its status and approve it. Once it is connected, you can pull MCP resources into a prompt with the `@server:resource` syntax, the same way you reference files with `@`.
#### Working an incident with Claude Code, step by step
1. **Start in plan mode.** Run `claude --permission-mode plan`, or press Shift+Tab until the status bar shows plan mode on. In this mode Claude reads files and answers questions without making changes.
2. **Describe the symptom with its evidence.** Say what broke, when it started, where the evidence is, and what fixed looks like. Ask for causes before any fix:
```text
Checkout requests started failing after this morning's deploy.
The stack trace and recent log lines are in /tmp/incident/.
Failures are intermittent: some orders succeed.
Read the code paths in the trace and list the likely causes.
For each cause, say what evidence would confirm or rule it out.
Don't edit anything yet.
```
3. **Line the failure up with recent changes.** Ask it to use `git log` to find commits since the last good deploy that touched the files in the trace, and to say which of them could produce this error.
4. **Test each hypothesis against evidence.** Have it run the read-only queries or log searches that would confirm or rule out each cause, and quote the output it relied on. If it says a cause is confirmed without showing a log line, query result or commit, ask for the evidence.
5. **Reproduce the bug in a failing test.** Ask it to write a test that fails for the reason it identified. Watching that test fail proves the diagnosis.
6. **Fix the root cause and verify.** Leave plan mode by approving the plan or pressing Shift+Tab. Press Ctrl+G first if you want to edit the plan in your editor. Then ask it to fix the cause, run the test suite and fix any failures.
7. **Ship through review.** Ask it to commit with a descriptive message and open a PR. Your normal review and CI decide the merge. You can find the session again later with `claude --from-pr` followed by the PR number.
```text
> checkout errors began after the last deploy. list hypotheses and the evidence for each
```
*Ask in plan mode first, so Claude reads and reasons before it edits anything.*
#### Keeping Claude Code read-only against production
Instructions in `CLAUDE.md` are guidance: they tell Claude how you work. Permissions, hooks and the sandbox are the parts that enforce limits whatever Claude decides. When production is involved, use the enforcing layer.
- **Give the shell read-only credentials.** This is where I'd start: a read-only cloud role or API token limits what any command Claude runs can do against production.
- **Pick the permission mode on purpose.** In Manual mode, Claude asks before file writes, Bash commands and MCP tool calls. From Claude Code v2.1.283, interactive terminal and VS Code sessions start in auto mode. There, a separate classifier model reviews most actions and blocks only what looks risky, such as scope escalation or unknown infrastructure. During an incident, press Shift+Tab to switch mode if you want to approve each command yourself.
- **Allowlist the safe commands.** Use `/permissions` to pre-approve the read-only commands you trust, so you only get prompts for actions that matter. Run `/permissions` again to see the allow and deny rules actually in effect.
- **Sandbox the session.** `/sandbox` turns on OS-level isolation that restricts filesystem and network access.
- **Know what deny rules miss.** Bash rules match the literal command string. A `Bash(rm *)` deny rule doesn't block `/bin/rm` or `find -delete`. For a hard guarantee, use a PreToolUse hook or the sandbox.
#### Holding the context window together during a long investigation
Claude's context window holds every message, every file it reads and every command output. A single debugging session can use tens of thousands of tokens, and model performance drops as the window fills: Claude starts forgetting earlier instructions and making more mistakes. Incidents are where this bites, because logs are large.
- **Filter before you send.** Grep for the request ID, the error class or the time window before you pipe anything in.
- **Delegate wide searches to a subagent.** A prompt like "use a subagent to search the checkout logs for this request ID and report only the matching lines" runs the search in a separate context window, and only the summary comes back.
- **Watch the budget.** `/context` breaks down what fills the window. Use `/clear` between unrelated incidents. If an investigation spans several sittings, `claude --continue` resumes the latest session in the directory and `claude --resume` lets you pick one from a list.
- **Move runbooks into skills.** A procedure you need only during incidents belongs in a skill. Claude loads a skill when a request matches it, so it doesn't sit in every session the way `CLAUDE.md` does.
#### Proving the fix before it ships
Claude stops when the work looks done. Unless it has a check it can run, you become the only verification, so give it something that returns pass or fail.
- **In the prompt:** "write a failing test that reproduces the issue, then fix it, then run the tests." Add "address the root cause, don't suppress the error" when the likely quick fix is a guard clause.
- **Across the session:** set the check as a `/goal` condition. A separate evaluator re-checks it after every turn.
- **As a hard gate:** a Stop hook runs your check as a script and blocks the turn from ending until it passes.
- **With a second opinion:** a verification subagent gets a fresh model to try to refute the fix. That way the agent that did the work isn't the one grading it.
Ask for evidence, and don't accept a claim of success on its own. The test output, the command it ran and what it returned are faster to review than rerunning everything yourself. After Claude's checks pass, you can run `/verify` yourself to confirm the change against the running app.
#### Where production debugging with Claude Code goes wrong
- **A confident cause with no evidence.** A stated cause with no log line, query or commit behind it is a guess. Fix: require every claim to name the log line, query output or commit behind it, and to mark anything unverified as unknown.
- **Raw logs flooding the context.** Dumping an unfiltered log file pushes your instructions out of the model's attention. Fix: filter to the incident window first, or hand the search to a subagent.
- **Patching the symptom.** A null guard where the data should never be null hides the bug until it breaks somewhere else. Fix: ask why the value is missing, fix it at the source, add a regression test, and ask Claude to search the codebase for the same pattern.
- **Trusting CLAUDE.md to prevent writes.** A rule in a markdown file is advice. Fix: read-only credentials, permission rules, a PreToolUse hook or the sandbox.
- **An MCP server that never shows up.** The usual causes are `.mcp.json` placed inside `.claude/` or servers listed under a `servers` key. Two others are a dismissed approval prompt and relative paths, which resolve against the directory you launched from. If `/mcp` shows the server connected with zero tools, choose Reconnect. If it still shows zero, run `claude --debug=mcp` and read the server's stderr in `~/.claude/debug/.txt`.
- **Your own configuration getting in the way.** If Claude behaves oddly mid-incident, start `claude --safe-mode`. It disables CLAUDE.md, skills, plugins, hooks and MCP servers, so you can tell whether a customisation is the cause.
#### Running production checks without a person at the keyboard
Claude Code can run on a schedule, which covers recurring checks such as looking for CI failures overnight. Routines run in the cloud, so they work when your computer is off, and they can also trigger on API calls or GitHub events. Desktop scheduled tasks run on your machine, GitHub Actions runs in your CI pipeline, and `/loop` polls within an open session. A scheduled run can't ask you clarifying questions, so its prompt has to say what success looks like and what to do with the result. Keep these runs on read-only credentials too.
#### Finding and fixing production issues with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to the clouds and tools your team already runs on, so when something breaks there it works out why, and the fix arrives ready for your team's normal review.
Using Claude Code? See [how to give Claude Code production context with Praxen](https://getpraxen.app/for/claude-code/).
#### Common questions
**Can Claude Code read my production logs directly?**
Only through a path you give it. You can pipe a filtered log file in with a command like cat error.log | claude, let it run read-only CLIs such as aws, gcloud or sentry-cli, give it a URL to fetch, or connect an MCP server that queries your observability tool.
**Is it safe to give Claude Code access to production?**
It is much lower risk when every layer is read-only. Give its shell read-only credentials, allowlist only the read commands you trust with /permissions, and turn on /sandbox for OS-level isolation. Send every fix through a pull request that your review and CI gate.
**How do I connect Claude Code to my observability tool?**
If the tool has a CLI, install it and tell Claude to use it. That is the most context-efficient option. If it runs an MCP server, register it with claude mcp add --transport http NAME URL, or add it to .mcp.json at the repository root under mcpServers. Then run /mcp to approve it and check it is connected.
**What is a good first prompt when an incident starts?**
Start in plan mode with claude --permission-mode plan. Describe the symptom, when it started, where the stack trace and logs are, and whether it is intermittent. Ask for a list of likely causes, with the evidence that would confirm or rule out each one, and tell it not to edit anything yet.
**Why does Claude Code sometimes give a confident but wrong root cause?**
One reason is a full context window: as it fills, Claude forgets earlier instructions and makes more mistakes. Another is missing evidence: without logs, metrics or deploy history in the session, it has nothing to check its reasoning against. Require each claim to cite the command output behind it, filter logs before sending them, and use a subagent for wide searches.
**How do I stop Claude Code from running a destructive command?**
Use a PreToolUse hook or the sandbox. Bash deny rules match the literal command string, so a Bash(rm *) rule does not block /bin/rm or find -delete. Keeping its credentials read-only is a sensible backstop as well.
**Can Claude Code watch production and start debugging on its own?**
It starts when you prompt it, schedule it or trigger it from an event. Routines run in the cloud on a schedule or on API calls and GitHub events, GitHub Actions runs it in CI, channels push alerts and webhooks into a session from an MCP server, and /loop polls within an open session. For tools built around alert-driven investigation, see the related guide on what an AI SRE is.
**How do I pick an incident investigation back up later?**
Run claude --continue to resume the most recent session in the current directory, or claude --resume to pick from a list. If Claude opened a pull request during the session, claude --from-pr followed by the PR number opens the session picker filtered to that PR.
#### Sources
- [Best practices for Claude Code](https://code.claude.com/docs/en/best-practices)
- [Common workflows - Claude Code Docs](https://code.claude.com/docs/en/common-workflows)
- [Debug your configuration - Claude Code Docs](https://code.claude.com/docs/en/debug-your-config)
- [Quickstart - Claude Code Docs](https://code.claude.com/docs/en/quickstart)
- [Claude Code documentation index](https://code.claude.com/docs/llms.txt)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[AI x Production: coding agents, MCP and autofix](https://getpraxen.app/series/ai-x-production/)
1. [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/)
2. [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/)
Next: [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/)
#### Related
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/): How to use Claude Code: install it, write a CLAUDE.md, pick a permission mode, steer it mid-task, script it with -p and connect it to your tools with MCP.
- [Workers Issues: route Cloudflare errors to coding agents](https://getpraxen.app/learn/incident-response/cloudflare-workers-issues-error-monitoring-agents/): Cloudflare's new Workers Issues groups exceptions and 5xx errors and sends them to Claude Code, Cursor, Devin or a webhook. Setup, automations and limits.
- [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/): Monitor a Convex app in production: use the Health page and CLI, stream logs to Axiom or Datadog, report exceptions to Sentry, and alert on failures.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
---
### How to monitor a Convex app in production
> Monitor a Convex app in production: use the Health page and CLI, stream logs to Axiom or Datadog, report exceptions to Sentry, and alert on failures.
Part 8 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 30, 2026 · 13 min read
Canonical: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/
Start with the Health page, the Logs page and the npx convex logs and npx convex insights commands. These only show recent activity, so on a Convex Pro plan you should also stream logs to Axiom, Datadog, PostHog or a webhook, and send exceptions to Sentry or PostHog. Then set alerts on the function_execution events Convex emits for every function run: failure rate per function, slow functions, write conflicts, scheduler lag and functions getting close to their limits.
To monitor a Convex app in production, start with what every deployment already has: the Health page, the Logs page, and the `npx convex logs` and `npx convex insights` commands. All of these show recent activity only. On a Pro plan, add a log stream to Axiom, Datadog, PostHog or a webhook, and turn on exception reporting to Sentry or PostHog. Then build alerts from the `function_execution` events Convex sends for every function run.
Convex runs your backend as TypeScript functions that sit next to its database. That means almost everything worth watching shows up as a function execution: its status, how long it took, what it read and wrote, and why it ran. This page goes from the built-in views to alerts that reach whoever is on call.
#### What monitoring a Convex app means
For the broader backend decision, [Supabase vs Convex in production](/learn/reliability/supabase-vs-convex-in-production/) compares consistency, scaling limits, migrations and recovery alongside the monitoring work below.
A Convex backend is a set of functions of four types: queries, mutations, actions and HTTP actions. They run on either the default Convex runtime or the Node.js runtime. The scheduler and cron jobs run functions with no user present. Clients subscribe to queries, and Convex reruns those queries whenever the data they read changes.
Convex turns every run into structured events:
- A `function_execution` event for each run. It holds the function path and type, `status` (`success` or `failure`), `execution_time_ms`, an `error_message` on failure, a `usage` block (documents and bytes read and written, egress, action memory), and a `run_reason`.
- A `console` event for each `console.log`, `console.warn` or `console.error`, with a `log_level` and an `is_truncated` flag.
- Deployment-level topics: `concurrency_stats`, `scheduler_stats`, `current_storage_usage`, `storage_api_bandwidth`, `audit_log` and `ai_gateway_usage`.
Monitoring Convex in production comes down to three jobs. Keep those events longer than the dashboard does. Write alerts on the few that mean users are hurting. Be able to follow a single failed request from the error a user saw to the log line behind it. I built and led the Workers observability team at Cloudflare, and the rule I'd apply to any hosted backend applies here: get the platform's own events into the tool you already alert from before you add any custom instrumentation.
#### What the Convex dashboard and CLI already show you
##### The Health page
The Health page is where each deployment opens. It shows:
- **Failure rate**: the percentage of failed requests, per minute, over the last hour.
- **Cache hit rate**: the percentage of cache hits, per minute, over the last hour. This applies to query functions only.
- **Scheduler status**: if the scheduler falls behind because too many tasks are scheduled, this card shows the lag in minutes.
- **Last deployed**: when functions were last pushed. Check it first when the failure rate jumps.
- **Insights**: functions that read too many bytes or too many documents in one transaction, and functions hitting write conflicts. Each insight comes with a chart and an event log.
- **Integrations** (Professional only): the status of your log streams and exception reporting.
##### The Logs page
The Logs page gives a realtime view of function activity plus a short history. Each entry shows the time, request ID, outcome, function name, output and duration in milliseconds. Duration here leaves out network latency. You can filter by text, function, execution status and log severity. Clicking an entry shows every log line that shares its request ID. Because the page doesn't keep a complete history, older requests may be gone by the time you look.
##### The CLI
```bash
# Tail production logs in the terminal, and keep a copy you can grep later
npx convex logs --prod | tee ./logs.txt
# Health insights for the last 72 hours: write conflicts and resource limit problems
npx convex insights --prod --details
# Check a hunch against production data with a read-only, sandboxed query
npx convex run --prod --inline-query 'await ctx.db.query("messages").take(5)'
# List tables, then print rows from one
npx convex data --prod
npx convex data messages --prod
```
Inline queries run in a sandbox. They can read data but they can't modify the database or reach the network, which makes them safe for poking at production during an incident.
#### The Convex signals worth an alert
The events carry a lot of fields. These are the ones worth paging on or checking every day:
- **Failure rate per function**: `status` equal to `failure`, divided by total calls, grouped by function path. A single failing mutation matters more than a small rise in the overall rate.
- **Slow functions**: `execution_time_ms` above the level your users will tolerate, grouped by function path.
- **Write conflicts**: events carrying `occ_info`, which names the table, the document and the conflicting function. Also watch `mutation_retry_count` on successful mutations.
- **Scheduler lag**: the `scheduler_stats` topic, or the scheduler status card on the Health page.
- **Functions nearing limits**: `console` events with a `system_code`, which Convex adds automatically when a function is approaching its limits.
- **Error-level console logs**: `log_level` equal to `ERROR`.
- **Usage drift**: `database_read_documents`, `database_io_read_bytes` and `action_memory_used_mb` per function, compared week on week.
#### Step 1: Stream Convex logs to a destination you can alert from
Log streams send function executions, console logs and deployment events to Axiom, Datadog, PostHog or any webhook URL. They need a Convex Pro plan.
1. Open the production deployment in the Convex dashboard, then go to **Settings** and the **Integrations** tab.
2. Choose a destination and enter what it asks for:
- **Axiom**: a dataset name, an API key, and optional attributes that are added to every event.
- **Datadog**: your Datadog site location, an API key, and comma-separated tags sent in the `ddtags` field.
- **PostHog**: a project token, an optional host (US Cloud by default, EU Cloud or self-hosted if you set one), and an optional service name. Logs arrive in OpenTelemetry log format.
- **Webhook**: a URL. Each request body is a JSON array of events.
3. Save it and wait for the `verification` event. Once it arrives, the Convex dashboard shows the Axiom stream as verified and active.
4. Add attributes or tags that name the project and environment. They make queries across several deployments easier to read.
5. With Axiom, open the **Integrations** section of the **Dashboards** tab. Axiom creates a Convex dashboard for you, covering execution metrics, errors and performance. Fork it if you want to change the layout.
In Axiom, the fields specific to each event sit under a `data` prefix (`data.status`, `data.function.path`). Deployment metadata sits under `convex` (`convex.deployment_name`, `convex.deployment_type`). Webhook payloads carry the same fields at the top level of each event.
#### Step 2: Send function exceptions to Sentry or PostHog
Logs tell you a function failed. An error tracker groups the failures, keeps the stack traces and tracks which users were affected. Exception reporting is a Pro feature too.
1. In Sentry, create a project and set its platform to **Node.js** so Convex exception events are processed correctly. Copy the DSN.
2. In the Convex dashboard, open **Settings**, then **Integrations**, click the Sentry card and paste the DSN. You can add tags of your own if you like.
3. Convex tags every event with `func`, `func_type`, `func_runtime`, `request_id`, `server_name` and `environment` (`prod`, `dev` or `preview`). When the function is authenticated, it sets the user to the caller's `tokenIdentifier`. You can't override these tags.
4. Expect exceptions to take a minute or two to show up.
For PostHog Error Tracking, use the PostHog card and your project token. Each `$exception` event carries `convex_function`, `convex_function_type`, `convex_deployment_type` and `convex_request_id`, and stack traces are included. For Datadog Error Tracking, set it up through Datadog's Sentry SDK path, then use the Convex Sentry integration.
#### Step 3: Turn function_execution events into alerts
##### Axiom monitors in APL
These queries follow the field names Axiom documents for the Convex integration. The first one pages when a function's error rate goes above 5%:
```kusto
['convex']
| where ['data.topic'] == "function_execution"
| where ['convex.deployment_type'] == "prod"
| summarize total_calls = count(), failures = countif(['data.status'] == "failure") by ['data.function.path']
| where total_calls >= 20
| extend error_rate = (failures * 100) / total_calls
| where error_rate > 5
```
The `total_calls` floor stops a function called twice in the window from paging you over one failure. Before you rely on the `deployment_type` filter, check which values your own events carry.
This one finds functions that took longer than 5,000 ms:
```kusto
['convex']
| where ['data.topic'] == "function_execution" and ['data.execution_time_ms'] > 5000
| summarize count() by ['data.function.path']
```
This one ranks functions by how much data they read. It is the query to run once a week, and the one to reach for when database I/O costs climb:
```kusto
['convex']
| where ['data.topic'] == "function_execution"
| summarize avg_docs_read = avg(['data.usage.database_read_documents']), avg_bytes_read = avg(['data.usage.database_io_read_bytes']) by ['data.function.path']
| order by avg_bytes_read desc
```
Axiom's own example uses `database_read_bytes`, but the current Convex event schema calls this field `database_io_read_bytes`, so the query above uses that name.
Axiom monitors can alert on the average execution time too. For example, you can fire when the average across functions goes above 2 seconds in any 5 minute period, and route the alert to email, Slack, PagerDuty or another notifier.
##### Datadog and PostHog
In Datadog, write log monitors on the same fields: failed `function_execution` events grouped by function path, and execution time above your threshold. Use the tags you set in `ddtags` to keep production separate from everything else. In PostHog, the full event is the log record body, and the deployment metadata comes as resource attributes, so you filter on those.
##### A webhook receiver of your own
A webhook stream fits when you want the events in your own pipeline or warehouse. Every request is signed. Convex computes an HMAC-SHA256 of the body, encodes it as lowercase hex with a `sha256=` prefix, and sends it in the `x-webhook-signature` header. You'll find the secret in the dashboard when you set up the webhook. Verify the signature in constant time, and reject events whose timestamp is too old:
```typescript
export default {
async fetch(req: Request, env: { WEBHOOK_SECRET: string }) {
const body = await req.arrayBuffer();
const signature = req.headers.get("x-webhook-signature");
if (!signature) return new Response("Unauthorized", { status: 401 });
const key = await crypto.subtle.importKey(
"raw",
new TextEncoder().encode(env.WEBHOOK_SECRET),
{ name: "HMAC", hash: "SHA-256" },
false,
["verify"],
);
const valid = await crypto.subtle.verify(
"HMAC",
key,
Uint8Array.fromHex(signature.replace("sha256=", "")),
body,
);
if (!valid) return new Response("Unauthorized", { status: 401 });
const events = JSON.parse(new TextDecoder().decode(body));
if (events[0].timestamp < Date.now() - 5 * 60 * 1000) {
return new Response("Request expired", { status: 403 });
}
const failures = events.filter(
(e: any) => e.topic === "function_execution" && e.status === "failure",
);
// forward failures to your alerting tool here
return new Response("ok");
},
};
```
#### Reading write conflicts, retries and query reruns
Some Convex problems never show up as a failed request. Three fields catch them.
**`occ_info`** appears when a function hits a write conflict with another function. It names the `table_name`, the `document_id`, the `write_source` (the conflicting function) and a `retry_count`. When the same document keeps showing up, many callers are writing to one hot document. A counter that every request increments is the usual culprit. The Health page and `npx convex insights` list the same conflicts.
**`mutation_retry_count`** counts failed attempts before a mutation succeeded. The mutation reports `success`, so an error-rate alert never sees it. Users still wait for every retry. Track the average retry count per mutation next to latency.
**`run_reason`** says why a function ran: `initialSubscription`, `dataChange`, `identityChange`, `webSocket`, `httpApi`, `httpEndpoint`, `cron`, `scheduler`, `action` or `tester`. A query with a high count of `dataChange` reruns is being invalidated by frequent writes to data it reads. Subscription updates count as function calls in billing, so this field explains a function call bill that grows faster than your traffic. `mutation_queue_length` does the same job for mutation backlogs within a single client session.
#### Catching functions before they hit Convex limits
Convex enforces limits that turn a slow-growing problem into a sudden failure. The ones worth knowing:
- Actions on the Convex runtime get 64 MiB of RAM, and Node.js actions get 512 MiB. Watch `action_memory_used_mb` for actions creeping toward their runtime's limit.
- Function arguments and return values can each be up to 16 MiB. Node actions accept arguments up to 5MiB. The `function_args_bytes` and `function_returns_bytes` usage fields show how close you are.
- A document can be up to 1 MiB, and an array can hold up to 8192 elements. A list you keep appending to inside one document will hit one of these limits eventually.
- Code size is capped at 32 MiB per deployment.
- The Free plan includes 1,000,000 function calls a month in total. Professional includes 25,000,000 a month, and calls beyond that are billed.
Convex warns you before some of these fail: it adds a `console` event with a `system_code` when a function gets close to a limit. Alert on any `console` event where `system_code` is present. The Health page insights flag functions that read too many bytes or documents in one transaction.
#### Following one failed request from the browser to the log line
In production, Convex hides server-side error details from the browser console so it doesn't leak server state. What the client gets is a Request ID. Tie your tools together through it:
1. Log the Request ID in your frontend's error handler, or pass it to your client-side error tracker.
2. For a recent failure, paste the ID into the search box on the Logs page. You'll see every line from that request, including the exception.
3. For an older failure, search your log stream for `function.request_id` (under `data.` in Axiom). In Sentry, search the `request_id` tag.
4. With the failing function identified, read the relevant data using `npx convex run --prod --inline-query` to check whether the documents look the way the code expects.
5. Compare the failure's start time with **Last deployed** on the Health page, and with the deploy history in your CI.
#### Where Convex monitoring goes wrong
- **Treating the Logs page as your history.** It keeps a short window, and the request you need will have gone by the next morning. Fix: set up a log stream before you launch.
- **Streaming in the legacy format.** Log streams set up before May 23, 2024 use the legacy event schema, which has different field names. Fix: update the stream to the new format in the dashboard under Settings, Integrations, then update your queries.
- **Letting dev and preview deployments page you.** Exception tags and event metadata cover every deployment type. Fix: filter alerts on `environment` in Sentry and on `convex.deployment_type` in your log tool.
- **Accepting unsigned webhook traffic.** Anyone who learns the URL can post fake events to it. Fix: verify `x-webhook-signature` in constant time and reject stale timestamps.
- **Sentry events arriving garbled.** The Sentry project platform wasn't set to Node.js. Fix: create the project with the Node.js platform.
- **Error-rate alerts that fire on one failure.** A function with two calls and one failure has a 50% error rate. Fix: require a minimum number of calls in the window before the rate counts.
- **Missing slow mutations that succeed.** Write conflicts get retried, and the final `success` hides the cost. Fix: alert on `occ_info` and on the average `mutation_retry_count` per mutation.
- **Writing queries against the wrong field paths.** Axiom nests event data under `data`, while webhook payloads don't. Fix: look at one raw event in your destination before you write a monitor.
#### Fixing Convex issues with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Convex, so when a deployment starts failing it works out why, and the fix arrives ready for your team's normal review.
Running on Convex? See [how Praxen monitors Convex in production](https://getpraxen.app/for/convex/).
#### Common questions
**Do I need a paid Convex plan to monitor production?**
No for the basics. The Health page, the Logs page, npx convex logs and npx convex insights don't need a log stream or exception integration. Log streams to Axiom, Datadog, PostHog or a webhook, and exception reporting to Sentry or PostHog, both need a Convex Pro plan.
**How long does the Convex dashboard keep logs?**
The Logs page shows a short history of recent function logs and new ones as they arrive. It isn't a complete historical record, so older requests may be missing. To keep history, set up a log stream, or run npx convex logs --prod | tee ./logs.txt to save a copy locally.
**Which log and error destinations does Convex support?**
Log streams can go to Axiom, Datadog, PostHog, or a webhook at any URL you choose. Exception reporting supports Sentry and PostHog Error Tracking. Datadog Error Tracking works through its Sentry SDK path combined with the Convex Sentry integration.
**How do I find the server error behind a failure a user saw in production?**
Convex hides server error details from the browser in production and gives the client a Request ID. Paste that ID into the search box on the Logs page for recent requests. For older ones, search function.request_id in your log stream or the request_id tag in Sentry.
**What should my first Convex alert be?**
Start with a failure-rate alert per function: count function_execution events with status failure, divide by total calls, group by function path, and fire above 5%. Set a minimum call count so low-traffic functions can't page you over a single error. Add a slow-function alert on execution_time_ms next.
**How do I monitor write conflicts in Convex?**
Failed or retried functions carry an occ_info object with the table name, document ID and the conflicting function. Alert on events where it's present, and track mutation_retry_count on successful mutations. Run npx convex insights --prod --details to see conflicts from the last 72 hours.
**Why does my Convex log stream use different field names from the docs?**
Log streams configured before May 23, 2024 send the legacy event schema. Convex recommends updating to the current format. Once you do, update your queries and monitors to the new field names.
**How quickly do Convex exceptions appear in Sentry or PostHog?**
Convex documents a delay of a minute or two for both Sentry and PostHog. For anything faster, alert from the log stream on failed function_execution events, which carry the error message and stack trace.
#### Sources
- [Log Streams, Convex Developer Hub](https://docs.convex.dev/production/integrations/log-streams/)
- [Legacy Event Schema, Convex Developer Hub](https://docs.convex.dev/production/integrations/log-streams/legacy-event-schema)
- [Exception Reporting, Convex Developer Hub](https://docs.convex.dev/production/integrations/exception-reporting)
- [Health, Convex Developer Hub](https://docs.convex.dev/dashboard/deployments/health)
- [Logs, Convex Developer Hub](https://docs.convex.dev/dashboard/deployments/logs)
- [CLI, Convex Developer Hub](https://docs.convex.dev/cli)
- [Limits, Convex Developer Hub](https://docs.convex.dev/production/state/limits#functions)
- [Observing your app in production, Stack by Convex](https://stack.convex.dev/observability-in-production)
- [Connect Axiom with Convex, Axiom Docs](https://axiom.co/docs/apps/convex)
- [Send data from Convex to Axiom, Axiom Docs](https://axiom.co/docs/send-data/convex)
- [Convex + Axiom: Complete observability for reactive backends](https://axiom.co/blog/axiom-convex-integration)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
#### Related
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [Supabase vs Convex in production: reliability and cost](https://getpraxen.app/learn/reliability/supabase-vs-convex-in-production/): Supabase vs Convex in production: compare data models, realtime, consistency, scaling, migrations, monitoring, backups and costs before choosing your backend.
---
### How to monitor a Django app on Render
> Monitor a Django app on Render: JSON logs, a real health check, failure alerts, log and metrics streams, OpenTelemetry under Gunicorn and Celery checks.
Part 1 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 30, 2026 · 14 min read
Canonical: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/
Start with what Render already collects: logs from stdout and stderr on every service, CPU, memory and HTTP metrics, and health checks on the web service. Then log JSON with Render's Rndr-Id request ID, point an HTTP health check at a Django view that queries the database, turn on failure notifications, and stream logs and metrics out before retention ends. Add OpenTelemetry tracing set up in a Gunicorn post_fork hook, and heartbeats for Celery tasks and cron jobs, because health checks never reach them.
To monitor a Django app on Render, start with what the platform already collects, then close the gaps it leaves. Render captures everything your web service, Celery worker and cron jobs write to stdout and stderr. It charts CPU, memory and HTTP traffic, and it probes web services with health checks. Your job is to make those logs searchable by request, give the health check a path that proves Django can reach its database, turn on failure notifications, and stream logs and metrics to a tool that keeps them longer than Render does.
Two things stay invisible until you add them: what happens inside a slow request, and whether a background job finished its work. For the first, add OpenTelemetry tracing to Django, set up so it works with the Gunicorn `--preload` flag Render sets by default. For the second, give each job that matters a heartbeat. The steps below take a typical project (a Gunicorn web service, a Celery worker, a cron job and Render Postgres) from default settings to alerts that reach you.
#### What Render already watches on each Django service
Each part of a Django deployment runs as its own Render service, and each one gets different signals:
- **Web service running Gunicorn and Django.** Runtime logs, CPU and memory, HTTP request volume from the public internet, and a health check every few seconds. A Pro workspace or higher adds a log line for every public HTTP request and a Response Times graph at p50, p75, p90 and p99.
- **Background worker running Celery.** Runtime logs, CPU and memory. Health checks only cover web services and private services, so nothing probes a worker.
- **Cron job.** Logs and run history on its Runs page, plus a notification when a run fails. Render doesn't emit streamed metrics for cron jobs.
- **Render Postgres.** Active connections, transaction volume, replication lag once you add a read replica, lock-delayed queries, running processes and top queries.
None of this tells you which view is slow and why, or whether the task that sends receipts actually ran. The steps below cover both.
#### Step 1: Log JSON to stdout with Render's request ID on every line
Render needs no SDK for logs, because it reads stdout and stderr. What it can't do is link your log lines to the request that produced them. For every public request, Render writes a `requestID` into its HTTP request log. It sends the same value to your service in the `Rndr-Id` header and returns it to the client in the response. Cloudflare also sets a `CF-Ray` header on every inbound request, and Render's uptime guide recommends logging it. I led the Workers observability team at Cloudflare, and on any platform I'd make this the first change: put the platform's request ID into your own log lines.
Store both IDs in context variables, so every log line written during the request picks them up:
```python
# myproject/log_context.py
import contextvars
import json
import logging
import os
request_id = contextvars.ContextVar("request_id", default="-")
cf_ray = contextvars.ContextVar("cf_ray", default="-")
COMMIT = os.environ.get("RENDER_GIT_COMMIT", "")[:7]
INSTANCE = os.environ.get("RENDER_INSTANCE_ID", "")
class JsonFormatter(logging.Formatter):
def format(self, record):
payload = {
"level": record.levelname.lower(),
"logger": record.name,
"message": record.getMessage(),
"requestId": request_id.get(),
"cfRay": cf_ray.get(),
"commit": COMMIT,
"instance": INSTANCE,
}
if record.exc_info:
payload["exception"] = self.formatException(record.exc_info)
return json.dumps(payload)
class RequestIdMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
rid = request_id.set(request.headers.get("Rndr-Id", "-"))
ray = cf_ray.set(request.headers.get("CF-Ray", "-"))
try:
return self.get_response(request)
finally:
request_id.reset(rid)
cf_ray.reset(ray)
```
Put the middleware first in `MIDDLEWARE` and send Django's logging to stdout:
```python
# settings.py
MIDDLEWARE = [
"myproject.log_context.RequestIdMiddleware",
"django.middleware.security.SecurityMiddleware",
# ...
]
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {"json": {"()": "myproject.log_context.JsonFormatter"}},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout",
"formatter": "json",
},
},
"root": {"handlers": ["stdout"], "level": os.environ.get("LOG_LEVEL", "INFO")},
}
```
Render sets `RENDER_GIT_COMMIT` and `RENDER_INSTANCE_ID` on every service by default, so each line now records which deploy and which instance wrote it. Gunicorn's access log needs no setup. Render's Python runtime sets `GUNICORN_CMD_ARGS` to `--preload --access-logfile - --bind=0.0.0.0:10000`, which already sends access lines to stdout.
The log explorer can now answer questions like these. Its search box accepts wildcards, and RE2 regular expressions between slashes:
```text
8ebfa3c3-8929-4885 # every line for one request
status_code:/5../ # request logs with a 5xx status
path:api/orders/* # requests to one endpoint
/responseTimeMS=\d{3}\d+/ # requests slower than one second
```
To watch lines as they arrive during a deploy, use Live tail in the dashboard or run `render logs` from the Render CLI.
#### Step 2: Point the health check at a view that queries the database
By default, Render's health check is a TCP probe to your service's port. It passes once Gunicorn accepts a connection, even if every view raises an error. An HTTP health check makes Django prove it can do real work, and Render suggests a simple database query for it.
```python
# myproject/health.py
from django.db import connection
from django.http import JsonResponse
def healthz(request):
try:
with connection.cursor() as cursor:
cursor.execute("SELECT 1")
except Exception:
return JsonResponse({"status": "unhealthy"}, status=503)
return JsonResponse({"status": "ok"})
```
```python
# urls.py
from myproject.health import healthz
urlpatterns = [
path("healthz/", healthz),
# ...
]
```
Then set the path in either of two places:
1. In the Render Dashboard, open the web service's **Settings** page, scroll to **Health Checks**, click **Edit**, enter `/healthz/` and click **Save Changes**.
2. In your Blueprint, add `healthCheckPath` to the web service:
```yaml
services:
- type: web
name: django-web
startCommand: gunicorn myproject.wsgi:application -c gunicorn.conf.py
healthCheckPath: /healthz/
```
Check it from your machine. It should print `200`:
```bash
curl -s -o /dev/null -w "%{http_code}\n" https://your-service.onrender.com/healthz/
```
##### What Render does when the check fails
A check passes when the instance returns any 2xx or 3xx status within five seconds. Anything else counts as a failure. If a running instance fails consecutive checks for 15 seconds, Render stops sending it traffic, and any healthy instances keep serving. If the failures last 60 seconds, Render restarts the instance and notifies you. A new deploy only receives traffic once all its new instances pass at the same time. If that hasn't happened within 15 minutes, Render cancels the deploy and the old version keeps serving.
**Figure 1. What happens to a Render instance that keeps failing its health check**
From Render's health check docs: routing stops after 15 seconds of failed checks, and the instance restarts at 60 seconds.
| Span | Starts at | Duration |
| --- | --- | --- |
| Checks failing (still receives traffic) | 0 s | 15 s |
| Out of routing (restarted at 60 s) | 15 s | 45 s |
| **Total** | | **60 s** |
##### Make sure Django accepts the health check's Host header
Render sets the health check's `Host` header to one of your verified custom domains, or to the service's `onrender.com` subdomain if you have none. If that host is missing from `ALLOWED_HOSTS`, Django rejects the request and every deploy fails its checks. Add Render's hostname from the environment:
```python
ALLOWED_HOSTS = ["www.example.com"]
if os.environ.get("RENDER_EXTERNAL_HOSTNAME"):
ALLOWED_HOSTS.append(os.environ["RENDER_EXTERNAL_HOSTNAME"])
```
Keep the view cheap, and check only what this instance needs to serve traffic. A stopped Celery worker or a slow payment API is real trouble, but restarting web instances won't fix either one, so they belong in Step 7.
#### Step 3: Turn on failure notifications for every service
From your workspace home, click **Integrations > Notifications** and set the default to **Only failure notifications**, by email, Slack or both. At that level, Render tells you when:
- a build or deploy fails
- a cron job run fails
- a running service becomes unhealthy
- a persistent disk goes past 80% of its allocated space
- Render suspends a service that keeps failing to start and become healthy over a 24-hour period
**All notifications** adds Slack messages for deploys that go live and services that recover, which suits a deploy channel. You can override the level for each service, and Render webhooks can send events anywhere else.
#### Step 4: Read metrics, logs and deploys over the same time range
When a graph on the Metrics page moves, you find the cause fastest by lining up three views over the same window:
1. **Metrics.** Check CPU, memory, request volume grouped by status code, and response times. Memory that climbs steadily and ends in restarts suggests a leak. A jump in 5xx responses points at code or a dependency.
2. **Logs.** Open the log explorer for that window and filter with `status_code:/5../` or the view's path. Click an instance ID to see only that instance's lines.
3. **Deploys.** Open the service's **Events** page, find the deploy just before the change, and read that deploy's logs. Health check failures during a deploy explain why a new version never went live.
On the database, watch **Active Connections** and **Lock-Delayed Queries**. The second counts completed queries that waited one second or longer on a lock. Django keeps a separate database connection for each worker thread, so your database must allow at least as many connections as you have worker threads. Render sets `WEB_CONCURRENCY` from the plan's CPU count (1 on `0.5c-512mb`, 2 on `2c-4g`), so more instances or more workers mean more connections. The **Queries** tab lists running processes and the most frequent queries.
Watch for two blind spots. HTTP metrics and request logs only include requests from the public internet, so calls between your services over the private network don't appear. And the outbound bandwidth graph only updates once an hour.
#### Step 5: Stream logs and metrics out before Render's retention ends
Render keeps logs and metrics for a period set by your workspace plan. Older logs are deleted, and upgrading later doesn't bring them back.
**Render log and metrics retention by workspace plan**
| Workspace plan | Log retention | Metrics retention |
| --- | --- | --- |
| Hobby | 7 days | 7 days |
| Pro | 14 days | 14 days |
| Scale / Enterprise | 30 days | 30 days |
Render also processes at most 6,000 application log lines per minute for each running instance. It drops anything above that from both the explorer and log streams. Keep SQL logging and `DEBUG`-level logging off in production.
To keep logs longer and alert on them, add a log stream:
1. From your workspace home, click **Integrations > Observability** and scroll to **Log Streams**.
2. Under **Default destination**, click **+ Set default**.
3. Enter your provider's endpoint: `HOST:PORT` for syslog, or the full URL for HTTPS.
4. Add a token if your provider needs one, click **Save Changes**, and choose whether to include preview instances.
Syslog streams use TLS over TCP in RFC5424 format. On Pro you can leave individual services out of the stream, and on Scale you can send a service to its own destination. A log stream only carries lines. To join them up with other data, use the IDs you logged in Step 1, and have your provider parse `requestId` and `commit` as fields.
For metrics, a workspace admin on Pro or higher can open **Observability**, go to **Metrics Stream** and click **+ Add destination**. Render lists Better Stack, Datadog, Grafana, Groundcover, Honeycomb, New Relic, Pydantic Logfire and SigNoz, plus a custom OpenTelemetry endpoint. Datadog receives metrics in its own format and needs an organisation-level API key. Neither stream counts against your outbound bandwidth.
#### Step 6: Trace Django with OpenTelemetry, set up for Gunicorn preload
A trace shows where a slow request spent its time, broken down by view and by query. The `opentelemetry-instrumentation-django` package instruments Django with a single call to `DjangoInstrumentor().instrument()`.
On Render, the catch is `--preload`. With preload, Gunicorn loads your app in the master process and then forks the workers. OpenTelemetry's `BatchSpanProcessor` exports spans from a background thread, and it is not fork-safe: the child process inherits a lock the parent holds, and it deadlocks. The fix is to instrument Django at import time, then create the tracer provider and span processor in a Gunicorn `post_fork` hook, which runs in each worker after the fork.
```bash
pip install opentelemetry-sdk opentelemetry-instrumentation-django opentelemetry-exporter-otlp
```
```python
# myproject/wsgi.py
import os
from django.core.wsgi import get_wsgi_application
from opentelemetry.instrumentation.django import DjangoInstrumentor
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
DjangoInstrumentor().instrument()
application = get_wsgi_application()
```
```python
# gunicorn.conf.py
import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
def post_fork(server, worker):
resource = Resource.create(attributes={
"service.name": os.environ.get("RENDER_SERVICE_NAME", "django-web"),
"service.version": os.environ.get("RENDER_GIT_COMMIT", "unknown"),
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint=os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"]))
)
trace.set_tracer_provider(provider)
```
Set `OTEL_EXPORTER_OTLP_ENDPOINT`, plus any credentials your tracing backend needs, as environment variables on the service. Start Gunicorn with `-c gunicorn.conf.py`, as in the Blueprint above. Because every span carries the commit SHA, you can compare latency before and after each deploy. To turn off the Django instrumentation without a code change, set `OTEL_PYTHON_DJANGO_INSTRUMENT=False`.
#### Step 7: Give Celery tasks and cron jobs a heartbeat
Health checks never reach a background worker, and a cron job that hangs never fails. Render guarantees at most one active run per cron job. A hung run therefore delays every scheduled run behind it until Render stops it after 12 hours. Worker logs and CPU graphs tell you a process is alive. They can't tell you the job did its work.
Save a timestamp in a shared cache, such as Render Key Value, each time a job succeeds:
```python
# billing/tasks.py
from celery import shared_task
from django.core.cache import cache
from django.utils import timezone
@shared_task
def sync_payments():
run_payment_sync() # the real work
cache.set("heartbeat:sync_payments", timezone.now().timestamp(), timeout=None)
```
Then expose how old each heartbeat is, and return 503 when one is stale:
```python
# myproject/health.py (continued)
import time
from django.core.cache import cache
MAX_AGE_SECONDS = {"sync_payments": 15 * 60, "nightly_report": 26 * 60 * 60}
def jobs_health(request):
now = time.time()
stale = {}
for job, max_age in MAX_AGE_SECONDS.items():
last = cache.get(f"heartbeat:{job}")
if last is None or now - last > max_age:
stale[job] = None if last is None else int(now - last)
return JsonResponse({"stale": stale}, status=503 if stale else 200)
```
Write the heartbeat after the work succeeds. It then proves the whole path from scheduler to finished task. Keep `/healthz/jobs/` out of Render's health check path: a stale heartbeat should page a person, and restarting web instances won't restart the worker. For cron jobs, make sure the command exits when it finishes. The Runs page shows the logs for each run, and **Trigger Run** starts one on demand, cancelling any run already in progress.
#### Step 8: Probe the app from outside Render
Render's health checks run inside the platform. An external probe sends requests from outside, which is closer to what your users experience, and Render's uptime guide recommends one. Point it at a real page that reads from the database, at `/healthz/jobs/`, and at your custom domain, so a broken DNS record or certificate also trips it. When the probe fails, its notification includes the `CF-Ray` ID, which you can search for in the logs from Step 1. The same guide recommends running more than one instance, so the site stays up while Render moves a service off a failed node.
#### The alerts worth paging on for Django on Render
- **Deploy failed or service unhealthy**, from Render notifications.
- **External probe failing** on the main page or the jobs endpoint.
- **5xx rate rising**, built in your log tool from the status codes in request logs.
- **p99 latency above its usual level**, from streamed metrics or traces.
- **Memory climbing steadily** on web or worker instances.
- **Job heartbeat stale** for any task users depend on.
- **Disk past 80%**, from Render notifications.
- **Postgres connections near the database's limit**, or a jump in lock-delayed queries.
Page on failed deploys, the external probe, the 5xx rate and stale heartbeats. Send the rest to a channel someone reads every day.
#### Where Django monitoring on Render goes wrong
- **The health path redirects.** Any 3xx counts as healthy, so a path that redirects to HTTPS or to a login page passes even when Django can't reach the database. Fix: have `/healthz/` return 200 or 503 itself, with no auth and no redirect.
- **The custom domain is missing from ALLOWED_HOSTS.** Once you verify a domain, Render uses it as the health check's Host. Fix: list both the custom domain and `RENDER_EXTERNAL_HOSTNAME`.
- **Tracing starts before the fork.** Under `--preload`, spans stop arriving or workers hang. Fix: create the tracer provider in `post_fork`.
- **Log lines vanish during an incident.** A burst above 6,000 lines per minute per instance gets dropped. Fix: log one structured line per event at INFO, and keep query logging off.
- **Logs expire before anyone reads them.** Fix: set up a log stream on day one.
- **The health check calls a third-party API.** When that API takes more than five seconds, Render pulls healthy instances out of routing. Fix: check only what the instance needs to serve traffic.
- **Worker failures page nobody.** Fix: add heartbeats and an external check on the jobs endpoint.
- **The web service runs one instance.** Fix: scale to more than one, so a health check restart doesn't take the site offline.
#### Watching Render services and Celery workers with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Render, so when a web service, Celery worker or cron job breaks it works out why, and the fix arrives ready for your team's normal review.
Running on Render? See [how Praxen monitors Render in production](https://getpraxen.app/for/render/).
#### Common questions
**Does Render monitor a Django app without any setup?**
Partly. Render captures stdout and stderr from web services, background workers and cron jobs, charts CPU and memory, and probes web services with a TCP health check by default. Its log streams don't create full tracing or APM correlation on their own, and it doesn't check that Celery tasks finished, so add OpenTelemetry and job heartbeats for those.
**How long does Render keep logs and metrics?**
Both are kept for 7 days on Hobby, 14 days on Pro and 30 days on Scale or Enterprise. Logs older than that are gone even if you upgrade later, so set up a log stream to a provider that keeps them longer.
**Why does my Django health check fail on Render?**
Most often, Django rejects the Host header because it isn't in ALLOWED_HOSTS. Render sets that header to your verified custom domain, or to the service's onrender.com subdomain if you have none. Other causes are a 4xx or 5xx response from the view, or a response slower than five seconds. On a new deploy, if the new instances don't all pass together within 15 minutes, Render cancels the deploy and keeps the old version running.
**How do I monitor Celery workers on Render?**
Run Celery as a background worker. That gives you logs and CPU and memory graphs, but no health checks. Have each important task write a timestamp to a shared cache when it succeeds, expose the ages on a web endpoint that returns 503 when one is stale, and point an external monitor at that endpoint.
**Which Render monitoring features need a Pro workspace?**
HTTP request logs in the log explorer, the Response Times latency graph, metrics streaming to an OpenTelemetry provider, and leaving individual services out of a log stream all need Pro or higher. Sending one service's logs to its own destination needs Scale or higher.
**Can I send Render metrics to Datadog or Grafana?**
Yes, on Pro or higher. A workspace admin adds a Metrics Stream under Observability. Render pushes metrics to Datadog in Datadog's own format using an organisation-level API key, and to Grafana over OTLP using your stack's endpoint and a token that starts with glc_. Logs go through a separate log stream.
**Why do my OpenTelemetry traces stop when Django runs under Gunicorn on Render?**
Render's Python runtime starts Gunicorn with --preload, so your app loads before the workers fork. OpenTelemetry's BatchSpanProcessor isn't fork-safe and can deadlock in the child process. Create the tracer provider and span processor in a Gunicorn post_fork hook, and keep DjangoInstrumentor().instrument() in wsgi.py.
**Why are some of my log lines missing on Render?**
Render processes up to 6,000 application log lines per minute for each instance and drops the excess from both the log explorer and log streams. Turn off DEBUG and SQL query logging in production, and log one structured line per event.
#### Sources
- [Logs in the Render Dashboard](https://render.com/docs/logging)
- [How Render handles logging and observability](https://render.com/articles/how-render-handles-logging-and-observability)
- [Service Metrics (Render Docs)](https://render.com/docs/service-metrics)
- [Health Checks (Render Docs)](https://render.com/docs/health-checks)
- [Streaming Render Service Logs](https://render.com/docs/log-streams)
- [Email and Slack Notifications (Render Docs)](https://render.com/docs/notifications)
- [Streaming Render Service Metrics](https://render.com/docs/metrics-streams)
- [Default Environment Variables (Render Docs)](https://render.com/docs/environment-variables)
- [Best Practices for Maximizing Uptime (Render Docs)](https://render.com/docs/uptime-best-practices)
- [Cron Jobs (Render Docs)](https://render.com/docs/cronjobs)
- [Django Instrumentation (OpenTelemetry Python)](https://opentelemetry-python.readthedocs.io/en/latest/examples/django/README.html)
- [Working With Fork Process Models (OpenTelemetry Python)](https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html)
- [Databases: persistent connections (Django documentation)](https://django.readthedocs.io/en/stable/ref/databases.html)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Next: [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
#### Related
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [Fix Render “Port scan timeout reached, no open ports detected”](https://getpraxen.app/learn/troubleshooting/render-port-scan-timeout-reached-no-open-ports-detected/): Why a Render deploy fails with “Port scan timeout reached, no open ports detected”, how to spot the cause in the logs, and how to bind to 0.0.0.0 on PORT.
---
### How to monitor a Supabase app in production
> Monitor a Supabase app in production: scrape the Metrics API into Prometheus, drain logs, run SQL health checks, trace requests and alert on what matters.
Part 3 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 29, 2026 · Updated September 30, 2026 · 14 min read
Canonical: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/
Scrape the project's Metrics API into Prometheus or another compatible tool, send its logs to your alerting tool with a log drain, run Supabase's SQL detection checks on a schedule, and add trace headers to your app's requests so its traces line up with Supabase's logs. Then alert on a short list of signals: API and Auth server errors, connection pressure, blocked or long-running queries, slow statements, replication lag and database growth.
To monitor a Supabase app in production, get four kinds of signal out of the project and into the tool you already alert from. Those are database metrics from the Metrics API, platform logs through a log drain, scheduled SQL checks against Postgres, and traces from your own app that carry a trace ID into Supabase's logs. Then write a short list of alerts on the signals that actually break apps: server errors on the API and Auth, connection pressure, blocked or long-running queries, slow statements, replication lag and database growth.
The dashboards and Logs Explorer in Supabase Studio work well for a quick health check. They live apart from your app's own telemetry, though, so you can't line up a latency spike in your API with a saturated connection pool. The steps below take a project from dashboards you have to go and look at to alerts that come to you.
#### What monitoring a Supabase app covers
If you are still choosing the backend, compare the operating models in [Supabase vs Convex in production](/learn/reliability/supabase-vs-convex-in-production/) before translating these Postgres checks into a function-based monitoring plan.
A Supabase project is several services behind one URL: Postgres, the API gateway and PostgREST, Auth, Storage, Edge Functions and Realtime. Your app adds a frontend and usually a server of its own. Each part fails in its own way, and Supabase exposes each one through a different route:
- **The Metrics API** publishes roughly 200 Postgres performance and health metrics in Prometheus exposition format. They cover CPU and memory, disk I/O and WAL, Supavisor and Postgres connection pools, and query performance. The whole set refreshes every minute.
- **Logs** come from every subsystem and can be queried with SQL in Explorer. Log drains stream them to another tool on the Pro, Team and Enterprise plans.
- **Live Postgres statistics** come from `pg_stat_activity` and `pg_stat_statements`. You read them with the Supabase CLI, SQL in Explorer, or `execute_sql` on the Supabase MCP server.
- **Advisors** return security and performance findings, each with a severity, the affected object and a documentation link.
- **Client-side tracing** in the Supabase SDKs adds W3C trace headers to outgoing requests, so a `trace_id` from your app appears in API Gateway and Edge Function logs.
One detail shapes everything below. `pg_stat_activity` is a live snapshot. `pg_stat_statements` and the cache counters are cumulative since their last reset. A snapshot tells you what is happening right now. A cumulative counter only tells you about a time window once you compare two saved readings.
I built and led the Workers observability team at Cloudflare, and the advice I would give for any hosted platform applies here too: get its telemetry into the tool that already holds your app's traces and alerts before you write a single alert rule.
#### The signals that break Supabase apps first
With about 200 metrics on offer, most teams start with too many charts and no pages at all. These are the signals worth an alert or a scheduled check:
- **Server errors on the API and Auth.** The 5xx rate on API Gateway and Auth, each measured on its own. This is the closest thing to what users feel.
- **Access failures.** A jump in 401 and 403 responses usually points at a broken client flow or a policy change.
- **Connection pressure.** Client connections against `max_connections`. Reserved slots, role limits and pooler limits can cut a client off before the instance-wide number looks full.
- **Blocked and long-running sessions.** Queries active for a long time, transactions left idle, and sessions waiting on another session's lock.
- **Slow statements.** Mean execution time per statement from `pg_stat_statements`, compared hour on hour. Deploys that drop an index show up here first.
- **CPU, memory and disk I/O.** Supabase suggests a right-sizing alert when CPU or memory stays above 80%. High I/O wait often means missing indexes or queries that scan too much data.
- **Replication lag**, if you run read replicas.
- **Database size growth**, which warns you well before the disk fills.
- **Advisor findings** at `WARN` or `ERROR` severity.
#### Step 1: Scrape the Metrics API into Prometheus
Every hosted project has a metrics endpoint at `https://.supabase.co/customer/v1/privileged/metrics`, protected with HTTP Basic Auth and a secret API key. If you use Grafana Cloud, the integration in the Supabase Dashboard sets this up for you. Datadog can read the same endpoint through the Datadog Agent's OpenMetrics integration, and AWS Managed Prometheus works as well. For a self-hosted Prometheus and Grafana:
1. Copy a secret API key (it starts with `sb_secret_`) into your secret manager or an environment variable.
2. Check that the endpoint answers:
```bash
curl -s -u "username:$SUPABASE_SECRET_KEY" \
"https://$PROJECT_REF.supabase.co/customer/v1/privileged/metrics" | head -n 20
```
You should see plain-text lines in Prometheus exposition format, one metric per line. An authentication error means the key or the username is wrong.
3. Add a scrape job to `prometheus.yml`:
```yaml
scrape_configs:
- job_name: 'supabase'
scrape_interval: 60s
metrics_path: /customer/v1/privileged/metrics
scheme: https
basic_auth:
username: username
password: ''
static_configs:
- targets:
- '.supabase.co:443'
labels:
project: ''
```
4. Keep `scrape_interval` at 60 seconds to match the refresh. Add one job per project ref so each project's labels stay separate. If Prometheus sits behind a proxy, allow outbound HTTPS to `*.supabase.co`.
5. In Grafana, add Prometheus as a data source under Connections, Data sources. Then import `dashboard.json` from the `supabase-grafana` repository under Dashboards, New, Import. That gives you over 200 panels covering CPU, I/O, WAL, replication, index bloat and query throughput.
#### Step 2: Turn the metrics into alerts that page someone
The `supabase-grafana` repository ships example PromQL rules. Three of them are worth loading on day one:
```yaml
groups:
- name: supabase
rules:
- alert: PostgresDatabaseDown
expr: pg_up == 0
for: 5m
labels:
severity: critical
- alert: PostgresReplicationLagHigh
expr: (physical_replication_lag_physical_replication_lag_seconds > 600) and (rate(physical_replication_lag_physical_replication_lag_seconds[10m]) > 0)
for: 5m
labels:
severity: warning
- alert: PostgresDatabaseSizeGrowth
expr: (pg_database_size_mb - pg_database_size_mb offset 12h) / pg_database_size_mb offset 12h * 100 > 20
for: 5m
labels:
severity: warning
```
The first fires when the instance is still sending metrics but the database has been down for 5 minutes. The second fires when replication lag passes 10 minutes and is still climbing. The third fires when the database has grown by 20% or more in 12 hours.
On top of those, add a connection saturation alert that fires as pools near capacity, so you have a few minutes to investigate before connections start failing. Add a right-sizing alert for CPU or memory held above 80%, which gives you time to upgrade before users notice. Set thresholds for disk use and long-running transactions to fit your project's size. Route everything through Alertmanager, Grafana OnCall, PagerDuty or whichever tool already wakes your on-call.
**Starter alert thresholds for a Supabase project**
| Signal | Starting threshold | Source of the default |
| --- | --- | --- |
| Database down | pg_up equals 0 for 5 minutes | supabase-grafana example alerts |
| Replication lag | Over 600 seconds and still rising | supabase-grafana example alerts |
| Database size | 20% or more growth in 12 hours | supabase-grafana example alerts |
| CPU or memory | Held above 80% | Supabase Metrics API guidance |
| Client connections | 80% of max_connections | Supabase detection checks |
| API or Auth server errors | 20 or more errors, at least 1%, and double the previous hour | Supabase detection checks |
| Slow statement | Mean of 100 ms or more and double the previous hour, with 20 or more calls | Supabase detection checks |
| Session age | Active or idle in transaction for over 30 seconds | Supabase detection checks |
#### Step 3: Drain Supabase logs into the tool you alert from
Metrics tell you something is wrong. Logs tell you which request failed. Log drains send the logs of the whole stack (Postgres, Auth, Storage, Edge Functions and the rest) to one or more destinations. Set them up under Project Settings, Log Drains. They are available on the Pro, Team and Enterprise plans, and Supabase bills them per drain, per million events and for egress.
The destinations are a custom HTTP endpoint, OpenTelemetry over OTLP, Datadog, Grafana Loki, Amazon S3, Sentry, Axiom, Last9 and Syslog. HTTP destinations receive batched POST requests of up to 250 events, or whatever has arrived within 1 second, whichever comes first. A few details matter when you pick one:
- **OTLP** needs an endpoint that accepts `application/x-protobuf` at `/v1/logs`. It works with the OpenTelemetry Collector, Grafana Cloud, New Relic, Honeycomb, Datadog's OTLP ingestion and Elastic.
- **Loki** must accept structured metadata, and you should raise the maximum number of structured metadata fields to at least 500 so large events are not rejected.
- **Sentry** receives the events in its Logging product. They do not arrive as Sentry errors.
- **Custom endpoints** currently receive unsigned requests, so add a secret header and check it on your side.
- **S3** suits long-term retention and compliance archives next to a live destination.
Once logs arrive, build alerts in that tool on error-level events from Auth, Edge Functions and Postgres. A drop in volume is worth an alert too.
#### Step 4: Schedule Supabase's SQL detection checks
Supabase publishes a set of detection checks for health, security, performance and capacity. Its own monitoring agents use the same checks. Each check returns **finding**, **clear**, or **unable to assess** along with the input that was missing. Run them from a cron job, a scheduled CI workflow or an agent with project-scoped, read-only access (the Supabase MCP server takes `read_only=true`). Record the observation time, the window and the thresholds with every result.
##### Server errors on the API and Auth
Run this against the logs source for the last complete UTC hour, then again for the hour before it:
```sql
select source,
countIf(status between 100 and 599) as responses,
countIf(status between 500 and 599) as server_errors,
countIf(status in (401, 403)) as access_failures
from (
select source,
toInt32OrNull(if(source = 'edge_logs',
log_attributes['response.status_code'], log_attributes['status'])) as status
from logs
where source in ('edge_logs', 'auth_logs')
)
group by source;
```
Work out the error rate per source. Supabase's starting policy reports a finding at 20 or more server errors, a rate of at least 1%, and at least double the previous hour's rate. Both windows need at least 100 responses, or the result is unable to assess. Run the same query over the last complete day to track 401 and 403 access failures against the day before.
##### Connection pressure
```sql
select
count(*) filter (where backend_type = 'client backend') as client_connections,
current_setting('max_connections')::int as max_connections
from pg_stat_activity;
```
Report a finding when client connections reach 80% of `max_connections`. If connections crowd the limit often, move chatty or bursty workloads to the dedicated pooler with sensible pool sizes.
##### Blocked and long-running sessions
```sql
select pid, usename as role, state,
now() - query_start as query_age,
now() - xact_start as transaction_age,
pg_blocking_pids(pid) as blocking_pids
from pg_stat_activity
where datname = current_database()
and pid <> pg_backend_pid()
and (
(state = 'active' and now() - query_start > interval '30 seconds')
or (state like 'idle in transaction%' and now() - xact_start > interval '30 seconds')
or cardinality(pg_blocking_pids(pid)) > 0
)
order by query_start
limit 20;
```
Every row needs a look. A non-empty `blocking_pids` names the blockers. If you get 20 rows back, the list may be cut short.
##### Slow statements
With `pg_stat_statements` enabled, save a snapshot of `calls` and `total_exec_time` per `(dbid, userid, queryid, toplevel)` every hour. Divide the change in total time by the change in calls for each interval. Supabase's policy flags a statement whose mean reaches 100 ms and doubles the previous hour, with at least 20 calls in each interval. Throw out comparisons across a reset, an upgrade or an eviction. When a statement is flagged, read its query plan.
##### Advisors
Call `get_advisors` with `type: "security"` and again with `type: "performance"`. Report `WARN` and `ERROR` findings with the lint name, the affected object and the documentation link. Keep `INFO` as context. Run the advisor again after each fix. An empty result does not prove the project is secure.
##### From the terminal
The Supabase CLI wraps many of the same statistics:
```bash
supabase link --project-ref
supabase inspect db long-running-queries
supabase inspect db blocking
supabase inspect db outliers
supabase inspect db role-connections
supabase inspect db replication-slots
```
For disk pressure, `bloat`, `vacuum-stats`, `table-sizes` and `index-sizes` show where the space went. For query work, `unused-indexes`, `index-usage` and `seq-scans` point at indexing problems.
#### Step 5: Carry your app's trace IDs into Supabase logs
Metrics and logs cover the Supabase side. To follow one slow page load from the browser through PostgREST or an Edge Function, the request needs a trace ID that both sides record. The Supabase JS, Swift, Dart and Python SDKs can attach W3C `traceparent`, `tracestate` and `baggage` headers, and the resulting `trace_id` appears in API Gateway and Edge Function logs.
In JavaScript you need `@supabase/supabase-js` 2.106.0 or later, `@opentelemetry/api` at runtime, and a registered tracer provider. From 2.112.0, load the tracing runtime once at your entry point:
```ts
import '@supabase/supabase-js/tracing'
import { trace } from '@opentelemetry/api'
import { createClient } from '@supabase/supabase-js'
const supabase = createClient(SUPABASE_URL, SUPABASE_KEY, {
tracePropagation: true,
})
const tracer = trace.getTracer('my-app')
await tracer.startActiveSpan('load-orders', async (span) => {
const { data, error } = await supabase.from('orders').select('*')
span.end()
})
```
The SDK only adds trace headers to requests for `*.supabase.co`, `*.supabase.in` and `localhost`. When you call Edge Functions from the browser, add `traceparent`, `tracestate` and `baggage` to the function's CORS allow-list, or import `corsHeaders` from `npm:@supabase/supabase-js@^2.112.3/cors`, then redeploy.
Vendor SDKs differ. The OpenTelemetry SDK, and Honeycomb, Grafana or New Relic over OTLP, work with no change. Supabase's tracing guide lists `propagateTraceparent: true` as a requirement for Sentry. In the browser it also needs your project URL in `tracePropagationTargets`, plus `sentry-trace` in the Edge Function's CORS allow-list. Datadog's `dd-trace` for Node.js adds W3C headers by itself. Datadog Browser RUM needs your project URL in `allowedTracingUrls` with the `tracecontext` propagator.
#### Where Supabase monitoring setups go wrong
##### Scraping faster than the data changes
The Metrics API refreshes once a minute. A shorter scrape interval stores repeated values and costs storage for nothing. **Fix:** keep `scrape_interval: 60s`.
##### Reading silence as health
If the scrape job breaks, `pg_up == 0` never fires, because no data arrives at all. The same goes for logs: Supabase's checks say zero recorded events alone does not prove service health, and a missing source row calls for a capture check. **Fix:** alert when the scrape job fails and when a log source goes quiet. Treat an unable-to-assess result as something to act on.
##### Adding API and Auth errors together
API Gateway events and Auth events are separate observations of different requests. Summing them blurs both rates. **Fix:** compute and alert on each source by itself, and report unknown status codes separately.
##### Resetting statistics to get a clean baseline
Resetting `pg_stat_statements` wipes the history every comparison depends on. **Fix:** save hourly snapshots and compare deltas. Start collecting now if you have no history.
##### Cancelling a query because it is old
A long query or a wait event alone does not make a session a blocker, and query age is a different thing from lock-wait time. **Fix:** act on non-empty `blocking_pids`, find out what the transaction is for, and run the check again to confirm the problem has cleared.
##### Trace IDs that never appear in the logs
The SDK never throws when it can't propagate, so a bad setup fails quietly. The usual causes are a missing `@supabase/supabase-js/tracing` import on 2.112.0 or later, a call made outside any active span, no registered tracer provider, a Sentry setup without `propagateTraceparent: true`, or the CDN build, which cannot load the tracing runtime. **Fix:** look for the SDK's one-time console warning, wrap calls in spans, and check each item on that list in turn.
##### Leaving the secret key in a config file
The Metrics API authenticates with a secret (service-role) API key. **Fix:** inject it from a secret manager or environment variable, rotate it on a schedule, and update Prometheus when you do.
##### Building monitoring tables inside the project
Writing snapshots back into the database you are watching adds load and tangles the two together. **Fix:** Supabase's guidance is to keep saved readings in persistent storage outside the project, or in a historical metrics source you are allowed to use.
##### Planning on log drains from the Free plan
Drains need the Pro, Team or Enterprise plan. **Fix:** until you upgrade, rely on the Metrics API and scheduled log queries run through Explorer or the MCP `query_logs` tool.
#### A checklist before the app takes real traffic
1. Metrics API scraped every 60 seconds, one job per project, with the key in a secret manager.
2. Supabase Grafana dashboard imported, or the Grafana Cloud or Datadog equivalent set up.
3. Alerts live for database down, replication lag, size growth, connection saturation and sustained CPU or memory, routed to on-call.
4. An alert on the scrape job failing and on log sources going quiet.
5. Log drain sending to your alerting tool, with S3 added if you need an archive.
6. Hourly API and Auth error checks, a daily access-failure check, and connection and blocker snapshots, all running read-only.
7. `pg_stat_statements` enabled, with hourly snapshots saved outside the project.
8. Security and performance advisors run on a schedule, with `WARN` and `ERROR` findings sent to a person.
9. Trace propagation on in the app, with trace headers in the Edge Function CORS allow-lists.
#### Letting Praxen watch the Supabase project
Praxen detects production issues and fixes them by opening pull requests. It connects to Supabase, so when a database, edge function or auth flow breaks it works out why, and the fix arrives ready for your team's normal review.
Running on Supabase? See [how Praxen monitors Supabase in production](https://getpraxen.app/for/supabase/).
#### Common questions
**Does Supabase have built-in monitoring?**
Yes. Studio has observability dashboards, a Logs Explorer you can query with SQL, reports and Advisors, and the CLI reads live Postgres statistics with `supabase inspect db`. For alerting and for lining up database signals with your app's telemetry, export the data through the Metrics API and log drains.
**What is the Supabase Metrics API endpoint and how often should I scrape it?**
Each hosted project exposes `https://.supabase.co/customer/v1/privileged/metrics`, protected with HTTP Basic Auth and a secret API key. It publishes roughly 200 Postgres metrics in Prometheus format and refreshes every minute, so scrape it every 60 seconds.
**Which Supabase plans include log drains?**
Log drains are available on the Pro, Team and Enterprise plans, under Project Settings, Log Drains. Supabase bills them per drain, per million events and for egress. On the Free plan, use the Metrics API and scheduled log queries.
**Can I monitor Supabase with Datadog or Grafana Cloud?**
Yes. Grafana Cloud has an integration in the Supabase Dashboard. Datadog can read the Metrics API through the Datadog Agent's OpenMetrics integration, and it is also a log drain destination that needs an API key and a region.
**What should I alert on first?**
Start with the database being down for 5 minutes, replication lag over 10 minutes and rising, and database size growing 20% in 12 hours, all from the supabase-grafana example rules. Then add client connections at 80% of max_connections, CPU or memory held above 80%, and API or Auth server errors of 20 or more at a rate of at least 1% that doubles the previous hour.
**How do I find which query is slowing the database down?**
Enable `pg_stat_statements`, save hourly snapshots, and compare the mean execution time per statement. Supabase's check flags a mean of 100 ms or more that doubles the previous hour with at least 20 calls. For a quick look, run `supabase inspect db outliers` and `supabase inspect db long-running-queries`, then read the query plan.
**How do I connect my frontend traces to Supabase logs?**
Use `@supabase/supabase-js` 2.106.0 or later, import `@supabase/supabase-js/tracing` at your entry point from 2.112.0, and create the client with `tracePropagation: true` while an OpenTelemetry span is active. The trace ID then appears in API Gateway and Edge Function logs. Add the trace headers to your Edge Functions' CORS allow-list.
**Is it safe to let an AI agent run these monitoring checks?**
Supabase's detection checks are written for read-only access: use the project-scoped Supabase MCP server with `read_only=true`, which runs all SQL as a read-only Postgres user. Also restrict the server's feature groups to the ones the checks need. Keep any fix behind human review.
#### Sources
- [Own Your Observability: Supabase Metrics API](https://supabase.com/blog/metrics-api-observability)
- [Supabase Docs: Detection checks](https://supabase.com/docs/guides/observability/detecting)
- [Supabase Docs: Inspect the database](https://supabase.com/docs/guides/database/inspect)
- [Supabase Docs: Metrics API with Prometheus and Grafana (self-hosted)](https://supabase.com/docs/guides/telemetry/metrics/grafana-self-hosted)
- [Supabase Docs: Log Drains](https://supabase.com/docs/guides/telemetry/log-drains)
- [Supabase Docs: Client-side tracing](https://supabase.com/docs/guides/telemetry/client-side-tracing)
- [Supabase Docs: Model context protocol (MCP)](https://supabase.com/docs/guides/ai-tools/mcp)
- [supabase-grafana: Example alerts](https://github.com/supabase/supabase-grafana/blob/main/docs/example-alerts.md)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
Next: [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
#### Related
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [How to monitor a vibe-coded app in production](https://getpraxen.app/learn/monitoring-coverage/vibe-coding-production-monitoring/): Monitor a vibe-coded app in production: surface swallowed errors, add a health route, tag deploys, watch cron jobs and alert on what users feel first.
- [Supabase vs Convex in production: reliability and cost](https://getpraxen.app/learn/reliability/supabase-vs-convex-in-production/): Supabase vs Convex in production: compare data models, realtime, consistency, scaling, migrations, monitoring, backups and costs before choosing your backend.
- [Fix “remaining connection slots are reserved” in PostgreSQL](https://getpraxen.app/learn/troubleshooting/postgres-remaining-connection-slots-are-reserved/): Why Postgres refuses new logins with “remaining connection slots are reserved”, how to find who holds the connections, and how to pool and cap them for good.
---
### How to monitor a vibe-coded app in production
> Monitor a vibe-coded app in production: surface swallowed errors, add a health route, tag deploys, watch cron jobs and alert on what users feel first.
Part 1 of [Vibe-coded apps in production](https://getpraxen.app/series/vibe-coded-apps-in-production/)
By Boris Tane, Founder of Praxen · Published September 30, 2026 · 12 min read
Canonical: https://getpraxen.app/learn/monitoring-coverage/vibe-coding-production-monitoring/
Get the vibe-coded app to report its own failures. Replace swallowed errors with structured logs, put a health route that queries the database behind an outside uptime check, and stamp every log line with the release that produced it. Then give scheduled jobs heartbeats, and alert only on what users feel: the app is down, server errors are rising, sign-in is failing, the database is out of connections, or a job has stopped.
To monitor a vibe-coded app in production, add the layer the generator left out. Every failure should raise an error and log a structured event. A health route should sit behind an outside uptime check. Each log line should carry the release that produced it, and every scheduled job needs a heartbeat. On top of that, write a handful of alerts for what users feel first: the app is down, server errors are climbing, sign-in is failing, the database is running out of connections, or a job has stopped running.
You don't need a new platform on day one. Your host's logs, a small logging helper, one health route and one external check cover most of it. The rest of this page takes you through each step with code you can paste, then covers the alerts worth having and the gaps that bite a few weeks after launch.
#### What production monitoring means for a vibe-coded app
A vibe-coded app is built mostly by prompting an AI tool such as Lovable, Bolt, v0, Replit, Cursor or Claude Code, then shipping what it produces once the preview works. Production monitoring is the set of signals that tell you the live app is failing before a user does, with enough detail to find the cause.
Those signals come in three kinds:
- **Logs**: one record per event, such as a failed payment or a request that timed out.
- **Metrics**: numbers over time, such as requests per minute, error rate or open database connections.
- **Traces**: the path of one request through your frontend, API, database and third-party calls.
For one app on one host with a hosted database, logs and a few metrics do most of the work. Traces start to pay off once a request crosses several services.
A vibe-coded app is different because nobody on the team has read every file. Monitoring is how you learn what the code actually does with real users and real data. I built and led the Workers observability team at Cloudflare, and the rule held on every platform we touched: a person or an agent can only debug what the code emits.
#### Why vibe-coded apps fail quietly after launch
Generated code is written for the path the prompt described. Check the other paths for gaps like these:
- A token that never expires, or an endpoint with no rate limit.
- An API call with no retry and no timeout. It fails or hangs when the other side is slow.
- A database connection pool sized for a demo, which runs out when real users arrive at the same time.
- A query with no index. It is fast on test data and slow on real data.
- A `catch` block that swallows the error and returns an empty result.
The last one is why these apps fail quietly. A failed request turns into an empty list on screen, the server logs nothing, and every dashboard stays green. The user sees a broken page and you see nothing.
These gaps can also stay hidden for a while. In one widely reported case, a founder ran a public coding trial on Replit. About nine days in, the agent ignored an instruction not to change anything without approval. It ran commands that wiped the production database and deleted records for more than 1,200 executives and over 1,190 companies. It then said a rollback wasn't possible, though a backup was available. Monitoring can't stop an agent that has write access to production. What it can do is tell you within minutes that the data is gone, and show you which release or action came just before.
#### The signals that tell you a vibe-coded app is broken
Start with the signals that match what a user feels. Each one maps to a gap in the list above.
1. **Reachability of the main route.** Can someone outside your host load the app and reach the database behind it?
2. **Server error rate per route.** Count 5xx responses by route. A rise straight after a deploy points at that deploy.
3. **Sign-in and access failures.** A jump in 401 and 403 responses usually means a broken auth flow or a changed access rule.
4. **Database pressure.** Open connections compared with the limit, and statements whose average time keeps climbing.
5. **Background work that stopped.** Cron jobs, queue consumers, webhooks and email sends. When these stop, nothing throws an error. The work just never happens.
6. **Third-party calls.** Error rate and latency for your payment provider and any AI API, plus the AI API spend. Spend climbs fast when a loop keeps retrying.
That's the whole starter list. Everything else can wait until one of these points you at it.
#### Step 1: Map what the generated app depends on
You can't monitor a dependency you don't know about. Ask your coding agent to list them, then check its answer against the environment variables set on your host.
```text
> list every external service this app calls and the env var each one reads
```
*Run it at the repository root, then compare the answer with your host's environment settings.*
Write the result into a short file in the repository, such as `OPERATIONS.md`. Record where the frontend is hosted, where the API and functions run, which database you use, which scheduled jobs exist and which third-party APIs the app calls. While you're in there, check that no secret key is read in client-side code. When the generator doesn't treat API keys as secrets, they can end up in the browser bundle.
#### Step 2: Turn swallowed errors into loud, structured logs
First, find the error handling that hides failures. A search catches most of it:
```bash
grep -rn -A3 "catch" src app lib | grep -E "return (\[\]|null|undefined)"
```
Next, add a small logger that writes one JSON line per event. Every line gets the release and the environment, so you can filter on them later:
```ts
// lib/log.ts
const release = process.env.RELEASE_SHA ?? "dev";
const env = process.env.APP_ENV ?? "development";
type Fields = Record;
function write(level: "info" | "warn" | "error", event: string, fields: Fields = {}) {
console.log(JSON.stringify({ ts: new Date().toISOString(), level, event, release, env, ...fields }));
}
export const log = {
info: (event: string, fields?: Fields) => write("info", event, fields),
warn: (event: string, fields?: Fields) => write("warn", event, fields),
error: (event: string, fields?: Fields) => write("error", event, fields),
};
```
Then rewrite each swallowed error. This is what generated code often looks like:
```ts
export async function getOrders(userId: string) {
try {
const res = await fetch(`${API_URL}/orders?user=${userId}`);
return await res.json();
} catch (e) {
return [];
}
}
```
This version has a timeout, a structured event and a real failure:
```ts
import { log } from "@/lib/log";
export async function getOrders(userId: string) {
const started = Date.now();
const res = await fetch(`${API_URL}/orders?user=${userId}`, {
signal: AbortSignal.timeout(8000),
});
if (!res.ok) {
log.error("orders.fetch_failed", { userId, status: res.status, ms: Date.now() - started });
throw new Error(`orders request failed: ${res.status}`);
}
log.info("orders.fetched", { userId, ms: Date.now() - started });
return res.json();
}
```
The rule is simple. Every failure is either handled on purpose or logged and raised. Log one event per request or job, with the fields you'd search by: event name, route, user or tenant id, status and duration. Keep passwords, tokens and personal data out of those fields, and check generated schemas and log lines for personal data stored in plain text.
To prove it works, throw a test error from a route in staging and check that it shows up in your host's logs with the route and the release attached.
#### Step 3: Add a health route that queries the database
A health route that returns success without doing anything tells you the process is up and nothing else. Make yours run one real query:
```ts
// app/api/health/route.ts
import { db } from "@/lib/db";
export async function GET() {
const started = Date.now();
try {
await db.query("select 1");
return Response.json({ ok: true, release: process.env.RELEASE_SHA, dbMs: Date.now() - started });
} catch (err) {
console.error(JSON.stringify({ level: "error", event: "health.db_failed", message: String(err) }));
return Response.json({ ok: false }, { status: 500 });
}
}
```
Check it from your own machine:
```bash
curl -s -w "\n%{http_code} %{time_total}s\n" https://your-app.example.com/api/health
```
A healthy app answers like this:
```text
{"ok":true,"release":"3f9c2a1","dbMs":12}
200 0.184s
```
Now point an external uptime checker at the route. It has to run outside your host's network, because a check on the same host goes down with it and stays silent.
#### Step 4: Stamp every log line with the release
When errors jump, the first question is what changed. If every log line carries the commit that produced it, the answer is a filter away. Set the variable in your build step, and make sure your host passes it through to runtime:
```bash
export RELEASE_SHA=$(git rev-parse --short HEAD)
npm run build
```
The logger from Step 2 already adds `release` to every line, and the health route reports it. This is the first thing I'd add to any app, generated or not. It turns "something broke this afternoon" into "errors started with release 3f9c2a1".
#### Step 5: Put heartbeats on cron jobs and webhooks
Background work fails by not happening, so you have to watch for something that is missing. Make each job report when it finishes:
```ts
import { log } from "@/lib/log";
export async function nightlyInvoices() {
const started = Date.now();
try {
const { sent } = await sendInvoices();
log.info("job.invoices.done", { sent, ms: Date.now() - started });
await fetch(process.env.INVOICES_HEARTBEAT_URL!, { method: "POST" });
} catch (err) {
log.error("job.invoices.failed", { message: String(err), ms: Date.now() - started });
throw err;
}
}
```
Point the heartbeat URL at any monitor that alerts when a ping doesn't arrive within the job's schedule plus a margin. For webhooks, log two separate events: one when the webhook arrives and one with the processing result. Alert when arrivals fall to zero during hours that normally see traffic.
#### Step 6: Write the few alerts worth waking up for
Page someone only for signals a user would notice. Send everything else to a daily summary.
- **App down.** The external health check fails on consecutive runs.
- **Server errors rising.** Supabase's published detection checks give a sensible starting rule: 20 or more server errors, a rate of at least 1%, and at least double the previous hour's rate. The same shape works for any API.
- **Database connections near the limit.** Those checks flag client connections at 80% of `max_connections`. On Postgres you can read both numbers with one query:
```sql
select count(*) filter (where backend_type = 'client backend') as client_connections,
current_setting('max_connections')::int as max_connections
from pg_stat_activity;
```
- **A slow statement getting slower.** Flag a statement whose mean time reaches 100 ms and doubles hour on hour, with 20 or more calls in each hour. A deploy that drops an index shows up here first.
- **A missed heartbeat** on any job that moves money, sends email or syncs data.
- **A jump in sign-in failures**, compared with the same window the day before.
If your app runs on Supabase, the Metrics API publishes Postgres health metrics that refresh every minute, and log drains can stream platform logs to another tool on the Pro, Team and Enterprise plans.
#### Getting your coding agent to add the instrumentation
The same agent that wrote the app can do most of Steps 2 to 5 in one pass, as long as the request is precise about what you want.
```text
> add structured logs, a database health route and heartbeats on every cron job
```
*Name the logger file and the event fields you want, so every route logs the same shape.*
Read the diff before you merge it. Look for three things: secrets or personal data in log fields, new catch blocks that still return empty results, and a health route that skips the database. Then run the test error from Step 2 again.
#### Where monitoring a vibe-coded app goes wrong
- **The health route never queries the database.** It stays green while every real request fails. Fix: run one real query, as in Step 3.
- **The uptime check runs on the same host.** When the host goes down, the check goes down with it. Fix: run it from outside.
- **Alerts land in an inbox nobody reads.** Fix: send pages to a phone or a chat channel someone owns, and keep that list short.
- **Preview deployments page on-call.** Fix: tag every event with the environment, and filter alerts to production.
- **Full request bodies get logged.** That puts tokens and personal data into your logs. Fix: log named fields only, and redact anything that looks like a secret.
- **Only the browser reports errors.** Server functions and jobs fail out of sight. Fix: send server logs somewhere you can search, and alert on them.
- **Platform logs are off or kept briefly.** Fix: confirm logging is on for every function, and drain logs to a store with the retention you need.
- **A regenerated file drops the logging.** Agents rewrite whole files. Fix: add a test that fetches the health route in CI, and check new code for the logger import during review.
- **The agent that built the app holds production write credentials.** Fix: give it read-only access, send every change through a pull request that a person merges, and test a backup restore before you need one.
#### A launch checklist for a vibe-coded app
- [ ] Every external service and secret is listed in the repository, and no secret is read in client code.
- [ ] No catch block returns empty data silently. A test error shows up in the logs with its route and release.
- [ ] A health route queries the database, and an outside checker calls it.
- [ ] Every log line carries the release and the environment.
- [ ] Every scheduled job and webhook has a heartbeat or an arrival alert.
- [ ] Alerts cover the app being down, rising server errors, connection pressure, slow statements, missed heartbeats and sign-in failures, and they reach a person.
- [ ] Production credentials held by coding agents are read-only.
#### Watching a vibe-coded app with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to the platforms your app already runs on, so when something breaks it works out why, and the fix arrives ready for your normal review.
Running on Supabase? See [how Praxen monitors Supabase in production](https://getpraxen.app/for/supabase/).
#### Common questions
**Do I need a paid observability tool to monitor a vibe-coded app?**
Not on day one. Your host's logs, a structured logger, a health route that queries the database and an outside uptime check cover the first signals. Add an error tracker or a log drain once you have real traffic and need to search several days of history.
**What should I monitor first on a vibe-coded app?**
Start with six signals. Can people outside your host reach the main route and the database? Watch server errors per route, spikes in 401 and 403 responses at sign-in, database connections and slow statements, background jobs that stopped, and errors and spend on third-party APIs. Each one matches a gap that generated code often leaves.
**How do I find out whether my app is swallowing errors?**
Search for catch blocks that return an empty array or null, and ask your coding agent to list every one. Then throw a test error from a route in staging and check that it appears in your logs with the route and release. If it never appears, something between the route and the logger is swallowing it.
**My vibe-coded app runs on Supabase. What changes?**
You get two extra sources. The Metrics API publishes Postgres health metrics in Prometheus format and refreshes them every minute. Log drains stream platform logs to another tool on the Pro, Team and Enterprise plans. Supabase's published checks flag connection pressure at 80% of max_connections, which makes a good first alert.
**Should the AI agent that built my app have access to production?**
Read-only access at most. In one reported Replit trial, an agent had been told not to make changes without approval. About nine days in, it wiped the production database anyway, deleting records for more than 1,200 executives and over 1,190 companies. Keep writes behind pull requests that a person merges, and test your backup restore before you need it.
**How many alerts should a small vibe-coded app have?**
Start with the few that match what users feel: the health check failing, server errors rising after a deploy, a missed job heartbeat, and database connections near the limit. Send everything else to a daily summary. An alert that fires often and needs no action teaches people to ignore the one that matters.
**Do I need tracing for a vibe-coded app?**
Usually not at first. With one frontend, one API and a hosted database, structured logs that share a request id answer most questions. Add tracing once requests cross several services and the logs can't tell you which hop is slow.
**Should preview deployments be monitored?**
Keep their logs, but route their alerts away from on-call. Tag every event with the environment and filter production alerts on that tag. A broken preview branch then never pages anyone.
#### Sources
- [Praxen documentation (full text)](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content, including How to monitor a Supabase app in production](https://getpraxen.app/llms-full.txt)
- [From Vibe Code to Production: How to Make AI-Generated Software Production-Ready (Rishabh Software)](https://www.rishabhsoft.com/blog/vibe-code-to-production)
- [Vibe Coding to Production: The Checklist (Slash)](https://slash.co/articles/vibe-coding-to-production-the-checklist-testing-ci-cd-monitoring-security/)
- [AI-powered coding tool wiped out a software company's database (Fortune)](https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/)
- [Vibe coding service Replit deleted production database (The Register)](https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/): Monitor a Supabase app in production: scrape the Metrics API into Prometheus, drain logs, run SQL health checks, trace requests and alert on what matters.
- [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/): Set up health checks, JSON logs, OpenTelemetry traces, monitors and crash alerts for a FastAPI service on Railway, with the commands and config to copy.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
---
### Cloudflare Hyperdrive connection errors: causes and fixes
> Fix Cloudflare Hyperdrive connection errors: config codes 2008 to 2016, pool exhaustion, connection_refused and stale clients reused across Worker requests.
Part 7 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 30, 2026 · 13 min read
Canonical: https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/
Most Hyperdrive connection errors come from one of three places: Hyperdrive can't reach or sign in to your database (config codes 2008 to 2016, or connection_refused), the origin pool is exhausted (Failed to acquire a connection from the pool), or your Worker reuses a database client across requests. To fix them, open the database to Cloudflare's IP ranges over TLS, keep transactions short and check the pool metrics before you raise the origin connection limit. Then create a new client inside every handler.
A Hyperdrive connection error tells you which of three hops failed. There is the hop from Hyperdrive to your database, the pool of origin connections Hyperdrive shares across your Workers, and the database client in your Worker code. Each hop fails with its own messages. Once you know which one you're looking at, the fix is usually one change.
This page lists every message Cloudflare documents for those hops. It shows how to confirm which cause is yours with the pool metrics and a quick database query, gives the fix for each, and covers the alerts that stop the errors coming back.
#### Where a Hyperdrive connection error comes from
Hyperdrive sits between your Worker and a regional Postgres or MySQL database. Your driver connects to Hyperdrive at the edge, next to your Worker, which is fast. Hyperdrive then runs the query over a pooled connection that sits close to your database. That saves the seven round trips a fresh connection needs: one for the TCP handshake, three for TLS and three for authentication. When the request ends, the Worker's client is garbage collected, and the origin connection stays in the pool for the next request.
That design gives you three kinds of failure:
1. **Configuration errors.** When you create or update a config, Hyperdrive runs a test connection and an empty test query (a `;` in PostgreSQL). If that fails, you get a numbered code from 2008 to 2016, or the message `Failed to connect to the provided database`.
2. **Runtime errors from Hyperdrive.** These arrive as `ErrorResponse` wire protocol messages, so your driver throws them from the query that caused them. Hyperdrive errors with no matching PostgreSQL code use the code `58000`. Errors sent by your database pass through unchanged where possible.
3. **Errors from your own code.** These are Workers runtime and driver errors, and they almost always mean a client was created in global scope or kept for another request.
#### Configuration codes 2008 to 2016: Hyperdrive can't reach or sign in to the database
These appear when you create or update a config. Each code points at one cause.
**Hyperdrive configuration error codes and their fixes**
| Code | What Hyperdrive found | Fix |
| --- | --- | --- |
| 2008 | Bad hostname, it could not be resolved | Confirm the hostname exists in public DNS |
| 2009 | Hostname does not resolve to a public IP | Use a public address, or connect privately through Workers VPC or Tunnel |
| 2010 | Cannot connect to host and port | Check the spelling and the public DNS record |
| 2011 | Connection refused | Allow connections from Cloudflare in your firewall or ACL |
| 2012 | Database does not support TLS | Turn on TLS, Hyperdrive requires it |
| 2013 | Invalid credentials | Check the username exists and the password case |
| 2014 | Database name does not exist | Use the database name, and check it exists |
| 2015 | Generic failure with no known reason | Open a Cloudflare support ticket |
| 2016 | Test query failed | Grant the user read and write permissions |
Code 2010 is a routing problem: check that the hostname has a public DNS record that resolves to a public IP address, and that it isn't misspelled. Code 2011 is the firewall code. Hyperdrive connects from Cloudflare's IP address ranges, which all Hyperdrive configurations share with other Cloudflare products. Cloudflare documents three ways to let it in:
1. Allow inbound connections from the whole public Internet.
2. Allow the public Internet, with an IP access control list that permits only Cloudflare's ranges.
3. Keep the database on a private network and run a Cloudflare Tunnel inside that network. Workers VPC is the other private option in the docs.
If you see `Failed to connect to the provided database` and none of the codes, look at certificates. `Server return error and closed connection` means the database demands client certificates and your config doesn't send them. `TLS handshake failed: cert validation failed` means you gave Hyperdrive a server CA certificate that didn't sign the certificate the database presented. Either the CA is wrong, or you are pointing at the wrong database.
#### Failed to acquire a connection from the pool
This is the runtime error most teams mean when they search for Hyperdrive connection errors. Hyperdrive waited for a free origin connection and timed out, or it couldn't connect at all. If it happens now and then, the pool is being exhausted: too many connections are held for too long.
##### Why the pool runs dry
Hyperdrive pools in transaction mode. A client holds one origin connection for the whole of a transaction and gives it back when the transaction ends. Hyperdrive doesn't limit client connections from your Workers, but it does cap origin connections per configuration:
- **Free plan:** about 20 origin connections per configuration.
- **Paid plan:** about 100 origin connections per configuration.
- **Every configuration:** at least 5.
The usual causes, in the order worth checking:
- **Long transactions.** Each open transaction holds a connection. An external API call or heavy computation between `BEGIN` and `COMMIT` keeps that connection away from every other Worker isolate.
- **Slow queries.** A query can run for up to 60 seconds before Hyperdrive kills it. A handful of those at once fills a small pool.
- **Durable Objects that keep a client open.** A Durable Object keeps state across requests. If it keeps a database client open, that connection stays allocated from the pool, and many long-lived objects doing the same can drain it.
- **Transactions used to keep SET state.** Hyperdrive runs `RESET` on a connection when it goes back to the pool. Wrapping many operations in one transaction to keep a `SET` alive holds the connection the whole time. Put the `SET` in each query or transaction that needs it.
- **A pool sized too small.** Someone lowered the origin connection limit, or traffic grew past it.
##### Confirm it with the pool metrics
Hyperdrive exports pool size metrics through the `hyperdrivePoolSizesAdaptiveGroups` dataset in the GraphQL Analytics API. You can also see them in the **Pool connections** chart on the **Metrics** tab of each configuration. Run this query for the window when the errors started:
```graphql
query HyperdrivePoolSizes(
$accountTag: string!
$configId: string!
$datetimeStart: Time!
$datetimeEnd: Time!
) {
viewer {
accounts(filter: { accountTag: $accountTag }) {
hyperdrivePoolSizesAdaptiveGroups(
limit: 10000
filter: {
configId: $configId
datetime_geq: $datetimeStart
datetime_leq: $datetimeEnd
}
) {
avg { currentPoolSize availablePoolSlots waitingClients }
max { maxPoolSize currentPoolSize waitingClients }
dimensions { coloCode }
}
}
}
}
```
Read it like this:
- **Waiting clients spike, and peak open connections sit at the pool maximum.** The pool is exhausted. Go on to the database check below.
- **Open connections are well under the maximum, but errors continue.** A likely explanation is that Hyperdrive can't open new connections. Look at the next section on `connection_refused`.
- **One `coloCode` is bad and the others are fine.** The pressure is in one location's pool. Hyperdrive is distributed, so a client that can't reach an existing pool gets a new pool with its own allowance. Connection counts can then briefly go over the listed limits.
Metrics are kept for 31 days, so you can compare the incident with a normal week.
##### Find what is holding the connections
On Postgres, list the sessions that have been busy or idle inside a transaction for a long time:
```sql
select pid, usename, state,
now() - xact_start as transaction_age,
left(query, 120) as query
from pg_stat_activity
where datname = current_database()
and state in ('active', 'idle in transaction')
and xact_start is not null
order by xact_start
limit 20;
```
Rows in the `idle in transaction` state are the ones to chase first. Some code opened a transaction and went off to do something else.
##### Fix the pool exhaustion
1. **Shorten the transactions.** Do only the essential queries inside the transaction. Move API calls and computation outside it:
```ts
// Before: the connection is held while the payment API responds
await sql.begin(async (tx) => {
const [order] = await tx`select * from orders where id = ${id} for update`;
const charge = await fetch(env.PAYMENTS_URL, { method: "POST", body: JSON.stringify(order) });
await tx`update orders set status = 'paid' where id = ${id}`;
});
// After: call the API first, then hold the connection for two quick statements
const charge = await fetch(env.PAYMENTS_URL, { method: "POST", body: JSON.stringify({ id }) });
await sql.begin(async (tx) => {
await tx`select 1 from orders where id = ${id} for update`;
await tx`update orders set status = 'paid' where id = ${id}`;
});
```
2. **Close Durable Object clients when they sit idle.** Use connection timeouts, and limit how many objects hold a database connection at all.
3. **Raise the origin connection limit, carefully.** Change it in the dashboard under **Settings**, then **Origin connection limit**, or with Wrangler:
```bash
npx wrangler hyperdrive update $HYPERDRIVE_ID --origin-connection-limit=60
```
The limit is soft. During network trouble, Hyperdrive may open more connections than you set, so keep it below your database's own maximum. If several Hyperdrive configurations point at the same database, add their limits together. If open connections keep reaching the plan limit, Cloudflare takes limit increase requests through a form linked from the limits page.
#### Server connection attempt failed: connection_refused
This runtime error means Hyperdrive can't open new connections to your origin. There are two causes. The first is a firewall or ACL rejecting Cloudflare, often after someone tightened the rules or the database moved. The second is your database provider refusing connections because you went over its connection limit.
To tell them apart, check the database's own connection count at the time of the errors. If it was at the provider's maximum, the soft pool limit, other clients sharing the database, or several Hyperdrive configs filled it. Lower the Hyperdrive limit, or raise the database's. If the count was low, fix the firewall with one of the three networking options above.
#### Errors that mean a client outlived its request
Workers don't allow I/O across requests. A client created in global scope, or cached in a variable and reused, belongs to a request context that has ended. The messages depend on the driver:
- **Workers runtime:** `Disallowed operation called within global scope` (connecting during script startup) and `Cannot perform I/O on behalf of a different request`.
- **node-postgres:** `Connection terminated`, `Connection terminated unexpectedly`, `Client has encountered a connection error and is not queryable`, `Client was closed and is not queryable`, `Cannot use a pool after calling end on the pool`, `Client has already been connected. You cannot reuse a client.`
- **Postgres.js:** `write CONNECTION_ENDED`, `write CONNECTION_DESTROYED` and `write CONNECTION_CLOSED`, followed by the host and port. The `code` property on the error holds the code.
- **mysql2:** `Can't add new command when connection is in closed state`, `Connection lost: The server closed the connection.` (`PROTOCOL_CONNECTION_LOST`), `Pool is closed.`
- **mysql:** `PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR`, `PROTOCOL_ENQUEUE_AFTER_QUIT` and `PROTOCOL_ENQUEUE_HANDSHAKE_TWICE`.
The fix is the same for all of them: create the client inside the handler, on every request. Don't create a driver-level pool (`new Pool()` or `createPool()`) in global scope. Hyperdrive already pools for you, so a new client per request is fast.
```ts
import { Client } from "pg";
export default {
async fetch(request, env, ctx): Promise {
const client = new Client({ connectionString: env.HYPERDRIVE.connectionString });
await client.connect();
const result = await client.query("SELECT id, status FROM orders LIMIT 10");
// No client.end() needed: the client is cleaned up when the request ends,
// and the origin connection stays in Hyperdrive's pool.
return Response.json(result.rows);
},
} satisfies ExportedHandler;
```
You don't need to call `client.end()`, `sql.end()` or `connection.end()`. Clients are cleaned up when the request or invocation ends, including when a Workflow or Queue consumer finishes, or when a Durable Object hibernates or is evicted.
#### Driver errors that look like connection failures
A few errors show up while you connect, but they come from the driver or the runtime:
- **`Uncaught Error: No such module "node:..."`**: your code or a library needs a Node module. Turn on Node.js compatibility for the Worker.
- **`Code generation from strings disallowed for this context`**: the driver calls `eval()`, which Workers don't support. This is common with mysql2. Configure the driver not to use it.
- **`Hyperdrive does not currently support MySQL COM_STMT_PREPARE messages`**: remove prepared statements from your MySQL queries.
- **`Internal error.`**: something broke on Cloudflare's side. Check for an ongoing Hyperdrive incident, contact support, and retry if retries are safe for your workload.
#### Recovering after a database failover
Hyperdrive detects and recovers from most failovers by itself. If connections still point at the old primary afterwards, restart the pool. Open the configuration in the dashboard, go to **Settings**, and select **Restart** under **Danger zone**. You need the Hyperdrive Admin role. A restart drops every active connection and rebuilds the pool, so in-flight queries may see brief errors. Do it once, then watch the pool chart refill.
#### Reproducing the error locally without fooling yourself
Plain `wrangler dev` connects straight to the database through `localConnectionString`, or through the `CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_` variable followed by your binding name. In that mode, pooling and caching don't take effect, so pool exhaustion and Hyperdrive's own errors won't reproduce.
`wrangler dev --remote` runs your Worker on Cloudflare with the deployed Hyperdrive config, so pooling and caching are active. Any writes land in the configured database, which may be production. Point a separate Hyperdrive config at a staging database before you load test pool behaviour.
#### Stale reads are a caching issue
If a read after a write returns old data, the connection is fine and the query cache is serving the read. Hyperdrive doesn't invalidate cached reads when you write. Check `cacheStatus` in the `hyperdriveQueriesAdaptiveGroups` dataset. For reads that must be fresh, such as sessions, permissions and reads right after a write, add a second binding to the same database with caching turned off. It still gets pooling and fast connection setup.
#### What keeps Hyperdrive connection errors coming back
- **Raising the limit to cover a leak.** A bigger pool buys time until the idle transactions fill it again. Fix the transaction first, then size the pool.
- **Setting the Hyperdrive limit equal to the database maximum.** The soft limit lets Hyperdrive go over it, which turns into `connection_refused`. Leave headroom.
- **Forgetting other clients of the same database.** Migrations, cron jobs, a second Hyperdrive config and admin tools all share the database's connection limit.
- **Caching a client in a module variable "for speed".** It saves nothing, because Hyperdrive already removes the setup cost, and it causes every error in the stale client section.
- **Load testing with local `wrangler dev`.** It bypasses the pool, so the test proves nothing about pool exhaustion.
- **Only watching errors.** By the time errors show, clients have already queued. Waiting clients show the problem earlier.
#### Alerts worth setting on a Hyperdrive config
Pull the two datasets on a schedule into whatever tool pages your team, and alert on:
1. **Peak waiting clients above zero for several minutes** in `hyperdrivePoolSizesAdaptiveGroups`. This is the earliest sign of contention.
2. **Peak open connections close to `maxPoolSize`** for a sustained period. Time to find the slow transaction or ask for more connections.
3. **Error share** by `eventStatus` in `hyperdriveQueriesAdaptiveGroups`, compared with the previous hour.
4. **Average `connectionLatency` rising.** Slow new connections to the origin often come before refusals.
I built and led the Workers observability team at Cloudflare, and my advice is the same for every pooled resource. Put the pool chart and your deploy markers on one timeline. A config change or deploy that shrinks the pool, or lengthens a transaction, is much easier to spot when the waiting clients line starts climbing right after it.
#### Tracing a Cloudflare alert back to its change with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Cloudflare, so when Hyperdrive connections start failing it works out why, and the fix arrives ready for your team's normal review.
Running on Cloudflare? See [how Praxen monitors Cloudflare in production](https://getpraxen.app/for/cloudflare/).
#### Common questions
**What does "Failed to acquire a connection from the pool" mean in Hyperdrive?**
Hyperdrive timed out waiting for a free origin connection, or couldn't connect at all. When it happens now and then, the pool is exhausted because connections are held too long, usually by long transactions or slow queries. Check waitingClients and currentPoolSize in the pool metrics, then look for sessions idle in a transaction on the database.
**How many database connections does Hyperdrive open?**
Each configuration gets about 20 origin connections on the Free plan and about 100 on a paid plan, with a minimum of 5. The limit is soft, so Hyperdrive may briefly open more during network trouble or high traffic. Set it below your database's own maximum.
**How do I change the Hyperdrive origin connection limit?**
In the dashboard, open the configuration, go to Settings, change Maximum connections under Origin connection limit and select Save. From the terminal, run npx wrangler hyperdrive update with your config ID and the --origin-connection-limit flag. You can also send a PATCH request with origin_connection_limit to the Hyperdrive REST API.
**Why do I get "Cannot perform I/O on behalf of a different request" with Hyperdrive?**
A database client created during one request is being reused in a later one, and Workers don't allow I/O across requests. Create a new client inside your fetch or queue handler on every request. Hyperdrive's pool already removes the connection setup cost.
**Do I need to call client.end() when using Hyperdrive?**
No. The client in your Worker is cleaned up when the request or invocation ends, and Hyperdrive keeps the origin connection in its pool for reuse. That also applies when a Workflow or Queue consumer finishes, or when a Durable Object hibernates or is evicted.
**Which IP addresses should I allow in my database firewall for Hyperdrive?**
Hyperdrive connects from Cloudflare's published IP address ranges, which all Hyperdrive configurations share with other Cloudflare products. You can allow the whole Internet, allow only those ranges in an ACL, or keep the database private and connect through Cloudflare Tunnel or Workers VPC.
**What does Hyperdrive error 2012 mean?**
The database doesn't support TLS, and Hyperdrive requires TLS to connect. Turn on TLS on the database server, then save the Hyperdrive configuration again so it reruns the test connection.
**Why can't I reproduce Hyperdrive pool errors with wrangler dev?**
Plain wrangler dev connects straight to the database through localConnectionString, so Hyperdrive's pooling and caching don't take effect. Use wrangler dev --remote to run with the deployed config. Any writes then reach the database that config points at, so use a staging config for tests.
#### Sources
- [Troubleshoot and debug, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/observability/troubleshooting/)
- [Connection lifecycle, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/concepts/connection-lifecycle/)
- [Connection pooling, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/concepts/connection-pooling/)
- [Limits, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/platform/limits/)
- [Hyperdrive exposes database connection pool size metrics, Cloudflare changelog](https://developers.cloudflare.com/changelog/post/2026-05-15-hyperdrive-pool-size-metrics/)
- [Metrics and analytics, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/observability/metrics)
- [Tune connection pooling, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/configuration/tune-connection-pool/)
- [Local development, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/configuration/local-development)
- [Firewall and networking configuration, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/configuration/firewall-and-networking-configuration/)
- [FAQ, Cloudflare Hyperdrive docs](https://developers.cloudflare.com/hyperdrive/reference/faq/)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
Next: [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
#### Related
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [Fix “remaining connection slots are reserved” in PostgreSQL](https://getpraxen.app/learn/troubleshooting/postgres-remaining-connection-slots-are-reserved/): Why Postgres refuses new logins with “remaining connection slots are reserved”, how to find who holds the connections, and how to pool and cap them for good.
---
### Cloudflare Workers error 1101: causes and how to fix it
> Error 1101 means your Cloudflare Worker threw an uncaught JavaScript exception. Find the exception in logs, match it to its cause, fix it and roll back fast.
Part 6 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 30, 2026 · 13 min read
Canonical: https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/
Error 1101 means your Worker threw a JavaScript exception it never caught, so it returned no response and Cloudflare served an error page. To find the exception message, run wrangler tail with the error status filter, or filter Workers Logs on the exception outcome. The message tells you the cause: an unresolved promise, a lost this reference, an I/O object shared between requests, or an ordinary bug in your code. If users are affected, roll back with wrangler rollback first, then fix the code and redeploy.
Error 1101 means your Worker threw a JavaScript exception and never caught it. The Workers runtime had no response to send, so it served its own error page. Cloudflare's support docs call this a rendering error. The cause is in your Worker code, or in a dependency your code calls.
Every fix starts in the same place: read the exception message in Workers Logs or `wrangler tail`. That message tells you which cause you have, from a promise that never settles to a method that lost its `this`. If users are hitting the error right now, roll back first. Then fix the code, reproduce the error locally to confirm the fix, and redeploy.
#### What error 1101 means for your Worker
When a production Worker hits an error that stops it returning a response, the client gets a Cloudflare error page with a code. Code 1101 means the Worker threw a JavaScript exception. The nearby codes each mean something different, so check which one you have before you start debugging:
- **1101**: the Worker threw a JavaScript exception.
- **1102**: the Worker exceeded its CPU time limit. The fix here is to do less work per request. It has nothing to do with exception handling.
- **1019**: the Worker hit the loop limit. A Worker cannot call itself or another Worker more than 16 times.
- **1027**: the Worker exceeded the free tier daily request limit.
- **1021, 1022, 1024 and 1042**: routing and subrequest problems, such as requesting a host the Worker cannot reach.
- **Other 11xx codes**: these usually point at a problem in the Workers runtime itself. Check the Cloudflare status page before you touch your code.
A Workers exception can also show up as a plain HTTP 500, so a 500 in front of a Worker deserves the same checks. The error page carries a Ray ID. Keep it: Cloudflare Support asks for the Ray ID, the Worker name, recent code changes and steps to reproduce.
Some failures never produce an error page. Cloudflare lists `Network connection lost`, memory limit errors while reading a stream, and `daemonDown` as runtime errors. The user doesn't see them, and you only find them in logs.
#### Find the exception behind your 1101
I built and led the Workers observability team at Cloudflare, and on a 1101 I look for the exception message before anything else. Every later step depends on it.
##### Check the Worker is writing logs
New Workers have observability turned on by default. Older Workers may not. Open your Wrangler config and make sure it contains this (Workers Logs needs Wrangler 3.78.6 or later):
```jsonc
{
"observability": {
"enabled": true,
"head_sampling_rate": 1
},
"upload_source_maps": true
}
```
If you had to add it, redeploy. `head_sampling_rate` sets the share of requests that get logged, from 0 to 1. The default is 1, which logs every request. If someone set it to 0.01 to save cost, 99 out of every 100 failing requests leave no log, and a rare 1101 can go unrecorded.
##### Filter Workers Logs down to the failing requests
1. In the Cloudflare dashboard, go to **Workers & Pages**.
2. Select your Worker, then **Observability**.
3. Filter on `$workers.outcome = "exception"` to list every request that ended in an uncaught exception.
4. For a wider net, filter on `$metadata.error EXISTS` to see every log line that has an error attached.
5. Open one event and read the exception name, message and stack.
For a quick check on scale, the Worker's metrics page has an **Errors by invocation status** chart. The **Uncaught Exception** series counts invocations that ended in an uncaught exception, which is the failure behind a 1101. **Exceeded CPU Time Limits** and **Exceeded Memory** are counted separately.
##### Tail the Worker live while you reproduce the error
`wrangler tail` streams invocation logs, your own `console.log` lines and uncaught exceptions as they happen. Filter it to errors, or on a busy Worker the stream fills with successful requests:
```bash
npx wrangler tail my-worker --status error --format pretty
# JSON output: exceptions are in the exceptions field
npx wrangler tail my-worker --status error | jq '.exceptions'
# Narrow to requests from your own machine while you reproduce
npx wrangler tail my-worker --status error --ip self
```
Each tail entry is a JSON object with an `outcome`, an `exceptions` array, a `logs` array and the request details under `event.request`. Once you've shipped a fix, keep the tail running to confirm the exceptions have stopped.
##### Turn on source maps so the stack trace points at your code
Most Workers are bundled and minified before deploy. Without source maps, a stack trace points at one minified JavaScript file, with wrong line numbers and no TypeScript file names. Set `upload_source_maps` to `true`, as in the config above, and Wrangler generates and uploads source maps on every `wrangler deploy` or `wrangler versions deploy`. This needs Wrangler 3.46.0 or later, and each source map can be up to 15 MB gzipped.
Cloudflare applies the source map after the invocation ends, and you see the mapped trace in real-time logs and Tail Workers. Logging `err.stack` from inside your Worker still prints the minified trace, because source maps aren't available inside the Worker at runtime.
##### Line up the first failures with a deploy
Most 1101 spikes start at a deploy. List recent versions and deployments, then tail a single version:
```bash
npx wrangler deployments list
npx wrangler versions list
npx wrangler tail my-worker --status error --version-id
```
If you use gradual deployments, traffic is split between two versions. Filtering by version ID shows whether only the new version throws.
#### Match the exception message to its cause
##### `The script will never generate a response`
The runtime found that all the code for the request had finished, with nothing left in the event loop, and yet no Response had been returned. Cloudflare names two causes.
**A promise that never resolves or rejects.** Somewhere on the path to your Response, a promise waits on something that never happens. In Node or a browser this code would hang forever. Workers throws an error so you can find it. Make sure every path settles the promise:
```ts
// Hangs if "ready" never fires: nothing settles the promise
function waitForReady(emitter) {
return new Promise((resolve) => {
emitter.on("ready", resolve);
});
}
// Every path settles
function waitForReady(emitter, ms = 5000) {
return new Promise((resolve, reject) => {
emitter.on("ready", resolve);
setTimeout(() => reject(new Error("ready timed out")), ms);
});
}
```
Check your dependencies too, since the promise that never settles is often inside a library. The typescript-eslint rule `no-floating-promises` reports promises that are created and never handled.
**A WebSocket whose server side is never closed.** If the `close` handler never calls `server.close()`, the connection stays open and the runtime throws this error. With the `web_socket_auto_reply_to_close` compatibility flag, the runtime completes the close handshake for you. The flag is on by default for compatibility dates on or after `2026-04-07`, so moving your compatibility date forward makes this cause much less likely.
##### `TypeError: Illegal invocation`
The full message says a function was called with an incorrect `this` reference. It usually comes from destructuring a runtime object whose methods depend on `this`, and `ctx` is the usual one:
```ts
// Throws: waitUntil has lost its this
const { waitUntil } = ctx;
waitUntil(promise);
// Works
ctx.waitUntil(promise);
// Also works, if you need a standalone reference
const waitUntilBound = ctx.waitUntil.bind(ctx);
```
Thrown on the request path and left uncaught, it becomes a 1101.
##### `Cannot perform I/O on behalf of a different request`
Streams, request bodies and responses belong to the request that created them. This error means one request tried to use an I/O object that another request created. The usual cause is caching a `Response` or `Request` in global scope. It needs a second request to show up, so a single test request won't trigger it. Send two requests in quick succession in local development and you can reproduce it. Keep the data in global scope and build a new Response each time:
```ts
let cachedBody = null;
export default {
async fetch(request, env, ctx) {
if (cachedBody) return new Response(cachedBody);
const upstream = await fetch("https://api.example.com/config");
cachedBody = await upstream.text();
return new Response(cachedBody);
},
};
```
If you need state shared across requests, use Durable Objects. For a cache shared across requests, use Workers KV.
##### A plain TypeError, ReferenceError or rejected promise
Cloudflare's support page lists the everyday causes: undefined variables or functions, type errors, promise rejections and failed network requests. Some common forms:
- Reading a property of `undefined`, often an `env` binding or secret that exists in one environment and is missing in another.
- Calling `await res.json()` on an upstream that returned an HTML error page, which throws a `SyntaxError`.
- A rejected `fetch` or binding call with no `try...catch` around it.
The stack trace, once it's source mapped, gives you the file and line. Compare the failing line with the diff of the most recent deploy.
##### `Network connection lost` from a fetch or binding
This is a connection failure. Cloudflare's advice is to catch the `fetch` or binding call and retry it. Put a small retry with a limit around calls to upstreams that sometimes drop connections.
#### Stop the errors now, then fix the code
##### Roll back to the last good version
A rollback immediately creates a new deployment from the version you pick, and that deployment serves traffic on all your routes and domains:
```bash
# Roll back to the version uploaded before the latest one
npx wrangler rollback
# Roll back to a specific version, skipping the prompts
npx wrangler rollback --message "Roll back: 1101 on /checkout"
```
In the dashboard, go to your Worker, then **Deployments**. Select the three-dot menu on the version you want and choose **Rollback**. If a split deployment is live, rolling back replaces both versions with the one you chose, at 100% of traffic.
Before you roll back, know these limits:
- You can only roll back to the 100 most recently published versions.
- A rollback does not change bindings or the resources they point to. If the data shape changed between versions, the old code can throw against the new data.
- A rollback is blocked if a Durable Object class migration happened in between, or if the target version binds to an R2 bucket, KV namespace or queue that no longer exists.
##### Fall back to your origin when the Worker sits in front of one
If your Worker proxies to an origin, call `ctx.passThroughOnException()` at the top of the handler. When your code throws an unhandled exception, the request then goes to the origin as if the Worker weren't there. It doesn't cover errors from the origin `fetch()` itself, so wrap that call in `try...catch`. It also can't replay a request body the failed attempt already read.
```ts
export default {
async fetch(request, env, ctx) {
ctx.passThroughOnException();
return fetch(request);
},
};
```
##### Reproduce the exception locally before redeploying
`wrangler dev` runs your Worker on `localhost:8787`, and `console.log` output and exceptions print in the terminal. Send the request that failed in production:
```bash
npx wrangler dev
curl -i http://localhost:8787/checkout
# If the bug depends on real data in KV, D1 or R2
npx wrangler dev --remote
```
Once you can reproduce it, write a test that fails for the same reason, fix the code, and watch the test pass before you deploy.
#### Catch exceptions at the top of the handler
A top-level `try...catch` turns an unexpected exception into a response you control, and logs the details as structured fields you can filter on:
```ts
export default {
async fetch(request, env, ctx) {
try {
return await handle(request, env, ctx);
} catch (err) {
console.log({
level: "error",
message: "unhandled exception",
error: String(err),
stack: err instanceof Error ? err.stack : undefined,
path: new URL(request.url).pathname,
});
return new Response("Something went wrong", { status: 500 });
}
},
};
```
Three details make this work:
- **Write `return await`.** A bare `return handle(...)` hands back the promise without awaiting it, so if it rejects, the rejection happens outside the current function. The `catch` misses it and the request still ends as a 1101. Awaiting the promise throws the exception into the current function, where the `catch` can handle it.
- **Log objects.** Workers Logs pulls out and indexes the fields of a JSON log line, so you can filter on `level` or `path` directly, without a text search.
- **Watch the outcome change.** A caught exception no longer counts as an exception outcome, so the `$workers.outcome = "exception"` filter stops finding it. Filter on your own `level` field to find these errors.
If you also send errors to an outside service, pass the request to `ctx.waitUntil()`. Otherwise it can be cancelled when the invocation ends.
#### What goes wrong while chasing a 1101
- **The logs are sampled or turned off.** A low `head_sampling_rate` drops most failing requests. Fix: set it to 1 while you investigate, or keep sampling and rely on `wrangler tail --status error` during the incident.
- **The tail enters sampling mode.** On a busy Worker, real-time logs drop messages and print a warning. Fix: add filters such as `--status error` or `--ip self`. It can take up to a minute to leave sampling mode after you add a filter.
- **Teammates can't open the tail.** At most 10 clients can view a Worker's logs at once, counting dashboard sessions and `wrangler tail` together. Fix: close idle sessions during an incident.
- **The stack trace is minified.** Fix: set `upload_source_maps` and redeploy.
- **The evidence ages out.** Workers Logs keeps logs for 3 days on the Free plan and 7 days on the Paid plan, and real-time logs store nothing. Fix: copy the exception and Ray ID into the incident notes, and export logs through OpenTelemetry, Logpush or a Tail Worker if you need them for longer.
- **You debug a 1102 as if it were a 1101.** A CPU limit is a different failure, and its fix is to do less work per request, so exception handling won't solve it. Fix: read the code on the error page and the **Errors by invocation status** chart before you start.
- **The rollback throws too.** The older code meets data or bindings that have changed since. Fix: check what changed between the two versions before you roll back.
#### Keeping 1101s out of production
- Observability is on, with `head_sampling_rate` at a level that still catches rare errors.
- `upload_source_maps` is set, so every exception maps back to a real file and line.
- The `no-floating-promises` lint rule runs in CI.
- The fetch handler has a top-level `try...catch` with `return await` and structured error logs.
- Calls to upstreams and bindings have a timeout, and a retry where it's safe to retry.
- Nothing request-scoped (Request, Response, streams) lives in global scope.
- The compatibility date is recent enough to pick up runtime fixes such as automatic WebSocket close.
- Risky changes go out as gradual deployments, and you watch errors per version with `--version-id`.
- Exception logs flow to the tool that pages your team, so a spike reaches you before users report it.
#### Watching Workers for uncaught exceptions with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to Cloudflare, so when a Worker starts throwing uncaught exceptions it works out why, and the fix arrives ready for your team's normal review.
Running on Cloudflare? See [how Praxen monitors Cloudflare in production](https://getpraxen.app/for/cloudflare/).
#### Common questions
**Is error 1101 caused by Cloudflare or by my code?**
Almost always by your code. Cloudflare defines 1101 as the Worker throwing a JavaScript exception, so the exception came from your code or a library it calls. Other 11xx codes usually mean a problem in the Workers runtime itself, and for those you should check the Cloudflare status page first.
**What is the difference between error 1101 and error 1102?**
1101 means the Worker threw an exception it didn't catch. 1102 means the Worker went over its CPU time limit. A 1102 has nothing to do with exception handling, and the fix is to cut the work done per request. The Errors by invocation status chart counts them separately, as Uncaught Exception and Exceeded CPU Time Limits.
**Why can't I find the exception in my Worker logs?**
Check that observability is enabled in your Wrangler config, and that head_sampling_rate isn't set so low that it drops most requests. Workers Logs keeps logs for 3 days on the Free plan and 7 days on the Paid plan, and real-time logs store nothing at all. Run npx wrangler tail with --status error while you reproduce the request to see the exception live.
**Why does my 1101 stack trace point at minified code?**
Your source maps aren't uploaded. Add upload_source_maps set to true in your Wrangler config (Wrangler 3.46.0 or later) and redeploy. Cloudflare then maps stack traces back to your original files in real-time logs and Tail Workers. A single source map can be up to 15 MB gzipped.
**How do I roll back a Worker that started throwing 1101 errors?**
Run npx wrangler rollback to go back to the version uploaded before the latest one, or pass a version ID from npx wrangler versions list. You can also use the Rollback option on the Deployments tab in the dashboard. You can roll back to any of the 100 most recent versions, but bindings aren't changed, and a Durable Object migration or a deleted R2 bucket, KV namespace or queue blocks the rollback.
**What does The script will never generate a response mean?**
It's a 1101 that the runtime raises when all your code has finished and the event loop is empty, but no Response was returned. Cloudflare names two causes: a promise that never resolves or rejects, and a WebSocket whose server side is never closed. Make sure every path settles its promise, and turn on the no-floating-promises lint rule to catch unhandled promises.
**Can I replace the 1101 error page with my own response?**
Yes. Wrap your handler in try...catch, use return await so a rejected promise throws inside the function where the catch can handle it, and return your own 500 response. If the Worker sits in front of an origin, ctx.passThroughOnException() sends the request to the origin whenever your code throws an unhandled exception.
#### Sources
- [Errors and exceptions (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/errors/)
- [Error 1101 (Cloudflare Support docs)](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1101/)
- [Workers Logs (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/logs/workers-logs)
- [Real-time logs (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/logs/real-time-logs/)
- [Source maps and stack traces (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/observability/source-maps/)
- [Wrangler Workers commands (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/wrangler/commands/workers/)
- [Rollbacks (Cloudflare Workers docs)](https://developers.cloudflare.com/workers/configuration/versions-and-deployments/rollbacks/)
- [await (MDN Web Docs)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen Cloudflare integration](https://docs.getpraxen.app/integrations/cloudflare)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
Next: [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
#### Related
- [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix.
- [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review.
- [Fix Cloudflare “Error 1102: Worker exceeded resource limits”](https://getpraxen.app/learn/troubleshooting/cloudflare-error-1102-worker-exceeded-resource-limits/): Why a Cloudflare Worker returns Error 1102, how to tell a CPU time overrun from a 128 MB memory overrun, and the code and Wrangler changes that fix each.
---
### Is it safe to give an AI SRE read access to production?
> Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist.
Part 1 of [Choosing AI for production](https://getpraxen.app/series/choosing-ai-for-production/)
By Boris Tane, Founder of Praxen · Published September 29, 2026 · 12 min read
Canonical: https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/
Yes, read access is a sound first step for an AI SRE, as long as you define read narrowly: resource metadata, logs, metrics, traces and code, with secret values and customer data stores left out. The agent cannot change anything with that access, so the risk that remains is disclosure: the AWS ReadOnlyAccess policy reads S3 and DynamoDB data, a Kubernetes read role that includes Secrets exposes their values, and whatever the agent reads flows into a model and anywhere it can send data. Scope the role, control egress, keep credentials outside the agent's runtime and check what the vendor stores.
Yes. Read access is the right way to start with an AI SRE, and for most teams it is safe enough, provided you define "read" narrowly: resource metadata, logs, metrics, traces and code, with secret values and customer data stores left out. The agent cannot change production with that access, so the risk that remains is disclosure: what the agent can see, where that text goes next, and who else can steer it.
That last part is where teams get caught. The AWS ReadOnlyAccess policy reads data in S3 and DynamoDB, and a Kubernetes read role that includes Secrets returns their values. An agent that can read also sends what it reads to a language model, may save it to memory, and may be able to call out to the internet. This page covers what read access really exposes, the four ways it goes wrong, the exact roles to grant, and a checklist to run before you connect a production account.
#### What "read access to production" actually covers
An AI SRE investigates by calling the same read APIs an engineer uses: list and describe resources, query logs and metrics, pull traces, read deploy history and code. Each call runs under a credential, and that credential's permissions decide what "read" means. Two defaults surprise people.
**Kubernetes.** In RBAC, `get` on Secrets reveals their values, and so do `list` and `watch`. The Kubernetes docs point out that a list response, such as the one behind `kubectl get secrets -A -o yaml`, includes the contents of every Secret. The second trap is `get` on the `nodes/proxy` subresource. It reaches the kubelet API, which can run commands in any pod on that node, and it bypasses audit logging and admission control. A role made only of `get` verbs can still be a remote shell.
**AWS.** The managed ReadOnlyAccess policy lists, gets and describes every resource in the account. AWS warns that it also reads data in storage services such as S3 buckets and DynamoDB tables, and it includes read access to IAM and Billing. The ViewOnlyAccess job function is narrower: it shows resources and basic metadata and cannot read resource content. AWS keeps its job function policies up to date as new services and capabilities launch, so a managed read role grows over time without anyone on your team changing it.
So "read-only" answers one question: can the agent change things? It tells you nothing about whether it can read your database rows or your API keys.
#### Why read-only access still carries risk: four ways it goes wrong
A public write-up of incidents from an operations agent that Microsoft runs in production is a useful guide, because none of the incidents needed an attacker. Its authors conclude that if the environment permits an action, the agent will eventually take it, whether by accident or because someone steered it.
##### Secret values come back from read calls
If the role can read Secrets, secret stores or parameter stores, a routine investigation pulls credentials into the agent's context. In the Microsoft write-up, the agent found a credential committed to a customer repository, quoted it in its findings and saved it to memory with a note never to use it. The note came too late. The secret already sat in an investigation summary and a memory store, and neither was covered by anyone's rotation playbook.
**Fix:** keep secret values out of reach. A pod spec shows which Secret a container references, which is enough to diagnose a missing or wrong reference. The Kubernetes guidance is to restrict `get`, `list` and `watch` on Secrets, and to grant `list` or `watch` only to the most privileged system components.
##### Customer data lands in prompts, transcripts and memory
Anything the agent reads goes into a model prompt. Depending on the product, it may also be stored in a transcript, a summary, a memory or a shared thread. Logs are the usual carrier: they hold whatever your code prints, which is why the Kubernetes docs tell developers to avoid logging secret data in the clear. An agent that can read S3 objects or DynamoDB items can pull customer records into the same places.
**Fix:** leave object and table contents out of the role. Ask the vendor where inference runs, how long transcripts and memories are kept, whether your data trains models, and whether you can use your own model provider.
##### Text in your logs can steer the agent
An agent that reads logs, tickets and alert payloads reads text that outsiders can influence. A request path, a user agent string or an error message can be written to look like an instruction. The Microsoft write-up makes the point that a hallucinated command and an injected one are the same command once they reach execution. The dangerous pairing is read access plus a way out. In one Microsoft case, the agent was asked to read a screenshot and had no vision tool, so it found a free OCR service on the public internet and posted the image to it. Nobody attacked anything, and customer data could still have ended up on a third party's server.
**Fix:** control egress at a boundary the model cannot modify. If the agent runs code, it should reach only the endpoints the task needs.
##### A read credential can be copied or replaced
If the credential sits on a filesystem the agent can read, the agent can reuse it, or rebuild it. In the same write-up, an agent's short-lived GitHub token expired. It read its own harness code, rebuilt the OAuth device-code flow, asked a researcher to finish the login and wrote the new access and refresh tokens to disk for reuse. The harness was supposed to decide what authority the agent had, and the agent replaced it from the inside.
**Fix:** keep credentials outside the agent's runtime and issue them short-lived and per task. The strongest pattern gives tools a handle that a proxy swaps for a real credential at the network boundary, so the sandbox can use a credential without ever holding one.
#### Scoping a Kubernetes read role that leaves Secrets out
1. **Create a dedicated ServiceAccount.** Never hand the agent a human's kubeconfig or an existing admin account. The Kubernetes docs advise against `cluster-admin` except where it is specifically needed.
2. **Name every resource.** Wildcards grant access to every object type that exists today and every type added later, Secrets included.
3. **Keep nodes in a separate small role**, and never include `nodes/proxy`. Container logs come from `pods/log`.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ai-sre-reader
namespace: ai-sre
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ai-sre-read
rules:
# Core resources by name. No secrets, no wildcards.
- apiGroups: [""]
resources: ["pods", "pods/log", "services", "endpoints", "events", "configmaps", "persistentvolumeclaims"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ai-sre-read-nodes
rules:
- apiGroups: [""]
resources: ["nodes"] # never nodes/proxy
verbs: ["get", "list", "watch"]
```
ConfigMaps are meant for non-confidential data. If your teams store credentials in them anyway, drop `configmaps` from the list until that is fixed.
4. **Bind per namespace where you can.** RoleBindings keep the grant inside the namespaces the agent should see. Nodes are cluster-scoped, so that one role needs a ClusterRoleBinding.
```bash
kubectl create rolebinding ai-sre-read --clusterrole=ai-sre-read --serviceaccount=ai-sre:ai-sre-reader -n payments
kubectl create clusterrolebinding ai-sre-read-nodes --clusterrole=ai-sre-read-nodes --serviceaccount=ai-sre:ai-sre-reader
```
5. **Verify the grant yourself.** Install charts and vendor docs drift. Test the permissions that must be absent.
```bash
SA=system:serviceaccount:ai-sre:ai-sre-reader
kubectl auth can-i list secrets --as=$SA -A
kubectl auth can-i get nodes/proxy --as=$SA
kubectl auth can-i create pods/exec --as=$SA -n payments
kubectl auth can-i delete pods --as=$SA -A
```
Each line should print:
```text
no
```
Then run `kubectl auth can-i --list --as=$SA -n payments` and read the full list. Repeat after every upgrade of the agent's chart.
#### Scoping an AWS read role that leaves data stores out
1. **Start from ViewOnlyAccess.** Skip ReadOnlyAccess: it reads S3 objects and DynamoDB items.
2. **Add the telemetry reads the agent needs by name.** ViewOnlyAccess covers listing and basic metadata, so grant log, metric and trace queries explicitly.
3. **Add an explicit deny for data and secret reads.** An explicit deny still holds if someone later attaches a broader policy to the same role.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TelemetryReads",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics",
"cloudwatch:DescribeAlarms",
"logs:DescribeLogGroups",
"logs:FilterLogEvents",
"logs:StartQuery",
"logs:GetQueryResults",
"xray:GetTraceSummaries",
"xray:BatchGetTraces"
],
"Resource": "*"
},
{
"Sid": "NoDataOrSecretReads",
"Effect": "Deny",
"Action": [
"s3:GetObject",
"dynamodb:GetItem",
"dynamodb:BatchGetItem",
"dynamodb:Query",
"dynamodb:Scan",
"secretsmanager:GetSecretValue",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath",
"kms:Decrypt"
],
"Resource": "*"
}
]
}
```
4. **Simulate before you trust it.**
```bash
aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:role/ai-sre-reader --action-names s3:GetObject secretsmanager:GetSecretValue logs:StartQuery --query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text
```
```text
s3:GetObject explicitDeny
secretsmanager:GetSecretValue explicitDeny
logs:StartQuery allowed
```
Log queries still return log lines, and those can contain personal data. That is the point of the access, so treat it as customer data and apply the controls in the next section.
For AWS investigations, define the evidence reads around the failure. [EC2ThrottledException](/learn/troubleshooting/EC2ThrottledException/) needs function state and EC2 activity; [AsyncEventsDropped](/learn/troubleshooting/AsyncEventsDropped/) needs async configuration, errors and throttles. An SQS [ChangeMessageVisibility failure](/learn/troubleshooting/ChangeMessageVisibility/) can be investigated from worker logs and queue configuration, while receiving or extending a message changes queue behavior and belongs to a separate operational identity.
Also separate the investigator from telemetry delivery. Installing [LambdaInsightsExtension](/learn/troubleshooting/LambdaInsightsExtension/) requires configuration changes and collector log-write permissions. Debugging [AWS_EMF_ENVIRONMENT](/learn/troubleshooting/AWS_EMF_ENVIRONMENT/) starts with the library's emitted JSON and its delivery path; read access to CloudWatch cannot repair a collector that never sends data.
#### Controlling where production data goes after the agent reads it
A tight role limits what the agent can see. These controls limit what happens next:
- **Egress:** default-deny outbound traffic from wherever the agent runs code, with a short allowlist such as package registries.
- **Credentials:** short-lived, issued per task, stored outside the agent's filesystem and memory.
- **Secret scrubbing:** tools that remove recognised secrets from tool output before it reaches the model.
- **Data terms:** where inference runs, retention for transcripts and memories, a model-training opt-out, and whether you can bring your own model provider key.
- **Memory hygiene:** a way to read, edit and delete what the agent saved, and a rule that any secret it saw gets rotated.
#### Keeping the read identity separate from any write path
Read access is usually a stage before write access, so design for the second stage now. Give writes a separate identity, so a leaked read credential cannot change anything and your audit logs show reads and writes under different names. The Microsoft team's production rule is a sensible default: reads run on their own, every write waits for human approval, and deletes are blocked.
The approval gate has to sit outside the agent and catch every write path. In one of their incidents, the agent was told to deallocate VMs only after five safety checks. The logging service went down after the third check, and the agent matched the situation to a past memory where deallocating had been safe and went ahead through a path the approval policy never intercepted. Code changes are the easy case: send them as pull requests, and your branch protection, reviewers and CI stay in charge of the merge.
#### Read-role mistakes that bite, and the fix for each
- **Attaching ReadOnlyAccess because the name sounds safe.** It reads S3 and DynamoDB data. Use ViewOnlyAccess plus named telemetry reads.
- **Wildcards in a ClusterRole.** A wildcard over core resources includes Secrets and every future type. List resources by name.
- **Granting `nodes/proxy` to reach container logs.** `get` on it can run commands in pods. Use `pods/log`.
- **Leaving a long-lived key where the agent can read it.** The agent can reuse or rebuild it. Issue short-lived credentials from outside its runtime.
- **Forgetting that the agent remembers.** Transcripts and memories outlive the investigation. Review them, and rotate any secret the agent saw.
- **Checking the role once.** AWS updates managed policies and charts change between versions. Re-run `kubectl auth can-i` and the IAM simulation after every upgrade, and review RBAC bindings on a schedule.
- **Treating a confident answer as a verified one.** Read access cannot break production, but the person who acts on a wrong diagnosis can. Require a citation to a query, log line or commit for every claim.
#### Checklist before you connect a production account
1. The agent has its own identity in every cluster and cloud account.
2. Kubernetes roles name resources explicitly and exclude Secrets, `nodes/proxy` and `pods/exec`.
3. AWS roles start from ViewOnlyAccess, add telemetry reads by name and explicitly deny data and secret reads.
4. `kubectl auth can-i` and `aws iam simulate-principal-policy` confirm the negatives.
5. Outbound network access from the agent's code sandbox is default-deny.
6. Credentials are short-lived and never stored in the agent's runtime.
7. You know where inference runs, what is retained, and how to opt out of model training.
8. You can read and delete the agent's memories.
9. Any write path uses a separate identity behind human approval, with deletes blocked.
10. You re-check all of the above after every upgrade.
Running on AWS? See [how Praxen monitors AWS in production](https://getpraxen.app/for/aws/).
#### Common questions
**Can an AI SRE read Kubernetes Secrets with a read-only role?**
Yes, if the role grants get, list or watch on Secrets. The Kubernetes docs note that list and watch reveal Secret contents as well as get, and a list response returns every Secret's values. Leave Secrets out of the role: a pod spec still shows which Secret a container references, which is enough to diagnose a missing reference.
**Is the AWS ReadOnlyAccess policy safe to give an AI SRE?**
It is broader than the name suggests. AWS states that ReadOnlyAccess can read data in storage services such as S3 buckets and DynamoDB tables, and it includes read access to IAM and Billing. Start from the ViewOnlyAccess job function, which shows resources and basic metadata, then add the log, metric and trace reads the agent needs by name.
**Can an agent with only read access still cause harm?**
It cannot change resources, but it can disclose them. The main paths are secret values pulled into its context, customer data stored in transcripts or memory, and data sent out if the agent can reach the internet while running code. Watch one Kubernetes exception: get on nodes/proxy can run commands in pods, so in practice it is a write permission.
**What should I ask a vendor before connecting production?**
Ask where model inference runs, how long transcripts and memories are kept, whether your data trains models and how to opt out, and whether you can bring your own model provider key. Also ask whether the agent can make outbound network requests when it runs code, and whether credentials ever sit inside the agent's runtime.
**How do I check what the agent's credentials can actually do?**
In Kubernetes, run kubectl auth can-i --list --as=system:serviceaccount:NAMESPACE:NAME, then test negatives such as list secrets and get nodes/proxy, which should print no. In AWS, run aws iam simulate-principal-policy against the role with actions such as s3:GetObject and secretsmanager:GetSecretValue. Repeat after every upgrade, because managed policies and install charts change.
**Do logs expose secrets to the agent?**
They can. Logs hold whatever your code prints, which is why the Kubernetes docs tell developers to avoid logging secret data in the clear. Treat log read access as access to customer data, fix code that logs tokens, and prefer tools that scrub recognised secrets before they reach the model.
**When should an AI SRE get write access?**
After it has a track record of correct diagnoses on read access, and then through a separate identity. Keep a human approval on every write, send code changes as pull requests through your normal review, and block deletes. Make sure the approval gate sits outside the agent and catches every write path.
#### Sources
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Role Based Access Control Good Practices (Kubernetes)](https://kubernetes.io/docs/concepts/security/rbac-good-practices)
- [Good practices for Kubernetes Secrets](https://www.kubernetes.io/docs/concepts/security/secrets-good-practices/)
- [AWS managed policies for job functions (AWS IAM User Guide)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html)
- [Stop restricting the agent. Start restricting its environment (Microsoft Command Line)](https://commandline.microsoft.com/azure-sre-agent-restricting-environment-ai-safety/)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [How to correlate incidents with deploys using evidence](https://getpraxen.app/learn/deployment-safety/how-to-correlate-incidents-with-deploys/): Learn how to correlate incidents with deploys using release markers, metrics, logs and verified GitHub, Vercel, Cloudflare and Kubernetes change events.
- [What is change intelligence? A production safety guide](https://getpraxen.app/learn/deployment-safety/what-is-change-intelligence/): What is change intelligence? Learn how deploys, config, flags, infrastructure and dependency changes connect to incidents, using tools you already have.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [AWS_EMF_ENVIRONMENT: valid values and missing EMF fixes](https://getpraxen.app/learn/troubleshooting/AWS_EMF_ENVIRONMENT/): AWS_EMF_ENVIRONMENT overrides EMF library detection. Check Node, Python and Java values separately, trace stdout or agent delivery, and fix missing metrics.
---
### What is an AI SRE? How it works and how to evaluate one
> An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
By Boris Tane, Founder of Praxen · Published September 28, 2026 · Updated September 29, 2026 · 5 min read
Canonical: https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/
An AI SRE is an AI agent that connects to your observability tools, cloud accounts and code. It triages alerts, investigates incidents and proposes fixes without waiting for someone to prompt it. It uses a large language model with tool access to query logs, metrics, traces and deploy history, and it should show the evidence behind each conclusion while a person approves risky actions.
#### AI SRE, defined
An AI SRE (AI site reliability engineer) is software that does the investigative part of on-call work. It watches telemetry and triages alerts. It works out what changed, finds the likely root cause and proposes a fix. It works as an autonomous agent: it decides what to query and acts on its findings without a human typing a prompt.
The term is new and loose. Many products now carry the label, so the label alone tells you little. What matters is what the tool does when an alert fires at night.
#### How it works
Most AI SREs follow the same loop, whatever the vendor.
1. **Build context.** The agent connects to your clouds, repositories, observability tools and chat. It maps services, their dependencies, recent deploys and past incidents, so it reasons about your actual system. Good tools ground the model with retrieval over your runbooks, post-mortems and service topology.
2. **Triage.** When an alert fires, the agent reads the data around it and groups related alerts into one incident. Then it decides whether the problem is real. Most alerts need no action, so this step removes a lot of noise.
3. **Investigate.** It forms several hypotheses, such as a bad deploy, a config change or a saturated dependency. It checks each one against logs, metrics, traces and code. Good tools test hypotheses in parallel and drop the ones the evidence rules out.
4. **Explain.** It writes up the cause as a chain: the signal, the change that triggered it, the code path and the blast radius. Each link should cite a query, log line or commit.
5. **Remediate.** It proposes a rollback, a config change or a code fix as a pull request. A person approves anything risky.
6. **Document and learn.** It drafts the timeline and post-mortem. It also saves confirmed findings so the next investigation starts faster.
#### AI SRE vs AIOps, copilots and runbooks
These tools overlap, so it helps to be precise.
- **Runbook automation** runs scripts someone wrote in advance. It handles only the failures you predicted.
- **AIOps** groups and deduplicates alerts with statistics. It tells you signals are related and leaves the cause to you.
- **Copilots** answer questions when you ask them.
- **AI SREs** start work when the incident starts and carry it from triage to a proposed fix.
Also watch for AI assistants built into a single observability platform. They can only reason over the data that platform holds, and many incidents cross a cloud, a database and a deploy.
#### Levels of autonomy
Autonomy comes in four levels, and most teams move up them one action type at a time:
1. **Observe and report:** the agent investigates every alert and posts findings with evidence.
2. **Advise:** it ranks remediations by confidence, and you approve or reject each one.
3. **Act with approval:** it prepares the fix and runs it after one human approval.
4. **Autonomous for known patterns:** it resolves well-understood, low-risk incidents on its own, within the policy you set.
Start at level one. Raise autonomy for an action type once the agent has been right on it many times. Keep anything that touches customer data behind approval.
#### How to evaluate an AI SRE
Run a trial on your own stack with these steps:
1. Pick a few resolved incidents where you already know the root cause. Ask the tool about the same time window and compare its answer with yours.
2. Check every claim for a citation. Click through to the query, log line or commit. A step with no evidence should be marked as unknown.
3. Connect with read-only credentials first. Confirm that write actions pause for approval and that code changes arrive as pull requests your reviewers and CI gate.
4. Count the noise. Track how many alerts it dismisses and spot-check those dismissals.
5. Test a cross-service incident, such as a deploy that exhausts a database connection pool. See whether it follows the path from the symptom to the change.
6. Measure the time from alert to a root cause you trust, and compare it with your current baseline.
#### Get your stack ready
An agent inherits the quality of your telemetry. You can improve results today with the tools you already have:
- Emit structured logs with a service name and request ID on every event.
- Record deploys with commit SHAs, so changes line up with metric shifts.
- Keep infrastructure-as-code and deploy manifests in the repository, so a tool can link code to the resources it ships to.
- Write down service ownership, for example in a CODEOWNERS file, and keep runbooks close to the code.
- Save the dashboards and queries your team trusts. Many tools reuse them as checks.
#### Common mistakes
- **Trusting a confident answer without reading the evidence.** Language models can produce plausible wrong root causes. The safeguard is a glass box: every suggestion links to the data behind it.
- **Granting write access on day one.** Earn autonomy per action type.
- **Judging each step in isolation.** Evaluate the whole run on a real incident, from alert to fix.
- **Expecting it to replace engineers.** It takes on the toil. People keep design, risk calls and new failure modes with no history to learn from.
#### Doing this with Praxen
Praxen detects production issues and fixes them by opening pull requests. It connects to the platforms your team already runs on, so it can see when something breaks there, works out why, and sends a fix ready for your normal review.
See [Praxen as your AI SRE](https://getpraxen.app/use-cases/ai-sre/).
#### Common questions
**Will an AI SRE replace my SRE team?**
No. It takes on the toil of triage, investigation and documentation, while people keep architecture, risk decisions and new failures with no precedent.
**Is it safe to give an AI SRE access to production?**
Start with read-only access and require human approval for any write. Let the agent propose fixes for engineers to approve, and send code changes through your normal review.
**How is an AI SRE different from AIOps?**
AIOps groups and deduplicates alerts with statistical models. An AI SRE uses a language model with tool access to investigate, explain the cause with evidence and propose a fix.
**How do I evaluate an AI SRE?**
Trial it on incidents you have already resolved: ask about the same time window and compare its answer with the root cause you found. Check that every step cites a query, log line or commit, connect it read-only first, and measure the time from alert to a root cause you trust.
**What should I have in place before adopting an AI SRE?**
Good telemetry, because the agent inherits its quality. Emit structured logs with a service name and request ID, record deploys with commit SHAs, keep infrastructure-as-code and deploy manifests in the repository, and write down who owns each service.
#### Sources
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### Related
- [Workers Issues: route Cloudflare errors to coding agents](https://getpraxen.app/learn/incident-response/cloudflare-workers-issues-error-monitoring-agents/): Cloudflare's new Workers Issues groups exceptions and 5xx errors and sends them to Claude Code, Cursor, Devin or a webhook. Setup, automations and limits.
- [How to build a memory system for proactive agents](https://getpraxen.app/blog/how-to-build-a-memory-system-for-proactive-agents/): Proactive agents run without a prompt, so each run has to start from what earlier runs learned. How we built memory for Praxen's agents on git: one repository per team, a nightly clean-up, and lessons from merged pull requests.
- [AsyncEventsDropped: diagnose Lambda async event loss](https://getpraxen.app/learn/troubleshooting/AsyncEventsDropped/): AsyncEventsDropped counts Lambda async events discarded after age or retry limits. Check throttling, destinations and DLQs, then verify processing recovery.
- [Fix Cloudflare “Error 1102: Worker exceeded resource limits”](https://getpraxen.app/learn/troubleshooting/cloudflare-error-1102-worker-exceeded-resource-limits/): Why a Cloudflare Worker returns Error 1102, how to tell a CPU time overrun from a 128 MB memory overrun, and the code and Wrangler changes that fix each.
- [Fix Vercel “FUNCTION_INVOCATION_FAILED” 500 errors](https://getpraxen.app/learn/troubleshooting/vercel-function-invocation-failed/): Why a Vercel Function returns 500 FUNCTION_INVOCATION_FAILED, how to find the crash in runtime logs, and how to fix throws, missing env vars, files and memory.
---
### How to monitor a FastAPI app on Railway: logs, traces and alerts
> Set up health checks, JSON logs, OpenTelemetry traces, monitors and crash alerts for a FastAPI service on Railway, with the commands and config to copy.
Part 2 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 29, 2026 · 13 min read
Canonical: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/
Give the FastAPI app a /health route that Railway checks on each deploy, write single-line JSON logs to stdout, and turn on tracing so Uvicorn runs under opentelemetry-instrument and requests show up on Railway's Traces page. Then add monitors on memory and CPU in the Observability dashboard (Pro plan), plus a project webhook that posts failed and crashed deploys to Slack or Discord. Railway does not collect request latency or error-rate metrics, and it never runs the healthcheck again after a deploy goes live, so use its HTTP logs or traces for latency and an outside uptime check for availability.
Monitoring a FastAPI app on Railway comes down to a few pieces: a `/health` route that Railway checks on every deploy, JSON logs written to stdout, OpenTelemetry traces sent to Railway's built-in collector, monitors on CPU and memory, and a project webhook that posts failed and crashed deploys to Slack or Discord. Railway graphs CPU, memory, disk and network for every service with no setup. It does not collect request latency or error rates as metrics, and it only calls your health check while a deploy goes live.
You add very little code to the app: a health route, a log formatter and one middleware. Tracing needs a package install and a new start command. The alerts are dashboard settings.
#### What monitoring means for a FastAPI service on Railway
Here, monitoring means you can answer these questions without opening a shell: is the service up, is it healthy under load, what did a slow or failed request do, and who hears about it when something breaks. Railway answers each one with a different feature, and each feature has its own limits.
- **Platform metrics.** Every service has a **Metrics** tab with CPU, memory, disk usage and network traffic. It keeps up to 30 days of data, and dotted lines mark each deploy, so you can see which commit started a memory climb. With several replicas, the **Sum** view adds them together and the **Replica** view shows each one separately.
- **Logs.** Railway captures anything your process writes to stdout or stderr, plus build and deploy logs. Railway's edge also writes an HTTP log for every request, with the path, status code and timings, and you can filter on all of them.
- **Traces.** For a traced service, Railway's edge starts a trace on each request and passes a W3C `traceparent` header to your app. If the app runs an OpenTelemetry SDK, its spans join the same trace on the **Traces** page. Tracing is a preview feature.
- **Alerts.** Project webhooks fire when a deploy changes state, including `Failed` and `Crashed`, and when a monitor alerts. Monitors on the Observability dashboard alert when CPU, RAM, disk or network egress crosses a threshold. Monitors need the Pro plan.
##### What Railway leaves to you
For a Python API, these gaps matter. Railway does not collect application metrics such as request latency, error rate or business counts. Its docs send you to a third-party tool for those. The healthcheck only runs at the start of a deploy, and Railway never uses it to keep watching the service. There is also no log drain setting, so sending logs somewhere else takes a forwarder or an SDK. The sections below close each gap with what Railway already gives you, then show when an outside tool is worth adding.
#### Add a health route and bind Uvicorn to Railway's port
Railway injects a `PORT` variable and uses it for health checks, so Uvicorn has to listen on `0.0.0.0` and `$PORT`. Add a cheap route that returns 200 once the app can serve requests:
```python
# main.py
from fastapi import FastAPI
app = FastAPI()
@app.get("/health")
async def health():
return {"status": "ok"}
```
Then set the health check path and start command. You can do this in the service's **Settings**, or in a `railway.toml` next to your code:
```toml
[deploy]
startCommand = "uvicorn main:app --host 0.0.0.0 --port $PORT --no-access-log"
healthcheckPath = "/health"
healthcheckTimeout = 300
restartPolicyType = "ON_FAILURE"
```
On each deploy, Railway calls `/health` until it gets a 2xx response, then moves traffic to the new version. If no 2xx arrives within the timeout, which is 300 seconds by default, Railway marks the deploy as failed. To give the app longer, for example when it loads a large model at startup, set the `RAILWAY_HEALTHCHECK_TIMEOUT_SEC` variable.
These details trip up FastAPI apps:
1. The check comes from the hostname `healthcheck.railway.app`. If you use `TrustedHostMiddleware`, add that host to the allowed list, or the check fails with a 400:
```python
from fastapi.middleware.trustedhost import TrustedHostMiddleware
app.add_middleware(
TrustedHostMiddleware,
allowed_hosts=["api.example.com", "*.up.railway.app", "healthcheck.railway.app"],
)
```
2. Keep `/health` shallow. If the route also queries Postgres, a short database blip turns into a failed deploy. Check dependencies from a separate route and watch that one from outside Railway (see the last setup section).
Railway has deprecated config-as-code files in favour of Infrastructure as Code. Existing `railway.toml` files keep working for legacy services until 2026-12-01. If you'd prefer not to keep a file, the same settings are in the dashboard.
#### Write one JSON log line per request to stdout
Railway treats a log line as structured when the whole JSON object sits on one line. It reads `message` and `level`, colours each line by level, and lets you filter on any other key as `@key:value`. Plain text lines are still stored. They get a level from their stream, info for stdout and error for stderr, so `@level` filters work on them, but they carry no custom attributes to filter on.
Python's `logging.StreamHandler` writes to stderr unless you tell it otherwise, and Railway converts every stderr line to `level: error`. Point the handler at stdout and healthy requests stop showing up in red:
```python
# logging_setup.py
import json
import logging
import sys
class JsonFormatter(logging.Formatter):
def format(self, record):
payload = {
"message": record.getMessage(),
"level": record.levelname.lower(),
"logger": record.name,
}
payload.update(getattr(record, "fields", {}))
if record.exc_info:
payload["exception"] = self.formatException(record.exc_info)
return json.dumps(payload, default=str)
def setup_logging():
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
```
The stack trace goes into a single `exception` field, so a crash stays as one entry you can open.
Next, log one event per request with the route, status, duration and trace ID:
```python
# main.py
import logging
import time
from fastapi import FastAPI, Request
from opentelemetry import trace
from logging_setup import setup_logging
setup_logging()
log = logging.getLogger("app")
app = FastAPI()
@app.middleware("http")
async def log_requests(request: Request, call_next):
start = time.perf_counter()
fields = {"method": request.method, "path": request.url.path}
try:
response = await call_next(request)
except Exception:
fields["duration_ms"] = round((time.perf_counter() - start) * 1000, 1)
log.exception("unhandled error", extra={"fields": fields})
raise
ctx = trace.get_current_span().get_span_context()
fields.update(
status_code=response.status_code,
duration_ms=round((time.perf_counter() - start) * 1000, 1),
trace_id=format(ctx.trace_id, "032x") if ctx.is_valid else None,
)
log.info("request", extra={"fields": fields})
return response
```
`trace_id` stays empty until you turn on tracing in the next sections. After that, every log line links to its trace.
I built observability tools at Baselime and then led the Workers observability team at Cloudflare, and this is the habit I push hardest: write one wide event per request, with enough fields that you never need a second query to explain it.
The `--no-access-log` flag in the start command matters. Railway's HTTP logs already record every request at the edge, and Uvicorn's plain-text access lines only eat into the logging limit of 500 lines per second per replica. Above that limit, Railway drops lines and prints a warning with the number dropped.
Once the app is deployed, these queries work in the Log Explorer (the **Observability** tab) and in each deployment's log panel:
```text
@level:error
@path:/orders AND @status_code:>=500
@duration_ms:>500
@level:error AND "unhandled error"
```
In a terminal, `railway logs` shows the latest deployment's logs.
Your plan sets how long Railway keeps logs, and traces are kept for the same period. If you upgrade, logs that had aged out come back straight away.
**Railway log and trace retention by plan**
| Plan | Retention |
| --- | --- |
| Free | 3 days |
| Trial | 7 days |
| Hobby | 7 days |
| Pro | 30 days |
| Enterprise | Up to 90 days |
#### Read latency and error rates from Railway's HTTP logs
Before you write any metrics code, check the edge's HTTP logs. They answer most latency and error questions. Each request has `@httpStatus`, `@responseTime` (time to first byte), `@totalDuration`, `@path`, `@method` and `@srcIp`, and the number fields accept comparisons and ranges:
```text
@httpStatus:500..599
@path:/api/v1/orders AND @httpStatus:>=400
@responseTime:>1000
@totalDuration:>5000 @httpStatus:>=500
-@httpStatus:200
```
Railway only fills `@responseDetails` when the application fails to respond, so it explains requests that never reached a FastAPI handler. These filters give you error counts and slow requests per route, which covers most of what a small team asks. They can't show where the time went inside the handler. Traces do that.
#### Turn on tracing and let OpenTelemetry instrument FastAPI
You turn tracing on per service and per environment. Once it's on, the edge traces every request to the service, and Railway adds the `OTEL_*` variables on the next deploy. There is no sample rate: Railway traces every request.
1. Enable tracing for the service in the **Tracing setup** panel on the **Traces** tab, or from the CLI:
```bash
railway trace enable --service api
railway trace status --all
```
2. Install the OpenTelemetry distribution and let it detect FastAPI and your other libraries:
```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```
`opentelemetry-bootstrap` installs the matching packages, such as `opentelemetry-instrumentation-fastapi`. Run it locally, then add what it installed to `requirements.txt` or `pyproject.toml` so the Railway build installs them too.
3. Put `opentelemetry-instrument` in front of the start command:
```bash
opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port $PORT --no-access-log
```
4. Railway's receiver only accepts traces, but the Python distribution exports metrics by default. Add these variables to the service:
```bash
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none
```
5. Redeploy, send a few requests, and wait for the **App** indicator in the Tracing setup panel to turn green.
Each request now produces a server span named after its method and route, with `http.route` and `http.response.status_code`. Calls made with `httpx`, `requests` or `aiohttp` become client spans. `asyncpg`, `psycopg2`, `SQLAlchemy` and `redis` produce spans that include the statement or command. To see your own slow work in the waterfall, wrap it in a span:
```python
from opentelemetry import trace
tracer = trace.get_tracer("checkout")
def calculate_total(cart):
with tracer.start_as_current_span("calculate-total") as span:
span.set_attribute("cart.items", len(cart.items))
return price_items(cart)
```
The Traces page and the CLI use the same filter syntax, so you can find slow routes and errors from either one:
```bash
railway trace list --since 30m --errors
railway trace list --all --filter '@http.route:/checkout @duration:>500'
```
When a user reports a failed request, ask them for the `x-railway-trace-id` response header, or include it in your error responses. Paste it into the **Trace ID** field to open that exact request.
To skip the SDK, use the **Automatic instrumentation** switch. It traces Python processes with no code changes, on a best-effort basis. The SDK gives you custom spans and full control.
Tracing has limits. Each replica can export 1,000 spans per 10 seconds, and a search returns up to 500 traces. On a busy service, set `OTEL_TRACES_SAMPLER=traceidratio` and put a fraction in `OTEL_TRACES_SAMPLER_ARG`. The default parent-based sampler follows the edge, which traces everything.
#### Put the numbers on one dashboard and add monitors
1. Open the **Observability** tab and check that the environment selector shows `production`. The dashboard only covers one environment.
2. Click **Start with a simple dashboard**. Railway creates widgets for spend, service metrics and logs.
3. Add a logs widget for the API service filtered to `@level:error`, so errors sit next to the memory graph.
4. On the memory widget, open the three-dot menu, choose **Add monitor**, and set a threshold above your normal peak. Do the same for CPU.
5. If the service has a volume, add a disk usage monitor. Add a network egress monitor if you watch bandwidth costs.
Monitors send email and in-app notifications. They also fire your project webhook, so they arrive in the same channel as failed deploys. For a Python service, the memory monitor matters most: a slow leak shows up on the graph well before the process crashes, and the alert gives you time to act.
#### Send failed and crashed deploys to Slack or Discord
You set webhooks per project, and they fire for every environment in that project.
1. Create an incoming webhook in Slack (a `hooks.slack.com` URL), or in a Discord channel under **Integrations**, then **Webhooks**.
2. In the Railway project, open **Settings**, then the **Webhooks** tab. Paste the URL, choose the events and click **Save Webhook**.
3. Trigger a real deploy to check it works. **Test Webhook** sends from your browser, so CORS on the receiving end can make a working endpoint look broken.
Railway recognises Slack and Discord URLs and reformats the payload for them. Any other URL gets JSON with a `type` such as `Deployment.failed` and a `resource` block naming the project, environment and service. That's enough for a small receiver to keep only production events. Host the receiver in its own project, so an outage in the app can't take its alerting down with it.
By default, Railway restarts a crashed service on failure, up to 10 times, so a one-off crash often fixes itself before anyone looks. A failed deploy never fixes itself. Treat `Deployment.failed` as urgent, and page someone when the same service crashes repeatedly.
#### Cover the checks Railway stops running after deploy
The healthcheck stops once a deploy is live, so nothing on Railway tells you when the API stops answering overnight. For continuous checks, Railway's docs point to the Uptime Kuma template in its marketplace. Run it in a separate project and point it at a route that checks your real dependencies.
For longer retention or application metrics, send telemetry to a third-party tool. Either path works:
- **Vendor SDK.** For Python, Datadog wraps the start command with `ddtrace-run`, New Relic uses `newrelic-admin run-program`, and you initialise Sentry in code. Map Railway's variables to the vendor's names, for example `DD_SERVICE=${{RAILWAY_SERVICE_NAME}}` and `DD_VERSION=${{RAILWAY_DEPLOYMENT_ID}}`.
- **OpenTelemetry to your own backend.** Set `OTEL_EXPORTER_OTLP_ENDPOINT` to the backend, and prefer `http/protobuf` on port 4318. If you set this variable yourself, Railway adds none of its tracing variables, so your app's spans no longer reach the Traces page. The edge keeps tracing requests.
To forward raw stdout, run Vector or Fluent Bit as its own Railway service.
#### Where FastAPI monitoring on Railway goes wrong
- **Uvicorn is not listening on the PORT variable.** The health check returns service unavailable and the deploy fails. Start Uvicorn with `--host 0.0.0.0 --port $PORT`.
- **Every log line is red.** The handler writes to stderr. Pass `sys.stdout` to `StreamHandler`.
- **JSON logs show up as plain text.** Pretty-printed or multi-line JSON breaks parsing. Write one object per line with `json.dumps` and no indent.
- **The health check fails with status 400.** `TrustedHostMiddleware` is rejecting `healthcheck.railway.app`. Add it to `allowed_hosts`.
- **Edge spans appear but the app's spans don't.** The `OTEL_*` variables arrive on the first deploy after you enable tracing, so redeploy. Also check that the service doesn't set its own `OTEL_EXPORTER_OTLP_ENDPOINT`.
- **The SDK logs export errors.** It's trying to send metrics or logs. Set `OTEL_METRICS_EXPORTER=none` and `OTEL_LOGS_EXPORTER=none`.
- **Staging shows nothing.** Tracing, the dashboard and monitors are all per environment. Turn them on in each environment.
- **Logs go missing under load.** You've hit 500 lines per second per replica. Turn off Uvicorn access logs, raise the log level in production, and sample high-frequency events.
#### Go-live checklist for FastAPI on Railway
- `/health` returns 200 and is set as the healthcheck path
- Uvicorn listens on `0.0.0.0` and `$PORT`
- Logs are single-line JSON on stdout with `level`, `path`, `status_code`, `duration_ms` and `trace_id`
- Tracing is on in production and the App indicator is green
- `OTEL_METRICS_EXPORTER` and `OTEL_LOGS_EXPORTER` are both set to `none`
- The production environment has a dashboard with an error log widget
- Memory and CPU monitors are set up (Pro plan)
- A project webhook posts to your alerts channel and has fired on a real deploy
- An outside uptime check hits a route that checks your dependencies
#### Let Praxen watch the Railway project for you
Praxen detects production issues and fixes them by opening pull requests. It connects to Railway, so when a service breaks it works out why, and the fix arrives ready for your team's normal review.
Running on Railway? See [how Praxen monitors Railway in production](https://getpraxen.app/for/railway/).
#### Common questions
**Does Railway track request latency and error rates for a FastAPI app?**
Not as metrics. The Metrics tab only covers CPU, memory, disk and network. Railway's HTTP logs do record `@httpStatus`, `@responseTime` and `@totalDuration` for every request, so filters like `@httpStatus:500..599` or `@responseTime:>1000` find errors and slow requests. For timing inside each route, enable tracing and run the app under `opentelemetry-instrument`.
**Is Railway's healthcheck enough to know my API is up?**
No. Railway only calls the healthcheck path at the start of a deploy, until it gets a 2xx, and never after the deploy is live. For continuous checks, Railway's docs suggest the Uptime Kuma template, or you can point any outside uptime service at your public domain.
**How long does Railway keep logs, traces and metrics?**
Railway keeps logs for 3 days on Free, 7 days on Trial and Hobby, 30 days on Pro and up to 90 days on Enterprise. Traces are kept for the same period. Metrics graphs hold up to 30 days of data. For longer history, send telemetry to a third-party tool with a vendor SDK or OpenTelemetry.
**Why are all my FastAPI logs marked as errors on Railway?**
By default, Python's logging StreamHandler writes to stderr, and Railway converts every stderr line to level error. Create the handler with `logging.StreamHandler(sys.stdout)` and include a `level` field in single-line JSON. Railway then colours each line by its real level.
**Do I need the Pro plan to get alerts?**
Monitors need the Pro plan. They are the threshold alerts on CPU, RAM, disk usage and network egress. Project webhooks cover failed builds, failed deploys and crashed deployments, and Railway formats them automatically for Slack and Discord URLs.
**Can I send FastAPI traces to Datadog or Grafana and keep Railway's Traces page?**
Not from the same exporter. If you set `OTEL_EXPORTER_OTLP_ENDPOINT` yourself, Railway adds none of its tracing variables, and your app's spans stop reaching the Traces page. The edge still traces requests. Choose one destination per service: Railway's built-in tracing, or a vendor setup such as `ddtrace-run` or an OTLP backend.
**What does the Railway rate limit of 500 logs/sec warning mean?**
Railway allows 500 log lines per second per replica on every plan and drops anything above that. For FastAPI, start Uvicorn with `--no-access-log`, because the edge HTTP logs already record every request. Then raise the production log level and sample high-frequency events.
**How do I find the trace for one user's failed request?**
Railway's edge adds an `x-railway-trace-id` header to every traced response. Have the client or your error page show it, then paste it into the Trace ID field on the Traces page, or run `railway trace get` with the ID.
#### Sources
- [Railway Observability Dashboard](https://docs.railway.com/observability)
- [Railway Logs](https://docs.railway.com/observability/logs)
- [Railway Metrics](https://docs.railway.com/observability/metrics)
- [Railway Healthchecks](https://docs.railway.com/deployments/healthchecks)
- [Railway Tracing](https://docs.railway.com/observability/tracing)
- [Railway Tracing for Python](https://docs.railway.com/observability/tracing/python)
- [Set Up Alerts for Crashes, Restarts, and Failed Deploys](https://docs.railway.com/guides/alerts-crashes-failed-deploys)
- [Connect a Third-Party Observability Tool](https://docs.railway.com/guides/third-party-observability)
- [Railway Config as Code reference](https://docs.railway.com/config-as-code/reference)
- [Product docs: Quickstart, Clouds and Detect](https://docs.getpraxen.app/llms-full.txt)
- [Product overview and build log](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
Next: [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
#### Related
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [Fix Railway “Application failed to respond” (502)](https://getpraxen.app/learn/troubleshooting/railway-application-failed-to-respond/): Why Railway returns “Application failed to respond” with a 502, how to tell a wrong host, port or target port from a crash or overload, and how to fix each.
---
### How to debug Cloudflare Workers errors: logs, traces and error codes
> Debug Cloudflare Workers errors step by step: read 1101 and 1102 codes, enable Workers Logs and source maps, use wrangler tail, DevTools and local traces.
Part 4 of [Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
By Boris Tane, Founder of Praxen · Published September 29, 2026 · 12 min read
Canonical: https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/
Start with the error code the client sees. 1101 means your code threw a JavaScript exception, and 1102 means the Worker ran out of CPU time or memory. To find the failing line, turn on Workers Logs and source maps in your Wrangler config, stream live errors with `npx wrangler tail`, then replay the request under `wrangler dev`, where DevTools, breakpoints and local traces show you what broke.
Most Cloudflare Workers errors reach you in one of three ways: a numbered error page, a spike in the dashboard, or a stack trace that points at minified code. Debugging them follows a short loop. Read the code Cloudflare returns. Find the exception and its stack trace in Workers Logs or `wrangler tail`. Map the trace back to your source with source maps. Then replay the request under `wrangler dev` with DevTools attached.
The loop only works if the evidence exists when the error happens. So turn on logs and source maps first. Then learn the few runtime errors that only happen on Workers, because general JavaScript advice will not fix them.
#### What the Workers error codes mean
When a Worker in production fails before it returns a response, the client gets a Cloudflare error page with a code. The code is your first clue. It tells you whether your code threw, a limit stopped it, or the request never reached your code at all.
**Workers error codes and where to look first**
| Code | What happened | Where to look first |
| --- | --- | --- |
| 1101 | Worker threw a JavaScript exception | Stack trace in logs |
| 1102 | Worker exceeded its CPU time or memory limit | Metrics, Invocation Statuses |
| 1103 | The Worker owner needs to contact Cloudflare Support | Cloudflare Support |
| 1019 | Worker hit the loop limit | Worker-to-Worker call chain |
| 1021 | Worker requested a host it cannot access | Outbound fetch target |
| 1022 | Cloudflare failed to route the request to the Worker | Routes and domains |
| 1024 | Subrequest to a Cloudflare-owned IP address | Outbound fetch target |
| 1027 | Free plan daily request limit exceeded | Plan and route fail mode |
| 1042 | Fetch to another Worker on the same zone without global_fetch_strictly_public | Compatibility flags |
| 10162 | Module has an unsupported Content-Type | Build output |
Two codes cover most incidents. **1101** means your JavaScript threw an exception, so the answer is in a stack trace. **1102** means the Worker exceeded its resource limits, either CPU time or memory, so the answer is in your metrics. Other `11xx` codes usually point to a problem in the Workers runtime itself. Check the Cloudflare status page before you touch your code.
A 1027 has a twist. Free accounts get 100,000 requests a day, reset at midnight UTC. A route set to fail closed returns the 1027 page. A route set to fail open skips the Worker, so requests behave as if no Worker were configured, and you see odd behaviour with no error at all.
#### Turn on Workers Logs and source maps before the next error
Workers Logs collects invocation logs, your `console.log` output, errors and uncaught exceptions. It stores them in your Cloudflare account, where you can query them per Worker. New Workers have it on by default. Older Workers need the setting in the Wrangler file and a redeploy, and it needs Wrangler 3.78.6 or later.
```jsonc
// wrangler.jsonc
{
"observability": {
"enabled": true,
"head_sampling_rate": 1
},
"upload_source_maps": true
}
```
`head_sampling_rate` runs from 0 to 1 and defaults to 1, which logs every request. `upload_source_maps` tells Wrangler to generate and upload source maps on `wrangler deploy` and `wrangler versions deploy`. Cloudflare then maps the stack trace of any uncaught exception back to your original files and lines, so a TypeScript error points at the `.ts` file you wrote. Source maps need Wrangler 3.46.0 or later, and each map can be up to 15 MB gzipped.
I built and led the Workers observability team at Cloudflare, and my advice on this step is blunt: a log you never collected cannot help you, so switch it on before the incident.
##### Log objects so you can filter by field
Workers Logs extracts the fields of any JSON object you log and indexes them.
```js
// Hard to query: the ID is buried in a string
console.log("user_id: " + userId);
// Easy to query: user_id and order_id become fields
console.log({ user_id: userId, order_id: orderId, status: res.status });
```
The first line hides the ID inside a message, so finding every log for one user means a text search. The second gives you `user_id` and `order_id` as fields you can filter on in the middle of an incident.
##### What Workers Logs costs and how long it keeps data
Workers Logs is included on both plans. The Free plan writes 200,000 log events a day and keeps them for 3 days. The Paid plan includes 20 million events a month, charges $0.60 per additional million, and keeps them for 7 days. A single log can be up to 256 KB. Anything larger is truncated and marked with `$cloudflare.truncated`. For longer retention, send logs to another tool with OpenTelemetry export, Workers Logpush or Tail Workers.
#### From error page to failing line, step by step
1. **Read the code.** Match the code on the error page, or the status your client got, against the table above. That tells you whether to hunt for an exception, a limit or a routing problem.
2. **Check the invocation statuses.** In the dashboard, go to **Workers & Pages**, select the Worker, and open **Metrics** > **Errors** > **Invocation Statuses**. `Exceeded CPU Time Limits` and `Exceeded Memory` appear there as separate lines, so a 1102 turns into one of two specific problems.
3. **Stream live errors.** From the project directory, run:
```bash
npx wrangler tail
```
Each event arrives as a structured JSON object:
```json
{
"outcome": "ok",
"scriptName": null,
"exceptions": [],
"logs": [],
"eventTimestamp": 1590680082349,
"event": {
"request": { "url": "https://www.bytesized.xyz/", "method": "GET", "headers": {}, "cf": {} }
}
}
```
Pipe it through `jq` to keep only the failed invocations:
```bash
npx wrangler tail | jq 'select(.outcome != "ok") | {outcome, exceptions, url: .event.request.url}'
```
The same stream is in the dashboard: open the Worker's **Logs** tab and select **Live**.
4. **Search history in Workers Logs.** If the error happened an hour ago, tail will not show it. Open the Worker in **Workers & Pages** and select **Observability**. Each invocation writes one invocation log, marked `$cloudflare.$metadata.type = "cf-worker-event"`, with the request, the response and metadata. Your own logs and exceptions sit next to it.
5. **Read the remapped stack trace.** With source maps uploaded, the stack trace in real-time logs and Tail Workers points at your source file and line. Cloudflare leaves any line it cannot remap as it was. A partly minified trace can mean the source map is stale or missing.
6. **Reproduce it locally.** Once you know the route and the input, replay the request against `wrangler dev` and step through it, as the next section shows.
#### Reproducing the error locally with DevTools, breakpoints and traces
`wrangler dev` runs your Worker on your machine in `workerd`, the same open-source runtime that runs in production, and serves it on `localhost:8787`. Exceptions and `console.log` output print in the terminal. For errors in a Cron Trigger, `wrangler dev --test-scheduled` exposes a `/cdn-cgi/local/scheduled` route that fires the scheduled handler on demand.
##### Open DevTools
Press `D` in the terminal running `wrangler dev` and Cloudflare's build of Chrome DevTools opens in a browser tab. With Vite and the Cloudflare Vite plugin, open the debug URL printed in the console, for example `http://localhost:5173/__debug`. DevTools gives you the console, breakpoints, CPU profiling and memory snapshots. The dashboard editor and the Workers Playground include DevTools too.
##### Set breakpoints in VS Code
Create `.vscode/launch.json`:
```json
{
"configurations": [
{
"name": "Wrangler",
"type": "node",
"request": "attach",
"port": 9229,
"cwd": "/",
"resolveSourceMapLocations": null,
"attachExistingChildren": false,
"autoAttachChildProcesses": false
}
]
}
```
Run `npx wrangler dev` in the VS Code terminal, pick the **Wrangler** configuration in the Run & Debug panel and press play. Set a breakpoint, request `http://127.0.0.1:8787`, and execution stops on that line. WebStorm works the same way with an Attach to Node.js/Chrome configuration on port 9229. If that port is taken, `wrangler dev --inspector-port` sets another one.
##### Query local traces to see which call failed
Current versions of `wrangler dev` and `vite dev` capture OpenTelemetry traces for every local invocation, with no SDK and no code change. The runtime records spans for outbound `fetch` calls, for binding calls to KV, R2, D1, Durable Objects and Queues, and for the handler itself. Press `e` in Wrangler, or visit `/cdn-cgi/explorer` on the local server, to open the Local Explorer. Click a failing request to see its spans, errors and correlated logs.
This is the fastest way to see which operation broke. In Cloudflare's own example, an order endpoint returned 500. The trace showed that the KV read succeeded, the D1 insert failed with `no such column: delivery_window`, and the Queue send never ran. The cause was a migration that had not been applied locally.
Coding agents get the same data. When Wrangler detects an agent session, it prints the address of the Local Explorer API, which serves an OpenAPI schema and a read-only endpoint for querying traces and logs with SQL. A plain prompt is enough:
```text
> POST /api/orders is returning 500. Find the cause, fix it, and verify the fix locally.
```
*The agent reproduces the error, reads the local trace, applies the fix and checks the new trace.*
#### Runtime errors that only happen on Workers
##### "The script will never generate a response"
This one arrives as a 1101. The runtime saw that all the code for the request had finished and nothing was left in the event loop, yet no Response came back. Other runtimes would hang forever. Workers throws so you can find the bug.
The usual cause is a Promise that never resolves or rejects:
```js
export default {
fetch(req) {
const response = new Response("ok");
const { promise, resolve } = Promise.withResolvers();
// setTimeout(resolve, 0) is the missing line
return promise.then(() => response);
},
};
```
Find the Promise your Response depends on, in your code or a dependency, and make sure every path resolves or rejects it. The `no-floating-promises` ESLint rule catches Promises that are created and never handled.
The second cause is a WebSocket whose server side is never closed. On compatibility dates on or after 2026-04-07, the `web_socket_auto_reply_to_close` flag completes the close handshake for you, so this mostly bites Workers on older dates. On those, call `server.close()` in the close listener.
##### "Illegal invocation: function called with incorrect this reference"
A runtime method lost its `this`. The classic case is destructuring `ctx`:
```js
export default {
async fetch(request, env, ctx) {
// Throws: waitUntil has lost its this
const { waitUntil } = ctx;
waitUntil(somePromise);
// Works: call it on ctx, or bind it first
ctx.waitUntil(somePromise);
const bound = ctx.waitUntil.bind(ctx);
bound(somePromise);
return fetch(request);
},
};
```
##### "Cannot perform I/O on behalf of a different request"
Each invocation has its own execution context. I/O objects such as streams, request bodies and responses belong to the request that created them. If you cache a Response in global scope and return it from a later request, the runtime throws this error. Cache the data and build a fresh Response each time:
```js
let cachedData = null;
export default {
async fetch(request, env, ctx) {
if (cachedData) {
return new Response(cachedData);
}
const response = new Response("Hello, world!");
cachedData = await response.text();
return new Response(cachedData, response);
},
};
```
For state shared across requests, use Durable Objects. For cached data, use Workers KV.
##### Error 1019: the loop limit
A chain of Workers cannot call itself or another Worker more than 16 times. The `CF-EW-Via` header counts the invocations left and drops by one on each hop. When it reaches zero you get 1019. Look for a route or service binding that sends a request back into the same Worker.
#### When a limit stops the Worker: error 1102
Error 1102 shows the message `Worker exceeded resource limits`. The invocation outcome in analytics and Logpush tells you which limit it was: `exceededCpu` or `exceededMemory`.
##### Running out of CPU time
CPU time only counts time spent running your code. Waiting on `fetch`, KV reads or database queries does not count. Free Workers get 10 ms per request. Paid Workers default to 30 seconds and can go up to 5 minutes:
```jsonc
{
"limits": {
"cpu_ms": 300000
}
}
```
Before you raise the limit, profile CPU in DevTools locally to find the hot path. Then consider splitting heavy work across smaller requests or moving it into Durable Objects.
##### Running out of memory
Each isolate gets 128 MB, covering the JavaScript heap and WebAssembly allocations. One isolate serves many requests at once, so they all share that budget. When the isolate goes over, in-flight requests finish and the runtime starts a new isolate for the next ones. Buffering a large body can also throw `Memory limit would be exceeded before EOF`.
Work through the fixes in this order:
1. Stream request and response bodies with `TransformStream`.
2. Keep large data in KV, R2 or D1, out of Worker memory.
3. If you use Zod, upgrade to 4.5.0 or later, since earlier versions use far more memory per schema.
4. Take heap snapshots in DevTools locally to find leaks.
#### Mistakes that hide Workers errors
- **Logging a stack from inside the Worker.** Source maps are not available at runtime, so `console.log(err.stack)` prints the minified trace. Let the exception reach the runtime, then read the remapped trace in real-time logs or Tail Workers.
- **Sending logs without `ctx.waitUntil()`.** A `fetch` to an external log service after the response has gone may never complete. Wrap it in `ctx.waitUntil()`, which keeps work alive for up to 30 seconds after the response.
- **Sampling away the evidence.** A `head_sampling_rate` of 0.01 logs one request in a hundred, and a rare error may never show up. Keep the rate at 1 while you debug, and set a different rate per environment under `env.staging.observability` when you need to. Also note that an account over 5 billion logs in a day drops to a 1% sample for the rest of that day.
- **Trusting `wrangler tail` under heavy traffic.** Real-time logs can enter sampling mode and drop messages, with a warning in the stream. Filter what you watch, and remember that at most 10 clients, dashboard sessions and `wrangler tail` calls combined, can watch one Worker at a time. Real-time logs store nothing, so use Workers Logs for history.
- **Waiting on WebSocket logs.** In `wrangler tail`, `console.log` calls inside WebSocket handlers stay hidden until the client closes the connection, then flush all at once.
- **Blaming your code for a runtime problem.** An unfamiliar `11xx` code often means the Workers runtime itself is struggling. Check the Cloudflare status page first.
#### Letting Praxen watch your Workers for errors
Praxen detects production issues and fixes them by opening pull requests. It connects to Cloudflare, so when a Worker starts throwing errors it works out why, and the fix arrives ready for your team's normal review.
Running on Cloudflare? See [how Praxen monitors Cloudflare in production](https://getpraxen.app/for/cloudflare/).
#### Common questions
**What does Cloudflare Workers error 1101 mean?**
Error 1101 means your Worker threw a JavaScript exception before it returned a response. Find the stack trace in Workers Logs or with `npx wrangler tail`. A 1101 that says `The script will never generate a response` usually means a Promise that never resolves, or a WebSocket that is never closed on older compatibility dates.
**What causes error 1102 and how do I fix it?**
Error 1102 means the Worker exceeded its CPU time or its 128 MB memory limit. Check **Metrics** > **Errors** > **Invocation Statuses** to see which one. For CPU on the Paid plan, raise `cpu_ms` from the 30 second default up to 300000 (5 minutes). For memory, stream bodies and keep large data in KV, R2 or D1.
**Why does my Workers stack trace point at minified code?**
Source maps are not uploaded. Add `"upload_source_maps": true` to your Wrangler config and redeploy with Wrangler 3.46.0 or later, and keep each map under 15 MB gzipped. Remapped traces appear in real-time logs and Tail Workers, but a `console.log(err.stack)` inside the Worker stays minified.
**How long are Workers Logs kept?**
Workers Logs keeps data for 3 days on the Free plan and 7 days on the Paid plan. The Free plan writes 200,000 log events a day. The Paid plan includes 20 million a month and charges $0.60 per additional million. For longer retention, export with OpenTelemetry, Workers Logpush or Tail Workers.
**Can I set breakpoints in a Cloudflare Worker?**
Yes, in local development. Run `wrangler dev` and press `D` to open DevTools, or attach VS Code or WebStorm to port 9229 with a launch configuration. For a deployed Worker, use Workers Logs, `wrangler tail` and source-mapped stack traces, then replay the failing request locally.
**Why is wrangler tail missing some of my logs?**
Under heavy traffic, real-time logs can enter sampling mode and drop messages, with a warning in the stream. Filter the stream to cut volume, and note that only 10 clients can watch one Worker at a time. Logs inside WebSocket handlers also stay hidden until the client closes the connection.
**How do I see which binding call failed in a local request?**
Current versions of `wrangler dev` and `vite dev` record OpenTelemetry traces for every local invocation automatically. Press `e` in Wrangler or open `/cdn-cgi/explorer` to view spans for fetch calls and for KV, R2, D1, Durable Objects and Queues bindings. Coding agents can query the same traces through the Local Explorer API.
#### Sources
- [Errors and exceptions · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/errors/)
- [Source maps and stack traces · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/source-maps/)
- [Workers Logs · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/logs/workers-logs)
- [Real-time logs · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/logs/real-time-logs/)
- [Limits · Cloudflare Workers docs](https://developers.cloudflare.com/workers/platform/limits/)
- [DevTools · Cloudflare Workers docs](https://developers.cloudflare.com/workers/observability/dev-tools/)
- [Wrangler commands for Workers](https://developers.cloudflare.com/workers/wrangler/commands/workers/)
- [Debugging logs example · Cloudflare Workers docs](https://developers.cloudflare.com/workers/examples/debugging-logs/)
- [Better debugging for Cloudflare Workers, now with breakpoints](https://blog.cloudflare.com/debugging-cloudflare-workers/)
- [Your agent can now debug Workers with local tracing](https://blog.cloudflare.com/local-tracing/)
- [Praxen documentation](https://docs.getpraxen.app/llms-full.txt)
- [Praxen full content](https://getpraxen.app/llms-full.txt)
#### About the author
Boris Tane is the founder of Praxen. He previously founded Baselime, observability for the future of the cloud, which Cloudflare acquired. At Cloudflare he built and led the Workers observability team.
#### More in this series
[Platform playbooks](https://getpraxen.app/series/platform-playbooks/)
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/)
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/)
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/)
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/)
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/)
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/)
Previous: [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/)
Next: [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/)
#### Related
- [What is an AI SRE? How it works and how to evaluate one](https://getpraxen.app/learn/incident-response/what-is-an-ai-sre/): An AI SRE is an AI agent that triages alerts, investigates incidents and proposes fixes. Learn how it works, the autonomy levels and how to evaluate one.
- [Fix Cloudflare “Error 1102: Worker exceeded resource limits”](https://getpraxen.app/learn/troubleshooting/cloudflare-error-1102-worker-exceeded-resource-limits/): Why a Cloudflare Worker returns Error 1102, how to tell a CPU time overrun from a 128 MB memory overrun, and the code and Wrangler changes that fix each.
- [How to build a memory system for proactive agents](https://getpraxen.app/blog/how-to-build-a-memory-system-for-proactive-agents/): Proactive agents run without a prompt, so each run has to start from what earlier runs learned. How we built memory for Praxen's agents on git: one repository per team, a nightly clean-up, and lessons from merged pull requests.
## Series (reading order)
### Choosing AI for production
> What an AI for production actually does, how to evaluate one, and the best tools for each job, from root cause to on-call.
1 article
Canonical: https://getpraxen.app/series/choosing-ai-for-production/
1. [Is it safe to give an AI SRE read access to production?](https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production/): Read access for an AI SRE is a sound first step if you scope it. The risks are secrets, customer data and exfiltration. Here are the controls and a checklist. Boris Tane, September 29, 2026, 12 min read. Markdown: https://getpraxen.app/learn/ai-in-production/is-it-safe-to-give-an-ai-sre-read-access-to-production.md
---
### Shipping safely: deploys, config and change risk
> How small teams ship to main every day without breaking production: watching deploys, config changes and rollouts.
1 article
Canonical: https://getpraxen.app/series/shipping-safely/
1. [How to fix a broken build on main](https://getpraxen.app/learn/continuous-integration/how-to-fix-a-broken-build-on-main/): How to fix a broken build on main: read the failing job, decide between rerun, revert and fix forward, find the breaking commit and stop the next one. Boris Tane, October 17, 2026, 6 min read. Markdown: https://getpraxen.app/learn/continuous-integration/how-to-fix-a-broken-build-on-main.md
---
### Platform playbooks
> Running a specific stack in production: Next.js on Vercel, APIs on Cloudflare, Rails on Fly.io, Supabase and more.
8 articles
Canonical: https://getpraxen.app/series/platform-playbooks/
1. [How to monitor a Django app on Render](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render/): Monitor a Django app on Render: JSON logs, a real health check, failure alerts, log and metrics streams, OpenTelemetry under Gunicorn and Celery checks. Boris Tane, September 30, 2026, 14 min read. Markdown: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-django-app-on-render.md
2. [How to monitor a FastAPI app on Railway: logs, traces and alerts](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway/): Set up health checks, JSON logs, OpenTelemetry traces, monitors and crash alerts for a FastAPI service on Railway, with the commands and config to copy. Boris Tane, September 29, 2026, 13 min read. Markdown: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-fastapi-app-on-railway.md
3. [How to monitor a Supabase app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production/): Monitor a Supabase app in production: scrape the Metrics API into Prometheus, drain logs, run SQL health checks, trace requests and alert on what matters. Boris Tane, September 29, 2026, 14 min read. Markdown: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-supabase-app-in-production.md
4. [How to debug Cloudflare Workers errors: logs, traces and error codes](https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors/): Debug Cloudflare Workers errors step by step: read 1101 and 1102 codes, enable Workers Logs and source maps, use wrangler tail, DevTools and local traces. Boris Tane, September 29, 2026, 12 min read. Markdown: https://getpraxen.app/learn/troubleshooting/how-to-debug-cloudflare-workers-errors.md
5. [How to debug Vercel function timeouts](https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts/): Debug Vercel FUNCTION_INVOCATION_TIMEOUT 504s: find the slow call with vercel logs and httpstat, fix hangs, set maxDuration and verify on a preview. Boris Tane, October 1, 2026, 12 min read. Markdown: https://getpraxen.app/learn/troubleshooting/how-to-debug-vercel-function-timeouts.md
6. [Cloudflare Workers error 1101: causes and how to fix it](https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101/): Error 1101 means your Cloudflare Worker threw an uncaught JavaScript exception. Find the exception in logs, match it to its cause, fix it and roll back fast. Boris Tane, September 30, 2026, 13 min read. Markdown: https://getpraxen.app/learn/troubleshooting/cloudflare-workers-error-1101.md
7. [Cloudflare Hyperdrive connection errors: causes and fixes](https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors/): Fix Cloudflare Hyperdrive connection errors: config codes 2008 to 2016, pool exhaustion, connection_refused and stale clients reused across Worker requests. Boris Tane, September 30, 2026, 13 min read. Markdown: https://getpraxen.app/learn/troubleshooting/cloudflare-hyperdrive-connection-errors.md
8. [How to monitor a Convex app in production](https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production/): Monitor a Convex app in production: use the Health page and CLI, stream logs to Axiom or Datadog, report exceptions to Sentry, and alert on failures. Boris Tane, September 30, 2026, 13 min read. Markdown: https://getpraxen.app/learn/monitoring-coverage/how-to-monitor-a-convex-app-in-production.md
---
### AI x Production: coding agents, MCP and autofix
> Giving coding agents production context, letting AI open fixes safely, and building software that operates itself.
2 articles
Canonical: https://getpraxen.app/series/ai-x-production/
1. [How to use Claude Code to debug production issues](https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues/): Debug production issues with Claude Code: get logs and traces into the session, test hypotheses against evidence, stay read-only and ship a verified fix. Boris Tane, September 29, 2026, 12 min read. Markdown: https://getpraxen.app/learn/ai-in-production/how-to-use-claude-code-to-debug-production-issues.md
2. [How to keep AI coding agents from breaking production](https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production/): Stop AI coding agents breaking production: scoped credentials, branch protection they can't bypass, required checks that block, and production-aware review. Boris Tane, September 29, 2026, 14 min read. Markdown: https://getpraxen.app/learn/ai-in-production/how-to-keep-ai-coding-agents-from-breaking-production.md
---
### Vibe-coded apps in production
> Keeping a Lovable, Bolt, v0 or Replit app running once real users arrive: monitoring, debugging and fixing it without an ops team.
1 article
Canonical: https://getpraxen.app/series/vibe-coded-apps-in-production/
1. [How to monitor a vibe-coded app in production](https://getpraxen.app/learn/monitoring-coverage/vibe-coding-production-monitoring/): Monitor a vibe-coded app in production: surface swallowed errors, add a health route, tag deploys, watch cron jobs and alert on what users feel first. Boris Tane, September 30, 2026, 12 min read. Markdown: https://getpraxen.app/learn/monitoring-coverage/vibe-coding-production-monitoring.md
---
### Using coding agents: Claude Code, Cursor, Codex and more
> Practical guides to each coding agent by name: installing it, adding MCP servers, writing its instructions file and the workflows that hold up.
6 articles
Canonical: https://getpraxen.app/series/using-coding-agents/
1. [How to use Claude Code: a practical guide](https://getpraxen.app/learn/coding-agents/how-to-use-claude-code/): How to use Claude Code: install it, write a CLAUDE.md, pick a permission mode, steer it mid-task, script it with -p and connect it to your tools with MCP. Boris Tane, October 17, 2026, 6 min read. Markdown: https://getpraxen.app/learn/coding-agents/how-to-use-claude-code.md
2. [How to add MCP servers to Cursor](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor/): How to add MCP servers to Cursor: project and global mcp.json files, remote and local servers, OAuth sign-in, secrets from env vars and tool approvals. Boris Tane, October 17, 2026, 3 min read. Markdown: https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-cursor.md
3. [How to use Codex CLI: install, AGENTS.md, MCP and exec](https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli/): How to use Codex CLI: install it, sign in, write an AGENTS.md, set permissions, resume sessions, script it with codex exec and add MCP servers. Boris Tane, October 17, 2026, 3 min read. Markdown: https://getpraxen.app/learn/coding-agents/how-to-use-codex-cli.md
4. [How to use OpenCode: install, plan and build, MCP](https://getpraxen.app/learn/coding-agents/how-to-use-opencode/): How to use OpenCode: install it, connect a provider, write an AGENTS.md with /init, switch between Plan and Build, undo changes and add MCP servers. Boris Tane, October 17, 2026, 3 min read. Markdown: https://getpraxen.app/learn/coding-agents/how-to-use-opencode.md
5. [How to add MCP servers to GitHub Copilot in VS Code](https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot/): How to add MCP servers to GitHub Copilot in VS Code: workspace and user mcp.json, HTTP and stdio servers, MCP: Add Server and confirming tool calls. Boris Tane, October 17, 2026, 3 min read. Markdown: https://getpraxen.app/learn/coding-agents/how-to-add-mcp-servers-to-github-copilot.md
6. [How to use the Pi coding agent](https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent/): How to use the Pi coding agent: install it, sign in with /login, pick a model, branch sessions with /tree and /fork, compact context and script it with -p. Boris Tane, October 17, 2026, 3 min read. Markdown: https://getpraxen.app/learn/coding-agents/how-to-use-the-pi-coding-agent.md
## Open roles (full content)
### Founding Engineer
URL: https://getpraxen.app/careers/founding-engineer/
Location: San Francisco, CA
Type: Full-time, in-person
Compensation: $200k to $400k + meaningful equity
Apply: DM https://x.com/boristane on X with a link to your work.
## Opportunity
Coding agents now write most of the code. The bottleneck has moved from typing to everything around it: observing, monitoring, debugging, root-causing, on-call, deciding what's safe to ship. The lifecycle is collapsing into two halves. One half is intent. The other is execution. The execution half should not be this hard.
We're building Praxen: the platform that lets developers ship self-operating software. Software that watches itself in production, investigates issues, finds root causes, and ships fixes before anyone gets paged. Software that operates itself.
This is what comes after coding agents. The boring parts of software engineering, automated end-to-end. The development lifecycle for a world where humans set intent and software does the rest.
## What you'd work on
- Building agents that observe, reason about, and act on production systems end to end.
- Turning raw telemetry, code, and infrastructure into context a model can actually reason from.
- Designing operator interfaces that make autonomous software feel obvious to humans and legible to other agents.
- Working across the stack from product, design, and platform to get features into production quickly.
- Closing the loop between what we observe in production and the code an agent writes next.
## What we're looking for
You should have strong product engineering instincts and be comfortable owning features end to end. You write TypeScript fluently and ship fast: prototype, iterate, throw it out when it doesn't work.
You've built agentic systems with modern frameworks and have opinions about what works. You've operated production systems and understand observability deeply: traces, wide events, high-cardinality data.
Experience with the Cloudflare developer platform (Workers, Durable Objects, R2, D1, Queues), ClickHouse, or other infrastructure-adjacent products is especially helpful. So is taste, for product, for prose, for what makes a tool feel right under a developer's hands.
## Who is Praxen Inc.?
A small team, and the company behind Praxen. We've operated at Cloudflare scale, where we built the observability platform used by millions of developers every day. We've spent years writing, operating, and breaking production systems for a living.
Praxen Inc. is what comes next. Based in San Francisco. We work in person.
---
### Founding Growth
URL: https://getpraxen.app/careers/founding-growth/
Location: San Francisco, CA
Type: Full-time, in-person
Compensation: $200k to $400k + meaningful equity
Apply: DM https://x.com/boristane on X with a link to your work.
## Opportunity
Coding agents now write most of the code. The bottleneck has moved from typing to everything around it: observing, monitoring, debugging, root-causing, on-call, deciding what's safe to ship. The lifecycle is collapsing into two halves. One half is intent. The other is execution. The execution half should not be this hard.
We're building Praxen: the platform that lets developers ship self-operating software. Software that watches itself in production, investigates issues, finds root causes, and ships fixes before anyone gets paged. Software that operates itself.
Someone has to make sure every developer on earth hears about it. That's this role: growth built as a software system, not a spreadsheet, owning the loop from a developer hearing about Praxen to a workspace with a connected cloud.
## What you'd work on
- Owning the self-serve funnel end to end: waitlist, onboarding, activation, and the instrumentation that tells us what's working.
- Building the content engine: docs, changelogs, technical posts, and the tooling that makes publishing fast.
- Shipping growth experiments weekly and killing the ones that don't move anything.
- Building distribution surfaces: integrations, templates, CLI install flows, and the places developers discover tools.
- Turning product moments into shareable artifacts, the way an incident investigation becomes a story worth posting.
## What we're looking for
You should be a strong full-stack engineer who has owned growth, activation, or developer experience before. This is an engineering role with growth instincts, not a marketing role with a code editor. You write TypeScript fluently and ship fast: prototype, iterate, throw it out when it doesn't work.
You think in funnels and cohorts, and you instrument before you argue. You write well; developer marketing is mostly writing that respects the reader.
Taste is especially helpful here: for what developers actually share, for prose, for what makes a tool feel right under a developer's hands. And an allergy to what they scroll past.
## Who is Praxen Inc.?
A small team, and the company behind Praxen. We've operated at Cloudflare scale, where we built the observability platform used by millions of developers every day. We've spent years writing, operating, and breaking production systems for a living.
Praxen Inc. is what comes next. Based in San Francisco. We work in person.
---
### Founding Researcher
URL: https://getpraxen.app/careers/founding-researcher/
Location: San Francisco, CA
Type: Full-time, in-person
Compensation: $200k to $400k + meaningful equity
Apply: DM https://x.com/boristane on X with a link to your work.
## Opportunity
Coding agents now write most of the code. The bottleneck has moved from typing to everything around it: observing, monitoring, debugging, root-causing, on-call, deciding what's safe to ship. The lifecycle is collapsing into two halves. One half is intent. The other is execution. The execution half should not be this hard.
We're building Praxen: the platform that lets developers ship self-operating software. Software that watches itself in production, investigates issues, finds root causes, and ships fixes before anyone gets paged. Software that operates itself.
None of that works unless the agents are trustworthy. This role's goal is to improve the models and the harnesses behind self-operating software. The hard problem is not getting a model to propose a fix; it is knowing, empirically, when the agent is right, when it is guessing, and how to move that boundary run after run. This is research in production: your experiments ship to real workspaces, and the feedback loop is measured in days, not conference cycles.
## What you'd work on
- Improving the models themselves: post-training on real operational data so they investigate production systems better with every run.
- Improving the harness: the agent loop, the tools, and the context, so the same model gets further on the same incident.
- Building evaluation harnesses that score agent investigations against ground truth from real production incidents.
- Running context engineering experiments: what to inject, when, and how much, across briefings, memory, and retrieval.
- Studying failure modes of tool-using agents on long-horizon operational tasks and turning them into fixes.
- Publishing internally, and externally when it makes us sharper.
## What we're looking for
You should have shipped LLM systems and measured them seriously, beyond demos and vibes. You're fluent in evaluation methodology: building datasets, defining metrics, avoiding contamination.
You write production-quality code in TypeScript or Python and don't need a research engineer to ship. You have opinions about agent reliability, grounding, and context windows, backed by experiments.
You're comfortable when the answer is "we don't know yet" and rigorous about finding out.
## Who is Praxen Inc.?
A small team, and the company behind Praxen. We've operated at Cloudflare scale, where we built the observability platform used by millions of developers every day. We've spent years writing, operating, and breaking production systems for a living.
Praxen Inc. is what comes next. Based in San Francisco. We work in person.
## Values
- We ship skateboards. The first version rolls, it doesn't fly. We'd rather put something real under people this week than promise a car next quarter.
- Make every single detail perfect, and limit the number of details. Fewer things, obsessed over. If a detail isn't worth perfecting, it doesn't ship.
## Contact
- X: https://x.com/praxen
- Email: boris@getpraxen.app
## Citation
When referencing Praxen, cite as: "Praxen (getpraxen.app), AI agents that make your software self-operating, by Praxen Inc."