A site reliability engineer (SRE) is a software engineer who owns the reliability of production systems and treats that reliability as a software problem rather than a manual one. Instead of logging into servers to restart things and fielding pages all night, an SRE writes code to make the system heal itself, and measures reliability with hard numbers, not gut feel.
The role exists because two teams usually want opposite things. Developers want to ship features fast, and operations wants the system to stop breaking, so the two end up in a quiet tug of war where every release is a fight. SRE resolves that tension with a shared measurement of reliability that both sides agree on in advance, then lets the data decide who gets their way this quarter.
This article explains what an SRE actually does, the three concepts that make the job different from traditional operations, and how the role compares to DevOps and platform engineering so you can tell a real SRE position from an ops job with a nicer title.
What an SRE does day to day
Picture a checkout service that starts returning 500s during a traffic spike. In a traditional ops setup, someone gets paged, SSHes into a box, restarts the service, and closes the ticket. The same thing happens next week, and the week after, because nobody had time to fix the underlying cause.
An SRE handles the same incident differently. They still mitigate it quickly, but afterward they treat the recurrence as the real bug. They'll write an automated remediation, add an alert that fires on the leading indicator instead of the symptom, and run a postmortem to find what let the failure reach users in the first place. The goal is that this class of incident never pages a human again.
That's the split that defines the work. An SRE spends part of their time on operations (responding to incidents, running on-call, doing capacity planning) and the rest building software that removes the need for the first part. The build side includes automation, internal tooling, deployment pipelines, and the observability that tells you whether the system is healthy before your users do. The role started at Google in 2003 under Ben Treynor Sloss, who described it as what happens when you ask a software engineer to design an operations function.
The three ideas that make it a distinct discipline: SLIs, SLOs, and error budgets
If a job posting says "SRE" but never mentions these, be suspicious. They are the mechanics that separate the discipline from operations with a rebrand.
A service level indicator (SLI) is a metric that reflects what your users actually experience. Not CPU utilization or disk I/O, which describe the machine, but things like "did the request succeed" and "did it respond within 300ms." A service level objective (SLO) is the target you set for that SLI over a window, for example 99.9% of checkout requests succeed over 30 days.
The error budget is where it gets interesting. It's the amount of unreliability you're allowed to spend, or 100% minus your SLO target. If your target is 99.9% over a 30-day month, your budget is the remaining 0.1%:
123456Month length: 30 days = 43,200 minutesSLO target: 99.9%Allowed downtime: 0.1% × 43,200 = ~43 minutes/month99.95% → ~21.6 minutes/month99.99% → ~4.3 minutes/month
This number turns a philosophical argument into an accounting one. As long as the service is inside its budget, developers can ship as fast as they want, because there's room to absorb the risk. Once the budget is spent, the policy kicks in and risky launches pause until reliability recovers. Nobody has to argue about whether the system is "reliable enough." The budget already answered it.
The insight underneath is that 100% is the wrong target for almost everything. Going from 99.9% to 100% costs enormous effort, and your users can't tell the difference because their own laptop, Wi-Fi, and ISP are already far less reliable than that. The error budget makes that tradeoff explicit and spends the slack on shipping instead of chasing nines nobody notices.
Toil and the 50% rule
Toil is the SRE term for manual, repetitive work that keeps the system running but never makes it better and grows linearly as the service grows. Restarting the same pod after every memory spike, copying artifacts between environments, triaging the same alert for the third time this week. If a machine could do it and a human is still doing it, it's toil.
Google puts a hard cap here: an SRE team should spend no more than 50% of its time on toil, leaving the other half for engineering that reduces future toil. It's an upper bound, not a target, and in practice Google's teams average closer to a third. The cap has a clever enforcement mechanism. When a team's operational load creeps past 50%, the overflow gets handed back to the product development team, which gives developers a direct incentive to build systems that don't page anyone.
This is the line that most distinguishes SRE from ops. An operations team's workload scales with the size of the system, so more services mean more people. An SRE team's workload is supposed to stay flat because the engineering half keeps automating the operational half away. If your "SRE" role is 90% firefighting with no protected time to fix root causes, it's an ops job wearing an SRE badge.
On-call, incident response, and blameless postmortems
SREs own production, which means they carry the pager. On-call is structured rather than heroic: a typical rotation has an engineer on primary and another on secondary as backup, and a healthy shift sees roughly two incidents. Far fewer and the rotation is wasted attention. Far more and the responder can't investigate anything properly or learn from it.
When an incident does happen, the SRE coordinates the response and mitigates user impact first. Increasingly, AI SRE agents handle some of that first-response investigation. They correlate signals and surface a likely cause before a human even opens the dashboard. Then comes the postmortem, the root cause analysis that gets past the symptom to why the system let the failure through. It's blameless by design, for a hard-nosed reason. If people fear being punished for an outage, they hide the details that would prevent the next one. A blameless culture assumes the engineer acted reasonably given what they knew, and points the investigation at the system that let a single mistake reach users.
To catch problems before they page anyone, SREs lean on the four golden signals of monitoring: latency, traffic, errors, and saturation. Watching those four across a service tells you almost everything about its health, and they map cleanly onto the logs, metrics, and traces an SRE instruments the system with.
SRE vs DevOps vs platform engineering
These three overlap enough to cause constant confusion, so here's the clean version.
DevOps is a culture. It's a set of principles about breaking down the wall between development and operations, shared ownership, and fast feedback. It tells you what good collaboration looks like but not exactly how to implement it. SRE is one concrete implementation of those principles, with prescribed practices: SLOs, error budgets, the toil cap, blameless postmortems. Treynor Sloss called SRE a specific implementation of DevOps with a few idiosyncratic extensions, and that framing holds up. DevOps is the philosophy, SRE is a working blueprint.
Platform engineering is the newer sibling and pulls in a different direction. A platform engineering team builds the internal developer platform, the self-service tooling, golden paths, and infrastructure abstractions that let product teams ship without filing tickets. The focus is developer experience. SRE's focus is production reliability. The two collaborate closely, and at smaller companies one team often does both, but the goals are distinct: platform engineering optimizes how fast developers can move, SRE makes sure what they ship stays up.
What it takes to become one
SRE sits at the intersection of software engineering and systems, so the skill set is broad. You'll want real coding ability, usually Python or Go, because the job is a development role, not scripting on the side. Solid Linux and networking fundamentals matter because cloud-native infrastructure runs on them. Beyond that, expect to work with containers and Kubernetes, infrastructure as code with tools like Terraform, at least one major cloud provider, and observability tooling built on open standards like Prometheus and OpenTelemetry.
The compensation reflects the scarcity of people who have all of that at once. Total pay for SREs in the US commonly lands in the roughly $133,000 to $210,000 range, with senior and staff roles climbing higher in major metros. Most people arrive at SRE from either software engineering or systems administration, then pick up the missing half on the job.
Final thoughts
A site reliability engineer is a software engineer who makes reliability measurable and then engineers it, rather than a sysadmin who reacts to whatever breaks. The tell is always the same: real SRE runs on SLOs, error budgets, and a protected engineering budget, while ops-with-a-badge just runs on pages. If you're building the practice, start by defining SLIs from what your users actually experience, then set an SLO you can defend, and the error budget follows from there.
None of that works without telemetry you can trust. SLIs are only as good as the signals underneath them, and stitching an SLO together from three disconnected tools is how error budgets quietly go wrong.
Dash0 is an OpenTelemetry-native observability platform that stores logs, metrics, and distributed traces together on open standards like OTLP and PromQL, so the SLIs behind your service level objectives come from real user-facing signals rather than proxy metrics. On top of that same data, Dash0 tracks error budgets and burn rate (currently in private beta), so incident response, capacity planning, and reliability all draw from one source.
Start a free trial to instrument your services and see your SLOs, error budgets, logs, and traces in one place. No credit card required.