Lessons from a decade of DevOps

What system administration teaches you that no AI framework will.

I came into DevOps through the unglamorous side of web work: servers that needed patching, deployments that worked everywhere except production, certificates expiring at inconvenient times, and scripts nobody wanted to touch because nobody remembered why they existed.

A decade later, the tooling has changed. Containers became normal. CI/CD became expected. Infrastructure moved into code. Now AI agents are starting to operate pieces of the stack.

The fundamentals have not changed much.

The systems still have state. The network still fails. Credentials still expire. A successful command can still leave you with a broken service. And the fastest way to make a small incident worse is to change three things before you understand the first one.

Here are the lessons that stuck.

The system is always more complicated than the diagram

Architecture diagrams are useful. They are also aspirational.

The real system includes:

  • the old cron job that still runs on a forgotten host
  • the load balancer rule nobody documented
  • the DNS record pointing at an IP address from three years ago
  • the manual configuration change made during an outage
  • the service account with an unclear owner
  • the deployment script that only works because of a file outside the repository

That is not a criticism. It is normal. Systems accumulate history.

The practical lesson is to verify before acting. Do not assume the service is running where the docs say it is. Check the active processes, listeners, logs, deployment version, DNS, and configuration currently in use.

“According to the documentation” and “according to the running system” are different kinds of evidence.

Make the safe path the easy path

People do not ignore process because they enjoy risk. Usually they ignore it because the approved path is slow, unclear, or harder than logging into a server and fixing the problem directly.

If deploying safely requires five manual steps and tribal knowledge, people will eventually bypass it.

Good operations work makes the right thing easier:

  • Deploy through one repeatable command or pipeline.
  • Keep configuration in a known location.
  • Make health checks visible.
  • Provide a documented rollback path.
  • Use least-privilege credentials by default.
  • Put common diagnostics where people can find them.
  • Make the production status obvious before anyone starts changing things.

The best runbook is not a long document. It is a workflow that removes opportunities for mistakes.

Backups are only real after a restore

Every team says they have backups. The important question is whether they can restore one.

A backup job can complete successfully while capturing the wrong data, writing to the wrong destination, or producing an archive nobody can actually use. A database dump without a tested restore procedure is evidence of good intentions, not disaster recovery.

Test restores on a schedule. Time them. Confirm the restored application can start and serve expected data. Write down what was missing, what took longer than expected, and what permissions or secrets were required.

The same goes for infrastructure state, deployment artifacts, and content files.

If recovery depends on one person remembering a sequence of commands, recovery has a single point of failure.

Observability is a debugging tool, not decoration

Dashboards can be beautiful and still leave you blind during an incident.

What matters is whether you can answer basic questions quickly:

  • Is the service up?
  • Is it serving the expected version?
  • Is the problem local or downstream?
  • When did the behavior change?
  • Which requests are failing?
  • Is the queue growing?
  • Did a recent deployment, configuration update, or certificate change line up with the failure?

Logs, metrics, traces, uptime checks, and alerts each answer different questions. None replaces the others.

Structured logs are especially valuable because they make it possible to connect events across systems. Include timestamps, service names, request IDs, operation IDs, and meaningful error details. Avoid dumping secrets or customer data into logs just because it is convenient.

When something breaks at 2 a.m., the goal is not to have the most telemetry. The goal is to reduce uncertainty fast enough to make a safe decision.

Idempotency saves you from normal failures

A surprising amount of operational reliability comes down to one question:

What happens if this runs twice?

Jobs are retried. Webhooks are duplicated. A worker may crash after the downstream system accepted its request but before it recorded the result. A person may press the button again because the first response timed out.

If the second execution creates another charge, another ticket, another deployment, or another destructive change, the system is fragile.

Build actions so they can be retried safely:

  • use stable idempotency keys
  • record operation state before external calls
  • read back results after writes
  • distinguish “not started,” “in progress,” “completed,” and “unknown”
  • make retries resume work rather than repeat it blindly

This matters for AI agents too. An agent does not make distributed systems less distributed. If anything, it creates more reasons for retries and partial failures.

Rollback beats confidence

No deployment process is perfect. No amount of testing eliminates every production failure.

The important capability is not believing a change will work. It is being able to reverse it quickly when it does not.

A deploy should answer these questions before it starts:

  • What version is live now?
  • What version are we changing to?
  • How do we verify success?
  • What signals mean the release is unhealthy?
  • How do we return to the previous version?
  • Who can authorize that rollback?

If rollback requires rebuilding a release, searching old chat messages, or reconstructing a server by hand, it will be slow exactly when speed matters.

Versioned releases, atomic symlink switches, feature flags, database migration discipline, and retained artifacts are not exciting. They are how you turn an outage from a long night into a controlled reversal.

Security is mostly ownership and defaults

Security failures are often described as sophisticated attacks. Plenty are much simpler:

  • a shared root password
  • an SSH key nobody rotated after an employee left
  • a token committed to a repository
  • a public admin endpoint with no meaningful authentication
  • a service account that can modify far more than it needs to
  • a backup stored beside the server it is supposed to recover

The cure is not one more security product. It is clear ownership and safer defaults.

Know who owns each credential. Use separate accounts for separate systems. Limit permissions. Rotate exposed secrets. Keep secrets out of repositories and browser code. Encrypt sensitive data at rest. Review the accounts and keys that still have access.

Every credential should answer: who uses this, for what, and how do we remove it safely?

Automation needs an operator mindset

Automation is valuable because it does repetitive work consistently. It becomes dangerous when it assumes the environment is always healthy and predictable.

A solid automation task checks preconditions, logs what it is doing, handles expected failures, and exits loudly when it cannot continue safely.

A fragile task swallows errors and keeps going.

The difference matters more as systems become more autonomous. An AI agent can plan and call tools, but it still needs the same guardrails as any other automation:

  • narrow permissions
  • explicit inputs and outputs
  • timeouts and retry limits
  • idempotent actions
  • current-state verification
  • audit logs
  • human approval for high-impact changes
  • a defined stop condition

Giving an agent a shell is not an operations strategy. Giving it a small set of well-designed tools can be.

Documentation is part of the system

The document nobody updates is not documentation. It is a historical artifact.

Useful documentation is close to the work:

  • setup instructions beside the code
  • runbooks linked from alerts
  • ownership recorded with services
  • environment variables described without exposing their values
  • rollback steps in the deployment process
  • architectural decisions written down when they are made

Keep it short enough to use. Keep it accurate enough to trust.

The test is simple: could someone unfamiliar with the system make a routine change or respond to a common alert without needing a private call with the person who built it?

If not, the system is still relying on memory.

The boring work compounds

Nobody gets excited about a clean service file, a tested restore procedure, a structured log event, or a deployment script that refuses to run without a health check.

Until something fails.

Then those details are the difference between controlled work and panic.

A decade of DevOps has made me skeptical of shortcuts that claim to remove operational complexity. New frameworks and new models can make teams faster. They do not repeal the rules of stateful systems.

The work is still the work: understand the system that exists, make changes in small reversible steps, verify outcomes, keep records, and leave the next person a safer path than the one you inherited.