Is Kubernetes Reliable? The Hard Truth After 8 Years in Production

I spent the first six months of 2020 convinced Kubernetes was a liability.

My team at SIVARO had just migrated a customer's core payment processing pipeline onto a 30-node cluster. Every Tuesday at 3:17 PM, the cluster would silently drop network policies. Not crash. Not error. Just stop routing traffic between services for exactly 47 seconds. The monitoring stack showed nothing. The logs showed nothing. We found it because a junior engineer, Ananya, stayed late and noticed a cron job's output looked wrong when she compared it to dashboard metrics by hand.

That's the real answer to is kubernetes reliable? — it depends on what you mean by "reliable." If you mean "does it crash?" — no, it almost never crashes. If you mean "does it do what you expect, predictably, without silent failures?" — absolutely not, unless you engineer for it.

I'm Nishaant Dixit. I've been running production Kubernetes clusters since 2018 at SIVARO, where we build data infrastructure and AI systems that process 200,000 events per second. I've broken more clusters than most people have deployed. Here's what I've learned.

What "Reliable" Actually Means for an Orchestrator

Most people think Kubernetes reliability is about uptime. They're wrong because uptime is the wrong metric.

Kubernetes has had 99.99% API server uptime across my clusters for the last three years. Zero unscheduled downtime from the control plane itself. That's not the problem.

The problem is that "reliable" for a distributed system means something different than "the software stays running." It means:

Predictable behavior during failures
Consistent performance under load
Deterministic outcomes from declarative configs
Silent failure detection — the hardest one

The Docker-in-production crowd from 2015 will tell you Kubernetes is overengineered. They're half right. But the alternative — rolling your own orchestration with Consul, Nomad, or raw systemd units — trades one set of reliability problems for another. I've done both. Kubernetes wins on the net, but only if you respect its failure modes.

The Reliability Taxonomy: 4 Kinds of Failure

I categorize Kubernetes reliability problems into four buckets. You'll recognize some of these if you've been in production longer than a month.

1. The Control Plane Is a Single Point of Failure (That Isn't Single)

The etcd cluster is the real bottleneck. Not the API server, not the scheduler. Etcd.

At first I thought this was a scaling problem — add more etcd nodes. Turns out it was a latency problem. Etcd's Raft consensus means writes must propagate to a majority of nodes. If your etcd cluster spans three availability zones and you get cross-AZ latency spikes above 10ms, your API server starts timing out. Deployments pause. Health checks fail. The cluster looks dead.

We tested this at SIVARO with a 5-node etcd cluster across us-east-1. Latency jitter between AZs hit 25ms during an AWS networking event in June 2025. The API server reported 503s for 12 minutes. The applications kept running — Kubernetes doesn't kill running pods when the control plane is down — but we couldn't deploy, scale, or roll back. That's a reliability failure even though no application crashed.

Fix: Run etcd in the same AZ. Or accept that multi-AZ adds latency risk. There's no free lunch.

2. Networking Is Where Reliability Goes to Die

CNI plugins are the most failure-prone component in any cluster I've seen.

Calico. Cilium. Flannel. Weave. I've run all of them in production. Cilium with eBPF has been the most reliable for us since switching in January 2024, but it's not bulletproof.

The specific failure we hit most often: network policy propagation delay. You apply a NetworkPolicy, it takes 3-12 seconds to propagate across nodes. In that window, traffic that should be blocked gets through. Or traffic that should be allowed gets dropped. Inconsistency window.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-except-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
    ports:
    - protocol: TCP
      port: 9090

That YAML looks clean. But apply it to a 50-node cluster running Calico v3.28 and you'll see a 6-second window where pods in the monitoring namespace can't reach the target, followed by a 3-second window where random other traffic still works. This is a reliability problem because your security model has a temporal gap.

We wrote a custom controller that checks policy propagation by probing endpoints before declaring a rollout complete. That's the kind of workaround you need.

3. Stateful Workloads: The Real Test

Stateless apps on Kubernetes? Fine. Reliable enough for almost anything.

Stateful workloads — databases, queues, anything with persistent volumes — are where is kubernetes reliable? becomes a serious question.

The issue is not Kubernetes itself. It's the ecosystem of CSI drivers and storage backends. We've used:

AWS EBS CSI driver: Good, but volume attachment limits per node (40) will bite you
Portworx: Expensive but actually reliable for stateful sets
Rook/Ceph: Don't. Just don't. We tried it in 2022 and gave up after 4 months

The StatefulSet controller itself is solid. But the combination of StatefulSet + PVC + PodDisruptionBudget + cluster autoscaler creates interactions that are hard to predict. Here's a real bug we hit:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-broker
spec:
  serviceName: kafka-headless
  replicas: 6
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

The podManagementPolicy: Parallel combined with maxUnavailable: 1 caused the controller to delete pods faster than the CSI driver could detach volumes. Two pods ended up with the same PV attached. Data corruption. June 2025. Never recovered fully — had to restore from backup.

Lesson: Don't use Parallel pod management with stateful workloads. Ever.

4. The Human Factor: Configuration Drift

The most common reliability failure I see isn't in Kubernetes code. It's in configuration drift across environments.

You've got a staging cluster running Kubernetes 1.28. Production is on 1.30. A developer writes a CronJob manifest that uses a beta API that's been removed in 1.30. It works in staging. It breaks in production. That's a reliability failure caused by version inconsistency.

We now run a policy engine (Kyverno, since April 2025) that validates manifests against the target cluster's API versions before deployment. But that's a band-aid. The real fix is keeping clusters within one minor version of each other.

The Security Angle: What Are the 4 C's of Kubernetes Security?

You can't talk about reliability without talking about security. An unreliable system that gets pwned is doubly useless.

What are the 4 c's of kubernetes security? Cloud, Cluster, Container, Code. That's the CNCF's official model.

Here's my take: the 4 C's are a useful mental model but they miss the point. The real security reliability question is: does your cluster survive a compromise without cascading failure?

We tested this at SIVARO in a controlled environment (August 2025). A compromised container with host network access. The attacker could reach the kubelet on port 10250. From there, they could exec into any pod on that node. The question was: could they break the control plane?

Answer: yes, because the kubelet credential was stored in a Secret that also had permissions to create ClusterRoleBindings. One compromised node, five minutes, and the attacker had cluster-admin.

The 4 C's model would have caught this if we'd audited Cloud (IAM roles) and Cluster (RBAC). But nobody does that audit as a pair. They check them separately. The gap between the layers is where reliability fails.

My rule: Every service account should have a PodSecurityPolicy (or OPA Gatekeeper constraint) that restricts its token's capabilities. If a container is compromised, the blast radius should be one node, not the cluster.

Is Kubernetes Production Ready?

This question comes up every year. It's been production-ready since Kubernetes 1.10 in 2018. But "production ready" doesn't mean "easy."

Let me give you the honest answer: yes, Kubernetes is production ready, but most teams aren't ready for Kubernetes.

The Kubernetes Learning Curve is real. We measured it at SIVARO. A senior engineer with 5+ years of backend experience takes 3-4 months to become productive with Kubernetes. Not "can deploy a pod" productive. "Can debug a failing cluster at 2 AM" productive.

That learning curve is a reliability risk. Because inexperienced operators make mistakes that look like Kubernetes failures but are actually human errors.

The Monitoring Gap: What You're Not Seeing

Standard Kubernetes monitoring (metrics-server, Prometheus, cAdvisor) shows you CPU and memory. That's table stakes. It doesn't show you:

API server request latency by verb (create vs get vs list)
Etcd commit latency
CNI plugin memory pressure
Kubelet heartbeat jitter

We built a custom dashboard after the 2020 network policy incident. Here's what we measure:

# Prometheus recording rules we use
- record: node:etcd_commit_latency_p99
  expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))

- record: cluster:kubelet_heartbeat_stale
  expr: count(time() - kubelet_node_status_update_timestamp > 60)

- record: cluster:network_policy_propagation_lag
  expr: max_over_time(cilium_network_policy_propagation_duration_seconds[1m])

If you're not measuring etcd_disk_backend_commit_duration_seconds, you don't know if your control plane is healthy. Period.

CI/CD Reliability: Where Clusters Go to Die

The most reliable Kubernetes cluster in the world is useless if your CI/CD pipeline deploys broken configs to it.

Kubernetes CI/CD Best Practices for Modern DevOps Teams recommends deployment strategies like canary and blue-green. We use them. But they only help if your pipeline validates the configs first.

Our worst production incident (October 2023) was caused by a CI/CD pipeline that deployed a ConfigMap with an invalid YAML value. The ConfigMap was valid YAML — the value was wrong. It set max_connections: 5 in a PostgreSQL config. The application crashed on startup because it expected an integer, not a string. The deployment rolled back, but the ConfigMap wasn't part of the rollback. 6 hours of downtime.

How Kubernetes Can Improve Your CI/CD Pipeline talks about using k8s for CI/CD infra. That's fine. But the real improvement comes from schema validation in your pipeline. We use kubectl diff --server-side with validation against OpenAPI schemas. If the pipeline can't validate the manifest against the live cluster's API version, it fails before deploying.

The Best Kubernetes CI/CD Tools: Top 7 Solutions In 2026 lists ArgoCD, Flux, and Harness as the top tools. We use ArgoCD. It's reliable if you configure it right. But don't let the tool fool you — the pipeline is only as reliable as the tests you run in it.

The Reliability Checklist: What I'd Ask a Vendor

If you're evaluating Kubernetes (or being sold a managed Kubernetes service), ask these questions:

What's your etcd backup strategy? If they say "managed by the platform," ask for RPO and RTO numbers. You should get sub-1-hour RPO for a single-AZ cluster.
How do you handle node failures in the control plane? Managed services like EKS and GKE handle this. Self-managed clusters need separate HA control planes.
What's your CNI plugin and what's its failure mode? Cilium has a "identity" mode that can cause connectivity issues during upgrades. Calico has issues with large route tables. Know which one you're getting.
What's the maximum pods per node? Default is 110 in kubelet. Most managed services set it lower (EKS defaults to 58 for some instance types). Exceeding this causes scheduling failures.
Can I get a Service Level Objective (SLO) for the API server? Not an SLA — an SLO. 99.9% availability with <200ms p99 latency. If they can't measure this, they can't operate the cluster reliably.

The Hard Truth

Is kubernetes reliable? Here's my one-sentence answer: Kubernetes is reliable as a platform, unreliable as a product.

As a platform — the APIs, the controller model, the reconciliation loop — it's one of the most solid distributed systems I've worked with. The control theory approach (desired state vs actual state) handles network partitions, node failures, and resource constraints better than anything else.

As a product — the YAML, the CNI plugin chaos, the CSI driver inconsistencies, the endless version churn — it's a mess. Every team that runs Kubernetes in production has a "war story" file. If they say they don't, they're either lying or not looking hard enough.

The difference between teams that succeed and teams that fail is observability and automation. You can't operate Kubernetes reliably by looking at dashboards 9-5. You need automated remediation, proactive scaling, and deep monitoring of the control plane internals.

At SIVARO, we run 47 production clusters across 3 cloud providers. We have automated runbooks for:

Etcd leader election failures
Kubelet certificate rotation issues
CNI agent restarts
API server throttling due to client misuse

The last one — client misuse — is the most common. Some developer writes a loop that calls list pods every 100ms instead of watching. The API server throttles. Everyone's deployments slow down. That's a reliability failure caused by bad client behavior, not by Kubernetes.

FAQ: Is Kubernetes Reliable?

Q: Is kubernetes reliable for stateful workloads like databases?
A: Yes, but you need to be careful. Use StatefulSets with OrderedReady pod management policy. Never use Parallel with PVCs. Test your CSI driver's behavior during node failures. We run PostgreSQL, Redis, and Kafka on Kubernetes. It works, but the operational overhead is higher than VMs.

Q: Is kubernetes production ready for small teams?
A: Yes, if you use a managed service (EKS, GKE, AKS). No, if you self-manage. The operational burden of etcd, networking, and upgrades will consume your team. I've seen 3-person teams fail because they spent 60% of their time on cluster management.

Q: What's the biggest reliability risk most teams ignore?
A: API version deprecation. Kubernetes removes deprecated APIs every 3-4 releases. If you're running manifests with extensions/v1beta1 Ingress (removed in 1.22), you're one upgrade away from broken deployments. Use kubectl convert to check. Or run kubent (Kubernetes Namespace Tool) in your CI pipeline.

Q: How do you measure reliability?
A: We use a custom reliability score based on: API server latency (p99 <200ms), etcd commit latency (p99 <10ms), kubelet heartbeat freshness (all nodes within 30s), and deployment success rate (>99.5%). If any of these drop below threshold, an incident is declared.

Q: Is kubernetes reliable for AI/ML workloads?
A: Yes and no. Batch jobs (training) work well with Volcano or custom schedulers. Inference serving with GPUs is trickier — NVIDIA's GPU operator has improved but still has bugs with multi-instance GPU partitioning. We use Kserve for inference and it's been stable since version 0.12.

Q: What's the one thing that would make Kubernetes more reliable?
A: Better testing of interactions between components. Most bugs I've seen are at the boundaries — CNI plugin + kubelet + CSI driver interaction. The Kubernetes community tests components in isolation. They need more integration tests across component boundaries.

Q: Should I migrate to Kubernetes in 2026?
A: Maybe. If you have less than 5 services, no. If you have 50 services with different scaling requirements, yes. If you're in between, consider Nomad or just run containers on VMs with systemd. Kubernetes has a fixed cost. Make sure your complexity budget covers it.

Conclusion

I've been doing this since 2018. I've built systems processing 200,000 events per second. I've broken clusters in every way possible. And I keep using Kubernetes.

Is kubernetes reliable? Yes, but only if you treat it like a distributed system that requires active management. Not a turnkey solution. Not a silver bullet. Not a set-it-and-forget-it platform.

The teams that say Kubernetes is unreliable are the ones that expected it to work like a managed database — deploy once, forget forever. It doesn't work that way. It's more like a nuclear reactor: powerful, complex, and you need trained operators watching the control rods.

The reliable operators I know share three traits:

They measure everything (especially the control plane internals)
They automate remediation (not just detection)
They accept that Kubernetes will fail in ways they can't predict — and they build for that

Kubernetes is the most reliable orchestration system I've used. It's also the one that requires the most work to make reliable. Those two statements aren't contradictory. They're the definition of engineering.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.