You’re walking into a cineplex to meet up with a friend you haven’t seen in a few years. However, the movie you agreed to see is showing in three different theaters. Your friend arrived first, and sent you a cryptic text saying “on the left.” The problem is that you don’t know if he means the theater is in the left wing or on the left side of the hallway. Essentially, you’re in the right place but have now had to sneak into several different theaters to figure out where your friend is.
When DevOps teams receive an error while working in Kubernetes, they face a similar problem. Like a cineplex, Kubernetes is a distributed system. Where the cineplex has different wings with numbered theaters on different sides of multiple hallways, Kubernetes has pods, nodes, control planes, network layers, and more. Since the errors often require manual searches across multiple locations, finding an answer to an issue becomes time-consuming.
By understanding some common errors and struggles, teams can implement best practices that improve the mean time to recovery (MTTR).
What Is Kubernetes Troubleshooting?
Kubernetes troubleshooting is the process of identifying, diagnosing, and resolving issues that arise in a Kubernetes cluster, including:
- A pod that fails to start.
- A service no one can reach.
- A node that stopped responding or reporting its status.
- An application technically running but not actually working.
Kubernetes is a distributed system with various moving parts that can include:
- Pods
- Nodes
- Control plane
- Networking layers
- Storage
- CI/CD pipeline that deployed the last change.
Effective troubleshooting requires moving rapidly from detecting that something broke to identifying quickly what changed, why the something broke, and the safest way to fix the issues.
What Are The Three Pillars of Kubernetes Troubleshooting?
Kubernetes incidents typically follow the same three step pattern that allow development teams to create repeatable processes:
- Understanding: Identifying the root cause requires knowing what changes before the error notification. Since the same symptoms can start in the pod, node, service layer, network, storage, or control plane, the first step in trouble shooting is pulling the signal from where it lives, like kubectl describe, pod and container logs, resource metrics, deployment history, and recent Git or config changes.
- Management: After understanding the likely cause, you need to focus on identifying the safest fix, not the fastest command. While a restart might fix the problem, it can cause a service disruption that impacts revenue so confirming the blast radius and identifying the changes allows you to apply the lowest-risk mitigation.
- Prevention: Prevention is about keeping the same issue from occurring in the future by updating runbooks with the exact symptom and fix, tuning alerts to surface the issue faster in the future, moving manual fixes into version-controlled configurations, and ensuring people have visibility into the failure mode.
Why Is Kubernetes Troubleshooting So Difficult?
For many development teams, kubernetes troubleshooting comes with unique challenges arising from:
- Distributed systems: Surface-level symptoms can come from anywhere within the system, including API server, etcd, a scheduler, controller managers, and any node’s kubelet working and talking with each other.
- Fragmented ownership: Clusters often host microservices built by different teams who may use different programming languages and have different dependencies with different logging conventions so determining whether an issue is a problem from DevOps or the application or both is a challenge.
- Uneven visibility: A pod’s logs disappear when the pod does so all metrics, events, and traces live in different places unless deliberately consolidated, requiring engineers to move between tools.
- Hidden failures: While self-healing features like automatic restarts and rescheduling help with errors, they become problematic when they fix a symptom before anyone notices and without addressing the underlying cause.
Common Kubernetes Errors and How to Fix Them
Kubernetes failures can occur at the pod, networking, cluster-access or deployment layers. By understanding some of the most common errors and how to fix them, you can troubleshoot faster.
CrashLoopBackOff
When the container starts and crashes, Kubernetes tries to restart it while an increasing delay occurs between attempts.
Some common causes include:
- Application-level bugs.
- Missing dependencies.
- Failed liveness probes.
- Bad startup commands.
- Container exiting immediately because no long-running process was defined.
When trying to troubleshoot this issue, you should consider:
- Reviewing kubectl describe pod <name> to see the Events section and restart count.
- Checking kubectl logs <name> –previous to see what the container printed right before it died.
- Deploying with a temporary sleep command to get a shell into the container before it exits.
ImagePullBackOff / ErrImagePull
Kubernetes is unable to pull an image specified for a container so it backs out because trying again.
Some common causes include:
- Typographic error in the image name or tag.
- Image fails to exist in the registry.
- Registry requires authentication that has not been configured, a missing or incorrect imagePullSecret.
When trying to troubleshoot this issue, you should consider:
- Reviewing kubectl describe pod <name> to identify the image Kubernetes is trying to pull and the failure reason.
- Verifying that the image name and tag are correct.
- Confirming that the registry is reachable.
- Checking that any required pull secret exists and is referenced correctly in the pod specifications.
OOMKilled (Exit Code 137)
While someone scheduled the pod, Kubernetes was unable to create the container.
The Linux kernel’s Out-Of-Memory (OOM) killer terminated a container because it exceeded its allotted memory limit, reporting the exit code 137 (SIGKILL) and the reason for the container’s termination as OOMKilled.
Some common causes include:
- Memory set too low for application’s actual needs.
- Memory leaks that grow usage over time.
- Unexpected traffic or data volume spikes.
When trying to troubleshoot this issue, you should consider:
- Checking that kubectl describe pod <name> names the missing object, like configmap “app-config” not found.
- Confirming the object exists in the correct namespace.
- Confirming that the name in the pod manifest matches exactly.
Node Not Ready
A node fails to report a healthy status, and any stateful pods on it become unavailable.
Some common causes include:
- The Kubelet process crashed.
- Node ran out of resources.
- Node lost network connectivity to the control plane.
When trying to troubleshoot this issue, you should consider:
- Reviewing kubectl get nodes for anything flagged with the NotReady
- Checking if the node recovers on its own within Kubernetes’ default five-minute window.
- Cordoning it to mark a Kubernetes node as unschedulable so the scheduler stops placing new pods in it.
- Draining if it is safe to move existing workloads
- Fixing the underlying kubelet or network issue.
- Replacing the entire node if it is not recoverable.
Pods Stuck in Pending / FailedScheduling
The API server creates or accepts the pod, but the scheduler fails to place it on a node.
Some common causes include:
- Insufficient CPU or memory across the cluster.
- Node affinity or taint/toleration rules that no node satisfies.
- Unbound PersistentVolumeClaims.
When trying to troubleshoot this issue, you should consider:
- Reviewing kubectl describe pod <name> to identify a FailedScheduling event explaining exactly what happened, often something like “”0/5 nodes are available: 5 Insufficient cpu.”
- Adjust resource requests.
- Relax scheduling constraints.
- Add capacity depending on what the message says.
DNS Resolution Fails Inside Pods
Pods fail to resolve service names or external domains, even though they run fine.
Some common causes include:
- Unhealthy or under-resourced CoreDNS pods.
- Unintentionally blocking DNS traffic with a NetworkPolicy.
- Misconfigured pod DNS config.
When trying to troubleshoot this issue, you should consider:
- Checking CoreDNS pod health using kubectl get pods -n kube-system -l k8s-app=kube-dns.
- Testing resolution directly from inside a pod with kubectl exec <pod-name> — nslookup kubernetes.default.
- Checking for NetworkPolicies that might be blocking traffic to port 53, if CoreDNS is healthy but resolution still fails.
- Confirming whether the failure is isolated to one namespace or cluster-wide to determine whether the issue arises from a policy or CoreDNS.
kubectl Fails to Reach the Cluster
Every kubectl command times out or fails, even though workloads may still be running fine inside the cluster.
Some common causes include:
- Wrong active context.
- Expired credentials.
- VPN or firewall blocking the connection.
- API server unavailable.
When trying to troubleshoot this issue, you should consider:
- Confirming that kubectl config current-context points to the right cluster.
- Checking kubectl config view –minify for the active configuration.
- Checking network connectivity to API server endpoint directly.
- Refreshing cloud provider authentication if using a managed service.
- Checking API server logs if you have infrastructure-level access.
Stuck Rollout
A deployment update fails to complete while both old and new pods are present, leaving the rollout unfinished.
Some common causes include:
- Checking whether New ReplicaSet’s pods fail readiness probes.
- Checking whether New ReplicaSet’s pods are unable to pull their image.
- Checking whether New ReplicaSet’s pods don’t have enough cluster resources to schedule.
When trying to troubleshoot this issue, you should consider:
- Reviewing kubectl rollout status deployment/<name> to identify any progress at all.
- Checking kubectl describe rs <new-replicaset> and kubectl get events for why the new pods fail to become read, with failed probes, image pull errors, or scheduling failures listed here.
- Using kubectl rollout undo deployment/<name> to roll back immediately so you can fix a genuinely broken underlying issue.
Other Exit Codes (1, 125, 139, 143)
Some errors and failures follow different patterns than the ones above. Some examples of these include:
Exit Code 1: General application error, code bug, or a bad reference to a file that fails to exist in the image.
Exit Code 125: Container runtime failed to run the container, often due to an invalid command or daemon-level failure, occurring before the application starts.
Exit Code 126: Command invoked cannot execute, typically a permissions issue or the binary not being marked executable.
Exit Code 127: Command not found, usually a typo in the entrypoint/command or a missing binary/path in the image.
Exit Code 139: Segmentation fault (SIGSEGV), typically pointing to a library compatibility issue, a code bug, or a hardware/architecture mismatch.
Exit Code 143: Container received SIGTERM, a shutdown signal, that means checking kubelet logs to identify what triggered the termination request if it happens unexpectedly.
Best Practices for Faster Kubernetes Troubleshooting
When development teams can rapidly identify an issue’s root cause, they improve customer satisfaction and, ultimately, improve revenue by spending more time on roadmap projects. To improve these processes, you can consider some of these best practices.
Centralize Kubernetes Logs Instead of Relying on Node-Local Storage
Aggregating Kubernetes logs from every pod, node, and control-plane component into a single searchable system streamlines investigation processes, especially when compared to leaving them scattered across ephemeral nodes. Centralizing logs removes the single point of data loss and enables incident correlation across the whole cluster instead of focusing on one node at a time.
When DevOps teams troubleshoot Kubernetes, they should ensure that they or their solution for managing incident data can:
- Forward logs through a node-level agent, like DaemonSet, into a centralized platform rather than depending on kubectl logs, since pod-local logs disappear the moment a pod is gone.
- Ingest and correlate logs across every namespace and node simultaneously, since distributed incidents rarely announce themselves on a single machine.
- Route control-plane logs into the same system as application logs, so one search can span both the symptom and the root cause.
Use Structured Logging to Make Kubernetes Logs Actually Searchable
Normalizing logs into machine-readable, key-value fields makes filtering, dashboards, and building alerts possible. When different teams and programming languages produce inconsistent log formats and unstructured text, investigations turn into manual processes.
When DevOps teams troubleshoot Kubernetes, they should ensure that they or their solution for managing incident data can:
- Index fields like pod name, namespace, container, and image so teams can filter straight to the failing resource instead of scanning raw text output.
- Parse and normalize logs from many different microservices and languages into a consistent schema for cross-service correlation.
- Build saved, parameterized searches for recurring failure signatures like CrashLoopBackOff and OOMKilled so the same investigation can be rerun instantly against a different pod or namespace.
Set Retention, Rotation, and Archiving Policies
Defining how long you keep logs, when they roll over, and when older data moves to cheaper archive storage help you keep important data while maintaining a budget. For example, verbose debug and trace logs may be relevant later so less expensive cloud storage creates cost-effective options. Kubernetes has no native long-term log storage, so a policy prevents node disks from filling up and critical evidence from disappearing.
When DevOps teams troubleshoot Kubernetes, they should ensure that they or their solution for managing incident data can:
- Configure index rotation and retention so verbose debug and trace logs stay available long enough to catch slow-building issues without runaway storage costs.
- Move aging logs to cheaper archive storage instead of deleting them, since a postmortem may need historic data.
- Apply separate retention policies to development and production clusters to prevent noisy dev logs from taking space production data needs during a live incident.
Make Search Fast Enough to Matter Mid-Incident
When responders can filter, save, share, and rerun searches across large volumes of Kubernetes log data in seconds, then turn searches into a live dashboard, they directly reduce response times. Speed to root cause depends on how fast someone can go from noticing something is broken to finding the log line that explains it.
When DevOps teams troubleshoot Kubernetes, they should ensure that they or their solution for managing incident data can:
- Save search filters by pod, namespace, or image so a one-off investigation becomes a repeatable query the whole team can reuse on the next incident.
- Push search results directly into dashboard widgets, since spotting a pattern is faster in a chart.
- Support parameterized searches so the same query can be rerun against a different pod or namespace without rebuilding it from scratch.
Build Alerts for Failure Signatures
Automatically triggering a notification when a defined condition appears in the log stream helps identify issues before they become incidents. For example, you could build alerts around a spike in restarts, a wave of OOMKilled events, or repeated failed image pulls. By implementing alerts, you can minimize the impact of Kubernetes self-healing behaviors that can mask a problem long enough for it to become a problem.
When DevOps teams troubleshoot Kubernetes, they should ensure that they or their solution for managing incident data can:
- Build event definitions for known failure patterns such as CrashLoopBackOff loops, repeated OOMKilled events, and ImagePullBackOff so the team is notified the moment the pattern starts.
- Correlate multiple related events into a single alert instead of sending one notification per pod that buries the real signal.
- Route alerts into the channels the team already uses so remediation can start immediately, including Slack, MS Teams, and email.
Control Log Access
Applying role-based access control (RBAC) so engineers only see logs for the services or namespaces they own and masking sensitive fields at ingest reduces both security risk and noise for the person troubleshooting. Kubernetes logs frequently carry secrets, credentials, and personally identifiable information through env vars and application output.
When DevOps teams troubleshoot Kubernetes, they should ensure that they or their solution for managing incident data can:
- Scope log access by namespace or service ownership so an on-call engineer sees exactly what is relevant to their incident.
- Mask or anonymize sensitive fields like credentials and PII at ingest.
- Maintain an audit trail of who accessed which logs during an incident, for compliance and for reconstructing the investigation.
Graylog: Lightning Fast Kubernetes Troubleshooting
Graylog enables organizations to centralize Kubernetes logs and manage them more effectively. With Graylog, organizations have several options for how to rotate Kubernetes logs, including easily setting up an index rotation and specifying when to delete, close, or archive logs. For archival purposes, you can move logs to a new location and compress them to save space. Graylog Illuminate provides parsing and streamlined log ingestion through OpenShift Content Packs.
With Graylog’s scalable, flexible platform, organizations can implement proper log retention and monitoring while minimizing the cost of Kubernetes log storage and management.
