Kubernetes troubleshooting lab

This is a lab. I break it so I have to find the layer. Nothing here was an incident for a customer.

Problem

Reading about kube-proxy is not the same as standing in front of a Service that will not answer. I wanted a cluster I could damage in specific ways: pods, DNS, network, limits, storage. Then put it back, and write down which command actually showed the fault.

Architecture

The workload is the public Online Boutique microservices demo, on a Kops cluster in AWS. The diagram is the path I walk, and the places I have deliberately broken.

What I built

A cluster, the demo installed, and a habit of changing one thing at a time. A debug pod for curls. Notes next to each break, because I do not trust my memory of which flag I flipped.

What broke

I have run these on purpose:

• CoreDNS pods pending after a node change. Every name looked down. The apps were fine.
• A NetworkPolicy that blocked the frontend from a backend. Endpoints were healthy. The request still died.
• A memory limit set too low. OOMKilled, then a crash loop, then a Service with nothing Ready behind it.
• A readiness probe on the wrong path. The pod stayed Running. The EndpointSlice went empty.
• A PVC stuck pending because the Kops cluster had no default StorageClass.
• Slow DNS from ndots and a short name used outside the namespace. The Service existed. The lookup was the wait.

Lessons

Change one thing. Write the command that proved it. Ready is not Running. A Service problem is often a pod problem, a policy problem, or a DNS problem wearing a Service's name.

The longer version of the Service path is a draft note in the failure log.