Test autoscaling like you test your app. Load-test it, break it, see what happens after an incident. Otherwise, you’ll find the limits when it hurts most.
Observability doesn’t matter unless it answers questions
Kubernetes has mountains of data. Things like logs, metrics, traces, events, audits, deployment history, container restarts, control plane noise, you name it. The real challenge isn’t collecting info, but actually it’s making sense of it. The CNCF and others have best practices for logging and telemetry, like centralizing logs and not leaking secrets. Those matter, but at the end of the day, engineers need answers, not just data. When something breaks, no one’s asking, “Is Kubernetes alive?” They want to know what changed. Did something roll out? Did a pod crash? Did autoscaling fire too late? Was a node unhealthy, a secret rotated, a network policy too tight, a downstream DB choking?
Observability should line up with real operational questions and not just ticking boxes for logs, or metrics. Dashboards need to match service ownership. Alerts need to mean something to end users. Telemetry should connect to deployments and incidents. Measure how quickly engineers spot the root cause, not just that you have the data somewhere.
