Debug a Distroless Kubernetes Pod With Ephemeral Containers

A production container built from a distroless or minimal image may have no shell, package manager, curl, or ps. That is often intentional: fewer files mean a smaller image and a reduced attack surface. But when an incident requires DNS, network, or process inspection, kubectl exec -it POD -- sh may fail with “executable file not found.” Rebuilding the production image just to add diagnostic tools is rarely the best first move. Kubernetes ephemeral containers provide another option: attach a temporary debugging container to an existing Pod.

This runbook explains when the technique works, its security implications, and how to validate a repair without casually changing the application deployment. Replace the example namespace, Pod, container, service, and port with values from your own cluster. Use it only in a cluster where you are authorized to debug workloads.

Symptoms: separate a missing shell from an application failure

A failed kubectl exec does not prove the application is unhealthy. It may simply mean that the image has no /bin/sh. Other symptoms include missing curl or nslookup, an inability to inspect a process in a minimal image, or an intermittent production issue that is difficult to reproduce locally.

Preserve evidence before restarting anything. First confirm your cluster context, Pod state, recent events, and application logs:

kubectl config current-context
kubectl -n demo get pods -o wide
kubectl -n demo describe pod app-pod
kubectl -n demo logs app-pod -c app --tail=100
kubectl -n demo get events --sort-by=.lastTimestamp

Root causes: DNS, policy, port, or process?

Do not treat every timeout as a DNS issue. Common causes include an incorrect Service hostname or namespace, a Service selector that matches no ready Pods, an egress or ingress NetworkPolicy, an external firewall, a wrong destination port, or an application that listens only on an unexpected interface. A debugger can help collect evidence, but it cannot decide which cause is responsible without checking the surrounding Kubernetes resources.

Preflight: permissions and debugging-image trust

Adding an ephemeral container uses the pods/ephemeralcontainers API subresource. Permission to read or even update an ordinary Pod does not automatically grant permission to add one. Check the specific permission:

kubectl auth can-i update pods/ephemeralcontainers -n demo
kubectl -n demo get pod app-pod -o jsonpath='{.spec.nodeName}{"\\n"}'

If this returns no, ask the cluster administrator for the approved, narrowly scoped access path. Do not work around RBAC by using a privileged account without authorization. Kubernetes Pod Security Admission evaluates ephemeral containers against the Pod Security Standards too; admission webhooks and custom policies should also inspect this subresource.

Use a debugging image approved by your organization and preferably pinned by digest from a trusted registry. The command below uses busybox:1.36 to make the syntax readable; in production, substitute your organization’s approved immutable image reference. Debug images can contain powerful utilities and should be treated as privileged operational tooling. Avoid adding broad capabilities or using a privileged profile unless a specific, approved diagnostic requires it.

Diagnosis: attach an ephemeral container

For a running Pod, a basic interactive session looks like this:

kubectl -n demo debug -it pod/app-pod \
  --image=busybox:1.36 \
  --target=app \
  -- sh

The --target=app option asks the runtime to target the application’s process namespace. Process visibility depends on runtime support and the Pod’s configuration; if the debugger cannot see the application’s processes, that does not by itself prove the process is absent. Containers in a Pod share the network namespace, so a debugger can often test the Pod’s network path even when process targeting is unavailable.

Inside the debugging shell, start with read-only checks:

cat /etc/resolv.conf
hostname
nslookup kubernetes.default.svc.cluster.local
wget -S -O /dev/null http://service-name.demo.svc.cluster.local:8080/health

Available utilities and flags vary by image. A minimal BusyBox build may not include every networking command or certificate bundle. If a tool is missing, use a vetted image that contains it; do not interpret “command not found” as proof of a network problem. Replace the example service, port, and health path with a real, non-mutating endpoint.

Interpret the results with Kubernetes resources

If DNS lookup fails, verify that the Service exists in the expected namespace:

kubectl -n demo get svc service-name -o wide
kubectl -n demo get endpointslices \
  -l kubernetes.io/service-name=service-name
kubectl -n demo get networkpolicy

A Service with no ready EndpointSlice addresses can indicate a selector mismatch or unhealthy backends rather than a DNS failure. A successful DNS lookup followed by a timeout points toward connectivity, policy, or an unavailable upstream. A connection refusal often means the destination was reached but nothing accepted the connection on that address and port. Interpret each result alongside application logs and the Service’s targetPort.

Remember that a successful request from the debugger proves only that one request path worked at that moment. It does not prove that every replica, node, tenant, or application code path is healthy. The debugger may also expose process metadata, filesystem views, or sensitive command-line arguments through shared namespaces and /proc. Restrict access, keep the session auditable, and never paste secrets or process environments into public tickets.

Fix: repair the source of truth

Once the root cause is known, change the durable configuration rather than the debugger. Correct a Service hostname in application configuration, fix mismatched selectors or labels, adjust the smallest necessary NetworkPolicy rule, or reconcile the application’s listening port with the Service targetPort and health probes. Commit the change through your normal GitOps or deployment workflow. Avoid relying on manual edits to a live Pod; a replacement Pod will not preserve them.

Before a change, capture the current Deployment and Service manifests in a secure location:

kubectl -n demo rollout history deployment/app
kubectl -n demo get deployment app -o yaml > app-before.yaml
kubectl -n demo get svc service-name -o yaml > service-before.yaml

These files can contain sensitive operational details. Store them with appropriate access controls and do not commit them blindly to a public repository. Make one scoped change at a time so the result is attributable.

Validation: verify the user-facing path

After applying the repair, check rollout status, Pods, endpoints, and logs:

kubectl -n demo rollout status deployment/app --timeout=180s
kubectl -n demo get pods -l app=app -o wide
kubectl -n demo get endpointslices \
  -l kubernetes.io/service-name=service-name
kubectl -n demo logs deployment/app --tail=100

Then repeat the original request from the same source workload or network path that failed. Compare error rate, latency, and readiness with the baseline. If the issue affected one node or tenant, test that scope explicitly. A green rollout alone is not proof that the user-facing issue is fixed.

Rollback and cleanup

If the change causes regressions, revert the reviewed Git commit or restore the previous manifest. kubectl rollout undo deployment/app can roll a Deployment back to a prior revision, but it does not undo independent Service or NetworkPolicy changes. Inspect rollout history and revert each changed resource deliberately.

Ephemeral containers cannot be individually removed or restarted after they have been added. Their specification remains on the Pod until that Pod is replaced or deleted. Exit the shell, record the session, and allow the workload controller to replace the Pod through a planned rollout when appropriate. Do not delete a production Pod solely to erase a debugger without checking availability, replicas, and disruption budgets.

When ephemeral containers are not the right tool

If the Pod is Pending or cannot start, begin with scheduling events, image pulls, and node health. Ephemeral containers are not supported for static Pods. If policy blocks the feature, request an approved alternative. A copied Pod or a node-debugging Pod is a different workflow with different isolation and privilege implications; do not treat it as an automatic substitute.

Operator checklist

• Confirm the cluster context and target Pod before running commands.
• Preserve events and current and previous logs.
• Check explicit permission on pods/ephemeralcontainers.
• Use a trusted, pinned debugging image and least-privilege profile.
• Run read-only DNS and connectivity checks first.
• Correlate results with Services, EndpointSlices, and NetworkPolicies.
• Fix version-controlled configuration, then validate real traffic.
• Revert each changed resource deliberately and plan Pod replacement for debugger cleanup.

Primary references

Kubernetes: Debug Running Pods; Kubernetes: Ephemeral Containers; kubectl debug reference; Kubernetes: Network Policies. Related ProdCraft guides: CoreDNS DNS troubleshooting and ImagePullBackOff diagnostics.