When Kubernetes replaces a pod or a process manager restarts a service, immediate exit can interrupt requests, lose queued jobs or leave partial operations. Graceful shutdown gives the service a bounded window to stop accepting work and finish what is already in progress.
Plan the workflow
Use signal.NotifyContext for SIGTERM and SIGINT. When cancellation occurs, stop accepting new requests and notify background workers. SIGKILL cannot be caught, so the application must finish before the platform termination deadline.
Configure the controls
Call http.Server.Shutdown with a deadline. It closes listeners and waits for active connections to become idle until the context expires. Check ListenAndServe errors explicitly. Long-lived streams and upgraded connections need separate handling and tests.
Validate failure cases
Stop polling for new jobs, finish or safely requeue active jobs, and close dependencies only after workers exit. Make operations resumable or idempotent if they cannot finish before the deadline. Coordinate readiness removal and Kubernetes terminationGracePeriodSeconds.
Production checklist
Test SIGTERM during normal requests, streams and jobs; test dependency failures during cleanup; ensure shutdown completes within the orchestrator budget. Related: Kubernetes rolling updates.
undefined
Architecture and prerequisites
Treat shutdown as a sequence, not a single process exit. First stop accepting new work, then allow the load balancer or orchestrator to stop routing requests, then drain active HTTP handlers, finish or requeue background jobs and finally close database and network clients. If dependencies are closed too early, in-flight handlers fail during cleanup. Give each stage a bounded timeout and log which stage exceeded its deadline. The sequence should be documented and tested for the service’s actual dependencies.
Configuration and implementation
Use signal.NotifyContext for SIGTERM and SIGINT and pass cancellation into worker loops. Do not use an unbounded background context for operations that should stop with the process. For HTTP servers, call Shutdown with a timeout and handle its returned error. Long-lived streaming connections, WebSockets and upgraded protocols need special treatment because they may not drain like ordinary requests. Decide whether to notify clients, allow a grace period or close them at a deadline.
Security and compatibility
Coordinate readiness with termination. A pod should stop receiving new traffic before the application exits, but endpoint and load-balancer updates take time to propagate. Use readiness checks that reflect whether the instance can accept work, and size terminationGracePeriodSeconds to cover propagation plus draining. A preStop hook may help in some deployments, but it consumes the same grace period. Avoid arbitrary sleeps without measuring the platform behavior.
Testing and validation
Background jobs need a clear shutdown contract. Stop fetching new jobs, finish safe in-flight work or release leases so another worker can retry. Persist checkpoints for long-running operations. Make side effects idempotent because a job may be restarted after a node failure even when graceful shutdown was attempted. Close message consumers only after their current acknowledgements are settled, and avoid acknowledging work before the result is durable.
Monitoring and troubleshooting
Test termination during normal requests, streaming responses, database writes and active jobs. Simulate a dependency that becomes unavailable during cleanup. Verify that the service exits within its orchestrator deadline and that logs explain unfinished work. Test SIGKILL separately as an ungraceful failure because no signal handler can intercept it. Recovery logic must remain safe even when graceful shutdown never runs.
Production operations
Track shutdown duration, active connections at signal time, jobs completed, jobs requeued and forced exits. Alert when a service repeatedly reaches its deadline or leaves work unfinished. Document how operators can drain one instance during maintenance and how to restore capacity after a failed rollout. Graceful shutdown should be an observable reliability property, not a code path that nobody has tested since the first deployment.
Operational readiness and lifecycle
Before rollout, define an owner for the configuration, supported versions, dependencies and escalation path. Put configuration under version control and require review for production changes. Verify the implementation with a representative workload and record a baseline so future changes can be compared. Test a failure scenario, a restart and a rollback rather than relying only on a successful happy-path test. Monitoring should expose both the component’s internal health and the user-visible outcome, with alert thresholds that have a documented response. Keep a concise runbook containing diagnostic commands, expected signals, safe retry guidance and recovery steps. Review the design after the first production cycle and whenever an upstream dependency changes. Remove temporary exceptions, rotate credentials where applicable and update compatibility notes. This ongoing ownership is what turns a working configuration into a maintainable production service.
Scenario-based verification
For Go shutdown, test SIGTERM while requests, streaming connections and background jobs are active. Confirm readiness is removed before the listener closes, the process exits inside the orchestrator deadline and unfinished jobs can be safely retried. Track forced exits and shutdown duration so regressions appear in monitoring.
Performance and capacity validation
Define a measurable service-level baseline before enabling the change. Record normal and peak throughput, error rate, latency distribution, resource consumption and recovery time. A single successful test does not show how the system behaves under concurrency or after a dependency becomes unavailable. Run load tests that resemble real request patterns and include burst traffic, slow clients and large payloads. Compare results with the baseline and investigate regressions rather than accepting them because the deployment completed. Keep test data representative but non-sensitive, and repeat the test after significant runtime or dependency upgrades.
Logging and alert design
Operational logs should make failures diagnosable without exposing credentials or customer data. Include a correlation identifier, component version, action outcome and safe resource reference. Avoid logging bearer tokens, private keys, full database records or unrestricted request bodies. Define retention and access permissions for diagnostic data. Alert on repeated failures, queue growth, resource saturation and missed scheduled work, but ensure every alert has an owner and an action. Review alert noise after rollout and remove rules that do not lead to useful investigation.
Recovery documentation
Document the recovery path as carefully as the setup path. Include prerequisites, permissions, exact verification commands, expected outputs, rollback steps and conditions that require escalation. Test the procedure with someone who did not build the original system; this exposes assumptions that are obvious only to the author. Keep the runbook next to the configuration or source repository and update it when versions or workflows change. Schedule a periodic review so temporary exceptions, obsolete settings and unsupported dependencies do not become permanent parts of production.
Apply these checks to Go Graceful Shutdown: Drain HTTP Requests and Background Workers: retain the test evidence, identify the owner who accepts the production risk and define the conditions that trigger rollback. Re-run the checks after major upgrades or changes in traffic patterns.
