systemd Start Request Repeated Too Quickly: Fix start-limit-hit

When a Linux service repeatedly exits during startup, systemd may eventually stop trying and report Start request repeated too quickly or start-limit-hit. Running systemctl reset-failed can clear the limit counter, but it does not repair the crash. If the service is still missing a configuration file, cannot bind its port, or exits because of a permission error, it will simply fail again.

This guide is a practical runbook for Ubuntu and other systemd-based Linux servers. It covers evidence collection, common root causes, a safe repair workflow, validation, and rollback. Replace myapp.service with the real unit name. Commands that change unit state or files should be run with appropriate administrative authorization.

Symptoms: identify the actual systemd failure

Check the unit state and recent journal entries before restarting it:

systemctl status myapp.service --no-pager -l
journalctl -u myapp.service -b --no-pager -n 150
systemctl show myapp.service -p Result -p ExecMainCode -p ExecMainStatus -p NRestarts

Look for the first application error before the repeated restart messages. The final start-limit message is often a consequence: systemd has rate-limited repeated attempts after the service failed too many times within its configured interval. The original cause may be several lines earlier in the journal.

Root causes to check

Application exits immediately

The process may fail because of invalid configuration, a missing environment file, a database connection failure, or an unhandled startup exception. Run the application’s documented configuration validation command, if it has one, and inspect its logs. Do not assume that increasing the restart limit makes an invalid configuration valid.

Wrong executable or working directory

A unit can point to an old binary path, a missing virtual environment, or a working directory that no longer exists. Inspect the loaded unit and its overrides:

systemctl cat myapp.service
systemctl show myapp.service -p ExecStart -p WorkingDirectory -p User -p Group

Confirm that the executable exists and is accessible to the configured service user. Avoid printing secrets from environment files into shared terminals or tickets.

Permissions and runtime dependencies

The service user may not be able to read configuration, write a socket or log directory, or access a required certificate. A dependency may also be unavailable when the process starts. Check ownership and permissions for the exact paths named in the journal; do not solve the issue with broad chmod -R 777 changes.

Port conflict or invalid listener

If the service cannot bind its configured port, identify the process already listening there:

sudo ss -ltnp

Compare the listener with the service configuration. Stop or reconfigure only the process that is confirmed to own the conflicting port. Killing a process based only on a port number can interrupt an unrelated production service.

Diagnosis: inspect the unit and its dependencies

Review the service definition, including drop-in overrides, and the dependency state:

systemctl cat myapp.service
systemctl list-dependencies myapp.service
systemctl show myapp.service -p After -p Requires -p Wants -p Restart -p StartLimitIntervalUSec -p StartLimitBurst

Check the current boot’s logs and the service’s exact exit status. If the application is a custom binary, test its built-in configuration check as the service user when safe. Do not manually run a second production instance using the same data directory or port just to see whether it starts.

Inspect recent changes: deployments, environment-file edits, certificate renewals, package upgrades, and firewall or database changes often explain why a previously healthy service began failing. Compare timestamps with the first failure rather than focusing only on the latest restart attempt.

Fix: correct the underlying failure first

Repair the specific issue shown by the evidence. Restore a missing file from the approved deployment artifact, correct the executable path, fix narrowly scoped ownership, free a port only after identifying its owner, or restore a required dependency. Keep the change in configuration management or version control so it is reproducible.

If the unit file itself needs modification, use a drop-in override instead of editing a vendor-managed unit under /usr/lib/systemd/system or /lib/systemd/system directly. For example, create an override editor session:

sudo systemctl edit myapp.service

Only add settings that match the actual diagnosis. After saving a unit change, reload systemd’s unit definitions:

sudo systemctl daemon-reload

Do not blindly add Restart=always or raise StartLimitBurst. Restart policies are useful for transient failures, but an aggressive retry loop can increase load, flood logs, or repeatedly execute unsafe startup behavior. Configure restart behavior according to the application’s failure modes and operational policy.

Reset the start limit only after the repair

When the underlying problem is corrected, reset the failed state and start-rate counter, then start the service:

sudo systemctl reset-failed myapp.service
sudo systemctl start myapp.service
systemctl status myapp.service --no-pager -l

reset-failed clears the failed state and resets the start-rate counter; it is not a repair command. If the service fails again, return to the new journal entries and diagnose the fresh failure instead of repeatedly resetting the counter.

Validation: check service behavior, not just Active

Confirm the unit remains active and that the application responds through its normal health check:

systemctl is-active myapp.service
systemctl show myapp.service -p ActiveState -p SubState -p Result -p NRestarts
journalctl -u myapp.service --since "10 minutes ago" --no-pager

Use the application’s documented local health endpoint or its own diagnostic command. Verify expected listeners with ss, confirm dependent services are reachable, and watch logs for repeated exits. A service that is briefly Active but restarts every few seconds is not healthy.

Rollback

If a configuration or unit override makes behavior worse, revert the exact change using your configuration-management workflow. For a systemd drop-in, inspect it with systemctl cat and remove only the override you created, then run systemctl daemon-reload. Restore the previous application configuration or package artifact from a known-good backup. Restart only after confirming that the rollback does not conflict with another running process or data migration.

Prevent the next start-limit incident

Make startup checks explicit: validate required configuration before deployment, use dedicated service users, create required directories with narrow permissions, and expose a meaningful health check. Alert on restart counts and service failure state, not only host availability. Record the first failure message and the deployment change that preceded it. Keep restart limits as a safety mechanism rather than disabling them to hide an unstable process.

Primary references

systemd.unit: Start rate limiting; systemctl: reset-failed; systemd.service: Restart behavior. Related ProdCraft guides: Run Python apps with systemd and Linux journalctl and logrotate.