Fail background jobs interrupted by shutdown

[msaroufim]

What changed

• catch worker cancellation while a background submission is active
• atomically mark the interrupted job as failed with a clear resubmission message
• update only pending/running rows so a completion racing with shutdown cannot be overwritten

Why

KernelBot stores accepted submissions in an in-memory worker queue. During a deployment, cancelling a worker propagated CancelledError through the active submission task, but that exception bypassed the normal failure handlers. The database row therefore remained running indefinitely, as happened to submission 873664.

Benefits

• deployment-interrupted submissions no longer appear to run forever
• users receive a terminal status and an actionable request to resubmit
• the conditional database update protects successfully completed jobs from shutdown races

This intentionally does not auto-retry submissions: automatic retry could duplicate an execution that already reached Modal before the local worker was terminated.

Validation

• uv run pytest tests/test_background_submission_manager.py -q (10 passed, including shutdown against real PostgreSQL)
• uv run pytest tests/test_leaderboard_db.py::test_leaderboard_submission_basic -q (1 passed)
• uv run ruff check src/libkernelbot/background_submission_manager.py tests/test_background_submission_manager.py
• git diff --check

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
background_submission_manager.py					288-289, 335-336
leaderboard_db.py
utils.py
Project Total

This report was generated by python-coverage-comment-action