Scheduling It, and Noticing When It Stops

By the end of this lesson you can choose how a job is triggered and how two runs are stopped from overlapping, and you can write the check that notices a job has not succeeded recently — which is the only kind of monitoring that catches a job that has stopped running altogether.

Here is the outage that every automation eventually has, and the reason it lasts so long.

A job runs every night and sends a report. In March, the machine it runs on is rebuilt and the scheduled task is not recreated. The job does not fail. It does not error. It does not run. No message is sent, and nobody notices, because the absence of a report at seven in the morning looks exactly like a quiet week.

In June, somebody asks where the March figures went.

Every alert that watches for errors is blind to this. An error requires something to run. The only thing that catches it is an alert on silence: not "did the run fail" but "when did a run last succeed, and is that longer ago than it should be".

The second half of this lesson is the same idea applied earlier: a run that starts before the previous one has finished. Two copies of a job writing the same output is not usually reported as an error either. Both runs succeed, and one of them overwrites the other's work.

Choosing the trigger. Three shapes, and they are not interchangeable:

• On a clock. Every hour, every night at 02:00. Simple, and the default. The interval has to leave room for the run's own deadline, which the services module set.
• On an event. A file arriving, a message queued. Better when the work is caused by something rather than due at a time, and it needs its own guard against the event arriving twice.
• On demand only. Run by a person. The right answer more often than people expect, particularly while a job is new.

Stopping overlap. A run that takes longer than its interval will eventually meet the next one. Two mechanisms, and using both is normal:

• The scheduler's own policy. Windows Task Scheduler has a setting for what to do when the task is already running — do not start a new instance, queue it, or stop the existing one. Choose deliberately rather than accepting whatever the default is.
• A lock the job itself holds. A lock record can contain the run id and start time, but the job must acquire it atomically and release only the lock it owns. Done correctly, this guards starts from any trigger, including a person running the job by hand while the schedule fires. Merely creating and removing a file is not an exclusivity guarantee.

A lock needs recovery rules because a killed run can leave it behind. An age such as three times the expected runtime can prompt investigation, but age alone is not proof the owner has stopped: reclaiming a live run's lock can create the very overlap it was meant to prevent. Verify the owner is gone or use a lease with renewal and fencing before reclaiming; otherwise one crash can block later runs.

On Windows, the built-in scheduler is driven from the command line by schtasks. Microsoft's own documentation for that command is the reference to check the details against; the shape is:

schtasks /create /sc daily /st 02:00 /tn "nightly-invoices" ^
         /tr "C:\path\to\python.exe C:\path\to\job.py --apply" /f

schtasks /query /tn "nightly-invoices" /v /fo LIST
schtasks /run   /tn "nightly-invoices"
schtasks /delete /tn "nightly-invoices" /f

/sc daily with /st is the trigger, /tn names the task, /tr is what runs, and /f suppresses the confirmation prompt when a task of that name already exists. A task that runs as you, on your own machine, does not need administrator rights; one that runs as the system account does. Microsoft documents that schtasks may prompt for your account password when creating a task, which is normal behaviour rather than a sign that something is wrong.

Two details that catch people out, both worth checking with /query rather than assuming: the task runs with a working directory you did not choose, so use absolute paths for everything; and a task set to run only when the user is logged on does not run when they are not.

On Linux and macOS, the equivalent is a cron entry, and cron has no overlap policy at all, so the lock is not optional there.

Alerting on silence. The run record from the first lesson of this module is the heartbeat. The check is arithmetic:

age of the most recent *successful* run  >  interval + grace   =>  stale

Three states, not two, and each needs a different message: healthy, stale, and never succeeded — a job that has never once completed is a different problem from one that stopped completing, and reporting both as "stale" sends whoever reads it looking for the wrong thing.

The grace period exists so that a run which is merely late does not alert. Make it a fraction of the interval, not a fixed number of minutes, or an hourly job and a monthly job end up sharing a tolerance that suits neither.

Two separate alerts. "The last run failed" and "no run has succeeded recently" are different questions with different answers, and a job can be in one state without the other: a job failing every night becomes stale when the time since its last success exceeds the interval plus grace, and a job that stopped running is stale while never having failed at all.

import datetime


def parse(moment):
    return datetime.datetime.fromisoformat(moment)


def health(records, now, interval_minutes, grace_minutes):
    """Is this job still succeeding often enough? Returns (status, message)."""
    completed = [record for record in records if record["status"] == "completed"]
    if not completed:
        return "never succeeded", "no completed run on record"
    latest = max(parse(record["finished_at"]) for record in completed)
    age = int((parse(now) - latest).total_seconds() // 60)
    allowed = interval_minutes + grace_minutes
    if age > allowed:
        return "stale", f"last success {age} minutes ago, allowed {allowed}"
    return "healthy", f"last success {age} minutes ago, within {allowed}"


RECORDS = [
    {"finished_at": "2026-03-01T02:04:00", "status": "completed"},
    {"finished_at": "2026-03-02T02:06:00", "status": "completed"},
    {"finished_at": "2026-03-03T02:01:00", "status": "failed"},
    {"finished_at": "2026-03-04T02:03:00", "status": "failed"},
]
NOW = "2026-03-04T09:00:00"

status, message = health(RECORDS, NOW, interval_minutes=1440, grace_minutes=120)
print("newest record: ", max(record["finished_at"] for record in RECORDS))
print("newest success:", max(r["finished_at"] for r in RECORDS if r["status"] == "completed"))
print("status: ", status)
print("message:", message)

This is a job that ran on all four nights and succeeded on the first two. The two printed timestamps are there to make the distinction visible: something happened seven hours ago, and nothing has worked in more than two days.

completed filters before taking the maximum, so a failed run does not count as a heartbeat. That one line is the difference between a check that notices a job failing every night and one that reports it as healthy because it ran.

max over the parsed timestamps rather than the last element of the list: records are not guaranteed to arrive in order, and "the newest" is a question about values, not positions.

The age is computed in whole minutes. In this simulation all timestamps use the same naive clock; a deployed checker needs timestamps normalized to a common timezone and a policy for future-dated records.

The job runs daily, so interval_minutes is 1440 and the grace is 120, allowing 1560. Predict all four lines.

newest record:  2026-03-04T02:03:00
newest success: 2026-03-02T02:06:00
status:  stale
message: last success 3294 minutes ago, allowed 1560

Delete the completed filter, so latest is taken over every record regardless of status. The simplest way is to replace that line with completed = list(records).

newest record:  2026-03-04T02:03:00
newest success: 2026-03-02T02:06:00
status:  healthy
message: last success 417 minutes ago, within 1560

An assistant was asked for "monitoring so we know if the nightly job is working". It produced this.

def check(log_path):
    lines = log_path.read_text().splitlines()
    if not lines:
        alert("no log output")
        return
    if "ERROR" in lines[-1]:
        alert("job failed: " + lines[-1])
    else:
        print("job healthy")

def check(records, now, interval_minutes, grace_minutes):
    status, message = health(records, now, interval_minutes, grace_minutes)
    if status != "healthy":
        alert(f"{status}: {message}")
    last = max(records, key=lambda record: parse(record["finished_at"])) if records else None
    if last and last["status"] not in ("completed", "refused"):
        alert(f"most recent run {last['status']}: {last.get('reason', '')}")

Write health, which decides whether a job is still succeeding often enough. It is run against three supplied jobs: one healthy, one that has not succeeded in far too long, and one that has never succeeded at all.

Loading this exercise…

Optional, and the transfer task for this lesson: schedule something real, on your own machine.

This is the local-Python path, and local-python-setup.txt has the exact commands, Windows first, including the ones for removing what you create.

Write a two-line script that appends the current time to a file, schedule it to run every five minutes with schtasks on Windows or a cron entry elsewhere, and watch the file for fifteen minutes. Then do the part that matters: delete the scheduled task and confirm with schtasks /query that it is gone.

If you would rather schedule something with more to look at, starter_daily_digest.py is this module's ideas in one file, with its completed version and sample data. It has a --health flag that reports exactly the staleness check you just wrote.

Sign in to track your progress on this exercise.

1. Why can no amount of error alerting detect a job that has stopped running?
2. Why must a staleness check count only successful runs?
3. Why are "stale" and "never succeeded" reported as different states?
4. Why does a job need its own lock even when the scheduler has an overlap policy?

Sign in to track your progress on this exercise.

Choose the trigger deliberately, decide what happens when a run is still going when the next one is due, and acquire a job lock atomically with a safe recovery rule so one crash does not disable the job permanently or let two live runs overlap. Then build the alert that error handling cannot provide: the age of the most recent successful run, compared against the interval plus a grace period, reported as healthy, stale, or never succeeded. Nothing in a browser can install any of this, and the local path is where you do it for real. The capstone now assembles everything: a scheduled reporting workflow with validation, a run record, safe reruns, tested recovery, and the documentation somebody else could run it from.