Alert on WhatsApp when a Kubernetes CronJob stops running

Kubernetes will tell you a Job failed. It will not tell you a CronJob stopped being created — and that is the more common outage. A suspended CronJob, a schedule with a typo, a node that cannot pull the image, a namespace cleaned up during a migration: all silent.

Prometheus can catch this with kube-state-metrics and an alert rule. If you do not already run that stack, a heartbeat gets you the same answer in one line of YAML.

A CronJob that reports in when it succeeds

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-export
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: export
              image: ghcr.io/acme/export:1.4.2
              command: ["/bin/sh", "-c"]
              args:
                - |
                  /app/export.sh &&
                  curl -fsS -m 10 "$HEARTBEAT_URL"
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: alertkite
                      key: heartbeat-url

Setting it up

Create a heartbeat monitor with a period matching the schedule, plus grace for image pulls.
Store the ping URL as a Secret rather than inlining it in the manifest.
Chain the curl after your command with && so only a successful run reports in.
Set concurrencyPolicy: Forbid unless overlapping runs are genuinely safe.

Why the URL belongs in a Secret

Anyone holding the ping URL can report the job healthy. That is a small blast radius, but manifests get committed to public repositories constantly, and a leaked heartbeat URL means silence exactly when you need noise.

startingDeadlineSeconds is the trap

If the controller cannot start a job within that deadline — during a node drain, say — it skips the run and records a miss. Without an external heartbeat, a cluster that skips every run looks identical to one that has nothing scheduled.

Grace has to cover the image pull

A cold node pulling a large image can add minutes before your command runs at all. Set the grace period against the worst case, not the steady state, or you will page yourself every time a node recycles.