Two things fail quietly until they hurt: a certificate that wasn't renewed, and a scheduled job that stopped running or keeps failing. Since 0.10, WatchLog finds both by itself, on Linux servers and on Kubernetes, and alerts before they become an outage. Nothing to configure: no list of hosts to check, no job to wrap in a ping.
Both are under Applications, next to the monitored services: the Certificates and Scheduled jobs tabs show the cluster or fleet selected in the top bar. The Overview and each server's page show their counts and problems, and the search (⌘K) finds them by name.
They need agent 0.10 or later (and, on Kubernetes, the agent chart's 0.10 permissions). A tab names the servers whose agent is older; upgrade them and their certificates and jobs appear within ten minutes.
Certificates
How they are found
On a server, the agent reads the certificate every listening TCP port presents, from the server itself, every 10 minutes:
- a TLS handshake on each port a service, process group or container listens on; a port that isn't TLS is left alone for an hour;
- STARTTLS on the well-known ports: SMTP (25, 587), IMAP (143), POP3 (110), PostgreSQL (5432) and MySQL (3306);
- the names in the server's certificate stores, asked for one by one on each TLS port (SNI), so
each virtual host's certificate is found: Let's Encrypt (
/etc/letsencrypt/live), acme.sh and Caddy. Only the stores' directory names are read; no file in them is opened.
On Kubernetes, the cluster-agent reads, from inside the cluster:
- each Service's TLS port (named
httpsor numbered 443) at its cluster address, the API server's owndefault/kubernetesincluded; - each Ingress's TLS hosts, asked for by name on its Ingress controller's Service.
No traffic is decrypted, no private key is read and no Secret is opened: WatchLog sees only what a client sees.
What the tab shows
Each certificate once, problems first, with its names, where it is served (a service and port on a server; a Service or an Ingress host in a cluster), its issuer, the days it has left with a bar of its lifetime, how it renews, and its status:
| Status | When |
|---|---|
| Expired | its validity ended |
| Expires soon | less than 3 days left |
| Expiring | less than 14 days left |
| Problem | it doesn't cover a name it is served for, or its chain is incomplete (the intermediate is missing) |
| Working | none of these |
Renewal. WatchLog keeps every certificate it has seen under the same names. A new one while the old one was still valid is a renewal, and the tab says when it last happened and how many days before expiry ("renews itself; last time 32 days ago"). Until it has seen one, it says "not seen renewing yet". When it sees the job that renews it on the same server (certbot's timer or cron line, acme.sh's), it names it, and says so when that job is failing. A certificate from a private CA or self-signed is marked as such.
A certificate's page lists every place it is served, its facts (issuer, validity, key, chain, fingerprint) and every certificate seen under its names.
Scheduled jobs
How they are found
- systemd timers, from systemd itself: every timer, enabled or not, its schedule as
systemd normalised it (
OnCalendar, or after boot), its random delay and whether it catches up after a reboot (Persistent), and each run of the unit it starts: when, how long, the exit code. - cron jobs, from the crontabs on the server (
/etc/crontab,/etc/cron.d/*and the users' crontabs), matched with the processes cron really starts: a small eBPF program sees each command cron's child runs, and its exit gives the code. Without eBPF, cron's own lines in a text log (/var/log/cron,/var/log/syslog) give each run, without its exit code. - Kubernetes CronJobs, from the cluster-agent: each Job a CronJob created is a run, and a
failed Job's pod gives the exit code.
spec.timeZoneis honoured.
Only a name leaves the server. A cron job's name is derived from its command: the script's
or program's file name (backup.sh, pg_dump), a framework's command (artisan schedule:run, manage.py clearsessions), or run-parts' directory (cron.daily). The
command line, its arguments, its environment, the user it runs as and its output are never
sent, so a password in a crontab line stays on the server. A timer's name is its unit's name.
States
| Status | When |
|---|---|
| Failing | its last run failed: the exit code, how many times in a row, the last good run |
| Missed | a run was due and none started, with the server and the agent up; or the timer was disabled after it was due |
| Slow | running, or ran, 3 times its usual duration and at least 5 minutes longer (once it has 5 good runs) |
| Running | a run is under way |
| Working | none of these |
| Disabled | a timer that is turned off and never ran |
A run that starts late because of a timer's random delay, a persistent timer's catch-up after a reboot, or the hour around a daylight saving change is never counted as missed. A run that was due while the server was off, or while the agent was stopped, is told apart and not counted either. Times are the server's own time zone.
The table shows each job's schedule in words and its next run, its last run and how long it took, and its last 14 runs as small bars. A job's page shows why it is in its state, its schedule as written, its runs and their durations, and for a systemd timer its unit's log on demand (Logs). A job that runs every 5 minutes or more often keeps its last 14 runs and an hourly count of its runs and failures, never one row or event per run.
Default alerts
On from the first day, in every edition, for Kubernetes and servers alike, and editable in Alerts like the others:
| Rule | When |
|---|---|
| Certificate expiring | 14 days left (warning); 3 days left or expired (critical) |
| Certificate name or chain wrong | a name it is served for isn't covered, or the chain is incomplete, for 10 minutes |
| Scheduled job failed | its last run failed (warning); 3 in a row (critical); a job every 5 minutes or more often only at 3 in a row |
| Scheduled job missed | a run was due and none started |
| Scheduled job slow | running, or ran, far longer than usual |
One alert per certificate and problem, and per job, not per run; a new certificate or the next good run resolves it. A certificate that renews itself on time never alerts. The message says what, where and since when: "backup.timer failed 3 times in a row on worker-1 · exit code 2 · last good run 2 d ago".
Events record a certificate renewed, about to expire or expired, and a job that failed, missed a run or recovered (one per job, reason and day, so a job every minute stays a few lines). The probable cause names an expired or mismatched certificate where a failing connection arrives, and the failing timer behind a service that failed.
What isn't covered
- A virtual host whose name is in no certificate store (a certificate copied in by hand under a name WatchLog can't know): only the port's default certificate is read.
- Ports bound to loopback only, with a
docker runinstall: the agent reads the host's ports through its network gateway, so127.0.0.1-only ports aren't reached. A systemd or package install reads them. - Beyond the caps: 50 ports and 100 certificates per server or cluster; the tab says how many more there were.
- Kubernetes: Ingresses without a TLS section, the Gateway API, and Services a NetworkPolicy keeps the cluster-agent from reaching (counted as unreachable). Without the chart's 0.10 permission to list Ingresses, only TLS Services are read.
- anacron and
at: anacron's own jobs run under cron asrun-parts, which is listed;atjobs aren't. - cron jobs inside containers, unless the container's cron runs in the server's process tree the agent sees.
- cron's
CRON_TZ=lines: such jobs are evaluated in the server's time zone. - cron runs without eBPF and without a text log (Debian 12's default keeps only the journal): its cron jobs are listed without runs and never alert.
- Certificates on UDP (QUIC, DTLS) and client certificates.
API and MCP
Read-only routes list a cluster's or fleet's certificates and scheduled jobs, and one of each
with its history or runs (API). The MCP tools list_certificates and
list_scheduled_jobs give an assistant the same answers (MCP): "Which certificates
expire this month?", "Did last night's backup run?".