Server health preset

Most outages on a single server are not the app crashing. They are the quiet things around it: an nginx config that no longer passes nginx -t and only fails at the next reload, a certificate renewal that has been broken for weeks, a disk that fills up and takes the database with it, a backup job that stopped producing files.

The Server health preset sets up a monitor for each of those in one step, and gives you the cron file that feeds them.

What it creates

The checks marked ✓ are selected by default. The web server and the database are yours to pick, since a server usually runs nginx or Apache and one database, and a monitor for a service that is not there would only alarm. mysqladmin uses the credentials of the user cron runs as: for root that is the unix socket or /root/.my.cnf.

You pick which checks you want. Each one is an ordinary ping monitor with an expected interval and a grace period, so a check that stops reporting alerts as missed, and a check whose command fails alerts as failed.

A preset only creates. If a ping monitor with the same name already exists, it is reported as existing and left untouched, so applying the preset again never resets settings you changed later. Names are per monitor type: an HTTP or SSL check called web1-nginx-config-test does not block the ping of the same name.

In the panel

1. Monitors → Add monitor → Presets → Server health.
2. Enter a host name (lowercase, e.g. web1). It becomes the prefix of every monitor name.
3. Tick your web server and database, adjust the backup directory and the disk threshold, and click Preview.
4. Click Create monitors, then copy the cron file shown below the preview.

On the server

The cron file calls the shell client, so install and configure it first:

curl -fsSL https://hellocron.com/hellocron.sh -o /usr/local/bin/hellocron
chmod +x /usr/local/bin/hellocron
sudo hellocron configure

Then save the generated file as /etc/cron.d/hellocron-server-health. It looks like this:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
7 * * * * root /usr/local/bin/hellocron run web1-nginx-config-test /bin/sh -c 'nginx -t'
*/15 * * * * root /usr/local/bin/hellocron run web1-disk-space-root /bin/sh -c 'test "$(df --output=pcent / | tail -n 1 | tr -dc 0-9)" -lt 85'

Three details that matter:

• Keep the PATH line. Cron gives a job a bare PATH (/usr/bin:/bin on Debian), and nginx, apachectl and mysqladmin live in /usr/sbin - without it a healthy server would report a failure every time.
• No dot in the file name. Cron on Debian and Ubuntu skips files in /etc/cron.d whose name contains a dot.
• Keep the trailing newline. Cron ignores a last line without one. The generated file already ends with a newline.

hellocron run sends the run ping, executes the command, and sends complete or fail with the exit code and error output. The command sits inside sh -c '...' so that pipes and redirects are part of the monitored run.

From the server, in one command

Instead of ticking the checks in the panel, let the host say what it runs. With the client installed and a management key saved (sudo hellocron configure --management-key, the key starts with mk_ and needs the write scope), run:

sudo hellocron discover-server-apps

It looks for the binaries behind each check (nginx, apachectl, certbot, pg_isready, mysqladmin, redis-cli, mongosh) and asks systemd, or the process list, whether the service is running. The result is a list you can adjust before anything is created:

Checks for this host ([x] = selected):
  [x]  1. nginx-config-test      nginx -t                                   running (/usr/sbin/nginx, service active)
  [ ]  2. apache-config-test     apachectl configtest                       not found
  [x]  3. certbot-renew-dry-run  certbot renew --dry-run                    installed (/usr/bin/certbot)
  [x]  4. disk-space-root        disk usage of / below the threshold        built-in
  [x]  5. backup-freshness       a file younger than 25h in the backup dir  built-in
  [x]  6. postgres-ready         pg_isready                                 running (/usr/bin/pg_isready, service active)
  [ ]  7. mysql-ping             mysqladmin ping                            installed (/usr/bin/mysqladmin, service not active)
  [ ]  8. redis-ping             redis-cli ping                             not found
  [ ]  9. mongodb-ping           mongosh ping                               not found

Toggle: numbers separated by spaces, a = all, n = none, Enter = continue, q = quit:

Web servers and certbot are ticked as soon as they are installed, because nginx -t works whether or not nginx is running. A database is ticked only when its service is active: an installed but stopped MySQL is usually a leftover, and a monitor for it would only alarm. Then the client asks for the host name (the prefix of every monitor name, defaulting to this host’s name), the backup directory and the disk threshold, shows what will be created and what already exists, and after your confirmation creates the monitors and prints the cron file.

After printing the file in a terminal, the client offers to save it to /etc/cron.d/hellocron-server-health (requires root). Confirmation installs the file owned by root:root with mode 0644. Replacing an existing file requires a separate confirmation, defaulting to no, and keeps a hidden backup that cron does not execute. Checks run at their scheduled times while the cron service is running.

The cron file is the only thing on stdout; the list, the questions and the summary go to stderr. The install prompt is skipped with --yes, -o or redirected stdout. For a non-interactive write (redirection needs a root shell):

sudo sh -c 'hellocron discover-server-apps --yes > /etc/cron.d/hellocron-server-health'

--yes accepts the detected selection and the defaults without asking. --items nginx-config-test,postgres-ready picks the checks yourself, --all takes every check, --host web1, --backup-dir /srv/backups and --disk-threshold 90 set the answers up front, --dry-run stops after the preview, and -o <file> writes the cron file to a path instead of stdout. Like the panel, the command only creates: monitors that already exist are listed as existing and left untouched. The command needs a plan with config as code; on a plan without it the client says so and points you to the panel.

Over the API

The same presets are available to provisioning scripts through the Management API (key mk_). Applying a preset is a POST and needs the write scope; GET /api/v1/presets is a read and needs read, so a named key used for both needs both scopes:

curl -s https://app.hellocron.com/api/v1/presets/server-health/apply?dry_run=true \
  -H "Authorization: Bearer $HELLOCRON_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"host": "web1", "params": {"backup_dir": "/srv/backups"}}'

The response lists what would be created and contains the cron file in cron; a dry run creates and changes no monitor and no event. dry_run accepts only true, false, 1 or 0 - a typo is a 422 with code dry_run_invalid, never a silent write. Drop ?dry_run=true to create the monitors. GET /api/v1/presets lists the available presets. Both are described in the OpenAPI spec.