An Automated Server Monitor

I realized some months ago, a major contributing factor to server downtime was "nobody noticing it was down". So I started checking the status of every server of every product I'm even vaguely responsible for every morning. This expanded to be quite a list over time and became a bit of a chore.

Previously, I'd write a script that hits every server API and presents summary statistics, then just run that every morning. I had a bunch of TM160 LED controller chips left over from the Catan game board though... So I figured I may as well use them. So I designed a board around the ESP8266 that sends a HTTP GET to an arbitrary list of servers, and checks whether it gets HTTP 200 or some other response.

The rounded corners and mounting holes make the board quite versatile.

How it Works

It stores an array of 9 servers -- one to check the Internet connection (I use CloudFlare) and 8 servers to monitor.

Once it connects to Wi-Fi, it turns the Wi-Fi LED on. Then it checks the first server (CloudFlare). If it gets HTTP 200, the Internet LED turns on.

After that it creates a HTTPS client, and hits the first server, processes the response, and closes the client. It repeats this for each server in the array. If The same HTTPS client just makes requests to all 9 servers, we end up with a lot of 'out of memory' crashes.

Even though it makes HTTPS requests, it does not check the security certificate. This mainly because I'm not really concerned about someone performing a MITM attack on a humble server monitor, but also because I'd have to manually load all the certificate files onto the device and checking them consumes memory.

The big patches of white silkscreen are there so you can write the server names in with whiteboard marker (if they change often), or permanent marker if they don't.

The device runs quite stable, gives me 8 fewer things to think about every morning, and is very cheap to produce. I'm quite happy with it!

You Know What Would be Better Though?

Since building this, I've started using business card sized circuits with rounded corners as a sort of standard format. I like it because it's a convenient format for sharing or shoving in a pocket. It's enough board surface to do most things I like to do. Also who doesn't like rounded rectangles? It looks good.

So I've redesigned this board to take advantage of the new format, and also add some small features.

Bit of a messy soldering job on the LEDs, I had put them on backward at first. Will clean it up at some point.

The board supports monitoring 6 things. The single strip of silkscreen between the left and right lights is a better use of board space -- if you're not using all of them, or one of the names is short, it frees up space for other text.

The mounting holes at the top are for M3 magnetic spacers. These are essentially hex spacers with magnets permanently stuck to them. This lets you stick the monitor to various surfaces. You can also bolt it to things normally of course. The USB plug in the bottom is for power, but also used to update firmware to the board.

Some new features on the verso as well!

I've also updated the electronics. Previously it used an ESP8266 D1 Mini, optimally the "pro" version. That worked fine, but cost around 5$. For just a little more (say 8$), I could get an ESP32S3 SuperMini which has a lot of advantages:

  1. A USB-C port
  2. Much more memory
  3. Much more processing power
  4. Better behaved I/O pins
  5. Similar form factor!

The new chip means that I can make the firmware do more advanced things without runningo ut of memory and crashing. I think Grafana integration for more sophisticated alert monitoring would be a good way to go, instead of just checking HTTP responses on health endpoints.

This way the colors can indicate different issues (not just up/down), and we can monitor more aspects of the same service, if that's what we want to do!