Self-Hosting Shenanigans

Automation And You - plus a cool hack for Substackers

I’ve been thinking recently about self-hosting - running your own servers and services to suit your needs and facilitate your personal and professional life. In an era where technology creeps into our homes from every angle (even the humble doorbell is now an internet-aware communications device!), it seems to me that seeking to coalesce and control this influx of binary butlers, silicon servants and artificial assistants on our own terms would be A Smart Thing To Do.

Whether we want to control and monitor devices in the home, back up our photos and videos, or synchronise notes, recipes, and documents between devices, our corporate overlords are keen to offer us ‘cloud’ solutions by the bucketload: naturally incompatible with their competitors and usually expecting you to forget any privacy concerns you might have about uploading your entire personal life to what is, and will forever be, just somebody else’s computer.

With the low barrier to entry and enticingly smooth user experience, most people are happy to tap the ‘Send to Tim Cook’ button and forget all about it.

Of course, privacy concerns aren’t the only factor which might push you towards self-hosting. Perhaps you want to take advantage of the choice and freedom to try any number of software solutions and services before settling on whatever suits you best, or maybe you want the ability to ensure you can still access what you need if the power goes out or the network goes down.

For those of us who enjoy tinkering with tech, self-hosting also provides an opportunity to try out new ideas, stitch things together in unexpected ways, and have fun without constantly having to worry about an endlessly growing list of service subscriptions breaking the bank.

In this post, I’ll walk you through a little of my setup - and read to the end for a little Substack-related surprise!

Taking Control

The first thing any self-respecting self-hoster needs to do is research which hardware is available to suit their requirements. The second thing they need to do is forget everything else and buy a Raspberry Pi instead - since you’ll be hard-pressed to find a more suitable and affordable piece of kit to start your self-hosting adventure. Of course, if you have an old laptop lying around then this could also serve the same purpose - but I favour the Pi for its tiny footprint (mine sits on top of a speaker) and low running costs. If you do opt for the Pi, it’s worth investing in a high-capacity microSD card, especially if you’re planning on using it for file backups.

Tiny machine, big ambitions

As an aside - if you’re not using a Raspberry Pi for your self-hosting experiment, I’d still highly recommend using Linux as your operating system. Though you’ll likely be able to use a Windows or Mac machine, most software you’re likely to want to play around with will be expecting a Linux machine.

Once you have your server machine ready, you’ll certainly want to give it a fixed local IP address. Your home WiFi router should allow you to assign a fixed IP to a specific device on your network, even if your external IP changes (most consumer ISPs don’t provide fixed IPs as standard - and acquiring one usually comes at an extra cost, if they’ll provide one at all).

A fixed local IP means you will always be able to connect to your self-hosted services at the same address. This is very important, so make this one of the first things you do!

Docker

Docker is a container system which allows you to quickly and easily deploy software in a standardised way. It enables software developers to provide ready-to-run versions of their software and all of its dependencies to their users, whilst greatly reducing the likelihood that an individual’s particular setup will interfere with things. This is ideal for software which provides networked services - since if you have a lot of users or your hardware changes often, you want to be able to adapt and scale with minimal fuss.

Most things you’re likely to want to play with initially will provide Docker images, though the specifics vary. You may find:

• Docker images on Docker Hub

  • This is the ideal scenario - allowing you to download and run the software with no fuss

• A Dockerfile in a developer’s GitHub repository

  • A little more involved - the Dockerfile contains instructions for Docker to build the required image, which can then be run as though it were pulled from Docker Hub

• A docker-compose.yml file

  • For more complicated software, which uses third-party dependencies e.g. a database system, a docker-compose file tells Docker which different pieces of software to run and how to connect them together. For a lot of self-hosting software, you’ll want to either find or write a docker-compose file to keep things nice and tidy.

Since the whole point of self-hosting is to run services within your home network, there can be a little network-related fiddling to get things up and running: mostly keeping track of which ports different services are listening on, but nothing too complicated. Docker will allow you to configure things easily - but you’ll need to refer to the setup guide of each piece of software to know what you need to do.

You can install Docker through your system’s package manager - though things will likely be easier if you use the terminal and simply run the following command (assuming you’re on a Raspberry Pi):

sudo apt-get install docker docker-compose

Nextcloud

Nextcloud is a file sync tool, collaboration tool, chat system, and office suite all rolled into one. It’s certainly not without its quirks - but it’s highly extensible and provides a ton of functionality which will get you started with the basics of self-hosting. It’s not the quickest software on the planet when running on a Raspberry Pi, but it does the job and is ridiculously simple to set up given all that it provides.

If you want to try Nextcloud, copy the text below into a docker-compose.yml file and change the passwords / usernames wherever you see something like ‘enter_password_here’. Even though you’re self-hosting and not (currently) exposing any of this to the wider internet, it’s still worth sticking to good security practices and picking a good, strong password.

version: '2'

volumes:
  nextcloud:
  nextcloud_db:

services:
  db:
    image: yobasystems/alpine-mariadb:latest
    restart: always
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    volumes:
      - nextcloud_db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=enter_root_password_here
      - MYSQL_PASSWORD=enter_user_password_here
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=enter_username_here

  app:
    image: nextcloud
    restart: always
    ports:
      - 8080:80
    links:
      - db
    volumes:
      - nextcloud:/var/www/html
    environment:
      - MYSQL_PASSWORD=enter_user_password_here
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=enter_username_here
      - MYSQL_HOST=db

Next, you’ll want to open the terminal, change to the directory where you’ve saved the docker-compose.yml file, and type

docker-compose up -d

You’ll then be able to open a browser, visit http://localhost:8080 and start playing around with your Nextcloud installation.

This is a very basic setup but will give you a taste of what Nextcloud offers. In addition to the base software, Nextcloud also has an integrated app store which allows you to add extra features; many of which are entirely free though some do require paying for a third-party service. Of the free offerings, one of my favourites is Cookbook - which allows you to paste a link to a recipe you’ve found and produces easy-to-follow instructions and timers:

This recipe is actually amazing

For a more up-to-date and in-depth guide to getting Nextcloud set up so you can access it from anywhere, check this guide from HowToGeek - though wherever you see a reference to Apache (a web server), I would recommend you instead use Caddy - which is much easier to configure.

Nextcloud also provides a mobile app so you can sync things directly from your phone and access your shared documents on the go.

Home Assistant

Home Assistant is where self-hosting really starts to shine. At its core, HA is intended to provide a central hub from which you can manage any of your smart devices such as sockets, lamps, switches, and essentially anything else with a network connection.

It can be a little opaque when you first use it; working out the difference between devices, entities, and helpers requires a bit of experimentation and reading - but once you’ve understood the basics, you’ll find that beneath its somewhat clunky surface, it’s actually a pretty elegant system which allows you to manage your home for any situation.

On top of its direct support for devices, there are also dozens of integrations with external services, meaning you’re able to do things like:

• Adjust the brightness of your lights / control your curtains or shade based on the elevation of the sun (yes, really)

• Turn on your smart TV when you walk into your living room

• Monitor and control your energy usage based on your bank balance

There’s also a vibrant community sharing tips and tricks on how to get the most out of your setup. Check out the Community Blueprints for inspiration!

Just like Nextcloud, Home Assistant is simple to set up with Docker Compose:

version: '3'
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - /PATH_TO_YOUR_CONFIG:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    network_mode: host

You can find more information about installation and configuration in the official Raspberry Pi guide.

A simple lamp brightness control in the Home Assistant dashboard

You’re also able to set up your own ‘helper’ entities which you can use to keep track of different values provided by other software or services - though this will require a bit more in-depth fiddling to become useful.

Home Assistant is just one of several available smart-home hubs, but it’s one I’ve stuck with for a while now. The flexibility it provides, and the community of users are fantastic - and combined will allow you to chain together devices, software and ‘the rest of the world’ in ways no paid-for service is currently able to offer. If you want to be the master of your own domain, then Home Assistant is for you!

As with Nexctloud, Home Assistant also provides iOS and Android apps for when you’re ready to expose your self-hosted installation to the wider internet.

n8n

A recent discovery for me, but something I’m definitely excited to play around with more, n8n (Which is, by the way, a terrible name) is software which allows you to build automation workflows using a dizzying number of services. What this means, roughly, is you can define a workflow using simple ‘if-this-then-do-that’ logic, using a simple drag-and-drop interface to connect nodes together.

Every day, pull down an RSS feed and transform it into an email-friendly newsletter, then send it

You can create workflows using a wide variety of trigger events - actions which occur on external services, or regular, scheduled tasks. There really is no limit to how complex you can get here - and if the integrations included with the base system don’t provide what you’re looking for, then there are a large number of community additions which you can install to provide extra functionality. It’s also relatively simple to write your own if you’re so inclined!

I haven’t yet explored everything n8n has to offer, but what I’ve seen so far has me excited to play around and create a digital Rube Goldberg machine of epic proportions.

n8n is probably only useful if you’re willing to expose your self-hosted services to the wider world, so refer to the instructions for a valid docker-compose setup. In my case, I removed the dependency on Traefik since I just use Caddy to manage proxying.

A Treat for Substack

In all my excitement at discovering n8n, I decided to see what I could do when it came to Substack.

As I’ve mentioned elsewhere, the fact that Substack doesn’t offer an API or really any integration options at all is a source of great frustration to me, but ever the optimist, I decided I’d get something working, by hook or by crook.

To that end, I’ve created a little system which does the following:

• Allows me to define subscriber goals in Home Assistant

• Monitors for ‘New Subscriber’ emails from Substack in n8n

• When a new person subscribes, increments the counter in Home Assistant, which then sets the brightness of a lamp to the percentage of my subscriber goal, and then starts a playlist on Spotify to celebrate!

Here’s what the n8n workflow looks like:

Look on my Works, ye Mighty, and despair!

And here, in all its glory, is my ‘Substack Subscriber Goals’ dashboard in Home Assistant:

Yes, I am a gigantic nerd

In the above image, you’ll see an input field which lets me set my current subscriber goal. If I set it to say, 1000 subscribers, then my ‘Subscriber Goal Progress’ immediately drops down to 3%, which also sets my lamp brightness down to 3%, and my meagre home-office setup becomes decidedly more gloomy.

I can only get any work done if you subscribe!

So, if you enjoyed this post, and want to see more like this, then help me keep the lights on (literally), share & subscribe!

If you’ve got your own self-hosting setup, or have any questions about mine, then feel free to comment below and let’s see what we can do to take back control of our tech!