Let me get one thing out of the way. This isn’t a humblebrag. I know most people in the IT space probably have a good grasp at the underlying technologies behind the project I’m about to talk about, and to them it’s probably a no-big-deal “yeah I know about this” sort of thing. I also understand though, that for the average person hosting a project that’s going to see two or maybe four users in its life span, this is kind of overkill but in a good way. This all started when my brother asked me if I was able to host a persistent Minecraft server for him. I knew in the past that he had some trouble with some shady hosting providers not being able to keep Minecraft servers safe from attackers, and they also seemed to have a problem with uptime/reliability. This is something I wished to remedy.
Most people who’ve self-hosted a Minecraft server have probably done so via drag-and drop windows-isms using an executable .jar file they double-clicked on to run, and when they shut their system off for the night the server was no longer reachable and there was still likely a port open in their router just waiting to be scanned. This just isn’t my style. Enter my operating system of choice: Linux. I have a personal server (sort of a glorified NAS at the moment) that was already running Ubuntu Server that would be perfect for the task. The only issue is separation of services. I wanted the Minecraft server to be portable, to be able to transferred to another system easily. Installing the MC server on the host just wasn’t going to cut it. For this, I was going to need a VM. In theory, an LXC container would have worked as well with less overhead, but I wanted to keep the beginnings simple.
“Simple” would end up being a very relative term, however. I was used to libvirt’s graphical user interface for virtual machine management on Linux, and to be honest I wanted to spend the majority of my time on the project, not setting the infrastructure for the project up. Cockpit became a necessary addition to my setup.
Cockpit, for those not already aware, is a wonderful piece of software written by the team at Red Hat Enterprise Linux that gives your headless server a webpage administration panel, and with some addons and tweaks, also allows you to manage VMs running on the host. The first step was to set up cockpit with the correct addons and to add a virtual bridge to the host. I’d never created a non-NAT bridge before, so this was a nail-biting experience. If I misconfigured it, I’d have to go and plug a VGA monitor and keyboard into my server to do an offline diagnosis, which was something I really did not want to do. Luckily, after reading the docs for what felt like a thousand times, I issued the final command and with bated breath restarted the SSH session. It worked! Now my VMs would get IP addresses on the local area network as if they were actually there!
Unfortunately for me, the version of Minecraft that was requested was not of the vanilla kind. You see, there are all kinds of different versions of Minecraft nowadays, from the veritable list of versions directly released by Mojang to super-modified versions of the game that are released by the modding community that they call “modpacks.” It was one of these kinds of servers that was requested by my brother. The problem is that the mod community behind this game aren’t very good with Linux. Or documentation, for that matter. And hosting providers take advantage of this to offer “one-click solutions” to gamers who just want a no-questions-asked way to play with their friends on these modpacks. This led to some frustration. Where are the server jars? Do I just grab the mod files and put them in a vanilla server directory? No one seemed to have any answers. That is, until I caved in and finally installed the community’s curse-forge launcher (ex: I hate installing unneeded launchers with a passion. Just one of my quirks.) and saw that there was a button to just simply download the server pack needed. Cool.
So now I have a VM, the correct version of OpenJDK and the correct server files. We should be nearing the end of this adventure, right? If you know me personally, you’ll know that the answer to that question is a resounding “NO.”
So now we have a basic modified instance of a Minecraft server up and running in a virtual machine. This is nearing the part in which conventional tutorials would all agree that all that is left to do is open up port 25565 to the outside internet, look up your external IP, and tell your friends “Hey guys, log onto -insert your external IP address here- and let’s get playing!” There’s just one very big problem with this; A little but not-insignificant issue called security. You see, there are machines that people around the world set up to scan every port (or at least every popular one) on every active IP address on the internet. I know it sounds like a conspiracy theory and a herculean task, but it happens, daily. I used to run a few services that required exposed external ports and every time I’d go to read the logs it would just be a long red list of failed login attempts from IP addresses in China, Russia, South Korea, etc. It just so happens Minecraft ports are very popular to scan. Not only can they be a vector to get into a local area network, but they’re a big target for script-kiddies looking for a quick bit of fun by logging into said public Minecraft server and burning your creation to the ground (hope you had backups). Needless to say, I did not want to open a port on my LAN if I could help it. If you’re still with me, this is where it starts to get good.
I could’ve just stuck with IP whitelisting, or even user whitelisting for that matter. But A.) I didn’t want to have to get into the server command line every time a new player needed to be added, and B.) I just couldn’t bring myself to expose that port regardless. I really wanted to keep my IP free of attackers (or at least, as much as possible). I’d heard about proxy servers but the online tutorials on how to set up a reverse proxy all looked daunting and very confusing until a good friend of mine explained how they worked in a way that my dense mind could understand. Once understood though, I really got into the weeds with this project. My next step was to log into my DigitalOcean account and spin up a droplet (a small virtual private server hosted in the cloud that you pay per month for) to mess around with.
A brief aside, as I can already feel anyone reading this going “well then why don’t you just host the server in a VPS, you dummy?” and the answer to that is twofold. One, I have hardware, lots of very capable hardware. Why would I pay for a VPS when I can use the hardware I’ve already paid for to host a service for a family member or friend? The other part is scalability: It costs a small amount per month to host a small VPS (as of writing, $5/month for 2vCPUs and 512mb of ram). Minecraft however, is a very performant application and the more mods you add the more CPU power and RAM it needs. Expanding the VPS to keep up can get costly, fast.
Getting back on track, I started messing around with Wireguard. The goal was to get a persistent encrypted tunnel going between this VPS in the cloud and the Minecraft server. This would require a constantly-online connection node on my end to talk to the VPS. I could’ve put this configuration into a linux container, but I wanted to be able to use this connection even when the host on my end was down, so I got out the old faithful: my Raspberry Pi 3B+ and installed PiVPN, which is a set of scripts that helps you quickly and easily set up a personal VPN server. It took me around a day after work of reading and messing around with configurations to figure it out and get the config I was looking for, which was a two-way tunnel between the machine on my network and the VPS, where the VPS only used the VPN exclusively for traffic requesting the Minecraft server’s IP. All other traffic would be served using the VPS’ own network.
Now that I had a way into the network for the VPS to see the Minecraft server, it was time for packet forwarding. Minecraft is quite nice in that respect in that it only uses TCP ports for its multiplayer packets, meaning that it’s a perfect candidate for normal proxy servers to handle. There are multiple projects for specialty Minecraft proxies for the use case of hosting multiple Minecraft servers on the same LAN. For this application, I decided to go with a project called Gate. I only needed the proxy to be a proxy. I didn’t need all of the other fancy features Minecraft proxies have, and Gate was lightweight and simple. After following the instructions in the doc, assigning a SRV record to my own personal domain and configuring DNS in the management panel for my domain, I had a fully-proxied server up and running that was configured to allow traffic from my domain! Now when users went to log into my server, they’d type “mc.example.net” instead of an IP address! Not only was this kind of fancy, but it also had a knock-on effect of effectively blocking people from trying to log in directly via the IP of the droplet. Any time I tried to do this it would kick me as the proxy was expecting traffic from the domain. This smells a little of security through obscurity, as anyone with the domain can still log in, but this satisfied my own personal risk tolerance for the time being.
Everything is set up now. Players can log in via the domain, systems are up and running to make sure the connection is secure, everything is running smoothly. That is, until you need to stop or reboot the system. What if there’s a power outage when I’m not home? What if I need to reboot the host and I only have a small amount of time? I’d have to log into the VM (at least that was already configured to start on the host bootup process) and manually start the Minecraft server in a tmux session. Hardly an elegant or simple solution. No, it was time to make Minecraft a Systemd service. Systemd is an init system for the Linux operating system. It’s responsible for the start up and maintaining of software necessary for the OS to run. You can call Systemd via the command line to start up and shut down applications at will, and query the status of said applications. You can also create scripts for Systemd to run at startup when you need a service to run at boot. This is exactly what I did, writing the Minecraft boot arguments into a script and setting that script to initialize on boot. Now it will start up with the system, but what about shutting down? Systemd has a “kill” ping that by default causes the Minecraft server to gracefully shut off when the OS is shutting off, but to the players in the server, it means the Minecraft world will go dark suddenly and without warning. That’s not very user-friendly. What if I accidentally sent the shut down command to the VM one day while not in contact with anyone? I wouldn’t want them to lose progress or be stuck in an area they didn’t want to be in when the server came back up.
To remedy this, I’d need the VM to be able to issue admin commands into the Minecraft server while it was running. I’d need something called an RCON client. RCON is a piece of software that hooks into many multiplayer game servers, or at least the ones that support it, and enables admins to remotely manage the gameplay side of things without having to be logged into the game itself. I had to pull the client from github and compile it from source, then move the compiled binary into /usr/bin to enable it to be called by the system without having to reference a directory. After that I wrote a shut-down script for systemd that changed the default shutdown timeout to 5 minutes, and each minute it would put in the game chat “The server will shut down for maintenance in X minutes” and then on the last tick, the server would shut off.
That’s right, I’m still not sure I’m done with this project. I know in order to consider it finished I still have to come up with solutions for backing the world up. I’d like to have a 1-2-3 solution where the server directory is zipped up and backed up to another directory in the VM, then to another computer on my LAN, and then finally to some cloud storage, but I haven’t decided if it’s going to be a directory backup or a vm snapshot backup, and I haven’t gotten the methodology down for that just yet. Again, I know this is overkill, but it’s a nice thing to learn regardless. If you’ve stuck with me this far, thanks for reading about my silly shenanigans, and I hope to be working on another project of this caliber really soon!