Game Servers (2013 - 2015)

In 2013, I started a public Garry's Mod game server. I was originally using a $5/month Digital Ocean VPS for the hosting. It grew quickly and became one of the most popular public Garry's Mod servers worldwide. It was my most profitable project until 2020 (when BestOfGleam overtook).

This was my first public Linux server (of any kind) and I have been hosting various public services since.

The game server itself was actually the least interesting component from a technical viewpoint. This page will talk about all the “helpers” and sub-services which were required to host a thriving community.

Garry's Mod requires additional files to be sent to players during the loading phase (and whenever the map changes; which happened every 30 minutes). To prevent the game server becoming overloaded Garry's Mod will severely rate limit the upload speed to very slow speeds. To overcome this limitation, a second server needs to be provisioned which can distribute the files over HTTP(S).

Originally I used a second Digital Ocean VPS to handle file distribution, but it became overwhelmed as the game server gained popularity. Specifically, 30 players all downloading large files simultaneously during a map change would consume the 1Gbps link which Digital Ocean gave me. I could have ignored this problem because it didn't prevent anything from working, it simply caused some players to join the game slower. I decided to fix it anyway because I wanted everyone to have a good experience.

I found another VPS provider who was offering 10Gbps servers at the time. I provisioned 2 of those and used DNS load balancing to split the traffic “equally” between them. Unfortunately, the majority of my players were using the same ISP and were physically located close to each other. The average player was using their ISP DNS provider which was caching the IP address and causing the round-robin DNS to have an approximately 90:10% split. eg, one server would be handling most of the traffic alone.

The new servers also couldn't handle 30 players simultaneously downloading assets due to slow spinning disks. I could have improved the load balancing by purchasing a hardware load balancer with my new provider. I also could have switched to SSD storage with the new servers, but the bill was already getting far too high. I opted to keep just 1x 10Gbps server and sign up for KeyCDN who distributed the files to my players; a single 10Gbps server was the source of the files which KeyCDN would access.

I had plenty of monitoring on my own servers but I was curious about what other Garry's Mod servers were doing. Specifically, I wanted to know which maps were the most popular. I also wanted to compare my server trends with the trends of every other server (eg, is only my server unpopular right now, or everyones?)

I signed up for a dedicated server with Kimsufi which had a 2TB disk so I could store lots of metrics. I wrote a Python script which contacted Steam's servers and asked for a list of all public Garry's Mod servers; this would run regularly and the results were saved into a database. I wrote a second script which would loop through every public server online and request information from them directly. This information included their current map, list of players, and some more information. This information was also saved into the database and was generating about 1GB of database entries a day.

Due to having information about every Garry's Mod server in the world at high resolution, I could make very good estimates to what was happening with players globally. Occasionally, I would see large drops in the overall player count worldwide caused by ISP outages or various events happening in the real world. This made it much easier to determine I had lost players due to something big happening outside of my control and not because I had done something wrong.

With the data I was collecting, I was able to generate a full list of every map ever played. I then wrote a script to download each of those maps from public sources. Eventually I had a large number of downloaded maps. I made a web page which allowed people to view a list of maps and download any of them, I also made a compressed archive available which included them all. Unfortunately, a map-maker took offence and threatened me with legal action because their private map was available in the archive. I took down the archive and the ability to download maps after a few weeks of being up. I found a list of the maps contained within the archive on Pastebin but the links will no longer work. New maps were discovered and downloaded by my system after this archive was made and I seem to recall it exceeded 1000 maps.

I was also able to monitor which servers any player was visiting, even if their Steam visibility was set to “appear offline”; this is because every Garry's Mod server in the world was being probed by my servers and returning a list of their players. This list didn't include an ID of the player (only their display name) but anybody with a sufficiently unique name could be tracked easily between servers. I wrote a script which found unique usernames who were swapping servers very regularly and would warn me if they joined my server; these players were often trolls getting banned from every server they visited within minutes. Having a warning about them beforehand was surprisingly useful because I could keep a close eye on them and ban them much quicker when/if they broke the rules.

Public game servers attract trolls. For some reason some of the trolls have high budgets meaning that they can evade bans quite easily. For example, I once banned somebody who re-joined the server with a total of about 25 different accounts and each of those accounts owned a paid copy of the game. I'm not sure if they stole the accounts or just loved trolling so much they bought the game 25 times. I generally consider IP address bans to be a bad thing for multiple reasons but many trolls had access to lots of IPs so that wouldn't be the solution anyway.

To combat exactly this kind of ban evasion, I wrote a plugin for Garry's Mod which ran on the PC of every player. It was very simple and consisted of about 10 lines in total. When somebody joined the server, a file would be created within the Garry's Mod data directory on their PC which contained a UUID. This UUID would be sent to my server upon each connection and would never change.

I created a server-side script which would store the generated UUIDs against specific accounts and monitor if that UUID got spotted on another account. Each UUID would be linked to a SteamID, and each SteamID would be linked to one or more UUIDs. Eventually, it created a “web” joining multiple computers and accounts together giving a good insight into players sharing computers and players with multiple accounts. If a player got banned, this service would look up their UUID, collate it with their secondary accounts, and ban them all at once. This was an incredibly powerful tool which worked very well.

Looking back, I can definitely see some ethical and privacy issues with the system. If I ever get into game server hosting again, I won't be re-making this particular feature despite knowing how powerful it is.

I would occasionally look at the UUIDs generated to ensure the system was working correctly and quickly made an interesting observation: many of the HTTP conversations were intercepted by malware and injected with malicious code. I could see this because the UUID was sent to the player via HTTP and then sent back via HTTP to my server. If I remember correctly (I no longer have the data), about 10% of my players had their HTTP sessions tampered with by malware or their ISP. This, surprisingly, didn't affect the performance of the system because the injected content simply became part of the UUID.

The particular Garry's Mod gamemode I was running was a first-person shooter and thanks to other FPS games, many players were interested in a “kill cam” (eg, a video showing how another player just killed them). I wanted to implement this feature partially because it would make my players happy, and partly because it could be used as evidence against cheaters.

I discovered a “take screenshot” function within the Garry's Mod plugin API so I wrote a plugin which would take a screenshot at the moment any player killed another player. This screenshot would then be sent to my server and stored for future use. This feature was quite popular; whenever a player got a good kill, they could access a screenshot of it on my website and use it for bragging rights. Unfortunately, recording videos of events seemed too technically difficult to implement.

One day, when looking through some of the screenshot images, I discovered a screenshot which also included the Steam Overlay and someone's private Steam conversation within. I updated the plugin to make this significantly less likely to happen again and added privacy controls to the screenshots so they wouldn't leak to people if it happened again.

I wanted to investigate the matter some more so I set up a private test server with a plugin that would take a screenshot each second and send it back to me. Sure enough, I was receiving screenshots showing the Steam Overlay and the private messages contained within whenever I opened it during testing. It would be trivial for a server owner to implement this on a public server and monitor whatever you do inside your Steam overlay (unless the Garry's Mod developers have fixed this flaw).

A huge issue with Garry's Mod servers is the lack of proper DNS support. Specifically, when somebody connects to a server, their game client only remembers the server IP and throws any DNS information away. This means that if you ever change IP address you will lose all of your players; of course your most dedicated players will look up your new IP, but many will not.

I migrated from DigitalOcean to another provider after a few months of operation and lost a big chunk of players. When I needed to migrate a third time, I came up with a solution to the problem.

It was/is possible to set up a second Garry's Mod server with a new IP and configure IPTables on the old server to act as a transparant proxy to the new IP. This means that anybody who connects to the original server will actually end up playing on the new server via the original.

In order for the original server to still appear in the server browser, it must be running Garry's Mod server as normal (this sends pings to the Steam “Master Server” service and keeps it considered alive). For that reason, the networking must still work for Steam communication so the IPTables rules need to be well written.

Garry's Mod doesn't even realise anything is unusual if a player connects to a proxy server, so we need to perform our next trick..

By writing a Garry's Mod plugin which runs on the PCs of the players, I could detect the IP address of the server. If the IP matched the old server, the plugin could then issue a reconnect command but to the new IP address instead. This caused the new IP address to appear in the “recently used servers” list. Without this proxy and redirect chain only the old IP would appear in the list and would vanish after the server migration was completed thus they would not find my server again without looking it up.

For a short while, the server would be “duplicated” in their list of servers; the old IP and new IP. This caused very mild confusion for some players but it was far easier and better overall than messaging everybody the new IP.

The proxy part isn't strictly needed, you could simply redirect all players who join server 1 to server 2 without the proxy. This has a huge caveat: the original server will always appear to be empty (because everybody gets redirected away from it). The majority of players will not join an empty server, and also the Garry's Mod client will show empty servers further down the list in the browser making it harder to find.

Upon realising how easy it was to manipulate the traffic, I tried spoofing the server responses using my test server. Sure enough, you could quite easily pretend your server was full of players despite being empty (players are more likely to join a full server instead of an empty one). I believe this flaw has since been fixed. This is because Garry's Mod server browsers probe individual servers to get information about the server status instead of some kind of secured central source of data.

Another nice feature I implemented was gameplay statistics. Whenever a player got a kill or did something notable inside the game, a database entry would be created. I wrote a website in PHP which allowed you to browse the data in this database with a clean interface.

Some of the stats pages have been saved by The Wayback Machine, so I can include a screenshot of my stats page:

Multiple tabs of stats were available when the system was online, but The Wayback Machine didn't archive those.

Here is a screenshot of the stats homepage: