Valheim dedicated servers break sometimes. Whether it's a change to where the world files are kept, or what the arguments are that the game executable will accept, if you host your own server, you'll inevitably deal with a server that won't start or won't stay on. Very frequently, you'll see lots of lines in the log about a NullReferenceException.

Haptic recently had an outage that caused our Valheim servers to not start for an extended period of time, which is a rare event for us. I've spent a lot of time perfecting our game hosting systems to avoid this specific situation, but even all that effort isn't enough to avoid the occasional downtime. In our case, we ran into a variation of the NullReferenceException, which has many causes.

This blog post is all about our experience with fixing this problem. My hope is that it helps someone else who encounters it when running their own server, but it's also meant to give our users a sneak peak into the work behind the scenes.

Haptic uses a technique called "containerization" to run our game servers. Specifically, we use a technology called Docker. There are a few reasons it is a good idea to run a game server in a container, but mostly we do it because it makes it easy to provide every game server with the same blank slate every time it starts up.

Usually, this makes the servers more stable, but it requires us to automate a bit of set up that is shared across all servers. This is done in what is called a Dockerfile. Luckily for us, Iron Gate provides a Dockerfile sample in the dedicated server bundle! To view it, go into the docker/ directory in the server bundle and Dockerfile with a text editor.

You might have noticed there is another file in that folder, called steamclient.so. What is that doing there, you ask? Well, it's there because the Dockerfile needs it. On line 10 in the Dockerfile, Iron Gate is recommending that we copy that file into /usr/lib, which is a place that the Linux operating system keeps libraries needed to run its applications. Seems sensible enough!

When I wrote the code that automatically updates Valheim server files, I took Iron Gate's recommendation and began using this steamclient.so file in our Docker container. Things ran great, we had no problems, and I forgot about it as we moved on to solving other problems.

If you are like me and have been using this steamclient.so file in your Dockerized Valheim server at home, you may have been nodding along up to this point. If you're like me, then you also have had broken Valheim servers since around mid-November.

The problem for us wound up being this little steamclient.so file. An .so file is a "shared object" file. In Steam's case, they appear to be using an ELF file format for their shared object file. When I dug around in our Valheim server logs, I discovered that we were consistently getting one brief message about this file: wrong ELF class: ELFCLASS32.

If you see this in your logs, it is a good bet that you are having the same problem as we were having, and it was quite a chase to figure out exactly what was required for the servers to start normally.

Of course, the wrong ELF class was not the first thing that investigated. We tried a couple of different things at first when the Valheim servers stopped starting, but none of them changed the behavior of the servers. The Valheim server logs are very noisy and there are a lot of errors that don't need to be fixed and messages that can be ignored. The challenge is to find the one that you can't ignore and to find it fast.

Eventually, after trying things related to the new PlayFab integration, we tried swapping out the copy of steamclient.so that we were using with a different copy that is also distributed with the server bundle.

Originally, we were using docker/steamclient.so. I noticed that this file hadn't changed for several months, but the one at linux64/steamclient.so had changed relatively recently. Just because a file has the same name doesn't mean that the content and purpose is the same, but I figured it was worth a try.

To try it out, I manually swapped the files on the server and then hit the start button. To my everlasting relief, it worked! Finally, a working Valheim server!

It's still not clear to me why the docker/steamclient.so file was the cause of this issue for us. My best guess is that one of the Iron Gate engineers created the docker/ folder and wrote that Dockerfile by hand to do a nice thing for the folks out there who would want to Dockerize their Valheim servers and never expected the underlying Valheim code to change so much that the shared objects had to be updated.

This makes the most sense to me. As an infrastructure engineer, I find that a lot of bugs result from the mixing of automated and un-automated processses. The Docker folder never got the rigorous treatment that the rest of the server bundle got and it was forgotten about, and when a big change happened it was never updated.

This means unfortunately that if you want to continue to use Docker to run your Valheim dedicated server, you'll need to write your own Dockerfile and manually move the linux64/steamclient.so file into your container's /usr/lib directory to make sure that it is accessible. This is what we ended up doing and we have had no problems with it so far.

I hope this article has been helpful to someone out there, or at least has been interesting for some of our technically-inclined users. If you have run into some problems hosting a dedicated server or are curious about what Haptic does to keep its servers running smoothly, come join our Discord!

While I might not share our deepest and darkest dedicated server lore (I have to preserve some Haptic mystique), we've had to solve lots of small problems on the way to building our platform and I'd be more than happy to share some tricks of the trade. I'm meex in our Discord and you can tag me into any conversation that requires a higher geek quotient. Hope to see you there!