Kernel update
EchO found out couple weeks ago that the default Linux kernel is not completely suitable for running high-FPS CS:S server. The solution would be to compile new kernel with few settings changed. Here’s how it went.
The reference to everything was post at steampowered.com. Most likely the author has not figured those things himself but copied them somewhere else, but it’s not important here.
Compiling a new kernel is relatively simple task. All you need to do is get kernel sources, change the settings you want, compile, install and reboot the system.
Anyway, most of the time something goes wrong. It took three attempts to get it right at Setti.
The first (failed) attempt was the easy way. Just recompile the current kernel without recompiling all modules. The idea was to use the modules of the kernel already installed. Everything should work because the kernel version is the same and the modules have been already compiled and being used by the current kernel. This way the possibility of doing something wrong would be minimized.
The kernel compiled fine. Everything was the same as the current system except the kernel. The system is running on SATA hard disk, which requires special drivers before it can be read. That’s why there’s also so called “initrd-image”, which gets loaded first in the boot process, which then loads the real kernel from the SATA drive. The other option would have been to compile SATA drivers inside the kernel, but the principle idea was to change as little as possible to minimize errors.
The system didn’t boot. Rescue-system saved the day. It was easy task to login to the rescue-system, change the old kernel back to the default kernel and reboot. The next attempt would be in the next day with the latest 2.6.20.7 kernel.
The second (failed) attempt was slightly better attempt. This time it wouldn’t matter what the old system was, because all new kernel modules must be compiled for the new kernel. After the first attempt there wasn’t any additional information why the system didn’t boot. It should have worked, but it didn’t. The problem might have been the kernel unable to load the old kernel modules, or it could have been the new kernel was unable to start with the old initrd-image.
The second attempt was actually in two parts. At the first part the server didn’t boot, just like in the first attempt. The rescue-system saved the day again. Now, instead of reverting back to the old kernel, new initrd-image was created for the new kernel. Reboot. Didn’t work. The reason why it didn’t boot was maybe because the new initrd-image wasn’t actually used because bootloader (lilo) configs were left the same. Well, then to the rescue-system again, revert to the old kernel, launch up the CS:S server and other applications and pretend as nothing happend :).
The third (successful) attempt went like it was planned. The kernel recompiled with SATA and reiserfs drivers inside. All the modules recompiled again just to be sure. Bootloader configs triple-checked. Reboot. Then one minute silence… and then “Reply from 213.239.209.221: bytes=32 time=52ms”. Success!
Now the current running kernel is version 2.6.20.7 with 1000 HZ timer frequency. The timer frequency makes sure the CS:S server is as responsive as possible, making the gameplay smoother than ever. The CS:S server tickrate is 66 tick. Players have already reported better hit registration and there doesn’t seem to be the old kind of laggyness every now and then.
The system is now as perfect as can be in the point of view of the kernel. There are few unnecessary drivers in the kernel, but they are not slowing down the system. The kernel scheduler, which tells which application gets how much CPU time, is optimized for interactive low-latency high-priority systems. The kernel, of course, optimized for AMD Athlon64 processors - instead of the “general x86-64″ optimizations with the default kernel. You don’t get it faster than this :).
UPDATE 2007-04-23
The kernel has been updated to 2.6.21-rc6-rt0. The version numbers says it’s normal linux development kernel version, which has been patched with real-time pre-emption patch. In addition to just bringing real-time support to the kernel, the patch also includes several other improvements to the default kernel. In fact, the real-time feature is not as critical as the other features - it’s not even necessary, so it’s not enabled.
To make sure the kernel does not do sloppy job, there are now statistics of FPS on the server (reference). The FPS should be steady around 300. Any fluctuations in the graph indicate that the CPU is not wholly utilized.
As opposed to the original reference on how the kernel should be patched, the kernel is now “tickless” (CONFIG_NO_HZ=y), although patched to “USER_HZ 1000″ in param.h just to be sure.
Currently the server seems to be able to do it’s job, so there’s no need to tweak the kernel forth. Despite that, it’d be still nice to test how the kernel would perform by enabling real-time support. Between the previous 2.6.20 kernel and the current, the real-time pre-emptive kernel was shortly tested, but then the param.h file wasn’t patched to 1000 HZ - although logically thinking it shouldn’t matter, because the “tickless” feature makes it obsolete. Another nice test would be to try non-tickless (ie. normal) real-time pre-emptive kernel with 1000 HZ param.h patch. The reference at steampowered.com tells to use desktop level pre-emption, which is one level less interactive from the real-time pre-emption.
The best source of information over anything has been valve mailinglist archive for linux server.
UPDATE 2007-05-04
In fact the kernel has been running at 300 HZ since 2007-04-25.
The 1000 HZ kernel with all possible configurations didn’t make it any better in the end, maybe even worse. As a last resort the kernel tickrate was lowered to 300 HZ, in case that 1000 HZ would have been too much to handle.
There’s no significant difference, if at all, so it’s all the same which kernel is running. It’s 300 HZ now, but the gaming experience is pretty much the same lagging as it has been since the day it started lagging - presumably the day when valve updated on 2007-03-13.