Dawn of the list
Setti server querier was born sometime in 2005. The first version listed handful of servers and allowed server admins to submit their own servers on the list. The main cause for the service was to help people keep track on servers allowing all players, ie. public, listed on different web sites.
Obviously, after a while, some servers went down and the list contained dead servers. The solution in all its simplicity was to add “Report broken” link next to the server address. The link made it possible for users to say the server is down. This system was active for few weeks. Not surprisingly the server list was very static and not very useful - still slightly more useful than lists on dozen of forums. After some time, quite expectedly, web crawlers started to terrorize report links and all servers gained “broken” points.
Enter QStat
QStat is program which is able to query many kinds of server types. It is possible to give list of servers as parameter and QStat will query all of them. It’s even possible to output the result as customized web page. This is how things were in the early days of Setti server list. The main page of Setti CSS server was generated every 10 minutes by QStat. All changes on the main page had to be done in the QStat template. Server querying was automatic, so from this point on Setti server list has been somewhat unique in its kind.
Old Setti CSS server main page. Generated by QStat.
Around that time the system started to store historical data too. It was started partly due to difficulties recognizing servers being down for only hour or two from servers being down for two weeks. The historical data allowed generation of very accurate server statistics.
At this point the server query system was functioning quite effortlessly. It was easy to submit servers ( “in the form at the bottom of the page” ) and dead servers were removed automatically. The system scanned hundreds of servers all over the world in minutes. The speed was sufficient for the time being. However, there were still problems keeping the list valid. Not all submitted servers were public. They had to be removed from the list by adding them manually in excluded servers list. At the time it wasn’t known how to automatically verify whether a server is accepting all players or not.
Enter client emulation
The biggest revolution in server querying happened on October 2005. Then bug.meh.not revealed the secrets of client-server-client communication when the client joins the server. The procedure allowed anyone to check whether a server will accept all players or not. Not long after his release the client emulation was reproduced for much greater use: verifying servers in Setti servers list. The effect of being able to tell a server from a public server is the most significant thing for server lists.
UPDATED RELEASE: Source/HL Cracked Server Query Tool
[ Posted: Monday 29th August 2005 11:06 pm ]
[ Posted: Sunday 23rd October 2005 01:09 pm ]
[DEV]Cracked Server Finder:SourceCode
[ Posted: Sunday 30th October 2005 01:55 pm ]
The posts at cs.rin.ru are also archived here.
Historical data
By this time the QStat based Setti server querier was to be decommissioned. New Perl based system featuring server status checking was to replace the old, somewhat messy system. The speed of server querying was slightly disappointing, due to lack of threads, but as there weren’t too many servers there wasn’t need for better system. Just as before.
Historical data for all servers had grown much, but all statistics based on the thousand log files were working good. There were slight problems because of changed syntax along the way. The very first historical data entry looked like this.
1128021324,fal.setti.info:27015,24/24,de_dust2,42,cstrike fal.setti.info CSS server
The line contains unixtime, server address, players, map name, ping, server name with server type. When the query system changed to Perl program, it was good time to add some more information which could be requested from the server.
1146333203,ru.setti.info:27015,26/25,de_dust2, 0/d/0/l/1.0.0.28,Setti CSS server
The line is almost the same, but ping has been replaced with “0/d/0/l/1.0.0.28″. Cleverly enough the delimiter is “/” and not “,”, so all statistics scripts still worked without changes. The first entry is ping, fixed to 0 for being no use, ‘d’ for dedicated, 0 for no VAC, ‘l’ for linux and 1.0.0.28 as game version. After months and several small improvements the syntax for historical server data reached its final form.
1153264488,ru.setti.info:27015,25/24,de_dust2,0/d/0/l/1.0.0.28/cstrikes/240/1,Setti CSS server
There is additional “/cstrikes/240/1″, which means ‘cstrike’ for game directory with appended ’s’ to indicate HL2/Source, 240 as AppID and 1 for number of bots.
The statistics files take whopping 2.7 GB disk space in 10000 files, ie. that many servers. Still, all stats for great number of servers were rendered in split second. Global server statistics and such were generated from the overwhelming number of log files by nifty handcrafted Perl programs. Sweet.
New, improved & database
The new database system is much faster than the old file based system. It’s because of threads. The latest file based system had threads, but they weren’t used as efficiently. The current database based server querier scans 3000 servers in 120 seconds, which is 25 servers per second. It’s good enough. Just as before, no need for better.
Some compromises had to be done with the database. There are three historical database tables: daily, weekly and monthly. Daily table contains data in ten minute intervals for one week, week table contains data in one hour intervals for month and monthly table contains data in four hour intervals infinitely - don’t mind the names. The file based queries stores all historical data in ten minute intervals. All! It means that in file based system it’s possible to see back in time in great accuracy. For example it’s possible to look at New Year on 2006 and see all aforementioned server statistics in ten minute intervals for all servers. The new system does only 4 hour accuracy. Good enough.
Additionally the new server query system scans a lot wider span of servers than all the other server query systems combined. The old system started to cough around 700 servers. The new could do 10000, but it seems that 3000 is all there is.
For the first time in the history of Setti server queriers the system in use is on solid ground. Confidence in the system is higher than ever. The system is easy to maintain and the database makes it easy to query different kind of results.
It’s fast. It’s reliable. It’s the product of necessity.
2009-10-02
