Free gameserver to whoever can help

(Ohmz) #1

The title is true, I will give a free gameserver for 1 month to anyone who can solve this problem I’ve been having with ET servers under Windows. You’ll be able to pick from our 4 datacenter locations for your free server (CA, TX, KS, or DC) and it can be up to 20 players and a public server.

Keep in mind, this is not likely to be an easy fix, as I am an advanced level user myself - I’ve been running gameservers since 1997. But you never know, it always helps to have a fresh set of eyes on a problem. Here goes…

We’ve been hosting ET under Linux since ET was released, and we’ve had no problems at all with the Linux servers. However, we recently started hosting ET on Windows 2003 servers as well. For the most part, they are fine, but every once in awhile the server resets. Unfortunately, I don’t know yet if it’s crashing or just restarting as it has not happened while I was watching the server. What I do know is that whatever is set in the watchdog timer is loaded when I come back to the server. For example, I have a 6 map rotation running on my test server, when I come back to it after a day or so it’s stuck on the map I set to load in the watchdog command - fueldump. So I know that the server stopped working at some point, and reset itself.

I won’t bother posting configs unless someone thinks that is absolutely necessary. The reason for that is that this same problem has occured on 3 different gameservers that we have hosted, all with different configs. It’s been tested with ShrubET and with no mods - the problem still happens any which way. The problem happens on both ET 2.55 and 2.56 (or 1.01 and 1.02 depending on which numbers you go by). Besides that, I’ve used configs that work without a hitch on our Linux servers.

The only files we setup are what is normally necessary for the server. We copy the following to the users home directory, so it’s not a complete game install, but this has never been a problem before:

user\et.exe
user\etmain\cgame_mp_x86.dll
user\etmain\qagame_mp_x86.dll
user\etmain\ui_mp_x86.dll
user\etmain\mp_bin.pk3
user\etmain\pak0.pk3
user\etmain\pak1.pk3
user\etmain\server.cfg
user\pb* (Punkbuster files)

I’m going to try copying the other files just in case, but I highly doubt it will make a difference, they are:
user\version.inf
user\etmain\campaigncycle.cfg
user\etmain\hunkusage.dat
user\etmain\lmscycle.cfg
user\etmain\objectivecycle.cfg
user\etmain\punkbuster.cfg
user\etmain\stopwatchcycle.cfg
user\etmain\description.txt

The server is launched with a command line such as:
user_et.exe +set dedicated 2 +set fs_game shrubet +set net_ip x.x.x.x +set net_port 27960 +exec server.cfg

Things I’ve already checked include: file comparison with known good copies - all files check out 100% using md5sum or diff, running the game in a background or foreground process - no difference, running the game as an admin user with full privileges or a regular user with limited privs - no difference, running the game without Shrub - no difference according to my clients.

I tried looking through the game logs, but without knowing when exactly the server crashed, it’s like looking for a needle in a haystack.

My clients tell me that when it happens, all players on the server get kicked off and have to reconnect. They say it happens sometimes every 20 mins and other times after several hours. They also sat that it doesn’t happen on any particular map. So as far as I know, it’s random.

Thanks in advance to anyone that can help us out. And in case you are wondering if this is legit, I guarantee you that it is. We host over 130 gameservers, so we have plenty of room to accomdate the free server.

-Ohmz

(Lekdevil.NL) #2

Interesting. As with all problems like this one, it’s quite impossible to come up with a definite solution right off the bat. So, I’m going to dump some ideas on ya that you might want to try.

  1. First of all, the “Windows 2003” is a big red flag to me. It’s new and, as far as I know, largely untested in the game server market. Can you install XP or 2000 on one of your systems, install ET the same way you did on the 2K3 system and see if that makes a difference?

  2. The fact that the watchdog timer kicks in means that the map has crashed but the server process itself kept on running. Do all your 2K3 ET servers have this problem or only the ones that get used regularly? In any case, it might be a memory leak problem. Try using the Windows Performance Monitor (or whatever it’s called in 2K3) to graph the ET process memory usage. You can also accelerate the map cycling by setting the g_userTimeLimit to a low value and disabling the warmup (g_dowarmup 0). This will let the server cycle through a lot of maps in a short time. If it is a memory leak type of thing, this should increase the rate at which the server crashes. You might want to have some test clients connect to the server to simulate player connections.

  3. Have you tested your systems for basic system stability? Are you able to run the Prime95 Torture Test for more than 8 hours without any errors? If you’ve got bad memory or a problem with memory access, the ET data structures might get corrupted, leading to a map crash.

  4. Are there any indications of problems in the Windows Event Viewer?

If I’ll get any more ideas I’ll post them here. Good luck!

PS: Oh, and your install looks fine. There is absolutely no need to add those extra files. I do miss the Shrub files, though. Where and how did you install those?

(DG) #3

Possibly setting nothing in the watchdog will make it easier for a test period, logs might be easier to look through to troubleshoot if it’s a problem with the game. Alternatively add something to watchdog that you can search the logfile for, e.g. I use
set com_watchdog_cmd “vstr d_initial ; say watchdog found no map running - restarted mapcycle”
(d_initial is the starting vstr for my map cycle)

Also, not at all sure this will specifically help this problem, but worth doing regardless.

If you’re running multiple gameservers off the one install, add “+set fs_homepath WhateverFolderName” to the shortcut/batch/cmd line/whatever you use to get it running, i.e.

user_et.exe +set dedicated 2 +set fs_game shrubet +set fs_homepath 27960 +set net_ip x.x.x.x +set net_port 27960 +exec server.cfg

FolderName should be unique to each gameserver running on the install, I like to use the port number for convenience (easy to prevent duplicates and easy associate which server is running from which folder).The sf_homepath essentially just tells ET to use it as the working directory, all your log files will be kept here and each will have it’s own PB (so you dont get problems when multiple servers try to update the same PB at the same time). Note initially the first players to join will get spammed about old or incorrect PB version, because it takes several mins for PB to download itself for each server.

Edit: noticing the +fs_game shrubet; I take it you followed the option2/normal way of installing mods, i.e. into shrubet folder, and didnt install into etmain and overwrite files (his reasoning for suggesting this is really stupid imo).

I’d suggest W2k aswell, runs well and stable for me.

(Lekdevil.NL) #4

Yeah, the fs_homepath might be an issue if you’re running several instances of the same executable to host multiple servers. From the file listing I do get the idea, however, that each user and therefore each server has its own et.exe. As long as you only launch it once, there will be no problems with shared log or PB directories, even though fs_basepath = fs_homepath in that case.

(DarkangelUK) #5

Is there any reason you’re hosting the server using Win2k3? I mean Linux obviously works. Not trying to be rude but i have the “aint broke so don’t fix it” saying in my head.

(Ohmz) #6

I’ll try to answer all of your questions in one post:

Lekdevil.NL:

  1. I’m not the only provider hosting off of Win2k3, I know of at least one other that uses only Win2k3 and hosts over 600 gameservers. Also, it has worked really well for every other game we host, such as Q3, RTCW, MoH, EF, JK2, CS, Halo, etc. None of those game exhibit the problem, so it seems ET specific. I may try the test on a 2k server as you mentioned, but I doubt that is the issue.

  2. As far as I know, all of the ET servers we host on Win2k3 have this same problem. Checking the memory is a good idea, I’ll write a script to log it once every 5 minutes.

  3. Yes, the systems have been tested and are very reliable. The memory has been tested with memtest86, and I used prime 95’s torture test to test the CPU and heat levels. I always do this before we go live with any new server. On top of that, we use ECC memory (error correcting). I’m certain the problem is not hardware related.

  4. Nothing in the Event Viewer relating to ET (nor any system issues).

You’re right, I didn’t mention the shrub install portion, I’m using the 1.2-TEST13 build of Shrub for ET. It’s file structure looks like:

user/shrubet/qagame_mp_x86.dll
user/shrubet/shrubbot.cfg

This build runs fine on our Linux servers.

DG:
Good point on the log. However, I was being lazy before, it wasn’t quite so much the needle/haystack scenario as I thought. A few simple searches got me the info I needed to pinpoint where it happens in the log. However, I don’t see anything out of the ordinary in it. Here’s a link with highlighted areas and some notes on the log file: http://www.preygrounds.com/misc/log.html

The fs_homepath is being automatically set on the Windows server. We normally set these on the command line with our Linux systems, but I noticed under Windows it already does this automatically, so I didn’t add it to the command line. I verified this by checking fs_homepath and fs_basepath from the console. And yes, each user gets their own foilder and separate exe, pk3’s, etc.

DarkangelUK:
The reason we are using some Win2k3 servers is because some other games we decided to host were Windows only. Originally, it was for CoD, but then they came out with Linux support for it later on. Also we host Halo, which being a M$ game will probably never have a Linux binary. Another game is EF2, Windows only. There may be some others that I’m missing, but that’s off the top of my head. I also found from my testing that the games ran really well under Windows 2003, some even performed better than Linux. And believe me I do not say that lightly, I much prefer to run Linux on all of my systems.

Thanks everyone for your help, as I mentioned, I didn’t expect it to be an easy fix, but you never know with these sort of things.

-Ohmz

(Lekdevil.NL) #7

running the game without Shrub - no difference according to my clients.

Can you independently confirm that claim? Do you have any log files of servers crashing that are not running ShrubET? It is a test version, so there just might be a bug in the game code (although ASE reports 138 servers running the same version).

Also, I noticed in the log output that your server is using the XP save feature. Do you execute an XP reset in the map rotation script? If so, does the XP reset correlate with the map crash? Posting the complete server config and some more log output might help.

(Lekdevil.NL) #8

I just pb_cvarval’ed through your server’s cvars. I noticed that you’ve set g_XPSave to 7, but are not using the ClearXP command in your map rotation script. Am I right about this? Do all the servers that crash have this type of configuration, or are some NOT using the XP save feature (or use XP save but ALSO ClearXP)?

(Ohmz) #9

Not yet, I am getting ready to run a test on that next actually. However, the problem does happen on a clients 2.55 ET running Shrub 1.0 and another client running 2.56 and Shrub version 1.1-ET1.02. I told both clients to try running the server without shrub and they said it still happens, but I’m not certain that they actually did it. So I’ll run an independent test and find out.

I’ll get back to you on the other items, however, this test server is XP saving indefinitely, no clearing. I don’t believe the other 2 servers are using XP save, but I will check. I’m actually using a slightly modified config of one of my clients servers that runs fine under Linux. I’ll post my configs shortly as well.

-Ohmz

(Ohmz) #10

That is correct, it’s a never clearing XP save. I checked the other clients and they are both using g_xpsave 7 as well. However, they are cleaing XP on the first map in their rotations. One client uses an 8 map objective map rotation, the other a 6 map stopwatch.

BTW, here’s a full log from my test server in case you want to look through it. http://www.preygrounds.com/misc/crash.zip.

-Ohmz

(Ohmz) #11

Here’s the configs for my test server. Most of this was copied off a perfectly working clients server under Linux. Very little tweaking has been done, just passwords and a slightly different map rotation.

server.cfg:
set sv_maxclients “24” // number of players including private slots
set g_password “” // set to password protect the server
set sv_privateclients “2” // if set > 0, then this number of client slots will be reserved for connections
set sv_privatepassword “password” // that have “password” set to the value of “sv_privatePassword”
set rconpassword “noidonthinkso” // remote console access password
set refereePassword “” // referee status password

// DL, RATE

set sv_maxRate “25000” // 10000 standard but poor for ET
set sv_dl_maxRate “8000” // increase/decerease if you have plenty/little spare bandwidth
set sv_allowDownload “0” // global toggle for both legacy download and web download
set sv_wwwDownload “0” // toggle to enable web download while disconnected from the server

// MOTD ETC

set sv_hostname “^1USA ^7West ^4XP Save”
set server_motd0 “Hosted by”
set server_motd1 “^1www.Preygrounds.com”
set server_motd2 “”
set server_motd3 “”
set server_motd4 “”
set server_motd5 “”

// CHARGE TIMES

set g_ltchargetime 28000
set g_medicchargetime 20000
set g_engineerchargetime 25000
set g_soldierchargetime 25000
set g_useralliedrespawntime 25
set g_useraxisrespawntime 25

// MISC SETTINGS

set g_heavyWeaponRestriction “100”
set g_inactivity “150”
set g_antilag “1”
set g_altStopwatchMode “0”
set g_autofireteams “1”
set g_complaintlimit “6”
set g_ipcomplaintlimit “3”
set g_fastres “0”
set g_friendlyFire “1”
//set g_gametype “4” // Game type should be set from map rotation script
set g_minGameClients “8”
set g_maxlives “0”
set g_alliedmaxlives “0”
set g_axismaxlives “0”
set g_teamforcebalance “1”
set g_noTeamSwitching “0”
set g_voiceChatsAllowed “5”
set g_doWarmup “1”
set g_warmup “30”
set g_spectatorInactivity “0”
set g_speed “330”
set sv_floodProtect “1”
set sv_allowDownload “1”
set sv_pure “0”
set sv_minping “0”
set sv_maxping “250”
set match_latejoin “1”
set match_minplayers “0”
set match_mutespecs “0”
set match_readypercent “0”
set match_timeoutcount “0”
set match_warmupDamage “2”
set team_maxplayers “0”
set team_nocontrols “1”
set pmove_fixed “0”
set pmove_msec “8”

// shrub

exec shrub.cfg

// LMS ONLY SETTINGS

set g_lms_teamForceBalance “1”
set g_lms_roundlimit “3”
set g_lms_matchlimit “2”
set g_lms_currentMatch “0”
set g_lms_lockTeams “0”
set g_lms_followTeamOnly “1”

// VOTING

set g_allowVote “0”
set vote_limit “0”
set vote_percent “75”
set vote_allow_comp “0”
set vote_allow_gametype “0”
set vote_allow_kick “0”
set vote_allow_map “0”
set vote_allow_matchreset “0”
set vote_allow_mutespecs “0”
set vote_allow_nextmap “0”
set vote_allow_pub “0”
set vote_allow_referee “0”
set vote_allow_shuffleteams “1”
set vote_allow_swapteams “0”
set vote_allow_friendlyfire “0”
set vote_allow_timelimit “0”
set vote_allow_warmupdamage “0”
set vote_allow_antilag “0”
set vote_allow_balancedteams “0”
set vote_allow_muting “0”
set vote_allow_shuffleteamsxp “0”

// PUNKBUSTER

// sv_punkbuster is a readonly variable. Use +set sv_punkbuster on command line or use the command pb_sv_enable in the console

pb_sv_enable
pb_sv_load pb.cfg
readconfig

// LOGGING

set g_log “test.log” // Game logging
// set g_logsync 0
set logfile 1 // Console logging ( 1: enable 2: enable and sync )

// MAP ROTATION

exec maps.cfg // Campaign mode
//exec objectivecycle.cfg // Objective mode
//exec lmscycle.cfg // Last Man Standing mode

// WATCHDOG

// in case the game dies with an ERR_DROP
// or any situation leading to server running with no map
//set com_watchdog 10 // defaults 60
// set com_watchdog_cmd “exec campaigncycle.cfg” // defaults to quit
set com_watchdog_cmd “map railgun”

shrub.cfg:
// set g_shrubbot shrubbot.cfg
set g_censor “bad words here”
set g_playdead 1
set g_ctf 0
set g_ammocabinettime 30000
set g_healthcabinettime 30000
set team_maxpanzers -1
set skill_lightweapons “20 50 90 140”
set skill_battlesense “20 50 90 140”
set skill_medic “20 50 90 140”
set skill_fieldops “20 50 90 140”
set skill_heavyweapons “20 50 90 140”
set skill_covertops “20 50 90 140”
set skill_engineer “20 50 90 140”
set g_misc 0
set g_teamdamagerestriction 25
set g_complaintlimit 0
set g_packdistance 3
set g_logadmin admin.log
set g_logoptions 1
set g_dropmsg “You have been kicked from the server.”
set g_intermissiontime 30
set g_drophealth 2
set g_shortcuts 1
set g_xpsave 7
set g_goomba 1
set g_weapons 6
set g_skills 6
set g_classchange 1
set g_dragcorpse 1
set g_shove 80
set g_nadewar 0
set g_msgs 60
set g_msg1
set g_msg2
set g_msg3
set g_msg4
set g_msg6
set g_msgpos 1
set g_privatemessages 1

maps.cfg:
set timelimit 20

set g_xpsave 7
set g_warmup 30
set g_gametype 2
set g_friendlyfire 1

set goldrush “set g_msg5 ^1Goldrush: ^7Next map is Fueldump; map goldrush; set nextmap vstr fueldump”
set fueldump “set g_msg5 ^1Fueldump: ^7Next map is Oasis; map fueldump; set nextmap vstr oasis”
set oasis “set g_msg5 ^1Oasis: ^7Next map is Radar; map oasis; set nextmap vstr radar”
set radar “set g_msg5 ^1Radar: ^7Next map is Railgun; map radar; set nextmap vstr railgun”
set railgun “set g_msg5 ^1Railgun: ^7Next map is Battery; map railgun; set nextmap vstr battery”
set battery “set g_msg5 ^1Battery: ^7Next map is goldrush; map battery; set nextmap vstr goldrush”

vstr goldrush

pb.cfg:
pb_sv_badname 30 Nazi
pb_sv_badname 30 Hitler
about 30 more of these

//CVARs for the PB server to check. Place checks to keep players from abusing CVARs.
pb_sv_cvarempty

pb_sv_cvar rate in 2500 25000
pb_sv_cvar snaps in 20 40
pb_sv_cvar cl_maxpackets in -15 100
pb_sv_cvar cg_bobup in 0 0.005
pb_sv_cvar r_picmip in 0 3
pb_sv_cvar r_overbrightbits in 0 4
pb_sv_cvar r_mapoverbrightbits in 0 4
pb_sv_cvar cg_shadows in 0 1
pb_sv_cvar r_rmse in 0 0
pb_sv_cvar cg_fov in 90 120
pb_sv_cvar r_allowextensions in 1
pb_sv_cvar r_softwareGL in 0
pb_sv_scorekick -9 //player gets kicked for having score below zero

pb_sv_CQC 1 //Client Query Capability (i.e. can they query the server cvars)

pb_sv_MsgPrefix “ET PB Server”
pb_sv_KickLen 5
pb_sv_LogSync 0
pb_sv_LogCeiling 1000
pb_sv_Sleep 100
pb_sv_PowerMin 10
pb_sv_PowerDef 1
pb_sv_PowerKickLen 5
//pb_sv_httpAddr 192.168.0.1 //enter Webtool address here
//pb_sv_httpPort 27970 //enter Webtool port here
//pb_sv_httpRefresh 60
//pb_sv_httpKey “Put Password Here”
//pb_sv_httpmaps radar railgun fueldump oasis battery goldrush //for the Web Tool map pull down menu

pb_sv_extChar 0
pb_sv_minName 1
pb_sv_dupNameGrace 30 //Prevents name stealing on the server.
pb_sv_changePeriod 10
pb_sv_changeMax 20 //Number of name changes before kicking a player.
pb_sv_emptyname 1
pb_sv_aliasmax 10

// Screenshots
PB_SV_AUTOSS 1

-Ohmz

(bigguy) #12

Hello,

I am an ET admin for my clans game server. It is running on windows XP Pro. I am having the same problem this guy is having.

I did notice a trend when this happens is that the plyers stop working for the Eng. building things.

I also have shrub running with xpsave set to 7. i had it set to 5 and it still happened.

I can’t offer the reward of a free games server, but would be greatly thankfull for the help in this matter.

(Rippin Kitten) #13

Try setting match_warmupdamage to 0. We’re running into the same issue with our shrub server. While this didn’t completely remove the crashes, it did lower their frequency to only once every two days or so.

RK

(bani) #14

there are a number of server crash bugs in etmain that we fixed in etpro. i am not sure, but i don’t believe shrub fixed them.

the first most common crash is caused by lots of players combined with lots of ‘sets’ in your server config.

the second most common crash is caused by lots of players and long campaigns, eg longer than 3 maps.

(Lekdevil.NL) #15

Unfortunately, neither of those issues seem to apply to his server’s config. Still, replacing ShrubET with ETPro on one of the servers and checking if that improves things might be a good idea.

BTW, I’ve taken a look at the log file. The output when the map crash occurs is identical to what would be shown if you’d execute the “killserver” command at that point. Alle players get kicked without any indication in the log and the “Players_Allies” and “Players_Axis” cvars remain unchanged. Not very useful, but it might help to track down the problem.

(bigguy) #16

Well I took the warmup damage off. It happend again. I got this message on the consol

ERROR: ‘trigger_objective_info’ targets multiple entities with targetname ‘axissidetowertarget’, the second one isn’t a ‘func_constructible’

Once agin it restarted after the plyers did not work.

When is the new version of shrub comming out. Currently i am running 1.2 test13.

This is getting really frustrating

This happens on diffrent maps, so it can’t be the pk3’s right?

(lennyballa) #17

What about trying to host with the full game? Do a fresh install of the normal wolf et, install the patch and then host. Set you’re hunkmegs also some higher, but not to high, since that will make wolf also crash. Try it with vanilla wolf, so no mod, so you can find out if it is wolf or a mod, or maybe you’re server. Did you try it on a other machine with win2k3? Hope this helps a bit finding out your trouble, good luck :slight_smile:

(bani) #18

you may not like this answer, but you could always try etpro and see if it solves your crashing problem.

(TFate) #19

I can see the headlines now.

“In a strategically brilliant move, bani has hijacked every ET server with his ETPro mod, forcing recognition and surrender worldwide.”

j/k :slight_smile:

(ETplayer) #20

bani likes pimping his stuff :clap: