Q3map2 cluster-processing support?

(SirVentolin) #1

I might be able to get my hands on a few high end pentium 3s and i already have a few pentium 4s laying around…is there a way i can cluster them with q3map2 so that they share the workload?

I saw another thread talking about splitting the work load by splitting up the bsp then rejoining them together at the end. Seems to me that’d make funny looking light maps were they join together,but i have no clue how it works.

Any tips, pointers, ect? Thanks

on the other hand…would i be better off trying to install linux on a Sun Microsystems Blade server? I have money laying around as well as CPUs…I hear the Sun blades have a LOVELY floating point capability, which i think would strongly aid the lightmapping process, no?

(SCDS_reyalP) #2

I might be able to get my hands on a few high end pentium 3s and i already have a few pentium 4s laying around…is there a way i can cluster them with q3map2 so that they share the workload?

No. If you are a skilled C coder, you might be able to make such a thing, but nothing like that currently exists.

on the other hand…would i be better off trying to install linux on a Sun Microsystems Blade server? I have money laying around as well as CPUs…I hear the Sun blades have a LOVELY floating point capability, which i think would strongly aid the lightmapping process, no?

How you spend your money is up to you, but that certainly doesn’t sound like the best bang/buck to me.

(SirVentolin) #3

Yeah, I just took a second look at the Sun website…meh…

Perhaps some Xeons would be better suited to the task?

What did you have in mind?

(SCDS_reyalP) #4

Well the whole idea of spending thousands just to speed up q3map2 compiles strikes me as fairly absurd. Unless you are working for a company on a serious deadline that compile times are keeping you from meeting…

You shouldn’t have to do full quality light compiles of your entire map very often. Even if it takes overnight (which would require a pretty complex map if you have a modern system and are using sane compile options), that shouldn’t be a big deal. Do a regular light -fast for basic testing. If you need to tune something at high quality, use regions or split that part of the map out.

If you really do want to spend on a system for q3map2, I would suggest the multi-core athlon 64s or opterons offer the best bang for the buck. Sun does actually have a pretty nice offering in this area, the V40z: http://store.sun.com/CMTemplate/CEServlet?process=SunStore&cmdViewProduct_CP&catid=116125

$35K for 4way/8 core isn’t a bad deal, although you could most certainly find something for less elsewhere. To pick a random link from google, maybe somewhere like this http://www.thinkmate.com/index.php?cPath=1967&tags=opthpc perhaps. (I have no information on that particular vendor)

(SirVentolin) #5

thousands? no no no, lemme give it to you straight, I have money, I spend it only on myself (yeah, mostly because of quake 3/doom 3) no girlfriend in sight anytime soon (if i listed all of the reasons behind that, i’d crash the forum) I have many resources, one objective, and more free time than mortal man was ever intended to have.

My only question now is what environment does Q3map2 flurish in? Would it do better with a core that could crank more floating point operations per second? Or with a core that handeled memory addressing and logic better? I noticed you listed only AMD based solutions, i’ve always been against them, partially due to their old school rep. of being unstable space heaters, and now they’re so pricey, and as far as i’ve seen, they’re still not commonly used in supercomputers for applications that require mostly floating point. I’d consider an Opteron system though, if Q3map2 was going to be 64-bit anytime soon.

thoughts?

(obsidian) #6

Building some crazy supercomputer to compile Q3Map2 maps is kind of like driving little Timmy to soccer practice in an Indy car while still respecting local speed limits.

Q3Map2’s ability to take advantage of such processors is very limited. At most, only the BSP and VIS stages can take advantage of multiple processors… but that usually only takes a few seconds to a few minutes anyway. The LIGHT stage can’t take advantage of multiple processors at all, and since this takes up 95% of the compile time, you won’t notice any kind of benefits by creating some sort of rendering farm.

Just get any relatively fast modern desktop processor and 1GB of RAM and that’ll be as good as it gets. At most, maybe get a dual core. Q3Map2 tops out with 1GB of RAM. I benchmarked by compiling q3dm1 with 512MB, 1GB and 2GB of RAM - no difference between 1GB and 2GB.

Your time (and money) is better spent optimizing your maps so that they compile more efficiently. A well constructed map makes a world of difference with regards to compile times. For example, take a map with 100% structural brushes and compile that, compared to a map constucted properly with caulk-hull and detail brushes - the difference for the VIS stage can be as large as a few seconds to several hours.

Oh, BTW… all modern processors are space heaters, so don’t look down on AMD for that. They’re not any better or worse than Intel.

Benchmark results… Take those figures with a grain of salt - compiled by different people on different machines with different software and setup.
http://ciole.net/quake_bench/

P.S. If you really have money to waste, mail me a cheque.

(SCDS_reyalP) #7

I’m inclined to agree with this.

Q3Map2’s ability to take advantage of such processors is very limited. At most, only the BSP and VIS stages can take advantage of multiple processors… but that usually only takes a few seconds to a few minutes anyway. The LIGHT stage can’t take advantage of multiple processors at all, and since this takes up 95% of the compile time, you won’t notice any kind of benefits by creating some sort of rendering farm.

I’m pretty sure you are mistaken. Light and Vis support multiple threads, and BSP does not (look for calls to RunThreadsOnIndividual in the code). How much it benefits, I don’t know, but the code is there to do a lot of the light compile on multiple threads. It should be quite parrallelizable.

Q3Map2 tops out with 1GB of RAM. I benchmarked by compiling q3dm1 with 512MB, 1GB and 2GB of RAM - no difference between 1GB and 2GB.

That surely depends on the map. q3dm1 is not a very complex map. There is no inherent reason q3map would top out at 1 GB, and on complex maps q3map2 can eat a LOT of memory. A number of people had problems running out of both physical RAM and swap.

SirVentolin:
My reason for suggesting the AMD x64 processors is that they are very strong in memory bandwidth and non-SSE floating point, which are both things q3map2 likes. The non-opteron models are also reasonably priced, and dual core is supported in most motherboards. At high end, the athlon64 achitecture is also superior for serious SMP systems. This http://www.anandtech.com/IT/showdoc.aspx?i=1982&p=3 tells you why.

they’re still not commonly used in supercomputers for applications that require mostly floating point.

25 of the current top 500 supercomputers run AMD. Including #10 and #11. Of course, intel has more than 10 times that number, which pretty much reflects their shares of the server market overall.

It might sound like I’m being AMD fanboy here, but thats just how I see the current price/performance situation. It has been the opposite in the past, and not doubt it will be again.

(SirVentolin) #8

Thanks for the feedback…that should help me out a lot in the future. It seems at the moment I don’t have quite as much money as I thought, so any of my silly…silly purchases will have to wait. For the sake of sanity though, I wasn’t planning on getting a dual Xeon JUST for compiling quake 3 maps…though…if the money had been right…who knows. The board I was looking at supported 8GB of ram, dual Xeon 3.6Ghz, and PCI 16x…which I believe would have been a lucrative setup for say…any game mankind may develop for the main stream market anytime in the next 4 years…easily. Compiling, coding, things of that sort are also important, and at the moment…more important to me than ever before. Anyway…I wondered what Obsidian meant about the Light process not being parallelized…I figured since the -threads switch was a major switch, it must effect every function of the compiler beneath that…meaning all BSP, VIS and Light calcs. Also considering that Light is the capitol bitch of mathematics…it’d be a very odd decision on Ydnar’s part to make that one process non-parallelizable.

Yeah, all moddern CPUs are space heaters, what’s up with that? You’d think they would have come out with better construction methods by now…last I heard IBM’s Cell processor was 60 or 65nm based…not too shabby. In effect of this inherent problem though, I was looking at getting the 2.8Ghz low-voltage option Xeons. AMD has made some company turn arounds in the past 5 years…granted, I certainly respect them more now than I did back in the K6 days…Still the issue stands that: Does anyone know the status of a potential 64-bit Q3map2 release? Also the issue was brought up that Q3map2 may not “see” the higher level functions supported by a processor like a Xeon. EM64 and Disable-bit instructions…sure…but the logic branching and basic floating point math are SO low level…it shouldn’t matter how Q3map2 was compiled/programmed, any program should be able to take advantage of those simple functions…which processors nowadays have in spades. Right?

I have a copy of TrueSpace 3, which was “designed for” 486s…or atleast that was the minimum requirement. The renderer for TS3 is one of the least efficient things I think i’ve ever been a witness to…but in any event…contrasting and comparing rendering performance between my old Pentium 1 200Mhz and my P4 3.0Ghz HT…yeah, there’s a difference in render time by as much as a week in some cases. This tells me that there’s a TON of unoptimised math going on, in which my P4 beat my P1 senseless, with no change to the original software.