HardLock with 32kx32k textures

[LordThrash]

I have 2.5b Pro and keep trying to get higher and higher res terrain, but every time a try any combination of mosaic tiles that will reach up to 32k x 32k my whole CPU will freeze during the terrain texture generation step. It looks to me like an infinte loop. No memory errors are reported.

I can make 16kx16k with no problems. I have tried mosaic tiles of 1024 and 512 and multiple maps. Maybe ten times now, and each one will crash.

I am using windows XP-64bit SP2 with 2 gigs of Ram on an AMD 4400+ dual core. My page ram is set to 8 gigs with raid-0 drives.

I'm hoping that this is something weird with just my OS and/or hardware. Suggestions?

[metalliandy]

Hey LordThrash,

I have spent the last 2.5 days rendering a 32x32 lightmap and it crashed halfway through the texture generation.

I have AMD x2 5600+...maybe its something to do with X2 cpu's :S

Though on second thoughts...prob. not...

Has anyone else had problems with this on AMD x2 cpu's?

[Aaron]

Hi LordThrash,

Thanks for the bug report. I've just setup a 32kx32k run to see if I can reproduce the problem. It's been quite a few months since I've tested the gigapixel texture generation performance/behaviour, so there's a chance I've put a bug in there somewhere. I'll let you know what I find, sometime later today.

Cheerio,
Aaron.

[Aaron]

Hello,

Well, it took my old AMD XP2600 about 11 hours to make a 32x texture map, but it completed without errors or crashes. LordThrash, could you please tell me the settings used for your map? In particular, I'd like to know the following settings:

• Climate used
• Heightfield size (in pixels)
• Attributes map resolution (AM/HF)
• Light map resolution (LM/HF)
• Texture map resolution (TN/HF)
• Texture map anti-aliasing radius.

It's possible (probable?) that I used different settings for my 32x map, so I'd like to test it again using your settings.

Best regards,
Aaron.

[LordThrash]

Was there an easy way to get that data? I was clicking through the wizard to look at the values for each layer. I noticed the numbers dont match up. It looks like the design map was 1k x 1k where everything else looks like it thinks its 2k x 2k

Climate used Temperate
Heightfield size (in pixels) 1024x1024, 512 tile, 2x2 mosaic,
Attributes map resolution (AM/HF) 8, mosaic 512 tile
Light map resolution (LM/HF) 8, (32 x 32 mosaic)LM size = 16384 x 16384 pixels
Texture map resolution (TN/HF) 16, TX size = 32768 x 32768 pixels
Texture map anti-aliasing radius. 2x

I get a lock up in less than 3 hours on the texturemap step
-Thrash

[LordThrash]

I switched to 2048x2048 tiled mosaic textures with a total of 32kx32k and it finished! yah!

[Madcowthomas]

I wanted to test the 32k since been working on the bare min for are project here.

Started last night on are 1024 map that were using, it has a horiz scale of 10.00

On the calc use the "For Best Results" on each, except on the texture ran anti-alisaing 8x and high resolution texture is set to create a 32x map

Start this last night at 11:30 and its 3:27 here and about halfway in the texture with a remaining time of 7hrs

Keep you all posted on my results

[Aaron]

Hi Thrash,

I ran a test using your settings, and it worked OK:





(note: these images weren't rendered at full texture res in Sapphire because my graphics card is a piece of junk.)

However, my testing box was a single-core system, and this bug is caused by the multi-core calculation manager. In particular, I just read this MSDN article, that mentions way down in the 'remarks' section that the Microsoft Win32 API can only allow up to 2048 threads to be created. I never knew that before, but that's rather important for multithreaded apps. Hmph!

At the moment in L3DT's calculation manager, a new thread is created for each tile and discarded once the tile is complete. At any given time, L3DT ensures that the thread count is no more than the number of logical processing units in your system (i.e. two for dual core, four for quad, etc), but what seems to count here is the total number of threads created, regardless of whether they have been destroyed since.

By increasing the tile size, you reduced the number of tiles, which had the effect of reducing the number of threads created during the lifetime of the calculation. If you had run the calculation again, it probably would have crashed then.

Anyway, the upshot is that I need to rewrite the calculation manager to reuse worker threads. I'll get right on it.

In the mean-time, one solution may be to disable parallel multithreading. To do this, go to 'settings->local settings' in the menu, then set 'calc->ThreadManager->MaxThreadCount' to 1 and set 'calc->ThreadManager->AutoDetectCoreCount' to false. To reverse these changes afterwards, simply set 'calc->ThreadManager->AutoDetectCoreCount' back to true.

Oh yeah, I nearly forgot:

Was there an easy way to get that data?

Yes there is. In the project directory, L3DT will have saved a file called 'YourMapName summary.html'. If you open this in a web-browser (i.e. double-click), it will show you all the settings down at the bottom of the page. Alternatively, you can go to 'settings->current project' in the menu, which shows you all the settings in a tree view. Finally, a third way to get the settings is to look in the 'YourMapName.def.xml' file. For users I would recommend the first two options, by I myself prefer the third option, because if I open that file in L3DT it will actually start re-creating the map based on those settings (a feature of the batch engine). That's awfully convenient when you're trying to reproduce a bug, and that's why I normally ask users for copies of their '.def.xml' files when they encounter a bug (although I seem to have forgotten to do that in this case, curiously.)

Cheerio,
Aaron.

[Madcowthomas]

Here is 2 shots from one of my terrains


Low Res




High Res 32k

[Q-dad]

Hi,

Just a subjective comment here... To me the low res picture looks much more natural (real) than the high res one... Is this only me, or could there be some objective "truth" to that observation...? I guess it could be like that for several reasons, i.e. that maybe the texture quality isn't "up to the task" for this high resolution, or maybe the pictures were taken from an unlucky viewpoint, or... anything else...? Just wondering...

(Probably not related, but I've always thought that relatively low res renderings of voxel terrains (Delta Force 2) looks much more natural (real) than high res renderings of polygon terrains (Black Hawk Down, etc.), and not until FarCry was released did I see natural (real) looking renderings of polygon terrains inside a game...)

Btw - the renderings posted by Aaron above looks very natural (real), but I guess those terrains wouldn't look like that in real time renderings of most game engines, would they...?

[Madcowthomas]

Both of those images are taken from TGEA engine

Been so busy have no time lately, due too the holidays

[Aaron]

Hi LordThrash,

I think I've fixed this problem (see announcement). Can you please try the latest developmental build on the Pro downloads page and let me know if it works?

Best regards,
Aaron.