Page 1 of 1

Memory allocation problems

PostPosted: Wed May 02, 2007 8:49 am
by kanetsb

I have the following problem. I need a large resolution texture for my terrain. As it happened - the resolution is 16k by 16k.

L3DT will handle that without any problems in the paging mode - but I want this, to be a single image for easier handling in Photoshop (for stuff like roads etc).

The problem is, that when I try to combine the blocks or just create it as a non-paged texture, I'm getting memory allocation errors like this:

L3DT_GetMemState error:
- cannot initialise performance monitor

CSubMap::InitMap error:
- calloc failed to allocate 768MB of RAM
- current state:

CMapWrap::CombineMosaic error:
- Cannot initialise submap

Is L3DT really unable to allocate and do this operation? I'm pretty sure I've got twice as much free physical RAM at this point.

Help... :(

PostPosted: Wed May 02, 2007 10:17 am
by kanetsb

Never mind - I guess it's the same as here:

Oh well... I guess I'll try making 8k/8k pages and join them manually in PS.


PostPosted: Wed May 02, 2007 3:10 pm
by holger
Had the same problem with more than 2Gb free physical ram...

PostPosted: Thu May 03, 2007 4:55 am
by Aaron
Hi Kanetsb and Holger,

The issue here is that the operating system is refusing a request for a large contiguous block of memory. This can happen because of memory fragmentation, small pagefile size, or other OS-dependent effects. You can for instance have 2GB free, but not have a contiguous block of 768MB free. Also, the 'Microsoft Foundation Classes' library (used by L3DT) limits the largest contiguous block allocated by a program to 1.2GB, irrespective of the actual amount of physical RAM.

These limitations on contiguous memory are why applications like L3DT and Photoshop use their own paging, so that many smaller non-contiguous blocks are used. Indeed, Photoshop has no greater success in allocating large contiguous blocks of memory than L3DT does; in the background it splits the images into smaller blocks stored in scratch files, much like mosaic maps in L3DT. Photoshop is always working in 'mosaic mode', but it's well hidden. At some point in the future, L3DT will also work purely in paged mode (all maps are mosaics), but for that to work I need to make some new image plugins, as described below.

Now, one difference between the way that Photoshop and L3DT do their paging is that L3DT cannot save large images as single files (it can do this for TER, HFF, without memory limitations, but not for images). This is because the 3rd party image library used to save image formats (FreeImage) requires that the image is resident in a contiguous block of RAM when loading or saving. To save non-contiguous (mosaic) maps as single files would require a new image library, or more accurately, the use of many different libraries.

Using libpng and libjpeg it is possible to save large non-contiguous maps as single files, so plugins could be written to save mosaics directly as single images without additional RAM overheads. I don't think there is an equivalent for BMP, but it's a simple format so I could write that myself. I have no idea about TGA, PCX or the other miscellanous image formats supported by L3DT. Anyway, the reason I have not switched over from FreeImage to these other libraries is that it becomes significantly more time-consuming to develop and support these format plugins. It is on the long-term to-do list, but that's all.

Best regards,

PostPosted: Mon May 14, 2007 9:58 pm
by Aaron
PS: I've fixed the out-of-memory error message in the latest dev buld to record your total allocated and currently available memory at the exact moment the memory request is refused.

PostPosted: Thu May 24, 2007 6:49 am
by kanetsb
I can give it a shake this weekend and see what comes out...

But anyway - I've written a simple C++ prog that just takes all the BMP files generated as a mosaic, and writes them to a single big RAW file... [opens a row of bitmaps, writes them row by row, closes, next row of bmps... and so on] Too lazy for headers etc... ;) Anyway - it works and you get a nice big file on which you can paint in Photoshop or whatever else. Maybe something like this should also be in L3DT? My code is of course crap, but it takes up something like 100k of memory, most of it are C++ libs etc. ;)

PostPosted: Sat May 26, 2007 2:29 am
by Aaron
Hi Kanetsb,

I'll bash together a plugin to do this. It's been on the to-do list for ages.


PostPosted: Sat May 26, 2007 5:19 am
by Aaron
Hi All,

I've finished the plugin. It can export bitmaps up to 4GB in size (the max allowed by the spec), and doesn't hold the map in RAM whilst doing so.

Here is the plugin page (with download link): ... _bigbitmap

If anyone is interested in making a jpg or png version of the plugin, the source code is provided on the plugin page also.

NOTE: This plugin may case L3DT to throw exceptions when shutting down. This is a problem in the way L3DT cleans up after calling threaded extension functions. I've fixed this in the next dev build.


PostPosted: Mon May 28, 2007 12:56 pm
by kanetsb
Thanks Aaron!
Works great! Exactly what was needed. :) It does crash l3dt at exit but... whatever. 8)

Other formats - I guess most of the people that would need these big sizes won't use jpg anyway as it tends to introduce some artifacts even at it's highest compression coefficient and png will only shrink this file to like 40% tops of the bmp so it will still be in around a 100MB for 16k/16k texture. Anyway you can always trans-code it to any format using any app like photoshop or gimp.

Thanks again!!!

PostPosted: Tue May 29, 2007 11:50 pm
by Aaron
Hi Kane,

I've just uploaded v2.5.1.2, which will fix the crashing problem. I forgot to include the plugin in the installation package, but I'll do that next time around.