L3DT users' community
Large 3D terrain generator

Memory fragmentation in L3DT Pro

Important stuff.

Memory fragmentation in L3DT Pro

Postby Aaron » Sat Jan 02, 2016 1:56 pm

Hi Everyone,

Firstly, Happy New Year!

Following a number of bug reports for L3DT Pro, I've identified a fault in the memory management that is causing heap fragmentation. The effect of this fault is to cause memory allocation to fail during calculations when large tile sizes are used in large maps; typically tile sizes of 2048 x 2048 pixels or larger, and map sizes 16k x 16k or larger.

Two fixes have been implemented, and are now in testing:
  1. L3DT will now allocate large map tiles as a series of separately allocated scanlines, rather than as one gigantic allocation for the whole tile. This change should effectively eliminate memory fragmentation, as the memory requests will be much smaller than before. Unfortunately, a number of plugins directly access the raw map/tile memory, and so need to be modified to support non-contiguous scanline based memory allocations. Some plugins have been successfully updated (e.g. Sapphire, L3DTio_HFZ), but the bulk of the plugins still need to be checked for this type of access, and updated if required. If they are not updated, L3DT will automatically convert to the previous contiguous allocation mode to allow the plugin to proceed safely, but it will throw an error message to indicate the plugin should be updated. Hopefully you'll never see one of these messages.
  2. For completeness I've also fixed a secondary cause of fragmentation that was inadvertently introduced with the over-caching feature in 2011. The mosaic memory table now holds onto memory pages for a little longer after a tile has been closed, just in case another tile is about to be allocated. Often, one tile is paged to disk to make space for another tile to come into RAM, and in this case it makes no sense to release the memory of the first tile just to re-allocate it for the second. This is mostly good housekeeping, as fix #1 should comprehensively some the fragmentation problem, but I'll sleep better knowing it's fixed.
Anyway, as mentioned above, I've a great many plugins to check and update. I'll upload a developmental build once done, and should these changes prove successful, we'll proceed fairly quickly to L3DT release v16.01 (or v16.02, if further tweaks are required).

Much gratitude goes to Demi, Tuomas, Sven, wdhwg001, and the other champion users who have taken the time to file bug reports for this memory allocation problem. Without their testing and reporting, this bug would have taken much longer to track down and fix. Thanks!

In other news, the Bundysoft store will be temporarily closed from the 4th to the 8th of January. I apologise for the inconvenience.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Re: Memory fragmentation in L3DT Pro

Postby Aaron » Fri Jan 22, 2016 1:10 pm

Hi Everyone,

The fixes described in the previous post are now available for testing in L3DT v16.01 build 1, which is on the Pro downloads page (download link is in your reg/sales e-mail.) This version should be much safer to use with large tile sizes (e.g. 2048px). However, please note that I still discourage the use of large tile sizes for calculations, as it limits the paging granularity and means L3DT uses more RAM to perform the same calculation at the same speed.

Please also note that, in addition to the two fixes described in the previous post, I have also made L3DT reduce its thread count when RAM (or address space, on 64 bit systems) is running low. This is because each calculation thread requires a certain number of tiles to be held in memory, so if L3DT needs to reduce the number of tiles in memory, it means reducing the number of threads too. If the memory situation improves, the suspended threads will automatically resume.

If you find any problems with the new build, please let me know in this thread, in the bug report forum, or by e-mail to aaron@bundysoft.com.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia


Return to Announcements

Who is online

Users browsing this forum: No registered users and 18 guests

cron