L3DT users' community
Large 3D terrain generator

Running out of memory making mip maps

Please report software faults here.

Running out of memory making mip maps

Postby demi » Mon Jan 27, 2014 1:21 am

downloaded 14.01 Pro beta 2 the other day.

I am working on a 16K map and it keeps bombing out with out of memory. Locks up map internally and does not release it back once it errors. Seems to be taking a lot of ram for a 512 mosaic.

Tested previous release 13.03 and it runs out of memory but it does release the ram and continues. It makes the mip maps when you zoom out. With 14.01 I have to select no on exit for save or it will continue to say map busy.
Last edited by demi on Mon Jan 27, 2014 1:48 am, edited 1 time in total.
demi
Oracle
 
Posts: 227
Joined: Thu Nov 24, 2005 4:56 am

Re: Running out of memory making mip maps

Postby Aaron » Mon Jan 27, 2014 1:35 am

Hi Demi,

Thanks for the bug report. Can you send me the log file, by any chance? I'll run some tests and get this sorted out quicksmart.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Re: Running out of memory making mip maps

Postby demi » Mon Jan 27, 2014 1:49 am

I probably over wrote it now. I'll reproduce the error and send it
demi
Oracle
 
Posts: 227
Joined: Thu Nov 24, 2005 4:56 am

Re: Running out of memory making mip maps

Postby demi » Mon Jan 27, 2014 1:58 am

Sent to sales. They are in different directories so it was still there
demi
Oracle
 
Posts: 227
Joined: Thu Nov 24, 2005 4:56 am

Re: Running out of memory making mip maps

Postby Aaron » Thu Jan 30, 2014 11:26 am

Hi Demi,

Thanks for the log. I'm off to a conference next week and won't be able to get back to this until the 2nd or 3rd week of Feb. However, I rather strongly suspect this is a problem with the over-caching algorithm, which is sort of a lazy garbage collector that only pages mosaic tiles back to disk when RAM is getting low, just in case they're needed again. It gives a nice performance boost to some calculations, but it's probably also the culprit if you're actually running out of RAM. Can you try the calculations again with over-caching turned off? To disable, select 'Settings->Local settings' in the menu, then set the 'MosaicCache->OvercacheFlag' setting to false.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Re: Running out of memory making mip maps

Postby demi » Thu Jan 30, 2014 10:17 pm

I give it a try but I looked at the log and I am not running out of ram. I still had at one point over250 meg when it error out. I'll report back here in a few days.
demi
Oracle
 
Posts: 227
Joined: Thu Nov 24, 2005 4:56 am

Re: Running out of memory making mip maps

Postby Aaron » Fri Jan 31, 2014 12:12 am

Hi Demi,

The problem is not normally about how much RAM is free, but whether the operating system can find enough free RAM in a contiguous block to meet the malloc request. So you can have 250MB free, but not, say, 10MB in a contiguous block, but just smaller fragments here and there. It's all about memory fragmentation, and there's no way (that I know of) to monitor this in real time, so you just ask the system for a block of memory, and the computer either says yes or no.

In fact, thinking about it, that's the problem with the mosaic cache handler; it just looks at the total amount of memory free and assumes that if there is sufficient space free then any request should succeed, without considering whether it's fragmented space or not. Instead, what I need to do when a malloc request fails is is run the memory compactor to drop tiles and then try malloc again, rather than just have L3DT throw up it's hands and complain that the operating system won't give it any more memory. I'll try this tonight, if time permits.

This may not be an issue in this case, but I think I may also take away from the user the choice of tile size for maps during calculations. When users decide to use large tiles (2k, 4k, etc.) then they massively increase the probability of running out of contiguous memory (on a fragmented system, the number of free 4MB blocks is not a quarter of the number of free 1MB blocks; it's much, much less). Users will still be able to choose tile sizes for export, however.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Re: Running out of memory making mip maps

Postby demi » Sun Feb 02, 2014 12:52 pm

I changed the setting as requested and the problems did go away so you're assumption is correct.
demi
Oracle
 
Posts: 227
Joined: Thu Nov 24, 2005 4:56 am

Re: Running out of memory making mip maps

Postby Aaron » Thu Feb 27, 2014 1:32 pm

Hi Demi,

In the latest version (L3DT Pro v14.02 beta 3) I've re-enabled the mosaic over-caching, but have made it much less aggressive in terms of memory consumed, so it shouldn't run into fragmentation problems. I've tested this on the mother of all batch scripts, and on a random assortment of really big maps, and it all seems OK. If you're interested in trying the beta release, please let me know if you find any problems.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia


Return to Bug reports

Who is online

Users browsing this forum: No registered users and 12 guests