L3DT users' community
Large 3D terrain generator

Disk Paging

Got a problem or need advice?

Disk Paging

Postby Rummy » Thu Mar 15, 2007 12:29 pm

Not sure where to put this since it may be a feature request, but here is my problem. I can't resample my very large heightfield because I seem to be running out of temp. storage space. Is it possible to select the disk l3dt pages to? I have 4 hard drives of which the smaller (faster) hard drive runs my applications and the others are for storage. My C: is rather precarious when it comes to space so I don't think I have enough room for the scratch disk. I guess my question is, how do I get my paging file to one of these other drives and can that only be done by installing L3dt onto one of them or finally am I totally wrong and thats not even my problem : ) L3dt is looking for a temp mosaic (sub map) file for re sampling that is not in the there by the way.......sorry no gogo juice in me yet. :roll:
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm

Postby Aaron » Thu Mar 15, 2007 12:58 pm

Hi Rummy,

Most of L3DT's paging takes place wherever you save your map. However, as of one of the earlier L3DT 2.5 beta builds, L3DT also pages some data (such as some temp maps) to your Windows temp directory. I can probably make this a user definable path in the next release (after v2.5). In the mean-time, you can if you so desire change your temp path* using the Control Panel, under the Advanced pane, then the 'Environmental variables' button, and then the TEMP option in the list. You have to restart for these changes to take effect.

L3dt is looking for a temp mosaic (sub map) file for re sampling that is not in the there by the way.......sorry no gogo juice in me yet.


Could you please send me the logfile and the map project file?

Best regards,
Aaron.

* Note I have no idea if this will cause other adverse effects. Follow this advice at your own risk.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Postby Rummy » Thu Mar 15, 2007 2:57 pm

Hello Aaron, yes it's the temp maps that it can't locate. I must have cleared the log since it doesn't seem to show the errors. I can easily reproduce it so I'll try to get you a log asap. When you say pages temp maps do you mean that as soon as this process begins it uses this directory? If it does then my problem is probably the tolerable size windows allows because L3dt gets well into the resampling process before this error occurs.
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm

Postby Aaron » Thu Mar 15, 2007 11:32 pm

Hi Rummy,

The log is cleared each time you start L3DT. Otherwise it would get really big.

When you say pages temp maps do you mean that as soon as this process begins it uses this directory?


Yes. L3DT doesn't wait until it's nearly out of memory before it starts paging; it's a continuous process so that the RAM usage in most calculations is farily stable whilst the the disk utilisation steadily rises. Off the top of my head, the following algorithms use temp maps in the temp directory:
  • Heightfield inflation
  • Heghtfield erosion
  • Map resampling (inc. during export)
  • Flooding seas/lakes/water table
  • Casting shadows
  • 2D rendering in main window (for large maps)
Some of these temporary maps can in fact be quite large (esp. the flooding ones), so I suppose to be on the safe side it would be recommended to have available space on the Windows drive (or wherever Docs and settings resides) that is around double the required space to store the map in its finished state. If you have Windows on a small partition then this will cause problems, so I'll move ahead with the development required to re-map the temp directory to a user-specified path.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Postby Rummy » Fri Mar 16, 2007 3:21 am

Thanks so much; I do appreciate that. I have about 7 gigs of free space on the c drive but the resampled HF based on some quick math will total about 33gb! Wow no wonder it won't finish.

Presently, I have cut the HF into a mosaic and exported whereby I resize rather quickly with a PS batch. My seams have gotten better but I think I've nailed the problem down to the duplicate border routine. I have ever so slight variations between matching tiles, usually just 1 shade of gray between pixels that when upsampled gives me fits. Thing is that many pixels on the 2 borders match (y axis is almost always perfect but x axis has those ever so slight variations. This is probably only obvious with the HF.

Again thanks for the help!
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm

Postby Aaron » Fri Mar 16, 2007 5:42 am

Hi Rummy,

I just remembered: If you've got a really recent version of L3DT (e.g. v2.5 RC3) you can use the new compressed heightfield format, which could potentially save you a lot of space for the swap maps.

To enable the new format, go to 'file->format preferences' in the menu, select 'project maps->heightfield->HFZ', and click on the 'use format' and 'make default' buttons. The space saving you get will depend on the vertical precision you require. Normally, the default precision is around 1cm (<1/2 in), which means that any vertical variations smaller than this are discarded. The larger you set this value, the smaller the files you get. I did a test with the precision set to 1m, and this gave a file that was 17% the size of the normal HFF. 1m may sound like lousy precision, but that was still more than twice as accurate as an 8-bit bitmap, and only ~35% of the size. To set the precision, go back to the HFZ page in file preferences, click on the 'options' button, then double-click on the 'precis' option. The values are in metres, so a value of 0.3 would be an accuracy of 30cm (~1ft).

Cheers,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Postby Rummy » Sun Mar 18, 2007 3:03 am

I noticed that new format a while ago :D . Anything under 2cm is a waste for me so that might workout well, however I will leave it at the default and let the game engine optimize. Any thoughts on the duplicate borders problem? Oddly it is more prevalent on the x axis of a tile, odd. I need the overlapping pixel, any thoughts on a work around? I should say this occurs with .png files prior to a photoshop inflation.
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm

Postby Aaron » Mon Mar 19, 2007 2:49 am

Hi Rummy,

No idea about the dup borders problem. I'll put it in the to-do list for v2.5a.

Oh, by the way, in the final release of v2.5 you will be able to change the temp directory path used by L3DT.

Cheers,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Postby Rummy » Mon Mar 19, 2007 12:59 pm

You're an ace! Thanks again.
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm

Postby Rummy » Mon Mar 19, 2007 3:35 pm

Ok, I'll use grow tiles by one because it seems to work properly.
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm

Postby Aaron » Wed Mar 21, 2007 10:17 am

Hi Rummy,

I've just uploaded RC3.3 (boy is this taking a long time!) Amongst other things, it lets you change the temp directory. To do this, go to 'Settings>File paths' then double-click on 'L3DT->TempDir' and select the desired temporary directory. You will have to re-start L3DT for the change to take effect.

Cheers,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Postby Rummy » Wed Mar 21, 2007 12:24 pm

8)
Rummy
Contributing member
 
Posts: 49
Joined: Wed Jan 31, 2007 2:28 pm


Return to Help and support

Who is online

Users browsing this forum: No registered users and 19 guests

cron