2.5b build 7: Exporting as hf2,hfz, etc

[monks]

Hi Aaron, I'm having trouble exporting a terrain as a hf2, hfz, hfs.gz- I kinda fuzzy on the differences between those right now - anyway, I've tried exporting the 20K terrain to all of these formats and they won't open in L3DT or GlobalMapper.
The terrain was originally imported into L3DT as a bt from GM- so no doubt it had projection info. I've tried altering the boolean for projection but it makes no difference.
Mike has implemented the non georeffed support and I've had an L3DT terrain in Global Mapper So before I querry GM, I need to make sure that I'm exporting/ re-importing correctly in L3DT.

The only differncees between the succesful and failed exports was:
size: 512 vs 20000 (20002 x 20003)
the terrain originally starting out as a bt.

monks

[monks]

Done a bit more testing -GM now has the georeffing support in place.

I think it's a problem specific to that terrain- whether that's a size thing or the odd dimensions. I did the following:

Create a 512 terrain in L3DT -> export HF2-GZ
import to GM: OK
Exported same terrain from GM as bt
Opened bt in L3DT: OK
Export from L3DT as HF2-GZ
reloaded HF2-GZ into L3DT: OK
loaded into GM: OK

So I don't think it's the fact that the terrain started out as a bt.

monks

[Aaron]

Hi Monks,

I've uploaded an updated HFZ plugin to:

http://www.bundysoft.com/wiki/doku.php? ... l3dtio_hfz

This plugin has an updated map loading function* that allows it to load single files that are too large to fit in memory. This is the same trick that I recently used to allow the BT plugin to load 20K x 20K files.

For any developers reading this: to make this easier, I've added two new medium-level functions to LibHFZ (hfzReadTile2 and hfzWriteTile2), which use handles to tile-sized memory blocks rather than map-sized memory blocks. I've uploaded the updated source code of the HFZ plugin for those interested (see link above).

Please let me know if there are any problems.

Cheerio,
Aaron.

* Note: there wasn't a problem with the export, as far as I can see.

PS: This plugin will only work with L3DT Pro 2.5.2.7 (that's RC 2.1, Sept 11 2007) or later, since it is compiled against an updated version of the plugin API.

[monks]

I've just read the specs so I understand the difference between the extensions.



Thanks Aaron, btw Mike has a question about void elevations. I've asked him to post in the TS thread, but I'll paste it later (and probably here too) if he doesn't get time.

monks

[monks]

Hi Aaron, there's still a problem with the import on hf2s.
I'm using the latest dev build and latest extensions (BT and HFZ).

When I import a previously written hf2, the event log gives:

CMapWrap::InitMap error
-cannot allocate SubMap

CL3DTio_HFZApp::LoadMapFile error
-cannot intiialise map

CMapWrap::SetPixel error
-invlaid coordinates

It looks like it's scrolling ad infinitum on the final SetPixel statement but I can't get access to the scroll bar on the dialog. I left it for 20 mins but it was still in the state as when I left it.

They open in GM, but only the first (bottom) row of tiles is correct, the rest looks garbled.

monks

[Aaron]

Hi Monks,

Whoops, sorry for that. I tested the plugin against a regular size (16k x 16k), but had omitted a bounds-check that was required when dealing with map sizes that are not a multiple of the tile size (20003?!). I've uploaded a fixed HFZ plugin.

Best regards,
Aaron.

[monks]

oooh you slacker

monks

[monks]

We're getting closer. I do get a file loading in, but I get this:

CMapWrap::InitMap error
- cannot allocate SubMap

CL3DTio_HFZApp::LoadMapFile error
- cannot initialise map

-but it still loads but I see this:

http://www.skindustry.net/medem/files/Temp/L3DTshot.jpg

It's happened twice in row.

-juas to make sure it's nothing my end:

I open the original bt big momma: import > heightfield
save active map: as hf2 (all settings @ default)

when I save the proj out and reload that, it open super quick, no probs.

import > heightfield > browse to hf2 file

monks

[Aaron]

Hi Monks,

I think this might take a day or two. The error messages were expected (I've since fixed that), but the apparent map corruption is very not-good. While I'm at it, I'm also adding new precision modes to HF2/HFZ (relative vs. absolute), and I'll take a crack at implementing support for void altitudes in LibHFZ.

Cheerio,
Aaron.

[Aaron]

Hi Monks,

The problem with the map looking wonky seemed to be in the automatic mipmap generation, which I think I've now fixed. For testing purposes, I've made myself a 20002 x 20003 pixel BT file, and I'm now running import/export cycles between the BT, HFZ and TER formats. I've updated those three plugins (BT/HFZ/TER) to quietly and seamlessly import ultra-large single files by splitting them to mosaics, without so much as a warning or error message. Testing should take a few more hours.

By the way, if possible, can I request that the next bug you find occurs on really small maps? Something like 37x25 pixels would be really nice

Ah, one more thing; the reason the map re-loaded more quickly after saving the project is because it was split to a mip-mapped mosaic. The tiles of such maps are only loaded as required, so assuming you don't have a 20k x 20k pixel monitor, L3DT won't need to load the whole full-res map. This is one of the many reasons I prefer to have large maps stored as many small tiles; it's way, way faster for LOD-able/selected-area things like 2D/3D rendering.

Cheerio,
Aaron.

[Aaron]

Hello again,

By the way, if possible, can I request that the next bug you find occurs on really small maps? Something like 37x25 pixels would be really nice

Hah, on a whim I took my own advice, and yes, I did find a bug. When using this size, either the HFZ exporter or the importer craps-out and corrupts the map data. Some more debugging to go here.

Cheerio,
Aaron.

[Aaron]

Wow, it is much easier when I can reproduce the bug on a file that's 1kb (37x25pxls), as opposed to one that's ~800MB (20003x20002). I don't have to wait for it to load!

Anyway, I've found the fault with the bung large HFZ export, and it was in the part that handles exporting mosaics as single HFZ files. Poor bounds checking and some dubious (safe, but wrong) buffer handling.

I'll upload the update tomorrow. For now, I'll be adding some new optional extended header blocks to the HFZ format to deal with variable precision. This is to fix another bug reported by e-mail that occurs when you have small horizontal scales (e.g. 0.5m), and insufficient vertical precision (e.g. the default 0.01m), resulting in odd steps and tiles in the HFZ files. This does not affect maps with 'normal' horizontal scales (e.g. 10m).

Cheerio,
Aaron.

[Aaron]

OK Carl,

I'm ready if you are. On the Pro downloads page is L3DT Pro 2.5b release candidate 3, which has updated BT/HFZ/TER plugins, all of which can now import/export very large files.

Note that the 20K HFZ file you exported was indeed corrupted, so you'll have to re-import the BT and then re-export as HFZ.

Best regards,
Aaron.

[monks]

I've updated those three plugins (BT/HFZ/TER) to quietly and seamlessly import ultra-large single files by splitting them to mosaics, without so much as a warning or error message. Testing should take a few more hours.

Cool- that's a safe pair of hands

This is one of the many reasons I prefer to have large maps stored as many small tiles; it's way, way faster for LOD-able/selected-area things like 2D/3D rendering.

Yeh, it's pretty impressive, especially when you compare side by side a 20K bt import and a project import.

Perfecto! It's now seamless internally and between L3DT and Global Mapper. How cool is that?!

Thanks,
monks

[monks]

Hey Aaron, did you get chance to put in the void elevation suppor?- no rush, but I'm asking just in case you did- apparently Mike was going to update on that.

Cheers,
monks