L3DT users' community
Large 3D terrain generator

.NET port for libhfz?

Any and all chit-chat regarding L3DT.

.NET port for libhfz?

Postby sidefx » Sat Nov 17, 2007 9:31 pm

Hi all,

Long time reader, first time poster, heh.

I work on a little tool for Neverwinter Nights 2 that allows users to import heightmaps (and a few other odds and sods) into the NWN2 toolset. One of the features is that it can import stuff direct from an L3DT project (heightmap, lightmap + alpha maps) Of course with the new version I need to build some sort of HF2\HFZ support in. I'm starting to look at porting libhfz into C# but was wondering if anyone had done this already, save me a bit of work. :) If anyone has could you contact me on sidefxboy(at)gmail(dot)com and I'll be eternally grateful.

peace
sidefx
New member
 
Posts: 2
Joined: Sat Nov 17, 2007 9:23 pm

Postby Aaron » Sun Nov 18, 2007 9:27 pm

Hi Sidefx,

Welcome to the forum. I've not heard of any other attempt to port libhfz to C#, so I think you're the first. If I can be of any assistance, please let me know.

If it turns out to be too hard (and I have no idea about this; I'm not very experienced at C#), a fallback position would be to make a plugin for L3DT that exports the maps in formats that your tool can already read. If you would like to try this option, please give me a list of the maps and their formats that your tool requires, and I'll bash together an exporter plugin.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Postby sidefx » Mon Nov 19, 2007 12:27 am

Thanks aaron. I've had a brief look at libhfz and it shouldn't be too much trouble to port across to C#. Not sure if I'll bother moving it to some sort of object oriented model, or just do a straight port of the basic functionality, but I don't think there should be too many issues with getting something that works. All I really need is the ability to read data, not save it, so just doing that bit should cut down the work a bit more.

I figured I might as well ask first though, avoid reinventing the wheel. :)
sidefx
New member
 
Posts: 2
Joined: Sat Nov 17, 2007 9:23 pm

Re: .NET port for libhfz?

Postby celludriel » Thu Mar 13, 2014 4:35 pm

Hey,

I have been trying to pick this up and give it a shot. However I'm a poor java developer with a basis of ANSI C knowledge. So however C# comes quite easy to me the C++ stuff is frightening. Especially when the memcpy comes into place and all the buffering. I'm used to create a Stream and just apply a writer on it and get going. So for the most part I can get around all the buffering stuff with the following.

Code: Select all
        public long hfzWriteHeader(HfzFile fs, HfzHeader fh)
        {
           // copy header into buffer
            MiscUtil.Conversion.EndianBitConverter endian = MiscUtil.Conversion.EndianBitConverter.Big;
            if (BitConverter.IsLittleEndian)
            {
                endian = MiscUtil.Conversion.EndianBitConverter.Little;
            }
         
            EndianBinaryWriter writer = new EndianBinaryWriter(endian, fs.pIoStream);
            writer.Write(Util.StringToByteArray("HF2", 4));
            writer.Write(fh.FileVersionNo);
            writer.Write(fh.nx);
            writer.Write(fh.ny);
            writer.Write(fh.TileSize);
            writer.Write(fh.Precis);
            writer.Write(fh.HorizScale);

           // put extended header into a buffer
            MemoryStream memStream = new MemoryStream();
           long rval = hfzHeader_EncodeExtHeaderBuf(fh, ref memStream, endian);
            if (rval < 0)
            {
                return rval;
            }

           // now write in header length of ext header
            writer.Write(fh.ExtHeaderLength);

            // write the extended header data
            Util.CopyStream(memStream, fs.pIoStream);

           return LIBHFZ_STATUS_OK;
        }


However when I started on the write tile methods it got a bit to complicated I just can't read what is written there to translate it to C#

Code: Select all
      for(i=i1+1; i<i2; i++) {

         // find max diff in line
         f = pTileData[(i-i1) + (j-j1) * fh.TileSize];
      
         TempInt=(long)((f-VertOffset)/VertScale);
         Diff = TempInt - LastVal;

         zi = i-i1-1;
         hfzMemcpy((char*)pDiffBuf+zi*4, &Diff, 4);

         LastVal = TempInt;

         MaxDev = MaxDev>abs(Diff)?MaxDev:abs(Diff);
      }


hfzMemcpy((char*)pDiffBuf+zi*4, &Diff, 4);

This line is giving me a lot of headache. What I believe is going on

- Diff is a long (Int32 in #)
- it has gotten some value earlier in the for loop
- then you are taking an array of chars ? start do some math on it ? and copying the bytes of the Diff variable into the bytes of the char array ?

I'm wondering why is it char* and not byte* and how can I translate this to a Writer implementation in C#. What I even find more peculiar is that yes pDiffBuf has some memory allocated to it in Tilsize*4 but is that memory filled with values or just junk from the memory ?

I would be most gratefull for a bit of enlightenment cause this has got me deadlocked
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Thu Mar 13, 2014 10:00 pm

I got through my deadlock with a lot of googling and asking on stackoverflow. So now I have a fully compilable library with a unit test to load one hf2.gz file. However the test is failing. So I must have made a bunch of mistakes. I'll carry on tomorrow.

Just wanted to post to give a small update on the status and that I'm not dropping it anytime soon until I got it working :) I'm kinda stubborn that way.
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Fri Mar 14, 2014 2:47 pm

Update:

I managed to write a unit test to read the header of an example file I had lying around.

Code: Select all
***** LibFhz.LibFhzTest.HelloWorldTest
FileVersionNo: 0
nx: 1024
ny: 512
TileSize: 256
HorizScale: 10
Precis: 0,01
ExtHeaderLength: 35
nExtHeaderBlocks: 1
pExtHeaderBlocks: System.Collections.Generic.LinkedList`1[LibFhz.HfzExtHeaderBlock]

BlockType: txt
BlockName: app-name
BlockLength: 11
pBlockData: System.Byte[]


Being the result. However this seems to be a fairly easy file. I wonder if anyone can point me to a very complex hf2 file with multiple different size headers ?
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Fri Mar 14, 2014 8:13 pm

New Update:

I'm now also able to read the actual mapdata. I just hope this is correct as it should. I used the demo file of the hf2 format specs.

following results:

Code: Select all
FileVersionNo: 0
nx: 1024
ny: 1024
TileSize: 256
HorizScale: 10
Precis: 0,01
ExtHeaderLength: 0
nExtHeaderBlocks: 0
pExtHeaderBlocks: System.Collections.Generic.LinkedList`1[LibHfz.HfzExtHeaderBlock]
mapData size: 16777216 bytes


I'm hoping a bit Aaron can confirm this is the amount of bytes for the float array of that specific file
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby Aaron » Sat Mar 15, 2014 12:25 pm

Hi celludriel,

Sorry for my slow reply, and well done on your efforts, it looks like you're nearly there!

Code: Select all
nx: 1024
ny: 1024
...
mapData size: 16777216 bytes


I'm hoping a bit Aaron can confirm this is the amount of bytes for the float array of that specific file


Doesn't quite look right to me (exactly four times too large). A float is only 4 bytes, so the size of the float array should be nx*ny*4 bytes in size, which is 4194304 bytes in this case.

I wonder if anyone can point me to a very complex hf2 file with multiple different size headers ?


This example has 5 headers; 4 relating to different aspects of georeferencing, and a 5th for the relative precision of the file.

http://www.bundysoft.com/L3DT/downloads/examples/baia_mare_HF.hfz

It's a 'hfz', which is exactly the same as a '.hf2.gz'.

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3696
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Re: .NET port for libhfz?

Postby celludriel » Sat Mar 15, 2014 3:26 pm

Thanks for the example Aaron I can really use that in my testing.

I got the bytes right now I read an existing file and then write it out to a new file and it's exactly the same in byte size. However I got precision errors all over the place which I still can't get fixed. I tried calculating the Diff with a lot of permutations I just can't get it right yet.

(Tried these with fTempInt and LastVal as float and as double, even tried decimal but that was waaaaaaaaaaaaay to slow)
Diff = (Int32)(fTempInt - LastVal);
Diff = (Int32)Math.Ceiling((fTempInt - LastVal));
Diff = (Int32)Math.Round((fTempInt - LastVal));

Here is a small extract result of one of these tries, when I start writing the tiles to the file.

Code: Select all

input file       written file       calculation going on
14                14                    TempInt 40,9996470648658(40,99965) - LastVal 26,9999443597892(26,99994) = 14
0                  0                      TempInt 40,9996470648658(40,99965) - LastVal 40,9996470648658(40,99965) = 0
22                22                     TempInt 62,9996158570527(62,99961) - LastVal 40,9996470648658(40,99965) = 22
-23              -22                    TempInt 39,9998045408186(39,99981) - LastVal 62,9996158570527(62,99961) = -22
33                33                     TempInt 72,9995669922573(72,99957) - LastVal 39,9998045408186(39,99981) = 33
88                88                    TempInt 160,998679213638(160,9987) - LastVal 72,9995669922573(72,99957) = 88
75                75                    TempInt 235,998312727673(235,9983) - LastVal 160,998679213638(160,9987) = 75
55                55                    TempInt 290,997662497616(290,9977) - LastVal 235,998312727673(235,9983) = 55
18                18                    TempInt 308,997498246247(308,9975) - LastVal 290,997662497616(290,9977) = 18
-20               -19                   TempInt 288,997977449521(288,998) - LastVal 308,997498246247(308,9975) = -19




You can see the divisions and subtractions going on ruining precision. I just can't get an angle on it how to get it right. I had an idea if the fTempInt or LastVal contain a .0 to increase the value with one , since it seems it's only numbers like 14,000008 are giving problems.

I'm not that great with floating point precisions , in Java I just use BigDecimal ... slower but it will fix them for me
Last edited by celludriel on Sat Mar 15, 2014 11:44 pm, edited 1 time in total.
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Sat Mar 15, 2014 9:03 pm

Just starting a new post the other one got a bit to big.

I started thinking how could I go out of memory no way that the image could be that big. Then I realized it, I was using the straight port of the c++ code that takes a float[] and keeps filling in the holes with data every Tile read. But that is not how I designed my class. I wanted to do a two parter:

- Read the header and save it in a Header object
loop over all the tiles
- save tile data as list of float in one big list

Now with this memory issue I figured a map would be a way better structure. I could make a method on the public interface, getTileData(nTile) and you would get the heightdata for that particular tile. So I'm going with that now. But now I had to rewrite my readTile method and with the latest test file of Aaron I got myself in a bind. I believe my logic of reading the Tile isn't right here is my method

Code: Select all
        private float[] hfzReadTile(EndianBinaryReader reader, HfzHeader fh, UInt32 TileX, UInt32 TileY, float[] pMapData)
        {
            UInt32 xOriginTile, xTileBorder, yOriginTile, yTileBorder, i = 0, j = 0;

            Int32 li;

            UInt32 TileSize = fh.TileSize;
            UInt32 mapWidth = fh.nx;
            UInt32 mapHeight = fh.ny;

            UInt32 xTiles = mapWidth / TileSize;
            UInt32 yTiles = mapHeight / TileSize;

            UInt32 xTileSize, yTileSize;

            /*           xOriginTile = TileX * TileSize;
                       yOriginTile = TileY * TileSize;
                       xTileBorder = xOriginTile + TileSize;
                       yTileBorder = yOriginTile + TileSize; */

                       if (TileX == xTiles)
                       {
                           xTileSize = 1;
                       }
                       else
                       {
                           xTileSize = TileSize;
                       }

                       if (TileY == yTiles)
                       {
                           yTileSize = 0;
                       }
                       else
                       {
                           yTileSize = TileSize;
                       }


            // read vert offset and sale
            char LineDepth = ' ';
            Int32 FirstVal = 0;
            try
            {
                float VertScale = reader.ReadSingle();
                float VertOffset = reader.ReadSingle();
                xOriginTile = 0;

                for (j = 0; j < yTileSize; j++)
                {

                    LineDepth = reader.ReadByte().ToString().Single(); // 1, 2, or 4
                    FirstVal = reader.ReadInt32();

                    float pixelValue = (float)FirstVal * VertScale + VertOffset;

                    // set first pixel
                    pMapData[xOriginTile] = pixelValue;

                    Int32 LastVal = FirstVal;

                    for (i = 1; i < xTileSize; i++)
                    {
                        if (TileX == xTiles)
                        {
                            System.Console.WriteLine("You should never get here");
                        }
                        switch (LineDepth)
                        {
                            case '1':
                                li = (Int32)reader.ReadByte();
                                break;
                            case '2':
                                li = (Int32)reader.ReadInt16();
                                break;
                            default:
                                li = reader.ReadInt32();
                                break;
                        }

                        pixelValue = (float)(li + LastVal) * VertScale + VertOffset;
                        LastVal = li + LastVal;

                        xOriginTile++;
                        pMapData[xOriginTile] = pixelValue;
                    }
                }
            }
            catch (Exception)
            {
                System.Console.WriteLine("TileSize: " + TileSize);
                System.Console.WriteLine("xTiles: " + xTiles);
                System.Console.WriteLine("yTiles: " + yTiles);
                System.Console.WriteLine("x: " + TileX);
                System.Console.WriteLine("y: " + TileY);
                System.Console.WriteLine("j: " + j);
                System.Console.WriteLine("i: " + i);
                System.Console.WriteLine("LineDepth: " + LineDepth);
                System.Console.WriteLine("FirstVal: " + FirstVal);
                throw;
            }

            System.Console.WriteLine("data size " + TileX + ", " + TileY + " : " + pMapData.Length);
            System.Console.WriteLine("last x: " + TileX);
            System.Console.WriteLine("last y: " + TileY);
            System.Console.WriteLine("last i: " + i);
            System.Console.WriteLine("last j: " + j);
            return pMapData;
        }


[UPDATE]

FOUND IT !!!! there is a unclarity in the spec imho though. It seems that a tile does not only have one pixel in it when it is an "extra" tile due to uneven map size. But every row in the tile first pixel needs to be there. After I had that fixed it worked ! I've updated the code with my method in the block above. I removed the error output to keep the forum clean.

[UPDATE WRITE]

I'm starting to doubt it will never be possible to write the file in C#. The loss of precission on floats is just staggering. The very first calculation on the very first write of a tile goes totally wrong.

Tile 0,0 first write of the Vertical offset
VertScale:0,009999821 is what it should be read from the origin file (byte array: VertScale:01001010110101100010001100111100 )
VertScale:0,009999896 is what the code gives me to write to the new file (byte array: VertScale:10011011110101100010001100111100)

Code: Select all
            float BlockLevels = (HFmax - HFmin) / Precis + 1;

           // calc scale
            VertScale = (HFmax - HFmin) / BlockLevels;
           VertOffset = HFmin;
            if (VertScale <= 0)
            {
                VertScale = 1.0f; // this is for niceness
            }


two divisions further and there is no more precision left

I'm starting to wonder if I even should read the bytes from the file to float and better store them into a decimal so I can calculate them better later on and then after the calculation save them back as a float ... I'm just guessing atm cause I can't pinpoint the precision problem is it on read or on write.

I've put my work upto now here : http://users.telenet.be/sunspot/hfzcsha ... /LibHfz.7z

Maybe someone else can take a look and see the bug. There are two implementations now a clean read namespace LibHfz, and the direct C++ port the clean namespace was based of LibHfz.CppPort. There is a Nunit test called LibFhzTest with the second test in the suite being the write test. I hardcoded my file locations in it though , so if you want to run the test you'll have to change the path of the files.
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Tue Mar 18, 2014 2:13 pm

Well small update again. I was getting worried I was calculating with different numbers so I wrote a simple c++ program that runs a read of one hf2 file and then check the very first read of the very first tile and get the following results.

Code: Select all
C++
====
Vertscale: 0.00999982096
VertOffset: 53.4127693
LineDepth: '\x2'
FirstVal: 27
Li: 14
LastVal: 41
f: 53.8227615

c#
====
VertScale: 0,009999821
VertOffset: 53,41277
LineDepth: 2
FirstVal: 27
Li: 14
LastVal: 41
f: 53,82276


So even at the start of the read the precision is lost since floats in c# only take up to 7 decimal precision. So now I'm starting to get very worried if I ever be able to transform the format to C#
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Thu Mar 20, 2014 7:04 am

Update
======

I finally managed to write a hf2 file. It took a lot of brainsweat and tears but it was eventually not a problem with the precision as it turned out the precision didn't matter at all ... The culprit was this little gem of code I've written

Code: Select all
                char LineDepth = '4';
                if (MaxDev <= 127)
                {
                    LineDepth = '1';
                }
                else
                    if (MaxDev <= 32767)
                    {
                        LineDepth = '2';
                    }

                writer.Write(LineDepth);


Aaron probably sees the epic fail I've written here straight away ... it took me a while of looking in totally the wrong place, before I noticed it ... doh !

Now I'm tryinig to put all the data in a nice class based data object instead of a procedural formed big array of floats, reading works writing not so much ... I have to reevaluate how I store the data it seems. I tried letting each tile have their own float array where all the data for one tile is stored. But atm it's not really working cause my VertOffset is calculated wrongly on the first tile , first line.

UPDATE
=======
I can now perfectly read and write dimensions that are exact. however when there are extra tiles for a nx: 2049, ny: 2049 map for example everything sill crashes :(. There is something I'm missing while reading the file once I reach TileX =8
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Sat Mar 22, 2014 4:27 pm

I finally had the extra pixel problem solved the "baia_mare_HF.hf2" file I was able to read and write. So I wrote a unit test to test all maps in the example folder and then I came across MtStHelens_10m.hf2. Is there any human mathematical reason why tile (X3, Y0) Only has 175 pixels in it's row while it's tilesize is 256 ?

It seems baia_mare_HF.hf2 worked because it had 9 * 9 tiles and it "accidently" fitted.

UPDATE
=======
Well I managed to put this formula together : TileSize-(((TileX+1)*TileSize)-mapWidth)

This gives me the right RowSize for the 3,0 Tile, but when I use this formula and try the baia_mare map well that one blows up in my face cause the result there is 255 instead of the expected 256 :(

Basically what I'm looking for is two formulas that will work on ANY map of ANY size of ANY well I don't know what else can be different. That gives you the exact rows and exact rowsize of a tile.

getRows(x,y) and getRowSize(x,y). This is the only thing that is currently blocking me on maps that are not the same dimension AND have one of those extra tiles due to the modulo operation.
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Sun Mar 23, 2014 10:15 am

Completed
=========

After some long evenings of trying to figure everything out. I had to learn c++ and c# syntax from scratch, considering I'm a java dev. Then I had to figure out the file format in more detail then just copy over c++ code to c#. Anyhow I managed to read all the demo files on the site, and write them back to disk. I created unit tests for that. It's not the fastest of code it seems. I like the use of lists over arrays but that comes with a performance hit. But considering what it is needed for (no batch processing) I think I can get away with it.

I do like to make this an opern source thing or something, but Aaron I probably need your approval for that, maybe you are willing to test it and if it passes your grade of contentment, you might consider hosting it ?

http://users.telenet.be/sunspot/hfzcsha ... /LibHfz.7z

Hope you like it and other people can get some use out of it.
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Re: .NET port for libhfz?

Postby celludriel » Tue Mar 25, 2014 10:27 am

Scope Expansion
==============

Opening HFZ files is nice and all, but then I realized I needed the AMF format as well as the WMF format. So that got me thinking. Why not expand the project to open all the L3DT binary files. However that is a lot of work. But never the less the basis is there for a manager that can open all files. Hence I started refactoring and coding in the opening and saving of AMF format. I renamed the lib to something more appropriate and put it here.

http://users.telenet.be/sunspot/hfzcsha ... Manager.7z

The main class that manages the loading of the files is

L3dtFileManager

currently it has the following methods

Code: Select all
       
        public HfzFile loadHfzFile(string fileName, FileFormat format)
        public void saveHfzFile(string fileName, FileFormat format, HfzFile file)

        public AmfFile loadAmfFile(string fileName, FileFormat format)
        public void saveAmfFile(string fileName, FileFormat format, AmfFile file)

        public DmfFile loadDmfFile(string fileName)
        public void saveDmfFile(string fileName, DmfFile file)

        public HffFile loadHffFile(string fileName)
        public void saveHffFile(string fileName, HffFile file)

        public WmfFile loadWmfFile(string fileName)
        public void saveWmfFile(string fileName, WmfFile file)


enjoy
celludriel
Member
 
Posts: 17
Joined: Thu Mar 13, 2014 4:23 pm

Next

Return to General discussion

Who is online

Users browsing this forum: No registered users and 6 guests

cron