====== HF2 / HFZ / HF2.GZ ====== ===== Overview ===== HF2 is a compressed heightfield format designed to replace uncompressed terrain formats such as HFF, TER and BT. In addition to smaller file sizes thanks to difference and gzip encoding, the HF2 format also includes an [[#extended header]] section for storing additional metadata such as georeferencing coordinates, comments, and the like. Thus, HF2 was designed with extensibility in mind. The compression in the basic HF2 files is difference encoding of lines. The HF2 format is seldom much smaller than the equivalent uncompressed data files. However, the HF2 files are efficiently compressed by the gzip algorithm, and so the HF2.GZ files (or HFZ, equivalently) may be significantly smaller than the uncompressed data (see [[#compression rates]]). By convention, the HF2 file extension refers to files that are compressed by the difference method only, whereas the HFZ or HF2.GZ extensions identifiy files compressed with both difference and gzip. HFZ or HF2.GZ files may be decompressed to recover the diff-encoded HF2 file using standard archiving software (e.g. [[http://www.7-zip.org|7-zip]]). An important feature of the HF2/HFZ format is that the vertical precision of the data may be user-defined (e.g. 1mm). This allows users to achieve very high accuracy (with large file sizes) or very small files (with lower accuracy). HF2 has a maximum vertical precision of 1/4294967296 of the height range (32 bits). For comparison, the precision of the nominally 'lossless' 16-bit TER/BT/HFF/PNG files is 1/65536 of the height range (16 bits). However, small variations may not be meaningful, and may therefore be discarded without data loss. For instance, a heightfield generated by the SRTM survey has an average error of 0.1-1.8m (depending on region)((See [[http://www2.jpl.nasa.gov/srtm/SRTM_D31639.pdf]].)), and so the SRTM data may be stored in a HF2 with a precision of 0.1m without causing the loss of meaningful data. Increasing the precision value in this way decreases the file size. Such variable precision and compression recommends the HF2 format over the uncompressed fixed bit-depth formats when greater accuracy or smaller file sizes are required. ===== Compression rates ===== The compression achieved by HF2/HFZ is a function of the randomness of the heightfield and the precision chosen by the user. The figures below represent the compression rates achieved for HF2/HFZ for a very noisy artificial heightfield generated by a diamond-square fractal algorithm (download as {{:l3dt:formats:specs:hf2:fractal-32bit.hff.zip|32-bit HFF}}, {{:l3dt:formats:specs:hf2:fractal-16bit.ter.zip|16-bit TER}} or {{:l3dt:formats:specs:hf2:fractal-16bit.hf2.gz|16-bit HF2.GZ}}). This heightfield had dimensions of 1024x1024 pixels, at 10m horizontal spacing, and a vertical range of 640m.
float IntRange = (max - min) / precis + 1; // number of integer height values required
float VertScale = (max - min) / range;
float VertOffset = min;
Following the tile header, the map data is stored in lines (rows) of pixels, again going west-to-east, with lines ordered south-to-north.
=== Line header ===
Each line has a 5-byte header of the following structure:
^ Name ^ Offset ^ Length ^ Type ^ Description ^
| Byte depth | 0 | 1 | byte | The byte-depth used for difference encoding in the line. May be 1, 2 or 4. |
| Start value | 1 | 4 | long | The starting value of the line, encoded as described below. |
=== Line data ===
The line data is stored as signed character, short or long integers, depending on the line byte depth specified in the line header. The line does not contain the first value, which is already specified in the line header. Hence the number of values in the line is TileSize-1 ([[#Map and tile sizes|See caveat on map/tile sizes]]).
== Reading ==
To calculate the second and subsequent pixel heights in the line, use the following equation:
long IntValue = FileValue + LastValue;
float h = (float)IntValue * VertScale + VertOffset;
...where //FileValue// is the integer value read from the file, //LastValue// is the integer value from the preceeding pixel. In the case of the second pixel in the line, //LastValue// is the //Start value// read from the line header. To read successive pixels, store //IntValue// in //LastValue// and iterate with the new //FileValue//. Please refer to [[:LibHFZ]] for an example implementation.
== Writing ==
Height values are written by first converting the floating-point height values into integers using the scale and offset values from the [[#tile_header|tile header]], as given in the formula below, and then using difference encoding to record the height change from pixel to pixel.
To calculate the height as an integer, use the following formula:
long HeightInt= (long)( (HeightFloat- VertOffset) / VertScale );
...where //HeightInt// is the output integer value, //HeightFloat// is the input height of the given map pixel as a floating point number, and //VertOffset// and //VertScale// are the scaling values for the current tile.
To write a line, you must first calculate the required bit depth for difference encoding. To do this, parse through the pixels in a line and use the equation above to calculate the //HeightInt// values. If the largest difference in integer values between adjacent pixels in the line is within the range of -128...127, the line may be encoded using a byte depth of 1. The maximum difference range for the supported byte depths are provided below:
^ Byte depth ^ Difference value range ^
| 1 | -128...127 |
| 2 | -65,536...65,535 |
| 4 | -2,147,483,648...2,147,483,647 |
In the line header, the scaled height value of the first pixel in the line is written in full 32-bit integer precision, as calculated by the above equation. Following the block header, the height values are written using difference encoding, where the delta between successive pixel heights is written to the file. The delta is calculated from taking the //HeightInt// value for a pixel (formula given above), and subtracting from this the //HeightInt// value from the previous pixel in the line.
===== Notes =====
==== Map and tile sizes ====
The tile size used for internal tiling in HF2 does not, and will not in general, be a whole number divisor of the map size. Maps of size 513x513 can, for instance, use an internal tiling of 64, even though the edge tiles will themselves only contain one pixel. These tiles are written with truncated size; no additional empty values are inserted to 'fill out' the tile.
Please refer to [[:LibHFZ]] for an example implementation.