Table of Contents

HF2 / HFZ / HF2.GZ

Overview

HF2 is a compressed heightfield format designed to replace uncompressed terrain formats such as HFF, TER and BT. In addition to smaller file sizes thanks to difference and gzip encoding, the HF2 format also includes an extended header section for storing additional metadata such as georeferencing coordinates, comments, and the like. Thus, HF2 was designed with extensibility in mind.

The compression in the basic HF2 files is difference encoding of lines. The HF2 format is seldom much smaller than the equivalent uncompressed data files. However, the HF2 files are efficiently compressed by the gzip algorithm, and so the HF2.GZ files (or HFZ, equivalently) may be significantly smaller than the uncompressed data (see compression rates). By convention, the HF2 file extension refers to files that are compressed by the difference method only, whereas the HFZ or HF2.GZ extensions identifiy files compressed with both difference and gzip. HFZ or HF2.GZ files may be decompressed to recover the diff-encoded HF2 file using standard archiving software (e.g. 7-zip).

An important feature of the HF2/HFZ format is that the vertical precision of the data may be user-defined (e.g. 1mm). This allows users to achieve very high accuracy (with large file sizes) or very small files (with lower accuracy). HF2 has a maximum vertical precision of 1/4294967296 of the height range (32 bits). For comparison, the precision of the nominally 'lossless' 16-bit TER/BT/HFF/PNG files is 1/65536 of the height range (16 bits). However, small variations may not be meaningful, and may therefore be discarded without data loss. For instance, a heightfield generated by the SRTM survey has an average error of 0.1-1.8m (depending on region)1), and so the SRTM data may be stored in a HF2 with a precision of 0.1m without causing the loss of meaningful data. Increasing the precision value in this way decreases the file size. Such variable precision and compression recommends the HF2 format over the uncompressed fixed bit-depth formats when greater accuracy or smaller file sizes are required.

Compression rates

The compression achieved by HF2/HFZ is a function of the randomness of the heightfield and the precision chosen by the user. The figures below represent the compression rates achieved for HF2/HFZ for a very noisy artificial heightfield generated by a diamond-square fractal algorithm (download as 32-bit HFF, 16-bit TER or 16-bit HF2.GZ). This heightfield had dimensions of 1024×1024 pixels, at 10m horizontal spacing, and a vertical range of 640m.

File accuracy File sizes HFZ compression rates
Bit depth Precis. (mm) HFZ HF2 HFF TER PNG BMP to float to TER to BMP
32 bit 0.0002 3.52MB 4MB 4MB 12%
30 0.001 3.42MB 14%
26 0.01 3.05MB 23%
23 0.1 2.45MB 3.13MB 39%
20 1 1.74MB 2MB 56%
16 bit 10 1.40MB 2MB 2MB 1.39MB 65% 30%
13 100 742kB 1MB 82% 64%
10 1000 325kB 92% 84%
8 bit 2500 231kB 1MB 198kB 1MB 96% 89% 77%

If you don't want to read all the numbers, the take-home messages are:

  1. HFZ provides continuous control over precision and compression, from 32-bit (12% compression) down to 8-bit (96% compression) and anywhere in between.
  2. A 16-bit HFZ file is 30% smaller than a TER file, and about the same size as a 16-bit PNG image.

Some more technical notes:

Source code

The canonical implementation of the HF2/HFZ formats is the LibHFZ library, the source code of which is freely provided under the terms of the LGPL. LibHFZ is written in the C programming language.

File structure

The HF2 file contains a fixed-length header, an optional variable-length extended header section, and finally the height data stored as difference-encoded tiles. The HFZ is merely a gzip of a HF2 file.

File header

The header of a HF2 file consists of 28 bytes of the following structure:

Name Offset Length Type Description
File ID 0 4 string Should be “HF2” (null terminated)
Version no. 4 2 unsigned short Should be 0.
Width 6 4 unsigned long Width of map in pixels.
Height 10 4 unsigned long Height of map in pixels.
Tile size 14 2 unsigned short Size of internal map tiles (8→65535). Default is 256.
Vert. precis. 16 4 float Precision of vertical scale, in metres. Must be greater than zero. Default is 0.01.
Horiz. scale 20 4 float Horizontal pixel spacing, in metres. Must be greater than zero. Default is 1.
Ext. header length 24 4 unsigned long Length of extended header, in bytes. Zero is default.

The values in the header may be read from a HF2/HFZ file without uncompressing the entire file using the functions provided in LibHFZ, particularly hfzReadHeader2.

Extended header

The extended header may contain additional application-specific data, stored in tagged blocks of variable length. The total length of the extended header is defined in the core header, described above.

Within the extended header, data blocks have the following byte structure:

Name Offset Length Type Description
Block type 0 4 string Null-terminated string describing type of block. Typically “txt”, “xml”, or “bin”. Not case-sensitive.
Block name 4 16 string Null-terminated string containing name of block. If null, the block should be read and stored but not interpreted.
Block length 20 4 DWORD Number of bytes in the block, immediately following block length. May be zero.
Block data 24 variable variable User defined data. Length is given by 'Block length'. May be text, XML, binary, whatever.

If a program does not “understand” a block, it should either store the block and re-write upon save, or else warn the user that data may have been lost.

Commonly used extended header blocks are listed on the extended header block page.

Map data

Map data is also stored in square tiles, with a side-length given by 'tile size' in the header. The tile order is in rows from west-to-east, with successive rows going from south-to-north. The tile ordering for a 4×4 example is shown below:

12 13 14 15
8 9 10 11
4 5 5 7
0 1 2 3

Tile header

Each tile has an 8-byte header of the following structure:

Name Offset Length Type Description
Vert. scale 0 4 float The vertical scaling of the data in the tile.
Vert. offset 4 8 float The vertical offset of the data in the tile.

These values are used when reading a HF2 to calculate the floating-point altitude from the integer values stored in the file, as shall be described below. When writing a HF2 file, these values are calculated from the min/max altitude in each block and the user-defined vertical precision, using the following equations:

float IntRange = (max - min) / precis + 1; // number of integer height values required

float VertScale = (max - min) / range;
float VertOffset = min;

Following the tile header, the map data is stored in lines (rows) of pixels, again going west-to-east, with lines ordered south-to-north.

Line header

Each line has a 5-byte header of the following structure:

Name Offset Length Type Description
Byte depth 0 1 byte The byte-depth used for difference encoding in the line. May be 1, 2 or 4.
Start value 1 4 long The starting value of the line, encoded as described below.

Line data

The line data is stored as signed character, short or long integers, depending on the line byte depth specified in the line header. The line does not contain the first value, which is already specified in the line header. Hence the number of values in the line is TileSize-1 (See caveat on map/tile sizes).

Reading

To calculate the second and subsequent pixel heights in the line, use the following equation:

long IntValue = FileValue + LastValue;
float h = (float)IntValue * VertScale + VertOffset;

…where FileValue is the integer value read from the file, LastValue is the integer value from the preceeding pixel. In the case of the second pixel in the line, LastValue is the Start value read from the line header. To read successive pixels, store IntValue in LastValue and iterate with the new FileValue. Please refer to LibHFZ for an example implementation.

Writing

Height values are written by first converting the floating-point height values into integers using the scale and offset values from the tile header, as given in the formula below, and then using difference encoding to record the height change from pixel to pixel.

To calculate the height as an integer, use the following formula:

long HeightInt= (long)( (HeightFloat- VertOffset) / VertScale );

…where HeightInt is the output integer value, HeightFloat is the input height of the given map pixel as a floating point number, and VertOffset and VertScale are the scaling values for the current tile.

To write a line, you must first calculate the required bit depth for difference encoding. To do this, parse through the pixels in a line and use the equation above to calculate the HeightInt values. If the largest difference in integer values between adjacent pixels in the line is within the range of -128…127, the line may be encoded using a byte depth of 1. The maximum difference range for the supported byte depths are provided below:

Byte depth Difference value range
1 -128…127
2 -65,536…65,535
4 -2,147,483,648…2,147,483,647

In the line header, the scaled height value of the first pixel in the line is written in full 32-bit integer precision, as calculated by the above equation. Following the block header, the height values are written using difference encoding, where the delta between successive pixel heights is written to the file. The delta is calculated from taking the HeightInt value for a pixel (formula given above), and subtracting from this the HeightInt value from the previous pixel in the line.

Notes

Map and tile sizes

The tile size used for internal tiling in HF2 does not, and will not in general, be a whole number divisor of the map size. Maps of size 513×513 can, for instance, use an internal tiling of 64, even though the edge tiles will themselves only contain one pixel. These tiles are written with truncated size; no additional empty values are inserted to 'fill out' the tile.

Please refer to LibHFZ for an example implementation.