News for March 2016

Hi All,

After much delay, I am pleased to finally announce that L3DT release 16.03 is available for download. Please read on for a description of the changes since the previous release (v11.11), which are many and varied. The full change log is available here.

At the outset I'd like to thank all the users who have given feedback and helped with testing since L3DT v11.11. My deepest gratitude goes out to all those who have helped to make L3DT even better through their suggestions, comments, testing, and bug reports. Thank you for your patience, too.

For most users, yes, this is a free update.

If you've purchased your license after L3DT v11.11 was released (i.e. after 30th of November 2011), the update period of your license key has been extended to support v16.03. This is in part to make up for the unusually long delay betwixt the releases. I've also extended the free update period for licenses to L3DT v16.03 from a 18 months to two years.

However, if you purchased your key on or before the 30th of November 2011, you will need to renew your license to be able to use this update (contact sales@bundysoft.com to discuss discounts). You are of course welcome to try-before-you-buy using the free 90 day trial.

As with all L3DT updates, this is a free update for all users of L3DT Standard Edition.

As I announced at the start of March, the 'L3DT for Torque' version of L3DT Professional that was distributed through the GarageGames website has been discontinued. Instead, users with licenses for L3DT for Torque may download and use L3DT Professional, which has exactly the same Torque support. Users should contact sales@bundysoft.com to receive an non-GarageGames off-line activation key and appropriate download instructions. Please quote your Ignition activation code from GarageGames.

By far the most complex and time-consuming change in L3DT 16.03 was in the way L3DT manages its memory and CPU usage. The previous release of L3DT Pro introduced the practice of over caching, whereby L3DT kept more of the map data in RAM to speed up some calculations, sometimes by as much as 3× faster. Unfortunately, for some large calculations on some systems, this resulted in running out of memory¹⁾, which never ends well. In this release, I have made L3DT moderate its memory usage when available RAM or address space starts to run low, even going so far as to automatically suspend some of the calculation threads to reduce the memory usage. Memory fragmentation has also been all but eliminated by having L3DT allocate large map tiles as non-contiguous arrays of scanlines.

Another major focus of the previous release was to increase the speed of the multi-threaded calculations. In hindsight this went too far, as users subsequently reported that multi-threaded calculations in L3DT Pro made their computer quite unusable for other tasks. In other words, the calculations were running too quickly, and L3DT wasn't pausing enough to let other processes have a turn. In this release, calculations will run at full speed when L3DT is in the foreground, but will automatically yield one processor core when the application is minimised or loses input focus (i.e. put in the background). You can of course also reduce the CPU usage manually using the 'Utilities→Throttle CPU' menu item, or by reducing the number of cores used in the 'Settings→Multi-thread settings' menu item.

The Sapphire 3D viewer has had a few nice changes in this release. Firstly, I've fixed support for super-large heightmaps, which previously caused Sapphire to choke. The size limit in the 3D viewer should now be the same as the rest of L3DT, i.e. 131,072 × 131,072 heightfield pixels. I've tested this release up to 32k×32k pixels, and Sapphire worked just fine, although it did slow down a bit when I pushed the horizon distance out to over 100km / 60 miles on a 10m resolution heightmap (see development log post, and image below).

32k×32k pixel heightmap with 64k×64k texture map in Sapphire. View distance is 108 km / 67 mi. Triangle count is 370k.

For 'normal' view distances (say, 2000 vertices), the frame rate and memory usage of Sapphire are reasonable, and do not change substantially with map size.

In other Sapphire news, some users reported that the mesh optimisation was frustrating when editing small features in the heightfield. In this release, Sapphire now does a so-called 'brute force' render of the triangles around the brush tool, so that you can see every raw vertex and triangle that you edit. You can see this in the image below, where dark line-drawn triangles are visible, showing every vertex.

Brute force triangle rendering

The heightfield tool window also now has a new tabbed interface, which allows users to more easily flip between tools and change brush settings. While I was at it, I also added 'strength' settings to all the applicable brush tools, which was a fairly common request.

Tool selection	Tool settings

Other little tweaks include the ability to set the movement speed ('Position→Speed', in Sapphire menu), and keys for zooming in and out (PGUP/PGDN) for when you don't have a scroll wheel at hand.

Over the years, many users have complained, quite rightly, that when editing the design map, it's too difficult to predict what what the final heightmap will look like. Previously, the only way to find out what the heightmap will look like after you edit the design map was to actually generate the full heightmap.

You can of course generate the full heightmap, but that takes a long time, especially for large maps. Worse still, if the heightfield doesn't turn out exactly as you liked, you have to go back and edit the design map, then re-generate the whole heightfield again. This could be a painfully slow cycle on large maps. In this release, you can now select an area of the design map using the select area tool, then select the 'Operations→Heightfield→Preview area from DM' menu option. This will generate the heightfield for just the selected area of the design map.

For example, I have selected the central valley in this design map:

…and then selected the magic menu item…

…whereon L3DT quickly generated the heightmap of only the valley that was selected:

2D	3D (from SW corner, looking NE)

Note that this option also allows you to generate the heightfield at different resolutions, so you can for example generate an extremely fast preview of a selected area (or the whole map) at 1/2 or 1/4 the normal heightfield resolution.

For more information, please have a look at the documentation.

Over the years, countless users have requested the ability to merge together two or more heightmaps with seamless borders. This is now supported by the atEdgeBlender plugin, and the 'stitching heightmaps' tutorial will show you how to use it. But that's not all; if your heightmap tiles have different vertical scaling, such as when imported from image files, you can use the 'Operations→Heightfield→Edge blending→Linear regression of tile scaling' option to automagically re-scale each tile so that they line up properly (for an example, see this forum thread).

This plugin can also help if you'd like to make infinitely tileable terrain from non-tiling heightmaps. Just select 'Operations→Heightfield→Edge blending→Wrap X and Y axes' in the menu, and the atEdgeBlender plugin will wrap the east/west and north/south borders on a heightmap so that they line up exactly.

If you're interested in scripting, the ZeoScript extension function for the edge blending calculation is calc.HF.BlendEdges, as described here. An example of this function in use is the cube wrapped heightfield script.

The feedback for the L3DT's mesh exporter is generally quite positive, but users have reported a few annoyances to do with tiled mesh exports, which have now been fixed in this release. The first was to do with how the mesh was optimised so that there were no seams at the borders between adjacent tiles. Whereas previous release used up a lot of unnecessary triangles in making sure edges line up, the new release has a more intelligent optimisation routine that uses far fewer triangles and still produces seamless borders. If you're interested, please have a look at this blog post, which shows the before and after shots.

The other change to the mesh exporter is the option to automatically resize the texture images for the mesh to the nearest power of two, as required by some game engines and renderers. It's a little thing, but it will cut out a heap of tedious image resizing for some users.

Another welcome addition in this release is the ability to modify the land type coverage in the attributes map using mask images. This feature can be used to 'bake' specific shapes or patterns onto the attributes map, or to re-calculate specific areas of the attributes map without clearing the whole map.

This is an example of changing the climate from temperate to arctic using a mask image over a selected area:

Attributes map (before, note selection)	Mask image	Attributes map (after, in 2D)	Attributes map (after, in 3D)

Note that for the pixels where the mask image was white, L3DT has re-calculated the attributes map land types using the arctic climate, resulting in snow (white) where it was once grass (green), and keeping the cliffs as cliffs (albeit the arctic climate versions). If we instead select the 'desert' climate, we get sand instead of grass/snow:

Attributes map (using desert climate)

The mask image doesn't have to be square, and the selected area (if used) also doesn't have to be square, or even the same aspect ratio as the mask, as L3DT will do the appropriate stretching for you. The mask image can be generated in any program (including L3DT's mask generator), and should be stored as 8-bit bitmap or PNG files. For more information about the mask overlay operation, please refer to the users' guide. Please also note that this feature is available as a script function, and as a filter graph in ZeoGraph (see atCalc_AM_MaskOverlay plugin page).

There are two big areas of improvement for the Torque T3D exporter in this release. One welcome change is that L3DT now supports setting the detail texture in T3D's materials, which means the terrain looks a little nicer out of the box:

Detail map in T3D

The other potentially interesting addition is the experimental support for exporting maps as multiple terrain blocks. This means you can now have T3D terrain that is larger than the usual T3D terrain size limit of ~4096×4096 pixels per block. However, please be aware that - last time I checked - the texture coordinate mode in T3D wraps the texture around at the tile edges, producing unsightly texture seams. There's nothing L3DT can do to fix that; it has to be fixed in the T3D rendering environment. I leave that as an exercise for the user.

There are a few other nice-to-haves in the updated T3D exporter, including:

• Texture DDS images are automatically resized to the nearest power of two.
• The default spawn point can no-longer be under water. The default spawn point, which is normally at the centre of the map, is now moved to the nearest land if the centre of the map is under water.

The major bug fixes in this release related to the usage of memory and CPU, which was discussed above. A goodly number of other bugs were also fixed in this version, as listed in the change log. If you were waiting for a bug fix that isn't on that list, please file a bug report in the forum or by e-mail to aaron@bundysoft.com, and I'll get right onto fixing it.

Plugin developers should note that this release of L3DT supports a new version of the Zeolite plugin API (v16.01). Plugins that use direct RAM access to map data via the zmap_GetDataPtr function or the CzMap::OpenDirectInterface / CzMapTile::OpenDirectInterface functions need to be updated, as L3DT now occasionally allocates maps and map tiles as non-contiguous arrays of scanlines (see zmap_IsContiguous, zmap_GetScanlinePtr). If you have any questions about updating to the new plugin API, please feel free to e-mail aaron@bundysoft.com.

To download the update, please follow the instructions below:

Please let me know what you think of the new release. Comments, requests and bug reports are all welcome, as well as suggestions for the next release. Please feel free to make use of this forum thread.

I'm going to hold off on any major development for a month or two, just in case any bugs are found that require a quick follow-up release. During this time I'll be listening to feedback, fixing any bugs as they are reported, tinkering with a few plugins, and generally collecting suggestions for the next build.

Best regards, Aaron.

Hi Everyone,

As part of the validation process for the forthcoming L3DT release 16.0X, I thought it might be fun to generate a gigapixel terrain map; i.e. one with 32k x 32k heightfield pixels. In the past when I've generated large terrain maps of this size, the objective was usually to verify that various algorithms in L3DT could handle large maps. I didn't care at all whether the results looked any good or not, as that was normally tested and tweaked much more quickly with smaller map sizes. Consequently, previous large terrain tests have often used settings optimised for speed, not quality, such as little or no erosion, no flooding of water table, etc.

This time around, I wanted to look at the results in the Sapphire 3D renderer to make sure it properly supported large maps. Since I was actually going to look at the results, instead of just verifying that the algorithms didn't crash, I thought I might make a little more effort to produce something nice. Erosion was enabled (at 30%), the water table was flooded (that took two days), and the attributes map, normals map, light map and texture maps were all generated at 2x resolution, meaning each was 64k x 64k pixels in size, or 4 gigapixels each. All up, the project is 82 gigabytes of raw data, which mercifully is only 29 gigabytes on disk thanks to file compression. It took about a week to generate on an old quad core box with only 4GB of RAM, but generate it did, and no bugs were encountered.

Anyway, here's a quick screenshot in Sapphire showing an aerial view from around 530m / 1700ft above ground level, with a horizon distance of 31km / 19 mi:

The view from 1,700ft.

Now, I'll freely admit that the image isn't stunning; without vegetation, roads or structures, you can easily see that the terrain is not particularly detailed (10m mesh spacing), and the texture is a little coarse (2x res only). This map represents the sort of level of detail you might find in a flight-sim, not an FPS game.

However, what was of interest to me what what happens when you fly a little higher, with a longer horizon distance. This is the same map, at the same location, from an altitude of 7000m / 23,000ft, with the horizon pushed out to 108 km / 67 mi:

The view from 23,000ft.

Now it's starting to feel like a large map. Note how the mountain that looked quite large in the the distance of the first image (indicated by red arrow below left), now looks rather puny in the second image (indicated below right):

1,700ft altitude, again.	23,000ft altitude, again.

I only really grokked the full scale of the map when I went looking for the same features in the 2D view. Note how the area we've been looking at, from the lakes to those those really distant mountains (purple arrow, > 100km away), is just a small part of the southwestern corner of the map.

The 2D view, same features marked (arrows at bottom left)

Truly, a 32k x 32k heightfield is a stonking big map to run around on.

Anyway, the take-home message from all of this is that L3DT 16.0X seems to work just fine with gigapixel maps, and Sapphire works fine too.

There should be a disclaimer, however; for the really long view distance shot (108km horizon), I was trying for a nice image, so I had both the geometry turned way up (370k triangles), and the texture level-of-detail bias disabled, so the frame rate dropped to only 0.7fps on account of all the the heightmap, water map and texture data that Sapphire had to process to render each frame. I could still move around, but only barely. This post should therefore not be interpreted as meaning Sapphire will be snappy when viewing so much data all at once, only that it can do it without crashing. That said, when I turned the horizon distance back to normal (from 100km to 10km) and the triangle count back to normal (from 370k to 32k), the frame rate bounced back to normal too (~10fps, on my ancient system.) With reasonable horizon distances, there is was no appreciable performance penalty of moving around on a 32k x 32k map, as compared to, say, a 'tiny' 2k x 2k map. Sapphire seems to scale well, which is nice.

Anyway, that's enough for now.

Cheerio, Aaron.

Hi Everyone,

In January of this year L3DT Professional gained the ability to allocate in-RAM maps as non-contiguous arrays of scanlines. This change eliminates memory fragmentation problems, and allows L3DT to operate much closer to system memory exhaustion without allocation errors. Unfortunately, this change also required an update to the 'Zeolite' plugin API:

• The function zmap_GetDataPtr now fails if the map is non-contiguous.
• The zmap_IsContiguous function was added to determine if the map is allocated as a contiguous block of RAM, or if it is instead allocated as an array of scanlines.
• The plugin function zmap_GetScanlinePtr may be used to directly get the memory pointer for an individual scanline in a map.
• The plugin function zmap_GetScanlineArray may be used to retrieve the array of pointers to all the scanlines in a map.

Likewise, if the map is allocated as a mosaic map, the tiles in each map may also be allocated as non-contiguous scanlines. Hence:

• The function zmap_tile_GetDataPtr now fails if the mosaic tile is non-contiguous.
• The zmap_tile_IsContiguous function was added to determine if the mosaic tile is allocated as a contiguous block of RAM, or if it is instead allocated as an array of scanlines.
• The plugin function zmap_tile_GetScanlinePtr may be used to directly get the memory pointer for an individual scanline in a mosaic tile.
• The plugin function zmap_tile_GetScanlineArray may be used to retrieve the array of pointers to all the scanlinesin a mosaic tile.

The direct access interface functions in the CzMap and CzMapTile classes have been updated accordingly. Furthermore, the CzMap::AsContiguousMap and CzMapTile::AsContiguousMap functions were added to allow plugins to create contiguous copies of non-contiguous maps or map tiles.

The updated plugin API is available here. Note that the v16.01 plugin API will not work with versions of L3DT older than v16.01, and hence the older plugin API is still available on the downloads page.

Please address all feedback regarding this update to aaron@bundysoft.com, or use the Plugins and scripts forum.

Best regards, Aaron.