L3DT development blog
Large 3D terrain generator

July 12

Ten years of L3DT - Part II

In the previous instalment of the Ten years of L3DT retrospective, we looked at L3DT releases 1.0 to 2.1, in which the whole map generation pipeline of L3DT was more or less developed. Today, we'll look at releases 2.2 and 2.3, which focussed on improving the user interface, improving calculation speed, and generally making L3DT more usable and useful.

v2.2, October 2004

Release 2.2 was a few steps forward with one step back. The step back was that mosaic heightfields were disabled, all on account of a pernicious bug in the mosaic water flooding algorithm, which wasn't fixed until L3DT 2.3. This bug cased the 'walls of blue jelly' bug in L3DT 2.1:

www.bundysoft.com_coppermine_albums_userpics_floodbug.jpg
Mosaic water flooding bug, in L3DT 2.1.

Although mosaic heightmaps and water maps were disabled in v2.2, support for mosaic light maps and textures was maintained.

The steps forward were many and varied:

  • Multiple climates were supported, much as they are today. This release included the familiar 'temperate' and 'arctic' climates. Alas, there was still no editor to be seen, so the climate files had to be edited manually using a text editor. A few intrepid users did master the dark art of manually editing climate files, but most were gravely injured in the attempt.
  • Erosion was added, producing much more realistic heightmaps (see below).
  • The water map flooding tool and water table modelling calculations were introduced. These allowed, for the first time, the ability to place mud or sand under and around lakes that weren't at sea level.
  • There was a big leap in the speed of all calculations, owing to me finding the 'optimisation' option in the compiler. Previous builds of L3DT had been compiled in 'debug' mode, would you believe.
  • L3DT supported its first compressed image file format (JPEG), and its first mesh file format (DirectX '.X'). Mind you, the mesh export was still a brute-force tessellation of all heightmap vertices, as the mesh decimator wouldn't be available until L3DT v2.9.
  • The batch generator was included. In hindsight, this feature has rarely been used, but it did force me to develop a better model for how settings were stored and recalled, and this later influenced the design of the settings/project XML files and the plugin API.

The result of these changes was that the terrain output from L3DT started to look like it could actually be used for games:

www.bundysoft.com_l3dt_gallery_r22_alps_cgalps1.jpg

After release 2.2 there were three revisions, 2.2a, 2.2b, and 2.2c. The only significant change from these revisions was that all map calculations were put in a worker thread, which meant that the program's user interface didn't lock up long calculations. This didn't make the program any faster, but it did at least give the user some confidence that the program hadn't hung. This change also allowed users to view the progress of calculations, and to cancel calculations, which was much appreciated and long overdue.

At this time, the very modest popularity of L3DT was too much for the truly pitiful download limits on GeoCities, so the website was moved to a new host, my then-ISP, TPG. The website was still laboriously written in static HTML though, as my subscription didn’t support PHP, ASP, or other server-side scripting. This became a bit of a limitation, and the website would soon move on to even grander digs for the next release (v2.3).

v2.3, June 2005

In the interests of simplicity, I'll group together the changes from L3DT release 2.3 with the four revisions that were released in the subsequent seven or so months (2.3a, b, c and d). Some of these revisions were pretty substantial, so if your interested in seeing which revision changed what, please follow the previous links.

User interface

Version 2.3 introduced a few changes to the user interface that should be immediately familiar to users of 'modern' L3DT. Firstly, version 2.3 introduced the 'wizard' user interface, which replaced a set of rather clunky control dialog boxes. There were a few teething problems with the initial wizard implementation, including no support for display DPIs other than 96, but these were fixed fairly quickly. Although some wizard panes would be tweaked, they are fundamentally the same now as they were then.

Making a new map in L3DT 2.3, using the wizard.

Version 2.3 also introduced the 'next' toolbar button, which was a direct precursor to the 'calculation queue' wizard that is in modern L3DT. This button looked at which maps had and had not been generated in the project, and opened the appropriate wizard pane for the next calculation in the process.

This release also introduced the 'fancy' threaded progress window, which included a live preview of the map output as it was begin calculated:

The calculation progress window in L3DT 2.3.

Hight and water maps

There were also some changes to the way L3DT generated heightmaps and water maps:

  • Thermal erosion was added to the heightmap generator. This more or less completed the 'design/inflate' heightmap generator, and there have only been tweaks since then.
  • A 'Perlin' heightfield generator was added. At the time I had the expectation of adding even more heightfield generator options, like ridged multi-fractal and the like, but there wasn't much demand from the users' community, so the plans remain on the shelf.
  • The speed of the water table modelling algorithm was improved by optimising the raster order. Like the height map generator, the water map generator in L3DT has changed only a little since this release.

Light, texture and alpha maps

In this version the light and texture map generators were re-written, and the alpha map generator was added. The light map generator received support for high-resolution, bump-mapped light maps with shadow anti-aliasing, which produced much more interesting results for cliffs and other rough land types (see below). The texture generator was modified to calculate the land type for every pixel when generating high-resolution textures, rather than just interpolating the land type as was done in previous releases. This resulted in much sharper transitions from one land type to the next, but was also quite a bit slower. Both the light map generator and the texture generator would be revised again in future releases (v2.4, v2.5 and v2.8), but the implementation in v2.3 was getting pretty close to the 'final' algorithms that we still use today in modern L3DT.

bumpshot3.jpg
The high-resolution bump-mapped light maps and textures of L3DT 2.3d.

Mosaic maps are back!

For version 2.3 I finally found and fixed the bug in the mosaic water flooding algorithm that caused the 'walls of blue jelly' problem, so I re-enabled mosaic map support for heightmaps and water maps. I also modified the design to use 64-bit addresses, which increased the size-limit for mosaic heightmaps to 128k × 128k pixels (16 Gpxls), and the size limit for texture/light maps to 4M × 4M pixels (16 Tpxls). With the return of mosaic height maps and water maps, L3DTVi2 was rebuilt by Stuart to support mosaic maps, and included a nice water texture:

The water texture in L3DTVi2 for L3DT 2.3.

File format changes

In this release, file format changes weren't very significant. The PNG image format was added, as was a RAW exporter for the heightmap. Some of the historical, proprietary formats used by L3DT were removed (LMF, TNF, etc.), and were replaced by either PNG or JPEG. Some settings that were stored in the map group file (the precursor of the project file) were split out to a separate map definition file, which could be run by the batch generator to make another map with the same settings. The map definition file, appallingly enough, used a text format in which all the settings were declared in a primitive home-baked scripting language, so that the file was executed by L3DT to load the settings. This nasty kludge would be fixed in the next major release, v2.4.

Scripting, really?

I found it amusing to note that the development plan for version 2.3 listed support a scripting language. Sadly, release 2.3 did not support scripting (other than the crazy settings files), but this feature remained on the development plan until delivered in release 2.5 with the 'cdPython' plugin (by Carl Douglas) and the 'ZeoScript' (by yours truly).

L3DT Professional

L3DT was split into 'Standard' and 'Professional' versions with L3DT v2.3b. Although the Pro version was still free to use at that time, the groundwork was being laid for the commercial sale of L3DT Professional that began with L3DT v2.4.

Installer

Would you believe that version 2.3 was the first release to have an installer? Prior to v2.3, the user had to unpack the zip archive and copy the program files manually, which wasn't so flash. My memory is a bit hazy, but I seem to recall that for version 2.3 I used some installer system from Microsoft (possibly Visual Studio?) to make an MSI installer. It turned out to be a little quirky, including - if I recall correctly - requiring the same drive mappings on the user's computer as was present on the system that compiled the installer (i.e. my computer). This wasn't great, so in release 2.3a, I changed to the Nullsoft Scriptable Install System (NSIS), which I still use today, and with which I am quite satisfied.

Website

The website was in quite a state of flux in the v2.3 era. Following v2.2, the amount of user feedback and questions I had been receiving had grown such that I needed a public forum to list questions and answers. For v2.3, I introduced the L3DT Yahoo! users group mailing list. There were a few really good discussions there, but it was not to last. Before even version 2.4 was released, I had moved the whole website to the new domain at bundysoft.com, and replaced the mailing list with the current forum, wiki, and image gallery, which have proved much more satisfactory. In other web news, the now-venerable fjord tutorial was added, giving users the first proper start-to-finish guide to using L3DT.

Coming up next...

Well, that's it for the v2.3 story. By this point, L3DT had moved from being fairly limited, crude, and unpolished, to being much more powerful, relatively refined, and generally useful. Stay tuned for the next instalment of 10 years of L3DT, in which L3DT goes on sale, hooks up with GarageGames, gets a new 3D renderer, gets a whole host of plugins, learns how to do 8 things at once (multi-core multithreading), and generally gets pretty awesome.

Until then, ciao, Aaron.

2013/07/03 02:09 · aaron
2013/07/12 05:54 · aaron

July 3

Ten years of L3DT

Hi Everyone,

L3DT version 1 was released ten years ago today, and it seemed to me like a good opportunity to take a stroll down memory lane to look at how the program has changed over the years, to reminisce about the more quaint aspects of the old user interfaces, and to chuckle at just how wildly inaccurate some of my development plans and projections were.

However, before I get all misty eyed talking about long-since deleted dialog boxes and the like, I should first like to thank all the users who have contributed to the development over the years, either by reporting a bug, requesting a feature, purchasing a license, asking or answering a question in the forum, telling other people about L3DT, or by helping the project along in other ways. Thank you all, I couldn't have done it without your encouragement and support.

Over the next week or so I'll post about each of the 14 major releases of L3DT to date, covering a couple of releases per post. I'll describe the significant changes, and cast a critical eye over what I got right, and not-so-right. Today's instalment covers the first year (and first four releases) of L3DT, in which the fundamentals of L3DT were established, including the heightfield and texture generators, and the whole processing pipeline.

Users of L3DT Professional can play along at home, as every one of the old releases of L3DT are still available for download from the 'archives' page on the L3DT Professional downloads site (see your reg/receipt e-mail for the link).

We begin, appropriately enough, with version 1:

v1.0, July 2003

The genesis of L3DT was, I think, a year or two before the first public release in July of 2003. The back story was that I'm a bit of a fan of real time strategy games like Total Annihilation, and thought I'd try my hand at making an RTS with really large 3D terrain maps. Something like Supreme Commander, only a few years beforehand. However, I fairly quickly found that the task was well beyond my skills and available time, so I focussed on a smaller part of the project: how to make large, realistic 3D terrain maps. By July 2003, I was (apparently) satisfied with what I had done thus far, and so proceeded to upload L3DT version 1 to my GeoCities website. Incidentally, that site was my very first experiment with HTML, and it showed (see below). Fortunately, I had a lot of help from my good pal Stuart Gooding, who is a web wizard.

L3DT website in 2003

L3DT release 1.0 had a remarkably basic user interface. Everything was menu-driven, and there were no manual editing tools, nor wizards, progress boxes, or any other mod-con that one might expect. This was one of my first Windows applications, and I clearly still had much to learn (and still do.) However, behind the crudely crafted dialog boxes of L3DT release 1 was a program architecture and work flow that would be familiar to L3DT users today. As with modern L3DT, maps were generated from a processing pipeline that started with a design map and ended with a texture map, via the heightfield, water map, attributes map, normals map, and light map. Some map names were different back then, but the contents and concepts continue through to today's version.

Creating a new map in L3DT 1.0 was a simple affair, with only a few settings to choose:

Creating a map in L3DT 1.0

The default map size was 1280×1280 heightfield pixels, and the maximum size was 12800×12800 heightfield pixels. It was even possible to generate larger heightfields using a very early implementation of mosaic maps. Mosaic maps were supported when you used the 'large map' generator, which was sort of like an even larger-scale design map for the design map. If you made a 'large map', the maximum final heightfield size was theoretically 51200×51200 pixels. I doubt anyone ever tested this limit, given the appalling speed of the v1.0 heightfield generator, and the relatively poor processor speeds at the time.

The heightfield generator in L3DT v1.0 was quite different to that in 'modern' L3DT; it simply added together thousands/millions of peaks to build up the height map, where the number and size of peaks in a region were determined by the design map. This was phenomenally slow. Worse still, it did not make very realistic terrain, since it didn't calculate erosion, cliffs or terraces, nor fractal noise. A typical heightfield, as shown in the 2D view, looked like this:

The heightfield display in L3DT 1.0

As with all other versions of L3DT, the colours in the texture were determined by the coverage of land types in the attributes map. However, in v1.0, solid colours were used for each land type, as texture images weren't supported until v2.1. Also, there were only four land types to be used (grass, sand, rock and snow), and they were all hard coded. The climate editor wouldn't arrive until v2.4. Typical texture output from L3DT v1.0 looked like:

The texture output of L3DT 1.0

One of the more popular features of L3DT these days is the broad range of supported file formats. This was not so in L3DT v1.0. All maps were stored in custom binary file formats like HFF. Texture maps could not be loaded or saved as part of the project, but could be exported as 32-bit bitmaps.

L3DT did come with a standalone 3D viewing application called 'L3DT Viewer', also written by Stuart Gooding, using Delphi and the GLScene library. It did a really good job of making my dull maps look interesting:

Stuart Gooding's 'L3DT Viewer'

That's about it for v1.0. Version 1.5 would soon follow…

v1.5, July 2003

Version 1.5 was released five days after v1.0, but was a major improvement. This update was much faster at generating heightfields, thanks to the use of a new interpolation routine that generated the heightfield from the design map. This wasn't quite the same as the fractal inflation routine used today, but the concept was similar. This version of L3DT also introduced fractal noise, cliffs and terraces, and mountain overlays. The net result was much more realistic terrain than the previous version, although the textures were still a bit rubbish:

This version was the first to allow manual editing of the design map (then called the 'intermediate map'). However, editing was extremely laborious, using the edit design map pixel window to edit individual pixels one by one. Although editing the design map now uses a brush tool, the old pixel editing window is still used today (with minor modifications, see below), and is the oldest control window that is still in L3DT.

Editing design map pixels in v1.5 Same window, ~10 years later (L3DT 13.04)

This version also added progress bars for calculations. However, these early progress boxes were pretty simple, and niceties such as cancel buttons and time estimates would have to wait for release 2.2a. Incidentally, the original progress boxes are still in use with the current release of L3DT when loading and saving map files.

Otherwise, the user interface of v1.5 was pretty similar to v1.0.

v2.0, August 2003

L3DT version 2 again improved the heightfield generation speed, and introduced the 'fractal inflation' algorithm that is more or less the same as is still used today (but without the erosion, which has to wait until v2.2). This version also removed the 'large map' generator from v1.0, which had never been developed to a stage where it was actually useful. This also meant that mosaic map support was dropped, to re-appear in v2.1.

Version 2 introduced the first brush-like tool window for editing maps; the 'design map pencil' (see below), which was later to become the design map brush still used today. This tool allowed users to paint changes onto the design map, albeit with a 1-pixel brush size.

The 'design map pencil' tool in L3DT 2.0

Shadow casting and water absorption effects also debuted in v2.0, producing more realistic and pleasing light maps:

Water effects in the light map

Finally, this version changed the layout of a number of the proprietary binary file formats, and generally broke backwards compatibility. Clearly, I still got some of the formats wrong, as they were changed again for v2.1 (which incidentally is largely compatible with the current release.)

Looking back, L3DT 2.0 was still clearly a long way from being particularly useful for game development. It could produce what were, at the time, large heightfield maps relatively quickly, but it as very difficult to plan out and design a heightmap, which is kind of a deal-breaker. The texture generator was also still quite primitive, using only solid colours for the different land types. That would change in v2.1…

v2.1, November 2003

Version 2.1 was a major improvement in a number of areas, and was - dare I say it - at the cusp of being useful for game development. Firstly, the texture generator was upgraded to use texture images for each land type, rather than just solid colours. Furthermore, the texture generator would now produce 'high resolution' textures, with greater detail than the heightmap. The results were pretty respectable at the time:

Texture output of L3DT 2.1.

Keen-eyed users may also notice that there were a few new buttons on the toolbar. Apparently by this release I had worked out how to use the resource editor in Visual Studio to change the toolbar, but judging by the number of greyed-out default buttons (cut, print, etc.), I still had more work to do. Nonetheless, progress is progress.

Version 2.1 re-introduced mosaic maps, which had been dropped in v2.0. This time around, they worked properly, and allowed maps to be generated that were too big to fit in RAM at once. The original size limit was 2 gigapixels (i.e. 32k × 32k). Mosaic maps were supported for the heightfield and all derived map types, including the texture map. Since big maps were now possible, the map display in L3DT 2.1 also now included the ability to zoom.

L3DT 2.1 still didn't have a climate editor or support for multiple climates, but the climate was now loaded from a file (standard.cli) rather than hard-coded, so it could be edited using a text editor if you were determined.

Useful file format options were added to L3DT for the first time with v2.1, including import and export to TerraGen's TER file format, the Virtual Terrain Project's BT file format, and bitmap images. The heightfield and water map file formats (HFF and WMF) were also updated, resulting in the file formats still used by L3DT.

Due to all the other changes in L3DT's file formats, Stuart Gooding released an all-new 3D viewer, entitled L3DTVi2. It was nice, and would continue to be used until Sapphire was introduced with L3DT v2.5.

L3DTVi2, the 'new' 3D viewer for L3DT 2.1, by Stuart Gooding

Stay tuned

In the next instalment of 'Ten years of L3DT', we'll cover L3DT release 2.2 and 2.3 (2004-2005), in which L3DT got threads, wizards, optimisations, an installer, alpha maps, a new website, and much much more. Stay tuned…

2013/04/29 04:52 · aaron
2013/07/03 04:29 · aaron

April 28

Updated heightfield tool window

Hi All,

The heightfield tools window in the Sapphire 3D renderer has been updated in the latest build of L3DT professional (v13.04 build 1, dated 28th of April 2013). The main change is that the tool settings can now be edited in a non-modal properties table, as shown below:

The new settings pane for the heightfield tool window.

Furthermore;

  • all tools allow you to edit the radius (or width, for bulldozer/leveller/etc.) of the tool brush, which previously was only controlled by mouse scroll wheel,
  • the camera zoom is now controlled by the page up / page down keys, meaning you can now zoom when in edit mode, and;
  • the sticky keys bug in Sapphire has been fixed.

If you find any problems with the new tool window, please report the faults in the bug reports forum (preferred), or e-mail aaron@bundysoft.com (not preferred.)

Cheerio, Aaron.

2013/04/28 06:32 · aaron
2013/04/28 06:34 · aaron

March 3

Bug fixes for multi-threading and memory management

Hi Everyone,

In September of 2011, L3DT received a major speed improvement thanks to a smarter way of managing when mosaic map tiles are paged from disk to memory and back. Unfortunately, that 'smarter' cache manager had a few subtle bugs that caused L3DT to run really slowly for certain calculations on 'large' maps, or even lock up entirely. Apparently these bugs didn't affect many users, but thanks to a bug report from Demi, they have now been tracked down and fixed.

The bug fix is included in L3DT Professional 13.03 developmental build 0, which is on the downloads page now (consult your sales/registration receipt e-mail for the link.) I would recommend this update to all users that generate 'large' maps (larger than 8k×8k pixels).

Cheerio, Aaron

2013/03/02 23:06 · aaron
2013/03/03 05:06 · aaron

<< Newer entries | Older entries >>

 
start.txt · Last modified: 2014/01/16 04:59 by aaron
 
Except where otherwise noted, content on this wiki is licensed under the following license:CC Attribution-Share Alike 3.0 Unported
L3DT Development Blog RSS Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki