News for March 2011

Hi Everyone,

[This post is aimed at plugin developers. It probably has no relevance to other audiences.]

Ever since the Zeolite plugin API was introduced in 2006, there has been a slight problem. It's slow. Well…not exactly slow, but slower. The same calculation written in a plugin runs more slowly than the same calculation written into the L3DT executable. This is because the plugin API - which plugins must use to access map data - performs some additional translation steps and safety checks that aren't required for code in the executable.

But no more…

In the latest update to the Zeolite plugin API (v2.9.6), I have included a new direct interface for maps and buffers that can potentially increase the speed of calculations in plugins by as much as 5x. The direct interface, which is implemented in the CzMap, CzMapTile (new!), and CzBuffer classes allows plugin-side code to directly access map and buffer memory without the overhead of calling plugin API functions.

The upshot is that, for suitable well-written algorithms, calculations performed by plugins can be as fast (or faster) than the same code running in the L3DT executable.

However, there is a downside to this interface; in fact there are several. Firstly, the direct interface does not do many useful housekeeping actions that are automagically performed by the conventional API functions, including:

1. Updating mipmaps,
2. Updating minimum / maximum altitudes in heightfields, and;
3. Marking mosiac tiles as changed after setting pixels (thereby forcing the tile to be saved when released).

Consequently, if plugin developers are changing pixel values using the direct interface they must:

1. Clear or update mipmaps,
2. Update minimum / maximum altitudes for heightfields, water maps and design maps, and;
3. Mark tiles as changed before releasing mosaic tile handles.

Note that these limitations do not affect reading pixel values; these interfaces are generally safe for reading, but more tricky for writing. Of course, you can use the direct interface to read quickly from one map, and the conventional API functions to write safely to another map, all at the same time.

The speed improvement of this interface is in part due to the fact that it skips all safety checks. Thus, it is extraordinarily easy to crash the application when improperly using this interface.

For example, range checking is omitted when getting or setting pixel values. The conventional API functions check the x and y coordinates given to a SetPixel / GetPixel call to ensure they're valid, and throw error messages if they're not. However, in the direct access interface there is no range checking, so invalid x or y coordinated will overrun the data buffer, dance a fandango on core, and crash the application. The risk is nil, of course, if the algorithm is well written and doesn't pass invalid coordinates in the first place. Thus, the direct access interface is only recommended for optimising rock-solid code that has already been thouroughly tested.

To use the direct interface, plugin developers must re-compile thier plugins and insert calls to the OpenDirectInterface member functions of the relevant CzMap, CzMapTile and/or CzBuffer objects. After calling OpenDirectInterface, the interface lasts only for the lifetime of the CzMap, CzMapTile or CzBuffer object, or until you call the CloseDirectInterface member function. Thus your calculation may use the direct interface when safe to do so, but not use it on other calculations where it might not be safe.

This interface will initially be rolled-out to select calculation plugins, as these are easiest to modify and will benefit most from such optimisations.

The first plugin to be updated was the atConvertWaterToHeight plugin. The calculation in this plugin gets input water map pixels and sets output heightmap pixels, with very little calculation in between, and thus is a good test of the pixel access speed. For this plugin, the effect of using the direct access interface was a speed-up of in-RAM maps of about 5x, and for mosaic maps of about 2x. The lesser improvement for mosaic map calculations is due the fact that mosaic calculations must perform disk paging, which is not affected by these optimisations.

The atCalc_AM_Image plugin has also been updated. This plugin is principally used for updating the terrain colour when painting with the attributes map brush in Sapphire. In this case the speed improvement was 30-40%. This improvement is less dramatic than with the atConvertWaterToHeight plugin because this calculation is less bound by simple pixel read/write speeds, and also because the optimisation applied to reading the the input values but not writing the output values. Furthert optimisation is possible here.

More complex plugins like the Sapphire 3D renderer may eventually be optimised thusly, but that may be some time away.

These changes are available Zeolite v2.9.6, which is available here. Please note that this API version is compatible with L3DT v2.9 build 12 or later.

If you would like to provide feedback on these changes, or any other features of the new plugin API, please feel free to reply to this forum thread.

Over and out, Aaron.

Hi Everyone,

On the downloads pages you may now find L3DT v2.9 build 11, in both the Standard and Professional editions. This developmental update includes a couple of bugfixes and some new features, as outlined below.

A week or two ago, some users in a not-to-distant corner of the internets were discussing troubles with L3DT's mesh export files. Specifically, they found that L3DT was exporting 3DS and OBJ mesh files with inverted normals. Of course, this was because they were using a renderer that uses a left-handed coordinate system whereas L3DT uses a right-handed coordinate system. Fortunately, Google Alerts notified me of this dire situation, and I've modified all the mesh exporters to allow mesh normals to be flipped during export. This applies to the 3DS, OBJ, X and B3D mesh exporter plugins.

To make use of this option in L3DT v2.9 build 11, go to the 'File→Format preferences' menu option, and navigate to the 'Project maps→Heightfield' tree option. Then, select the mesh format you wish to use (e.g. 3DS, OBJ, X, or B3D), and press the 'Options' button. In the options window, double-click on the 'InvertFaceNormals' option to set it to true. After this, you may close all windows and proceed with your export as per normal.

PS: Although I do try to follow L3DT bug reports on other forums via Google Alerts, I would still recommend users report bugs in the official bug reports forum, as you're much more likely to get a timely response and a workaround or bugfix.

The L3DTio_BigBitmap plugin, which provides L3DT with BMP support, has been updated to handle superfluous padding in the data section, which had been found in bitmaps saved by PhotoShop.

Menu options can now be added to L3DT using script files, whereas previously this could only be done using a compiled plugin. Menu options are defined in ZMENU files, which are stored in the following directory:

The file(s) in this directory contain one or more menu definitions, which are defined using the ZeoScript language. An example file is given below, in which I show how the 'Operations→Heightfield→Invert' menu option is defined, and following the example file that I will explain the meaning of the contents.

VERSION 1

# comment lines begin with a hash char
# comments within script segments may also use the // comment delimiter

MENU OPERATIONS "Heightfield.Invert"

// the action script just runs another script file, which contains the instructions for inverting the heightfield
int rval
RunScriptFile "Invert Heightfield.zs" rval

ONUPDATE

hvar hmap
set hmap <GetMap "HF">
if <not hmap>
  // disable the menu option if we can't get the heightfield
  set MenuEnable false
  return -1
endif

if <not <map.GetWidth hmap>>
  // disable the menu option if the heightfield has zero width (i.e. is uninitialised)
  set MenuEnable false
  return -1
endif

if <calcman.IsBusy>
  // disable the menu option if the calculation manager is busy
  set MenuEnable false
  return -1
endif

// otherwise, enable the menu option
set MenuEnable true
return 0

END

#more MENU definitions can follow here...

Each menu item beings with a MENU tag, followed (on the same line) by a menu context (e.g. OPERATIONS), and the name of the menu item (e.g. “Heightfield.Invert”, where '.' delimits submenus). The following menu contexts are supported here:

The menu item action script begins on the line following the MENU tag. This script contains the commands to be executed when the user selects the menu option. The script is terminated by either an END tag, or by an ONUPDATE tag if the menu definition includes an event handler to modify the menu item display state.

The ONUPDATE event handler script, if defined, is called as L3DT is preparing to show the menu. This script may change the menu states for enable (true = enabled, false = greyed/disabled), check (0 = unchecked, 1 = checked, 2 = indeterminate) and radio (true = radioed, false = unradioed). The ONUPDATE script must be terminated by an END tag. Thereafter, another MENU may be defined. Each file may contain any number of menu definitions.

If you have any questions or comments about any of the above changes, please feel free to make use of the forum.

Best regards, Aaron.