My own opinion...

[mauronen]

Hi Aaron.

Resuming the discussion started in New 3D renderer thread i try to give my 2 cents contribution.

This is an UML Component Diagram that sketchs my view of how L3DT could be:
http://www.bundysoft.com/coppermine/displayimage.php?pid=183&fullsize=1

Briefly, L3DT is a plugin coordinator that manages their artifacts through a Workflow Manager.
L3DT exposes himself to the world through a CPI (Common Plugin Interface) that allows external plugins to "plug" yourself to L3DT, in a way that they can work in symbiosis. Each plugin produces an artifact that is took in load by L3DT that loads an artifact's stack. L3DT's Workflow Manager dispatch each artifact to own manager (Heightfield artifact is managed by an Heightfield Manager, Water artifact is managed by a Water Manager and so on.
The Entry Point is a Design or an Import artifact.
The Exit Point toward a renderer is a Landscape artifact.

CPI is a set of interfaces that are the "card" of L3DT to the world.
They are:
CPFII = Common Plugin File Import Interface
CPFEI = Common Plugin File Export Interface
CPHCI = Common Plugin Heightfield Creation Interface
CPHFI = Common Plugin Heightfield Filter Interface
CPWI = Common Plugin Water Interface
CPCI = Common Plugin Climate Interface
CPVCI = Common Plugin Vegetation Coverage Interface
CPOPI = Common Plugin Object Placement Interface
CPSI = Common Plugin Sky Interface
CPLI = Common Plugin Light Interface
CPEEI = Common Plugin Environment Effect Interface
CPLEI = Common Plugin Landscape Export Interface

Naturally this is my personal opinion. If you consider this argument interesting, then i will continue. Otherwise feel free to drop this thread.
I'm an IT architect (specialized in J2EE environments), but developer too and i know that is very easy to trample a developer's pride.

Cheers.

Mauro.

[Aaron]

Hi Mauro,

I'm sorry I haven't replied to this yet. I'll try to get my thoughts on all of this together on the weekend (so much to think about too!)

Best regards,
Aaron.

[Aaron]

Hi Mauro,

Thanks for your post and schematic. This has certainly raised a few good ideas for future directions.

From a system architecture point of view, I can certainly see what you're getting at. However, I'm a bit cautious at the moment to specify so many plugin interfaces because I’m not yet sure what they’ll do, and what will and won’t be required. Presently I have two classes of plugins; a general plugin, and a file I/O plugin. The only difference is that file I/O plugins expose a few functions to load/save map files (ExtSaveMapFile, etc.), but are otherwise the same as general plugins.

Many of those interfaces you've specified I think could be (or rather, will be) combined into the generic interface of all plugins. For instance, a filter plugin (such as atSphericalDistort) adds an option to the menu, which when selected calls a function in the plugin that opens a simple UI, retrieves some maps from L3DT, does some calculations, then re-draws L3DT's main window to show the changes. A file export plugin such as the 'atExportArea' plugin I'm currently developing also adds an option to the menu, which when selected retrieves a list of file formats from L3DT, opens a file dialog, copies an area of a map to another map, and saves that map using L3DT's map saving routines (which incidentally call other plugins to do this.) These two examples do very different things, but they both use the same interfaces, and because they are both 'self contained,' L3DT doesn't need to know anything about what they actually do.

This 'do what you will' plugin architecture will, I think, work for the vast majority of plugins. However, I probably will have to define some conventions for plugins for, say, mouse tools (which will need callbacks), or terrain algorithms to be included in the 'algorithms' wizard (which will need a specific set of functions.) However, these requirements will be ‘weak’ insofar as to add a mouse tool the plugin will only have to call an API function like ‘mousetool_Create’ and pass some handles to the appropriate callback functions. There is nothing stopping such a plugin from doing lots of different things also. I could, if I so desire, bundle the erosion routine and the design map brush into the HFF format plugin, and they would work perfectly well. By this I mean to illustrate that plugins can do whatever they want, and generally don't have to conform to strict types or classes of plugins.

However, that's not to say that such classifications might not naturally evolve. It might make sense for a number of plugins that do similar things to do them in similar ways, with similar function exports and user interfaces. This could amount to a de facto but non-binding 'plugin interface'. Of course, I'd rather let these things evolve according to the wishes of the plugin developers than specify "you must do this Aaron's way", on the off chance that I might make a silly decision.

Another talking point is the managers you’ve specified in L3DT. I’ve got a plugin manager, and a combined calculation/thread manager of sorts. I suppose a set of functions I have amount to a format manager and a climate manager (though it’s a bit grandiose to call them that.) I’ve certainly got nothing I would call a heightfield/water map/etc manager, though. Hmmm…maybe I should. What sort of functionality would you expect from these managers? I might also add that some of these managers are likely to be implemented in plugins, particularly for non-core stuff like vegetation mapping or object handling.

Okay, now to the things that you plan has inspired me to do:

1) Plugins must be able to add mouse tools. This is quite a timely thought too, as I was planning on implementing heightfield and texture tools in the next release (2.5, that is), so they would give me a good test-case to ensure the API is fully functional.

2) Plugins must be able to define their own data types (a ‘class factory’.) This would allow developers to add their own data types to the API over and above the ones I’ve defined, which could be helpful in doing things like 3D rendering, where you might want a ‘trimesh’ class, for example. I had a plugin-able class factory going in my first attempt at the plugin API last year, so it’s no biggie to add this.

3) Plugins must be able to access the full range of project settings (calculation flags, etc.) This will be required for plugins to add things like object descriptions to the map definition file, or add extra files references to the project file. This is pretty easy to do, fortunately.

4) Plugins might need to be able to add wizard panes. This could be quite hairy, depending on how I do it, so I’ll have to put a fair bit of thought into a sensible implementation, and probably write a few example plugins just to be sure (e.g. push an existing ‘core’ wizard into a plugin.)

5) I’ll probably have to sort out a template in the wiki/docs for the plugin pages so that plugin developers can list, in a consistent way, the functions exported from their plugins (return type, arguments, etc.) This would allow plugins to more easily call on other plugins.

6) Alternatively, I could add a 'function explorer' that will allow developers to browse the available function definitions. I do have this already in my version (have you ever wondered what 'settings->explore all' does?), but a more focussed and easy to use implementation could be a great help.

7) Better documentation in general. On the plugin API documentation page I have a pretty good representation of the available functions, but there are no overviews or tutorials. This will have to be fixed.

Anyhoo, I’ve probably ranted quite enough for now. I hope I’ve not strayed too far from the original theme of your post.

Cheerio,
Aaron.

[mauronen]

Sorry Aaron for long waiting, but actually i'm very busy.
I'll try to reply soon.

Cheers.

Mauro.

[monks]

Very interesting post-

I like the modularity- this is the obvious way to go imo.

I'd also add GIS related tools- projection manager?
Also, (in the distant future of course) a native Google Map export.

monks

[mauronen]

Hi Aaron.

Thanks for your exhaustive answers.

As i said before, i'm an IT Architect and then my architectural analysis is based on patterns (well defined solutions for common problems).
My opinion is that L3DT could be depicted as a sequence of little subsystems that collaborate togheter. Each of them receives an input artifact, process it and produce, as output, same artifact processed, that is an input for subsequent subsystem.
This process could be modeled with two well-know patterns:

Façade Design Pattern:provide a unified interface to a set of interfaces in a subsystem. Façade defines a higher-level interface that makes the subsystem easier to use
Pipes and Filters Enterprise Integration Pattern:divide a larger processing task into a sequence of smaller, independent processing steps (Filters) that are connected by channels (Pipes)

So, my Component Diagram sketch is based on this 2 patterns. In brief:
- each Manager is a little subsystem that expose a unified interface. The subsystem can change or be replaced with another that implements same interface
- a Manager manages a small and independent process (because is a subsystem), but is only a step of a wider process. Each Manager is connected with previous and next Managers by a channel.
- Managers exchanges messages of same type (a landscape)

An scenario example could be:
1- A plugin produces an artifact. It is an empty or processed (by another plugin/filter pair) landscape. L3DT is a subsystem that expose well-know set of interfaces (CPI)
2- L3DT, via PluginManager, intercept this artifact and pass it to WorkflowManager
3- WorkflowManager receives this landscape; it knows process workflow in the chain and, therefore, is responsible to drive the wide process
4- WorkflowManager pass landscape to subsequent FilterManager (that are IOManager, HeightfieldManager, WaterManager, ClimateManager, ObjectManager, SkyManager, LightManager, EffectManager, TextureManager)
5- Each FilterManager process the landscape, model it according to own responsibility. It calls CalculationManager that calls ThreadManager and, finally, FilterManager gives processed landscape back to WorkflowManager
6- WorkflowManager give back control to next plugin that, again via PluginManager, intercept signal
7- Step from 1 to 6 are executed until last FilterManager is executed
8- Last step produces final landscape that is delivered by IOManager that, again via PluginManager, is exported to an external tool

Clearly the scenario could be much more complex as i described in Component Diagram. However the purpose is:
- decouple a wide process into smaller and simpler processes
- design interfaces so that each subsystem could easily cooperate each other
- hide subsystem's complexity, again, with a well-know interface that cooperate with corresponding plugin

Thanks to this 2 patterns is possible to:
1) Aaron could switch Filters
2) Aaron could add or delete Filters
3) Aaron could enable or disable plugin/filter pair
4) An external developer could change a plugin or create another new (according to plugin interface)
5) Aaron could change a FilterManager (accroding to filter interface)

Presently I have two classes of plugins; a general plugin, and a file I/O plugin. The only difference is that file I/O plugins expose a few functions to load/save map files (ExtSaveMapFile, etc.), but are otherwise the same as general plugins.

Try to see CPI as general plugin interface and I/O plugin interface as CPFII/CPFEI pair, that are subinterfaces of CPI. So, any other CPI-compliant interface could be a subinterface of CPI that could be enabled/disabled.

Again, if you consider this thread interesting i'll continue refining model and designing more complex scenarios.

Cheers.

Mauro.

[Aaron]

Hi Monks,

I'd also add GIS related tools- projection manager?

Certainly. However, managers don't need to be in L3DT proper - they can be in plugins, and any non-core managers (i.e. not calc/plugins/file I/O) will probably be implemented out there in pluginland. This will allow for users to upgrade/replace managers, and also allow for more rapid decentralised development (i.e. I'm not necessarily involved.)

The first example of a manager-in-a-plugin will probably be the 'terrain algorithm' manager, which will allow developers to add their own terrain algorithms to the 'select terrain algorithm' wizard. However, this may take several weeks to complete, as I'll have to add to the plugin API the ability to add wizard panes, and call the full suite of L3DT terrain filter functions. Nonetheless, this is the direction the goodship L3DT is heading.

Also, (in the distant future of course) a native Google Map export.

This can be done in plugins now, and I've added it to the plugin ideas and requests page. Any takers?

Cheers,
Aaron.

[monks]

I wish I could help you Aaron, I really do, but I simply don't have the time- I'd have to re-learn a number of things besides the PDK (or whatever).

Apparently ProFantasy [Fractal Terrains (FT)] are looking to include this in a future release- others have already moved in this direction (3DNature:WCS).

monks

[Aaron]

Hi Mauro,

There's sort of two issues here; the API, and L3DT. I’ll try to break them apart:

API

I suppose the main paradigm of the API when it comes to interaction between modules is functional programming (though it’s object-oriented when dealing with data.) Indeed, the idea of a ‘module’ in this context is simply a handy container for functions, and modules are effectively transparent to most of the API. The extension of this is that ‘mangers’ are themselves an abstract idea, being just a function or a collection thereof.

So, having just poured water on module-driven design, let’s describe how modules interact:

Each plugin exports functions to the API (L3DT does this too), which may be called by themselves or other plugins, or indeed by L3DT. Other than to define how functions may call other functions, I’m not going to specify interfaces or patterns for the API in general. This is because the patterns and interfaces used in different managers will vary greatly from manager to manager, depending on what is most convenient for what they do, and on the tastes of the developer in question (remembering that managers don’t have to be in L3DT.) Patterns will be emergent properties, not uniform across the landscape.

Here are some examples:

To run a calculation in parallel on a map, I pass to the thread manager (actually, just a function) a handle to the map(s), a handle to the function I want each thread to run, a handle to a list of tiles (defining areas for each thread), and a handle to a list of function arguments. The thread manager then processes the tiles in order, calling as many threads as it sees fit, each of which run the function passed to the thread manager. The manager doesn't know the details of the maps or calculations it will be running, and doesn't care. It doesn’t even know the function arguments (they are provided ‘from above’). It just runs the function given to it with the list of arguments provided on the maps and tiles provided. This way basically abstracts the thread management from the calculations themselves, which is a neat way to do things in parallel (see Google’s MapReplace algorithm.)

On the other hand, the I/O manager has a specific interface. You may call the file I/O functions (with a well-specified set of arguments), those functions check the arguemnts to find the right file I/O function to save/load the map, and call them with the appropriate arguments. There is no flexibility in this interface, because there is no need, and because it would only make things messy.

Two opposing ways to do things, but both appropriate for their task. Neither are particularly module-centric, though.

L3DT

By ‘L3DT’ I really mean the ‘L3DT pipeline’, i.e. the functions and procedures and interfaces that I’ve built atop the API, which are now or will be exposed to the API. Of course, here I have set patterns and interfaces, and on the whole I lean towards the facade DP above others. I steer away from pipes and filters in particular because I find the pipework to be restrictive in terms of parallel processing, or non-linear (forked) processing. All operations are still generally broken down into small parts and done in order (or in parallel), but each operation generally doesn’t know what came before or will come after or is happening at the same time. That’s all handled ‘from above’.

In any case, irrespective of my choices developers have the freedom to make their own managers and interfaces. One could, for instance, make a whole World-Machine-esque interface, calling on whatever functions of L3DT or other plugins that may be useful, and writing the wrapping code to make them behave like pipes and filters. The API lets you do this, but it’s not how ‘the L3DT pipeline’ works.

- each Manager is a little subsystem that expose a unified interface. The subsystem can change or be replaced with another that implements same interface

Agreed. Being able to swap managers (or rather, replace functions) is a necessary feature. Not implemented yet, but will be soon, probably using a plugin (doesn’t need to be a core API feature).

- a Manager manages a small and independent process (because is a subsystem), but is only a step of a wider process. Each Manager is connected with previous and next Managers by a channel.

I don't necessarily agree here. If I were to have a pipeline pattern such as this, it would only be because I have an over-reaching manager that calls them in order (in fact, I do in L3DT.) Generally, each component has no idea about ‘where it is’; it's the pipeline manager's job to keep track of this.

- Managers exchanges messages of same type (a landscape)

I don't do much, if any message passing at the moment. Each plugin/manager/function/whatever simply does what they need to do, and if something needs to be stored/flagged for use later or elsewhere, they set an appropriate value in the project settings tree. You can see many such examples of this in the 'def.xml' files. Each calc/plugin/manager/dialog/etc stores and recalls the settings they need, which are accessible to each other module/function/etc, but are otherwise decoupled from one another.

Clearly the scenario could be much more complex as i described in Component Diagram. However the purpose is:
- decouple a wide process into smaller and simpler processes

Check (though sometimes it’s not convenient.)

- design interfaces so that each subsystem could easily cooperate each other

Cooperation is via functional programming, so the developer just needs to check the function arguments and return values, and probably any comments left by the developer responsible.

- hide subsystem's complexity, again, with a well-know interface that cooperate with corresponding plugin

I do this wherever possible, but not so as to compromise flexibility. Bearing in mind that I've got a huge amount of plugin programming ahead of me, I'm shaping the API to make that as easy as possible.

Thanks to this 2 patterns is possible to:
1) Aaron could switch Filters
2) Aaron could add or delete Filters
3) Aaron could enable or disable plugin/filter pair
4) An external developer could change a plugin or create another new (according to plugin interface)
5) Aaron could change a FilterManager (accroding to filter interface)

The goal will be to allow anyone to switch filters. With a suitably well designed plugin, it will be possible for non-developers to switch filters (i.e. click and drag).

Presently I have two classes of plugins; a general plugin, and a file I/O plugin. The only difference is that file I/O plugins expose a few functions to load/save map files (ExtSaveMapFile, etc.), but are otherwise the same as general plugins.

Try to see CPI as general plugin interface and I/O plugin interface as CPFII/CPFEI pair, that are subinterfaces of CPI. So, any other CPI-compliant interface could be a subinterface of CPI that could be enabled/disabled.

I see your point, but I think this is a bit too module-centric. As I said up above (I think), the paradigm is really FP rather than OOP. Modules are just a means to put functions into the API, so rather than disabling a module’s interface (or a part thereof), you disable the functions it has exported. That said, I don’t yet have a mechanism by which functions may be disabled, but I’ll add it should the need arise (a plugin could do this anyway.)

Anyway, thanks again for the insight. As you can probably guess, I haven’t done much (formal) work with patterns, so I find it quite interesting to see your perspective. This sort of planning might come in very useful in instances where OOP turns out to be the dominant paradigm, but I think we’ll have to wait and see what and where this happens. Thanks again for your insight.

Cheers,
Aaron.

[mauronen]

I'm sorry again, Aaron, for this long delay reply, but actually i've a lot of troubles (work and parents).
As soon as i could have some time, i'll reply to this thread.

Sorry again.

See you soon.

Mauro.