L3DT for Spring, anyone?

[Aaron]

Hello everyone,

I'm tinkering with the idea of an 'L3DT for Spring' installer, for users of the hugely awesome open-source game TA Spring. This version, based on L3DT Pro, would include Spring-friendly climates and default settings. In the future, I would imagine that Spring-specific plugins could be included, such as a mapconv interface. Also, plugins not particularly useful to Spring mappers would not be included by default, so the download will be smaller.

Anyway, please reply if you think it's a good/bad idea. If you do like the idea, please feel free to add your input on what should and should not be included, or on what the default settings should be.

Cheers,
Aaron.

[carldt]

To me this sounds like a bad idea if the extra time you would spend on maintaining another version of the L3DT installer could be better spent on bug fixing/features in standard L3DT. Also I think L3DT should be kept as a more general tool rather than game specific.

In my experience, the fewer versions there are to support and maintain, the better. I think you will avoid some support issues by not branching into a spring version as well as a standard version...

But I think it would be great if L3DT standard could produce all the necessary artifacts required by TA Spring maps out of the box.

I'm inferring from your post that you are have or are adding the ability to generate metal/feature maps for spring? (This means being able to add user layers that can be drawn on using a paint brush type tool.) This is a good thing, especially if this feature goes in the standard (free) version.

You are probably aware that I was inspired to write the cdPython plugin as a way a of driving mapconv. In fact I posted a Python script which nearly goes all the way there - it's just missing the lines to export a metal map from L3DT because user painted layers were unavailable when I was working on that.

One feature that may be useful to Spring, would be the ability to carve roads through/across/around hillsides, to make the maps more vehicle friendly.

[JavaJones]

Aaron, I like the idea of supporting game communities like that with special attention, however I don't really see much value in making a "cut down" version. Most people who will be doing Spring maps probably have broadband so the download size shouldn't be an issue, and if nothing else you can always just make groups in your climates and other functions (plugins/scripts) that specify particular uses so anyone who wants to ignore non-Spring stuff can just stick to the relevant groups. If they were using default climates they would only look in the "TA-Spring Climates" group, for example. So you would include those climates with the standard installer and everyone could potentially benefit. There's little to lose by doing so - a somewhat larger installer perhaps, but it's still pretty darn small compared to some web downloads!

- Oshyan

[Aaron]

Hello,

Okay, I've been convinced. Thanks for the feedback; you've given me some ideas for how to better achieve that goal.

if nothing else you can always just make groups in your climates and other functions (plugins/scripts) that specify particular uses so anyone who wants to ignore non-Spring stuff can just stick to the relevant groups

That's a great suggestion, thanks. I'll put this on the dev-plan.

I'm inferring from your post that you are have or are adding the ability to generate metal/feature maps for spring? (This means being able to add user layers that can be drawn on using a paint brush type tool.) This is a good thing, especially if this feature goes in the standard (free) version.

I'm looking into that, yes. Once v2.5 is out of the way, I'll start experimenting with custom map layers and editors. There shouldn't be any major hurdles, but it will probably take a bit to make it 'nice'. (In the mean-time, can you feed in metal/feature maps made in another program?)

Cheers,
Aaron.

[carldt]

Hi,

I'm looking into that, yes. Once v2.5 is out of the way, I'll start experimenting with custom map layers and editors. There shouldn't be any major hurdles, but it will probably take a bit to make it 'nice'. (In the mean-time, can you feed in metal/feature maps made in another program?)

Yeah sure it's no problem to make the feature maps in another app like Gimp for example. However I'd like to make the Spring procedure a little more automated with the help of L3DT - so I don't have to open up different apps myself and browse to the various places to import output files from other programs. Jumping between applications stifles the creative process...

These are my options:

Control the process from L3DT, create all map layers in L3DT, and compile the map using a plugin.

Control the process from another application, and call L3DT on the command line to generate the texture map. Something like:


Code: Select all

L3DT.exe --heightmap hm.bmp --out texture.bmp --climate-file myclimate

Ultimately it would be great to have both options - a way to draw bitmap layers in L3DT, and a way to use L3DT in batch mode on the command line. Providing both would make L3DT extremely flexible.

Thanks.

[Aaron]

Hi Carl,

I'm not really in favour of having L3DT process complex command line arguments. There are an awful lot of options that require user input, and some of those don't have safe defaults. I'd be more comfortable with something like:

Code: Select all

l3dt.exe myscript.py


...where the python script gets passed through to cdPython for processing (I'll implement this now). This way L3DT doesn't have to assume any particular settings; they must be provided in the script, or else the the script author (whether human or a calling program) may choose to load the defaults. A more powerful way to do things, m'thinks. Now, I'll admit that you can't build a whole map this way just yet, as not all of L3DT's map building functions are accessible through the plugin API. However, I do intend to make this possible in v2.5a.

I suppose a python script for applying all settings and running all calcs could be overly complex, particularly if you have to fetch zeofunc handles, set arguments, call zeofuncs, collect return values, etc. Following Rummy's request earlier today for a command line-ish interface, I'm seriously considering a really cut-down scripting plugin, with syntax something like:

Code: Select all

project.ImportHF mymap.bmp            // import the heightfield

calc.HF.SetHorizScale 10            // change the horiz scale
calc.HF.SetVertRange -200 560         // changing the vertical range

project.LoadCalcPreset WM.Default      // load default water map settings
calc.GenWM                        // generate water map

project.LoadCalcPreset AM.Default
project.LoadClimate desert
project.SetSetting "AM.AM_in_HF" int 8 // make high-res attrib. map
calc.GenAM

project.LoadCalcPreset TN.Default
calc.GenTN

project.LoadCalcPreset LM.Default
calc.GenLM

project.LoadCalcPreset TX.Default
project.SetSetting "TX.TX_in_HF" int 8 // make really high-res texture
calc.GenTX

map.ExportFile TX mymap.jpg  // now save the texture

shutdown    // close L3DT


In the above example, the first element of each line is the zeofunc to be called, and the rest of the line are the space-delimited arguments. Since zeofuncs have registered argument prototypes, I don't need a complex parser to guess the type of argument being passed. Most of the work has already been done by the plugin API. Anyway, that's the idea at this stage, and if these scripts could also be passed as a command-line argument to l3dt.exe, then I think we'll be well on the way.

Cheers,
Aaron.

[carldt]

Hi Aaron,

It would be ideal to run L3DT with a Python script on the command line in a non-gui mode!

I suppose a python script for applying all settings and running all calcs could be overly complex, particularly if you have to fetch zeofunc handles, set arguments, call zeofuncs, collect return values, etc. Following Rummy's request earlier today for a command line-ish interface, I'm seriously considering a really cut-down scripting plugin, with syntax something like:

Instead of implementing another scripting plugin, I think perhaps we could take better advantage of cdPython and write some wrapper classes around the zeolite.py module that comes with cdPython to simplify things even further. The wrapper module could do things like fetch zeofunc handles on initialization so the script author need not have to bother with all the "plumbing".

I don't mind having a crack at this, but let me know what your plans are in this area so there is no duplication of effort.

Thanks,
Carl

[Aaron]

Hi Carl,

Okie dokie, it's done. In the next release, L3DT will run python scripts using cdPython when passed from the command line or when dropped on the icon or exe (assuming cdPython is installed). The app will still run in GUI mode, but at some point in the future I'll allow you to either hide the GUI or minimise it.

Instead of implementing another scripting plugin, I think perhaps we could take better advantage of cdPython and write some wrapper classes around the zeolite.py module that comes with cdPython to simplify things even further. The wrapper module could do things like fetch zeofunc handles on initialization so the script author need not have to bother with all the "plumbing".

I'm interested in the idea of pre-fetching some zeofunc handles. Actually, what I was thinking of doing was adding another set of c++/h files to the API for calling commonly-used extension functions. That way, the extension functions can be made to look like the 'normal' API functions, which may make their use appear a bit less mystical. This is actually what I'm doing in some of the API functions already (if you look in the code for CExtAPI::format_GetActiveFormat, you'll see its calling a zeofunc). Would a greater library of functions wrapped this way help, do you think?

Anyway, I was a little intrigued in the idea of a simpler scripting language, so I went ahead and implemented the plugin. It's still really basic, but it demonstrates the concept of calling extension functions in a command-line-esque script. For it to be useful I will need to add a whole bunch of zeofuncs (these will be useful for cdPython too, of course), and I'll need to improve the ZeoScript parser a little. It needs to handle, in some very basic way, piping functions return values to other functions. Something like:

Code: Select all

// calling map_SaveFile(map hMap, string lpFileName, format hFormat, bool SetFlags, bool ShowProgress)

map.SaveFile <z:project.GetMap HF> "C:\test map\this is a test.bmp" <z:format.GetByExt HF 0 bmp> false true


...where <z:...> is another zeoscript call, the return value of which is popped into the arg list of the parent function. For extra shorthand, I'd probably also do <m:> for map-get (e.g. <m:HF>, short for project.GetMap), <v:> for variable-get (short for var_GetByName), and <f:> for format-get (short for format_GetByExt). If I do that, the above line becomes:

Code: Select all

map.SaveFile <m:HF> "C:\test map\this is a test.bmp" <f:HF 0 bmp> false true


That's about as far as I'll go though. This scripter is not meant for complex things. It certainly won't have fancy stuff like object handles, and probably won't ever have logical flow control (if/else, etc). Above all, I don't want to spend time duplicating the goodies that are already available with cdPython, particularly since I have to implement stuff like custom map layers . However, what I do want from ZeoScript is a really simple, terse and command-oriented script interface, which I don't think you could easily get with real programming languages like python because their syntax is more complex.

Oh, by the way, python scripts will be able to run ZeoScripts using the 'ZeoScript.RunScript' extension function, and likewise ZeoScripts will be able to run python scripts using 'cdPython.RunScript'. Thus we'll be able to mix and match languages in the same script, where you could do lots of routine things more tersely in ZeoScript, and more complex and structured things in python.

Cheers,
Aaron.

[carldt]

Hi Aaron,

I'm interested in the idea of pre-fetching some zeofunc handles. Actually, what I was thinking of doing was adding another set of c++/h files to the API for calling commonly-used extension functions. That way, the extension functions can be made to look like the 'normal' API functions, which may make their use appear a bit less mystical. This is actually what I'm doing in some of the API functions already (if you look in the code for CExtAPI::format_GetActiveFormat, you'll see its calling a zeofunc). Would a greater library of functions wrapped this way help, do you think?

Yes I think adding more zeofunc wrappers to the library is a good idea. It seems pretty pointless and time consuming to have to write zeofunc fetching code for every plugin, especially when you know that this function is always available. If you add these wrappers to the C++ objects in zeolite, then swig will pick them up and cdPython will get these wrappers too.

I just had a thought: I could write a tool (a bit like SWIG) that would automatically generate C/C++ wrappers as output given a plugin or source file as input. This would mean that plugin developers could run the tool on their plugin, and automatically get a wrapper for the functions their plugin exports, saving others having to write zeofunc fetching code to use those functions. Having a tool like this would save on maintenance of wrapper code as well, you could use such a tool to generate the Zeolite classes.

[Aaron]

Hi Carl,

...especially when you know that this function is always available.

That's the thing; one of the main reasons I'm adding functions via zeofuncs rather than to CExtAPI is that I don't think they necessarily will always be available. The 'grand plan' is that someday I will spin all of zeolite out into it's own DLL so that other applications can use the framework, and indeed any plugins written for it. I did this back in '05 as a test, and the concept worked pretty well. However, at the time the task was a bit ahead of my ability, and I never got it debugged. Anyway, having zeolite running in other apps means that many of these extension functions won't be available (e.g. the L3DT calculation functions). This is why the functions in CExtAPI are really just the core ones for data/map/format/function handling - this is the minimal set that all zeolite plugins or zeolite-based applications will need to access. Anything more than that goes in as an extension function.

I just had a thought: I could write a tool (a bit like SWIG) that would automatically generate C/C++ wrappers as output given a plugin or source file as input. This would mean that plugin developers could run the tool on their plugin, and automatically get a wrapper for the functions their plugin exports, saving others having to write zeofunc fetching code to use those functions. Having a tool like this would save on maintenance of wrapper code as well, you could use such a tool to generate the Zeolite classes.

Ooh, I do like that idea. Here's a related but possibly insane idea: what about a plugin that takes a list of zeofuncs (e.g. the whole lot, or a sub-set from just one plugin, for example) and builds up wrapper C code for them? The process of generating wrapper code for a zeofunc is fairly automatic, so I guess it would be possible to automate in that way. I've already kind of done some of this work for the atFuncBrower plugin, which generate C-ish strings for the function prototypes. A plugin like this could then be used to generate the complete C wrapper code for all the available zeofuncs. An idea, anyway.

There are also two other approaches I'm considering for making zeofuncs easier to use. The first is an API function to combine the calls for getting and executing the zeofunc:

Code: Select all

bool CExtAPI::zeofunc_GetAndExec(const char* lpFuncName, ZLIST hArgList, ZVAR hRval);


The second idea is a few functions to run scripts directly through the API:

Code: Select all

bool CExtAPI::script_Run(const char* lpLanguage, const char* lpScript, ZLIST hArgList, ZVAR hRval);
bool CExtAPI::script_RunFile(const char* lpScriptFile, ZLIST hArgList, ZVAR hRval);


For instance, you could use this to call a python script to run extension functions:

Code: Select all

theAPI.script_Run("py", "import zeolite\r\nf = zeolite.CzFunc()\r\nf.GetFunc("project.SaveProjectAs")\r\ns = zeolite.CzStr()\r\ns.Attach(f.m_args.GetItem(0))\r\ns.SetText("C:\\tempdir\\this is a test.proj")\r\nf.Execute()\r\n", NULL, NULL);


That's about the same length of code that would be required to run that calc in C++. However, if you had the script saved to disk you could use the proposed script_RunFile function, which is shorter:

Code: Select all

theAPI.script_RunFile("project_SaveProjectAs.py", NULL, NULL); // I'll detect language based on script extension


Similarly, you could run a ZeoScript to do the same thing:

Code: Select all

theAPI.script_Run("zs", "project.SaveProjectAs \"C:\\tempdir\\this is a test.proj\"", NULL, NULL);


Either way, I think running py/zs scripts from C++ code could really simplify calling zeofuncs. One challenge will be how to pipe arguments into the scripts and retrieve return values. I think I'll be able to recover return values from ZeoScripts OK, but passing arguments to scripts will be a bit rustic (e.g. to use a map in a function argument, you have to format the string to contain the correct <m:...> call to retrieve the handle). It may be possible to do a better job with python - I guess we'd need some functions for pushing/popping values onto a globally accessible list/stack that can be called by the script. Anyway, even if we don't get the args and rvals working in the first instance, I'm sure we'll be able to do so eventually, and the end result will be a rather nifty programming tool.

Cheers,
Aaron.

[carldt]

Ooh, I do like that idea. Here's a related but possibly insane idea: what about a plugin that takes a list of zeofuncs (e.g. the whole lot, or a sub-set from just one plugin, for example) and builds up wrapper C code for them?

Hehe that sounds like a good candidate for a Python script! I'll give it a go. A python script that produces C code wrappers for zeofuncs. Then I can add the generated C code wrappers to cdPython, and SWIG will create Python wrappers for those C code wrappers... The end result: having all the zeofuncs accessible from Python without needing to do any zeofunc fetching.

[Aaron]

Hi Carl,

A python script that produces C code wrappers for zeofuncs. Then I can add the generated C code wrappers to cdPython, and SWIG will create Python wrappers for those C code wrappers...

I think you've just won the new "Idea of the Week" competition! What a technical feat that would be: converting C++ functions to C wrapper code via a python script which is then converted to python via SWIG. My only reservation is what will undoubtedly happen a year down the track, when a new user comes along and has some scripting problem that requires all this to be explained. "Well, you see, the script function you're calling is actually some FORTRAN 77 code that gets converted into a gypsy curse by a voodoo spell, then...[five minutes later]...which is converted to COBOL in Japanese...[another five minutes]...encrypted by ROT13...[maybe ten more minutes]...bounced off mirrors on the surface of the moon...[indefinite period]...and finally, converted to Python code. I know it sounds a little complex, but...". Actually, that kind of sounds like fun.

Cheers,
Aaron.

[carldt]

It's done! I looked at the atFuncBrowser plugin source, and wrote a Python script which does pretty much the same thing, except instead of displaying a tree control, it writes a C file.

Here's the script (excuse the lack of structure):

http://l3dtpython.googlecode.com/svn/trunk/Scripts/zeofuncGenerator.py

Here's the output:

http://l3dtpython.googlecode.com/svn/trunk/cdPython/zeofuncapi.c

I've taken zeofuncapi.c, and added it to cdPython, so that these zeofunc wrappers are all available from Python. Now, we can call a zeofunc without having to do all the zeofunc fetching and CzVar code...

Here's a slightly twisted example: a Python script that calls the cdPython plugin's RunScript function, passing a Python program as a string:

Code: Select all

zeolite.cdPython_RunScript("print 'zeofunc API test successful!'")


C/C++ plugin developers may find the generated zeofuncapi.c file useful in their projects, moreover, they may want to use the zeofuncGenerator.py to create a more current set of wrapper functions.

[DeathTwister]

OK, I am a bit late on this one, but I second the motion Javabones said, I like the way he put it, and I think one way or another spring climates would be sweet, weather you download them all in one, or get the climate files one at a time it is fine with me, good Idea brother Aaron by the way/smiles.
DT

[Forboding Angel]

Ack dangit, missed this one...

AAron, if you could link l3dt in some way with spring I would be eternally grateful. Also if you plan or are even willing to intigrate with the compiler I have a request...


Basically when mapconv was written, it was done by a dev with very little vision and it has a lot of silly things that cause us a lot of problems...

For example, we can only use bmp or jpg images with it. Also, there is a 600 meg limit on the images that it can process. The entire process is much akin to a solid kick in the nuts and we badly need png support.

One thing that would help immensely is if the intigrated compiler were to be able to use L3DT mosaic maps for the texture, instead of us having to put them together and then export.

Also, one more thing, while I'm investing in pipe dreams... Can you create a command line process for stitching huge mosaic maps together without loading the entire thing into memory and have it export as a jpg? Thing is, I'm working on a monstrosity 50x50 map. I can't export the entire image from L3dt else it craps all over itself. SO then I went with my last ditch option which is a program that a community member made called grout.

Grout does what I asked for above, but it has serious limitations. For one, the output file is always a bmp, that's bad, especially considering that with an image that huge I can never hope to get photoshop to open it, which means that everything has to be done via command line options. Last problem... Grout craps all over itself when it has to do with huge mosiac maps.


Help