L3DT users' community
Large 3D terrain generator

Multi-core support

Any and all chit-chat regarding L3DT.

Multi-core support

Postby Dretch » Mon Jan 30, 2017 2:07 am

Hello Aaron,

I was wondering if future updates of L3DT would raise the current maximum number of 32 cores.

I want to create a really massive map and was wondering if I created a beowulf cluster could the erosion algorithm calculations be shared among many cores? During my trial I found the process took days and sometimes didnt give the result I wanted.

Cheers.
Dretch
New member
 
Posts: 2
Joined: Mon Jan 30, 2017 1:50 am

Re: Multi-core support

Postby Aaron » Wed Feb 01, 2017 1:22 pm

Hi Dretch,

I apologise for the delay, and thank you for your question. Unfortunately, I think my answer is going to be disappointing, but here goes...

Alas, the erosion algorithm is not multi-threaded at all, so I will have to parallelize it before you can take advantage of a greater core count. This is on my to-do list. Multithreading has so far been applied to the calculations for the texture map, light map, normals map, and attributes maps, as these are typically generated at higher resolution than the heightfield, and therefore take longer than generating the heightfield. They were also relatively easy to parallelize, as each pixel is not dependent on its neighbours. This is unfortunately not the case for erosion, where each region strongly affects its neighbours, which requires inter-thread communication and synchronisation, so multithreading erosion will be more complex to implement, and won't scale as well to many cores anyhow.

As for increasing the supported core count; unless I resort to dark voodoo, the core count is limited to the size of the thread affinity mask for the process, which is 32 threads on a 32 bit system and 64 threads on a 64 bit system. Even then, I suspect the algorithms would not scale well to so many threads on a single system, as you'd most likely run into problems with disk bandwidth or thread-sync overheads. As far as my testing goes, most of the multithreaded calculations scale OK out to 6-8 cores, but thereafter there does seem to be diminishing returns (things may be better on newer systems, though).

Regarding Beowulf clusters; unless virtualisation has come a long way without me noticing (a distinct possibility), to run on a cluster L3DT would need to run separate instances on each system, and synchronise the jobs over the network. This is quite different to just multi-threading within one process on one system, and is quite independent of the core limit of the L3DT process. Anyhow, about a 8 years ago, I did actually develop code for running L3DT as a render farm across multiple systems, but as far as I can tell, I never quite finished it (the record shows I got distracted by scripts, graphs, 64-bit builds, other performance optimisations, etc.) It may be possible to take this work up again, but since it's a niche feature, it's not likely to be a high priority for some time. My guess is that it will also likely be significantly more difficult to support (networks are trouble), so if I do finish it, it is unlikely to be included with a normal 'indie' license (or trial, for that matter).

Best regards,
Aaron.
User avatar
Aaron
Site Admin
 
Posts: 3655
Joined: Sun Nov 20, 2005 2:41 pm
Location: Melbourne, Australia

Re: Multi-core support

Postby Dretch » Wed Feb 01, 2017 11:32 pm

Thank your response, even if it isnt what I wanted to hear lol. I couldnt program myself out of a paper bag but I have looked into the theories as I have a world I want to build and need a way to build it. So does your algorithm for erosion only use CPU's? Could it not take advantage of GPU processing like in this paper?

http://www-ljk.imag.fr/Publications/Bas ... n_PG07.pdf

Just curious, thanks for you time!
Dretch
New member
 
Posts: 2
Joined: Mon Jan 30, 2017 1:50 am

Re: Multi-core support

Postby Tominywa » Thu Dec 14, 2017 9:34 am

I hope it will have good support. I want to use it like nothing wrong.
Tominywa
Newbie
 
Posts: 1
Joined: Thu Dec 14, 2017 4:48 am


Return to General discussion

Who is online

Users browsing this forum: No registered users and 2 guests