FOSS gamedev and creative worlds

It's a known fact that adding too many nodes at once from Lua can be very slow. The Big Trees mod had issues because of this, and my Structures system also doesn't work as quickly as it could, while the Lua floatlands can take up to a minute to generate. Worse than that, I frequently get incomplete building spawns with my Structures mod, which proves the current method can even fail.

Currently, the only way to add blocks in Lua is with the minetest.env:add_node() function. Using it to spawn a lot of blocks at once is very laggy and even unsafe. I assume it's because it sends a lot of signals to create, notify and store a lot of blocks in a short amount of time... which the code can't properly group and doesn't have the time to handle.

My suggested alternative is allowing a list of blocks to be sent. The reason why trees and landscapes generated by the mapgen are so fast is that spawning a lot of blocks simultaneously is a dozen times faster in C++, given they can be organized and grouped accordingly while the hard work is done once for all nodes rather than per-node. With such a function, adding structures from Lua wouldn't require spamming the code with nodes, but sending a single list which the engine can queue and manage optimally.

If this can result in less lag, then I am all for it. Paramat's mods and moretrees would benefit greatly from this.

I have this old branch flying arround: https://github.com/PilzAdam/minetest/commits/set_nodes_area
Its almost twice as fast as doing it in Lua.

PilzAdam wrote:I have this old branch flying arround: https://github.com/PilzAdam/minetest/commits/set_nodes_area
Its almost twice as fast as doing it in Lua.

That looks like it might be useful. Though looking at the code, it's not what I had in mind here. My idea isn't a way to fill an area with one type of node, but to rez multiple nodes at various positions in a faster way (with one call for the whole set). That code is much simpler than I thought however, so maybe I can code this myself using its example. Thanks for posting it.

Anyway, I think this would be very helpful and important, especially since it should be an easy thing to code. It would prolly benefit a lot of mods, especially the naturally spawning structures.

MirceaKitsune wrote:

PilzAdam wrote:I have this old branch flying arround: https://github.com/PilzAdam/minetest/commits/set_nodes_area
Its almost twice as fast as doing it in Lua.

That looks like it might be useful. Though looking at the code, it's not what I had in mind here. My idea isn't a way to fill an area with one type of node, but to rez multiple nodes at various positions in a faster way (with one call for the whole set). That code is much simpler than I thought however, so maybe I can code this myself using its example. Thanks for posting it.

Anyway, I think this would be very helpful and important, especially since it should be an easy thing to code. It would prolly benefit a lot of mods, especially the naturally spawning structures.

IIRC Jeija had a branch for it. Somone might want to look in his repo.

It would be great if there'd be a better - safer and faster - way to spawn large structures! My random_buildings mod goes to a lot of trouble to make sure that the area where the building wants to spawn is loaded entirely. Mauvebic also had a lot of trouble with that aspect and gave me some valuable hints while I was writing that part. It is stil not satisfying.

My current approach would be to abuse the rollback function. At least that *does* seem to work pretty reliably - it does load unloaded blocks if necessary, so it might reliably load a huge structure. Under certain circumstances. It would be very abusive :-(

A function to set an entire chunk from lua at once would be great! It ought to work regardless of weather the area is loaded or not. Something very close to the real map file format, but with a well documented interface and all parts split up into variables passed to lua and staying consistant over diffrent mapfile versions.

Spawning huge buildings or part of another map could be extremly fast - as long as it does not have to be adopted to the sourroundings or moved a couple of blocks around. In theory, just copying entries from one database to another would do.

I'm currently working on adding this using PilzAdam's set_nodes_area for inspiration. It's taking a bit since I'm figuring out how to extract tables from tables in the Lua API, since the format will be { {pos1, node1}, {pos2, node2}, ... }. Will post here when it's ready.

@Sokomine, could you explain how you force the loading of blocks will rollback? And does it work with rollback disabled? About force loading there is my branch https://github.com/minetest/minetest/pull/606 that nobody has merged yet, but that is less hacky than that.

@PilzAdam, why was the Jeija branch not merged?

Nore wrote:@PilzAdam, why was the Jeija branch not merged?

Dunno, he never opened a pull request for that. I guess he wasnt completly done and is busy with other things.
It will probably need a rebase now, but since Taoki is working on it now its not really needed, I guess.

What is that, then? https://github.com/minetest/minetest/pull/440

Nore wrote:What is that, then? https://github.com/minetest/minetest/pull/440

Oh, didint know about that one / have forgotten it.

Success! Much of the credit goes to the people in #lua for helping solve the nested table readings.

https://github.com/MirceaKitsune/minetest/tree/node_group

https://github.com/minetest/minetest/pull/771

The new function has the same functionality as set_node, only that it sends a list of nodes instead. It uses 3 layers of nested tables, which may seem weird at first. A correct example of the raw format is:



When using a correct formula to add nodes to the table, that makes a lot more sense. This is closer to how it's really meant to be used... consider you might use a pattern loop there also:



Feel free to try it and compare it to the classic "set_node spam". I tested that it works but not how much faster it is (likely a considerable amount). If there are no problems and it's merged, I'll immediately adapt my Structures mod to use it and can do some heavier comparison there.

Unfortunately, I just noticed the other pull request Nore linked. Even more unfortunately, both pull requests have been closed because of (I'm sorry to say) how close minded some of the developers are. Seems mods like Structures, Lua floatlands, etc. will have to struggle with set_node when a perfect solution exists, because that's the way some of the project maintainers understand to do things.

https://github.com/minetest/minetest/pull/440

https://github.com/minetest/minetest/pull/771

The reasons cited are that this function would benefit a single mod (far from the truth) and that some of the admins are expecting a fancy "voxel manipulator" for the Lua API instead. I've no idea what that broad term is even supposed to mean or why it would be needed. All I know is those mods would benefit from a simple way to add a lot of nodes at once more quickly.

But that's the Minetest administration for you. Enjoy your limits and stagnation.

[EDIT] Ok, that's probably more harsh than needed... sorry. I'm simply confused as to why a clear and IMHO correct fix to a considerable problem is cut off for a distant idea which might even not happen. I also don't see how and in what sense it could be better. If anyone knows what the voxel manipulator system is meant to be like, feel free to post about it... it's fair and constructive to listen to any opinion and maybe someone can do that instead and also solve the problem this code was aiming to.

[EDIT 2] Found the link for the suggested Voxel system for everyone interested: http://dev.minetest.net/TODO#Lua_MapVoxelManipulator_Interface

Thanks for the effort MirceaKitsune. It's good to see that the slowness of large structure generation is recognised in the todo list, i look forward to any improvement. I am trying to learn all the tricks for more speed, but increasingly i feel the need for fast on-generated adding of most of the nodes in an entire 80x80x80 chunk. Upgrading from 0.4.4 to 0.4.6 has increased structure generation speed by roughly 50% which makes me happy.

Nore wrote:@Sokomine, could you explain how you force the loading of blocks will rollback? And does it work with rollback disabled? About force loading there is my branch https://github.com/minetest/minetest/pull/606 that nobody has merged yet, but that is less hacky than that.

As far as I understand it (and could see in the code), rollback does force-load unloaded chunks if it has work to do in there. That's also pretty logical because otherwise it would not do any good - a griefer whose "work" is rolled back may not have just griefed in the spot right in front of you. He might have wandered around a bit, destroying more on his way.
Using rollback for spawning of structures thus ought to work. Instead of rolling back a real griefer, the mod that wants to spawn something would fake the actions of a griefer and write a false log - i.e. claiming that there was a brick block at a certain point and that there's now air due to digging actions of our imaginary griefer. It would be a very unclean, abusive, desperate attempt to handle large structures, but right now it's the only way I know of that guarantees to some degree that a structure will be spawned completely.

I've talked to hmmm about mapgen v7 and that lua mapvoxelmanipulator feature. The way he described it, the internal format seems to be very close to how mapfile chunks are saved. It ought to be a compact format. Although it seems to be aimed at including more natural structures (strange shaped trees, structures in caves, each node with a random value), it ought to be extremly useful for spawning houses as well.

I hope it will be finished soon so that we can use it. No matter how it is implemented internally, such a function as MirceaKitsune described here (and what the mapvoxelmanipulator will hopefully be, perhaps among other things) is needed. Not only by one mod but by many - WorldEdit, Node Deployment, StructureIO, random buildings etc.