Page 1 of 1

3 Memory use optimisations for Lua mapgens

PostPosted: Thu Dec 15, 2016 05:57
by paramat
Many players are reporting OOM (Out Of Memory) errors when using LuaJIT, which has a low memory use limit.
Lua mapgens can use excessive memory if not using these 3 optimisations, and most Lua mapgens, even most of mine, are not using these.

Examples of these optimisations are from my mod 'stability' https://github.com/paramat/stability which is currently the best mod of mine to demonstrate good practice in Lua mapgen.

//////////////////////////////////////////////////////////////////////////////

1. Perlin noise objects: Only create once

The noise object is created by 'minetest.get_perlin_map()'
It has to be created inside 'register_on_generated()' to be usable, but only needs to be created once, many mapgen mods create it for every mapchunk, this consumes memory unnecessarily.

Localise the noise object outside 'register_on_generated()' and initialise it to 'nil'.
See the code below for how to create it once only.
The creation of the perlin noise tables with 'get3dMap_flat()' etc. is done per mapchunk.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
-- Initialize noise objects to nil

local nobj_terrain = nil
local nobj_biome   = nil

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   nobj_terrain = nobj_terrain or minetest.get_perlin_map(np_terrain, chulens3d)
   nobj_biome = nobj_biome or minetest.get_perlin_map(np_biome, chulens2d)
   
   local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
   local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

...
...

end)


////////////////////////////////////////////////////////////////////////////////////

2. Perlin noise tables: Re-use a single table

The Lua table that stores the noise values for a mapchunk is big, especially for 3D noise (80 ^ 3 = 512000 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
For each of the functions with an optional `buffer` parameter:  If `buffer` is not
nil, this table will be used to store the result instead of creating a new table.

#### Methods
* `get2dMap_flat(pos, buffer)`: returns a flat `<size.x * size.y>` element array of 2D noise
  with values starting at `pos={x=,y=}`
* `get3dMap_flat(pos, buffer)`: Same as `get2dMap_flat`, but 3D noise


Localise the noise buffer outside 'register_on_generated()'
Use the buffer parameter in 'get3dMap_flat()' etc.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
-- Localise noise buffers

local nbuf_terrain
local nbuf_biome

...
...

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
   local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

...
...

end)


/////////////////////////////////////////////////////////////////////////////////////////

3. Lua voxelmanip table: Re-use a single table

The Lua table that stores the node content ids for a mapchunk plus the mapblock shell is big (112 ^ 3 = 1404928 values).
Many Lua mapgens are creating a new table for every mapchunk, while the previous tables are only cleared out slowly by garbage collection, resulting in a large and unnecessary memory use.

A 'buffer' parameter was added in 0.4.13 to avoid this, a single table is re-used by overwriting the former values.

Very recently a 'buffer' parameter was also added to 'get_param2_data()', so is only usable there in latest Minetest dev or in Minetest 0.4.15 stable or later.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
* `get_data([buffer])`: Retrieves the node content data loaded into the `VoxelManip` object
    * returns raw node data in the form of an array of node content IDs
    * if the param `buffer` is present, this table will be used to store the result instead
* `get_param2_data([buffer])`: Gets the raw `param2` data read into the `VoxelManip` object
    * Returns an array (indices 1 to volume) of integers ranging from `0` to `255`
    * If the param `buffer` is present, this table will be used to store the result instead


Localise the data buffer outside 'register_on_generated()' as an empty table.
(Why as a table? Because only tables are passed by reference).

Use the buffer in 'get_data()' or 'get_param2_data()'.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
-- Localise data buffer

local dbuf = {}

-- On generated function

minetest.register_on_generated(function(minp, maxp, seed)

...
...

   local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
   local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
   local data = vm:get_data(dbuf)

...
...

end)

Re: 3 Memory use optimisations for Lua mapgens

PostPosted: Thu Dec 15, 2016 20:30
by Wuzzy
Thanks for this guide. I think it would be useful to add this text to the Developer Wiki:
http://dev.minetest.net/Main_Page

Re: 3 Memory use optimisations for Lua mapgens

PostPosted: Fri Dec 16, 2016 01:03
by paramat
Feel free to, i have no wiki access.

Re: 3 Memory use optimisations for Lua mapgens

PostPosted: Sun Dec 18, 2016 18:00
by Spaghetti Developer
Thanks Paramat! a long time I noticed this problem, the chink are struggling to generate and often they crash, then I have to walk to get closer so that you generate the missing area. I noticed a strong lag, due to the excessive vegetation in the jungle biome, and biome valley. Too many trees my friend! should not lower the density of vegetation? often biomes are full of trees excessively. This improves the generation of the chunk and making the most beautiful biomes and enjoyable, especially in multiplayer. I hope I explained myself :D

PostPosted: Sun Dec 18, 2016 20:48
by Hybrid Dog
> *** Note a buffer is only usable in 'get_param2_data()' in latest Minetest dev or in Minetest 0.4.15 stable ***

The buffer doesn't work for 'get_data()', so using it there doesn't hone memory usage, does it?

Re: 3 Memory use optimisations for Lua mapgens

PostPosted: Mon Dec 19, 2016 02:13
by paramat
The buffer does work for 'get_data()' and 'get_param2_data()'.

Re: 3 Memory use optimisations for Lua mapgens

PostPosted: Sat Apr 01, 2017 19:00
by paramat
Note i have been corrected on point 3, the initialisation of dbuf must define it as a table:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
 local dbuf = {}


Otherwise memory usage is not reduced.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
    -- Localise data buffer

    local dbuf = {}

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
       local area = VoxelArea:new{MinEdge = emin, MaxEdge = emax}
       local data = vm:get_data(dbuf)

    ...
    ...

    end)

Re: 3 Memory use optimisations for Lua mapgens

PostPosted: Sat Apr 01, 2017 19:45
by paramat
Just realised this probably applies to point 2 also:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
    -- Localise noise buffers

    local nbuf_terrain = {}
    local nbuf_biome = {}

    ...
    ...

    -- On generated function

    minetest.register_on_generated(function(minp, maxp, seed)

    ...
    ...

       local nvals_terrain = nobj_terrain:get3dMap_flat(minpos3d, nbuf_terrain)
       local nvals_biome = nobj_biome:get2dMap_flat(minpos2d, nbuf_biome)

    ...
    ...

    end)