Infinite world size

[Krock]

causing the edges to wrap

mapgens could have troubles with wrapping but it would be a very nice feature.

[spootonium]

Pushing back the horizon by a few blocks or a few thousand can only serve to delay the inevitable.

I would suggest that the under-utilisation of vertical space that we already have is the greater limitation. It's easy to forget that there's so much potential in the ~30km of y-axis either side of sea-level. What I mean is, that it is certainly easier to wander the surface until you blunder into an invisible wall than it is to mine even 1 kilometre down, or construct a cloud-piercing Obsidian Spire of Awesome.
Paramat (among others, of course) has taken many great strides to remedy that, by demonstrating that Minetest can as easily generate a Rama, Rapture, Ark or archiepelago. But, while we're waiting for him to release whatever mind-bending Möbius-world he's cooking up at the moment, I think it's important to remember that there's already plenty of exploration (and construction) to be done in the cave systems beneath our invisible feet.

[SegFault22]

most people find 64km^3 just large enough. I have big ideas and work with megastructures and the world's just big enough for me

Of course, currently people find the world just big enough. But, also consider that there are only a few dozen players (maximum) online on any multiplayer server in Minetest. We frequently discuss how we want a bigger, better community, and ways to make that so. But, if this issue is not addressed, people will start leaving when they find that there is not enough room to build a house, because mega-faction-guy built a huge wall just outside of the spawn protection area, or something.
Consider Minecraft's Gontroller server. Regularly, they have over 300 players online, with enough slots for 500 players. The Minecraft world is around 18 million blocks wide/long, so there is no issue there. But when we consider Minetest's relatively tiny 62000 block wide/long/tall cube, there comes issues if we ever intend to have a huge server like Gontroller (pvp or not). Eventually, cities/claimed territories will pile up everywhere, until there is no room left. The lifetime of the world is inversely proportional to the number of active players online.
Also, we must consider that not all pieces of land are good for cities and whatnot. A huge amount of surface area is ocean, and not many people build over water. Some players are picky enough to keep running around until they find the "perfect spot", but there are not many perfect spots in any Minetest world. Eventually, after many players come and go, after a lot of the land is either claimed or stripped of resources, one will not be able to find a place free from the influence of players. This prevents us from being able to go out and find a place free from the influence of players, and where the chances of someone visiting and ruining the place are negligible. Making the world size much bigger, or simply infinite, will fix that.

[paramat]

Minetest's core mapgen dev hmmmm is planning core skylands and underworld realms, and might also work on the lighting system to avoid higher realms shadowing lower realms, so even without mods there will hopefully be at least 3 stacked realms to inhabit, or even more if i can convince him to make multiple skyland realms possible. If anyone starts to run out of space in a world just think vertical, sky and underground cities seem more interesting to me than the huge but very flat MC worlds.

[Sokomine]

But, if this issue is not addressed, people will start leaving when they find that there is not enough room to build a house, because mega-faction-guy built a huge wall just outside of the spawn protection area, or something.

That rarely happens. What happens is that the tiny spawn area soon gets crowded and chaotic due to everyone building/claiming land there. When new players join, they stay close to spawn and hit only land that has already been built on or protected by someone. Many complain that they find no place to build on. But that's not true. All it would take is walking a few meters. Even Redcrabs server, that has been up for a long time already and is probably the largest, has more than enough free land for decades to come. Furthermore, good buildings do take some time to construct. Admittedly, sometimes you want a house with nice, untouched landscape around. But there are so few builders out there who can actually take advantage of that that it doesn't matter in the long run.

The problem with "the world is too small" is entirely a psychological one. Perhaps turning the player around by 90 degrees when he hits the edges might help already :-) Of course borders draw attention. If players know that there's the end at ca. 30 km, they'll set off for there.

Minetest's core mapgen dev hmmmm is planning core skylands and underworld realms, and might also work on the lighting system to avoid higher realms shadowing lower realms, so even without mods there will hopefully be at least 3 stacked realms to inhabit, or even more if i can convince him to make multiple skyland realms possible. If anyone starts to run out of space in a world just think vertical, sky and underground cities seem more interesting to me than the huge but very flat MC worlds.

Hope that gets implemented. We might have a "normal" world at ground level, a sky world with a flat mapgen (that's more convenient for most players who want to build), and perhaps a third one which is more survival-orientated.
Apart from that, perhaps it might be a good idea to provide an expandable sky platform as a mod. Whenever someone wants a place to build on, the platform could be automaticly extended in one direction and the land protected for the new player. Or a new player just gets a platform 500m above the last one - we could stack 30-60 such tiny worlds on top of each other.

[spootonium]

Minetest's core mapgen dev hmmmm is planning core skylands and underworld realms ...

Well, this is good news! Especially about the "shadowing" problem.

On another train of thought: perhaps, if all that folk really want is more useable surface area, then generating the world as a cube-shaped planetoid might be an interesting way to provide that. Per-face physics and the co-ordinate system would be a problem, though. :-\

Another possibility might be to implement something like a "Stargate" between multiple maps of the same gamemode on the same (or indeed different) servers. That would provide theoretically infinite world size, limited only by the capabilities of the server(s).

Still another blue-sky thought to throw out there: if it is indeed true, that the functional lifetime of a map is inversely proprtional to the number of players, then perhaps rather than a server-client architecture, with the entire map located on one machine, perhaps something approaching a peer-to-peer model. In such a model, the active (populated) chunks of the map could be divided and swapped out on the players' machines, as well as distributing the load of generating & maintaining those chunks. The world size (and rate of generation) would then be scaled to the number of players.

If anyone starts to run out of space in a world just think vertical, sky and underground cities seem more interesting to me than the huge but very flat MC worlds.

Still the most elegant solution.

[glenalec]

While I also am not convinced that 64km3 is too small*, you could (at likely expense of a lot of recoding and module breakage) bump the x and y to 24 bits each, giving a world 16,000km square and 64km high. My only motivation for sececting those particular values is 24+24+16 bits fits neatly into a 64bit integer, which may or may not be useful.

*at 1m resolution, at least. I have always wondered what a voxel-world might be like at higher resolutions, such as foot or decimetre blocks.

[thetoon]

While I also am not convinced that 64km3 is too small*,

You all make it sound like I'm the only fool in need of more horizontal space. Fair enough, but there's plenty of cases where it might be useful, and if possible it should be done (that's what I was initially asking : pointers to where I could work improving it). Something like http://www.voxelmars.com/ (based on FOSS VoxelJS) or what Denmark did with Minecraft is currently impossible with Minetest, which is a shame.

I've seen here and there people talking about OSM->MT converters : this is a no-go if we're stuck to something the size of the Paris area.

[aldobr]

Replace current map model with a delta based one.

Now you are not limited by storage.

Maybe use 80bit float values to store coordinates, that would make for a pretty big world.

[sfan5]

Replace current map model with a delta based one.

Now you are not limited by storage.

Maybe use 80bit float values to store coordinates, that would make for a pretty big world.

I know of no processors that natively calculate with 80bit floats.

[aldobr]

Basically, all processors since 8087 are able to do 80bits :

http://en.wikipedia.org/wiki/Extended_precision

If you need integers, you can use quads instead of integers...

its not outside the realm of possibility, but needs a rework on the map storage method, or else maps can grow huge even for todays hardware.

a delta engine only stores what was changed from the chunk generator, so most visited chunks will not be stored in the disk.

[spootonium]

delta engine only stores what was changed from the chunk generator, so most visited chunks will not be stored in the disk.

Do I understand correctly, that this will mean that each chunk will need to be re-generated on-the-fly (and changes applied) every time that chunk is visited?

[aldobr]

Yes, but usualy this means that the chunk will be generated client side...

[spillz]

And you can do client side caching and other tricks to improve performance.

[thetoon]

Replace current map model with a delta based one.

Not even mandatory. Having a big adress space doesn't mean more space would be visited, let alone modified. A simple garbage-collector-like mechanism could simply delete unmodified blocks older than a given threshold.

I'm not sure there's already generation time included in blocks, but it would be much simpler than delta-based solution. And in most cases, probably not less efficient.

Maybe use 80bit float values to store coordinates, that would make for a pretty big world.

Sure enough, but that would make a lot of changes, wouldn't it? Map storage, protocol, server, client...

[BrandonReese]

delta engine only stores what was changed from the chunk generator, so most visited chunks will not be stored in the disk.

Do I understand correctly, that this will mean that each chunk will need to be re-generated on-the-fly (and changes applied) every time that chunk is visited?

This sounds like a really bad idea. So every unloaded chunk would go through mapgen again as they are visited. Some mapgen's do more than just generate terrain. They create villages, they place special objects, they spawn entities, (things that can't reliably be done client side), with no client side lua, lua mapgens wouldn't work. Complex mapgens can take several seconds per chunk to generate. That would bring servers to a grinding halt. Even if it's cached on the client, it still has to generate the chunks for every new client. At one point I had over 20,000 player files on my server. That means at least the chunks around spawn would have been regenerated over 20,000 times.

[Krock]

At one point I had over 20,000 player files on my server. That means at least the chunks around spawn would have been regenerated over 20,000 times.

If one of them changes the chunk, it will be saved and the mapgen will take no longer CPU time.

[spillz]

This sounds like a really bad idea.

The point of a delta engine is that you offload the terrain generation to the client (i.e. someone would need to add client side support for lua terrain gen). For special cases like villages etc. that could still be generated by the server and sent to clients. But this approach would still mean a lot less CPU and data used by server (at the cost of quite a lot more complexity).

BUT most importantly, it should be possible to do it both ways. Some game types would really benefit from client side terrain generation.

[thetoon]

If one of them changes the chunk, it will be saved and the mapgen will take no longer CPU time.

Only if one of them changes the chunk. As world size increases, more and more blocks will be "just generated" and never modified, statistically.

[thetoon]

The point of a delta engine is that you offload the terrain generation to the client (i.e. someone would need to add client side support for lua terrain gen).

"someone" might have better things to do. I'm all for more and more features, especially interesting ones (client-side Lua sure would be nice to have). But I don't think we'd need to wait for client-side Lua to have bigger world sizes. As someone put it before, it's basically bigger coordinates, plus some mechanisms to prevent server worlds to grow up insane, be it :

* as simple as limiting coordinate-size from config file - so that bigger coordinates would have no effect on unwilling servers,
* or as complicated as variable compression formats, with different levels of detail
* and of course everything inbetween.

What I'd propose - and might gradually try to implement, unless someone else wants to - is the following plan :

Step1 : Increase coordinates' size. This is by far the harder part, as there might be references to them everywhere. Someone more used to MT's internals might think it easy, but it needs to understand the big pictures before being done.

Step2: Test it. Just to be sure we haven't left some references to the old coordinates format somewhere. Just to be safe, the old format should still be available, at least from a compile option (and preferably from a config file option).

At that point, we should be able to have much bigger worlds. Efforts should be on preventing disk size from getting out of control. Hence, Step3.

Step3: As seen from the previous posts, there's a lot of different options here, but here's mine. Let's add (optional) timestamp to blocks when writing them to disk (no matter what database backend is used) with a special value (0? MAX_VALUE?) for blocks which have been modified. Don't touch the block loading code.

Step4: On a regular basis (cron on a server, could be plugged to game start/stop on singleplayer), an external script (probably trivial, though it might fit some backends more than others) should be able to delete blocks older than a given or calculated timestamp, leaving the "special value" ones alone, as well as the "no timestamp" ones if any. An admin could decide to make the script adjust the timestamp based on disk usage, for instance.

[Calinou]

Adding the ability to not save unmodified MapBlocks would help for file size, but increase global CPU usage since parts of the world may be generated several times.

[aldobr]

Just use a cache.

Current model:

1st time a player enters a chunk, it asks the server to generate it.
server sends the chunk to the client.
server stores the chunk into the database.
doesnt matter if the chunk is exactly as it was when generated. Just passing by it makes it be stored into the database.

Delta engine:

client enters a chunk.
chunk is generated client side/server side (maybe using GPU) at the same time (no transfer via net needed).
client asks the server for deltas to apply to that chunk.
chunk is placed into a lru cache (least recently used chunk is discarded).
when player breaks/places a node, deltas are generated by the client and sent to the server.
deltas are stored into the database back end.

This is a case of storage x cpu.

Supposedly storage would be faster, but all data must be stored in a data structure (in that case a database) that has finite ammount of access time.

When you reach 10gb sized databases, no matter how efficient the database is, there WILL be huge overheads, reaching the case where database storage is LESS efficient than on-the-fly generation (at least when compared to C++ mapgen)...

You can use other tricks to improve database access in the current model...

Distributed database :

add mariadb or postgres database backend (it would be slower than sqlite initially).
split the world in superchunks and store each superchunk in a different mariadb server.
ie.: make a distributed database system and share the load accross multiple servers connected via internet.

[BrandonReese]

split the world in superchunks and store each superchunk in a different mariadb server.

Could this be done with sqlite also? I've wondered if something like this would be beneficial.

[aldobr]

It can be done (if someone is willing to work on it)

but sqlite is not a RDBMS

so you would need a server at each machine accepting connections and storing that in sqlite (ie.: you end up remaking a RDBMS).

Its easier to work with a RDBMS to start with...

Using a checkerboard distribution of chunks across multiple RDBMSs.

it would be faster and cheaper (for a small server) to use a RAID setup and distribute the load across multiple disks at the sector level or use a SSD, because most overhead for huge maps is at the disk i/o.

[spillz]

The point of a delta engine is that you offload the terrain generation to the client (i.e. someone would need to add client side support for lua terrain gen).

But I don't think we'd need to wait for client-side Lua to have bigger world sizes. As someone put it before, it's basically bigger coordinates, plus some mechanisms to prevent server worlds to grow up insane, be it [/snip]

hmm... let's see. 10 players in different unexplored areas of the map. (And if you think that's unrealistic, it's not clear to me why you want unlimited worlds anyway.) Each decides to do some exploring. Map gen is now doing 10x as many requests. Lag fest. You can't get around having enough CPU and, more importantly, bandwidth, to do map gen in game types that involve a lot of exploration. IMHO, the client approach is the only way to really get the game to scale given the current state of technology. With reasonably modern client hardware you can pretty much lock 60 fps and a long draw distance with terrain generation going on in the background and never see map pop-in.

[Sokomine]

Step3: As seen from the previous posts, there's a lot of different options here, but here's mine. Let's add (optional) timestamp to blocks when writing them to disk (no matter what database backend is used) with a special value (0? MAX_VALUE?) for blocks which have been modified. Don't touch the block loading code.

YES, please! Do this. I've been hoping to get this feature for some time and even considered implementing it myshelf but didn't get around to it. A timestamp for "generated at" and "last modified at" at database level (add new columns for those) would help to clean up the map from time to time. With a server growing, more and more territory is explored - while the likelyhood of someone ever passing by there again decreases. Backup and distribution of a map also gets more difficult if the size is in the GB range. Just look at my "manual" copy of Redcrabs world: around 100 MB, while the actual world file is several tens of GB.

Whenever an existing chunk is changed, the server could do a backup of the old chunk on database level, thus saving an edit history and allowing to restore individual chunks to previous states.

Backups/downloads of the map could be limited to chunks that where actually changed, resulting in a very small map file, saving bandwidth and storage space.

When a mod is added that changes mapgen significantly, it wil be possible to delete all the old unmodified blocks done by the old mapgen, while keeping all the structures players built. The actual mapgen changes would show up in existing but unmodified areas and not force players to walk further and further away from spawn.

All the above could be done with pretty simple changes. There's something else I'd like to have as well: Client-side mapblock cache. Download of mapblocks is still slow, and it destroys immersion if you notice after half an hour that there's a giant mountain right in front of you, or that the lake you considered to be an ocean is smaller than expected. The copy of the map you explored last time will still be accurate in most areas - and where it's outdated, the server'd send an update anyway once it gets around to. Such a client-side caching is already done with sfan5's special map-saving client. Now let that one store its mapblocks for each server independently and let it load mapblocks from there when connecting to a server - and immersion ought to increase significanlty. There will be a world to look at instead of white fog.

Both of the two wishes above have nothing at all to do with increasing the square area MT may cover. In my opinion, around 30km in each direction are far more than what even huge servers use up. The players are just drawn to the borders of the map because those borders exist. Sure it would be nice to have a larger area, but it's not really that urgent.

[Wuzzy]

I am not taking any “The Minetest world is too small and needs to be larger.” discussion serious until anyone comes with a REAL example which proofs there is actually a real need for larger worlds. The examples which have been shown have all been pretty hypothetical.

For example, a Minetest world where JUST the surface is almost completely claimed/built upon is yet to be seen …
Can anyone show me a Minecraft server where a surface of 66000×66000 is almost completely covered?

[Thermal_Shock]

Can anyone show me a Minecraft server where a surface of 66000×66000 is almost completely covered?

I don't think it exists. Plus most large Minecraft servers limit the size of their maps to less than that anyway in order to keep the maps filesize down.

Even this server with a few thousand active players on a town server barely makes use of their 16000x16000 map.

The potential size of a Minecraft map sounds great as a feature. But Minetests current map limit seems to fall on the more practical side of things.

[Sokomine]

I am not taking any “The Minetest world is too small and needs to be larger.” discussion serious until anyone comes with a REAL example which proofs there is actually a real need for larger worlds. The examples which have been shown have all been pretty hypothetical.

It's a psychological problem, not a physical one. The area around spawn gets crowded, and then suddenly players start out calling for a new map even tough far less than 100x100m around spawn has ever been built on.

[TenPlus1]

I like the idea of vertical realms appearing in the world so players can explore upwards or downwards to find new blocks to play with... As for map functions, I'd rather have time spent on making maps load/save quicker instead of making them larger so that multi-player servers could use a headerless setup and maybe even use the GPU to do all the work...