[Mod] Synchronize Worlds [WorldDiff]

[Casimir]

You might, or might not, know about the Tinas project. It's a collaborative map where everyone can share their new versions. The general problem is that - until now - you had to upload the whole map.
So I made this mod. It automatically saves all the changes you made inside a map into .we files. You can then share those files and others reinsert them into their own copy of the map. Quite similar to the version control system git, everyone has a local copy of the map and all the changes they know of, making it fully distributed and redundant.

This Topic is in Modding General because the mod is in development and you have to expect many changes. The naming of the files will probably change too, and there will be no support for old files. Make a backup of your world before using this mod.

Usage:
Download install and activate this mod.

Person Nr. 1
Play. In the debug you will see the message "[worlddiff] X nodes saved".
In your world folder you will find a folder named "wd" and inside it one named "output".
Compress that folder (e.g. as .zip or .tar.gz) and share it with others.

Person Nr. 2
In your worldfolder create a folder named "wd" and inside it one named "input".
Extract the archive.
Copy its contend into "input".
Play. In the debug you will see the message "[worlddiff] X nodes loaded".

The plan is, that you will be able to synchronize worlds with many many other players automatically over the internet. Using software like owncloud or freenet. You could have several servers using the same world at the same time. Or different worlds sharing only certain areas. Or you synchronize worlds on your own harddrive. I would like to extend this mod, so you can use it for many more thinks I can't imagine. And I would be happy if there are some people helping with this.
Ideally, we will soon have one big world, that is shared over many different channels, so this world becomes one giant distributed server that is always up.

In my testing I found that the worldedit format compresses very well. In one world where I saved every Mapblock as a .we file I had this results:
raw:

• wd_output 41.3MB
• map.sqlite 6.7MB

compressed:

• wd_output.tar.gz 971.7kB
• map.sqlite 3.1MB

Now add that you only need to share those segments you alter, the data volume to transmit is quite small. A screenshot of minetest with 1363x768 pixel is about the same size (1MB to 2MB) as the compressed we files of this testing map.

git: https://github.com/casimirkapazi/WorldDiff
download:
.zip
.tar.gz

depends: nothing
license: AGPLv3

[Casimir]

In the case someone wants to try it. Here is some input for Tinas.

[Casimir]

I read a bit into freenet and they actually have an protocol that lets you interact with it from within another program. But I'm not a programmer, here I reached the end of my limited skills. I could start to learn the bits I need, but I even don't know what programming language would be best to use.
The plan would be:
Try to push into and get things from within freenet.
Automate the packaging of the we files and uploading.
The same thing backwards, download and unpack.
Find some mechanism to know what to download and what not (timestamps or something).

Alternative I will try to set it up so far that it can be used by hand.

I know that very few people are using freenet, and very few use Minetest. The reason I want to do this, is just because it is possible. And as soon it works it will be beautiful from within.

Some warnings about freenet:
It might be illegal to run it in your country.
Heavy heavy disk use. Better use an old harddrive for it.
High up and download.

[Casimir]

At the moment the mod is terribly broken.
Using the normal worldedit it only loads new nodes. When one node should be replaced with air, it does keep the old one. So I set the whole area with air before loading the .we file. But when loading the file was faster then setting the air, it just deleted the part of the world. Both is far away from what I want - just replace all of the changed nodes.
I don't know if it is better to change worldedit, or to copy the relevant code into worlddiff and change it there.

[hoodedice]

At the moment the mod is terribly broken.
Using the normal worldedit it only loads new nodes. When one node should be replaced with air, it does keep the old one. So I set the whole area with air before loading the .we file. But when loading the file was faster then setting the air, it just deleted the part of the world. Both is far away from what I want - just replace all of the changed nodes.
I don't know if it is better to change worldedit, or to copy the relevant code into worlddiff and change it there.

For now, I think it is best to fork WE and do your stuff. When stuff gets stable, merge. I'm looking forward to this mod getting stable =)

[Casimir]

I made some big changes.
First of all it now saves all the nodes in the segment including air. This makes the .we files bigger, but fixes the problem described above. Also input and output folders are now in "worlds/[yourworld]/wd/input" and "worlds/[yourworld]/wd/output". Now this mod no more depends on worldedit, so you just need to install worlddiff.

[lunisol]

I'm probably one of the few people here that use both minetest and freenet. Like you said freenet is very heavy, disk usage wise. and even small files can take forever to load depending on several factors such as the files popularity(how many others have gained access to it), how many nodes(machines) you are connected to within freenet, and what security level freenet you are running.
Although I would be quite happy to share my freenet node with anyone here in the interests of making both our connections faster.

[Casimir]

For those interested in this project here is a little update.

First, what I am trying here is heavily pushing the boundaries of what I am capable of. I'm not a programmer and never had a real programing course. I also work on many other projects in my free time. So don't expect that anything happens here soon. But still, I wont give up on the idea of a fully decentralized server for a game like Minetest.

The current date todo list:

• The whole project should be build in modular parts. This way they can be easily replaced and reused for other projects. We will at least have: something to get and apply the data (=WorldDiff), a common data format - at best human readable (=worldedit format), a decentral data storage (=freenet), an application to up- and download the files to the datastore (yet to be made, see below), optional a kind of filter that manages the versions and decides which to apply.
• worlddiff actually is an ugly hack. It would be much better to get the data directly out of the database. But this has several issues: We would have to decide on one backend or do the work of supporting all the things minetest offers. We can not apply the changes in a running game, for something like this we would have to write a patch for the engine. Even then it is something tricky.
• If we still use worlddiff we need to fix bugs (there are many) and make it work with the new mod security (this should be no problem).
• Freenet is getting much better lately and I'm confident that this is the best choice. One of the developers did write up some instructions on how to create clients for up- and downloading to freenet (1,2). With this it does not seem to complicated to create something for our purpose. He even calles it "suitable for anything from filesharing over chat up to decentrally hosted game content like level-data" Freenet also offers a handy versioning for files and by design unused files will be forgotten.
• Once we put the world data up on freenet we should decide on a scalable data structure that allows to access just the part of the world one needs in that moment, without having to search a list of all the mapblocks. Maybe some treelike structure organized in folders and textfiles that only contain the links to the mapblocks.