Un-explore the world ?

[Marshall_maz]

Hi everyone ,

Is there a way to un-explore unused part of the world and get it removed from the world database.? ( As if you have never been to that part of the world).

When I started my world I did a lot of exploring opening big parts of the world. There is currently big parts of the world unused. Now when I add mods that add things like new trees etc. it will only be created when I get to unexplored areas and that part of the world get's created and added into the database. So now I have to travel very far to get any newly added items as I have to travel to unexplored areas for them to get created. So I would rather un-explore the current unused areas of the world ( if at all possible ) and re-explore them after adding new mods. Starting a new world is not an option as there is way too many work and buildings already put into this world.

Thanx

[Krock]

There was already the idea to not save unchanged chunks.I don't know about the progress of this.
For sqlite3 databases (default), you can use a sqlite3 database editor. A MT-specific tool for this doesn't exist AFAIK.

[solars]

For my Karsthafen map, I had written a quick script to shrink my map. I wrote it only for my personal use, so its quick and dirty... In this version, I have translatet the error-messages to english.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

#!/usr/bin/perl

use strict;
use CGI;
use DBI qw(:sql_types);
my $y;
my $z;
my $a;
my $b;
my $cgi=CGI->new();
my $Game=$cgi->param('game')?$cgi->param('game'):'.';
my $Out=$cgi->param('out')?$cgi->param('out'):'map.sqlite';

if(-e "$Game/map.sqlite" && -e "$Game/savegame.txt"){
  my $dbGame=DBI->connect("dbi:SQLite:$Game/map.sqlite",'','')||die 'Cannot open map.sqlite';
  my $dbSave=DBI->connect('dbi:SQLite:dbname=:memory:','','')||die 'Cannot create inmemory database';
  my $dbInput=$dbSave->prepare("CREATE TABLE `blocks` (`pos` INT NOT NULL PRIMARY KEY,`data` BLOB)");
  $dbInput->execute();

  open(FILE,"<$Game/savegame.txt")||die 'Cannot open savegame.txt';
  foreach(<FILE>){
    my @line=split(/\,/,$_);
    if($line[5]){
      $line[0]=int($line[0]/16-1);
      $line[1]=int($line[1]/16+1);
      $line[2]=int($line[2]/16-1);
      $line[3]=int($line[3]/16+1);
      $line[4]=int($line[4]/16-1);
      $line[5]=int($line[5]/16+1);
      print 'Writing ',$line[6];
      for(my $x=$line[0];$x<=$line[1];$x++){
   for(my $y=$line[2];$y<=$line[3];$y++){
     for(my $z=$line[4];$z<=$line[5];$z++){
       my $pos=minetest_get_block_as_integer($x,$y,$z);
       my $dbData=$dbGame->prepare("SELECT * FROM `blocks` WHERE `pos`=?");
       $dbData->execute($pos);
       my $rows=$dbData->fetchall_arrayref;
       if($rows->[0][1]){
         my $dbTest=$dbSave->prepare("SELECT * FROM `blocks` WHERE `pos`=?");
         $dbTest->execute($pos);
         my $test=$dbTest->fetchall_arrayref;
         unless($test->[0][1]){
      $dbInput=$dbSave->prepare("INSERT INTO `blocks`(pos,data) VALUES (?,?)");
      $dbInput->execute($rows->[0][0],$rows->[0][1]);
         }
       }
     }
   }
      }
    }
  }
  print "\n";
  $dbSave->sqlite_backup_to_file($Out);
  $dbSave->disconnect();
  $dbGame->disconnect();
}else{
  print "Minetest savegame with savegame.txt not found.\n\nUsage of this script:\n";
  print "nt_clean.pl game=[path to the savegame directory] out=[path for the new database file]\n\n";
  print "Width no use of the game parameter, the data is read from the actual directory.\nWidth no use of the out parameter, the database is written to 'map.sqlite'.\n\n";
  print "If you use the script from the savegame directory with no set parameters, the game database will be overwritten with no warning!\n\n";
  print "The file savegame.txt is a CSV file with no quotation marks and with comma as separator and following table columns:\n";
  print "1 - x from\n2 - x to\n3 - y from\n4 - y to\n5 - z from\n6 - z to\n7 - Name of the block\n\n";
}


sub minetest_unsigned_to_signed{
  if($_[0]<$_[1]){
    return $_[0];
  }else{
    return ($_[0]-2*$_[1]);
  }
}

sub minetest_get_integer_as_block{
  my $int=shift;
  my @return;
  $return[0]=minetest_unsigned_to_signed($int%4096,2048);
  $int=int(($int-$return[0])/4096);
  $return[1]=minetest_unsigned_to_signed($int%4096,2048);
  $int=int(($int-$return[1])/4096);
  $return[2]=minetest_unsigned_to_signed($int%4096,2048);
  return @return;
}

sub minetest_get_block_as_integer{
  return $_[2]*16777216+$_[1]*4096+$_[0];
}

If you save the code to 'mt_clean.pl' and put a savegame.txt to the directory with your map.sqlite-file, you can use the script with (if your savefile is in '.minetest/worlds/world' and a copy for the new version in '.minetest/worlds/newworld':
perl mt_clean.pl game=.minetest/worlds/world out=.minetest/worlds/newworld/map.sqlite

Bevor you use the script, you have to put a savegame.txt-file in the directory with your map.sqlite-file. As an example:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

-100,100,-20,50,-100,100,Center of the World
20,300,200,250,-120,-50,My Skycastle

Every Line is a definition of a area, that should be saved in the new database. The second line is an area from 20,200,-120 to 300,250,-50.
It will save a bigger area as you write in the file, it put the 16x16x16 Fields big blocks in the new database. But big landscapes outside this areas will be unexplored after that.

I hope this script can help you. It works from version 0.4.10 of minetest. In older versions, the landscape-generator destroys the buildings.

PS: Dont use the script form the Savegame-Directory without saving the map beforehand. You can use the script without parameter from the path with your map.sqlite-File and it will overwrite it with the belittled file without asking. With an faulty savegame.txt, your map can be destroyed. To copy the map and use the script on the copy without parameters make the new map in a simple way.

[Marshall_maz]

I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.

[Krock]

I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.

Mostly, I delete the file map.sqlite in the world folder to reset the terrain.
But seriously, those worlds aren't that big, mostly I have file sizes between 5 and 10 MB.

[Marshall_maz]

I tried this script but doesn't work. I get sqlite db modules failures.

I can't believe there is no tool to do this. This must be a problem for lots of players all the time.

Mostly, I delete the file map.sqlite in the world folder to reset the terrain.
But seriously, those worlds aren't that big, mostly I have file sizes between 5 and 10 MB.

But if I do that I loose everything already build in the map ?

Mine is 185mb , we are 2 users playing in the game and both of us already has lots of buildings and mines and other stuff. Don't want to loose all that

[solars]

I tried this script but doesn't work. I get sqlite db modules failures.

Is your perl without the dbi-modules?

Then you can install the dbi modules. At debian, you can do it as root with

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

aptitude install libdbi-perl

The dbi-perl package will be found in any linux-distribution.
If not, you can download ist there: http://dbi.perl.org/

And the path at the parameter 'game' must be right, if not, the script can't found the sqlite-database of your map.

Your world is 185mb big? My actual shrinked map is 50mb big... The original map: 6358mb!!! Without my script, I havn't a chance to build a download.

[prestidigitator]

I think this would be a GREAT addition to the API. We have a minetest.forceload_block(pos) call now. I think it would work well to also have:

• minetest.delete_block(pos)
  Marks the block containing pos for deletion from the database when it would normally be unloaded (may not be until server restart). This will not happen while there are active players in the nearby area. Removes the block from the forceload list if it is currently in it (also, adding a block to the forceload list cancels deletion).
  
• minetest.cancel_delete_block(pos)
  Unmarks the block containing pos for deletion. Has no effect if the block has already been deleted. Simply serves to cancel a pending delete.
  
• minetest.register_on_blockdelete(function(pos))
  Registers a callback to be called when (after) the block is actually deleted. On server startup, all mods are initialized before these callbacks are made.

Small side note though: It may not help space consumption much, but for pure-Lua mapgens you could effectively do this by having a hook in the mod called both during map generation and when someone wants to "regenerate" an area to clear changes, etc. Yet another plus for doing mapgen entirely in Lua. You could probably even fake an "ongenerate" callback to other mods....

[Sokomine]

Problem with that is that you don't want blocks to be deleted once players started building there. And finding out if they did is not that easy. I wished for a timestamp for last block change (which might be easy to do in sqlite and other sql databases, but not that easy in leveldb), but...blocks do get changed by abms as well. Especially the intresting surface blocks. While growing flowers are nice, they don't count as buildings worth preserving. The only way I can currently see is to have a database right from the beginning which counts player modifications per block.

[prestidigitator]

Problem with that is that you don't want blocks to be deleted once players started building there. And finding out if they did is not that easy. I wished for a timestamp for last block change (which might be easy to do in sqlite and other sql databases, but not that easy in leveldb), but...blocks do get changed by abms as well. Especially the intresting surface blocks. While growing flowers are nice, they don't count as buildings worth preserving. The only way I can currently see is to have a database right from the beginning which counts player modifications per block.

True, MOST of the time you probably wouldn't want to do it on its own. You'd probably also want to make use of events like on_dignode and on_placenode and maybe even functions like minetest.find_nodes_in_area and/or minetest.get_objects_inside_radius. However, adding to the actual API isn't about constraining possibilities, so better to add functionality that can be used generally than to add something very special-purpose. Who is to say someone wouldn't want to create a mod that allows an admin to blast away sections of the map even if players HAVE built there? Let them build in safeguards (such as warnings to players or cancelling the block deletion on certain events or requesting a custom mapgen to compare against what it WOULD generate for the block) if they like.

[mike]

It would be easier to remove big chunks from the world database instead of excact coordinates.

If you have 5 areas where people act, then you should preserve them.
I havent looked into the sqlite3 yet but if the world is a part of chunks you could just preserve some and delete the rest.

[lag01]

This simple Python script is as far is i got now:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

#!/usr/bin/env python

import sqlite3
import time

source = r'<PutYourMinetestFolderPathHere>/worlds/<PutYourMapNameHere>/map.sqlite'
target = r'<PutYourMinetestFolderPathHere>/worlds/<PutYourMapNameHere>/map.sqlite.rebuilded'

sourceconn = sqlite3.connect(source)
targetconn = sqlite3.connect(target)
sourcecursor = sourceconn.cursor()
targetcursor = targetconn.cursor()
targetcursor.execute("CREATE TABLE IF NOT EXISTS `blocks` (`pos` INT NOT NULL PRIMARY KEY, `data` BLOB);")

i=0
for row in sourcecursor.execute("SELECT `pos`, `data` "+" FROM `blocks`;"):
    pos=getIntegerAsBlock(row[0])
    if pos[0]**2 + pos[2]**2 < (1600/16)**2 and pos[1]>(-60/16):    #1600 blocks in each direction and 60 blocks deep
        targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))
        if ++i > 100:
            targetconn.commit()
            time.sleep(1)
            i=0

targetconn.commit()
sourceconn.close()
targetconn.close()


I think it is possible to use mapformat and minetestmapper as example to look inside blocks and actually sort them by their content.

One problem i found is that different mapgen versions are generating world differently, so boundary between old map blocks and new map, can look ugly.

[lag01]

I did that! Github :)

[mike]

how does it work?
can you set some areas which should be preserved?

[lag01]

how does it work?
can you set some areas which should be preserved?

It preserves round area around map center:
remap.py

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

if pos[0]**2 + pos[2]**2 < (160/16)**2 and pos[1]>(-60/16):
    targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))


Also it preserves blocks(16x16x16 cubes), which contains cobble, chests etc.

If that not enough for you, then you need to alter script code. For example add:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

elif pos[0]>=(200/16) and pos[0]<=(250/16) and pos[2]>=(100/16) and pos[2]<=(150/16) and pos[1]>(-16/16):
    targetcursor.execute("INSERT OR IGNORE INTO `blocks` VALUES (?, ?);", (row[0], row[1]))


That will save area around x 200 and z 100 and y above ~0.

[mike]

it would be nice if there would be a special preserve node.
if you place such a node in an area and set the max distance of protection it would work
so you can go to your important "world" inside your world and set this node (admin only).
then the script knows what to protection and its visual :)
its nearly the same function as you already coded, but just a special block with distance for each x,y,z, coord.

[lag01]

its nearly the same function as you already coded, but just a special block with distance for each x,y,z, coord.

Well, finding nearby blocks could be little harder that what i have already coded, especially with variable distance.
I will look what i can do, meantime you can test if current code do the job for you?

[mike]

iam not sure if i can test ist.
currently i have many worlds in my world.
every world has special coordinates like (e.g.):
one 0,0,0
2nd -10000,0,0
3th -20000,0,0
4th -10000,0,-10000
5th 10000,0,0

i have to preserve all of these worlds and for all of these worlds i need to set an area.

so i have to create an sql command that preserves all nodes which are 1000 or more near to these coords.
otherwise i will loose most of the worlds.

[lag01]

iam not sure if i can test ist.
currently i have many worlds in my world.
every world has special coordinates like (e.g.):
one 0,0,0
2nd -10000,0,0
3th -20000,0,0
4th -10000,0,-10000
5th 10000,0,0

i have to preserve all of these worlds and for all of these worlds i need to set an area.

so i have to create an sql command that preserves all nodes which are 1000 or more near to these coords.
otherwise i will loose most of the worlds.

I think, current script can save most parts of all 5 your worlds, you just need to specify correct path to your sqlite file and file where to copy data (i hope you use sqlite?)
Otherwise i can write code with coordinates you specified.

[mike]

i wrote the coords to you via pm.

for some world it could be nice to reset underground someday to uncripple the mines.

[lag01]

i wrote the coords to you via pm.

for some world it could be nice to reset underground someday to uncripple the mines.

Ok, i sent you PM with script with your coordinates calculated.
You did not answered if you tested original script, i think that is mean of you...

[mike]

which is the "original" script?
many solutions are posted here, but all are hard to use.
i havent taken time to try out this because i have to take much time because i dont want to notice db damage after some weeks.

[lag01]

which is the "original" script?
many solutions are posted here, but all are hard to use.
i havent taken time to try out this because i have to take much time because i dont want to notice db damage after some weeks.

You can download zip from Github https://github.com/AndrejIT/map_unexplore/archive/master.zip
Then you need to write correct path to your minetest world folder in two places, something like

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

source = r'~/.minetest/worlds/world1/map.sqlite'

About DB damage i have not though... DB table is quite simple, script just takes data as string from one DB and writes same unchanged string to another DB. The only chance that something will be broken is if sqlite and python handle string encodings incorrectly, on my computer that is not happening.

[aldobr]

minetest map file format needs to be appended with two fields per chunk:

update -> the date of the last update this chunk received.
modified -> weather this chunk was modified from the one generated by the mapgen.

with both field we can write map shrinkers.

[lag01]

minetest map file format needs to be appended with two fields per chunk:

update -> the date of the last update this chunk received.
modified -> weather this chunk was modified from the one generated by the mapgen.

with both field we can write map shrinkers.

Would be good addition. That could help to preserve blocks(chunks) where no player crafted blocks are added, for example tunnel parts where is no torches or buildings built completely from natural resources like dirt, sand, raw wood etc.

Upd.
Actually there already is "timestamp" field, i wonder what that do...
Upd2.
Bot we need to deal with blocks modified only by water/lava flow then :(

[Ivà]

@solars:
I tried your script and works like a charm. But I had to install libdbd-sqlite3-perl on a debain system to avoid the db modules failures.

@lag01:
I tried your script too and it does what you say :-), but I think it would be better if it saves a configurable area around all the blocks defined in useful_block_evidence. People tends to modify the surroundings of the things they build, with this script as is, these modifications are'nt preserved. If we could define an area around it would be fantastic.

[eli_3]

Hello,
I minetest version 0.4.10 and I would like to install the 0.4.12.
If I install the 0.4.12 I duplicate the worlds of 0.4.10?

[Don]

Hello,
I minetest version 0.4.10 and I would like to install the 0.4.12.
If I install the 0.4.12 I duplicate the worlds of 0.4.10?

When you install a new version of minetest it does not change existing worlds. They will still be there.

[eli_3]

thanks

[prestidigitator]

There's a somewhat easier way to do this. Export the areas you DO want to keep using a mod. Create a new world with the same seed, mapgen parameters, mapgen-influencing mods, etc. In the new world, import the whitelisted blocks with a mod when their locations are generated, and let the normal map generation take care of the rest. It's obviously still some work, but doesn't require mucking with the database directly.