map.sqlite backup with database locking

User avatar
vitaminx
Member
 
Posts: 49
Joined: Tue Dec 30, 2014 15:08
GitHub: vitaminx

map.sqlite backup with database locking

by vitaminx » Thu Jan 22, 2015 22:59

Hello everyone,

I'm currently looking into methods to do daily backups of my Linux minetest server and I ran into a difficulty.

If you do just a simple file copy of map.sqlite while the server is running, you end up with a corrupted backup.
This is because while the database is copied it is changed by server writes.

One way to do a sane copy is to stop the server for as long as the backup takes, and this seems to be the way most server admins do it.
That also means that all players get disconnected!

But I've tested another method:

I just leave the server running and tell the backup script to lock the database before copying it.
Once finished, the transaction is rolled back and the lock is released.

My test results for a 565MB map.sqlite file are pretty promising.
The script takes 12 seconds on my server to lock the database, copy the file and roll back.

There is no noticeable delay or lag for the players.

EDIT: I've modified the scripts to make a backup of the whole world directory, now it takes 19 seconds but still there's no lag at all for the players.

Here's how it works:

First, install the package "sqlite3" on your server.
You need at least version 3.8, the 3.7 version from Debian wheezy doesn't have the ".shell" command yet!

Save this script as "minetest-backup.sh" and make it executable (chmod 755 minetest-backup.sh):

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
#!/bin/bash

DEST="/opt/backups/minetest"
TMP="/tmp/minetest/world"
MAP="/home/minetest/.minetest/worlds/world/map.sqlite"
SQL_SCRIPT="/opt/scripts/bin/minetest-backup.sql"
FILE=$(/bin/date -I)

#############################
# remove temporary directory

function remove_tmp {
    if [ -e "$TMP" ]; then
        rm -rf "$TMP"
    fi
}

#############################
# create temporary directory

function create_tmp {
    mkdir -p $TMP

    if [ "$?" -ne 0 ]; then
      echo "Can't create backup directory '$TMP'. Aborting." >&2
      exit -1
    fi
}

#############################
# create backup directory

function create_dest {
    if [ ! -e $DEST ]; then
        mkdir -p $DEST

        if [ "$?" -ne 0 ]; then
            echo "Can't create backup directory '$DEST'. Aborting." >&2
            exit -1
        fi
    fi
}

#############################
# backup minetest world

function backup {
    echo ".read $SQL_SCRIPT" | time sqlite3 -echo "$MAP"
    nice tar -czf "${DEST}/world_${FILE}.tar.gz" "$TMP"
    chown -R backup: $DEST
}

#############################
# remove old backups

function remove_old {
    find $DEST -type f -mtime +6 -exec rm -f {} \;
}

remove_tmp
create_tmp
create_dest
backup
remove_old
remove_tmp


Then save the following snippet as "minetest-backup.sql":

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
begin immediate;
.shell cp -r /home/minetest/.minetest/worlds/world /tmp/minetest/
rollback;


Of course you need to change the paths to your personal preference, but I hope you get the idea!

Tell me your experience and if it works for you, and if not, why :)

Bye!
vitaminx
Last edited by vitaminx on Fri Jan 23, 2015 20:36, edited 2 times in total.
 

User avatar
Krock
Member
 
Posts: 3598
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker

Re: map.sqlite backup with database locking

by Krock » Fri Jan 23, 2015 17:41

So you only update the map but not the players' files?
I compress the complete world to a .7z file while the server is offline.
This results in backups à 300 MB of a 1 GB big map.
Newest Win32 builds - Find a mod - All my mods
ALL YOUR DONATION ARE BELONG TO PARAMAT (Please support him and Minetest)
New DuckDuckGo !bang: !mtmod <keyword here>
 

User avatar
vitaminx
Member
 
Posts: 49
Joined: Tue Dec 30, 2014 15:08
GitHub: vitaminx

Re: map.sqlite backup with database locking

by vitaminx » Fri Jan 23, 2015 20:38

Hi Krock, you're absolutely right, the map and player files need to stay consistent.
I've changed the scripts to copy the whole world directory while the database is locked, not just map.sqlite.

Greetings,
vitaminx
 

Sokomine
Member
 
Posts: 2980
Joined: Sun Sep 09, 2012 17:31

Re: map.sqlite backup with database locking

by Sokomine » Thu Jan 29, 2015 21:30

I'd love to have a more extensive backup and i.e. store blocks that where changed every 15 minutes or so. This won't work due to quite a lot of mods changing nodes constantly (technic, pipeworks, plants, ...). Blocks where players did anything ought to be marked for later backup.
A list of my mods can be found here.
 

User avatar
lag01
Member
 
Posts: 190
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag

Re: map.sqlite backup with database locking

by lag01 » Sat Jan 31, 2015 22:33

Just copying whole directory sounds simple, nice!
The backup process on my server is asking for improvement for quite a long time already. I just hope, it will not be too cruel to run server non stop, without giving players breaks to rest :)
So begin immediate; and rollback; are sqlite commands, will seamlessly lock and release database... Maybe that should be incorporated in my script: viewtopic.php?f=14&t=10333

Sokomine wrote:I'd love to have a more extensive backup and i.e. store blocks that where changed every 15 minutes or so. This won't work due to quite a lot of mods changing nodes constantly (technic, pipeworks, plants, ...). Blocks where players did anything ought to be marked for later backup.

Well, in theory it is possible to use minetest mod, which monitors player activity, like placing/digging blocks, or intercept interacting with chests and signs etc. and collect information about which blocks should be backed up.
 

Apo
Member
 
Posts: 19
Joined: Thu Dec 18, 2014 16:36
GitHub: apoleon
IRC: apo
In-game: apo

Re: map.sqlite backup with database locking

by Apo » Sun Feb 01, 2015 22:34

A lot depends on the file system. You can backup your server without shutting it down, if you use a file system like btrfs. It enables you to create snapshots of your filesystem while your Minestest server is still online.
 

User avatar
bphenix
New member
 
Posts: 4
Joined: Tue Jul 29, 2014 04:00
In-game: bphenix

Re: map.sqlite backup with database locking

by bphenix » Thu Jul 23, 2015 15:36

Hello.

vitaminx wrote:If you do just a simple file copy of map.sqlite while the server is running, you end up with a corrupted backup.
This is because while the database is copied it is changed by server writes.
[...]
Tell me your experience and if it works for you, and if not, why :)

Bye!
vitaminx

So do I.

I also would appreciate a live backup of the server, especially the world sqlite file.
I didn't found any alternative to yours, but to shut-down the server during the backup.

I tried your solution, but it crashed the server.
Also, version 3.8 is not sufficient. As example sqlite3.8.2 doesn't have the .shell command neither. But I downloaded the latest version from the web.

More than that, I'm not convinced by the principle: you say you lock the database, then you copy it, then you unlock it (rollback). In RDBMS world, "transaction"/"commit" or "rollback" doesn't has the same meaning as a global semaphore. Only the data that are accessed can cause some locks to be taken between concurrent clients writing in the database... And here, you start a transaction, but you actually don't lock anything, as you don't touch anything. Maybe I'm wrong, but then it means your solution relies on some specific sqlite magic. In this case, can you explain ?

As second question, why isn't it possible to use sqlite ".backup" command ? I tried it but it also crashed the minetest server... I suspect it's because the hardware on which it is running is limited, but maybe also because of locks that, in this case, will happen, and that the minetest server doesn't accept to live with for too long. Am I right ?

Do you test your sqlite file status after such a backup (using the sqlite "PRAGMA main.integrity_check; ") ?
What is the result ?
Did you already tried to launch a (test) server running on one of those worlds ?

And last but not least, if I organise an automated weekly backup with minetest server shutdown, I would appreciate to inform the users by a "shout" message, but I didn't found yet a way to do so from a bash script or a cron job. Any idea ?

Thanks in advance.
Moderated extremist.
 

User avatar
lag01
Member
 
Posts: 190
Joined: Sun Mar 16, 2014 03:41
GitHub: AndrejIT
IRC: lag01
In-game: lag

Re: map.sqlite backup with database locking

by lag01 » Thu Jul 23, 2015 17:31

bphenix wrote:..
And last but not least, if I organise an automated weekly backup with minetest server shutdown, I would appreciate to inform the users by a "shout" message, but I didn't found yet a way to do so from a bash script or a cron job. Any idea ?
...

Idea is simple: with script write to file, where minetest mod can read+write. From minetest mod, regularly check for that file and if it exists - delete it and do shutdown from inside minetest.
 

mystik
Member
 
Posts: 56
Joined: Tue Oct 28, 2014 22:16
GitHub: jasonjayr
In-game: mystik

Re: map.sqlite backup with database locking

by mystik » Thu Jul 23, 2015 22:00

A Disk snapshot is probably the best way to consistently capture the database + player files togeather.

If you're running on windows -- you *MAY* be able to take advantage of the "Volume Shadow Copy" system (aka VSS) (https://msdn.microsoft.com/en-us/librar ... 49(v=VS.85).aspx).

This snapshots your hard drive @ the filesystem level, and is how Windows is able to take consistent backups of important databases that are constantly in use (like, the registry, or an Active Directory Database). There are "VSS Providers" that databases can register on the system, so they can cleanup and store things consistently. For Something like minetest + sqlite the backup will be point-in-time consistent, but the snapshot might capture an in-flight transaction.

If you're running on linux, the same snapshotting can be achieved with LVM (or btrfs, or zfs). With LVM, you simply do:

lvcreate -n minetest-snapshot -L 1G -s /dev/vgname/lvname

Then you can mount /dev/vgname/minetest-snapshot and copy it off as slowly as you want. The actual snapshot only takes a fraction of a second, but the same caveat applies -- it's only 'freezing' what's on disk at that point in time, and may capture a half-written transaction to the sqlite journal.

Incidentally, this is a useful technique for running a mapper or anything else that , since it runs into the same issue with contention reading from sqlite.
 

User avatar
bphenix
New member
 
Posts: 4
Joined: Tue Jul 29, 2014 04:00
In-game: bphenix

Re: map.sqlite backup with database locking

by bphenix » Thu Jul 30, 2015 10:28

Thank you for your reply.
For sure, I'm running on Linux... What else ?
But my problem is that the system is existing and running (as often for Linux systems) on a weak, low-level hardware. So that I'm not sure I'll have the resources for the the LVM stuff.
More than that, I'm managing this Minetest server very remotely (I'm not even sure where it is physically) so I will have difficulties to re-install it on LVM.

But your answer is perfectly pertinent. Thanks again for it.
Moderated extremist.
 

User avatar
CraigyDavi
Member
 
Posts: 565
Joined: Sat Aug 10, 2013 13:08
GitHub: davisonio
IRC: davisonio or CraigyDavi
In-game: davisonio or CraigyDavi

Re: map.sqlite backup with database locking

by CraigyDavi » Mon Aug 10, 2015 11:38

This is a nice idea, I've always wanted to do this.

I once had a problem where some player files got cleared while modifying them while the server was since online, this might have been fixed though by now.

Your results are promising with a 565MB world, I will test this later and see how well it works with my 7.5GB world.
 


Return to Minetest Servers

Who is online

Users browsing this forum: No registered users and 4 guests

cron