Improving server performance / Database backends

[bark]

I've been running my server, Barkhouse, for almost a year now, and haven't experienced any particular issues with lag during this time. In the last month, however, I have been starting to get lag spikes about every 5-10 seconds (mainly when players are moving into unloaded chunks), lasting a couple of seconds to maybe 20 seconds every time. This started happening around the time new regions of the map was explored recently, as far as 9000 nodes from spawn.

Server specs
Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
6 GB RAM DDR3.
OS: Trisquel 7 (ubuntu derivative)

A priority should probably be to upgrade the RAM

But I have noticed that whenever the lag spikes occur, CPU usage peaks to 100% on one of the server cores. So maybe a RAM increase won't even fix this problem on its own?

edit: I have adjusted my minetest.conf according to this thread, that significantly reduced the lag spikes.

Database backends
Switching the database backend of minetest can apparently give the server a huge performance increase. I found the Database backends page in the minetest wiki useful, but also a bit scarce on information.

The default database backend for minetest is SQLite3, which is "Acceptable" in Speed and Reliability. The two other ones, LevelDB and Redis are either "Good" or "Very Good" (see table on wiki page).

So which one to choose? LevelDB seems like the safest way to go, based on

LevelDB is mostly used on servers because of its improved reliability and the ability to use more than 4 GB of space. You need to install it additionally.

But then a search for "leveldb minetest" returns mainly this post from LazyJ's blog, titled "LevelDB in Minetest is Just Not Worth it" which basically says that it's a terrible idea to use it.

In addition, I can't seem to find any documentation on how to install and activate LevelDB for minetest. Any further help and/or pointers on this would be greatly appreciated.

See also: Can SQLite3-databases NOT grow bigger than 4 gb? What would happen if they did?

+ Spoiler

Reply from sfan5 in LevelDB Test Server thread.

IIRC SQLite files larger than 4 GB don't perform very stable on some systems
Anyway, LevelDB is still faster and is more stable in general.

Then there is Redis, which supposedly is even faster, but also is more volatile, because the database is stored in memory only, and written to harddrive occasionally (ccould someone please elaborate on this?)

In addition, a server with Redis database backend can run MTSatellite for a live webmap which is a huge plus in my opinion.

Redis also has more install instructions than the latter, but not much. I did find this thread, Database Backend Questions, extremely useful in this regard, especially the thorough walkthrough on the install reply by joshburt;

+ Spoiler

In order to get redis support working (on linux - which is does). I had to :

1. Setup and properly configure a redis server. Go do that first or find a hosted one you can use.

2. Build the c library used to link to during minetest compiling and runtime:
a. git clone and build this: https://github.com/redis/hiredis

3. Next I had to build Minetest with redis support enabled:
cmake . -DRUN_IN_PLACE=1 -DBUILD_CLIENT=0 -DUSE_REDIS=1 -DREDIS_LIBRARY=/home/josh/hiredis/libhiredis.so -DREDIS_INCLUDE_DIR=/home/josh/hiredis/

These options are the special sauce:
-DUSE_REDIS=1 -DREDIS_LIBRARY=/home/josh/hiredis/libhiredis.so -DREDIS_INCLUDE_DIR=/home/josh/hiredis/
(Where the REDIS library and directory are correctly updated for your environment).

4. Make the hiredis library available during runtime.
Afterwards I copied the libhiredis.so library to the base of the minetest directory and created a sym link for the version specified via an 'ldd' on the resultant minetestserver binary:
libhiredis.so.0.11 -> libhiredis.so

5. Configure minetestserver to use the redis server
Update the world data storage by updating the world.mt as follows:
backend = redis
redis_address = 192.16.0.38
redis_port = 6379
redis_hash = minetest_dev

(The port and hash are optional parameters). The backend and server portal are REQUIRED.

I then launch minetestserver normally (from the root minetest directory: ./bin/minetestserver). If you missed anything it core dumps on start-up. Otherwise it acts just like a normal server but block data is stored in redis. Pure awesome! This was the steps needed for redis support in Linux. Your milage will vary and depending on the platform you're building for you might need to perform additional steps.

Feedback on this is greatly appreciated. Which database should I choose? Are there other steps I can take in addition to this?

[burli]

there is now also a Postgresql backend. You should try this. But it is not listed in the wiki. Search in the forum

[TenPlus1]

To make a leveldb world create a new map then run it once only, then close minetest and edit the world.mt file inside of the world folder, change the following line:

backend = sqlite3

..to..

backend = leveldb

..and run the map, it will now be copied across to a level db database and used from then on.

[bark]

Thanks for the reply. I guess I made this seem more difficult in my head than it was.

So if I instead want to migrate an existing world I can run (with minetest not running)

minetestserver --migrate leveldb path/to/world?

I have discovered that LevelDB must be installed, and then I have to compile minetest using the DENABLE_LEVELDB=1 flag.

In Debian and Ubuntu, LevelDB is available as libleveldb-dev. In Fedora, it is leveldb-devel.

[bark]

Ok, so I installed libleveldb-dev, and managed to successfully compile minetest from source using the DENABLE_LEVELDB-flag with the help of Calinou's 1-line compile script (Thanks Calinou!)

When I'm trying to migrate my existing map to LevelDB, I get the following error:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

$ minetestserver --migrate leveldb worlds/testmap/
ERROR[main]: Migration to leveldb is not supported


Does this mean that existing sqlite databases simply cannot be converted to leveldb? Or did I do something wrong?

@burli: Thanks for the advise. I have checked out the thread on Postgresql, and got a bit scared from the initial sentence, as I don't have much (actually no) experience with Postgresql or any other database for that matter.

The PostgreSQL backend is not for everyone. It requires some experience with
PostgreSQL or a willingness to learn. This is not something for the casual user.

Still might try this eventually.