Server don't work but still running

[amadin]

Anybody can't connect after ~ 5 hours of normal work, but server and redis is still running in processes list and server disappears from the list of servers (servers.minetest.net). If i restart minetestserver - it work again ~ 5 hours and the situation is repeated.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

5: Bind failed: Адрес уже используется
2015-08-09 06:18:04: ERROR[main]: UNRECOVERABLE error occurred. Stopping server. Please fix the following error:
2015-08-09 06:18:04: ERROR[main]: Failed to bind socket (port already in use?)

In thread 7f02a9f09780:
/root/minetest/src/server.cpp:511: void Server::step(float): A fatal error occurred: Failed to bind socket (port already in use?)
Debug stacks:
DEBUG STACK FOR THREAD 7f029affb700:
#0  virtual void* CurlFetchThread::Thread()
DEBUG STACK FOR THREAD 7f029b9fc700:
#0  virtual void* ServerThread::Thread()
#1  void Server::Receive()
(Leftover data: #2  void Server::SendBlocks(float))
(Leftover data: #3  void ItemStack::serialize(std::ostream&) const)
DEBUG STACK FOR THREAD 7f02a9f09780:
#0  int main(int, char**)
#1  Dedicated server branch
#2  void dedicated_server_loop(Server&, bool&)
#3  void Server::step(float)
2015-08-09 07:34:12: ERROR[ServerThread]: suspiciously large amount of objects detected: 50 in (-61,1,-16); removing all of them.
fields1:{
   title = "",
   text = "",
   save = "Save",
   quit = "true"
}
data1:nil
fields:{
   title = "",
   text = "",
   save = "Save",
   quit = "true"
}
data:{
   owner = "Diana",
   title = "",
   text = ""
}
fields1:{
   title = "Молодоженам.",
   text = "Дорогие мои молодожены.Вам подарлк.Это ваша дача.Будьте счастливы!!",
   save = "Save",
   quit = "true"
}
data1:nil
fields1:{
   title = "Молодоженам.",
   text = "Дорогие мои молодожены.Вам подарлк.Это ваша дача.Будьте счастливы!!",
   save = "Save",
   quit = "true"
}
data1:nil
fields:{
   title = "Молодоженам.",
   text = "Дорогие мои молодожены.Вам подарлк.Это ваша дача.Будьте счастливы!!",
   save = "Save",
   quit = "true"
}
data:{
   owner = "ogyrechiki",
   title = "Молодоженам.",
   text = "Дорогие мои молодожены.Вам подарлк.Это ваша дача.Будьте счастливы!!"
}
fields1:{
   title = "",
   text = "",
   save = "Save",
   quit = "true"
}
data1:nil
2015-08-09 10:32:50: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (-253,5,-118) (block (-16,0,-8))
fields1:{
   title = "Поздравляем",
   text = "В сей знаменательный день поздравляем молодоженов со свадьбой!!!",
   save = "Save",
   quit = "true"
}
data1:nil
fields:{
   title = "Поздравляем",
   text = "В сей знаменательный день поздравляем молодоженов со свадьбой!!!",
   save = "Save",
   quit = "true"
}
data:{
   owner = "amadin",
   title = "Поздравляем",
   text = "В сей знаменательный день поздравляем молодоженов со свадьбой!!!"
}
10:40:46: WARNING: Assignment to undeclared global "text" inside a function at ...t/bin/../games/minetest_game/mods/playereffects/init.lua:488.
2015-08-09 10:41:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-294) (block (3,2,-19))
2015-08-09 10:41:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-293) (block (3,2,-19))
2015-08-09 10:41:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-292) (block (3,2,-19))
2015-08-09 10:41:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-291) (block (3,2,-19))
2015-08-09 10:41:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-290) (block (3,2,-19))
2015-08-09 10:41:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-289) (block (3,2,-19))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-293) (block (3,2,-19))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-292) (block (3,2,-19))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-291) (block (3,2,-19))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-290) (block (3,2,-19))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (48,37,-289) (block (3,2,-19))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_willow" at (48,37,-288) (block (3,2,-18))
2015-08-09 10:41:44: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_willow" at (48,37,-287) (block (3,2,-18))
2015-08-09 10:41:58: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:willow_twig" at (52,39,-288) (block (3,2,-18))
2015-08-09 10:41:58: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (53,37,-286) (block (3,2,-18))
2015-08-09 10:41:58: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (54,37,-286) (block (3,2,-18))
2015-08-09 10:41:58: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (55,37,-286) (block (3,2,-18))
2015-08-09 10:41:58: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:fence_acacia" at (56,37,-286) (block (3,2,-18))
fields1:{
   title = "Поздравляем",
   text = "В сей знаменательный день поздравляем молодоженов со свадьбой!!!",
   save = "Save",
   quit = "true"
}
data1:nil
fields:{
   title = "Поздравляем",
   text = "В сей знаменательный день поздравляем молодоженов со свадьбой!!!",
   save = "Save",
   quit = "true"
}
data:{
   owner = "kuku_ept",
   title = "Поздравляем",
   text = "В сей знаменательный день поздравляем молодоженов со свадьбой!!!"
}
2015-08-09 10:49:08: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:mushroom_trunk" at (49,42,-291) (block (3,2,-19))
2015-08-09 10:49:08: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:mushroom_trunk" at (49,43,-291) (block (3,2,-19))
2015-08-09 10:49:08: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:willow_wood" at (50,43,-291) (block (3,2,-19))
2015-08-09 10:49:08: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:willow_wood" at (51,44,-291) (block (3,2,-19))
2015-08-09 10:49:08: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:willow_twig" at (52,39,-288) (block (3,2,-18))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (52,47,-293) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (53,47,-293) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (51,47,-292) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (52,47,-292) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (53,47,-292) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (54,47,-292) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:mushroom_trunk" at (51,45,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:mushroom_trunk" at (52,45,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (50,46,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (51,46,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (52,46,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (53,46,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (51,47,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (52,47,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (53,47,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (54,47,-291) (block (3,2,-19))
2015-08-09 10:53:20: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:acacia_wood" at (55,47,-291) (block (3,2,-19))
2015-08-09 11:17:24: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:mushroom_trunk" at (-697,14,-106) (block (-44,0,-7))
2015-08-09 11:17:24: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:redwood_wood" at (-697,15,-105) (block (-44,0,-7))
2015-08-09 11:18:22: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "stairs:stair_palm_wood" at (-689,4,-66) (block (-44,0,-5))
2015-08-09 11:18:22: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "default:torch" at (-689,6,-66) (block (-44,0,-5))
2015-08-09 11:18:22: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "stairs:stair_palm_wood" at (-687,4,-65) (block (-43,0,-5))
2015-08-09 11:19:37: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:banana_wood" at (-797,1,-99) (block (-50,0,-7))
2015-08-09 11:19:37: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "ethereal:banana_wood" at (-796,1,-99) (block (-50,0,-7))
2015-08-09 11:19:38: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "default:ladder" at (-799,2,-98) (block (-50,0,-7))
2015-08-09 11:19:38: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "default:ladder" at (-799,3,-98) (block (-50,0,-7))
2015-08-09 11:19:38: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "default:ladder" at (-799,4,-98) (block (-50,0,-7))
2015-08-09 11:29:58: ERROR[ServerThread]: Map::setNode(): Not allowing to place CONTENT_IGNORE while trying to replace "default:torch" at (-936,-128,-63) (block (-59,-8,-4))
2015-08-09 11:46:14: ERROR[ServerThread]: Item "memorandum:letter_written" not defined
2015-08-09 11:46:14: ERROR[ServerThread]: Item "memorandum:letter_written" not defined
2015-08-09 11:46:16: ERROR[ServerThread]: Item "memorandum:letter_written" not defined


"Адрес уже используется" - translated as "Adress already in use" but my autorestart script don't started (it must write in log about this) and server still runing (i use htop) but don't work for players and not allow connect. Before update server and migration to redis from leveldb server always shutdown on errors itself. But now server do not shutdown itself on error and freezes 2-3 times a day and last time without errors.
0.4.12-dev last update about 2-4 days ago from github, debian jessy x86_64, redis 3.0.3 (no errors in log).
*added 10.08. I think it is redis because i install old minetestserver (0.4.12-dev june from github) and still see this errors but server work fine.

My autorestart script:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

#!/bin/bash
RESTART="/etc/init.d/minetest start"
STOP="/etc/init.d/minetest stop"
SERVICE="minetestserver"
WORLD="world"
USERNAME="user"
MTPATH="/home/$USERNAME/minetest"
MAPPATH="/home/$USERNAME/minetest/worlds/world"
CPU_COUNT=2
INVOCATION="/home/$USERNAME/minetest/bin/$SERVICE"
OPTIONS="--worldname $WORLD"
MAPPER="$MTPATH/util/minetestmapper.py"
LOGPATH="/home/user/minetest"
LOGFILE="$LOGPATH/minetest.log"
BACKUPPATH="/home/$USERNAME/backup"
PIDFILE=/var/run/$SERVICE.pid

# Путь к pgrep команде
PGREP="/usr/bin/pgrep"

# httpd имя демона,
# RHEL/CentOS/Fedora - httpd
# Under Debian 4.x - apache2
HTTPD="minetestserver"

# поиск httpd pid
$PGREP ${HTTPD}

if [ $? -ne 0 ] # if not running
then
 # перезапуск
echo "Restart" $(date +"%y-%m-%d %T") >> /home/user/minetest/minetest-crash.log
rm $PIDFILE
sh /etc/init.d/minetest start >> /home/user/minetest/minetest-crash.log
sleep 5

fi
     

[Clouds]

I have just started experienceing this earlier today as well thus the server crashes

[amadin]

Today all blocks in the world are mixed together, and spawn came under ground in the stones. So i use backup now. I think it is redis because i install old minetestserver (0.4.12-dev june from github) and still see this errors but server work fine.

[benrob0329]

If the other Minetest session is still running, run "killall minetest"

That will kill Minetest and you can rester it.

[est31]

amadin, can you tell me the current git commit, and the git commit you used before, is that possible?
You can show a git history with "git reflog".

[est31]

amadin, which error do you get on 0.4.12-dev june builds too? the spawn error, or the hanging error?

[Clouds]

est31: They one Im using on my server was compiled on July 31if that helps any at all

[amadin]

amadin, can you tell me the current git commit, and the git commit you used before, is that possible?
You can show a git history with "git reflog".

I use 2 versions: 1-10 june and last 2-4 days, i do not remember exactly because in readme.txt no data or number of git commit. Now i fix one warning in redis log, so i must wait 24 hours to see the difference.

[amadin]

amadin, which error do you get on 0.4.12-dev june builds too? the spawn error, or the hanging error?

Only this strings, server not crashed (shutdown), only freez:
2015-08-10 00:31:03: ERROR[ServerThread]: ServerEnv: Trying to store id=12548 statically but block (-8,2,-11) already contains 49 objects. Forcing delete.

or

2015-08-09 00:30:47: ERROR[EmergeThread1]: WARNING: loadBlock: loading block (-17,-2,-8) failed: LOADING Redis is loading the dataset in memory

In redis log after start was this warning:
Redis can't set maximum open files to 10032 because of OS error: Operation not permitted.

[amadin]

On server version on june from github hang (freezing) after 3-5 hours too but i don't see any errors (server is running but not response), hangs begin after i installed redis. I want to curse because migrate back not work to leveldb or mysql.

[ExeterDad]

@amadin
I seem to remember getting errors similar to yours when using redis. I think I resolved it by adjusting "maxmemory" in my redis.conf and restarting the redis server.

If I remember correctly... my database (all Minetest) had grown to about the same size as maxmemory so needed parts of the map were not loaded into memory when needed by Minetest.

Hope that is helpful :)

[amadin]

Thanks but size of redis database is 800 Mb and i have 4gb RAM, minetest+redis i saw used maximum 2 gb. Fucking redis - i tryed different versions of redis, i tryed different versions of server but always after 5 hours server hangs (freezes) with no errors. On redis load:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

Server started, Redis version 2.8.17
[19293] 12 Aug 00:40:53.689 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

But i can't set sysctl vm.overcommit_memory because hoster protect this and i have 4gb ram.
If in redis.conf i set timeout 1800 (0 is default) then after 5 hour it freezes with errors:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2015-08-11 18:47:50: ERROR[EmergeThread0]: ERROR: An unhandled exception occurred: Redis command 'HGET %s %s' failed: Server closed the connection
2015-08-11 18:47:50: ERROR[ServerThread]: ERROR: An unhandled exception occurred: Redis command 'MULTI' failed: Server closed the connection

If in redis.conf i set tcp-keepalive 60 (0 is default) it is not help. And migrate from redis to mysql or leveldb not supported for my public server with 10-20 players online. I have only one question - who added this redis support in minetest?????????????

[amadin]

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31583 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31587 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31589 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31594 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31597 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31603 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31626 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: ServerEnv: Trying to store id=31627 statically but block (-56,-74,-22) already contains 49 objects. Forcing delete.
2015-08-12 06:52:03: ERROR[ServerThread]: Server::ProcessData(): Canceling: No player for peer_id=737 disconnecting peer!
2015-08-12 06:52:03: ERROR[ServerThread]: Server::ProcessData(): Canceling: No player for peer_id=736 disconnecting peer!
2015-08-12 06:52:03: ERROR[ServerThread]: Server::ProcessData(): Canceling: No player for peer_id=739 disconnecting peer!
2015-08-12 06:52:03: ERROR[ServerThread]: Server::ProcessData(): Canceling: No player for peer_id=752 disconnecting peer!
2015-08-12 06:52:04: ERROR[ServerThread]: Server::ProcessData(): Canceling: No player for peer_id=740 disconnecting peer!

Once after ~6 minutes of this errors server was freezed.
Today uptime was 11 hours (when i restart it) yohuuuu!!!! Next time i will not restart it. If it will work without freezing i put here my redis config.

[TenPlus1]

Each map block can contain only 48 entities which include dropped items, mobs, signs, itemframes and once it hits that limit anything over will show an error and be deleted like above... It's an in-game cleanup that keeps servers from grinding to a halt... You CAN increase the number to say 100 but it still means saving more objects in map...

[amadin]

deleted