FOSS gamedev and creative worlds

by **ShortCircuit** » Fri Sep 19, 2014 16:30

Running into an unusual problem on my Minetest 0.4.10 server. For some reason, users will periodically time out for no apparent reason.

At first, I would have thought this was due to network connectivity, however, it is occurring both on clients connected to the same LAN as well as a localhost client. (The server is a dedicated server hosted on my PC)

What is odd is that the server will log that the client has timed out, but the client will continue to run, however, other players will freeze in place and blocks destroyed will reappear, but blocks placed will still appear to take. So, apparently, the client isn't getting the message that it has lost the connection. Also, the client will not cleanly shutdown after the timeout, attempting to shut it down results in the client locking up and having to be killed. So, it appears possible this may be a client problem not a server problem, but not sure. Upon killing the client and reconnecting, everything is fine, until the timeout occurs again.

I have not been able to definitely find a way to reproduce the problem every time, however, I have found that it seems to be triggered most often by rapid placement or creation of blocks - the most common time I see it is when crafting, and I rapidly craft a large number of blocks very quickly. While it doesn't eliminate the problem entirely, trying to click more slowly on crafting seems to greatly reduce the number of occurrences of the problem. So possibly caused by some kind of race condition?

I have a number of mods loaded on the server, but I am planning to run a test server with no mods and see if I can reproduce the problem to eliminate the mods as the source of the trouble. However, I had the same mods loaded with 0.4.9 and did not encounter this problem. I will post back here once I have performed additional testing to rule out a mod causing trouble.

No matter the source of the issue, I would urge the developers to look into a way for the client to gracefully shutdown on a connection timeout, with an indication of the error, and an option to try and reconnect or exit, so that if this problem does occur, it doesn't require killing the client to restart it.

by **rubenwardy** » Fri Sep 19, 2014 16:37

When you press f5 twice on a client it shows the amount of packets (parcels of data) that are arriving from the server.

I had requested some indication of time outs before.

by **Sokomine** » Fri Sep 19, 2014 17:08

Krock has added a formspec on his server that informs players of server shutdowns. Such an information message might be something for all servers. After having received the message, and the server has shut down, you can still run around in that part of the world that was loaded before. Which is sometimes an advantage.

Trying out latest git might also help. Timeouts seem to occour from time to time, but they're rather rare and server-specific.

by **Krock** » Fri Sep 19, 2014 19:22

Sokomine wrote:Krock has added a formspec on his server that informs players of server shutdowns.

Yes but this only helps as long the player can receive the formspec. A "normal" time-out will be invisible without checking a command or looking at F5.

by **ShortCircuit** » Mon Sep 22, 2014 17:15

Sokomine wrote:Trying out latest git might also help. Timeouts seem to occour from time to time, but they're rather rare and server-specific.

This timeout is definitely not rare, occurs every single time we play. If we play for 2 or 3 hours, typically will see 5 or 6 timeouts. Very frustrating to players.

I do think the issue is most likely client side, since the client will freeze when a shutdown is attempted after the timeout has occurred. Server continues to run normally, and the player can reconnect with no issues once their client is killed and restarted.

Does anyone know a way I can attempt to gather more information about what is happening when this issue occurs so that I can try and narrow down what might be causing it?

by **ShortCircuit** » Mon Sep 22, 2014 17:20

Krock wrote:
Sokomine wrote:Krock has added a formspec on his server that informs players of server shutdowns.

Yes but this only helps as long the player can receive the formspec. A "normal" time-out will be invisible without checking a command or looking at F5.

This issue isn't a server shutdown, the server is still running fine, but the client is timing out even though the connection is still fine. I have noticed that the client will stop receiving new data from the server about a minute or two before the server log shows the client as timing out.

by **ShortCircuit** » Sun Sep 28, 2014 16:35

OK, got more info on the problem finally. Started minetest from a terminal window so I could see output. Everything looks normal, until when the timeout occurs this is the output:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all: *** Error in `minetest': malloc(): memory corruption (fast): 0x00007fd124009b7f ***

So, issue is definitely occurring on the client side. Anyone have any ideas on what could be causing this and a fix/workaround?

by **rubenwardy** » Sun Sep 28, 2014 19:27

Try running using gdb

by **ShortCircuit** » Fri Oct 03, 2014 20:28

rubenwardy wrote:Try running using gdb

OK, running using gdb gives the following:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all: Starting program: /usr/bin/minetest [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Irrlicht log: Irrlicht Engine version 1.8.0 Irrlicht log: Linux 3.11.0-26-generic #44-Ubuntu SMP Mon Jul 7 13:18:24 UTC 2014 x86_64 Irrlicht log: Using renderer: OpenGL 4.4.0 Irrlicht log: GeForce GTX 750 Ti/PCIe/SSE2: NVIDIA Corporation Irrlicht log: OpenGL driver version is 1.2 or better. Irrlicht log: GLSL version: 4.4 [New Thread 0x7fffeae40700 (LWP 28503)] [Thread 0x7fffeae40700 (LWP 28503) exited] [New Thread 0x7fffeae40700 (LWP 28504)] [New Thread 0x7fffe663e700 (LWP 28505)] [New Thread 0x7fffe5e3d700 (LWP 28506)] [New Thread 0x7fffe563c700 (LWP 28507)] [New Thread 0x7fffe4e3b700 (LWP 28508)] [New Thread 0x7fffcffff700 (LWP 28509)] [Thread 0x7fffe663e700 (LWP 28505) exited] [Thread 0x7fffeae40700 (LWP 28504) exited] [Thread 0x7fffcffff700 (LWP 28509) exited] [Thread 0x7fffe4e3b700 (LWP 28508) exited] [Thread 0x7fffe563c700 (LWP 28507) exited] [Thread 0x7fffe5e3d700 (LWP 28506) exited] [New Thread 0x7fffcffff700 (LWP 28637)] [New Thread 0x7fffe4e3b700 (LWP 28638)] [New Thread 0x7fffe563c700 (LWP 28639)] [New Thread 0x7fffe5e3d700 (LWP 28640)] [New Thread 0x7fffc67ab700 (LWP 28642)] Font size: 8 15 *** Error in `/usr/bin/minetest': malloc(): memory corruption (fast): 0x00007fffd000b03f *** [Thread 0x7fffc67ab700 (LWP 28642) exited] [Thread 0x7fffe5e3d700 (LWP 28640) exited] 0x00007ffff63851f8 in pthread_join (threadid=140737041909504, thread_return=0x7ffffffda398) at pthread_join.c:92 92 pthread_join.c: No such file or directory.

Doesn't seem to have any more information than I had before, but maybe you can spot something I'm not.

BTW, I tried using the latest daily build as someone suggested, and I couldn't even get it to run, it would crash with a ton of errors every time I tried to start it.

by **sfan5** » Sat Oct 04, 2014 12:10

Run a debug build of minetest in valgrind.

by **ShortCircuit** » Sat Oct 04, 2014 20:09

I ran minetest with the --trace option, and got this info when it crashed:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all: 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c3e 0100 3237 c500 1106 0002... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c41 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c45 0100 3237 c500 1106 0002... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c4b 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=13, data= 4f45 7403 0001 0003 3c4e 0100 2927 f800 0119 40 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c4f 0100 323c 6900 1106 0000... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c50 0100 3237 b300 1106 0002... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c52 0100 323c 6900 1106 0000... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c55 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c56 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c58 0100 323c 6900 1106 0000... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c59 0100 323c 6900 1106 0000... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c5c 0100 3237 c500 1106 0000... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c5d 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c5f 0100 4700 0800 0003 e800... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c61 0100 3237 b300 1106 0000... 10 <- 127.0.0.1:7530, size=c, data= 4f45 7403 0001 0003 3c62 0002 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c65 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c66 0100 4700 0800 0003 e800... 10 <- 127.0.0.1:7530, size=13, data= 4f45 7403 0001 0003 3c69 0100 2928 ca00 0119 40 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c6a 0100 3237 b300 1106 0000... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c6b 0100 3237 b300 1106 0002... 10 <- 127.0.0.1:7530, size=22, data= 4f45 7403 0001 0003 3c6d 0100 3237 b300 1106 0002... 10 <- 127.0.0.1:7530, size=c, data= 4f45 7403 0001 0003 3c71 0002 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c72 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=89, data= 4f45 7403 0001 0003 3c6e 0100 4700 0800 0003 e8ff... 10 <- 127.0.0.1:7530, size=d6, data= 4f45 7403 0001 0101 0032 37b3 002f 0100 0295 2200... 16:04:23: INFO[main]: ClientMap: Unloaded 2 blocks from memory, 1224 blocks in memory. 16:04:24: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:24: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:27: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:27: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:28: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:29: INFO[main]: Client: avg_rtt=0.002 16:04:29: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:30: INFO[main]: OpenALSoundManager: Creating positional playing sound 16:04:31: INFO[main]: OpenALSoundManager: Creating positional playing sound Terminated

I notice the OpenALSoundManager messages didn't start showing up until the timeout occurred, could the problem be sound related?

by **Sol** » Sat Oct 04, 2014 21:35

I got memory corruption in server today.

by **CWz** » Sun Oct 05, 2014 11:37

could this be caused by outdated clients such as the illegal buildcraft/freeworld/zombiesurvival ports?

by **ShortCircuit** » Tue Oct 07, 2014 00:08

Sol wrote:I got memory corruption in server today.

The server side has been fine on mine through this, it's just the client that has been crashing, at least partially.

by **ShortCircuit** » Tue Oct 07, 2014 00:10

CWz wrote:could this be caused by outdated clients such as the illegal buildcraft/freeworld/zombiesurvival ports?

I do have a couple 4.9 clients connecting, but they are not crashing, and the 4.10 client that matches the server is crashing even when the other clients have not been connected - and the server isn't crashing either, just the client.

FOSS gamedev and creative worlds

Users timing out in 0.4.10

Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10 *** Error in `minetest': mall

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Re: Users timing out in 0.4.10

Who is online