FOSS gamedev and creative worlds

by **blert2112** » Sat Jan 02, 2016 03:30

This is driving me flippin' crazy! I have been hammering on this all day and have gotten nowhere. I continually get Lua Out-Of-Memory crashes when using some of the mods I like the most.
Using a two day old build (or sfan5's latest):
running only Minetest_Game, mg_villages and handle_schematics results in an almost immediate crash,
running only Minetest_game, biome_lib, plantlife and moretrees results in a crash after a couple minutes.
So, I figured I would try the Valleys C mapgen to see if I could get around using these big mods. Worked fine until I installed the valleys_c support mod... OOM crash.
I tried recompiling many times with two different LuaJit libraries (the one you get from buildbot and the one installed in my MinGW setup). No joy.
I recompiled without Luajit and the OOM crashes go away but that opens up to other mods failing because, being written for LuaJit (which has a few enhancements found in Lua5.2), they crash when they come up against Lua5.1.
Any help would be greatly appreciated.
PC specs:
Core I5-5200U 2.2GHz
8GB RAM
Win10 x64
nothing of consequence running that would eat resources

edit: new info
Using sfan5's builds, I went back a few months to see if some recent change is causing my troubles. No luck. For grins I decided to try a 32bit build. Guess what? No crash! Ok, that was Krock's 32 bit MSVC build... What if it has something to do with MinGW builds not getting along with Win10? So I tried sfan5's latest 32bit build... No crash!
Something is going on here that is beyond my knowledge. For now I will have to go with a 32bit build and maybe do a bit of research.

edit2: more new info
Here's the thing... If I tell the video driver to use the NVidia 940M w/4GB there is no longer an OOM crash. So...
On my rig:
- 64bit build with the integrated Intel video = OOM crash
- 32bit build with the integrated Intel video = no crash
- 64 or 32bit build using the NVidia = no crash
It seems that memory sharing with the integrated Intel video is getting in Lua's way even though it can only grab at most 2GB out of 8 and there is plenty left over just sitting idle. I assume this won't be a problem once I move the game onto my server as graphics won't be an issue for it. But, I guess I am going to have to do some testing on my kid's laptops as they don't have the option to use dedicated graphics.

edit3: yet again
Spoke too soon. :(
OOM crash again with 64bit build of Valleys_c mapgen + valleys_c lua supplement using dedicated graphics. Beginning to wonder if this is worth the effort.

by **GeneticAlgorithm** » Sat Jan 09, 2016 21:53

I have been having this problem too. I'll put more info on it soon. I've had to stay with 4.12.

I have used:
minetest-0.4.13-win64-msvc
(I think this is the main download)

Specs
Intel Pentium 2020M
4GB RAM
Integrated Intel graphics
Win 8.1 x64

EDIT: I downloaded the latest version from sfan5, build bd40ee2-win64. So far, the error hasn't occurred.

EDIT2: Occurs with new world made with recent release of Dreambuilder. '80 mb'

EDIT3: Tried 'Mod Profiling'. Appeared to speed things up for a bit, but got this

Lua: OOM error from mod 'technic_worldgen' in callback environment_OnGenerated(
): not enough memory
Current Lua memory usage: 52 MB

by **duane** » Sat Jan 09, 2016 23:48

Is there really that much difference between the 32 and 64 bit builds? I never noticed much in the experimentation I did back when everyone was switching over; I just went to 64 bit by default.

by **GeneticAlgorithm** » Sat Jan 09, 2016 23:54

Not sure if it does. I just received it again with the 64bit. Haven't tested the 32bit yet, will update. However, I think the error will appear again because Dreambuilder is a large subgame with lots of lua code and objects. I googled Lua memory adjustment. Found info adjusting Lua memory for Elder Scrolls Online. I think we may need that feature for Minetest.

I think the problem is literally a memory restriction for lua. So this error may be common for heavy subgames and those with lots of big mods (like technic)

by **paramat** » Sun Jan 10, 2016 00:18

It's the lower memory restriction for LuaJIT. Mods shouldn't be written to only work with LuaJIT.
Complex lua mapgens (like some of mine and valleys mapgen) use up a lot of memory when using noise, especially 3D noise, enough to hit the LuaJIT ~2GB limit.
For now just don't use LuaJIT.

by **GeneticAlgorithm** » Sun Jan 10, 2016 01:55

Thanks!
PS: I'm a big fan of you're mods!

by **blert2112** » Mon Jan 11, 2016 01:06

I found a work-around. Instead of playing a single player game, I fire up a server and connect to it by running the client on the same machine, so two processes instead of one. It's a pain when testing mods but it's better than constantly crashing. Probably not what you want to do if you don't have at least four cores though.

by **Don** » Mon Jan 11, 2016 02:06

GeneticAlgorithm wrote:Thanks!
PS: I'm a big fan of you're mods!

Who isn't a fan? Paramat has done some amazing work.

by **xeranas** » Thu May 19, 2016 09:16

paramat wrote:Mods shouldn't be written to only work with LuaJIT

How does this looks like? How I can tell that mod only works with LuaJIT?

paramat wrote:For now just don't use LuaJIT.

Ok, how?
I downloaded windows build from minetest dowload page (0.4.14 64bit) and I getting OOM when hitting 12MB?

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all: 2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory 2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB

Do I miss put magic parameter somewhere?

My Mods and Stuff · by **azekill_DIABLO** » Thu May 19, 2016 15:41

12 MB??! that a lot man!

i have a world with 40 mods with no bugs (and very heavy mods like mymonths)

by **blert2112** » Fri May 20, 2016 00:04

xeranas wrote:
paramat wrote:Mods shouldn't be written to only work with LuaJIT

How does this looks like? How I can tell that mod only works with LuaJIT?

Unless you comb through the code you will not be able to tell. Most (if not all) mods seems to be made forgetting that someone may not be using LuaJIT. LuaJIT is more like Lua5.2 than Lua5.1: for instance, math.random() can break under Lua5.1 because it will wrap back around into negative numbers if the numbers fed to it are too large (Lua5.2 and JIT do not have this problem).

xeranas wrote:
paramat wrote:For now just don't use LuaJIT.

Ok, how?
I downloaded windows build from minetest dowload page (0.4.14 64bit) and I getting OOM when hitting 12MB?

You will need to compile MineTest without LuaJIT. I thought there might be a way to turn it on and off ( jit.on()/jit.off() ) but it doesn't work.

xeranas wrote:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory 2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB

Do I miss put magic parameter somewhere?

This means the memory usage was 12MB BEFORE something caused the OOM error. It does not reflect an actual accurate metric of memory usage.

What are you doing when the crash happens? Just playing a normal game or cruising around in fast mode? There are a few workarounds you can try: my Stop Lua OOM Crashes mod, remove some heavy mods, (my personal goto) run the server and client as separate processes.

One big thing that mod creators need to start doing... stop using globals for everything (globals are NOT tended to by the GC and are never cleared from memory even if they will never be used again). All those leftover somemod.register_something() functions and giant global tables that are no longer needed after the game loads and should be cleared. On my personal server I have totally re-written every mod in Minetest_Game (along with HomeDecor, Valleys_c helper, and a bunch of others) and have done away with almost every global function, table and var unless it is totally necessary.

by **anuser** » Tue May 24, 2016 18:51

I have the same problem. With 64 version of newest engine i have out of memory error just a few sec after begining. This make this game not playable. And it happens only when I have worldgen of technic mode turned on. Also the game load more slowly then. It is not related to how many other mods I run. But with the 32 version of the same engine this problem not exist: I can run all mods I have with full technic and there is no problem. But this not mean that 32 version is OK.

It can be also not playable an crash just after the world is loaded at begining. But this time this happened only when I have Sphax TestBDcraft textures turned on. Without them evrything seem to be all right. Of course I know that this is not related to technic or textures packages. With older engine release there was no problem with that modes/textures at all. In this case there is no message about out of memory, this look like some other error.

Details of 32 version problem

+ "Show details..."

I guess that this out of memory crashes were not present in earlier versions and it shows up only with recent versions. If I am right, maybe some people smarter than me could review the last changes in code to find the problem. It must be there, with recent changes. There is always possibility to compile engine without such single recent-code-update to see if the problem is still present. Maybe I do not understand how it works and I am wrong here, posting some stupid advices. But the problem exist and is really serious.

I am not coder but I think this is critical error that make the game not playable and reduce it to work-in-progress-without-public-release version. Sorry, if anyone will ask me now about Minetest, I can only tell him - in this moment - do not even try this, just forget about it. I mean this problem with out of memory crashes should make red alert here and be the main topic right now.

Edit: I am talking about the singleplayer mode and sorry for my language.

by **Fixerol** » Wed May 25, 2016 12:59

Ha! I've heared this "no OOM crash on 32bit builds" from maikerumine, interesting.

by **Fixerol** » Wed May 25, 2016 13:05

2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory
2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB

I have a custom subgame (for testing) that gives this kind of error, but for 'darkage' mod, interesting. Here it is:

2016-05-25 16:11:28: ERROR[Main]: ServerError: Lua: OOM error from mod 'darkage' in callback environment_OnGenerated(): not enough memory
2016-05-25 16:11:28: ERROR[Main]: Current Lua memory usage: 22 MB

by **blert2112** » Wed May 25, 2016 23:34

@Fixerol: Which DarkAge are you using? One that is using the VoxelManipulator I would have to guess. The problem is not from the DarkAge mod itself. BTW, try my version of DarkAge, it doesn't use the VM, it uses the ore_gen.

I get these LuaJIT OOM crashes when using the VoxelManipulator. Especially if more than one mod is using it. I can't say if there is a memory leak in the code (because I am not good enough at C++ to tell what's going on), in fact, there probably isn't. The issue looks to be with LuaJIT. The Lua garbage collector is not running all the time. If I recall correctly, it runs about every two seconds by default. A lot can happen in two seconds and the VoxelManipulator object and associated tables are quite large and can take up a good chunk of LuaJIT's memory. If more objects and associated tables are created before some of the others are garbage collected then the memory usage can add up quick. Add to this the fact that the GarbageCollector is quite inefficient when it has to deal with a lot of objects. I have successfully moved the DATA and AREAINDEX tables outside of LuaJIT's memory limit but it doesn't seem to help much because the functions to create these tables consume the same amount of memory as the table itself does. Yes, that memory can be set to NIL but it doesn't make a difference because nothing is actually freed until the garbage collector runs again. "Run the garbage collector more often" you may say... Well, you can but doing so can actually slow everything down and cause other issues and trying to run the GC while it is currently running can cause real bad things to happen. LuaJIT3 is supposed to have a brand new GarbageCollector with no memory limit but as far as I have seen it is not even on the horizon and there is no beta or alpha of it either. The best things that can be done to help alleviate this problem is to: 1) stop using globals for everything, 2) if you have to use a global clear it when it is no longer needed (example: mymod:register_something() ... clear this in a minetest.after(0) ), 3) reuse local objects (create the object outside of the loop instead of declaring 'local somevar' every iteration). Hopefully some of this will help someone but there is no way around the LuaJIT memory limit until version3 comes out... unless a table is large and needs to be kept in memory all the time, then it can be moved outside the memory limit quite easily.

by **Fixerol** » Wed May 25, 2016 23:39

It uses VoxelManipulator.

FOSS gamedev and creative worlds

Lua OOM crashes

Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Re: Lua OOM crashes

Who is online