Lua Threads for Computer Mod

[ninnghazad]

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

local str = fs.read(path):getdata()

nice, this will suffice.

...don't see myself ever using this. Every time I've ever used a file in lua, I've either read the whole thing...

i have used it a lot, and already have plans for it. although reading parts would be more important than writing parts.
it's not really needed, but without would make small random access to very large files rather slow and complicated. pathfinding data from drones for example grows fast and can easily grow above multiple hundred MB. not an every-day usage-case ofc.

ramdisks

ye, thought about caching stuff too. but it really is just frosting on the cake.

The device should use coroutines to slowly read the file in chunks, then piece the chunks together and send the whole thing at once. If you want a big file (or a big chunk of a file), it will work without lagging the server but will take a while. There should also be a maximum file size.

exactly what i mean.

How?

seems xpcall in minetest can somewhat handle OoMs - when xpcall runs out of memory it will not call the error-callback, but just return {false,"out of memory"}. far from a system that can actually limit/determine memory of a single (computer's) coroutine reliably, which seems to be a real pain to be done from inside lua.
so for now it should work if we change computer.lua's
local cr = coroutine.create(function() xpcall(bootstrap,error_handler) end)
to
local cr = coroutine.create(function() return xpcall(bootstrap,error_handler) end)
and check for the "out of memory" message in the crashed coroutine's return-value.
not a final solution, but could be a first step to stop players from crashing the server with memleaks.
sadly this would not enable per-computer-limits, but would ofc just trigger when minetest's global lua mem-limits get reached by a computer's coroutine.
meaning a computer could just fill 31.9GB of 32GB RAM and then wait for minetest itself to go OOM and crash without this measure doing anything.
i was thinking about using collectgarbage,__method and __gc_methods and so on to fiddle something up, maybe to calculate memory-size of userspace-table (ugh, would mean counting whole userspace basically everytime anything is done). but even if so we still need to make sure all of the queues/storages the player may access in any matter, are confined by max-limits. current example: make the event-queue(s) reject/pushover in case of too many queued events, so players cannot put a peripheral in a state where it will blow up minetest with a billion-events-queue or smth. same for console_histories (gets clipped to 4096 chars), and probably other stuff.


btw: with the proto, what happens to pending request when the requesting device has crashed before retrieving results? or was mined away or whatever.

[electrodude512]

i have used it a lot, and already have plans for it. although reading parts would be more important than writing parts.
it's not really needed, but without would make small random access to very large files rather slow and complicated. pathfinding data from drones for example grows fast and can easily grow above multiple hundred MB. not an every-day usage-case ofc.

I added file ranges to the protocol specification. When dronetest works, can I help with your pathfinding system? I was working on one in ComputerCraft before I switched and I had mapping and basic pathfinding working.

ye, thought about caching stuff too. but it really is just frosting on the cake.

It will be very significant, especially if you've mounted a remote device at /rom/bin or /bin. It's still pretty low priority, though. I'll add a max cache time field and something similar to "HTTP 304 Not Modified" to get_ack.

so for now it should work if we change computer.lua's
local cr = coroutine.create(function() xpcall(bootstrap,error_handler) end)
to
local cr = coroutine.create(function() return xpcall(bootstrap,error_handler) end)

I already did that. If you ever have a function like that, please, always put the return statement in. I had to do this in about 5 places before I could figure out why my event system wouldn't work.

i was thinking about using collectgarbage,__method and __gc_methods and so on to fiddle something up,
maybe to calculate memory-size of userspace-table (ugh, would mean counting whole userspace basically everytime anything is done). but even if so we still need to make sure all of the queues/storages the player may access in any matter, are confined by max-limits.

I'm not doing that. It would all be too slow and error-prone. If we must do something, I would go with a coroutine outside of the sandbox that gets each computers' environment, iterates through everything (using debug.getlocal to get locals), and makes sure it's not too big and then kills the computer if it is too big. It would go slowly, only doing a little bit of work at a time, so it doesn't make everything lag.

I don't think lua 5.1 supports __gc, but luajit might. I hope it does, for other reasons as well.

Most things that would eat all of memory in any short amount of time would get killed for not yielding often enough.

current example: make the event-queue(s) reject/pushover in case of too many queued events, so players cannot put a peripheral in a state where it will blow up minetest with a billion-events-queue or smth. same for console_histories (gets clipped to 4096 chars), and probably other stuff.

Console histories should be clipped. To prevent overflowing event queues, we should implement something like luacontroller_mesecons's overheating system

btw: with the proto, what happens to pending request when the requesting device has crashed before retrieving results? or was mined away or whatever.

The response is sent by the requestee, which then forgets about the request because it's done with it, and then the response doesn't go anywhere because the requester is gone. The requester is gone so it doesn't care, and the requestee has already responded so it doesn't care either. There's no second acknowledge to acknowledge acknowledges.

[electrodude512]

Here's a status update because I haven't committed anything for a while but I have lots of changes.

I have removed all APIs except for sys, term, drone, and peripheral. I'm going to make term, drone, and peripheral normal userspace libraries, but I'm not going to do that until everything else works. I moved any functionality from an API that I turned into a library into sys, for now. I have changed the calling convention for stuff in sys from sys:foo() to sys.foo() for security reasons (so you can't do sys.foo(fakesys) instead of sys:foo()), but sys:foo() still works (there's an if statement to fix it if you do it the old way). I'm moving rom reading into sys.readRom(path), and I'm going to remove internal storage from computers for the moment, but will re-add it after stuff mostly works. The main thing that's been taking so long is my filesystem mount manager. I've added a class system and many (probably too many) classes such as:
- device: wrapper for digilines that stores its address and has a function that the digiline daemon calls when it gets a packet for the device
- connection: same as protocol connection, except this one doesn't have to be formally opened, subclass of volume
- request: digilines request, in charge of sending itself and of handling the response (called by digiline daemon), returned by connection:get or connection:put
- volume: mountable thing, has higher level methods other than get and put such as volume:isDir
- ramdisk: subclass of volume
- rom: wrapper for sys.readRom, subclass of volume
- fs mount manager: unix mount thing, subclass of volume

By the way, if you want to work on anything but don't want to because of all of the changes I've made, it should be pretty easy to merge anything derived from commit 074c83581, so branch off of that (or merge that into master) if you want to do anything.

I'll push my changes as soon as the shell and lua work again.

[electrodude512]

Folders (combined ramdisks and fs mount managers) work. The rom device still doesn't work, so os.loadlib still uses the old sys.readFile. I pushed everything, but it's not ready to be merged yet. I've barely tested anything, but shell, mkdir, and ls work.

[octacian]

Is this dead? I really hope not. Looked like it was moving along quite well. I would like to see this in Minetest too, so am really interesting in seeing progress. If there will not be any more progress, can someone give me links to the Github repository(s) where this mod was developed? It would be really useful as a reference for me when I begin work on better computers.

[electrodude512]

Is this dead? I really hope not. Looked like it was moving along quite well. I would like to see this in Minetest too, so am really interesting in seeing progress. If there will not be any more progress, can someone give me links to the Github repository(s) where this mod was developed? It would be really useful as a reference for me when I begin work on better computers.

The code for Dronetest can be found at https://github.com/ninnghazad/dronetest. Ninnghazad seems to have done some more work on it since I lost interest in it.

I've made some more attempts on my own at a computer mod. I have a working Digilines hard drive block, floppy drive block, and floppy disk item. You can change the floppy disk item in the floppy drive, and the contents will change. Files stored on both types of disks are stored on your computer's hard drive - each in-game disk gets a directory in minetest/worlds/${worldname}/disk/ . I haven't released it yet because, although the floppy disk item has a texture, neither the hard disk block nor the floppy drive block have textures yet, because I'm a terrible artist. If anyone wants to make suitable textures that I approve of, I'll use them and publish the two blocks as the first part of a ComputerTest mod. These Digilines storage blocks are perfectly usable by standard Luacontrollers, in case a proper computer mod never appears.

One of the main problems with the Dronetest mod was the lack of a keyboard entry system. Ninnghazad made a pull request to Minetest to add this functionality, but it was never merged. I think I'll just add a formspec block that allows a computer to present any player who clicks on the block with a formspec user interface. It won't have ease of use and interfacing of the command line, but it will at least be something. Maybe I could also try using the F10 dropdown chat console somehow.

I also have a modified Luacontroller with coroutines and semi-persistent storage(i.e. until the block gets unloaded). I got it to load a file off of an external floppy disk in a floppy drive block over Digilines and then load the file into a function and run it in a coroutine, which then gets passed every event the modified luacontroller receives. That boot file could load a complete coroutine-based multitasking operating system, similar to that found in ComputerCraft for MC. I haven't released it yet because I'm not convinced it's safe yet, it requires the hard disk and floppy disk blocks that I haven't release yet, and because I'm not sure if it's implemented how it should be implemented.

I want to experiment with a Lua Lanes-based system that runs computers outside of the main Minetest thread. It will allow taking advantage of LuaJIT, and will surmount the problem of infinite loops (which is currently solved by luacontrollers using debug.sethook and non-JIT'd lua) by giving each player his own thread. All communications with the computer threads - which will all be through digilines - would occur through Lua Lanes' lanes. If a player has a computer run an infinite loop, only his own computers will suffer. I'm pretty sure there's a way to kill a Lua Lanes thread, so if a player's thread stops responding (i.e. goes into an infinite loop), the main Minetest thread will eventually kill that player's thread (i.e. all of that player's computers!) and notify the player via chat or digiline message (and maybe punish the player if it happens too much).

This is all assuming I ever find the time to start doing Minetest modding again. I hope I do eventually, but it won't be soon.