Lua Threads for Computer Mod

[ninnghazad]

in case you are wondering why nothing is happening, i'm waiting for https://github.com/minetest/minetest/pull/1737.
and to be true, i don't think anybody actually feels responsible for pull requests from outside, on the contrary - there are >100 pull requests and most of these will be wasted time.
and i'm not bored enough to care, and i don't care enough to beg, so until stuff moves forward imma do something else.

[Evergreen]

in case you are wondering why nothing is happening, i'm waiting for https://github.com/minetest/minetest/pull/1737.
and to be true, i don't think anybody actually feels responsible for pull requests from outside, on the contrary - there are >100 pull requests and most of these will be wasted time.
and i'm not bored enough to care, and i don't care enough to beg, so until stuff moves forward imma do something else.

I'll see if I can annoy some of the devs about this once I get home. It seems like an extremely useful feature.

[Neuromancer]

ShadowNinja gave the reason on the pull request that this would make the engine too laggy. I commented on the thread that why not then make this an optional feature that could be turned on and off like shaders which is really laggy. I really hope this pull request gets accepted because I really think that this mod and the feature it needs (keypress) would open up really cool functionality to Minetest.

[electrodude512]

It didn't make Minecraft laggy (and I have a five year old computer), why would it make Minetest laggy?

I know barely anything about the C internals of Minetest, so I wouldn't be surprised if there is some good reason that this would cause significant lag.


In the meantime, after Nov 1 when I have free time again, I'm going to start reworking the internals of dronetest to be more secure, efficient, and readable, and might possibly even add some peripherals, regardless of what happens with this pull request.

Ninnghazad, in case you're wondering, here's a very incomplete list of plans I have, roughly in priority order:

• make a function that returns a new sandboxed computer environment, instead of copying a table
• sandbox stuff more properly - have two tables for each computer - one the computer can wreck as much as it wants, the other that it has no access to (except maybe through setter and getter functions)
• make sys not have security holes (you might have already fixed this)
• plug-and-play drivers - standard protocol for a computer to read a string from a peripheral, compile it as a function sandboxed inside another sandbox inside the computer's sandbox so it can't wreck the computer, and use whatever table that function returns after calling it as the method table for the peripheral. This allows anyone to add their own peripherals without having to put stuff in /rom/ or needing any real standards, and also makes protocols between computers trivial.
• I/O expander peripheral, should have bundled cable support if there are any mods that add bundled cables
• external floppy drive
• external hard drive - will only do if you will agree with me that all hard drives should be external and computers should have no internal non-volatile storage except channel ids - there's no internal terminal , why should there be internal anything? It should just be a CPU block.
• wireless modem peripheral

I made a TODO tag for github's issue tracker a while ago, so we can also abuse the issue tracker as a todo list. We should probably use it, but I don't have time to decide on this or add stuff to it now.

[ninnghazad]

laggy

i have commented about this on the pull request - it does not cause lag, it causes additional traffic. we don't need to discuss this here - if we want that feature, it has it's price, simple as that. PRs need 2 coredevs approving, we have like 1,5 - so getting another dev's approval is what we need on the matter.

the textures not unloading is a bigger issue performance-wise. i looked into this and it's not a bug, it's a missing feature - textures just do not get unloaded in minetest.
could add something for that, but i'll wait until the other PR is through.

@electrodude
good to see you're still around. and good to see you have plans.
you may have noticed that i'm rather picky when it comes to coding =), but most stuff i would note, you'll come along anyway.

about the external/internal drives: it is not that i am fixed on having internal hdds - i just do not want to over-complicate the mod for anti-features. to the user, the programs/tools that come with each system need to reside somewhere. of course they are not really on each computer. but to the user, the system will have a root-filesystem even without drives, like on a linux box, where the programs are.
maybe it squenches your thirst for modularity if we say that the root-filesystem of a computer is always read-only. we explain that to the user by saying the computer have a ROM-flash-bios. that way we still have it simple (easily accesible root-fs), but if the user wants to write own files and read them, they need a external hdd. which would then be practically every case where a user wants to run a custom program. so that external-hdds would be needed for practical use, but one would not have to bother about it when developing apis or ROM-programs.
thats good.

the pnp-drivers sound pretty much like how it currently is, only in a different wording. call peripheral and in the end get table with functions to use it.

about the sandboxing, that's also how it is now, with a different wording. accessible per-comp environment user can wreck, and global environment that is only accessible through functions (apis).
with a more ... controlled ... creation (your first point) of the wreckable-table, or userspace-environment, i think we will have a solid system. so we can be sure that there are only deep copies/new tables, no references in each userspace-env.

about the wireless-modem, well, the transceiver somewhat is a special wireless-modem for dronetest-peripherals. for a "real" digilines-wireless-modem i'd say thats digiline's job. and we just use it like other digiline-devices.
the transceiver however needs more love - it needs bulletproofing for cases like feedback-loops, multiple transceivers on the same net and and and.

and finally something new:
how about energy-consumption? ideas/wishes about that? i wanted to make computers use energy each coroutine.resume of their main routine. so intense tasks cost more energy (and laggy stuff with that also). computers would just not resume and wait until energy is there. so i don't want computers that crash everytime there's a powerfluke. they should just get slow, or halt. not sure about drones yet, maybe like some akku or something.
i didn't like CC's turtles burning coal, i want them to be fully electric - mass to power conversion needs to be done with other stuff.

[electrodude512]

@ninnghazad:
I don't see any reason we can't move forward with the core while we're waiting for our PR. Even though probably no one else would, I would still use this mod even if our PR is never accepted and all code editing had to be done in an external editor.

I'm working on some of the innards now. I'm making a sandboxed environment factory function, instead of using table.copy. I already did env and am now working on env.sys.

I can't understand why this piece of code is so complicated: (sys.lua, line 49)

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

dronetest.sys.loadApi = function(name)
   local api = getApi(name, dronetest.sys.sandbox)
   local env_save = table.copy(dronetest.userspace_environment)
   local env_global = getfenv(api)

   -- [[ debug code removed for post brevity ]]

   -- Make drone/computer's env available to APIs, with id and stuff
   --env_save.dronetest = getfenv(0).dronetest
   env_save.sys = getfenv(2).sys
   env_save.print = function(msg) dronetest.print(env_save.sys.id,msg) end
   setfenv(api,env_save)
   return api()
end


What's wrong with something like this?

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

sys.loadApi = function(name)
   local apifunc = getApi(name, sys.sandbox)

   --following line now unnecessary, replaced by closure from containing function
   --local env = table.copy(dronetest.userspace_environment)
   
   local apienv = setmetatable({}, {__index=env})

   setfenv(apifunc, apienv)
   
   return apifunc()
end


I <3 metatables.

[ninnghazad]

@electrodude512
You are right, i guess i just got a little frustrated with not being able to move on my own pace.

That code by no means has to be that ugly, plx fix if you like. Just one of those functions that grew ugly. Do make sure you test out stuff after you change that however, maybe try to run the quarry-demo-prog or so, i guess that uses most stuff we have right now.

If you have stuff written already, please commit changes - i don't mind lots of (partly useless) commit-messages.

[electrodude512]

ShadowNinja added loops and functions to luacontrollers. I'll investigate his changes soon. We should either get to work or just write an OS and lots of peripherals (including an external HDD for fs support) for luacontrollers.

[Jeija]

Just wanted to tell you that if you want to add HDDs, more sensors, peripherals etc. to digilines, I'm open for it!
Also, if my pull request for minetest (https://github.com/minetest/minetest/pull/1869) gets merged, the luacontroller can have an internet database for code snippets and such and it would even allow for stuff like real internet access via digilines.

[electrodude512]

@Jeija:

I'm going to wait to see what ninnghazad thinks before I abandon the project I told him I would help him with. Regardless of that, I have two ideas for features I think luacontrollers should have, whether or not ninnghazad and I decide to use it. I made a fork and will PR it soon, hopefully tonight or tomorrow. I'm adding two features - do you want two separate PRs? I've never done a PR on github before, so we'll see what happens.

[ninnghazad]

Heho.

@electrodude512
I do this for fun, i don't and don't want to feel in competition about this stuff. i think it's great the luacontroller grows.
however, i also do not think digilines and dronetest are redundant.
digilines is to me more of a lower-level foundation - i think of the luacontroller as a programmable microcontroller,
whereas dronetest is in it's usage more like an old pc.
and i do imagine the existence of both, and the connectivity between both, to be a combination of epic amounts of possibilities and strange contraptions.
i would use the luacontroller to build fast, tight circuits,
and the dronetest-computer to access a bunch of these and controll em with a gui or analyze data.
or, well, to control a drone =).

@Jeija
appreciate the offer.
dronetest is currently requiring digilines, and uses it to connect computers/drones with each other
and the only existing "peripherals" (display, transceiver - a modem to talk to drones).
the only thing different to plain digilines peripherals about these is that they have a tiny protocol to present a safe ingame-api to the player for use in ingame-code based on a simple table of functions in each peripheral.
the player uses a wrapper-table generated from that info to call and use the peripheral over digilines in the ingame-code.

that means that i don't plan to reimplement sensors or other stuff that digilines offers,
but i think i do not get around having some special dronetest-peripherals, like the transceiver, or the display.
these would be accessible through digilines, but would talk protocol.
writing specific wrappers for digilines-peripherals i would generally like to avoid, but to use the luxury of that automatic wrapping i might have to in some cases.

i am not informed about the current development of digilines,
and maybe there is already some minimal protocol in the talks (diginet maybe ;) ?), maybe it's out of scope for digilines.
i do want people using dronetest to be able to hook up to some peripheral and start writing ingame-code using it, without having to visit some api-webpage just to get a list of possible functionality - but i would really like to be able to do that with any digilines-peripheral (without having to build own versions of everything or have like a parallel world of peripherals).

[electrodude512]

@ninnghazad:
I am not giving up on dronetest having its own CPU yet. I'm going to do this PR and see what happens. Both the dronetest CPU and the luacontroller CPU have their own advantages and disadvantages based on how they work internally. However, I don't think the deciding factor for which one to use is that one is best for fast/simple stuff (luacontrollers) and one best for slow/complex stuff (dronetest cpu). They should both be fast and powerful enough to do either.

If we switch to using the luacontroller for our CPU, dronetest won't be obsolete/redundant because all of the peripherals will still be extremely useful.

If I end up not convincing you that we should use the luacontroller CPU for dronetest, I'll most likely stick with the dronetest CPU.

Finally, I think we should develop a standard peripheral protocol on top of digilines that can easily be used by both dronetest and luacontrollers.

@Jeija:
I have semi-persistent states working in mesecons_luacontroller- you can now put anything, including functions, coroutines, self-referential tables, and whatever else you want, in the sandbox's global environment and it will stay until the computer reboots. I think mem still works as usual (i.e. completely persistent but you can only put certain things in it), but I haven't tested it yet.

[ninnghazad]

@electrodude512
heho, i thought about it and i wouldn't want to drop the dronetest cpu/core. although that is in the end just based on the fact that i like writing and thinking about it, and i wouldn't want to have to compromise about some stuff that may or may not come up.

still, yay to a common protocol for digiline-devices to be used by dronetest and others.

[electrodude512]

I have committed a draft of the dronetest protocol to github/ninnghazad/dronetest/dronetest_protocol.md. Feel free to make any modifications you like, but if you want to seriously change things, please consider talking to me about it first or doing it on another branch.

I'm committing my changes to the userspace environment on another branch and will merge them back into master once I test them more and fix anything that got broken along the way.

[electrodude512]

I committed my changes to the userspace environment on a new branch "newenv" and fixed a ton of security holes. Your use of getfenv(number) led to all sorts of fun, but I removed it all.

The only reason I didn't commit it to master is because you said not to push stuff without testing drones first, but I don't think we've really reached the point where we should be worrying about drones yet. We should get the basic computer working, with no security flaws and a somewhat stable OS and peripheral interface, and then worry about drones. If you like my changes, feel free to merge them into master.

Can we also get a real event system? This will make things like digiline message receiving, timers, input, etc. much easier, and will make a multitasking kernel practical. My immediate reason for needing a multitasking kernel is to allow registering packet acknowledgement callbacks and packet send retries for my digilines connection protocol. It will also be useful for computers that are supposed to simultaneously have a shell and do stuff over digilines. I can write the kernel - I wrote one for ComputerCraft that had tons of features and worked very well.

We should talk on IRC about stuff soon. If it's not too late today, can you get on #minetest-mods today?

[ninnghazad]

@electrodude512
heho, didn't notice your post yesterday. will hang around irc today till bout 1700 CET, problably for the next days.
i merged and tested your branch, just a few tiny fixes and i was able to run test-quarry without problems. so that went well.
talking bout ugly getfenvs, maybe you could take a look at shell.run and replace that getfenv-stuff with your new methods as well.

event-system yes, well there actually is one. some things just bybass it, like computer/drone communication to a part iirc. and multitasking works quite well using coroutines. there is however no event-duplication or stuff like that for coroutines, like in CC's parallel api. thought about adding something like that that handles event-duplication for coroutines. so i am not sure what exactly you mean by multitasking kernel.
as for digilines-callbacks and event-callbacks, take a look at dronetest.events.register_listener and dronetest.events.wait_for_receive.
it is used for example in the shell to listen for key-events "asynchronously" or to wait for digilines response.
dronetest.events is used for keyboard, digilines, input-box and any other event i came across until now and contains all basic stuff to send/listen/poll events.
the events themselves are stored in a table for each computer, dronetest.active_systems[ID].events
it was a little tricky to get callback/listeners to not trigger lua's "call across c stack" error, but works rather well atm, and its pretty fast.

so, ofc i also read your protocol-draft. i like it. not quite sure about the open/send/close yet, but i'm sure you had plans with that. did i miss the action that just calls a method of a peripheral and returns the result? anyway, i think its a solid foundation, and it won't take much to change whatever stuff (drones) there is atm to follow the protocol. probably only a few changes to peripheral.lua to get the current stuff working, and a few more to have peripheral api fully expose all protocol features.
something minor though - i think message/object-identifiers should have a fixed type in the proto, string in my opinion. for the simple reason that i would not want to have to check X possible combinations of identifiers or stuff just to match a message. something like "CLIENTIDENT-SERVERIDENT-UUID" (https://github.com/Tieske/uuid pure lua uuid gen). i think if somebody wants tables as identifiers, he can just add some on top. seeing how even a few drones can talk quite a lot over digilines, large setups may have to match thousands of messages per tic.
currently msg_id with some random id or something in it is used to match messages of a single request/answer set, something like that plus the object-identifiers would be needed, so n-computers could talk to n-peripherals in as many combinations as they want. not sure if both identifiers were in the draft, forget it if so.

EDIT: when i say bypass the event system i do not mean by mistake. this only applies to certain actions of the drones/transceiver, which i would consider a unique case in that there was no direkt way to have an entity communicate with digilines with nodes. everything else just uses the event system as one would expect.

EDIT2: just to clarify, you can use shell and send stuff over digilines at once right now. it's not blocking each other. hell, you can have 10 drones all blasting tons of digiline-messages and events in parallel using one computer.

[electrodude512]

@ninnghazad:

To send commands to a drone, you open a connection to it and then send it commands. Any response data from the drone goes in the send_ack, although I didn't really explicitly say that in the protocol draft.

Object identifiers do not identify the recipient. They are local to the recipient. For example, if we make a digilines peripheral to control drones over radio (which we should do), you would use different object identifiers to specify which drone the radio should talk to, even though all messages to drones would be sent to the same digilines channel (that of the radio). I see no reason to disallow table identifiers - they are much easier to deal with because they don't need to be parsed.

Object identifiers are only used to initiate a connection (with open). Once a connection is opened, a connection id, which should be a uuid, is used (with send).

The multitasker kernel I want to implement is basically a really fancy version of ComputerCraft's parallel. The difference is that you can dynamically add, delete, pause, reconfigure, send fake events to, etc. processes while running, as well as register functions (again, dynamically while running) that get called on certain events and automatically get deleted when they say that they are done (i.e. they got the event they were waiting for and handled it).

All external peripherals, including drones, should be controlled through digilines. Drones should be controlled through a digilines-controlled radio. Nothing should bypass the event system. Everything that needs a callback should be done through an event.

I am going to start centralizing the event system.

EDIT: I see that you give rom programs access to the minetest table. This is a very bad idea. Any scary things like this should be in apis. In fact, we should probably find a safer way than the apis folder.

[ninnghazad]

To send commands to a drone, you open a connection to it and then send it commands. Any response data from the drone goes in the send_ack, although I didn't really explicitly say that in the protocol draft.

i see... but why not still have a direct method-call without open? would save some overhead.

control drones over radio (which we should do)

we do, the transceiver is a drone-control-radio-digilines-peripheral. it connects per digiline-wire to the diginet, and per its own "wireless" with the drones.

Object identifiers are only used to initiate a connection (with open).

k, i guess thats what i meant with message-id, i just noticed the badness of my naming, as it doesn't just describe a single message but a session of communication.

i don't have my mind fixed on anything with the protocol, probably am ok with whatever you think up. i can see you are knowledgable about it, so go nuts. my "complaints" just stem from not wanting to add too much overhead to simple tasks.

All external peripherals, including drones, should be controlled through digilines.

they are. even the drones. i have not explained myself well enough. the drones aren't nodes, like other digilines-stuff, but entities, like mobs.
entities cannot connect directly to digilines-devices, mainly because digilines does not support that.
the callback that is added to nodes to implement digilines, is not called on entities.
and if it would be, it probably wouldn't be called for entities floating around somewhere like drones.
this is where the transceiver comes into play. it is a node and has the callback.
it hangs in the digilines-network and listens for pakets addressed to drones and receives those for the drones.
the transceiver then acts as the drone's callback, and the drone executes it's stuff like any other peripheral.
it's not that i think this is so cool, a kind of modem-node simply is needed atm to make it work.
the transceiver at least makes it transparent, the player needs it, but does not have to talk to the transceiver,
but can act as if the drone's were simply connected by digilines-wires.
oh, and all digilines stuff in dronetest right now uses dronetest.events.

centralizing the event system

it's central, there's dronetest.events.* and all event-functions sit there. some have wrappers for userspace-access.
the point in storing the events that get passed around in active_systems is obvious i think.
keeping a shared table is error prone and has exponential loss of speed per device added when searching and so on.

I see that you give rom programs access to the minetest table. This is a very bad idea. Any scary things like this should be in apis. In fact, we should probably find a safer way than the apis folder.

ROM is not player writable. whatever is there comes with the mod or was placed there by an admin.
everything in /rom to me is to be treated like an api security-wise. open-backend, secure-frontend.
that's why i meantioned implementing your api-loader in shell.run, to have all calls to /rom handled exactly like apis.
of course that means that all progs/bins/tools whatever we put into /rom must be as securely written as an api,
like not return the minetest object and such stuff.
that last point may not be the case atm (lots of debug stuff in there), but thats the plan.
i can't see any insecurety in a ROM /rom folder itself, and like the approach.
if there are no bugs then the player cannot write to /rom, and has no chance to use any of the goodness without calling premade programs or apis from userspace.
and those apis and programs are no more insecure than any other userspace-bridge stuff.
if system commands or apis fail and give out sensible stuff, thats an error, which is to be avoided in any kind of userspace-bridge.

multitasker kernel I want to implement is basically a really fancy version of ComputerCraft's parallel.

yes yes yes. registering event callbacks from userspace is already possible though. dronetest.events gets partly wrapped to userspace sys.events (or smth like that). but as mentioned with just this coroutines do not each get an event, and cannot listen on the same event. call it parallel, i totally want something like that. ComputerCraft's parallel was much more comfortable to use as a player than plain coroutines with events. a layer atop that with its own event duplication/redistribution will come in handy for lots of possible player-built-projects.

[electrodude512]

super long post warning:

i see... but why not still have a direct method-call without open? would save some overhead.

The overhead only happens when you initiate a connection, which should be pretty rarely.

we do, the transceiver is a drone-control-radio-digilines-peripheral. it connects per digiline-wire to the diginet, and per its own "wireless" with the drones.

so THAT's why drones won't work for me!!! (facepalm) I really should read code before complaining about it...

k, i guess thats what i meant with message-id, i just noticed the badness of my naming, as it doesn't just describe a single message but a session of communication.

If you come up with a better name, by all means, please change it. I wasn't happy with the names and you've shown that they're obviously confusing. The protocol also should probably have a name other than "dronetest protocol" or "digilines protocol" so it doesn't get confused with dronetest or digilines, an easy confusion to make because it involves both but is neither.

i don't have my mind fixed on anything with the protocol, probably am ok with whatever you think up. i can see you are knowledgable about it, so go nuts. my "complaints" just stem from not wanting to add too much overhead to simple tasks.

In real life, telling a drone to do something isn't really a simple task. Also, if you use connections properly, it's one digilines message to send a command to a drone.

I should add a way to bundle messages together, so a single message can serve as both the ack to a send as well as a response send that needs to be ack'd. However, it's probably a better idea to just use get for things like this. I didn't say this in the protocol specification, but get and put can also have (opened) connection ids.

they are. even the drones. i have not explained myself well enough. the drones aren't nodes, like other digilines-stuff, but entities, like mobs.
entities cannot connect directly to digilines-devices, mainly because digilines does not support that.
the callback that is added to nodes to implement digilines, is not called on entities.
and if it would be, it probably wouldn't be called for entities floating around somewhere like drones.
this is where the transceiver comes into play. it is a node and has the callback.
it hangs in the digilines-network and listens for pakets addressed to drones and receives those for the drones.
the transceiver then acts as the drone's callback, and the drone executes it's stuff like any other peripheral.
it's not that i think this is so cool, a kind of modem-node simply is needed atm to make it work.
the transceiver at least makes it transparent, the player needs it, but does not have to talk to the transceiver,
but can act as if the drone's were simply connected by digilines-wires.
oh, and all digilines stuff in dronetest right now uses dronetest.events.

I think there should definitely be a modem node. It makes much more sense than digilines messages, which normally travel through wires, magically propagating through the air just because its destination moves around.
Why aren't drones blocks? Mesecons movestone is a block, so it's not impossible to make moving blocks. They should still use radios even if they are made into blocks.

it's central, there's dronetest.events.* and all event-functions sit there. some have wrappers for userspace-access.
the point in storing the events that get passed around in active_systems is obvious i think.
keeping a shared table is error prone and has exponential loss of speed per device added when searching and so on.

I don't see why the callbacks and listeners aren't done in userspace. I don't see how there's any searching involved or what's error prone about it, either. It takes O(n) time to remove everything from a queue and O(nlog(n)) time for a priority queue, which are both way less than exponential.

Should we use the mesecons action queue instead of our own queue? dronetest depends on mesecons anyway, so we might as well use it.

I'm going to go ahead and implement my own event queue and kernel alongside the existing event system. I'll leave the existing one in place until mine works reasonably well.

ROM is not player writable. whatever is there comes with the mod or was placed there by an admin.
everything in /rom to me is to be treated like an api security-wise. open-backend, secure-frontend.
that's why i meantioned implementing your api-loader in shell.run, to have all calls to /rom handled exactly like apis.
of course that means that all progs/bins/tools whatever we put into /rom must be as securely written as an api,
like not return the minetest object and such stuff.
that last point may not be the case atm (lots of debug stuff in there), but thats the plan.
i can't see any insecurety in a ROM /rom folder itself, and like the approach.
if there are no bugs then the player cannot write to /rom, and has no chance to use any of the goodness without calling premade programs or apis from userspace.
and those apis and programs are no more insecure than any other userspace-bridge stuff.
if system commands or apis fail and give out sensible stuff, thats an error, which is to be avoided in any kind of userspace-bridge.

There should only be one way to get anything that's not sandboxed. It should be from a single function that accepts an identifier string, not a path. Non-sandboxed things should not be in the sandbox filesystem (see the next paragraph for why). The fewer points of contact between the inside and outside of the sandbox, the fewer opportunities for someone hosting a minetest server with our mod getting their home folder (or hopefully only there server sandbox) wiped. It is critical that there are as few channels through the sandbox wall as possible, and those few channels be very carefully checked for security flaws.

Putting non-sandboxed things in the normal filesystem won't work for the filesystem driver I'm writing that supports mounting arbitrary devices in arbitrary folders, similarly to how unix mounts work. It would allow you to remount /rom somewhere else and mount whatever you wanted in /rom, and cause all sorts of fun.

Why is shell.lua an api and not a sandboxed program or at least a sandboxed library?

I have an idea: we should make it so computer formspecs have an inventory that you can put upgrades into, such as removable drives and certain other peripherals such as one that exposes a C encryption library. This would work well with my filesystem manager. Internal peripherals would be accessed through digilines, with channels of the form computerchannel:slotid or something, except that they probably should not be accessible from outside of the computer. There should also be a backplane block that can hold more item peripherals, in case you need one item peripheral to be accessible by multiple computers or in case they don't all fit in one computer.

yes yes yes. registering event callbacks from userspace is already possible though. dronetest.events gets partly wrapped to userspace sys.events (or smth like that). but as mentioned with just this coroutines do not each get an event, and cannot listen on the same event. call it parallel, i totally want something like that. ComputerCraft's parallel was much more comfortable to use as a player than plain coroutines with events. a layer atop that with its own event duplication/redistribution will come in handy for lots of possible player-built-projects.

Yes, but it should be done in userspace. My kernel adds the layer you speak of, except in userspace. Computercraft's parallel used events. Anything that would run under CC parallel would run under my kernel.

I'm so concerned about security because, if it isn't highest priority, people are going to start using our mod before security is working properly and end up getting their servers wiped and getting mad at us for writing a horrible mod with no security.

[ninnghazad]

we do produce quite the walls of text =)

The overhead only happens when you initiate a connection, which should be pretty rarely.

i guess in the drones case this would mean opening a connection when wrapping the drone, so no additional hassle for the player and only little overhead.

so THAT's why drones won't work for me

=). place computer and transceiver, either together or connected with digiwire. place drone nearby, but a little below the computer (so it doesn't dig that). switch on computer and type "quarry 1 4 9" and click exe. drone will speedily dig through a 9x9 area, spitting out some ores when done. if you place a chest above it, it will put the mined stuff in it.
if you use the version of minetest from my fork (with the formspec-stuff in it) you can even try the keyboard-feels and type in commands instead of using that inputfield/exebutton-thing.

If you come up with a better name

object-id is already better then message-id. maybe connection-id would be fitting. not sure. for the protocol's name... phew... idk

it's not impossible to make moving blocks.

nono, that wasn't really the problem. actually the drones started out as blocks. but thinking about how to have them interact with nodes/mobs/players i ended up changing them. entities are better to animate, and other entities, like mobs, can interact/notice them with ease. also rotation and stuff just was more natural with entities.
there is a drone-node that looks like a drone, but it's only used by players to place drones.
that node turns into a real drone once placed.
the drone's actually perform really well.
however, as you like the modem-approach, i'd say things are well on this front.

O(n)

ofc, but i meant the fact that a single list of events for all systems (not split by system) would mean O(numSystems*numSystems*n) seeing how every system needs to search n*numSystem events.

callbacks and listeners aren't done in userspace

well, there are some wrappers to provide the event-functions to userspace. this is simply because i wanted quick access to events from userspace. that doesn't mean that userspace couldn't have an additional "real" event-layer (even outside of parallel i mean). currently userspace-event functions just fetch/send events from the mod's main event-queues, where currently all events/signals pass through. i'm not sure about the advantage of having another layer (outside of parallel,where it is obviously needed for duplication and stuff) but it's ofc possible.
i did not use mesecon's (or any other 3rdparty) queue just because of speed considerations. i neither wanted to "pollute" the mesecons queue, nor loose control over how the internal queue works. a few peripherals and a complex piece of software running will result in tons and tons of events, i want to reduce tic-based latencies as much as possible - meaning events that trigger callbacks shall do so immediately, and not wait one tic for a pollingmechanism to trigger. without this, for example, a player typing on a keyboard-peri would have multiple tics of delay for his actions to appear on a display. it was like this at some point, and it was just terrible to use, so i made sure callbacks get called right away on sending events.

Why is shell.lua an api and not a sandboxed program or at least a sandboxed library?

well, to me thats pretty much the same in this case. api may be a bad name for this, but shell will contain more then just the run().

non-sandboxed things in the normal filesystem won't work...

well now the mounting filesystem is an argument you got me with. it'd be insane to have to put exceptions for /rom everywhere in a unix-style filesystem with remounting.

maybe we should call the whole possible contents of /rom "modules", and not have it accessible through fs. that would in the end just mean that programs like ls would have to be in userspace, and be written as userspace-code, and stuff like the quarry or the tests (just as examples right now) would have to be turned into modules and a corresponding userspace-prog. ls would be userspace code that accesses fs-module. quarry would be userspace-prog that accesses peripherals-module, maybe a special drone-module with convenience-functions and other modules.
this would dictate the single point of entry as the module-loader (which would be your api-loader basically).
this would also relieve us of the api/lib/prog/tool/yomom namings, all that stuff is a modules.
they don't reside in fs at all, and only exist to players as the output of lets say dronetest.load_module().
the more i write "modules" the more i like the idea =). modulesmodulesmodules....
how about "modules", no /rom? such safe, very compromise?

Yes, but it should be done in userspace

oh, i was talking about userspace. that /rom/api/coroutine.lua is userspace-coroutines. nothing fancy, just wanted to make sure there is a way to have them without having luajit/minetest choke. but that changes what i said - these userspace-coroutines are no userspace-parallel for the same reasons i mentioned before. userspace-parallel totally welcome, u can look to the simple coroutines to see whatever it was i did to them to not throw cross-boundary errors (that happens because luajit and coroutines in coroutines in sandboxes and stuff, but as i said we can wiggle around that).

I'm so concerned about security because

well, your argument is sound. i'm a chaotic person, but i absolutely get your concern and for that can surely agree with you to give security more early priority. i wouldn't argue any other way, but wouldn't have cared much for what happened before a public release. but thats a bit of ignorance on my side, it's github. it was public from day 1.

...inventory....removable drives....peripherals....

ye, i also have that in mind. but lets put that back a little and focus on basic peripherals while keeping internal ones in mind. there may be some items to stick into computers that aren't digilines-devices, like power-related stuff. not sure about the whole power thing really yet.
but yes, lets have that, but i would say it's not on the top of the list atm.


i think this is growing to be a really nice mod.
it already is fun, one can make little chains of production using just chests and drones and a computer.
the ugly way a drone currently spits out items into the sky when there is no chest just cracks me up.
after like 5 minutes there are still ore's raining down from the sky. to good to fix =).

[electrodude512]

i guess in the drones case this would mean opening a connection when wrapping the drone, so no additional hassle for the player and only little overhead.

yes

if you use the version of minetest from my fork (with the formspec-stuff in it) you can even try the keyboard-feels and type in commands instead of using that inputfield/exebutton-thing.

I'll try your fork.

nono, that wasn't really the problem. actually the drones started out as blocks. but thinking about how to have them interact with nodes/mobs/players i ended up changing them. entities are better to animate, and other entities, like mobs, can interact/notice them with ease. also rotation and stuff just was more natural with entities.
there is a drone-node that looks like a drone, but it's only used by players to place drones.
that node turns into a real drone once placed.
the drone's actually perform really well.
however, as you like the modem-approach, i'd say things are well on this front.

I think the modem should also be used for wireless digilines.

a single list of events for all systems (not split by system)

I don't want a single list of events for all systems. I want one list per system. It seems this is what there already is, but you implementation searches the queue instead of just popping the oldest element off.

well, there are some wrappers to provide the event-functions to userspace. this is simply because i wanted quick access to events from userspace. that doesn't mean that userspace couldn't have an additional "real" event-layer (even outside of parallel i mean). currently userspace-event functions just fetch/send events from the mod's main event-queues, where currently all events/signals pass through. i'm not sure about the advantage of having another layer (outside of parallel,where it is obviously needed for duplication and stuff) but it's ofc possible.

My kernel does this, but in userspace. It's not another layer, it's the same layer in a different place.

i did not use mesecon's (or any other 3rdparty) queue just because of speed considerations. i neither wanted to "pollute" the mesecons queue, nor loose control over how the internal queue works. a few peripherals and a complex piece of software running will result in tons and tons of events, i want to reduce tic-based latencies as much as possible - meaning events that trigger callbacks shall do so immediately, and not wait one tic for a pollingmechanism to trigger. without this, for example, a player typing on a keyboard-peri would have multiple tics of delay for his actions to appear on a display. it was like this at some point, and it was just terrible to use, so i made sure callbacks get called right away on sending events.

Two separate queues is probably slower than one bigger queue. I'm thinking of rewriting the mesecons action queue and sending them a pull request that makes it use a heap priority queue (O(n*log(n) time instead of O(n^2)) and allows events that get run on the same tick they get handled. If they don't like events that get handled on the same tick, we can just use the mesecons action queue (with a heap so it's fast) but have our own wrapper function that normally sends things to the mesecons action queu but handles events that should be handled the same tick specially.

but shell will contain more then just the run().

Like what? Why? The shell should definitely be in userspace.

maybe we should call the whole possible contents of /rom "modules", and not have it accessible through fs. that would in the end just mean that programs like ls would have to be in userspace, and be written as userspace-code, and stuff like the quarry or the tests (just as examples right now) would have to be turned into modules and a corresponding userspace-prog. ls would be userspace code that accesses fs-module. quarry would be userspace-prog that accesses peripherals-module, maybe a special drone-module with convenience-functions and other modules.
this would dictate the single point of entry as the module-loader (which would be your api-loader basically).
this would also relieve us of the api/lib/prog/tool/yomom namings, all that stuff is a modules.
they don't reside in fs at all, and only exist to players as the output of lets say dronetest.load_module().
the more i write "modules" the more i like the idea =). modulesmodulesmodules....
how about "modules", no /rom? such safe, very compromise?

Yes, we should do this. The top few layers, such as the mount manager, will be in userspace. Why does the quarry test need a module? It should just use digilines->modem->drone.

oh, i was talking about userspace. that /rom/api/coroutine.lua is userspace-coroutines. nothing fancy, just wanted to make sure there is a way to have them without having luajit/minetest choke. but that changes what i said - these userspace-coroutines are no userspace-parallel for the same reasons i mentioned before. userspace-parallel totally welcome, u can look to the simple coroutines to see whatever it was i did to them to not throw cross-boundary errors (that happens because luajit and coroutines in coroutines in sandboxes and stuff, but as i said we can wiggle around that).

How about we make the userspace loadstring so that it enables jit and uses the on-function-call hook instead if it can determine that the function will call another function inside of every loop? This might be tricky. For example, this can use jit:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

while true do foo() end


because there is a function call every time through the loop.

...inventory....removable drives....peripherals....

ye, i also have that in mind. but lets put that back a little and focus on basic peripherals while keeping internal ones in mind. there may be some items to stick into computers that aren't digilines-devices, like power-related stuff. not sure about the whole power thing really yet.
but yes, lets have that, but i would say it's not on the top of the list atm.

It's not on top of my list either, but it is fairly high up. They will be implemented as fake digilines devices.

the ugly way a drone currently spits out items into the sky when there is no chest just cracks me up.
after like 5 minutes there are still ore's raining down from the sky. to good to fix =).

I'll have to try that! After we fix it, we should make an internal peripheral for drones that makes them do this the old way, just for fun. It doesn't need a crafting recipe (and probably shouldn't have one).

[ninnghazad]

The shell should definitely be in userspace.

ye ofc, i didn't mean any complicated stuff, just the stuff that comes with a normal shell. idk, like setting a custom prompt or little things like that, that thematically belong in shell.* - all just userspace. it currently gets its key-events from dronetest.events' userspace-wrappers, but that could be any other way.

but you implementation searches the queue instead of just popping the oldest element off

ye, thats ugly. i think i left it like that cause of the filters. so receive({"key","mouse"}) would eat up key and mouse events, but would not pop any other it is not supposed to return.

My kernel does this, but in userspace.

k, the userspace-wrappers for dronetest.events are ugly anywho. we just tweak dronetest.events to immediately feed userspace-events that come from outside ("key"-events ie.) to the userspace-queue or something. so that shell.* for example has access to key events, not just those generated in userspace.

Why does the quarry test need a module? It should just use digilines->modem->drone.

that quarry thing is just kind of a test for the other parts of the mod and if they work together. so currently it uses stuff that is not accessible to players, but that shall all end up in the drone's peripheral-methods at one point. i thought about a module for whatever helper-functions and stuff may be left that just don't fit well with the drone itself. but after reading the quarry code again i think everything that's in there can be formulated (without getting weird) as a method of the drone-peripheral. just a few function like snapRotation or stuff would be left, those don't need own module.

while true do foo() end

well, i'm not too happy with jit.off(). but this way was the only foolproof way i found at the time. other stuff would mean having to parse and scan all player-lua using game-lua, and parse well too - so many ways to hide nasty loops and recursions. also this would mean that all player-lua's loops and whiles have to contain function-calls. and ah, that is kind of strange, one could work with it but having to write a function for every loop... meh.
it's ugly as it is now, but it works - i would prefer this over having to tell players they cannot just write plain loops.
would prefer a jit-on way ofc, but if you read around luajit, (ex-)coco and the whole mess, i'm rather happy we can yield across coroutines at all =).
and as userspace-coroutines prove, even if ugly, we can do what we need with it. i think the parallel module will work with the same tricks no prob.

[electrodude512]

ye, thats ugly. i think i left it like that cause of the filters. so receive({"key","mouse"}) would eat up key and mouse events, but would not pop any other it is not supposed to return.

I'm designing my kernel so you can set the filter by passing it through coroutine.yield or kernel.setfilter(filter) (which the kernel calls when you give it a filter through coroutine.yield). Passing nothing to coroutine.yield just uses the old filter. If you set a filter like {"key", "mouse"}, it will get converted to {["key"] = true, ["mouse"] = true} by kernel.setfilter and then the new filter will be saved in the process table, so the kernel only needs to do a hash lookup and not a linear search to determine if a process wants an event.

k, the userspace-wrappers for dronetest.events are ugly anywho. we just tweak dronetest.events to immediately feed userspace-events that come from outside ("key"-events ie.) to the userspace-queue or something. so that shell.* for example has access to key events, not just those generated in userspace.

Yeah, pretty much.

having to parse and scan all player-lua using game-lua, and parse well too - so many ways to hide nasty loops and recursions. also this would mean that all player-lua's loops and whiles have to contain function-calls. and ah, that is kind of strange, one could work with it but having to write a function for every loop... meh.

If the parser isn't sure, then the function doesn't get to use jit. It will still run, it just won't get to use jit. If there are no functions in a loop, the userspace loadstring can add a dummy one to the beginning. It might be easier just to add a dummy function call after the "do" of every loop instead of tracing all possible execution paths (good luck!). You can enable jit per-function, right? Also, maybe modules can say they want jit without having to be checked first.

If we do this, we should have a table that's global to all computers whose keys are strings of the original functions and whose values are strings of the processed and jit-safe versions. That way, the first load of some commonly used file will take longer because it will have to parse and fix it, but the rest of the times that function is loaded will go quickly because it just has to get the processed string out of the hash table.

[ninnghazad]

Passing nothing to coroutine.yield

why pass stuff to yield()? i think thats an odd spot to put event handling. if that is what it is. yield() should yield and nothing more. kernel.events.receive[_async](filter)/send(type) + kernel.events.* or whynot?

can you give a rough example code how u would imagine using that parallel kernel like, to call 2 functions at once? (lets just assume those would use some events the kernel would have to provide)

about the debughook/jit.off():
take a look at the coroutine.lua module, it is like 20 lines of codes. a bit strange, and not luajit.
however, this solves 2 problems:
- endless loops
- cross boundary yielding
while the solution you mentioned would ideally solve the loop-problem, it would not solve the yielding-problem.
without it solved there are no userspace coroutines and thusly no parallel kernel possible.
it absolutely is necessary to have proper yielding.
furthermore, it would introduce other problems in debugging: the debugging information the player may get when writting bugging code will not be accurate, and point to wrong lines/non-existing stuff that may be added by the extra-parser.
the yielding-problem basically needs the jit.off for a systems main coroutine and/or userspace coroutine before the coroutine's function has been used.
the loop-problem needs the debughook that counts instructions.
and actually i think the debughook for the endless-loops isn't so bad, it's rather simple yet solid. i do not quite remember if the debughook mechanism worken without jit.off(), but doesn't really matter as we need the yielding.

EDIT: if however you would see a need for an extra loadstring with extra parser that has no coroutines/yielding or player-debugging and want to write the mentioned parser, go nuts =)

[electrodude512]

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

local queue = os.loadlib("queue")

local q = queue.new()

local popper = kernel.spawn("pop",
   function()
      while true do
         coroutine.yield({"queue_push"})
         local i = q:pop(i)
         if i then
            print(i)
         end
      end
   end
)

local pusher = kernel.spawn("push",
   function()
      local i = 1
      while true do
         q:push(i)
         i = i + 2
         kernel.send(popper, "queue_push")
         kernel.yield()   -- like coroutine.yield, but guarantees that we get a turn next tick
      end
   end
)

kernel.wait(pusher)

kernel.wait(popper)


coroutine.yield is better because it forces processes to yield in a reasonable amount of time. It also makes it easier to determine whether or not a computer or process actually needs a turn. I'll add a feature to my kernel so if ten programs all want a dummy event because they want a turn next tick, only one dummy event will get queued but all of the processes that want one will still get their turns.

kernel.events.receive[_async](filter)/send(type) + kernel.events.*

Fetching events through coroutines is how computercraft does it, btw.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

function os.pullEventRaw(...)
   return coroutine.yield(...)
end


How about we wait until everything generally works before worrying about using jit?

[electrodude512]

one thing about the example i don't get yet: kernel.spawn("bla" < whats the bla for? or is it just the name for the function?

Yeah, it's the process name. PIDs are unique and are used as process handles (because that's what they are). Names aren't unique but mean something to humans.

[electrodude512]

I pushed some stuff to github on a new branch called "newevents". Don't merge it yet - most of it never gets loaded and hasn't been tested yet. I mainly pushed it so you can see what I have so far for my kernel. The version I pushed is completely untested. I don't know about the version I have now - it compiles and loads, but there's an error somewhere else, in os.loadlib (userspace library loader, I pushed a horribly broken version of it).

I'm going to be unavailable all weekend. I'm going to push all of my very WIP changes to newevents, even though it doesn't run, in case you want to do anything. Don't merge newevents into master yet, but I guess you can edit newevents if you want, even though it doesn't run and probably won't make any sense because everything is currently very inconsistent.

[electrodude512]

I got the userspace library loader working and changed bootstrap to load my kernel and tell it to run the shell. The shell program says "Welcome to dronetest shell.\n#" but doesn't work yet.

I'm going to implement things like the get function for my protocol (it needs a name!) to return a response handle instead of waiting for the response and then returning the actual response. That way, you can send 20 requests to one device, but none of them will get sent until your program yields. The last thing the kernel runs before yielding back to minetest will be a program that collects all queued dronetest messages, put them into bundles as appropriate (one bundle per channel), and sends them. The next time the computer gets a turn, the first thing the kernel runs will be a program that waits for a digilines response, separates bundles into individual responses or requests, and puts all of the requests into their request handles. That way, you can request 20 files, yield once, and get all 20 responses back at once, instead of having to yield 20 times, once per object.

I'm going to push everything to branch newevents once the shell runs, which should be pretty soon.

EDIT: I pushed everything. It runs, but it takes a while to boot (probably just not getting a turn because no events and so no turns or something silly). I barely tested it, but shell and lua work.

[ninnghazad]

heho electrodude.
thats some good news. i did not yet find the time to test the newevents branch.
request/response-bundling is a good idea. but make one version with immediate yielding. so that the player can have a single request with built in yield, or have the bundled request to be send upon the next manual/auto-yield. makes player's code a little shorter in cases of lots of conditionally depending peripheral-calls, like with drones.
will test the branch in the next days and start the pull.

"request 20 files":
what about requesting really large files in chunks? i think C's basic filehandle-operations like fseek,fput/fwrite should be supported by protocol, and should be supported seamlessly by a fs-module using an implementation of the protocol.
so that to a player a disk/mem-peripheral used through the shell may feel like a hdd on linux (mount, filehandles and stuff).
nice idea for peripheral btw, have hdds for permanent storage and mem for fast shared (parallel) storage that does not get saved but with same/similar peripheral-interface.
at this point we will also have to consider how long it may take to read a large file in a single go - we will need to make sure that actual reading of files on the server happens in a chunked and buffered manner with possible yields between chunks. this will make reading of large files slower, but is the proper way to go i think - otherwise players would be able to introduce uncontrollable io-load and could make the server choke. size of rw-buffers must be strictly limited ofc.

also i have an idea for preventing users from trying to overflow memory, but will have to try stuff first.

[electrodude512]

request/response-bundling is a good idea. but make one version with immediate yielding. so that the player can have a single request with built in yield, or have the bundled request to be send upon the next manual/auto-yield. makes player's code a little shorter in cases of lots of conditionally depending peripheral-calls, like with drones.

I don't think a separate function or a flag for the same function is worth it. Instead of using

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

local f = fs.read(path)
-- wait, do other stuff
local str = f:getdata()


you can just use

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

local str = fs.read(path):getdata()


which will do what you want. I'm using weak tables where appropriate, and removing a request object from its device's request table is all that has to be done to clean it up, so you don't have to worry about closing the request.

"request 20 files":
what about requesting really large files in chunks? i think C's basic filehandle-operations like fseek,fput/fwrite should be supported by protocol, and should be supported seamlessly by a fs-module using an implementation of the protocol.
so that to a player a disk/mem-peripheral used through the shell may feel like a hdd on linux (mount, filehandles and stuff).

I don't see myself ever using this. Every time I've ever used a file in lua, I've either read the whole thing into a string first or iterated over it by lines. This will only be useful for very large files; a string copy taking ten times longer is much better than sending ten requests. I will add a way to specify a byte range for the get and put commands to the specification, but it's near the bottom of my list of things to actually implement right now.

nice idea for peripheral btw, have hdds for permanent storage and mem for fast shared (parallel) storage that does not get saved but with same/similar peripheral-interface.

Inter-process communication will normally be done through tables. However, there will be a fake connection type that's basically a unix pipe (it will support all methods, not just send) between two processes. Ramdisks are only for when you want to keep a bunch of actual files in ram - an example would be as a filesystem cache to avoid constant remote fetching of stuff, or for a filesystem that has to be fast but can't just be a table or pipe because one of the processes involved expects to have a real filesystem. It will be possible to place a pipe (or any device or connection, for that matter) in a ramdisk; I'm not sure yet how this would be done or if it makes any sense on a real filesystem. I'm 99.99% sure on Linux that if you make a named pipe, write stuff to it, and reboot your computer, the stuff in the pipe will be gone.

at this point we will also have to consider how long it may take to read a large file in a single go - we will need to make sure that actual reading of files on the server happens in a chunked and buffered manner with possible yields between chunks. this will make reading of large files slower, but is the proper way to go i think - otherwise players would be able to introduce uncontrollable io-load and could make the server choke. size of rw-buffers must be strictly limited ofc.

The device should use coroutines to slowly read the file in chunks, then piece the chunks together and send the whole thing at once. If you want a big file (or a big chunk of a file), it will work without lagging the server but will take a while. There should also be a maximum file size.

also i have an idea for preventing users from trying to overflow memory, but will have to try stuff first.

How?