And I didn't realize this was a fairly old thread until I was half way through writing this post. So I'm going to go ahead and post it and hope I don't get tar and feathered.There are lots of ideas in this area all of which need more research.
For example you could probably get away from the security and, to a lesser extent, platform issues by using a C/C++ JIT compiler(may not be the right term though my google results are what I was thinking of). I've come across micro c/c++ compilers that are intended to be embedded into a program and used during runtime. So you distribute your c++ mod and it will be compiled on the platform in a sand-boxed environment. The modder will still need to make sure it runs well on all platforms but we already have that in Lua to a lesser extent (esspecially if your interacting with the file system).
Another idea you could do right now is create a C++ program that runs alongside minetest and use a mod to bridge the two programs. For example make some sort of C++ program for AI then let it interact with the minetest world through a lua mod (or maybe even the network protocol). This should work well with servers but not so well with singleplayer since you would need to run the extra program.
Quick disclaimer the following are just my thoughts. I have done C++ programming, but certainly do not know what all would go into re/designing the current engine's architecture around Core Mods.I'm liking the idea of core mods as well. Maybe it could use a C++ JIT compiler.
Ferk wrote:I think the precompiled coremods idea would have some problems:
- They will not be portable to all platforms Minetest supports (like it was already pointed out in this thread).
This is true but just means the burden is put on modders to make sure it works on multiple platforms. If they are making a Core Mod then it is likely they know what they are doing and know it will need more testing which is the price for performance.
Ferk wrote:- There are not that many mod developers that would use it. A lot of the people making mods are not even programmers. I'd guess most of the people willing to write a C++ mod that implements an AI are already working in the engine code.
I think that is the point. Most mods would be in lua stil with only a few mods that need heavy processing being made by more experienced mods. That might even be a good learning tool, you work your way up from registering simple nodes to working on more complecated lua mods and then when you are more experienced and start pushing the limmits of what lua can do you switch to making Core Mods. The you go off and make your own engine. :P
Ferk wrote:- You can already patch the engine if you wanted. It's open source.. if you want to code an AI in C++ that can be enabled from within Lua you can just write it in the current engine codebase.. you'll even have more freedom this way because you'll have the whole engine code exposed for you to change and study instead of having to depend on a limited API. I bet the devs would be happy to accept a PR that provides optional AI behaviors for the entities or many of the other features you mentioned were they nicely implemented (and if they are nicely implemented but still not accepted.. maybe go with that code to one of the forks, the day such significant features were officially rejected in core might be the day a Minetest fork would be able to really set itself apart).
I see a couple problems with this thinking.
First, not everyone will want the same features, but if you add all features that one could want, the engine will be slow.
Second, there is *always* politics when trying to get a PR accepted: someone doesn't like the way you did it, another person says it breaks compatibility so shouldn't be added, ect. This like you said causes forks which is ok, but that makes the features less accessible to the "masses" and harder to find. I think it is best, rather then fragmenting the community with forks that add some C++ feature, let players choose to include Core Mods that does what he wants.
Core Mods also gives modders some autonomy without decreasing the quality of the engine code.
Ferk wrote:- Creating an API is not easy and it's time consuming. I think Minetest dev resources are already limited.
- We would have 2 different modding APIs for the server.. why would we want to maintain both? one of the APIs is inevitably gonna fall behind and be disregarded.
I think part of the idea is to allow the Core Mods to interface with the engine more closely extending the C++ code directly. You may not need too much of an API to allow them to interface with the engine. And we would want to maintain both so that we can have a better experience and more flexability. If one does fall out of use it would probebly be the Core Mods and we would know that they were a bad idea.
Ferk wrote:- Precompiled code could be dangerous for people who just pick whatever binary from the internet and run it.
- Since you can't directly see the code as you can with lua, it won't be as "free as in freedom"-friendly (even though this is not as important since I can see this community really understands and I'd guess those mods will still be FOSS).
Both of these could probebly be solved with some sort of sandboxed JIT C++ compiler.
Ferk wrote:On top of that.. I don't think the bottleneck for the performance is in the Lua language. I think the problem is that all the processing is done in the server and most of the lag is network latency. I do not experience any problems with mobs when I play singleplayer or in a local network. I think client-side Lua modding is a more approachable solution.
With Core Mods you might beable to strip out some of the code from the engine that slows it down. Maybe you could even pull all the legacy code into Core Mods and say "If you don't want this feature broken add in this code mod and it will be just like it was."
I guess in my mind I am thinking that Core Mods could make it so: the engine is treated as a stable core that is only tinkered with occasionally, core mods are treated as performant APIs that add gameplay mechanics but are constantly changing and innovating, and Lua Mods are treated as content creators more or less how they are used now. So that it is really easy to add content, a bit harder to design good core mechanics, and really hard to effect the engine itself.