FOSS gamedev and creative worlds

How do they work? Do I have to make core changes instead then rebuild Minetest?

I think so. Minetest doesn't have any C++ plugin system or anything at the moment.

You can make C++ Mods, but they won't be plugin mods, they'll just be patches. Like MC. You will have to recompile though

Lua allows to load compiled libraries (*.so, *.dll) into a mod but doing it that way is just an ugly workaround and slower than a direct function in Minetest to include them.