FOSS gamedev and creative worlds

I wonder if it is possible to configure a minetest server to perform authentication using some external system, like a BD or a REST endpoint instead of the auth file?

This and this may give you a starting point.

As to how to connect/query/whatever the authentication backend, you will need to code a dynamic lib (.dll/.so) exposing the low-level functionality to be loaded from Lua, and code the high-level layer directly in Lua. I'll leave the googling to you on how to do that.