FOSS gamedev and creative worlds

Are there any cool game modes out there? like better mobs n stuff?

There are no real "mods", however there are patchsets adding some additional content. They're usually harder to install, as they require modyfing the C++ game code then recompiling it (where in Minecraft, the mods' code is already compiled).

As Calinou said, you have to fork the project to add features and mods.
If you want to test unofficial features, you can browse github by looking for Minetest's forks and compile them.