> Is it possible to change the default mapgen flags for a subgame, so the user can still overwrite them?
I doubt it.
> I think of this: If Minetest fails to place any node (except those with probability 0), the entire decoration is not placed. Think of it as an “all-or-nothing decoration”. :D
Is this possible?
Possible but complex and slow, it would have to be optional because with trees we want them to intersect and overlap, decoration code has of course to be lightweight and fast because it acts during mapgen. So no doing this is not liely to happen.
As explained you can already avoid this using order of registration.
> But I still have large plants over ferns (=jungle grass), maybe because it also comes from v6
This is because the mgv6 1-node decorations are registered in default/mapgen.lua before your mod decorations so you have the order of placement wrong. It's not a bg.
You need to 'minetest.clear_registered_decorations' then re-register all (including mgv6 decorations) in a new order.