DBMS driver, DBMS player persistence, persistence API

[GauVeldt]

two ideas I'm considering in my project's MT fork:

1. altering the database code to use MariaDB (or Postgre SQL or even MySQL) or similar distributed-server-friendly dedicated DBMS; and

2. altering the player persistence code to use DB tables rather than filesystem files which update more slowly and probably contribute to the "inventory lag" and are not friendly to multiple-server deployments; and

3. providing a persistence API for scripts to persist player data in some standardized manner. creating "inventories" in a mod to store a new health/hunger/money/whatever stat seems hackish at best. A possible ocnsideration is a generalized API that initializes a template as an abstracted player-centric persistence object usable in lualand via attributes and having under-the-hood code persisting attribute alteration (to the DBMS or files depending on feasibility of point #2) on the object automatically.

Any thoughts on these ideas?

EDIT2:
example API use case:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

--
-- prototype: playerdata.new(modname,template,resetFlag)
--
--    modname - the name of the mod and should be the same as the name used initially
--              by the mod to register the mod with the engine
--
--    template - an object whose keys will used to synchronize return object
--               with the corresponding persisted data
--
--    resetFlag - false: values are defaults if not persisted data currently exists
--                       (data survives server restart)
--              -  true: values always reset to match whether current data exists or not
--                       (data reset on server restart)
-- mystats["this"] and mystats["foo"] become per-player persistent attributes

mystats=playerdata.new({"this"="that","foo"="bar"},false)

-- mod may now use/alter mystats["foo"] and mystats["this"]
-- as desired, values are per-character and will automatically
-- persist unless resetFlag was true (in which case they reset
-- to the defaults on server restart)

-- EDIT: as far as I know the internal api code knows only
-- which player is involved with a current script's execution
-- but not the mod originating the call

-- mystats should be kept as a global by the mod's init.lua
-- (or a script called from init.lua) within the mod's namespace
-- (allowing a uniform way for other mods to access
-- eg: thismod["mystats"] accessed from thatmod)


[rubenwardy]

Inventory lag is nothing to do with the time it takes to write player files. It is all in the RAM, and is saved every server save interval - 30 secs, I think.

Once the game is loaded, it is fairly hard to tell which mod is calling a function. Remember, also, that some mods contain library functions which are called by other mods, so getting the calling mod name won't be accurate.

[kaeza]

IIRC, the "inventory lag" is not because of slow I/O, but rather because everything that must switch to Lua (which is basically everything in the engine), must be more or less serialized, because Lua is not thread-safe. For example, moving an item causes the engine to call into Lua code (`on_metadata_inventory_move` and friends), but if some part of the map is generating at the moment (`on_generate` and friends), the inventory move cannot be actually performed until the `on_generate` callback returns and the main loop regains control (and actually calls `on_metadata_inventory_move`), and vice versa.

[rubenwardy]

because Lua is not thread-safe.

Just to expand: You can run multiple threads in Lua. You can't access the VM from multiple threads in CPP without locking it. The Minetest API isn't thread safe. It is feasible for Lua map gen to be running in a different Lua thread, and for the map gen to keep locking and checking on it. It might be better to make a separate VM though, and not allow the map generator access to any Lua globals except from LVM. And perlin stuffs. But - breaking mods.

[GauVeldt]

IIRC, the "inventory lag" is not because of slow I/O, but rather because everything that must switch to Lua (which is basically everything in the engine), must be more or less serialized, because Lua is not thread-safe. For example, moving an item causes the engine to call into Lua code (`on_metadata_inventory_move` and friends), but if some part of the map is generating at the moment (`on_generate` and friends), the inventory move cannot be actually performed until the `on_generate` callback returns and the main loop regains control (and actually calls `on_metadata_inventory_move`), and vice versa.

Hmm. What if the inventory prediction heuristic is changed perhaps then?

Instead of waiting for the server to confirm the inventory change, start a prediction from the point of user interaction and rollback the predicted changes (similar to a DB rollback) if the server subsequently overrules the change in its (lagged) future reply. The visual might "snap" at that point but that is either (a) a cheating client or (b) a client incorrectly predicting the inventory manipulation or (c) a mutliplayer (async) interaction messes with the client's predicted (expected) result server-side, (a) we don't care about them having visual glitches as a result of their (failed) attempt at cheating and (b) is more a bug in the prediction than a failure of the prediction itself, and (c) we can't do anything about and the result looks awkward visually whether exhibiting the current "lag" behavior or the corrective "snap" when prediction doesn't observe the hidden (multiple players taking stuff out of a furnace at the same time for instance) state change(s). The normal case will be the server response confirming the predicted state and no visual "snaps" nor lag would be observed. As an attempt to simplify the rollback the client could mark such inventory slots "dirty" and subsequently disallow interaction on "dirty" slots (they could be temporarily greyed out visually) until the server confirms the changes to those slots and they are then marked "clean" by the client (and the slots cease to be greyed out and once again may interact to the user).

EDIT: If no MT devs are interested in having the client predict this way I'll consider it in my MT fork. If the devs later change their minds they'll have to wade through my commits and find out what I changed :P

Anyways wouldn't DB persistence work better in a multiple-server architecture facilitating one server to see any data changes on another? This isn't so trivial if using the filesystem (DBMSes utilize transactions to handle async). Though as a hack I guess you could NFS or Windows fileshare the player data directories but that is, again, a hack, and riddled with all manner of async issues.

[GauVeldt]

because Lua is not thread-safe.

Just to expand: You can run multiple threads in Lua. You can't access the VM from multiple threads in CPP without locking it. The Minetest API isn't thread safe. It is feasible for Lua map gen to be running in a different Lua thread, and for the map gen to keep locking and checking on it. It might be better to make a separate VM though, and not allow the map generator access to any Lua globals except from LVM. And perlin stuffs. But - breaking mods.

Yeah the architecture can probably be adjusted to at least make the map generation scripts apartment-safe to mod scripts which would run in a different thread apartment. If mapgens needed to see modspace then they'd need a lock or message passing system to exchange data with mod scripts. Or they could use a DBMS-mediated (transactional) datastore. :P

PS: My fork will break a lot of mods anyway since there will be a new blockiverse engine mode (space the final block-tier) in addition to core (planetary) - similar to core but no sky, clouds, fog and coordinates stored sparsely in a DB table for celestial objects, planets (planetary runs in core so the typical -31000..31000 world axes) and blockoids (smaller fixed-size chunkspaces representing various sizes of asteroids, dwarf-planets, ships, etc and blockoids in terms of a normalized (integer) bounding sphere thus containing floor((2*r)^3) or floor(8*r*r*r) chunks, r chunks from the origin in any direction). Yeah in blockiverse it'll be about going from fully explorable block-planet-to-block-planet travelling through space in hastily constructed blockships hoping you didn't mess up the fuel calculations. :P (and yes the 3D sparse data will go MUCH farther than 62000 blocks -- it absolutely has to -- some scaling unit chosen (a dual value maybe using int32:X*pc+float32:Y*au) that allows astronomical navigation coordinates to be stored in terms of minetest's existing serializable datatypes).

[GauVeldt]

Once the game is loaded, it is fairly hard to tell which mod is calling a function. Remember, also, that some mods contain library functions which are called by other mods, so getting the calling mod name won't be accurate.

Okay then I'll edit the prototype and use case in the OP to make it necessary for the mod to pass it's modname string as an argument.

[rubenwardy]

The player's inventory works in the way you suggested - moves the item in the client, and then moves back if it failed. Formspec inventories - such as chests - don't work like this, for some unknown reason.

[GauVeldt]

The player's inventory works in the way you suggested - moves the item in the client, and then moves back if it failed. Formspec inventories - such as chests - don't work like this, for some unknown reason.

So I'd propose that the prediction logic should should be implemented as I described for Formspec'ed inventories. Seems like those take a bit longer to validate so the greying-out of the affected slots wouldn't be such a bad idea while waiting for the server to confirm the transaction.

[GauVeldt]

IIRC, the "inventory lag" is not because of slow I/O, but rather because everything that must switch to Lua (which is basically everything in the engine), must be more or less serialized, because Lua is not thread-safe. For example, moving an item causes the engine to call into Lua code (`on_metadata_inventory_move` and friends), but if some part of the map is generating at the moment (`on_generate` and friends), the inventory move cannot be actually performed until the `on_generate` callback returns and the main loop regains control (and actually calls `on_metadata_inventory_move`), and vice versa.

So a bit of philosopher's dinner going on while stuff gets softlocked waiting for serialization :P

[sfan5]

Minetest supports multiple database back ends, take a look at database-redis.cpp

[thetoon]

1. altering the database code to use MariaDB (or Postgre SQL or even MySQL) or similar distributed-server-friendly dedicated DBMS; and

Don't. The whole idea is great, but don't use a screwdriver to hit a nail. If you're going to add a dependance to an external storage engine, you'd be much better with something from the NoSQL world (CouchDB, ...).

Mind you, relational DB is clearly overkill where all you're going to have is "give me block at X/Y/Z".

[GauVeldt]

1. altering the database code to use MariaDB (or Postgre SQL or even MySQL) or similar distributed-server-friendly dedicated DBMS; and

Don't. The whole idea is great, but don't use a screwdriver to hit a nail. If you're going to add a dependance to an external storage engine, you'd be much better with something from the NoSQL world (CouchDB, ...).

Mind you, relational DB is clearly overkill where all you're going to have is "give me block at X/Y/Z".

Someone doesn't like SQL much. :P

A lot of these non-transactional DBMS backends probably don't behave well in server clusters (async/locking/philospher-dinner issues) where multiple world servers access the same DB. The space faring fork I'm making appears very likely that it may necessitate support for server clusters for large player volumes (space travel seems to be a rather popular gamer's mode of transport.

Also relational starts to make sense if the player data is also going into the DB as random-access rows rather than lines in sequential filesystem text files. Other things like mob/player reputation matrices practically scream for it.

[thetoon]

Someone doesn't like SQL much. :P

Sure, but that someone wouldn't be me. It's just SQL is overkill when you want key/value storage. Minetest's legacy storage scheme is really an abuse of SQL, for instance.

A lot of these non-transactional DBMS backends probably don't behave well in server clusters (async/locking/philospher-dinner issues) where multiple world servers access the same DB. The space faring fork I'm making appears very likely that it may necessitate support for server clusters for large player volumes (space travel seems to be a rather popular gamer's mode of transport.

RDBMS aren't too good at that either, unless you go for Oracle which would probably be overkill - beside behind highly un-convenient. CouchDB's model isn't actually too bad, but there might be much better ones.

Also relational starts to make sense if the player data is also going into the DB as random-access rows rather than lines in sequential filesystem text files. Other things like mob/player reputation matrices practically scream for it.

I'm all for "common" data storage, but then again I don't see how rigid RDBMS would deal with the variety of mods' needs - beside reimplementing key-value on top of SQL, like it's too often seen.

[thetoon]

Maybe I'm stating the obvious, but you might want to take a look at MTSatellite : https://bitbucket.org/s_l_teichmann/mtsatellite/

And especially its mtredisalize component : https://bitbucket.org/s_l_teichmann/mts ... redisalize

[Krock]

I would like to see a more Windoze-friendly world-backend as "alternative" to leveldb.
MySQL is not a good soulation because we only need a single table, therefore sqlite was a good chose.

The map saves all X seconds, you can define this in your minetest.conf. I use ~120s for this and haven't got any big problems/lags with sqlite3.

There is already a mod to collect player data (number of dug/placed nodes, ..)

[GauVeldt]

I would like to see a more Windoze-friendly world-backend as "alternative" to leveldb.
MySQL is not a good soulation because we only need a single table, therefore sqlite was a good chose.

The map saves all X seconds, you can define this in your minetest.conf. I use ~120s for this and haven't got any big problems/lags with sqlite3.

There is already a mod to collect player data (number of dug/placed nodes, ..)

Is the mod using a database backend (I didn't think there was any api for that) or does it use the existing filesystem based storage? The filesystem based storage won't work for a clustered server arrangement. It would be like attempting to run lua scripts unserialized.

In the case of my fork I want it possible for clustered servers sharing data so transactional access to not only map data but transactional player and other data is also necessary thus meaning maps won't be the only thing in DB meaning something like MariaDB (or MySQL and friends) becomes more suitable.