FOSS gamedev and creative worlds

I' ve just started a new project, the 3m - Minetest Mod Manager, which is meant to become kind of Package Manager for Minetest mods with official mod list, ability to sync it from network, check versions, update, install and remove mods.

If you want to contribute, just let me know. I will prefer the "shared repository" model of contribution rather than forks + pull requests.

Github here: https://github.com/Phitherek/3m

EDIT: Do not expect fast development, as I do not have so much time to do this, but it will slowly get its shape.

There are two modmanagers (those I know) right now:
Ingame: http://minetest.net/forum/viewtopic.php?id=2568
In python: http://minetest.net/forum/viewtopic.php?id=618

What's different about your mod manager?
I thought mine sounds pretty much the same, you would maybe have to add a little more functionality (only update which is basically check if new version is out-->remove mod-->install mod)
It already syncs modlists from a server.
The only bug about is that I didn't suceed in porting it to MS Windows, but that is just because I don't have a lot of experience with Windows.

My idea for 3m is slightly different. Although I really appreciate your work (especially in-game mod manager), my mod manager won' t have a "central repo" with zip files which will be maintained by me, but instead there is meant to be a single modlist file, which will point on the modinfo files managed by the mod developers, which are meant to point on the ORIGINAL repositories. Then, although the mod contributors will have to host and manage the modinfo files themselves, the main manager will depend only on one modlist file with a list of modinfo files and the updates could be done straightly by mod contributors. The main program is meant to gather information from all modinfo files (but not with every run, only by the "sync" option) and writing it to a "local modlist" and then compare versions and use the mods from this local modlist and local repo info file.

It will be also a chance for me to learn more. For most of my projects this is a main goal yet.

Phitherek_ wrote:[...]Although I really appreciate your work (especially in-game mod manager), [...]

The Python Mod Manager was written by me

Phitherek_ wrote:My idea for 3m is slightly different. Although I really appreciate your work (especially in-game mod manager), my mod manager won' t have a "central repo" with zip files which will be maintained by me, but instead there is meant to be a single modlist file, which will point on the modinfo files managed by the mod developers, which are meant to point on the ORIGINAL repositories. Then, although the mod contributors will have to host and manage the modinfo files themselves, the main manager will depend only on one modlist file with a list of modinfo files and the updates could be done straightly by mod contributors. The main program is meant to gather information from all modinfo files (but not with every run, only by the "sync" option) and writing it to a "local modlist" and then compare versions and use the mods from this local modlist and local repo info file.

It will be also a chance for me to learn more. For most of my projects this is a main goal yet.

See my mod database.

http://tinyurl.com/mtmdfrm

It could work as your "modlist".

You can use the existing APIs or tell me the format for a new one.

For now, the idea for the format of the modlist is:
- regular text file, filename "modlist.3mml" - and this file is to be maintained by me on the official server and maybe by others (I think I will make it configurable). The format is for now like this:
{mod_name}
[server]
address_of_the_server_with_modinfo
[modinfo]
/path/to/modinfo/on/the/server
{end}
for every mod. I will tell you the final version when it will be working ;).

And also I will have a look on your APIs. I would love to make 3m compatible with your DB ;).

EDIT: I see that your APIs are for Python and Lua, but I will code the whole thing in C++, so... I don' t know how I could use your APIs in that, sadly :(.

EDIT 2: The modlist is finally in the format that I described in this post, there is a parser that checks format for this - branch master, directory filetesters, file modlistparser.cpp.

Current status of the project:
I wrote all the most important parsers (for configuration, remote modlist list, remote modlist and modinfo files), they are all included in the main 3m file and also as separate file testers in the "filetesters" subdirectory of main repo. This means that I have defined format for all the files created by users, mod developers and modlist hosters. This also means that I finally got the external data into the program and this is in my opinion one of most important things to do for a start ;).

Next thing: the -S option - connecting to the modlist and modinfo files and gathering information.

You can use any language with sockets (internet download) capability to access the api's.

It is called "python" and "ingame" because it is sent in the format those programs/mods require, which you can easily use as well.

All it is is a file on my server that you download.

eg: api/python/getmodbyname.php?id=mesecons

(Actual Link)

will return this:

This is really great! However, for this format I would need a separate parser (when I already have 4 written and 2 more in plans), which will be harder to implement for me than the parsers that I have now. Also, there are two problems with this format, because 3m will need more information. Firstly, 3m will natively support not only archive files, but also git repositories, so there need to be a repository type information (already defined in my modinfo format). Also, 3m is not meant to analyze the versions as they are - there is something that I call "package release" instead, and it is a regular int number that tells 3m if there was another package release for it to upgrade (and it is release 1, 2, 3 etc.). I can also add a information-only and non-functional "version" field, how about that?
One thing for me: DEPENDENCIES! How could I forget to define dependencies in modinfo?! I am going to add it as soon as possible.
EDIT: So if you could make another, separate version of getmodbyname script that will generate output in modinfo format, I would be really grateful (eg. api/3m/getmodbyname.php). For testing you can use the modinfoparser.cpp from 3m repo (I will update it with dependencies soon). If not, you can send me the code and I will try to modify it to suit my modinfo format (I know PHP quite good, just look at my GitHub to know, that I had been playing with it a little ;)).
EDIT 2: Dependencies added ;). Now I need your opinions about the "version" field idea.

I look in ideas, and saw three format types

modlist.3mml
modlist.3mmi
modlist.3mlm
modlist.3mri

Can you give me an example?

At a guess, i would say the format is like this for getmods.php:



And then a modinfo for each (getmodbyname.php?id=name):

There are 6 files to be used by 3m: config - to store configuration, remote_modlists - to configure the modlists (one or many), modlist.3mml (the remote modlist), modlist.3mlm (the local modlist), modinfo.3mmi (remote modinfo, for every mod), and repoinfo.3mri (information of local repository). The 2 of them are for your interest: modlist.3mml and modinfo.3mmi. The names and extensions are just my invention, what is really important is what' s inside the file. And here are the formats:

For remote modlist (modlist.3mml) you got it right. Format for modinfo (modinfo.3mmi) slightly changed though:



So, the real example on which am I going to test 3m looks like this:
modlist.3mml:

This is only one test mod, which modinfo looks like this:
modinfo.3mmi:


EDIT: The parsers from the filetesters directory are already updated to these formats, you can use them for testing.

I have done it.

(modlist.3mml) api/3m/getmods.php
(modinfo.3mmi) api/3m/getmodbyname.php?id=mod_name

Please check over it for me, and test it.

At the moment release = version.

Question:
If the user changes the description is the release number increased? or is it only increased when the version is changed or the download changed? (like a linear (integer) version of the version number)

1. I would rather have a modname like in the Minetest' s mods folder (i mean e.g. mesecons, moreores, mobf etc.), and real mod name and description should be in the description field. In othere words, my modname is your ID ;). And it is for both modlist and modinfo. It will be also better for your scripts, because "minetest/forum/api/3m/getmodbyname?id=Minecraft like item drop/pick up" would be rather problematic URL... Also, my parsers are quite strict ;).
2. The default dependency should also be listed in modinfo. Or "none" if it really depends on nothing.
3. The repotypes. Since mesecons is on a Github, the best solution would be setting repotype to git and a public clone URL as its address.
4. And I really need those releases. And release changes ONLY when content of the mod changes.

Project status: SYNC IS WORKING! I would be grateful if someone else would test it. Here is a quick guide to configuration (currently only for Linux):
1. In your home directory create a .3m directory.
2. In ~/.3m create two files: config and remote_modlists
3. In config write your settings in the following format:

Note: The files above don' t have to exist, but their directories must be writeable.
4. In remote_modlists write your remote modlist list in the following format:

5. Compile 3m - "make" or "make 3m"
6. run ./3m -S
7. If everything is set up correctly, you should find a generated local modlist.3mlm in the directory, that you choosed.
File examples:
config:


remote_modlists:


Rubenwardy, you can now test your API yourself, just see if and how is the local modlist generated ;). And thank you for collaboration, because with your DB is one more, easy to maintain repository and the mod developers are more likely to register there than create modinfos for 3m themselves ;).

Rubenwardy: I need your help, because your server gives me unexpected response... It may be your hosting. See, when I give 3m following data in remote_modlists:

and then run sync, it outputs me a HTTP 302 on your server:

Do you know how to handle this?

EDIT: From Wikipedia: The HTTP response status code 302 Found is a common way of performing a redirection. - I need a direct address...

EDIT 2: Worse, it' s 404


EDIT 3: It' s all right, one admin from my hosting (cba.pl) gave me the correct HTTP request clue for this kind of hostings and now it works with everything after last update (yes, it connects with rubenwardy' s API for 3m!).

(to clear it out: 3m downloads the whole modlist, not only this part).

Phitherek_ wrote:Rubenwardy: I need your help, because your server gives me unexpected response... It may be your hosting. See, when I give 3m following data in remote_modlists:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

{rubenwardy}
[server]
multa.bugs3.com
[path]
/minetest/forum/api/3m/getmods.php
{end}


file...
(to clear it out: 3m downloads the whole modlist, not only this part).

Must be cause the links are wrong, and it goes to multa.bugs3.com//minetest/forum/api/3m/getmodbyname.php, and so a 404 error is activated which redirects the page to multa.bugs3.com/error.php?id=404

Ok, on 3m' s side all bugs has been resolved so far (these were mostly incomplete requests and uninitialized strings and so on).

Rubenwardy' s Mod Forum / Browser compatibility notice:
For your mod to work, do NOT follow rubenwardy' s guidelines in description, but rather just add a simple one-line description. In other case, 3m parser will not accept the format and your mod will not get into the local modlist and as a result it won' t be on 3m' s "database"!

Rubenwardy: Sorry for this notice to ignore your guidelines, but that is true. 3m requires short one-line description instead of the HTML or BB or WHATEVERcode (I created e. g. SMPBNS Code that looks identically for my SMPBNS project). Keep in mind that it is a commandline program and it won' t parse the code in description. Also - because of modinfo format, the [] are PART OF THIS FORMAT and obviously are not allowed elsewhere. And one more thing - 3m reads only one line of the description, descriptions with more lines are treated as not in the right format (because they are). From your database only my TestModPh and the item_drop were parsed successfully. The other ones - failed. I recommend separating the "Description" and the (proposed by me) new field "Overwiew". Description should be short, one-lined. Overwiew will be in this code of yours and can be displayed as HTML.

EDIT: Latest -S (sync) run with remote modlists: local - my local modlist, official - 3m official modlist and rubenwardy - Rubenwardy' s Mod Forum / Browser:

3m console output:


Generated local modlist.3mlm:

Phitherek_ wrote:Ok, on 3m' s side all bugs has been resolved so far (these were mostly incomplete requests and uninitialized strings and so on).

Rubenwardy' s Mod Forum / Browser compatibility notice:
For your mod to work, do NOT follow rubenwardy' s guidelines in description, but rather just add a simple one-line description. In other case, 3m parser will not accept the format and your mod will not get into the local modlist and as a result it won' t be on 3m' s "database"!

Rubenwardy: Sorry for this notice to ignore your guidelines, but that is true. 3m requires short one-line description instead of the HTML or BB or WHATEVERcode (I created e. g. SMPBNS Code that looks identically for my SMPBNS project). Keep in mind that it is a commandline program and it won' t parse the code in description. Also - because of modinfo format, the [] are PART OF THIS FORMAT and obviously are not allowed elsewhere. And one more thing - 3m reads only one line of the description, descriptions with more lines are treated as not in the right format (because they are). From your database only my TestModPh and the item_drop were parsed successfully. The other ones - failed. I recommend separating the "Description" and the (proposed by me) new field "Overwiew". Description should be short, one-lined. Overwiew will be in this code of yours and can be displayed as HTML.

EDIT: Latest -S (sync) run with remote modlists: local - my local modlist, official - 3m official modlist and rubenwardy - Rubenwardy' s Mod Forum / Browser:

3m console output:

I am going to add a new entry called "overview" and it will not support BBCode. So you can use it.

I need just what you call a "Simple description" in your guidelines to be one-line and in another field, that can be used in API. Then I could finally make the API fully compatible with 3m.

EDIT: After adding the field, I completed overview for all mods and finally got a full modlist in 3m. Here are the last -S results:
Console output:


Generated local modlist:


Sync works perfectly. Time to get to Query ;).

Quite big problem arose recently. I' ve made the local modlist parser, but while I am trying to parse my local modlist I am getting SEGFAULT just out-of-nowhere. Can the amount of data that I am trying to store in program' s memory be the case? If so, I must think of another solution (reading and analyzing single entries from files instead of loading the whole data to the memory).

EDIT: Tried differrent solutions but everytime an error arises - sometimes it' s SEGFAULT, sometimes it' s SIGABRT with core dump... There is something wrong with a structure I am using to collect data and I REALLY don' t know what, as the structure is similar to the one I used before and it worked. The further development is in question because of that, until I will find something that resolves these problems.

EDIT 2: Bug found, problem resolved.

The Query listing is here! Now I need to add searching. Here is the last Query run console output:

The Query action is now finished! Next... I think I must finally get to installing ;).
Here is a sample query for "Minetest":


Enjoy!

Just ran the first successful -I for the test mod entry! WARNING: It is still not deeply tested, mostly the dependencies need more testing.
Here is the output:

Ok, so now I have working -I option for "archive" repositories with recursive dependency checking!
Output below:


And here is the generated repoinfo file:


Now I will need some testers... ;)

Finally, Git repotype also works with -I now! This means the install option is practically finished, now it just needs some testing, and in that field I count on you, Minetest community ;).
Here is the output of the successful -I of the mod from Git repository:

IMPORTANT NOTICE: For now, 3m' s "archive" repotype handles only DIRECT FILE URLS! So GitHub zipball or MediaFire URL WILL NOT work. I will work on that in further versions.

EDIT: Solution for now - for Github use "Git" repotype, for MediaFire use "URL with Filename".

Are you still working on this?

*ping*

Are you still working on this?

Ping!

Yeah, I must admit, that the project was dead since months, but you must understand me, as I have quite demanding IT studies on my head and I do not have much time. I was also waiting until I finish C++ in object oriented way subject. My plans were to rewrite it in more OOP way, so the code would be cleaner and easier to understand etc. I think I can come back to working on 3m in a while, especially now, when I created the NetSocket++ library, which will make work easier for me and anyone that wants to work on it. So - was it dead? Yeah. Will it? I don' t think so. I am planning to get to work on it as soon as I have time.

Yeah, when it is remade it would be nice in a class format.

Good look with your studies.