numpy-izing minetestmapper

[spillz]

Paramat: very cool. I guess some of the blocks/chunks haven't been generated, which creates some of the discontinuity. Really like the look of the second one. Your different trees and biomes show up really nicely.

[paramat]

Yes those were sideviews of worlds instead of cross sections. Here's a central cross seciton of a flexrealm mod dyson sphere 2600x1200px, the rings are interference patterns in the clouds.

[spillz]

Yes those were sideviews of worlds instead of cross sections. Here's a central cross seciton of a flexrealm mod dyson sphere 2600x1200px, the rings are interference patterns in the clouds.

Wow... 8D

(And I had no idea that clouds get drawn -- or is this a feature of you mod?)

[spillz]

Latest update to the mapper. Let's go caving...

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

 python minetestmapper-numpy.py worlds/stampyworld/ stampyugb.png --drawunderground --minheight -32


Had no idea caves were so numerous. (Though maybe just bugs in my code?)

[paramat]

^ Good feature.

(And I had no idea that clouds get drawn -- or is this a feature of you mod?)

Yeah those are stationary cloud nodes added by my mapgen.

[thetoon]

Am I the only one having trouble on 64bits python?

[spillz]

Am I the only one having trouble on 64bits python?

What's the problem?

EDIT: If using python3, try pulling the latest version from my github repo. (Left in a python2 print statement)

[thetoon]

What's the problem?

EDIT: If using python3, try pulling the latest version from my github repo. (Left in a python2 print statement)

I'm using Python 2.7, so it shouldn't be a python3 problem.

Still, when running the script, I got hundreds of the following statement :

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

Traceback (most recent call last):
  File "./minetestmapper-numpy.py", line 707, in generate_map_info
    plist = map_block(mapdata, version, ypos, maxy, plist, cdata, hdata, dnddata, day_night_differs, id_map, ignore, air, face_swap_order)
  File "./minetestmapper-numpy.py", line 343, in map_block
    cdata[po] = content[opaques]
TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
Error at (-72,-1,-47): Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'
Traceback (most recent call last):
  File "./minetestmapper-numpy.py", line 707, in generate_map_info
    plist = map_block(mapdata, version, ypos, maxy, plist, cdata, hdata, dnddata, day_night_differs, id_map, ignore, air, face_swap_order)
  File "./minetestmapper-numpy.py", line 343, in map_block
    cdata[po] = content[opaques]
TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'

And the generated image contains nothing but blue sky (or whatever blue means).

[spillz]

Still, when running the script, I got hundreds of the following statement :

Try pulling the latest version and testing again. I suspect it won't fix, but you never know...

What OS are you using? What version of minetest was the map made with? Have you tried on a different world? Also, try using the default colors.txt if you haven't already.

[thetoon]

Holy smokes, Batman! It works. Just pulling the latest version, and it worked. Out of the box. No need to change colors.txt or anything.

Congrats on the quickest fix ever :)

[paramat]

spillz, the cross section images facing north, south etc. work but have been reversed left to right, try comparing a cross section with an overhead view and you'll see.

[spillz]

spillz, the cross section images facing north, south etc. work but have been reversed left to right, try comparing a cross section with an overhead view and you'll see.

I thought I had fixed those issues. But thanks, I will look into it.

[spillz]

Try the latest on my github repo (see here). It looks like north/south were reversed but not east/west, which I don't really understand. (Suspect there's another bug somewhere else)

[paramat]

Seriously, this mapper is the best thing to happen to Minetest since voxmynips.

EDIT tested and works, thanks.

[spillz]

Update: Added LevelDB support

[Krock]

Update: Added LevelDB support

Yay!


minetestmapper-numpy.py sais:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

Traceback (most recent call last):
  File "E:\Programme\minetest\util\minetestmapper-numpy.py", line 30, in <module
>
    import cStringIO
ImportError: No module named 'cStringIO'

please use

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

import io

cStringIO.StringIO replace with io.StringIO

Edit: Any working py-leveldb out there for python 3.3?

[spillz]

Thanks re stringio.

This is probably as good as it gets but you will have to point it at a levedb dll and i would have to add support because the API is different: https://code.google.com/p/leveldb-py/

[dgm5555]

Using the code from the repo I get the following fatal error for a variety of commandlines :-(:
(running Ubuntu 14.04 python 2.7)

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

~/.minetest/mods/minetest/util$ python minetestmapper-numpy.py ~/.minetest/worlds/stampyworld/ stampyugb.png --drawunderground --minheight -32
Result image (w=752 h=832) will be written to stampyugb.png
Error at (-23,3,-23): initial_value must be unicode or None, not buffer
Traceback (most recent call last):
  File "minetestmapper-numpy.py", line 619, in generate_map_info
    f = db.get(ps)
  File "minetestmapper-numpy.py", line 429, in get
    return io.StringIO(r[0])
TypeError: initial_value must be unicode or None, not buffer
Block data: Traceback (most recent call last):
  File "minetestmapper-numpy.py", line 1005, in <module>
    main()
  File "minetestmapper-numpy.py", line 1000, in main
    world.generate_map_info(str_to_uid)
  File "minetestmapper-numpy.py", line 781, in generate_map_info
    for c in r[0]:
NameError: global name 'r' is not defined

As a wild shot, I also tried python3 (after installing python3-numpy and python3-pil, but also errored with:-

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

~/.minetest/mods/minetest/util$ python3 minetestmapper-numpy.py ~/.minetest/worlds/stampyworld/ stampyugb.png --drawunderground --minheight -32
Traceback (most recent call last):
  File "minetestmapper-numpy.py", line 1005, in <module>
    main()
  File "minetestmapper-numpy.py", line 987, in main
    uid_to_color, str_to_uid = load_colors()
  File "minetestmapper-numpy.py", line 198, in load_colors
    f = file("colors.txt")
NameError: name 'file' is not defined


Using minetestmapper.py instead of minetestmapper-numpy.py produces a png though:
python minetestmapper.py -i ~/.minetest/worlds/stampyworld/ stampyugb.png --drawunderground --minheight -32

[dgm5555]

BTW, is there any way of creating a mod which displays the pregenerated png as a map with player location (or even without would be cool).
Also wondering how difficult it would be to use the script to produce a 3D model which could be output to blender?

[spillz]

Try changing all occurrences of StringIO to BytesIO and file to open.

Re pre generated map I am not sure. There are already in game mappers.

Re blender, probably not hard just need something to handle blender output

[spillz]

Try the latest version. Should work with both python 2 and 3. Thanks for reporting the bugs.

[dgm5555]

I made a quick mod for displaying a minetestmapper png in-game. I've put it as a new post to avoid hijacking the numpy post, but it relies on your minetestmapper script, so thanks. https://forum.minetest.net/viewtopic.php?f=9&t=9219

[dgm5555]

Is there any way of mapping the generated png to minetest world coordinates - ideally from within lua, but if not from the script. This latter could perhaps be acheived having an argument to modify the output name to include location of topL and bottomR coordinates in the world (if that's accessible), and perhaps automatically save the world data in the Comment field of the metadata within the png so it's always available even with name changes. I'm trying to figure out how to place a player location icon correctly on the map while playing the game (for my mapit tool). Cheers

[dgm5555]

I've modified the .py slightly to generate a couple of tEXt Textual data chucks to contain the desired output (I think, I'm not entirely happy with my understanding of how the regions are process, but have made a best guess). Given that I'm not a programmer, it needs some tidying up. I haven't forked your github, as it seemed pointless creating a 40MB fork for a 40kb file. The full edited version (with more debugging code) is on my repo: https://github.com/dgm3333/mapit.
The following code replaces the im.save just before main

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

    # region is incorrect if world limits are exceeded. I attempted to correct this, but it didn't work
    # as this code generates incorrect values if the requested region is greater than the world dimensions
    #pngminx = int(args.region[0]/16)*16 if args.region[0] < minx*16 else minx*16
    #pngmaxx = int((args.region[1]+16)/16)*16 if args.region[1] > maxx*16 else maxx*16
    #pngminz = int(args.region[2]/16)*16 if args.region[2] < minz*16 else minz*16
    #pngmaxz = int((args.region[3]+16)/16)*16 if args.region[3] > maxz*16 else maxz*16

    # worldlimits are measured in cubes of 16x16x16
    pngminx = minx*16
    pngmaxx = maxx*16
    pngminz = minz*16
    pngmaxz = maxz*16
    pngregion=[pngminx, pngmaxx, pngminz, pngmaxz]

    print("Saving to: "+ args.output)
    print("PNG Region: ", pngregion)
    print("Pixels PerNode: ", args.pixelspernode)
    print("border: ", border)

    # This saves data in tEXt chunks (non-standard naming tags are allowed according to the PNG specification)
    im.info["pngRegion"] = str(pngregion[0])+ ","+ str(pngregion[1])+ ","+ str(pngregion[2])+ ","+ str(pngregion[3])
    im.info["pngMinX"] = str(pngminx)
    im.info["pngMaxZ"] = str(pngmaxz)
    im.info["border"] = str(border)
    im.info["pixPerNode"] = str(args.pixelspernode)
    pngsave(im, args.output)

    # Now create a square 'thumbnail' for display on square faces (which turns out to benefit from quite high resolution).
    thumbSize = 512
    imSize = im.size
    if imSize[0] > imSize[1]:
      reSize=(thumbSize, int(thumbSize*(int(imSize[1])/imSize[0])))
    else:
      reSize=(int(thumbSize*(float(imSize[0])/imSize[1])), thumbSize)

    thumbBorder=((thumbSize-reSize[0])/2, (thumbSize-reSize[1])/2, thumbSize-(thumbSize-reSize[0])/2, thumbSize-(thumbSize-reSize[1])/2)
    thumbIm = Image.new("RGB", (thumbSize,thumbSize), args.bgcolor)
    thumbIm.paste(im.resize(reSize),thumbBorder)
    thumbIm.save(args.output.replace(".png", "_thumb.png"), "PNG")

#                                                                                                                                     
# wrapper around PIL 1.1.6 Image.save to preserve PNG metadata
#
# public domain, Nick Galbreath                                                                                                       
# http://blog.client9.com/2007/08/28/python-pil-and-png-metadata-take-2.html                                                                 
#                                                                                                                                       
def pngsave(im, file):
    # these can be automatically added to Image.info dict                                                                             
    # they are not user-added metadata
    reserved = ('interlace', 'gamma', 'dpi', 'transparency', 'aspect')

    # undocumented class
    from PIL import PngImagePlugin
    meta = PngImagePlugin.PngInfo()

    # copy metadata into new object
    for k,v in im.info.iteritems():
        if k in reserved: continue
        meta.add_text(k, v, 0)

    # and save
    im.save(file, "PNG", pnginfo=meta)


[spillz]

@dgm5555: that looks useful, so I will probably commit it in the next couple of days. thanks!

If you want more mapper speed you should try the CPP one (it also has the advantage of having fewer dependencies once compiled). But it's a PITA to modify... which is why I'll stick with python myself. There are probably a few things that could be done to eke a bit more speed out of the python version but bear in mind the some features are slow (e.g. caves).

[dgm5555]

@spillz: I may have overthought the calculation of world boundaries, so I've returned to the simple version (edited code above), but comments in PNG are working fine. Speed isn't a great issue for me, and my CPP is quite rusty, so I'll pass on that offer. Worst case I might have to make a cup of tea while waiting for map generation.

BTW: I'm wondering about adding an option to generate a thumbnail with particular dimensions and fill the edges with background. I've made a "map block", but unless the world has been explored in a square, the aspect ratio of the displayed map is off. If I'm right the following would work to enable saving of a 'thumbnail' (but I'm not on the right computer to try it at the moment):-

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

size = 128, 128
im.thumbnail(size, Image.ANTIALIAS)
im.save(args.output+ ".thumbnail", "PNG")


EDIT: This doesn't work, the working code has been edited into the snippet above

I'd also vote for water *semi* transparency if you're doing it, and just realised I've possibly done your ToDo #2. GUI. Have you got any other suggestions for it?

[spillz]

@dgm555: I added a slightly modified version of your patch (added a command line option for the thumb, and replaced an int() with a float())

https://github.com/spillz/minetest/comm ... 2322e6e5df

[spillz]

I'd also vote for water *semi* transparency if you're doing it, and just realised I've possibly done your ToDo #2. GUI. Have you got any other suggestions for it?

The semi-transparent water is straightforward enough, just more ugly code like the underground stuff. One thing I'm thinking about is that it should be possible to define various blocks as transparent in colors.txt and apply that color with a specified alpha level to blocks beneath. I'll add this feature sometime this week, probably.

Re GUI, the main thing I had in mind was a small app that draws the top view map, then makes it easy to select cross sections to view. Not sure that's all that useful (or easy to implement) for your in-game map.

[dgm5555]

Re GUI, the main thing I had in mind was a small app that draws the top view map, then makes it easy to select cross sections to view. Not sure that's all that useful (or easy to implement) for your in-game map.

In any case, I don't use the cross-section views, but as there seems to be no way of hooking the mouse clicks from lua, I got around it by making a square of 10x10 transparent buttons over the map, but I think (haven't tried it yet) that there is no way of forcing a reload of a texture in minetest (if there was, I would occasionally call the function in the background to update the map).
In any case a frontend gui should be pretty easy with tkinter if you wanted one.

[dgm5555]

One thing I'm thinking about is that it should be possible to define various blocks as transparent in colors.txt and apply that color with a specified alpha level to blocks beneath.

Just a thought. If you do this, could you keep the list separate from the main colors.txt data (eg a separate block at the beginning of the file) - it would be nice to be able to copy and paste it to an updated colors.txt in the future (when/if it gets updated), and if the data is distributed it will be more difficult to extract...
Actually if you were using this system, the same could be done with leaves and glass too, which could be interesting, though probably too small a scale to make much difference.