numpy-izing minetestmapper

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

numpy-izing minetestmapper

by spillz » Wed Mar 05, 2014 03:06

Updated Mar 14, 14

What is it? an improved version of the python minetestmapper:

1. overhauled command line arguments
2. control the drawn region (--region, --minheight, and --maxheight options)
3. draw fog to have node further from the view point fade (--fog to control strength, --fogcolor to choose color)
4. draw cross-sections as well as top down view (--facing)
5. draw nodes that are larger than a single pixel (--pixelspernode option)
6. faster: 8 to 10 times faster than the original minetestmapper.py (but still slower than the c++ code)

The script is here: https://github.com/spillz/minetest/blob/master/util/minetestmapper-numpy.py
(repo is here: https://github.com/spillz/minetest)

You must have a recent python and numpy (see www.scipy.org) installed.

Performance

As a test I tried out the original and numpy versions of minetest mapper the on the Feb 25 posting of Stampy's world on Ubuntu laptop:

minetestmapper.py
Time 1m23s
Memory Usage: 52MB

minetestmapper-numpy.py
Time 10s
Memory Usage: 13MB

Examples

Examples using Morn76's Stampy's world:

Cross section (5 pixels per node)

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
python minetestmapper-numpy.py --facing east --bgcolor lightblue --region 48 128 -384 -128 --drawscale --maxheight 64 --minheight -16 --pixelspernode 5 ~/.minetest/worlds/stampyworld stampye.png


Image

Top view (3 pixels per node)

Image

Command line arguments
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
./minetestmapper-numpy.py -h


Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--fogcolor COLOR] [--ugcolor COLOR]
                               [--drawscale] [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--drawunderground-standalone]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               [--pixelspernode PPN]
                               [--facing {up,down,north,south,east,west}]
                               [--fog FOGSTRENGTH]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --fogcolor COLOR      set the color for fog (default grey)
  --ugcolor COLOR       set the color for underground areas (default purple)
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas overlaid on the map
  --drawunderground-standalone
                        draw underground areas as a standalone map
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes, default = -2000 2000 -2000 2000)
  --maxheight YMAX      don't draw above height YMAX (default = 500)
  --minheight YMIN      don't draw below height YMIN (defualt = -500)
  --pixelspernode PPN   number of pixels per node (default = 1)
  --facing {up,down,north,south,east,west}
                        direction to face when drawing (north, south, east or
                        west will draw a cross-section)
  --fog FOGSTRENGTH     use fog strength of FOGSTRENGTH (0.0 by default, max
                        of 1.0)


TODO:
1. Isometric view
2. GUI
3. Water transparency
4. Requests???
Last edited by spillz on Sat Apr 19, 2014 04:39, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 04:41

Updated, numpy-ized the drawing code too, so that now the stampyworld render takes about 10s.
 

paramat
Member
 
Posts: 2662
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat

by paramat » Wed Mar 05, 2014 12:08

Ah awesome i wil try this later, i use the python mapper but find it slow, processing-intensive, and i can't seem to adjust the area it maps, it refuses to go beyond 2000 nodes from world centre, are you able to make it map any chosen area?
I edited the code to this for 4000x4000 maps, but can't go beyond, if i do the map will generate but will not be saved.
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
sector_xmin = -2000 / 16
sector_xmax = 2000 / 16
sector_zmin = -2000 / 16
sector_zmax = 2000 / 16


A suggestion, make the background colour black not white, map colours look so much better against black and the glare is bad for the eyes.
Last edited by paramat on Wed Mar 05, 2014 12:16, edited 1 time in total.
I rely on donations to help provide an income https://forum.minetest.net/viewtopic.php?f=3&t=14935
 

User avatar
rubenwardy
Member
 
Posts: 4500
Joined: Tue Jun 12, 2012 18:11
GitHub: rubenwardy
IRC: rubenwardy
In-game: rubenwardy

by rubenwardy » Wed Mar 05, 2014 12:34

Awesome, almost three times as fast.
 

User avatar
Krock
Member
 
Posts: 3598
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker

by Krock » Wed Mar 05, 2014 12:58

Again a mapper? Please add LevelDB support. Thanks.
spillz wrote:(repo is here: https://github.com/spillz/minetest)

Ah, here we've got some kind of link destroyer --^

Good work anyway!
Last edited by Krock on Wed Mar 05, 2014 13:00, edited 1 time in total.
Newest Win32 builds - Find a mod - All my mods
ALL YOUR DONATION ARE BELONG TO PARAMAT (Please support him and Minetest)
New DuckDuckGo !bang: !mtmod <keyword here>
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 14:52

Krock wrote:Again a mapper? Please add LevelDB support. Thanks.


How is it different from the SQL database? I.e. is the layout identical? I don't know anything about LevelDB but so long as it has a python binding it shouldn't be too hard to add support for.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 14:53

rubenwardy wrote:Awesome, almost three times as fast.


Should be more like 8 to 10 times with latest version but YMMV
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 15:15

paramat wrote:Ah awesome i wil try this later, i use the python mapper but find it slow, processing-intensive, and i can't seem to adjust the area it maps, it refuses to go beyond 2000 nodes from world centre, are you able to make it map any chosen area?
I edited the code to this for 4000x4000 maps, but can't go beyond, if i do the map will generate but will not be saved.
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
sector_xmin = -2000 / 16
sector_xmax = 2000 / 16
sector_zmin = -2000 / 16
sector_zmax = 2000 / 16



Maybe an overflow issue in the hex lookups (not my code), I will take a look.

A suggestion, make the background colour black not white, map colours look so much better against black and the glare is bad for the eyes.


By default? I don't have strong feeling either way, but it should be made easy for anyone to permanently set their preference without the need to set command line args every time.
 

User avatar
sfan5
Member
 
Posts: 3636
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5

by sfan5 » Wed Mar 05, 2014 15:30

Krock wrote:
spillz wrote:(repo is here: https://github.com/spillz/minetest)

Ah, here we've got some kind of link destroyer --^

Good work anyway!

Just remove the ) from the URL
Mods: Mesecons | WorldEdit | Nuke
Minetest builds for Windows (32-bit & 64-bit)
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 17:12

@paramat: the range bug might be a 64 bit issue. No such problem on my 32 bit laptop, but it is present on 64 bit desktop. Can you confirm? (Which is a pain because I do most of my spare time coding on the 32 bit laptop)
 

paramat
Member
 
Posts: 2662
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat

by paramat » Wed Mar 05, 2014 20:03

My system:
Processor Intel® Core™2 CPU T7400 @ 2.16GHz × 2
OS type 32-bit
I rely on donations to help provide an income https://forum.minetest.net/viewtopic.php?f=3&t=14935
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Wed Mar 05, 2014 20:37

That's not the problem then. Hmm...
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Thu Mar 06, 2014 04:31

Latest version cleans up the code a bit (no more globals), changes the command line syntax plus adds new options

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
./minetestmapper-numpy.py --help

usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--drawscale] [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas (NOT IMPLEMENTED!)
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes)
  --maxheight YMAX      don't draw above height YMAX
  --minheight YMIN      don't draw below height YMIN


Next steps... will try to eke slightly better performance out of it, fix rendering glitches, look into levelDB, and think about mapping caverns and cross-sections
Last edited by spillz on Thu Mar 06, 2014 04:42, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Thu Mar 06, 2014 14:14

@paramat: can you try again with latest version using --region switch? thx
Last edited by spillz on Thu Mar 06, 2014 14:23, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sun Mar 09, 2014 06:46

Experimenting with cross-sections. Odd looking... (vertical axis label should be y instead of z)

[img=Cross sectional map image]http://s8.postimg.org/m8vxe91yd/mapx3.png[/img]
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sun Mar 09, 2014 07:05

And here's karsthafen, around the city:

[img=cross section of karsthafen]http://s2.postimg.org/4tpsan0nd/mapkarst.png[/img]
 

4aiman
Member
 
Posts: 1208
Joined: Mon Jul 30, 2012 05:47

by 4aiman » Sun Mar 09, 2014 07:36

Always wanted to ask... Is it possible lo use lua VoxelManipulator to read terrain?
Maybe then it would be possible to make different kind of maps w/o hardcoding it in C?
 

User avatar
Krock
Member
 
Posts: 3598
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker

by Krock » Sun Mar 09, 2014 07:48

@spillz
Image
Or is that link an archivement, when reaching 666 posts?
Since there are 5 posts of you in a row, it should be possible to edit one.

@4aiman
I think the voxelManipulator is for mods only and the direct reading with sqlite3 in the map should be the fastest way.
Last edited by Krock on Sun Mar 09, 2014 07:50, edited 1 time in total.
Newest Win32 builds - Find a mod - All my mods
ALL YOUR DONATION ARE BELONG TO PARAMAT (Please support him and Minetest)
New DuckDuckGo !bang: !mtmod <keyword here>
 

User avatar
sfan5
Member
 
Posts: 3636
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5

by sfan5 » Sun Mar 09, 2014 08:39

Krock wrote:direct reading with sqlite3 in the map should be the fastest way.

You will not be able to support all upcoming map backends when you read directly from the map db.
Mods: Mesecons | WorldEdit | Nuke
Minetest builds for Windows (32-bit & 64-bit)
 

User avatar
Krock
Member
 
Posts: 3598
Joined: Thu Oct 03, 2013 07:48
GitHub: SmallJoker

by Krock » Sun Mar 09, 2014 09:24

sfan5 wrote:
Krock wrote:direct reading with sqlite3 in the map should be the fastest way.

You will not be able to support all upcoming map backends when you read directly from the map db.

How much backends are planned yet?
Newest Win32 builds - Find a mod - All my mods
ALL YOUR DONATION ARE BELONG TO PARAMAT (Please support him and Minetest)
New DuckDuckGo !bang: !mtmod <keyword here>
 

User avatar
sfan5
Member
 
Posts: 3636
Joined: Wed Aug 24, 2011 09:44
GitHub: sfan5
IRC: sfan5

by sfan5 » Sun Mar 09, 2014 10:28

Krock wrote:
sfan5 wrote:
Krock wrote:direct reading with sqlite3 in the map should be the fastest way.

You will not be able to support all upcoming map backends when you read directly from the map db.

How much backends are planned yet?

Planned, none (apart maybe adding sophia db.
But everyone could add a backend (open source yey!), I e.g. added a backend which used lua to do thing (you could use lua to connect to mysql for example).
Mods: Mesecons | WorldEdit | Nuke
Minetest builds for Windows (32-bit & 64-bit)
 

paramat
Member
 
Posts: 2662
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat

by paramat » Sun Mar 09, 2014 17:21

spillz wrote:@paramat: can you try again with latest version using --region switch? thx

Yep i will, i just realised i have to install numpy so that may have been the problem, i love the cross-section feature.
I rely on donations to help provide an income https://forum.minetest.net/viewtopic.php?f=3&t=14935
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sun Mar 09, 2014 19:27

paramat wrote:
spillz wrote:@paramat: can you try again with latest version using --region switch? thx

Yep i will, i just realised i have to install numpy so that may have been the problem, i love the cross-section feature.


Cross-sec still buggy so not committed yet. Will try to fix it soon.

@krock: I will try to contain myself (maybe my last thread post was unnecessary, but I have always thought it legitimate to post updates as replies because the edit button doesn't show up as new activity and is easily missed). But also feel free not to read this thread ;)
Last edited by spillz on Sun Mar 09, 2014 20:18, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Thu Mar 13, 2014 22:36

I pushed latest changes to github, with new command line options "--facing", "--fog" and "--fogcolor"

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
python minetestmapper-numpy.py --help


Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
usage: minetestmapper-numpy.py [-h] [--bgcolor COLOR] [--scalecolor COLOR]
                               [--origincolor COLOR] [--playercolor COLOR]
                               [--fogcolor COLOR] [--drawscale]
                               [--drawplayers] [--draworigin]
                               [--drawunderground]
                               [--region XMIN XMAX ZMIN ZMAX]
                               [--maxheight YMAX] [--minheight YMIN]
                               [--facing {up,down,north,south,east,west}]
                               [--fog FOGSTRENGTH]
                               world_dir [output]

A mapper for minetest

positional arguments:
  world_dir             the path to the world you want to map
  output                the output filename

optional arguments:
  -h, --help            show this help message and exit
  --bgcolor COLOR       set the background color (e.g. white or "#FFFFFF")
  --scalecolor COLOR    set the ruler and text color for the scale
  --origincolor COLOR   set the color for the map origin
  --playercolor COLOR   set the color for player markers
  --fogcolor COLOR      set the color for fog
  --drawscale           draw a scale on the border of the map
  --drawplayers         draw markers for players
  --draworigin          draw the position of the origin (0,0)
  --drawunderground     draw underground areas (NOT IMPLEMENTED!)
  --region XMIN XMAX ZMIN ZMAX
                        set the bounding x,z coordinates for the map (units
                        are nodes)
  --maxheight YMAX      don't draw above height YMAX
  --minheight YMIN      don't draw below height YMIN
  --facing {up,down,north,south,east,west}
                        direction to face when drawing (north, south, east or
                        west will draw a cross-section)
  --fog FOGSTRENGTH     use fog strength of FOGSTRENGTH (0.0 by default, max
                        of 1.0)



Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
python minetestmapper-numpy.py --region 0 660 300 600 --fogcolor lightblue --bgcolor lightblue --minheight 0 --maxheight 700 --drawplayers  --draworigin --facing south --fog 0.3 --drawscale karsthafen map.png


Sample output (karthafen at z position 600, facing south):

[img=cross section of karsthafen]http://i.imgur.com/NtJS7jO.png[/img]

Still has a few bugs and thinking about other features. e.g. gui, isometric view, drawing caves and transparent glass/water

PS: You need to install numpy for this to work (www.scipy.org)
Last edited by spillz on Thu Mar 13, 2014 22:58, edited 1 time in total.
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Fri Mar 14, 2014 03:54

(Last unsolicited post from me for a while, krock, promise)

A few bug fixes and here's one more option --pixelspernode

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
python minetestmapper-numpy.py --drawscale --facing east --bgcolor lightblue --region 48 128 -384 -128 --maxheight 64 --minheight -16 --pixelspernode 5 ~/.minetest/worlds/stampyworld stampye2.png


Each node takes the specified number of pixels but the edge shading operates at the 1 pixel level:

Stampy's world. Cross Section (5 ppn)... grr... bug with "sky" looking like it is in front of the nodes:

Image

Stampy's world. Topo view (3 ppn):

Image
Last edited by spillz on Fri Mar 14, 2014 04:12, edited 1 time in total.
 

jenova99sephiros
Member
 
Posts: 158
Joined: Sat Aug 03, 2013 15:16
In-game: Jenova

by jenova99sephiros » Fri Mar 14, 2014 06:39

Please red ver!
I english google translator
 

paramat
Member
 
Posts: 2662
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat

by paramat » Fri Mar 14, 2014 18:37

Amazing work! pixels per node will be very useful, look forward to using this once i install numpy.
I rely on donations to help provide an income https://forum.minetest.net/viewtopic.php?f=3&t=14935
 

spillz
Member
 
Posts: 138
Joined: Thu Feb 13, 2014 05:11

by spillz » Sat Mar 15, 2014 01:50

paramat wrote:Amazing work! pixels per node will be very useful, look forward to using this once i install numpy.


Cool! I encourage you to post images... Bug reports and feature requests welcome too.

I guess python is the ugly step child here. Too bad for me that you guys chose lua over python.
 

paramat
Member
 
Posts: 2662
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat

by paramat » Sun Mar 23, 2014 04:09

Image


Mandelbox mapgen plus paragenv7 mod biomes.
Finally got this working, it's amazing, really like the sideview and fog options, thanks for this.
I rely on donations to help provide an income https://forum.minetest.net/viewtopic.php?f=3&t=14935
 

paramat
Member
 
Posts: 2662
Joined: Sun Oct 28, 2012 00:05
GitHub: paramat

by paramat » Sun Mar 23, 2014 06:30

Image


Mountain found in paragen mod.
I rely on donations to help provide an income https://forum.minetest.net/viewtopic.php?f=3&t=14935
 

Next

Return to Minetest-Related

Who is online

Users browsing this forum: No registered users and 23 guests

cron