Real world terrain (mapgen idea)

[Novacain]

I was just kinda thinking about the current mapgen, and the issues associated with the current default mapgen, and just kinda wondered if we could have the mapgen based off real locations. My thoughts are these: topographical maps exist with the hights already in them. Would it be too hard to have some code to turn that map into real terrain? it could have a co-ordinate set point with longitudes and latitudes. example: 38.9536 and 120.0939 with an altitude correction of 1,890 meters to generate 0,0,0. just throwing this idea out there, hoping to inspire someone who works on mapgen.

[prestidigitator]

Not difficult at all. However, it might be worth considering scale. Minetest's coordinates span from like -32k to +32k in each dimension I believe, which if you consider a scale of 1m per node would only give you a 64km square area to play with. Unless you don't use a 1:1 scale, of course.

It would be interesting to also use real geological data if it could be found to create the underground strata and ore deposits (probably with a decent amount of fudging).

[pandaro]

I was just kinda thinking about the current mapgen, and the issues associated with the current default mapgen, and just kinda wondered if we could have the mapgen based off real locations. My thoughts are these: topographical maps exist with the hights already in them. Would it be too hard to have some code to turn that map into real terrain? it could have a co-ordinate set point with longitudes and latitudes. example: 38.9536 and 120.0939 with an altitude correction of 1,890 meters to generate 0,0,0. just throwing this idea out there, hoping to inspire someone who works on mapgen.

I like this idea, I would like to see it implemented

[twoelk]

as has been done for mc
here using OrdnanceSurvey OpenData?

Check this developing bitmap mapgen mod that might use hightmaps
and the older Image Loader by Kaeza that Mauvebic has used to import map outlines.

[Novacain]

Not difficult at all. However, it might be worth considering scale. Minetest's coordinates span from like -32k to +32k in each dimension I believe, which if you consider a scale of 1m per node would only give you a 64km square area to play with. Unless you don't use a 1:1 scale, of course.

It would be interesting to also use real geological data if it could be found to create the underground strata and ore deposits (probably with a decent amount of fudging).

Scale is a good point. real world proportions tend to be a little... large for a voxel sandbox game like minetest. so maybe account for a 3:1 scale? wouldn't want to remove too much :P

The code could also probably be used to create life size if someone happened to want to make a Mt. Everest map to climb it (hey, that could be a game!!).

as has been done for mc
here using OrdnanceSurvey OpenData?

Check this developing bitmap mapgen mod that might use hightmaps
and the older Image Loader by Kaeza that Mauvebic has used to import map outlines.

that is similar, but it looks like it lacks the ability to make an actual playable world. the size on it is such that it seems kind of pointless.

[rubenwardy]

I started writing a map generator like this for Minetest about a month ago, but the api does not support the connecting to the internet.

It used data from maps.google.com - they have a nice API.

[Novacain]

I started writing a map generator like this for Minetest about a month ago, but the api does not support the connecting to the internet.

It used data from maps.google.com - they have a nice API.

would it be possible to download the map to solve the connection issue? I really wish I knew more about programming so I could contribute.

[twoelk]

maps.google.com sends elevation data?

which api?

OpenStreetMap has free data

and scale is indeed a problem as in you have to decide what you want or rather you have to nail down the details of a nice foggy idea.

[BrandonReese]

I think you would need to write a separate program/script to download all of your data from whatever source you choose and parse it into a usable format (maybe a sqlite database?), and then feed that info to the mapgen.

[philipbenr]

Just throwing this idea out there, hoping to inspire someone who works on mapgen.

Talk to paramat, the magpen king. ;) After you figure out exactly what you have in mind, tell him, and if he likes it, he probably could whip out something awesome.

[Novacain]

Just throwing this idea out there, hoping to inspire someone who works on mapgen.

Talk to paramat, the magpen king. ;) After you figure out exactly what you have in mind, tell him, and if he likes it, he probably could whip out something awesome.

Yes, certainly the "magpen" king :P. And I did talk to him. using real-world data isn't very interesting to him. I have the feeling it's not complicated enough for him (based on his previous amazing projects).

[bobomb]

There is a format of elevation data called DEM which is already in a raster form. Publicly available DEMs are usually 10m or 30m resolution with elevation resolution to 1m. However, translating one pixel of DEM to one minetest voxel might be a simple way to scale the real world data.
There are also freely available land cover data sets in raster form, which could be used to create the land cover and tree distribution. digital scans of topo maps might also be used for this, but correlating the spatial reference system of one dataset to another can be tricky.
As for underground strata, there are datasets for this too, but probably is overkill for a fun game. However, there aren't (m)any mapgens with "strata" in minetest, the focus seems to be more about caves, etc. exposed rock faces with strata visible would be cool.

But I have been interested in creating a mod or simply a tool to load DEM data into .mts format for some time now. Freely available elevation data may be available from google or OSM but it would require further processing since it is in vector format. DEMs are so well suited for block world games as they are already rasters. Also, you would not need to worry about the spatial reference if all you wanted was real-world elevation data. as soon as you access OSM or other datasets, or attempt to combine them, you will need GIS libraries to "sync" them or to convert between real-life spatial coordinates and game pixels -- DEMs are usable as-is for initial experimentation.

canadian dem data is available at: http://geogratis.gc.ca/api/en/nrcan-rnc ... E738E.html

in the US: http://earthexplorer.usgs.gov/

a freeware program called MicroDEM is useful for automatically downloading and merging DEMs:
http://www.usna.edu/Users/oceano/pguth/ ... crodem.htm

an example of a vegetation dataset already in raster format but only at 250m resolution might be useful for determining BIOMES if the spatial reference is the same as your elevation data: from MODIS: http://modis.gsfc.nasa.gov/data/dataprod/mod13.php

anyway, I would be willing to help with any project along these lines, though my lua and minetest modding is only in its infancy at this point.

[Sokomine]

As for underground strata, there are datasets for this too, but probably is overkill for a fun game. However, there aren't (m)any mapgens with "strata" in minetest, the focus seems to be more about caves, etc. exposed rock faces with strata visible would be cool.

Yes, that would be nice. There are sometimes large mapgen artefacts where huge stone cliffs are visible due to either mapgen change, change of parameters, or errors in the mapgen. Such places could be produced deliberately and show diffrent colored rocks. Might even be useful as a ressource for digging it up and building more colorful buildings with it. I think there's at least one mod in development that aims at that direction.

anyway, I would be willing to help with any project along these lines, though my lua and minetest modding is only in its infancy at this point.

With voxelmanip, it boils down to a bit of code you can copy from other mods plus manipulating a kind of 3d array. That in theory allows you to do all you might ever want. However, the needs of terrain for gameplay may differ considerably from RL terrain. Maybe the easiest way would be to take a flat mapgen, add the occasional river and proclaim that it's terrain from a flat region. Larger geological structures like mountains or mountain ranges are invisible due to not enough land beeing loaded in the client.

[bobomb]

Such places could be produced deliberately and show diffrent colored rocks.

Re: strata. One way could be to make a flat map, with layers of stone, clay, coal, etc, then the map could be lifted and sunk to the surface levels desired, which would expose the strata, and create runs and veins of minerals in a novel way. I'm not as interested in new mineral types, yet, this is just a mapgen idea.

However, the needs of terrain for gameplay may differ considerably from RL terrain. Maybe the easiest way would be to take a flat mapgen, add the occasional river and proclaim that it's terrain from a flat region.

This misses the point of this thread entirely. The idea is to get actual real world terrain into the game, not to tweak existing mapgen settings. (Time spent in a virtual world could actually have the incidental benefit of familiarizing yourself with real world topography.)

Frankly the notion of using a heightmap to set the game map surface height should be getting everyone's panties wet, whether for importing real world terrain or having an instant "level editor" for minetest with nothing more than ms paint: one texture for elevation, one for surface cover (simple color code for biome, water source, road and village coverage), and even one for mineral probability below ground.

Larger geological structures like mountains or mountain ranges are invisible due to not enough land beeing loaded in the client.

These issues of scale are easy to address. The real world DEM pixels can be scaled to represent only a single in-game square, a 16x16 block, or whatever size you want to handle with voxelmanip. Also, vertical relief can be exaggerated. A common tactic with DEMs is to "normalize" the elevation ranges, so that whatever the range is, it is spread to a max and min value, with or without clipping. Another simple solution could just be a vertical exaggeration factor, where 1m real world elevation would mean 3m in game height.

help needed
What is really needed next is a way to read large files into the mapgen, without loading the entire file, in order to set the surface height. This lua function might help: http://www.gamedev.net/topic/572784-lua ... try4662735
Then I have no idea where the mapgen sets the final surface height. Is it something that is set first then filled below, or is it set as the result of some other algorithm? I might use the novagen mod as a basis for this project: https://github.com/Novatux/mg/
Another possibility that bypasses mapgen could be adding a worldedit command that can read from the heightmap/DEM.

[paramat]

If you get stuck i could edit a quick simple mapgen for you, then just add your own line of code that returns a height for a given (x, z) co-ordinate.

[prestidigitator]

What is really needed next is a way to read large files into the mapgen, without loading the entire file, in order to set the surface height. This lua function might help: http://www.gamedev.net/topic/572784-lua ... try4662735

You could break them up into files with name patterns related to what areas they cover (map_0_0, map_0_1, map_1_0, map_1_1, etc.). Or, for raw data (and probably uncompressed image data like BMP files too, but I can't remember right now) you can seek within files and read a limited amount of data using Lua's file API (see file:seek() and file:read(); might need to io.open in binary mode to be as portable across platforms as possible).

Then I have no idea where the mapgen sets the final surface height. Is it something that is set first then filled below, or is it set as the result of some other algorithm? I might use the novagen mod as a basis for this project: https://github.com/Novatux/mg/

Yeah, that is definitely a problem. I think it comes up particularly when automatically spawning players that have died, as I suspect the fact that they sometimes spawn underground for custom mapgens indicates its using the ground height behind the scenes. I don't think there's any way to set the ground height from Lua. You can get the ground height using a "heightmap" mapgen object, but I don't think you can set it. This is still quite an obstacle to full Lua mapgens.

[bobomb]

paramat has provided me with a basic magen mod that I have used to get some initial results. (the code is in this gist: https://gist.github.com/paramat/389ee65ce9378d9dad25 It. makes things like setting heights, water levels, and the ease with which strata could be implemented plain as day.

My main hurdles right now are properly decoding the binary .tiff (geotiff) file format. I found some code to decypher the header, and some other code to interpret 16-bit signed integers, but I am still working out some details. I would really like to overcome this, because I think image maps for things like biome variables (moisture, heat, etc) and other ground-cover considerations would be awesome, but if it gets too complicated, I might switch to using the plaintext .dem format. .dem files cover much less area than the mosaiced geotiff I am using, which covers 9601m east/west and 4801m north/south and encompasses the entire "e" tile of the 082 section of the southern BC freely available dem data. I am doing a 1:1 pixel scaling, so the terrain is exaggerated by a factor of about 30:1. There are steep areas on the map where I think i might be interpreting the 16-bit value incorrectly. Switching to a system like presti mentioned where the mod is able to switch to different .dem tiles as we move further from the origin might mean that using the .dem file format might be better for this project than geotiff. at this time the origin of the map is the center of the tiff tile. no effort has been made to translate the real world coordinates yet. This information needs to be extracted from the header. also because the actual pixels are sometimes rotated to the real world, the edges of .dem and geoTiffs are often packed out with "transparent" pizels set to -32767 which would need to be handled from the next tile.

the code for this is at: https://github.com/bobombolo/realterrain

this is very experimental code, maybe someone can help me write a better bitmap reading function

[bobomb]

more progress...



having to discard every other row, and still having major incongruity at z = 1203, 3 and -1199

[bobomb]

I have discarded the 16-bit dem and reduced it to 8-bit, this makes the dem tiff substantially smaller in size and eliminates all the complications I was having with regard to interpreting the two byte elevation (which was basically getting scaled out anyway). Here is the result, with no incongruities that I have found:

[twoelk]

so what part of the real world are we seeing or are these simple fictional hightmaps?

will you offer maps like these? TopoMC: Minecraft Worlds From Real Data
or something combined with OSM data? Minecrafting with OS OpenData You could use the OSM data to simply drop roads onto the otherwise generated terrain.

Anyways a simple way to import pixelmap hightmaps, ficional or not, would really be nice

[Dartmouth]

I like how the mountains go through the clouds :)

[bobomb]

so what part of the real world are we seeing or are these simple fictional hightmaps?

will you offer maps like these? TopoMC: Minecraft Worlds From Real Data
or something combined with OSM data? Minecrafting with OS OpenData You could use the OSM data to simply drop roads onto the otherwise generated terrain.

Anyways a simple way to import pixelmap hightmaps, ficional or not, would really be nice

This is not fictional, this is southern British Columbia. As for importing data from an online service, I think the hurdle there is the ability to access the internet through the Lua scripting API. Not sure that is possible, at least not in a cross-platform way. It's just what I've heard (top of this thread?).

At this point, I am going to make nice normalized DEMs for some of the major cities and geological formations of North America (Vancouver, The Grand Canyon, New York, Bent Armpit Wyoming), then they can be chosen by typing their name in the "SEED" section when you create a new map. In the same way you can drop any of your own heightmaps in the mod folder and put the filename in the seed field and voila.

As for roads and landcover, I am going to experiment with an indexed-color bitmap for doing things like "dropping roads" and rivers and determining the vegetation a little bit. that way a heightmap and a colormap that cover the same extent could do the job you're looking for, albeit offline (albeit, possibly fictional, unlike OSM data)...

[bobomb]

just messin around:

[paramat]

Good stuff.

[prestidigitator]

As for importing data from an online service, I think the hurdle there is the ability to access the internet through the Lua scripting API. Not sure that is possible, at least not in a cross-platform way. It's just what I've heard (top of this thread?).

So don't, then. Create a separate daemon that either works on its own or communicates with the Minetest mod through the filesystem (e.g. requests for data go in one location, and downloaded data goes in another).

[afflatus]

nice

[bobomb]

can we use something like this: https://github.com/Lua-cURL/Lua-cURLv3 for interacting with webservice APIs?
there may be useful data available from ESRI: http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Services_from_Esri/02r300000267000000/

also people interested in this project might also want to know about some of the awesome opensource Geospatial data tools such as:

http://www.gdal.org/
http://www.qgis.org/en/site/
these are all available in ubuntu repositories, but if you are in windows the best way to get all these tools and more is https://trac.osgeo.org/osgeo4w/

here is a tutorial for minecraft that is a modified version of a tutorial that has been a source of easy steps for getting dem's without going onto various data warehouse websites... using microdem
http://www.minecraftforum.net/forums/archive/tutorials/930401-mapping-using-real-world-terrain-data
http://www.usna.edu/Users/oceano/pguth/website/microdem/microdem.htm

also it would be nice to leave the tiffs uncompressed (uncheck LZW compression because it makes the file so much more difficult to decipher) but then zip them (they compress heavily) and have lua able to read inside the zipped files... also many of the freely available data is zipped or gzipped tars.

http://luaforge.net/projects/gzio/ ?
http://luaforge.net/projects/lar/ ?

[rubenwardy]

You can make http requests like this: https://github.com/rubenwardy/minetest_ ... ua#L18-L47

Although I'm not sure if that includes downloading files, I imagine it does.

Note the insecure environment bit at the top, whether you use cURL or my method, you'll need that.

[bobomb]

i have posted this mod as a work in progress: https://forum.minetest.net/viewtopic.php?f=9&t=12666



this actually wants me to make a "graphing calculator mod" where the map is generated algorithmically and forget perlin noise, what is that anyway? i mean i read wikipedia but screw it, let's just write a quadratic equation to generate awesome planar surfaces. or as the mandelbrot lake image implies, actually make the plane a fractal dive, in the complex number plane.

tomorrow i will read this: https://nbickford.wordpress.com/2012/12/21/creating-fake-landscapes/

and of course the awesome cover art from autechre's album, Amber is always an inspiration:

[bobomb]

You can make http requests like this: https://github.com/rubenwardy/minetest_ ... ua#L18-L47

Although I'm not sure if that includes downloading files, I imagine it does.

Note the insecure environment bit at the top, whether you use cURL or my method, you'll need that.

i was really hoping not to have to change the security setting.

With curl downloading files might only be necessary with the data warehouse approach, we could use the webservice APIs to simply query a value: elevation, precipitation, total solar radiation, land cover, etc...