Automating Maperitive generation and MapBox upload

I recently blogged about integrating MapBox maps within the Smallworld GIS.  I included a video clip that showed how I connected Smallworld to contour and hillshading maps hosted on MapBox.  Those contour and hillshaded maps were created using Maperitive.  In this post, I want to share how I automated the generation and upload of these maps.

If you are familiar with Maperitive you will know that it has had a scripting language for a long time and recently it also introduced Python scripting.  For this post I am still using the old scripting framework.  I imagine it can easily be ported to Python.

The Maperitive script references some custom rulesets that I created to allow me to explicitly style the contour lines.  Following is the code I used to automate map generation and upload.

• contours-white.mrules
• a Maperitive ruleset styling white contour lines with transparent background
• contours.mrules
• a Maperitive ruleset styling brown contour lines with transparent background
• contours.mscript
• the Maperitive commands required to generate brown contours, white contours and hillshading.  Note that color-depth=32.  This is required to allow transparent backgrounds in the tiles stored in the MBtiles database
• create_and_upload.bat
• The batch file that first runs the Maperitive script and then automates upload into MapBox using TileMill.  The upload requires syncAccount and syncAccessToken for your MapBox account.  See http://boulderalfmaps.blogspot.com/2013/01/tilemill-sync-info.html for details about how to do this.

	features
	lines
	contour major : contour[@isMulti(elevation, 100)]
	contour minor : contour[@isMulti(elevation, 20) and not @isMulti(elevation, 100)]
	properties
	map-background-color : #F1EEE8
	map-background-opacity : 0
	font-weight : bold
	font-family : Verdana
	text-max-width : 7
	text-halo-width : 25%
	text-halo-opacity : 0.75
	text-align-horizontal : center
	text-align-vertical : center
	font-stretch : 0.9
	rules
	target: contour*
	define
	line-color : #ffffff
	line-opacity : 0.35
	curved : true
	if : *major
	define
	map.rendering.contour.label : true
	min-zoom : 9
	line-width : 11:0.1;11.9:1;12:2
	font-size : 10
	font-style : italic
	font-weight : normal
	text-halo-width : 35%
	text-halo-opacity : 1
	text-halo-color : #F1EEE8
	else
	define
	min-zoom : 12
	line-width : 1
	draw : contour

view raw contours-white.mrules hosted with ❤ by GitHub

	features
	lines
	contour major : contour[@isMulti(elevation, 100)]
	contour minor : contour[@isMulti(elevation, 20) and not @isMulti(elevation, 100)]
	properties
	map-background-color : #F1EEE8
	map-background-opacity : 0
	font-weight : bold
	font-family : Verdana
	text-max-width : 7
	text-halo-width : 25%
	text-halo-opacity : 0.75
	text-align-horizontal : center
	text-align-vertical : center
	font-stretch : 0.9
	rules
	target: contour*
	define
	line-color : #7f3300
	line-opacity : 0.35
	curved : true
	if : *major
	define
	map.rendering.contour.label : true
	min-zoom : 9
	line-width : 11:0.1;11.9:1;12:2
	font-size : 10
	font-style : italic
	font-weight : normal
	text-halo-width : 35%
	text-halo-opacity : 1
	text-halo-color : #F1EEE8
	else
	define
	min-zoom : 12
	line-width : 1
	draw : contour

view raw contours.mrules hosted with ❤ by GitHub

	change-directory C:\ifactor\products\workspace\Virtual_Earth_Connector\MaperitiveCreateContours\blog
	clear-map
	set-setting name=map.decoration.grid value=False
	set-geo-bounds -5.18793847256578,56.6581368928296,-4.90722891535273,56.7097180989809
	generate-contours
	use-ruleset location=contours.mrules as-alias=contours
	export-bitmap file=cont.png
	// use color-depth=32 so we can make use of the transparency in the ruleset
	generate-mbtiles file=cont.mbtiles color-depth=32 minzoom=1 maxzoom=16
	// we do not need to regenerate the contours. Simply apply a different styling.
	use-ruleset location=contours-white.mrules as-alias=contours-white
	apply-ruleset
	export-bitmap file=cont-white.png
	// use color-depth=32 so we can make use of the transparency in the ruleset
	generate-mbtiles file=cont-white.mbtiles color-depth=32 minzoom=1 maxzoom=16
	clear-map
	set-setting name=map.decoration.grid value=False
	set-geo-bounds -5.18793847256578,56.6581368928296,-4.90722891535273,56.7097180989809
	generate-relief-igor
	use-ruleset location=contours.mrules as-alias=contours
	export-bitmap file=shading.png
	// use color-depth=32 so we can make use of the transparency in the ruleset
	generate-mbtiles file=shading.mbtiles color-depth=32 minzoom=1 maxzoom=16

view raw contours.mscript hosted with ❤ by GitHub

	set MAPERITIVE_EXE=C:/Maperitive/Maperitive.exe
	:: make note of the original directory. That is where the MBTiles will be
	set ORIG_DIR=%~dp0
	:: first create the MBtiles
	cmd %MAPERITIVE_EXE% %ORIG_DIR%contours.mscript
	:: now upload the MBTiles to MapBox. The MapBox web form allows you to upload MBtile files
	:: with max size of 5MB. If you want to upload larger files, you will need to use something
	:: like TileMill's upload feature.
	:: make sure that you are in the TileMill directory so we can use the appropriate node.exe
	cd C:\Program Files (x86)\TileMill-v0.10.1\tilemill
	:: see http://boulderalfmaps.blogspot.com/2013/01/tilemill-sync-info.html
	:: for instructions about how to determine syncAccount and syncAccessToken
	set SYNCACCOUNT=<syncAccount>
	set SYNCACCESSTOKEN=<syncAccessToken>
	:: upload the MBTiles file to your MapBox account.
	:: you will need to specify your own <syncAccount> and <syncAccessToken> values
	node index.js export cont %ORIG_DIR%cont.mbtiles --format=upload --syncAccount=%SYNCACCOUNT% --syncAccessToken=%SYNCACCESSTOKEN%
	node index.js export cont-white %ORIG_DIR%cont-white.mbtiles --format=upload --syncAccount=%SYNCACCOUNT% --syncAccessToken=%SYNCACCESSTOKEN%
	node index.js export shading %ORIG_DIR%shading.mbtiles --format=upload --syncAccount=%SYNCACCOUNT% --syncAccessToken=%SYNCACCESSTOKEN%
	:: finally change back to the ORIG_DIR
	cd %ORIG_DIR%
	:: NOTE: MapBox makes use of the mbtile's (a sqlite file) metadata.name field to label the map
	:: after it is uploaded. Maperitive always sets this field value to "My Map" so after uploading the
	:: MBTiles to MapBox you will need to manually change the map name to something meaningful.
	pause

view raw create_and_upload.bat hosted with ❤ by GitHub

tilemill-sync-info

In my previous post, I described the steps required to automate a TileMill project for rendering and then uploading an MBTiles file to MapBox.

	:: make sure that your current directory is the one that contains the TileMill-specific node.exe
	cd "C:\Program Files (x86)\TileMill-v0.10.1\tilemill"
	:: select the TileMill project name
	SET TM_PROJECT=chaco
	:: I want to export to MBTiles so I need to specify where to render the file
	SET TM_FILE=%~dp0/%TM_PROJECT%.mbtiles
	:: render the project
	node index.js export %TM_PROJECT% %TM_FILE% --format=mbtiles
	:: upload the MBTiles file to your MapBox account.
	:: you will need to specify your own <syncAccount> and <syncAccessToken> values
	node index.js export %TM_PROJECT% %TM_FILE% --format=upload --syncAccount=<syncAccount> --syncAccessToken=<synchAccessToken>

view raw TileMill_process.bat hosted with ❤ by GitHub

The one important part that was not covered in the post was the authentication method.

As part of the TileMill upload process, you are required to set --syncAccount and --syncAccessToken parameters.  This information is discoverable if you dig deep into TileMill caches.  To make it more easily accessible I created a TileMill plugin called tilemill-sync-info.  It simply exposes your sync information on the Application Settings page...

Now you can copy/paste that information into your batch upload scripts.

Productivity Tip: I use TileMill for both work and personal projects and host these projects on separate work and personal MapBox accounts.  To keep from having to constantly reauthorize TileMill between these two accounts prior to upload, I simply create batch upload scripts for each TileMill project with account-specific upload credentials.  Then I can author my projects in TileMill and when I am ready to upload to MapBox I simply run the appropriate project upload script and it sends my new map to the correct account.

I was inspired to create a TileMill plugin based on @willwhitedc's MapBox presentation at JS.Geo13 (#jsgeo13)

Automating TileMill in Windows

I have a need to live update TileMill projects similar to this poster's question.  I thought I had a solution when I found projectmill (a node.js project) but then I ran into a few problems.  Did I mention I am running on Windows 7?

(I discuss the challenges first, so if you are just looking for an answer, jump to the end of the post)

node-sqlite3 build problems

projectmill is a node.js project that has some dependencies.  npm install tried to install and build projectmill's node-sqlite3 dependency.  It turns out that I didn't have the correct Visual Studio build environment on my box and therefore the build didn't work.  Looking at projectmill's use of sqlite3, I discovered that it only used sqlite to update some metadata in the MBTiles file.  So instead of taking the time to learn all I needed to about Visual Studio to solve a relatively small part of projectmill, I cloned projectmill and tweaked my version to not require node-sqlite3.  Then I refactored the code to use the "shell command" approach (click the images to see full size)...

So that worked, but then I ran into the next problem.

projectmill not compatible with windows

Projectmill was likely never meant for Windows OS, but I found out the hard way.  The application does things like copying TileMill project files.  Unfortunately, it is expecting Unix-style pathnames and Windows lettered drive names cause it to break.  I could code around that, but then the next problem was that projectmill tried to launch tilemill with the Unix nice wrapper.  Again, this is not available in Windows, so that broke, too.  I could code around that, too, but it seemed that I was modifying too much of projectmill.

So I decided to take another approach.

Calling TileMill directly from Windows command line

I discovered that you can call TileMill directly from the Windows command line.  The important thing to remember is that you need to use the node.exe found in <TileMill Program Directory>\tilemill and not the node.exe that you might have installed for your other node projects.

This Windows bat code snippet gives an example of how you can construct calls to TileMill to automate rendering and upload processes.

	:: make sure that your current directory is the one that contains the TileMill-specific node.exe
	cd "C:\Program Files (x86)\TileMill-v0.10.1\tilemill"
	:: select the TileMill project name
	SET TM_PROJECT=chaco
	:: I want to export to MBTiles so I need to specify where to render the file
	SET TM_FILE=%~dp0/%TM_PROJECT%.mbtiles
	:: render the project
	node index.js export %TM_PROJECT% %TM_FILE% --format=mbtiles
	:: upload the MBTiles file to your MapBox account.
	:: you will need to specify your own <syncAccount> and <syncAccessToken> values
	node index.js export %TM_PROJECT% %TM_FILE% --format=upload --syncAccount=<syncAccount> --syncAccessToken=<synchAccessToken>

view raw TileMill_process.bat hosted with ❤ by GitHub

The projectmill application does basically the same thing the bat code does, but wraps it in a node.js framework.  It probably wouldn't take much to create a Windows-friendly node.js or python wrapper for TileMill automation.  The command line parameters for uploading to MapBox require a --syncAccount and --syncAccessToken  While the syncAccount is easy to know, it is not very easy to determine your syncAccessToken.  In a subsequent post I will show how to tweak TileMill to show that information more clearly to you.


Automating Tile Generation with TileMill over at the Fuzzy Tolerance blog gives additional details.

#OpenStreetMap + #Maperitive + @MapQuest = BoulderBikeNetwork

I like cycling. I like OpenStreetMap. I discovered Maperitive a few months ago and thought I would try my hand at rendering my own custom-styled map tiles. I don't have a PostgreSQL database nor do I currently have the time to learn it. It turns out, that I don't need to if I am working on a city-scale project.

Together with Maperitive's scripting language and some Python scripts that I wrote, I have put together a framework that anyone can implement fairly easily to create their own tiles for a localized area and then host it on a web server (I am using Amazon S3 for my tile server). I hope to publish my steps in the next month or two.

And then not too long ago I discovered that the MapQuest Open Initiative made bicycle routing freely available based on the OpenStreetMap data. How cool is that! So now I can ride my bike while surveying for OSM and if I tag the paths correctly, everyone can route on them properly within a few days of me uploading the data.

You can see my BoulderBikeNetwork sandbox area at https://s3.amazonaws.com/boulderalf/boulderbikenetwork/google_map.html

Note: just today I saw this map http://www.cyclelicio.us/map/ that does alot of similar things that I have been thinking about. It is exciting to see the many uses for the OSM data!

Why Parking Aisles are a Good Thing

I have been mapping parking aisles in my neighborhood. I was hoping that someday some OSM-based routing engines would tell me where I could take a short cut on my bike or on foot that would take me through the parking aisles and foot paths rather than around the parking lot.

I found such a situation in my neighborhood today at the Table Mesa Mall. You can see that I have added parking lot, parking aisles (not very straight but I will clean them up), buildings, shops and even stairs.

My scenario is that I am leaving King Soopers grocer on the west side of the complex and want to travel to Savers on the east side. I put the query to both open.mapquest.com (which uses OSM data) and maps.google.com (which uses data from somewhere else). Open.Mapquest sends me via the shortest route through the parking lot... Google sends me around the periphery.

Clearly, as Google gets more data into their dataset, their routing will likely get better... but that's the point, isn't it... that OSM also didn't have the data but then I came along and added it... and then MapQuest took that vector data and updated their routing web service.

Go OSM!!! Good job MapQuest!!!

Here are the two results (you can click on the image to get a larger view)...

OpenStreetMap Survey using iPad and GaiaGPS

There are many good OSM mapping techniques described at http://wiki.openstreetmap.org/wiki/Mapping_techniques.

The ideal mapping approach involves having tools that let you record new features (e.g., GPS and some kind of note taking device such as pen and paper) and also let you see what has already been mapped in your survey area (e.g., Walking Papers).

I prefer to use as little paper as possible, so when I discovered this solution that works for GPS-enabled iPad or iPhone devices, I wanted to share it.

You will need to download GaiaGPS (http://www.gaiagps.com/) from the iTunes App Store. Once you have installed the application, you can download the current OSM maptiles for the area you want to survey and then you can go into the field and start your work. As you start your surveying, you can place pins (aka Waypoints) and label them with notes relevant to the feature you are capturing. If find labelling/annotating waypoints in GaiaGPS orders of magnitude easier than trying to do the same thing with a standard GPS unit. You may also want to consider turning on a track (GaiaGPS calls it a "trip") so that you can get the path of your survey for later upload to OSM.

Once you have finished your survey, you can get the .gpx data out of GaiaGPS by e-mailing it to yourself. Once on your desktop, you can import the .gpx data with annotated waypoints and tracks into a tool like JOSM to guide you in adding OSM features.

Here is a 5 minute video that demonstrates the key points. The video quality is mediocre but you should get the idea of how to use the tool. I am not a representative of TrailBehind (the makers of GaiaGPS). I am only a very satisfied customer that has learned to use this app to simplify my OSM surveys.

Viewing OSM tiles using an iOS 4.2 device

I have not been able to get the slippymap used by sites like http://www.openstreetmap.org to play well with my iPad.  The slippymap provided by OpenLayers doesn't seem to recognize the touch gestures for zooming and panning.  I noticed, however, that the Google Maps API does recognize the touch gestures.  So when I saw in "OpenStreetMap: Using and Enhancing the Free Map of the World" the example to load OSM tiles into the GMaps API, I thought I would give it a try.

You can see the results here.  Nothing special, but now at least I can navigate the OSM data easily with my iPad.  The example I am using makes use of tiles that I am building for a bicycle route map of Boulder, CO, USA.  It is my first rendering project and I am using Maperitive for it.  I will post more on that later.