Automating Maperitive generation and MapBox upload

I recently blogged about integrating MapBox maps within the Smallworld GIS.  I included a video clip that showed how I connected Smallworld to contour and hillshading maps hosted on MapBox.  Those contour and hillshaded maps were created using Maperitive.  In this post, I want to share how I automated the generation and upload of these maps.

If you are familiar with Maperitive you will know that it has had a scripting language for a long time and recently it also introduced Python scripting.  For this post I am still using the old scripting framework.  I imagine it can easily be ported to Python.

The Maperitive script references some custom rulesets that I created to allow me to explicitly style the contour lines.  Following is the code I used to automate map generation and upload.

• contours-white.mrules
• a Maperitive ruleset styling white contour lines with transparent background
• contours.mrules
• a Maperitive ruleset styling brown contour lines with transparent background
• contours.mscript
• the Maperitive commands required to generate brown contours, white contours and hillshading.  Note that color-depth=32.  This is required to allow transparent backgrounds in the tiles stored in the MBtiles database
• create_and_upload.bat
• The batch file that first runs the Maperitive script and then automates upload into MapBox using TileMill.  The upload requires syncAccount and syncAccessToken for your MapBox account.  See http://boulderalfmaps.blogspot.com/2013/01/tilemill-sync-info.html for details about how to do this.

	features
	lines
	contour major : contour[@isMulti(elevation, 100)]
	contour minor : contour[@isMulti(elevation, 20) and not @isMulti(elevation, 100)]
	properties
	map-background-color : #F1EEE8
	map-background-opacity : 0
	font-weight : bold
	font-family : Verdana
	text-max-width : 7
	text-halo-width : 25%
	text-halo-opacity : 0.75
	text-align-horizontal : center
	text-align-vertical : center
	font-stretch : 0.9
	rules
	target: contour*
	define
	line-color : #ffffff
	line-opacity : 0.35
	curved : true
	if : *major
	define
	map.rendering.contour.label : true
	min-zoom : 9
	line-width : 11:0.1;11.9:1;12:2
	font-size : 10
	font-style : italic
	font-weight : normal
	text-halo-width : 35%
	text-halo-opacity : 1
	text-halo-color : #F1EEE8
	else
	define
	min-zoom : 12
	line-width : 1
	draw : contour

view raw contours-white.mrules hosted with ❤ by GitHub

	features
	lines
	contour major : contour[@isMulti(elevation, 100)]
	contour minor : contour[@isMulti(elevation, 20) and not @isMulti(elevation, 100)]
	properties
	map-background-color : #F1EEE8
	map-background-opacity : 0
	font-weight : bold
	font-family : Verdana
	text-max-width : 7
	text-halo-width : 25%
	text-halo-opacity : 0.75
	text-align-horizontal : center
	text-align-vertical : center
	font-stretch : 0.9
	rules
	target: contour*
	define
	line-color : #7f3300
	line-opacity : 0.35
	curved : true
	if : *major
	define
	map.rendering.contour.label : true
	min-zoom : 9
	line-width : 11:0.1;11.9:1;12:2
	font-size : 10
	font-style : italic
	font-weight : normal
	text-halo-width : 35%
	text-halo-opacity : 1
	text-halo-color : #F1EEE8
	else
	define
	min-zoom : 12
	line-width : 1
	draw : contour

view raw contours.mrules hosted with ❤ by GitHub

	change-directory C:\ifactor\products\workspace\Virtual_Earth_Connector\MaperitiveCreateContours\blog
	clear-map
	set-setting name=map.decoration.grid value=False
	set-geo-bounds -5.18793847256578,56.6581368928296,-4.90722891535273,56.7097180989809
	generate-contours
	use-ruleset location=contours.mrules as-alias=contours
	export-bitmap file=cont.png
	// use color-depth=32 so we can make use of the transparency in the ruleset
	generate-mbtiles file=cont.mbtiles color-depth=32 minzoom=1 maxzoom=16
	// we do not need to regenerate the contours. Simply apply a different styling.
	use-ruleset location=contours-white.mrules as-alias=contours-white
	apply-ruleset
	export-bitmap file=cont-white.png
	// use color-depth=32 so we can make use of the transparency in the ruleset
	generate-mbtiles file=cont-white.mbtiles color-depth=32 minzoom=1 maxzoom=16
	clear-map
	set-setting name=map.decoration.grid value=False
	set-geo-bounds -5.18793847256578,56.6581368928296,-4.90722891535273,56.7097180989809
	generate-relief-igor
	use-ruleset location=contours.mrules as-alias=contours
	export-bitmap file=shading.png
	// use color-depth=32 so we can make use of the transparency in the ruleset
	generate-mbtiles file=shading.mbtiles color-depth=32 minzoom=1 maxzoom=16

view raw contours.mscript hosted with ❤ by GitHub

	set MAPERITIVE_EXE=C:/Maperitive/Maperitive.exe
	:: make note of the original directory. That is where the MBTiles will be
	set ORIG_DIR=%~dp0
	:: first create the MBtiles
	cmd %MAPERITIVE_EXE% %ORIG_DIR%contours.mscript
	:: now upload the MBTiles to MapBox. The MapBox web form allows you to upload MBtile files
	:: with max size of 5MB. If you want to upload larger files, you will need to use something
	:: like TileMill's upload feature.
	:: make sure that you are in the TileMill directory so we can use the appropriate node.exe
	cd C:\Program Files (x86)\TileMill-v0.10.1\tilemill
	:: see http://boulderalfmaps.blogspot.com/2013/01/tilemill-sync-info.html
	:: for instructions about how to determine syncAccount and syncAccessToken
	set SYNCACCOUNT=<syncAccount>
	set SYNCACCESSTOKEN=<syncAccessToken>
	:: upload the MBTiles file to your MapBox account.
	:: you will need to specify your own <syncAccount> and <syncAccessToken> values
	node index.js export cont %ORIG_DIR%cont.mbtiles --format=upload --syncAccount=%SYNCACCOUNT% --syncAccessToken=%SYNCACCESSTOKEN%
	node index.js export cont-white %ORIG_DIR%cont-white.mbtiles --format=upload --syncAccount=%SYNCACCOUNT% --syncAccessToken=%SYNCACCESSTOKEN%
	node index.js export shading %ORIG_DIR%shading.mbtiles --format=upload --syncAccount=%SYNCACCOUNT% --syncAccessToken=%SYNCACCESSTOKEN%
	:: finally change back to the ORIG_DIR
	cd %ORIG_DIR%
	:: NOTE: MapBox makes use of the mbtile's (a sqlite file) metadata.name field to label the map
	:: after it is uploaded. Maperitive always sets this field value to "My Map" so after uploading the
	:: MBTiles to MapBox you will need to manually change the map name to something meaningful.
	pause

view raw create_and_upload.bat hosted with ❤ by GitHub

tilemill-sync-info

In my previous post, I described the steps required to automate a TileMill project for rendering and then uploading an MBTiles file to MapBox.

	:: make sure that your current directory is the one that contains the TileMill-specific node.exe
	cd "C:\Program Files (x86)\TileMill-v0.10.1\tilemill"
	:: select the TileMill project name
	SET TM_PROJECT=chaco
	:: I want to export to MBTiles so I need to specify where to render the file
	SET TM_FILE=%~dp0/%TM_PROJECT%.mbtiles
	:: render the project
	node index.js export %TM_PROJECT% %TM_FILE% --format=mbtiles
	:: upload the MBTiles file to your MapBox account.
	:: you will need to specify your own <syncAccount> and <syncAccessToken> values
	node index.js export %TM_PROJECT% %TM_FILE% --format=upload --syncAccount=<syncAccount> --syncAccessToken=<synchAccessToken>

view raw TileMill_process.bat hosted with ❤ by GitHub

The one important part that was not covered in the post was the authentication method.

As part of the TileMill upload process, you are required to set --syncAccount and --syncAccessToken parameters.  This information is discoverable if you dig deep into TileMill caches.  To make it more easily accessible I created a TileMill plugin called tilemill-sync-info.  It simply exposes your sync information on the Application Settings page...

Now you can copy/paste that information into your batch upload scripts.

Productivity Tip: I use TileMill for both work and personal projects and host these projects on separate work and personal MapBox accounts.  To keep from having to constantly reauthorize TileMill between these two accounts prior to upload, I simply create batch upload scripts for each TileMill project with account-specific upload credentials.  Then I can author my projects in TileMill and when I am ready to upload to MapBox I simply run the appropriate project upload script and it sends my new map to the correct account.

I was inspired to create a TileMill plugin based on @willwhitedc's MapBox presentation at JS.Geo13 (#jsgeo13)