Tile Server (2011)
Obsolete Page
This page describes setting up a Map Tile Server under Ubuntu 10.04 for use with the Planimate Map object. It is now obsolete because the tools have evolved, particularly mapnik and the map_tile module.
The most recent guide is available here.
This page is kept for historical reasons.
You could use the 2012 procedure under Ubuntu 10.04 but you have to substitute the older version of postgres as described in this guide.
Setting up the VM
I created the VM by pointing VMWare to a Ubuntu 10.04 ISO (ubuntu-10.04-server-amd64.iso) in its new VM wizard. I initially set 3.5GB of RAM, 20GB HDD and bridged networking. Selected defaults for the rest.. In a few minutes I was at a login prompt.
This was a first test, only for the Australia region. In the end, about 6GB of space has been used (not including tiles). With the current growth of the world osm file, you'll need 600GB of disk space.
I gave the VM plenty of RAM for the osm2pgsl import stage (see below). Since then I've shrunk the VM's RAM configuration to 1GB which is more than plenty for a small scale test server.
You'll need to know the IP address of your server. Running 'ifconfig' should tell you this and the "eth" device being used. If you want to give your server a static IP address, edit /etc/network/interfaces, substituting your network details.
sudo vim /etc/network/interfaces iface eth0 inet static address 192.168.0.18 netmask 255.255.255.0 gateway 192.168.0.100
After this execute
sudo ifdown eth0; ifup eth0
You can then use a terminal like PuTTY to log in.
Update operating system
sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade
At this time you might like to edit /etc/hostname and /etc/hosts to give your tile server a name. I called mine 'maps'. From other machines, I referred to it by ip-address.
After this, I rebooted (sudo reboot) so the latest kernel was resident.
Get some system tools
We'll need subversion to get the latest updates from OpenStreetMap and other places. Munin makes pretty pictures of activity on the server. I like screen. htop is neat-o.
sudo apt-get install subversion autoconf screen munin-node munin htop unzip
Create work folders
We'll be using svn to retrieve the latest versions of key tools instead of using packages. In this guide we keep the files in our home directory.
cd ~ mkdir src bin planet
Install the OSM Server
Downloading OpenStreetMap data
If you're building a small test server, you want an extract. Otherwise you can download a complete planet.osm file here.
In July 2011 the entire planet file is some 18GB; dont bother with it unless you have 1TB of disk space as the database gets huge when you import the data.
This downloads the latest planet file:
cd ~/planet wget http://planet.openstreetmap.org/planet-latest.osm.bz2
If you're intending to keep your database up to date, you might want to download your first planet file from planet.openstreetmap.org (or a mirror), keeping track of its date in its filename, eg: planet-110803.osm.bz2. Knowing the date is important if you intend to set up incremental updates (not covered in this article).
NOTE: don't decompress the file, the tools work on it directly.
Prepare the postGIS database
Use the PostGIS extensions to postgresql for all sorts of geographical goodness. Install the postGIS and prerequisites.
sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4 sudo apt-get install postgresql-server-dev-8.4 sudo apt-get install build-essential libxml2-dev libtool sudo apt-get install libgeos-dev libpq-dev libbz2-dev proj
Install osm2pgsql from the repository
The latest version of osm2pgsql has the most goodies, so we'll use that rather than a package.
cd ~/bin svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/ cd osm2pgsql ./autogen.sh ./configure make
Configure the PostGIS database
edit /etc/postgresql/8.4/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.
shared_buffers = 128MB # 16384 for 8.1 and earlier checkpoint_segments = 20 maintenance_work_mem = 256MB # 256000 for 8.1 and earlier autovacuum = off
Edit kernel parameter shmmax to increase maximum size of shared memory.
sudo sysctl -w kernel.shmmax=268435456 sudo sysctl -p /etc/sysctl.conf
Restart postgres to enable the changes
sudo /etc/init.d/postgresql-8.4 restart
It should restart as above.
* Restarting PostgreSQL 8.4 database server ...done.
Create a database called "gis". Some of our future tools presume that you will use this database name. Substitute your username for "username" in two places below. This should be the username that will render maps with mapnik.
sudo -u postgres -i createuser username # answer yes for superuser createdb -E UTF8 -O username gis createlang plpgsql gis exit
Set up PostGIS on the postresql database.
psql -f /usr/share/postgresql/8.4/contrib/postgis.sql -d gis
This should respond with many lines ending with
... CREATE FUNCTION COMMIT ... DROP FUNCTION
Note: other guides have postgis.sql in a different location. This is where I found it.
Substitute your username for "username" in two places in the next line. This should be the username that will render maps with mapnik.
echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
Should reply with
ALTER TABLE ALTER TABLE
Set the Spatial Reference Identifier (SRID) on the new database.
psql -f ~/bin/osm2pgsql/900913.sql -d gis
Should reply with
INSERT 0 1
Import planet data into the database with osm2pgsql
Before you start the import, review the parameters.
-S sets the style name.
--slim is mandatory as osm files are large these days. It also allows future incremental updates.
-d sets the data base name.
-C sets RAM cache size in MB. If your VM has 3.5GB of RAM, 2048 is OK. adjust it accordingly if your machine has less/more RAM.
Replacee ~/planet/filename.osm.bz2 with the location of your planet/extract file.
cd ~/bin/osm2pgsql ./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/filename.osm.bz2
Loading the planet file will take between an hour for a country and a few days for the whole world. Fast disks and a lot of RAM help most.
The import starts with logs like this:
Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point NOTICE: table "planet_osm_point" does not exist, skipping NOTICE: table "planet_osm_point_tmp" does not exist, skipping<br>Setting up table: planet_osm_line NOTICE: table "planet_osm_line" does not exist, skipping NOTICE: table "planet_osm_line_tmp" does not exist, skipping Setting up table: planet_osm_polygon NOTICE: table "planet_osm_polygon" does not exist, skipping NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping Setting up table: planet_osm_roads NOTICE: table "planet_osm_roads" does not exist, skipping NOTICE: table "planet_osm_roads_tmp" does not exist, skipping Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192 Setting up table: planet_osm_nodes NOTICE: table "planet_osm_nodes" does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes" Setting up table: planet_osm_ways NOTICE: table "planet_osm_ways" does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways" Setting up table: planet_osm_rels NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"
Don't be concerned by the "NOTICE:" entries above. All normal.
Next, osm2pgsql will start reading the compressed planet file.
Reading in file: /home/nerd/planet/planet-100217.osm.bz2
As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.
Processing: Node(10140k) Way(0k) Relation(0k)
As the import proceeds, the "Node" number will update a couple of times per second until complete, then the "Way" number will update not quite so quickly, roughly every second or two. Finally the "Relation" number will update but at a slower rate, roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.
Processing: Node(593072k) Way(45376k) Relation(87k) Exception caught processing way id=110802 Exception caught processing way id=110803 Processing: Node(593072k) Way(45376k) Relation(474k)
The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.
The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this.
Node stats: total(593072533), max(696096737) Way stats: total(45376969), max(55410575) Relation stats: total(484528), max(555276) Going over pending ways processing way (752k)
The "processing way" number should update approximately each second.
Going over pending relations node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97% Committing transaction for planet_osm_roads Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Sorting data and creating indexes for planet_osm_line Sorting data and creating indexes for planet_osm_roads Sorting data and creating indexes for planet_osm_polygon Committing transaction for planet_osm_point Sorting data and creating indexes for planet_osm_point Stopping table: planet_osm_nodes Stopping table: planet_osm_ways Stopping table: planet_osm_rels Building index on table: planet_osm_rels Stopped table: planet_osm_nodes Building index on table: planet_osm_ways Stopped table: planet_osm_rels Completed planet_osm_point Completed planet_osm_roads Completed planet_osm_polygon Completed planet_osm_line Stopped table: planet_osm_ways
This should mean that you import is complete and successful.
Install Mapnik library
The Mapnik library is the first of two items sometimes called "Mapnik". The other item is a collection of tools that OpenStreetMap uses to invoke Mapnik.
The official and up-to-date Mapnik Installation Instructions are here.
You might find that this procedure works as well.
Get some dependencies for building the Mapnik library.
sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev sudo apt-get install libboost-python1.40-dev python-cairo-dev python-nose sudo apt-get install libboost1.40-dev libboost-filesystem1.40-dev sudo apt-get install libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev sudo apt-get install libboost-program-options1.40-dev libboost-python1.40-dev sudo apt-get install libfreetype6-dev libcairo2-dev libcairomm-1.0-dev sudo apt-get install libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2 sudo apt-get install libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev sudo apt-get install libgdal1-dev python-gdal sudo apt-get install imagemagick ttf-dejavu
Build Mapnik library from source.
cd ~/src svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik cd mapnik python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/ python scons/scons.py sudo python scons/scons.py install sudo ldconfig
Confirm that Mapnik library is installed.
python >>> import mapnik >>>
If python replies with the second chevron prompt ">>>" and without errors, then Mapnik library was found by Python. Congratulations.
Install Mapnik tools
The Mapnik tools are the second item sometimes called "mapnik". This is a collection of tools from OpenStreetMap for making effective use of the Mapnik library.
cd ~/bin svn co http://svn.openstreetmap.org/applications/rendering/mapnik
Install prepared world boundary data
Mapnik uses prepared files to generate coastlines and ocean for small scale maps. This is faster than reading the entire database to render zoom levels from zero to nine.
This section now includes the additional shape files that were added to OpenStreetMap default styles in mid-2010. Beware of the long, strange looking links with the repeated http. They are unlikely to copy / paste directly. Use copy link location or equivalent.
cd ~/bin/mapnik mkdir world_boundaries wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz tar xvzf world_boundaries-spherical.tgz wget http://tile.openstreetmap.org/processed_p.tar.bz2 tar xvjf processed_p.tar.bz2 -C world_boundaries wget http://tile.openstreetmap.org/shoreline_300.tar.bz2 tar xjf shoreline_300.tar.bz2 -C world_boundaries wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip unzip 10m-populated-places.zip -d world_boundaries wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip unzip 110m-admin-0-boundary-lines.zip -d world_boundaries
Render your first map
The database is loaded and the tools are installed.
Customise generate_image.py with the following changes. This is used to create the XML file we'll use later next.
mapfile = "my_osm.xml" bounds = (144.37, -38.0, 144.39, -38.2)
- The bounds selected here are the latitude & longitude for the Geelong, Victoria region.
Create a customised OSM file (Remember to replace "username" with your username).
cd ~/bin/mapnik ./generate_xml.py osm.xml my_osm.xml --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --user username --accept-none
Let's test everything together.
cd ~/bin/mapnik ./generate_image.py
View image.png to confirm that you have rendered a map of Geelong. Congratulations.
Web Server
Now we set up the tile server. The mod_tile module uses a multithreaded version of apache.
sudo apt-get install apache2 apache2-threaded-dev
Build Apache Tile Module & renderd
Make sure you are the user that can access the DB, not root.
cd ~/src svn co http://svn.openstreetmap.org/applications/utils/mod_tile/ cd mod_tile make sudo make install sudo mkdir /var/lib/mod_tile sudo chown username /var/lib/mod_tile touch /var/lib/mod_tile/planet-import-complete sudo mkdir /var/run/renderd sudo chown username /var/run/renderd
(if directories already exist, make sure they have write access to the user).
Edit renderd configuration
The make install above copied /etc/renderd.conf from the src/mod_tile directory. In this setup, both Apache and the renderd program use this configuration file.
sudo vim /etc/renderd.conf
Edit as follows, adjust the home directory as appropriate:
[renderd] socketname=/var/run/renderd/renderd.sock num_threads=4 tile_dir=/var/lib/mod_tile ; DOES NOT WORK YET stats_file=/var/run/renderd/renderd.stats [mapnik] plugins_dir=/usr/local/lib/mapnik/input font_dir=/home/username/src/mapnik/fonts font_dir_recurse=1 [default] URI=/tiles/ XML=/home/username/bin/mapnik/my_osm.xml #HTCPHOST=proxy.openstreetmap.org
TIP: You can add alternate tilesets/layer configurations by adding another section eg:
[tiles2] URI=/tiles2/ XML=/home/username/bin/mapnik/my_osm2.xml
In my test I copied my_osm.xml to my_osm2.xml and disabled some layers by adding attribute status="off" to the Layer elements in the second half of the file.
Set up Apache configuration
Copy the configuration file for mod_tile then edit it:
cd ~/src/mod_tile sudo cp mod_tile.conf /etc/apache2/mods-available sudo vim /etc/apache2/mods-available/mod_tile.conf
The changes are in the first few lines:
- comment this out, we'll load it separately
#LoadModule tile_module modules/mod_tile.so
- change the servername and comment out the alias line
ServerName maps
- update the document root
DocumentRoot /var/www/
- enable the use of the config file in /etc
LoadTileConfigFile /etc/renderd.conf
Thats it for the edits.
Now create a load file for the module and create the links in mods-enabled.
sudo su root echo LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so > /etc/apache2/mods-available/mod_tile.load ln -s /etc/apache2/mods-available/mod_tile.load /etc/apache2/mods-enabled/mod_tile.load ln -s /etc/apache2/mods-available/mod_tile.conf /etc/apache2/mods-enabled/mod_tile.conf exit
You can now restart apache and check the module is loaded:
sudo apache2ctl restart
Start renderd
For debugging, do this in a separate window - as regular user, not root.
~/src/mod_tile/renderd -f
We put this (without the -f) into rc.local later.
Testing
From a web browser, navigate to the address of your new server,
You should see something like:
NoResp200: 0 NoResp304: 0 NoResp404: 0 NoResp503: 0 NoResp5XX: 0 NoRespOther: 0 NoFreshCache: 0 :
Now fetch a tile
http://yourserver/URI/0/0/0.png
Where URI matches the value entered into /etc/renderd.conf
If things dont work, check:
- Can renderd write to /var/lib/mod_tile and /var/run/renderd? Have files been created there?
- Check apache's error log
- Is font path correct in renderd.conf
- Any activity in the renderd console window?
Starting Automatically
I inserted this in /etc/rc.local
mkdir /var/run/renderd chown username /var/run/renderd sudo -u username /home/username/src/mod_tile/renderd
Adding a map to view tiles in a browser
Download and install OpenLayers:
cd ~ wget http://openlayers.org/download/OpenLayers-2.10.tar.gz tar -xzvf OpenLayers-2.10.tar.gz cd OpenLayers-2.10 sudo cp OpenLayers.js /var/www sudo cp -r style /var/www sudo cp -r theme /var/www sudo cp examples/style.css /var/www sudo cp examples/osm.html /var/www
Now customise osm.html to our site
sudo vim /var/www/osm.html
The following HTML will give you a full window map with on screen controls.
Change the URI as appropriate.
<html>
<head>
<title>OpenLayers Demo</title>
<style type="text/css">
html, body, #basicMap {
width: 100%;
height: 100%;
margin: 0;
}
</style>
<script src="http://www.openlayers.org/api/OpenLayers.js"></script>
<script>
function init() {
var options = {
projection: new OpenLayers.Projection("EPSG:900913"),
displayProjection: new OpenLayers.Projection("EPSG:4326"),
units: "m",
maxResolution: 156543.0339,
LonLat: new OpenLayers.LonLat(144.37228,-38.11116),
maxExtent: new OpenLayers.Bounds(-20037508.34, -20037508.34,
20037508.34, 20037508.34),
numZoomLevels: 20,
controls: [
new OpenLayers.Control.Navigation(),
new OpenLayers.Control.PanZoomBar(),
new OpenLayers.Control.Permalink(),
new OpenLayers.Control.ScaleLine(),
new OpenLayers.Control.MousePosition(),
new OpenLayers.Control.KeyboardDefaults()
]
};
map = new OpenLayers.Map("basicMap",options);
var newL = new OpenLayers.Layer.OSM("Default", "/URI/${z}/${x}/${y}.png", {numZoomLevels: 19});
map.addLayer(newL);
map.zoomIn();
}
</script>
</head>
<body onload="init();">
<div id="basicMap"></div>
</body>
</html>
You can add: set layer.attribution="whatever"; if you like.
Change the default co-ordinate to somewhere in Australia
...new OpenLayers.LonLat(144.38, -38.10)
Now navigate to http://yourserver/osm.html and you should have a map.
Adding Custom Shape Files
Export the shape files to somewhere accessible by mapnik (eg /home/username/bin/mapnik/ShapeDir)
Add appropriate style rules and layers to the map xml data file (mapnik/my_osm.xml)
<Style name="BerthStyle"> <Rule> <LineSymbolizer> <CssParameter name="stroke">#000000</CssParameter> <CssParameter name="stroke-width">0.4</CssParameter> </LineSymbolizer> </Rule> <Rule> <TextSymbolizer name="name" fontset_name="book-fonts" size="10" fill="rgb(0,0,51)" halo_radius="1" wrap_width="20"></TextSymbolizer> </Rule> <Rule> <PolygonSymbolizer> <CssParameter name="fill">#404040</CssParameter> <CssParameter name="fill-opacity">0.6</CssParameter> </PolygonSymbolizer> </Rule> </Style> <Layer name="geelong-berths-line" srs="+proj=latlong +datum=WGS84"> <StyleName>BerthStyle</StyleName> <Datasource> <Parameter name="file">/home/username/bin/mapnik/geelong/BERTHS/BERTHS_line</Parameter> <Parameter name="type">shape</Parameter> </Datasource> </Layer>
Restart renderd and view the map.
There is some useful information available here which relates to different geo spatial reference systems. To ensure everything will work correctly with OSM then it is best to have the data referenced using the WGS84 datum.