Tile Server (2012)

From Planimate Knowledge Base
(Redirected from Tile Server)
Jump to: navigation, search

The Planimate Map object enables the use of tiled map data in models. These notes describe the setup of a map tile web server which can be used with Planimate or other open Street Maps projects.

This is an update to the 2011 Tile Server Guide, which is now obsolete. The references section contains sources from which this content was derived.

Setting up the VM

Start with Ubuntu 12.04 Server (AMD 64). I used the 'mini' CD image, using a network install. My initial VM settings: 3GB of RAM, 32GB HDD and bridged networking. I did the most basic install.

For the Australia region only, you can probably get by with 20GB HDD. For the world, you'd need over 1TB.

I gave the VM plenty of RAM for the osm2pgsl import stage (see below). Since then I've shrunk the VM's RAM configuration to 1GB which is more than plenty for a small scale test server.

Login at the console to finish initial setup, you want to lock down its IP address. ('ifconfig' gives you current). Edit /etc/network/interfaces, substituting your network details.

sudo apt-get install vim openssh-server
sudo vim /etc/network/interfaces

iface eth0 inet static
address 192.168.0.18
netmask 255.255.255.0
gateway 192.168.0.100

After this execute

sudo ifdown eth0; ifup eth0

You can now use a terminal like PuTTY to log in.

At this time you might like to edit /etc/hostname and /etc/hosts to give your tile server a name. I called mine 'maps'. From other machines, I referred to it by ip-address.

Setting Up The Environment

Installation

We use the mapnik repo to get the latest boost library.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get install subversion autoconf unzip
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:mapnik/boost
sudo apt-get update
sudo apt-get install libboost-dev libboost-filesystem-dev \
libboost-program-options-dev libboost-python-dev \
libboost-regex-dev libboost-system-dev libboost-thread-dev \
g++ cpp libicu-dev libboost-filesystem-dev \
libboost-program-options-dev \
libboost-python-dev libboost-regex-dev \
libboost-system-dev libboost-thread-dev \
python-dev libxml2 libxml2-dev \
libfreetype6 libfreetype6-dev \
libjpeg-dev \
libltdl7 libltdl-dev \
libpng-dev \
libgeotiff-dev libtiff-dev libtiffxx0c2 \
libcairo2 libcairo2-dev python-cairo python-cairo-dev \
libcairomm-1.0-1 libcairomm-1.0-dev \
ttf-unifont ttf-dejavu ttf-dejavu-core ttf-dejavu-extra \
git build-essential python-nose libtool libgeos-dev libpq-dev \
libbz2-dev proj \
libgdal1-dev python-gdal \
postgresql-9.1 postgresql-server-dev-9.1 postgresql-contrib-9.1 \
postgresql-9.1-postgis libsqlite3-dev

NOTE: You can use this to check the boost version

apt-cache policy libboost-dev

Create work folders and fetch Map Data

cd ~
mkdir src bin planet

If you're building a small test server, you want an extract. Otherwise you can download a complete planet.osm file here.

Don't let the size of the file fool you, the database expands by 50x its size.

This downloads the latest planet file:

cd ~/planet
wget http://planet.openstreetmap.org/planet-latest.osm.bz2

Or here's an Australia extract:

cd ~/planet
wget http://download.geofabrik.de/osm/australia-oceania.osm.bz2

NOTE:If you're intending to keep your database up to date, you might want to download your first planet file from planet.openstreetmap.org (or a mirror), keeping track of its date in its filename, eg: planet-110803.osm.bz2. Knowing the date is important if you intend to set up incremental updates (not covered in this article).

NOTE:Don't decompress the file, the tools work on it directly.

Prepare the postGIS database

The osm2pgsql tool imports the osm data into the database, the latest source is needed as mapnik expects certain columns.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
./autogen.sh
./configure
make

Configure the PostGIS database

Edit /etc/postgresql/9.1/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

Edit kernel parameter shmmax to increase maximum size of shared memory.

sudo vim /etc/sysctl.conf

Add or change this line:

kernel.shmmax = 268435456

Then save and load into kernel:

sudo sysctl -p /etc/sysctl.conf

Restart postgres to enable the changes

sudo /etc/init.d/postgresql restart

It should report:

* Restarting PostgreSQL 9.1 database server

Create a database called "gis". Some of our future tools presume that you will use this database name. Substitute your username for "username" in two places below. This should be the username that will render maps with mapnik.

sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
exit

Set up PostGIS on the database.

psql -f /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql -d gis

This should respond with many lines ending with

...CREATE FUNCTION
COMMIT
...
DROP FUNCTION 

Substitute your username for "username" in two places in the next line. This should be the user account that will render maps with mapnik.

echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis

Should reply with

ALTER TABLE
ALTER TABLE

Set the Spatial Reference Identifier (SRID) on the new database.

psql -f ~/bin/osm2pgsql/900913.sql -d gis

Should reply with

INSERT 0 1

Import data into the database with osm2pgsql

In case you have problems, these are the parameters

-S sets the style name.
--slim is mandatory as osm files are large these days. It also allows future incremental updates.
-d sets the data base name.
-C sets RAM cache size in MB. If your VM has 3GB of RAM, 2048 is OK. adjust it accordingly if your machine has less/more RAM.

Replace ~/planet/filename.osm.bz2 with the location of your planet or extract file.

cd ~/bin/osm2pgsql
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/filename.osm.bz2

Loading the planet file will take between an hour for a country and a few days for the whole world. Fast disks and a lot of RAM help most.

The import starts with NOTICES which are normal. Next, osm2pgsql will start reading the compressed planet file and give progress reports. It will take days for the entire world.You'll see:

Processing: Node(10140k) Way(0k) Relation(0k)

You may see errors like this:

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

This is OK, just minor errors in the OSM data. It should all end with some Completed lines and a Stopped Table message.

Install Mapnik library

The Mapnik library is the first of two items sometimes called "Mapnik". The other item is a collection of tools that OpenStreetMap uses to invoke the library.

This compiles the mapnik library. It takes quite a while. I do it in parallel with importing the map data.

cd ~/src
git clone http://github.com/mapnik/mapnik
cd mapnik
./configure
make
sudo make install

Confirm that Mapnik library is installed.

python
>>> import mapnik
>>>

If python replies with the second chevron prompt ">>>" and without errors, then the library was found by Python. Congratulations.

Install Mapnik tools

The Mapnik tools are the second item sometimes called "mapnik". This is a collection of tools from OpenStreetMap for making effective use of the Mapnik library.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/rendering/mapnik

Install prepared world boundary data

Mapnik uses prepared files to generate coastlines and ocean for small scale maps. This is faster than reading the entire database to render zoom levels from zero to nine.

This section now includes the additional shape files that were added to OpenStreetMap default styles in mid-2010. Beware of the long, strange looking links with the repeated http. They are unlikely to copy / paste directly. Use copy link location or equivalent.

cd ~/bin/mapnik
mkdir world_boundaries
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
tar xvzf world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
tar xvjf processed_p.tar.bz2 -C world_boundaries
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
tar xjf shoreline_300.tar.bz2 -C world_boundaries

wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places.zip
unzip ne_10m_populated_places.zip -d world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip
unzip ne_110m_admin_0_boundary_lines_land.zip -d world_boundaries

Render your first map

The database is loaded and the tools are installed.

Customise generate_image.py with the following changes.

mapfile = "my_osm.xml"
bounds = (144.37, -38.0, 144.39, -38.2)

The bounds selected here are the latitude & longitude for the Geelong, Victoria region.

Create a customised OSM file (remember to replace "username" with your username).

cd ~/bin/mapnik
./generate_xml.py osm.xml my_osm.xml --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --user username --accept-none


Now the big test.

cd ~/bin/mapnik
./generate_image.py

View "image.png" to confirm that you have rendered a map of Geelong.

Web Server

Now we set up the tile server. The mod_tile module uses a multithreaded version of apache.

sudo apt-get install apache2 apache2-threaded-dev

Build Apache Tile Module & renderd

Make sure you are the user that can access the DB, not root.

git clone https://github.com/openstreetmap/mod_tile/
cd mod_tile
./autogen.sh
./configure
make
sudo make install
sudo make install-mod_tile
sudo ldconfig
sudo mkdir /var/lib/mod_tile
sudo chown username /var/lib/mod_tile
sudo mkdir /var/run/renderd
sudo chown username /var/run/renderd

(if directories already exist, make sure they have write access to the user).

Edit renderd configuration

If its not there already, copy ~/src/mod_tile/renderd.conf to /etc In this setup, both Apache and the renderd program use this configuration file. Edit /etc/renderd.conf as follows.

[renderd]
socketname=/var/run/renderd/renderd.sock
num_threads=4
tile_dir=/var/lib/mod_tile
stats_file=/var/run/renderd/renderd.stats

[mapnik]
plugins_dir=/usr/local/lib/mapnik/input
font_dir=/home/username/src/mapnik/fonts
font_dir_recurse=1

[default]
URI=/tiles/
XML=/home/username/bin/mapnik/my_osm.xml
#HTCPHOST=proxy.openstreetmap.org

TIP: You can add alternate tilesets/layer configurations by adding another section eg:

[tiles2]
URI=/tiles2/
XML=/home/username/bin/mapnik/my_osm2.xml

In my test I copied my_osm.xml to my_osm2.xml and disabled some layers by adding attribute status="off" to the Layer elements in the second half of the file.

Set up Apache configuration

Copy the configuration file for mod_tile then edit it:

cd ~/src/mod_tile
sudo cp mod_tile.conf /etc/apache2/mods-available
sudo vim /etc/apache2/mods-available/mod_tile.conf

The changes are in the first few lines:

  • comment this out, we'll load it separately
#LoadModule tile_module modules/mod_tile.so
  • change the servername and comment out the alias line
ServerName maps
  • update the document root
DocumentRoot /var/www/
  • enable the use of the config file in /etc
LoadTileConfigFile /etc/renderd.conf

Thats it for the edits.

Now create a load file for the module and create the links in mods-enabled.

sudo su root
echo LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so > /etc/apache2/mods-available/mod_tile.load
ln -s /etc/apache2/mods-available/mod_tile.load /etc/apache2/mods-enabled/mod_tile.load
ln -s /etc/apache2/mods-available/mod_tile.conf /etc/apache2/mods-enabled/mod_tile.conf
exit

You can now restart apache and check the module is loaded:

sudo apache2ctl restart

Start renderd

For debugging, do this in a separate window - as regular user, not root.

~/src/mod_tile/renderd -f

We put this (without the -f) into rc.local later.

Testing

From a web browser, navigate to the address of your new server,

http://yourserver/mod_tile

You should see something like:

NoResp200: 0
NoResp304: 0
NoResp404: 0
NoResp503: 0
NoResp5XX: 0
NoRespOther: 0
NoFreshCache: 0
:

Now fetch a tile

http://yourserver/URI/0/0/0.png

Where URI matches the value entered into /etc/renderd.conf

If things dont work, check:

  • Can renderd write to /var/lib/mod_tile and /var/run/renderd? Have files been created there?
  • Check apache's error log
  • Is font path correct in renderd.conf
  • Any activity in the renderd console window?

Starting Automatically

I inserted this in /etc/rc.local

mkdir /var/run/renderd
chown username /var/run/renderd
sudo -u username /home/username/src/mod_tile/renderd

Adding a map to view tiles in a browser

Download and install OpenLayers:

cd ~
wget http://openlayers.org/download/OpenLayers-2.10.tar.gz
tar -xzvf OpenLayers-2.10.tar.gz
cd OpenLayers-2.10
sudo cp OpenLayers.js /var/www
sudo cp -r style /var/www
sudo cp -r theme /var/www
sudo cp examples/style.css /var/www
sudo cp examples/osm.html /var/www

Now customise osm.html to our site

sudo vim /var/www/osm.html

The following HTML will give you a full window map with on screen controls.

Change the URI as appropriate, in examples above I used "tiles"

<html>
<head>
<title>OpenLayers Demo</title>
<style type="text/css">
html, body, #basicMap {
width: 100%;
height: 100%;
margin: 0;
}
</style>
<script src="OpenLayers.js"></script>
<script>
function init() {
var options = {
projection: new OpenLayers.Projection("EPSG:900913"),
displayProjection: new OpenLayers.Projection("EPSG:4326"),
units: "m",
maxResolution: 156543.0339,
LonLat: new OpenLayers.LonLat(144.37228,-38.11116),
maxExtent: new OpenLayers.Bounds(-20037508.34, -20037508.34,
20037508.34, 20037508.34),
numZoomLevels: 20,
controls: [
new OpenLayers.Control.Navigation(),
new OpenLayers.Control.PanZoomBar(),
new OpenLayers.Control.Permalink(),
new OpenLayers.Control.ScaleLine(),
new OpenLayers.Control.MousePosition(),
new OpenLayers.Control.KeyboardDefaults()

]
};
map = new OpenLayers.Map("basicMap",options);
var newL = new OpenLayers.Layer.OSM("Default", "/URI/${z}/${x}/${y}.png", {numZoomLevels: 19});
map.addLayer(newL);
map.zoomIn();
}
</script>
</head>
<body onload="init();">
<div id="basicMap"></div>
</body>
</html>

You can add: set layer.attribution="whatever"; if you like.

Change the default co-ordinate to somewhere in Australia

...new OpenLayers.LonLat(144.38, -38.10)

Now navigate to http://yourserver/osm.html and you should have a map.

Adding Custom Shape Files

Export the shape files to somewhere accessible by mapnik (eg /home/username/bin/mapnik/ShapeDir)

Add appropriate style rules and layers to the map xml data file (mapnik/my_osm.xml)

<Style name="BerthStyle">
<Rule>
<LineSymbolizer>
<CssParameter name="stroke">#000000</CssParameter>
<CssParameter name="stroke-width">0.4</CssParameter>
</LineSymbolizer>
</Rule>
<Rule>
<TextSymbolizer name="name" fontset_name="book-fonts" size="10" fill="rgb(0,0,51)" halo_radius="1" wrap_width="20"></TextSymbolizer>
</Rule>
<Rule>
<PolygonSymbolizer>
<CssParameter name="fill">#404040</CssParameter>
<CssParameter name="fill-opacity">0.6</CssParameter>
</PolygonSymbolizer>
</Rule>
</Style>

<Layer name="geelong-berths-line" srs="+proj=latlong +datum=WGS84">
<StyleName>BerthStyle</StyleName>
<Datasource>
<Parameter name="file">/home/username/bin/mapnik/geelong/BERTHS/BERTHS_line</Parameter>
<Parameter name="type">shape</Parameter>
</Datasource>
</Layer>

Restart renderd and view the map.

There is some useful information available here which relates to different geo spatial reference systems. To ensure everything will work correctly with OSM then it is best to have the data referenced using the WGS84 datum.

References

This guide was originally based on material from here and here.

If you have problems building the mapnik library, check the guide here for changes:

If you have problems building mod_tile, check here. At the time of writing it did not include the fix for the missing #include.

Thanks to the authors of these guides.