Tile Server (2011)

From Planimate Knowledge Base
Revision as of 14:19, 9 August 2011 by Dave.withnall (talk | contribs)
Jump to navigation Jump to search

The Planimate Map object enables the use of tiled map data in models. These notes describe the setup of a map tile web server which can be used with Planimate or other open Street Maps projects.

The bulk of the install is described here and credit to the author for this guide, which gets you from a bare linux install to locally creating map tiles.

Setting up the VM

I created the VM by pointing VMWare to a Ubuntu 10.04 ISO (ubuntu-10.04-server-amd64.iso) in its new VM wizard. I initially set 3.5GB of RAM, 20GB HDD and bridged networking. Selected defaults for the rest.. In a few minutes I was at a login prompt.

This was a first test, only for the Australia region. In the end, about 6GB of space has been used (not including tiles). With the current growth of the world osm file, you'll need 600GB of disk space.

I gave the VM plenty of RAM for the osm2pgsl import stage (see below). Since then I've shrunk the VM's RAM configuration to 1.8GB which is more than plenty for a small scale test server.

You'll need to know the IP address of your server. Running 'ifconfig' should tell you this and the "eth" device being used. If you want to give your server a static IP address, edit /etc/network/interfaces, substituting your network details.

sudo vim /etc/network/interfaces

iface eth0 inet static
address 192.168.0.18
netmask 255.255.255.0
gateway 192.168.0.100

After this execute

sudo ifdown eth0; ifup eth0

You can then use a terminal like PuTTY to log in.

Update operating system

sudo apt-get update
sudo apt-get upgrade

Get some system tools

We'll need subversion to get the latest updates from OpenStreetMap and other places. Munin makes pretty pictures of activity on the server. I like screen. htop is neat-o.

sudo apt-get install subversion autoconf screen munin-node munin htop unzip

Organize the file system a bit

cd ~
mkdir src bin planet

Install the OSM Server

Get the latest "Planet" - OpenStreetMap data file

new "planet" file is published approximately each week. The mirror and archives of the "planet" files are here. In March 2010 the planet file was about 8.2GB in length. If you are not interested in the entire planet you can choose to download an extract file instead.

Facing an 8.2GB or larger download, this is an excellent time to consider using screen if you haven't used it before. Screen allows you to operate several terminal windows through one ssh connection. And more. Have a look this screen tutorial if you haven't used it before. Or wait for your download to complete, or use another ssh session to continue.

cd planet
wget http://planet.openstreetmap.org/planet-latest.osm.bz2

Prepare the postGIS database

Use the PostGIS extensions to postgresql for all sorts of geographical goodness. Install the postGIS and prerequisites.

sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4
sudo apt-get install postgresql-server-dev-8.4
sudo apt-get install build-essential libxml2-dev libtool
sudo apt-get install libgeos-dev libpq-dev libbz2-dev proj

Install osm2pgsql from the repository

The latest version of osm2pgsql has the most goodies, so we'll use that rather than a package.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
./autogen.sh
./configure
make

Configure the PostGIS database

edit /etc/postgresql/8.4/main/postgresql.conf in four places. These changes help with the large quantities of data that we are using.

shared_buffers = 128MB # 16384 for 8.1 and earlier
checkpoint_segments = 20
maintenance_work_mem = 256MB # 256000 for 8.1 and earlier
autovacuum = off

Edit kernel parameter shmmax to increase maximum size of shared memory.

sudo sysctl -w kernel.shmmax=268435456
sudo sysctl -p /etc/sysctl.conf

Restart postgres to enable the changes

sudo /etc/init.d/postgresql-8.4 restart

It should restart as above.

* Restarting PostgreSQL 8.4 database server
...done.

Create a database called "gis". Some of our future tools presume that you will use this database name. Substitute your username for "username" in two places below. This should be the username that will render maps with mapnik.

sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
exit

Set up PostGIS on the postresql database.

psql -f /usr/share/postgresql/8.4/contrib/postgis.sql -d gis

This should respond with many lines ending with

...
CREATE FUNCTION
COMMIT
...
DROP FUNCTION 

Note: recent change requiring a new path to postgis.sql, is now reflected above.

Substitute your username for "username" in two places in the next line. This should be the username that will render maps with mapnik.

echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
  1. Should reply with
ALTER TABLE
ALTER TABLE

Set the Spatial Reference Identifier (SRID) on the new database.

psql -f ~/bin/osm2pgsql/900913.sql -d gis

Should reply with

INSERT 0 1

Load planet into the database with osm2pgsql

Your planet file will have a different date. Use an extract file if your interest is limited to a smaller portion of the planet. This operation will take 30 hours or longer. It is very I/O intensive and you can speed it up with very fast disks.

The osm2pgsql command that follows has several parameters. Because this process takes a long time to complete, it is worthwhile to take a moment to consider before starting.

-S
style file name.
--slim
Use slim mode. Do this. Trust me. Slim mode is required if you can't hold the complete node data in RAM during the import. Even if you have enough RAM and can hold all the node data, you probably want to use slim mode. Slim mode is required if you plan to later apply updates to your data base rather than reloading from scratch.
-d
data base name.
-C
RAM cache size in MB. Make this as large as you can.
~/filename.osm.bz2
Location of your planet file.

cd ~/bin/osm2pgsql
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-100217.osm.bz2

Loading the planet file will take some time. How much time it will take depends primarily on the speed of your hard drive system and on the configuration of your system and your available memory. For more details on tuning your Mapnik stack for better performance, see the SotM 2010 session and follow up by Frederik Ramm of GEOFABRIK.

In the interim, let's have a look at your planet import. The first part of the osm2pgsql output looks scary, but is normal.

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping<br>Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192
Setting up table: planet_osm_nodes
NOTICE: table "planet_osm_nodes" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE: table "planet_osm_ways" does not exist, skipping
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Don't be concerned by the "NOTICE:" entries above. All normal.

Next, osm2pgsql will start reading the compressed planet file.

Reading in file: /home/nerd/planet/planet-100217.osm.bz2

As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.

Processing: Node(10140k) Way(0k) Relation(0k)

As the import proceeds, the "Node" number will update a couple of times per second until complete, then the "Way" number will update not quite so quickly, roughly every second or two. Finally the "Relation" number will update but at a slower rate, roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.

The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this.

Node stats: total(593072533), max(696096737)
Way stats: total(45376969), max(55410575)
Relation stats: total(484528), max(555276)

Going over pending ways
processing way (752k)

The "processing way" number should update approximately each second.

Going over pending relations

node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97%
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes
Building index on table: planet_osm_ways
Stopped table: planet_osm_rels
Completed planet_osm_point
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Stopped table: planet_osm_ways

This should mean that you import is complete and successful.

Install Mapnik library

The Mapnik library is the first of two items sometimes called "Mapnik". The other item is a collection of tools that OpenStreetMap uses to invoke Mapnik.

The official and up-to-date Mapnik Installation Instructions are here.

You might find that this procedure works as well.

Get some dependencies for building the Mapnik library.

sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev
sudo apt-get install libboost-python1.40-dev python-cairo-dev python-nose
sudo apt-get install libboost1.40-dev libboost-filesystem1.40-dev
sudo apt-get install libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev
sudo apt-get install libboost-program-options1.40-dev libboost-python1.40-dev
sudo apt-get install libfreetype6-dev libcairo2-dev libcairomm-1.0-dev
sudo apt-get install libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2
sudo apt-get install libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev
sudo apt-get install libgdal1-dev python-gdal
sudo apt-get install imagemagick ttf-dejavu

Build Mapnik library from source.

cd ~/src
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik
cd mapnik
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
python scons/scons.py
sudo python scons/scons.py install
sudo ldconfig

Confirm that Mapnik library is installed.

python
>>> import mapnik
>>>

If python replies with the second chevron prompt ">>>" and without errors, then Mapnik library was found by Python. Congratulations.

Install Mapnik tools

The Mapnik tools are the second item sometimes called "mapnik". This is a collection of tools from OpenStreetMap for making effective use of the Mapnik library.

cd ~/bin
svn co http://svn.openstreetmap.org/applications/rendering/mapnik

Install prepared world boundary data

Mapnik uses prepared files to generate coastlines and ocean for small scale maps. This is faster than reading the entire database to render zoom levels from zero to nine.

This section now includes the additional shape files that were added to OpenStreetMap default styles in mid-2010. Beware of the long, strange looking links with the repeated http. They are unlikely to copy / paste directly. Use copy link location or equivalent.

cd ~/bin/mapnik
mkdir world_boundaries
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
tar xvzf world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
tar xvjf processed_p.tar.bz2 -C world_boundaries
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
tar xjf shoreline_300.tar.bz2 -C world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip
unzip 10m-populated-places.zip -d world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip
unzip 110m-admin-0-boundary-lines.zip -d world_boundaries

Render your first map

The database is loaded and the tools are installed.

Customise generate_image.py with the following changes. This is used to create the XML file we'll use later next.

mapfile = "my_osm.xml"
bounds = (144.37, -38.0, 144.39, -38.2)
  • The bounds selected here are the latitude & longitude for the Geelong, Victoria region.

Create a customised OSM file (Remember to replace "username" with your username).

cd ~/bin/mapnik
./generate_xml.py osm.xml my_osm.xml --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --user username --accept-none

 Let's test everything together.

cd ~/bin/mapnik
./generate_xml.py --dbname gis --user username --accept-none
./generate_image.py

View image.png to confirm that you have rendered a map of Geelong. Congratulations.

Installing PostGIS, Mapnik

Follow the guide linked above after reading the amendments below. Its useful to ssh in and copy/paste in a line at a time.

  • I only downloaded Australian data, so my import was different. Also notice the -C parameter. This is the amount of RAM the import uses for caching, you might want to reduce it if your VM has less than 3GB of RAM.
cd ~/bin/osm2pgsql
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/australia.osm.bz2

Final amendment - before the last step of running ./generate_image.py, perform these steps:

  • Customise generate_image.py to use the XML file we'll create next. Mapnik has evolved and now uses templates. I also edited some of the other py files, like generate_tiles.py, but this isn't necessary. For generate_image.py, the changes are:
mapfile = "my_osm.xml"
bounds = (144.37, -38.0, 144.39, -38.2)
  • Create a customised OSM file.

This file tells mapnik (and renderd) about your data:

cd ~/bin/mapnik
./generate_xml.py osm.xml my_osm.xml --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --user rick --accept-none

Running ./generate_image.py now should create image.png showing a region in Victoria.

Web Server

Now we set up the tile server. The mod_tile module uses a multithreaded version of apache.

sudo apt-get install apache2 apache2-threaded-dev

Build Apache Tile Module & renderd

Make sure you are the user that can access the DB, not root.

cd ~/src
svn co http://svn.openstreetmap.org/applications/utils/mod_tile/
cd mod_tile
make
sudo make install
sudo mkdir /var/lib/mod_tile
sudo chown rick /var/lib/mod_tile
touch /var/lib/mod_tile/planet-import-complete
sudo mkdir /var/run/renderd
sudo chown rick /var/run/renderd

(if directories already exist, make sure they have write access to the user).

Edit renderd configuration

The make install above copied /etc/renderd.conf from the src/mod_tile directory. In this setup, both Apache and the renderd program use this configuration file.

sudo vim /etc/renderd.conf

Edit as follows, adjust the home directory as appropriate:

[renderd]
socketname=/var/run/renderd/renderd.sock
num_threads=4
tile_dir=/var/lib/mod_tile ; DOES NOT WORK YET
stats_file=/var/run/renderd/renderd.stats

[mapnik]
plugins_dir=/usr/local/lib/mapnik/input
font_dir=/home/rick/src/mapnik/fonts
font_dir_recurse=1

[default]
URI=/tiles/
XML=/home/rick/bin/mapnik/my_osm.xml
#HTCPHOST=proxy.openstreetmap.org

TIP: You can add alternate tilesets/layer configurations by adding another section eg:

[tiles2]
URI=/tiles2/
XML=/home/rick/bin/mapnik/my_osm2.xml

In my test I copied my_osm.xml to my_osm2.xml and disabled some layers by adding attribute status="off" to the Layer elements in the second half of the file.

Set up Apache configuration

Copy the configuration file for mod_tile then edit it:

cd ~/src/mod_tile
sudo cp mod_tile.conf /etc/apache2/mods-available
sudo vim /etc/apache2/mods-available/mod_tile.conf

The changes are in the first few lines:

  • comment this out, we'll load it separately
#LoadModule tile_module modules/mod_tile.so
  • change the servername and comment out the alias line
ServerName maps
  • update the document root
DocumentRoot /var/www/
  • enable the use of the config file in /etc
LoadTileConfigFile /etc/renderd.conf

Thats it for the edits.

Now create a load file for the module and create the links in mods-enabled.

sudo su root
echo LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so > /etc/apache2/mods-available/mod_tile.load
ln -s /etc/apache2/mods-available/mod_tile.load /etc/apache2/mods-enabled/mod_tile.load
ln -s /etc/apache2/mods-available/mod_tile.conf /etc/apache2/mods-enabled/mod_tile.conf
exit

You can now restart apache and check the module is loaded:

apache2ctl restart

Start renderd

For debugging, do this in a separate window - as regular user, not root.

~/src/mod_tile/renderd -f

We put this (without the -f) into rc.local later.

Testing

From a web browser, navigate to the address of your new server,

http://yourserver/mod_tile

You should see something like:

NoResp200: 0
NoResp304: 0
NoResp404: 0
NoResp503: 0
NoResp5XX: 0
NoRespOther: 0
NoFreshCache: 0
:

Now fetch a tile

http://yourserver/tiles/0/0/0.png

If things dont work, check:

  • Can renderd write to /var/lib/mod_tile and /var/run/renderd? Have files been created there?
  • Check apache's error log
  • Is font path correct in renderd.conf
  • Any activity in the renderd console window?

Starting Automatically

I inserted this in /etc/rc.local

mkdir /var/run/renderd
chown rick /var/run/renderd
sudo -u rick /home/rick/src/mod_tile/renderd

Adding a map to view tiles in a browser

Download and install OpenLayers:

cd ~
wget http://openlayers.org/download/OpenLayers-2.10.tar.gz
tar -xzvf OpenLayers-2.10.tar.gz
cd OpenLayers-2.10
sudo cp OpenLayers.js /var/www
sudo cp -r style /var/www
sudo cp -r theme /var/www
sudo cp examples/style.css /var/www
sudo cp examples/osm.html /var/www

Now customise osm.html to our site

sudo vim /var/www/osm.html

Change the layer = line to use our server instead of open street maps.

layer = new OpenLayers.Layer.OSM("RickTiles", "/tiles/${z}/${x}/${y}.png", {numZoomLevels: 19});

You can add: set layer.attribution="whatever"; if you like.

Change the default co-ordinate to somewhere in Australia

...new OpenLayers.LonLat(144.38, -38.10).transform(...

Now navigate to http://yourserver/osm.html and you should have a map.