class: center, middle, title # Imposm .center[<img src="imgs/opossum.png" alt="Opossum" style="width:30%"/>] ### The other OSM → PostGIS import tool .small[SotM-EU 2014, Karlsruhe] <br> .small[Oliver Tonnhofer `<tonnhofer@omniscale.de | @oltonn>`] --- # About me ### Oliver Tonnhofer * Omniscale GmbH & Co. KG, Oldenburg, Germany * Open Source development * MapProxy development and support * OpenStreetMap services --- # OpenStreetMap services - commercial OSM services since 2009 - custom WMS/tile services for - official German broadband atlas - telcos - energy supply companies - real estate management --- # maps.omniscale.com <img src="imgs/maps.omniscale.com.png" alt="maps.omniscale.com" style="width:30%; float: right;"/> <img src="imgs/osm.omniscale.de.png" alt="osm.omniscale.de" style="width:30%; float: right; clear: right;"/> - WMS and tile services - self-service - high-speed live rendering - any projection, any scale, HiDPI/retina - .small[(English website will follow)] --- # History - Built first OSM based demo for MapProxy in 2008 - Started with osm2pgsql + OSM Mapnik XML style - Soon wanted our *own* maps --- # osm2pgsql .center[Built for openstreetmap.org,<br>not to solve everyone's problems.] -- - Slow for medium imports - Not flexible enough - Lots of room for improvements --- # Write a new tool to <br>IMPort OSM?! .center[How hard could it be?™] -- .center[Harder than we thought!] -- - But we got it working: Imposm 1 - Released as OpenSource in 2011: Imposm 2 - Now working on Imposm 3 --- # Design goals for Imposm - Efficient import - Time and memory - Custom database schema - Optimized for rendering --- class: center, middle # Time efficient import --- class: center, middle .center[<img src="imgs/cpu.png" alt="CPU" style="width:90%"/>] --- # Multi-CPU support - osm2pgsql: was single threaded, some parts are now parallel - Imposm: every thing is parallel - parsing - geometry building - importing --- class: center, middle # Memory efficient import --- # Planet file .muchcode[```xml <node id="1" lat="45" lon="8"/> <node id="2" lat="45" lon="9"/> <node id="3" lat="46" lon="10"/> ... <node id="2902437215" lat="-105" lon="30"/> ```] -- .muchcode[```xml <way id="1"> <nd ref="15684965"/> <nd ref="265654823"/> <nd ref="6548221"/> <tag k="railway" v="rail"/> </way> ... ```] -- - Need to access referenced nodes in random order - XML/PBF do not support random access --- # Node cache - Cache nodes in random access storage - in memory (dictionary/hash map) - key-value store - database .small[Also applies to ways (for building relations).] --- # osm2pgsql - normal mode: cache in memory - slim mode: store in PostgreSQL - flat-nodes: *new* mode, like normal mode but on disk --- # osm2pgsql (normal mode) - 22GB RAM for current planet file - stores coordinates in blocks of 65k nodes - empty blocks are not allocated - small extracts - larger extracts will fill most blocks with at least one node --- # osm2pgsql (slim mode) - Cache all nodes in PostgreSQL table - Low RAM usage - High IO and disk usage - Slow compared to memory cache or modern key-value storages --- # Imposm 2 nodes cache - Tokyo Cabinet key-value store - C-library - No server (like SQLite) - Additional optimizations - Compression - Delta encoding --- # Imposm 2 nodes cache - 8-10 bytes per node (~2GB for Germany) - Fast write speed (+200k-500k nodes/s) --- class: center, middle # Custom database schema --- # osm2pgsql DB schema - planet_osm_point - planet_osm_line - planet_osm_polygon - (planet_osm_roads) --- # osm2pgsql DB schema planet_osm_polygon: | id | landuse | building | leisure | waterway | *more* | *tags* | way | |:--:|---------|----------|---------|-----------|--------|--------|----------------| | 1 | | yes | | | | | `POLYGON(...)` | | 2 | park | | | | | | `POLYGON(...)` | | 3 | | | stadium | | | | `POLYGON(...)` | | 4 | | | | riverbank | | | `POLYGON(...)` | | 5 | forest | | | | | | `POLYGON(...)` | | 6 | | yes | | | | | `POLYGON(...)` | | 7 | | | | riverbank | | | `POLYGON(...)` | | ...| ... | ... | ... | ... | | | `POLYGON(...)` | -- Need to render all forests in level 5, 6 or 7?<br> -- → Table scan of polygon table to select forest from all other polygons. --- # Imposm 2 DB schema - Define DB schema with Python - Select `key=values` for each table -- ```python Points( name = 'drinks', mapping = { 'amenity': ('pub', 'bar') } ) ``` --- # Imposm 2 DB schema - support wildcards (`__any__`) - add other properties (fields) - filter/convert values - convert yes/true/1/false/no/0 to booleans .muchcode[ ```python LineStrings( name = ‘roads’, mapping = { ‘highway’: ‘__any__’, }, fields = ( ('name', StreetNames()), ('tunnel', Bool()), ('bridge', Bool()), ('oneway', Direction()), ) ``` ] --- # Custom filter .muchcode[ ```python Points( name = 'towers', mapping = { 'man_made': ( 'tower', 'water_tower', ) } fields = ( ('height', Meters()), ) ) ``` ] - 10m → 10.0 - 30ft → 9.144 --- class: center, middle # Optimized for rendering --- # Generalizations .center[<img src="imgs/generalizations.png" alt="generalizations" style="width:90%"/>] --- # IO Optimizations .center[<img src="imgs/clustering1.jpeg" alt="cluster" style="width:90%"/>] .copyright[Cluster graphics CC-By-NC-SA 3.0, Boundless] --- # IO Optimizations .center[<img src="imgs/clustering2.jpeg" alt="cluster" style="width:90%"/>] .copyright[Cluster graphics CC-By-NC-SA 3.0, Boundless] --- # IO Optimizations .center[<img src="imgs/buildings.png" alt="buildings" style="width:90%"/>] --- # PostgreSQL `CLUSTER` - Reorders data on disk - Order based on index --- # Spatial index .center[<img src="imgs/clustering3.jpeg" alt="cluster" style="width:90%"/>] .copyright[Cluster graphics CC-By-NC-SA 3.0, Boundless] --- # GeoHash index .center[<img src="imgs/clustering4.jpeg" alt="cluster" style="width:90%"/>] .copyright[Cluster graphics CC-By-NC-SA 3.0, Boundless] --- # Limit to .center[<img src="imgs/limitto.png" alt="limitto" style="width:60%"/>] --- # Design goals for Imposm - Efficient import - Efficient cache - Multi-CPU support -- - Custom database schema - Flexible mapping configuration -- - Optimized for rendering - Custom database schema - Generalized tables - Clustering --- # What's missing? - Update (diff) support - More performance --- # Performance - Imposm 2 is written in Python - CPU heavy stuff already in C - No *low hanging fruits* left --- # Imposm 3 - Rewrite from scratch - Shares most of the architecture - Optimize/reconsider all parts --- # Imposm 3 - Implemented in Go - Much faster than Python - Easy to learn - Easy C integration - Simple deployment --- # Imposm 3 nodes cache - Based on (Hyper)LevelDB instead of Tokyo Cabinet - HyperLevelDB is a LevelDB fork with optimizations for larger datasets. - Low level optimizations to save memory/reduce IO - Delta encoding - Encode common tags as single Unicode<br> `highway=residential` → U+E001 - High performance - 8 bytes per node - Fast write speed >1-4.000.000 nodes/s --- # PostgreSQL - `COPY FROM` bulk import - Use schema names instead of `osm_new_` prefix for import - Less fragile than renaming tables/indices/sequences --- # Diff support ```xml <create> <node id="1" lat="45" lon="8"/> <node id="2" lat="45" lon="9"/> <way id="1"> <nd ref="1"/> <nd ref="2"/> <tag k="railway" v="rail"/> </way> </create> ``` Regular OSM XML wrapped in `create`, `modify` and `delete` actions. --- # Diff support Update pub location? ```xml <modify> <node id="3" lat="53" lon="7"> <tag k="amenity" v="pub"/> </node> </modify> ``` Update point with ID 3 in DB. --- # Diff support Modify way geometry? ```xml <modify> <node id="2" lat="45" lon="7"/> </modify> ``` Update line string in DB, but which one? --- # Diff support Additional cache files with reverse index: - node ID → way IDs - way ID → relation IDs ``imposm3 diff``: - removes all deleted/modified elements - updates cache and reverse index - (re)inserts modifies/creates elements --- # Diff support - Handles generalized tables - Not performance optimized, but fast enough for now - Processing minutely updates (Europe extract): - worst case 20s - 95% < 5s - 80% < 1s - 65% < 0.5s --- # Imposm 3 Mapping - JSON instead of Python - Most Imposm 2 mappings can be converted to identical mapping - No custom filters possible --- # Current status of Imposm3 - In production >9m (>3m w/ diff) - Mapping might change - No documentation - blocker for official 3.0 --- # Future - HStore/JSON support - Other databases - (Lua?) scripting for custom filters --- # Imposm 2? - Maintenance mode - No new features planned --- # Summary - osm2pgsql: - works well for openstreetmap.org - Imposm: - more flexible for other use-cases - implements lots of optimizations - much faster, esp. for extracts and slower hardware --- name: fin # Fin .center[Thanks! Questions?] <br> <br> <br> * Oliver Tonnhofer * <img src="imgs/omniscale_logo_4c_cymk.png" alt="Omniscale" style="display:inline; height: 1.4em;"/> * `tonnhofer@omniscale.de` * `@oltonn`