gosmparse

README

OpenStreetMap PBF Parser in Go

gosmparse uses a callback driven API, which is stable (Documentation).

It has been designed with performance and maximum usage convenience in mind; on an Intel Core i7-6820HQ with NVMe flash it is able to process ~75 MB/s, so a planet file can be processed in under 10 minutes. If you find possible speed-ups or other improvements, let me know.

Characteristics

• fast
• panic-free
• tested with different files from different sources/generators
• more than 80% test coverage and has benchmarks for all hot spots
• one dependency only: gogo protobuf package (a few more are used by tests and are included in the module)
• can read from any io.Reader (e.g. for parsing during download)
• supports history files

Install

go get -u github.com/mattes/gosmparse

Example Usage

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func ExampleNewDecoder() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}

Download & Parse

It is possible to parse during download, so you don't have to wait for a download to finish to be able to start the parsing/processing. You can simply use the standard Go net/http package and pass resp.Body to the decoder.

resp, err := http.Get("http://download.geofabrik.de/europe/germany/bremen-latest.osm.pbf")
if err != nil {
	panic(err)
}
defer resp.Body.Close()
dec := gosmparse.NewDecoder(resp.Body)
err = dec.Parse(&dataHandler{})
if err != nil {
	panic(err)
}

Did it break?

If you found a case, where gosmparse broke, please report it and provide the file that caused the failure.

Documentation

Overview

Package gosmparse is a library for parsing OpenStreetMap binary PBF files.

It has been designed for very fast, flexible, streamed parsing of small and large files.

Index

• type Decoder
  • func NewDecoder(r io.Reader) *Decoder
  • func NewDecoderWithInfo(r io.Reader) *Decoder
  • func (d *Decoder) Parse(o OSMReader) error
  • func (d *Decoder) ProcessedBytes() uint64
• type Element
• type Info
• type MemberType
• type Node
• type OSMReader
• type Relation
• type RelationMember
• type Way

Examples

• NewDecoder

Types

type Decoder

type Decoder struct {
	// QueueSize allows to tune the memory usage vs. parse speed.
	// A larger QueueSize will consume more memory, but may speed up the parsing process.
	QueueSize int
	Workers   int
	// contains filtered or unexported fields
}

A Decoder reads and decodes OSM data from an input stream.

func NewDecoder

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a new decoder that reads from r.

Example

package main

import (
	"os"

	"github.com/mattes/gosmparse"
)

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func main() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}

func NewDecoderWithInfo

func NewDecoderWithInfo(r io.Reader) *Decoder

NewDecoderWithInfo returns a new decoder similar to NewDecoder, but will populate the Info field in the elements. Use this if you need meta data.

func (*Decoder) Parse

func (d *Decoder) Parse(o OSMReader) error

Parse starts the parsing process that will stream data into the given OSMReader.

func (*Decoder) ProcessedBytes

func (d *Decoder) ProcessedBytes() uint64

ProcessedBytes returns and resets processed bytes

type Element

type Element struct {
	ID   int64
	Tags map[string]string

	// Info is only populated if you use NewDecoderWithInfo.
	Info *Info
}

Element contains common attributes of an OSM element (node/way/relation).

type Info

type Info struct {
	Version   int
	Timestamp time.Time
	Changeset int64
	UID       int
	User      string
	Visible   bool
}

Info contains the metadata of an element.

type MemberType

type MemberType int

MemberType describes the type of a relation member (node/way/relation).

const (
	NodeType MemberType = iota
	WayType
	RelationType
)

type Node

type Node struct {
	Element
	Lat float64
	Lon float64
}

Node is an OSM data element with a position and tags (key/value pairs).

type OSMReader

type OSMReader interface {
	ReadNode(Node)
	ReadWay(Way)
	ReadRelation(Relation)
}

OSMReader is the interface that needs to be implemented in order to receive Elements from the parsing process.

type Relation

type Relation struct {
	Element
	Members []RelationMember
}

Relation is an OSM data element that contains multiple elements (RelationMember) and has tags (key/value pairs).

type RelationMember

type RelationMember struct {
	ID   int64
	Type MemberType
	Role string
}

RelationMember refers to an element in a relation. It contains the ID of the element (node/way/relation) and the role.

type Way

type Way struct {
	Element
	NodeIDs []int64
}

Way is an OSM data element that consists of Nodes and tags (key/value pairs). Ways can describe line strings or areas.