gosmparse

README

OpenStreetMap PBF Parser in Go

Gosmparse works already, but the API may change (Documentation).

It has been designed with performance and maximum usage convenience in mind; on an Intel Core i7-6820HQ with NVMe flash it is able to process 67 MB/s, so a planet file can be processed in less than 10 minutes. If you find possible speed-ups or other improvements, let me know.

Characteristics

• fast
• panic-free
• tested with different files from different sources/generators
• more than 80% test coverage and has benchmarks for all hot spots
• one dependency only: Go protobuf package
• can read from any io.Reader (e.g. for parsing during download)

Install

go get -u github.com/missinglink/gosmparse

Example Usage

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func ExampleNewDecoder() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}

Download & Parse

It is possible to parse during download, so you don't have to wait for a download to finish to be able to start the parsing/processing. You can simply use the standard Go net/http package and pass resp.Body to the decoder.

resp, err := http.Get("http://download.geofabrik.de/europe/germany/bremen-latest.osm.pbf")
if err != nil {
	panic(err)
}
defer resp.Body.Close()
dec := gosmparse.NewDecoder(resp.Body)
err = dec.Parse(&dataHandler{})
if err != nil {
	panic(err)
}

Did it break?

If you found a case, where gosmparse broke, please report it and provide the file that caused the failure.

Documentation

Overview

Package gosmparse is a library for parsing OpenStreetMap binary PBF files.

It has been designed for very fast, flexible, streamed parsing of small and large files.

Index

• func FeatureEnabled(flag string) bool
• type BlobIndex
  • func (i *BlobIndex) BlobOffsets(memtype string, id int64) ([]int64, error)
  • func (i *BlobIndex) FirstOffsetOfType(memtype string) (int64, error)
  • func (i *BlobIndex) ReadFrom(tap io.Reader) (int64, error)
  • func (i *BlobIndex) ReadFromFile(path string)
  • func (i *BlobIndex) SetBreakpoints()
  • func (i *BlobIndex) WriteTo(sink io.Writer) (int64, error)
  • func (i *BlobIndex) WriteToFile(path string)
• type BlobInfo
• type Decoder
  • func NewDecoder(r *os.File) *Decoder
  • func (d *Decoder) AutoSaveIndex()
  • func (d *Decoder) AutoloadIndex()
  • func (d *Decoder) Parse(o OSMReader, skipHeaderCheck bool) error
  • func (d *Decoder) ParseBlob(o OSMReader, offset int64) error
  • func (d *Decoder) SeekToOffset(offset int64)
• type GroupInfo
• type MemberType
• type Node
• type OSMReader
• type Relation
• type RelationMember
• type Way

Examples

• NewDecoder

Functions

func FeatureEnabled

func FeatureEnabled(flag string) bool

FeatureEnabled - return true if feature is enabled, else false

Types

type BlobIndex

type BlobIndex struct {
	Blobs       []*BlobInfo
	Breakpoints []uint64
}

BlobIndex - an index of all blocks in the file

func (*BlobIndex) BlobOffsets

func (i *BlobIndex) BlobOffsets(memtype string, id int64) ([]int64, error)

BlobOffsets - find the start offset of blob(s) containing desired element

func (*BlobIndex) FirstOffsetOfType

func (i *BlobIndex) FirstOffsetOfType(memtype string) (int64, error)

FirstOffsetOfType - find the first offset of blob of desired type

func (*BlobIndex) ReadFrom

func (i *BlobIndex) ReadFrom(tap io.Reader) (int64, error)

ReadFrom - read from destination

func (*BlobIndex) ReadFromFile

func (i *BlobIndex) ReadFromFile(path string)

ReadFromFile - read from disk

func (*BlobIndex) SetBreakpoints

func (i *BlobIndex) SetBreakpoints()

SetBreakpoints - set the breakpoints for node/way/relation boundaries

func (*BlobIndex) WriteTo

func (i *BlobIndex) WriteTo(sink io.Writer) (int64, error)

WriteTo - write to destination

func (*BlobIndex) WriteToFile

func (i *BlobIndex) WriteToFile(path string)

WriteToFile - write to disk

type BlobInfo

type BlobInfo struct {
	Groups []*GroupInfo
	Start  uint64
	Size   uint64
}

BlobInfo - store info about each block

type Decoder

type Decoder struct {
	// QueueSize allows to tune the memory usage vs. parse speed.
	// A larger QueueSize will consume more memory, but may speed up the parsing process.
	QueueSize int

	Mutex     *sync.Mutex
	BytesRead uint64
	Index     *BlobIndex
	Triggers  []func(int, uint64)
	// contains filtered or unexported fields
}

A Decoder reads and decodes OSM data from an input stream.

func NewDecoder

func NewDecoder(r *os.File) *Decoder

NewDecoder returns a new decoder that reads from r.

Example

package main

import (
	"os"

	"github.com/missinglink/gosmparse"
)

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func main() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}

func (*Decoder) AutoSaveIndex

func (d *Decoder) AutoSaveIndex()

AutoSaveIndex - automatically save index file if feature is enabled

func (*Decoder) AutoloadIndex

func (d *Decoder) AutoloadIndex()

AutoloadIndex - automatically load index file if one if available

func (*Decoder) Parse

func (d *Decoder) Parse(o OSMReader, skipHeaderCheck bool) error

Parse starts the parsing process that will stream data into the given OSMReader.

func (*Decoder) ParseBlob

func (d *Decoder) ParseBlob(o OSMReader, offset int64) error

ParseBlob - parse a single blob

func (*Decoder) SeekToOffset

func (d *Decoder) SeekToOffset(offset int64)

SeekToOffset move read pointer to byte offset

type GroupInfo

type GroupInfo struct {
	Type  string
	Count int
	High  int64
	Low   int64
}

GroupInfo - store info about each group

type MemberType

type MemberType int

MemberType describes the type of a relation member (node/way/relation).

const (
	NodeType MemberType = iota
	WayType
	RelationType
)

type Node

type Node struct {
	ID   int64
	Lat  float64
	Lon  float64
	Tags map[string]string
}

Node is an OSM data element with a position and tags (key/value pairs).

type OSMReader

type OSMReader interface {
	ReadNode(Node)
	ReadWay(Way)
	ReadRelation(Relation)
}

OSMReader is the interface that needs to be implemented in order to receive Elements from the parsing process.

type Relation

type Relation struct {
	ID      int64
	Members []RelationMember
	Tags    map[string]string
}

Relation is an OSM data element that contains multiple elements (RelationMember) and has tags (key/value pairs).

type RelationMember

type RelationMember struct {
	ID   int64
	Type MemberType
	Role string
}

RelationMember refers to an element in a relation. It contains the ID of the element (node/way/relation) and the role.

type Way

type Way struct {
	ID      int64
	NodeIDs []int64
	Tags    map[string]string
}

Way is an OSM data element that consists of Nodes and tags (key/value pairs). Ways can describe line strings or areas.