pbf

README

This is a library for extracting OSM entities from PBF files. It is intended to be used for simple one-off tasks, searching an area of no more than a few square kilometers, where setting up a PostgreSQL database would be a disproportionate effort.

It performs reasonably well when working with OSM extracts, such as the ones obtained from download.geofabrik.de. With modest hardware (e.g. an old ThinkPad T480) PBF files can be read at roughly 80MiB/s to 300MiB/s. When reading through an extract of the Czech Republic (828MiB), roughly 320MiB of RAM are used (but this will depend on the amount of cores on your CPU).

Performance can be improved, by changing the compression inside PBF files to zstd. This can be done with the zstd-pbf tool.

Find the full documentation of this library at godocs.io/github.com/codesoap/pbf.

Example

filter := pbf.Filter{
	Location: func(lat, lon int64) bool {
		// A square filter matching the city center of Bremen, Germany.
		return lat >= 53_071_495_496 &&
			lat <= 53_080_504_504 &&
			lon >= 8_799_510_372 &&
			lon <= 8_814_489_628
	},
	Tags: map[string][]string{
		// Find bicycle shops.
		"shop": {"bicycle"},
	},
}
// wget https://download.geofabrik.de/europe/germany/bremen-latest.osm.pbf
results, err := pbf.ExtractEntities("/tmp/bremen-latest.osm.pbf", filter)
resultCount := len(results.Nodes) + len(results.Ways) + len(results.Relations)
fmt.Printf("Found %d bicycle shop(s) in the center of Bremen.\n", resultCount)

Development setup

To generate some protobuf related code, you need the protoc tool, the protoc-gen-go tool and the protoc-gen-go-vtproto tool; the latter two can be installed like this:

go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto@v0.6.0

Ideas for the Future

For now, ways and relations will be incomplete, if they only partially lie within the location filter or their members didn't match the tag filter. This is undesirable, for example, when trying to render an extracted area as an image. Thus it would be useful to have a flag that makes the library return "ancillary entities". An idea for the interface can be found at commit 9b673e35bd0510e4ad53a2875dcf4a7ea4ca085a.

Documentation

Overview

package extractor provides functions for extracting OSM entities within a given area from PBF files. The entities can further be filtered by their tags.

The entities, as well as the filter options, do not contain all available data and metadata in order to simplify the interface and reduce resource use.

Index

• type Entities
  • func ExtractEntities(pbfFile string, filter Filter) (Entities, error)
• type Entity
• type Filter
• type LocationFilter
• type Node
  • func (n Node) Coords() (lat, lon int64)
  • func (n Node) ID() int64
  • func (n Node) Tags() map[string]string
• type Relation
  • func (r Relation) ID() int64
  • func (r Relation) Nodes() []int64
  • func (r Relation) Relations() []int64
  • func (r Relation) Tags() map[string]string
  • func (r Relation) Ways() []int64
• type Way
  • func (w Way) ID() int64
  • func (w Way) Nodes() []int64
  • func (w Way) Tags() map[string]string

Types

type Entities

type Entities struct {
	Nodes     map[int64]Node     // Nodes by their ID.
	Ways      map[int64]Way      // Ways by their ID.
	Relations map[int64]Relation // Relations by their ID.
}

func ExtractEntities

func ExtractEntities(pbfFile string, filter Filter) (Entities, error)

ExtractEntities extracts all entities matching filter from the given pbfFile.

type Entity

type Entity interface {
	ID() int64
	Tags() map[string]string
}

An Entity is an OSM entity: either a Node, Way or Relation.

type Filter

type Filter struct {
	// Location filters results by geographic location. If it is nil, this
	// filter is inactive.
	//
	// Note that ways and relations which only pass through Location, but
	// have no nodes there, will be missed.
	//
	// Example:
	//     myFilter.Location = func(lat, lon int64) bool {
	//         return lat >= 50_000_000_000 &&
	//             lat <= 50_010_000_000 &&
	//             lon >= 10_000_000_000 &&
	//             lon <= 10_010_000_000
	//     }
	Location LocationFilter

	// The Tags filter will filter entities by their tags. Keys are tags
	// and values are accepted values. If a value is an empty slice, every
	// value for this tag is accepted, but a value for this tag must be
	// present.
	Tags map[string][]string

	// If ExcludePartial is true, the filter will not match ways and
	// relations, where only some of their nodes lay within Location. This
	// can improve performance.
	ExcludePartial bool
}

Filter is a filter for OSM entities.

type LocationFilter

type LocationFilter func(lat, lon int64) bool

LocationFilter is a function that takes a latitude and longitude in nanodegrees and returns true if the given coordinates match the filter.

type Node

type Node struct {
	// contains filtered or unexported fields
}

func (Node) Coords

func (n Node) Coords() (lat, lon int64)

func (Node) ID

func (n Node) ID() int64

func (Node) Tags

func (n Node) Tags() map[string]string

type Relation

type Relation struct {
	// contains filtered or unexported fields
}

func (Relation) ID

func (r Relation) ID() int64

func (Relation) Nodes

func (r Relation) Nodes() []int64

func (Relation) Relations

func (r Relation) Relations() []int64

func (Relation) Tags

func (r Relation) Tags() map[string]string

func (Relation) Ways

func (r Relation) Ways() []int64

type Way

type Way struct {
	// contains filtered or unexported fields
}

func (Way) ID

func (w Way) ID() int64

func (Way) Nodes

func (w Way) Nodes() []int64

func (Way) Tags

func (w Way) Tags() map[string]string