gosm

README

Gosm

Gosm is a golang library which implements writing OSM pbf files. The initial idea is that when we use osmosis tool to dump data from database to pbf file, it causes high load of the database during that time. We need to maintain a large size of database just for the dump. For osmosis, we do not have control on how to query the database, so we implemented this library in golang. With this library, we can do the database query by ourselves during the data dump phase and we managed to reduce half size of the original database to finish the data dump.

So if you are looking for a full control during the data dump in your golang code, gosm is your choice. This is running in our production usage and under a stable version right now.

Installation

go get github/grab/gosm

Usage

Quick start

	// initialize an encoder
	wc := &myWriter{}
	encoder := NewEncoder(&NewEncoderRequiredInput{
		RequiredFeatures: []string{"OsmSchema-V0.6"},
		Writer:           wc,
	},
		WithWritingProgram("example"),
		WithZlipEnabled(false),
	)
	errChan, err := encoder.Start()

	// write some nodes
	nodes := []*Node{
		{
			ID: 7278995748,
			Latitude:  -7.2380901,
			Longitude: 112.6773289,
			Tags: map[string]string{
				"node": "node1",
				"erp":  "no",
			},
		},
		{
			ID: 6978510772,
			Latitude:  -7.2381273,
			Longitude: 112.6775354,
		},
	}
	encoder.AppendNodes(nodes)
	encoder.Close()

End to end demo

This is not a fully runnable code, would like to demo how to use the library.

You can replace oriNodes, oriWays, oriRelations with your own data.

	encoder := gosm.NewEncoder(&gosm.NewEncoderRequiredInput{
		RequiredFeatures: []string{"OsmSchema-V0.6", "DenseNodes"},
		Writer:           f,
	},
		gosm.WithWritingProgram("wp1"),
		gosm.WithZlipEnabled(true),
	)
	defer func() {
		_ = encoder.Close()
	}()

	errChan, err := encoder.Start()
	if err != nil {
		return err
	}

	var errs []error
	go func() {
		for e := range errChan {
			errs = append(errs, e)
		}
	}()

    sorts := make([]*Node, len(oriNodes))
	i := 0
	for _, node := range oriNodes {
		sorts[i] = node
		i++
	}
	
	// write nodes order by id 
	sort.Slice(sorts, func(i, j int) bool {
		return sorts[i].ID < sorts[j].ID
	})

    // at most write 8000 nodes in one batch
	nodes := make([]*gosm.Node, 0, gosmWriteElementsMax)
	count := 0
	for _, n := range sorts {
		nodes = append(nodes, n.ToGosmNode())
		count++
		if count == gosmWriteElementsMax {
			encoder.AppendNodes(nodes)
			nodes = make([]*gosm.Node, 0, gosmWriteElementsMax)
			count = 0
		}
	}
	if len(nodes) > 0 {
		encoder.AppendNodes(nodes)
	}
	// remember to flush
	encoder.Flush(gosm.NodeType)

    // write ways order by way id
	sorts := make([]*Way, len(oriWays))
	i := 0
	for _, way := range oriWays {
		sorts[i] = way
		i++
	}
	sort.Slice(sorts, func(i, j int) bool {
		return sorts[i].ID < sorts[j].ID
	})

	ways := make([]*gosm.Way, 0, gosmWriteElementsMax)
	count := 0
	for _, w := range sorts {
		ways = append(ways, w.ToGosmWay())
		count++
		if count == gosmWriteElementsMax {
			encoder.AppendWays(ways)
			ways = make([]*gosm.Way, 0, gosmWriteElementsMax)
			count = 0
		}
	}
	if len(ways) > 0 {
		encoder.AppendWays(ways)
	}
	encoder.Flush(gosm.WayType)
	
	// write relations by relation id
	sorts := make([]*Relation, len(oriRelations))
	i := 0
	for _, rel := range oriRelations {
		sorts[i] = rel
		i++
	}
	sort.Slice(sorts, func(i, j int) bool {
		return sorts[i].ID < sorts[j].ID
	})

	relations := make([]*gosm.Relation, 0, gosmWriteElementsMax)
	count := 0
	for _, r := range sorts {
		relations = append(relations, r.ToGosmRelation())
		count++
		if count == gosmWriteElementsMax {
			encoder.AppendRelations(relations)
			relations = make([]*gosm.Relation, 0, gosmWriteElementsMax)
			count = 0
		}
	}
	if len(relations) > 0 {
		encoder.AppendRelations(relations)
	}

	if len(errs) > 0 {
		...
	}

Notes

1. When you use AppendNodes, AppendWays or AppendRelations, at most 8000 items are allowed in one append operation
2. Use encoder.Flush(memberType MemberType) when you finished writing of one member type.

Contributing

Please raise an issue with your problem or new ideas. MRs are not accepted so far.

License

We are using MIT license. Please check LICENSE.md

Documentation

Index

• type Encoder
  • func NewEncoder(input *NewEncoderRequiredInput, opts ...Option) *Encoder
  • func (e *Encoder) AppendNodes(nodes []*Node)
  • func (e *Encoder) AppendRelations(relations []*Relation)
  • func (e *Encoder) AppendWays(ways []*Way)
  • func (e *Encoder) Close() error
  • func (e *Encoder) Flush(memberType MemberType)
  • func (e *Encoder) Start() (chan error, error)
• type Member
• type MemberType
• type NewEncoderRequiredInput
• type Node
• type Option
  • func WithBbox(b *gosmpb.HeaderBBox) Option
  • func WithLogger(l logger) Option
  • func WithOptionalFeatures(optionalFeatures []string) Option
  • func WithWritingProgram(w string) Option
  • func WithZlipEnabled(v bool) Option
• type Relation
• type Way

Types

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder contains all the needed context when generating pbf file.

func NewEncoder

func NewEncoder(input *NewEncoderRequiredInput, opts ...Option) *Encoder

NewEncoder initializes an OSM pbf encoder.

func (*Encoder) AppendNodes

func (e *Encoder) AppendNodes(nodes []*Node)

AppendNodes will append nodes to the buffer, when it meets the limit(8000 entities or 32MB) it will convert the nodes to dense nodes and write to the writer.

func (*Encoder) AppendRelations

func (e *Encoder) AppendRelations(relations []*Relation)

AppendRelations writes the relation data to the buffer.

func (*Encoder) AppendWays

func (e *Encoder) AppendWays(ways []*Way)

AppendWays writes OSM way data to buffer.

func (*Encoder) Close

func (e *Encoder) Close() error

Close will stop consuming the channel and close the writer.

func (*Encoder) Flush

func (e *Encoder) Flush(memberType MemberType)

Flush consume the remaining data from the buffer immediately and writes to the pbf file.

func (*Encoder) Start

func (e *Encoder) Start() (chan error, error)

Start will write the header file to the writer and start consuming data channel and write to the writer.

type Member

type Member struct {
	ID   int64
	Type MemberType
	Role string
}

Member is used to describe the entity in one relation.

type MemberType

type MemberType int

MemberType enum

const (
	// NodeType ...
	NodeType MemberType = iota
	// WayType ...
	WayType
	// RelationType ...
	RelationType
)

type NewEncoderRequiredInput

type NewEncoderRequiredInput struct {
	RequiredFeatures []string
	Writer           io.WriteCloser
}

NewEncoderRequiredInput contains the required parameters to initialize an encoder

type Node

type Node struct {
	ID        int64
	Latitude  float64
	Longitude float64
	Tags      map[string]string
}

Node is the struct for OSM node.

type Option

type Option func(e *Encoder)

Option type provides different options when initializing the encoder.

func WithBbox

func WithBbox(b *gosmpb.HeaderBBox) Option

WithBbox provides the pbf bbox information in the header block, just an information, does not limit the writing if the data is out of the bbox.

func WithLogger

func WithLogger(l logger) Option

WithLogger sets a logger.

func WithOptionalFeatures

func WithOptionalFeatures(optionalFeatures []string) Option

WithOptionalFeatures sets optional features for the encoder

func WithWritingProgram

func WithWritingProgram(w string) Option

WithWritingProgram sets the writing program, does not impact the encoding behaviour.

func WithZlipEnabled

func WithZlipEnabled(v bool) Option

WithZlipEnabled enabled zlib when writing data to pbf file.

type Relation

type Relation struct {
	ID      int64
	Tags    map[string]string
	Members []Member
	Info    gosmpb.Info
}

Relation is the struct for OSM relation.

type Way

type Way struct {
	ID      int64
	Tags    map[string]string
	NodeIDs []int64
}

Way is the OSM way struct.