ecpush

README

ecpush

ecpush is a Go package for subscribing to real-time meteorological data feeds from Environment Canada.

Documentation

Please visit the corresponding GoDoc entry for all necessary documentation, including parameter definitions and program defaults.

Goals

The main goal of ecpush is to provide a simple and lightweight client that can be used for receiving real-time data events directly from Environment Canada's meteorological product feed.

The client can directly fetch the published products, or it can just provide a notification channel containing the product location (HTTP URL to Environment Canada's Datamart). The client has also been designed to automatically recover from any connection or channel interruptions.

Usage

To create a new client, create a Client struct. The only required field is the Subtopics array. Default values for other fields are listed in the struct definition. An example configuration is shown below (subscribing text bulletins, citypage XML and CAP alert files).

Please see subtopic amqp pattern for formatting subtopics.

client := &ecpush.Client{
    Subtopics: &[]string{
        "alerts.cap.#",
        "bulletins.alphanumeric.#",
        "citypage_weather.xml.#",
    },
    DisableEventLog: false,
    FetchContent:    false,
}

Calling Connect(ctx) will return an error if no subtopics are provided. The function will block until the initial connection with the remote server is established. When the client is provisioned, an internal Goroutine is created to consume the feed.

// create context for closing client
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)

err := client.Connect(ctx)
if err != nil {
    panic(err)
}

To consume the events, call Consume() on the client. This returns an Event and an indicator if the client is still actively consuming from the remote server.

for {
    event, closed := client.Consume()
    if closed {
        // not actively consuming
        return
    }
    log.Println(event)
}

To close the client, call the cancel function on the context provided to the client. This will gracefully close the active channels and connection to the remote server.

close()

Examples

A fully functioning client can be found in the example directory.

Acknowledgements

I would like to thank Sean Treadway for his Go RabbitMQ client package. I would also like to thank Environment Canada and the awesome people at Shared Services Canada for their developments and "openness" of MetPX and sarracenia.

License

Copyright (c) 2019 Tanner Ryan. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Sean Treadway's Go RabbitMQ client package is under a BSD 2-clause license. Cenk Alti's Go exponential backoff package is under an MIT license. Once again, all rights reserved.

Documentation

Overview

Package ecpush is a package for subscribing to real-time meteorological data feeds from Environment Canada.

Goals

The main goal of ecpush is to provide a simple and lightweight client that can be used for receiving real-time data events directly from Environment Canada's meteorological product feed.

The client can directly fetch the published products, or it can just provide a notification channel containing the product location (HTTP URL to Environment Canada's Datamart). The client has also been designed to automatically recover from any connection or channel interruptions.

Usage

To create a new client, create a Client struct. The only required field is the Subtopics array. Default values for other fields are listed in the struct definition. An example configuration is shown below (subscribing text bulletins, citypage XML and CAP alert files).

Please see https://eccc-msc.github.io/open-data/msc-datamart/readme_en/ for formatting subtopics.

    client := &ecpush.Client{
        Subtopics: &[]string{
			"*.WXO-DD.citypage_weather.ON.#",
			"*.WXO-DD.bulletins.alphanumeric.#",
        },
        DisableEventLog: false,
        FetchContent:    false,
    }

Calling Connect(ctx) will return an error if no subtopics are provided. The function will block until the initial connection with the remote server is established. When the client is provisioned, an internal Goroutine is created to consume the feed.

// create context for closing client
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)

err := client.Connect(ctx)
if err != nil {
    panic(err)
}

To consume the events, call Consume() on the client. This returns an Event and an indicator if the client is still actively consuming from the remote server.

for {
    event, closed := client.Consume()
    if closed {
        // not actively consuming
        return
    }
    log.Println(event)
}

To close the client, call the cancel function on the context provided to the client. This will gracefully close the active channels and connection to the remote server.

close()

Examples

A fully functioning client can be found in the example directory.

Acknowledgements

I would like to thank Sean Treadway for his Go RabbitMQ client package. I would also like to thank Environment Canada and the awesome people at Shared Services Canada for their developments and "openness" of MetPX and sarracenia.

License

Copyright (c) 2019 Tanner Ryan. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Sean Treadway's Go RabbitMQ client package is under a BSD 2-clause license. Cenk Alti's Go exponential backoff package is under an MIT license. Once again, all rights reserved.

Index

• type Client
  • func (c *Client) Connect(ctx context.Context) error
  • func (c *Client) Consume() (*Event, bool)
• type Event

Types

type Client

type Client struct {
	Subtopics       *[]string // Subtopics are array of subscribed subtopics (see documentation for formatting)
	DisableEventLog bool      // DisableEventLog disables the event log (default: false)
	FetchContent    bool      // FetchContent enables HTTP content fetching (default: false)
	// contains filtered or unexported fields
}

Client contains the ecpush consumer client.

func (*Client) Connect

func (c *Client) Connect(ctx context.Context) error

Connect establishes the AMQP channel for receiving products located on provided subtopics. It blocks until the initial connection is established with the remote server. Context is passed for closing the client. Connect returns an error if no subtopics are provided.

func (*Client) Consume

func (c *Client) Consume() (*Event, bool)

Consume returns the next event and an indicator if the client is not actively consuming from the remote server.

type Event

type Event struct {
	URL            string // URL of the product (located on Datamart)
	MD5            string // MD5 of product
	SHA512         string // SHA512 of product
	Route          string // Route is AMQP routing key of event
	Content        string // Content is event contents (if FetchContent is true)
	ContentFailure bool   // ContentFailure indicates if event fetching failed
}

Event is a received payload from Environment Canada's datamart.