snap

README

snap ^{_{A powerful telemetry framework}}

---

1. Overview
2. Getting Started

• In Active Development
• System Requirements
• Installation
• Running Snap
• Load Plugins
• Running Tasks
• Building Tasks
• Plugin Catalog

1. Documentation

• Examples
• Roadmap

1. Community Support
2. Contributing

• Author a Plugin

1. License
2. Contributors

• Initial Authors
• Maintainers

1. Thank You

Overview

Snap is an open telemetry framework designed to simplify the collection, processing and publishing of system data through a single API. The goals of this project are to:

• Empower systems to expose a consistent set of telemetry data
• Simplify telemetry ingestion across ubiquitous storage systems
• Improve the deployment model, packaging and flexibility for collecting telemetry
• Allow flexible processing of telemetry data on agent (e.g. filtering and decoration)
• Provide powerful clustered control of telemetry workflows across small or large clusters

The key features of snap are:

• Plugin Architecture: snap has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The open plugin model allows for loading built-in, community, or proprietary plugins into snap.

  • Collectors - Collectors consume telemetry data. Collectors are built-in plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see Plugin Authoring below). Telemetry data is organized into a dynamically generated catalog of available data points.
  • Processors - Extensible workflow injection. Convert telemetry into another data model for consumption by existing consumption systems (like OpenStack Ceilometer). Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
  • Publishers - Store telemetry into a wide array of systems. snap decouples the collection of telemetry from the implementation of where to send it. snap comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows snap to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures.

• Dynamic Updates: snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (e.g. get /intel/psutil/load/load1/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.

• Snap tribe: snap is designed for ease of administration. With snap tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node communication process. Administrators can control all snap nodes in a tribe agreement by messaging just one of them. There is auto-discovery of new nodes and import of tasks and plugins from nodes within a given tribe. It is cluster configuration management made simple.

Some additionally important notes about how snap works:

• Multiple management modules including: CLI (snapctl) and REST API (each of which can be turned on or off)
• Secure validation occurs via plugin signing, SSL encryption for APIs and payload encryption for communication between components
• CLI control from Linux or OS X

Snap is not intended to:

• Operate as an analytics platform: the intention is to allow plugins for feeding those platforms
• Compete with existing metric/monitoring/telemetry agents: Snap is simply a new option to use or reference

Getting Started

In Active Development

The master branch is used for new feature development. Our goal is to keep it in a deployable state. If you're looking for the most recent binary that is versioned, please see the latest release.

System Requirements

Snap deploys as a binary, which makes requirements quite simple. We've tested on a subset of Linux and OS X versions.

Installation

You can get the pre-built binaries for your OS and architecture at Snap's GitHub Releases page. This isn't the comprehensive list of plugins. Right now, snap only supports Linux and OS X (Darwin).

Running Snap

We're going to assume you downloaded the latest packaged release of Snap and its plugins from here on. If you prefer to build from source, follow the steps in BUILD_AND_TEST.md.

Untar the version of Snap you downloaded and move its binaries, snapd and snapctl, to a place in your path. Here is an example of doing so (on Linux):

$ tar -xvf snap-v0.13.0-beta-linux-amd64.tar
$ mv snap-v0.13.0-beta/bin/* /usr/local/bin/
$ rm -rf snap-v0.13.0-beta

Start a standalone Snap agent (snapd):

$ snapd --plugin-trust 0 --log-level 1

This will bring up a Snap agent without requiring plugin signing (trust-level 0) and set the logging level to debug (log level 1). Snap's REST API will be listening on port 8181. To learn more about the snap agent and how to use it look at SNAPD.md and/or run snapd -h.

Snap can also be run in a clustered mode called tribe. Checkout the tribe documentation for more info.

Load Plugins

Snap gets its power from the use of plugins. The Plugin Catalog is a collection of all known plugins for Snap with links to the binaries. You can download individual plugins or pull down a package of starter plugins for your operating system under GitHub Releases. This isn't the comprehensive list of plugins, but they will help you get started.

Open a separate window from the one running snapd, then unpack the downloaded plugins (example is on Linux):

$ tar -xvf snap-plugins-v0.13.0-beta-linux-amd64.tar.gz
$ mkdir -p ~/snap/plugins/
$ mv snap-v0.13.0-beta/plugin/* ~/snap/plugins/
$ rm -rf snap-v0.13.0-beta

Next, load the plugins. This can be achieved through the REST API directly or by using the helper command snapctl.

Using the API directly with cURL:

$ cd ~/snap/plugins/
$ curl -X POST -F plugin=@snap-collector-mock1 http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@snap-processor-passthru http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@snap-publisher-file http://localhost:8181/v1/plugins

Every interaction with snapd can be done through the REST API. To see what else you can do with the API, view our API Documentation. We will continue on using snapctl:

$ cd ~/snap/plugins/
$ snapctl plugin load snap-collector-mock1
$ snapctl plugin load snap-processor-passthru
$ snapctl plugin load snap-publisher-file

Let's look at what plugins you have loaded now:

$ snapctl plugin list
NAME             VERSION         TYPE            SIGNED          STATUS          LOADED TIME
mock             1               collector       false           loaded          Tue, 17 Nov 2015 14:08:17 PST
passthru         1               processor       false           loaded          Tue, 17 Nov 2015 14:16:12 PST
file             3               publisher       false           loaded          Tue, 17 Nov 2015 14:16:19 PST

You now have one of each plugin type loaded into the framework. To begin collecting data, you need to create a task.

Running Tasks

Tasks are most often shared as a Task Manifest and is written in JSON or YAML format. Make a copy of this example task from the examples/tasks/ directory on your local system and then start the task:

$ cd ~/snap
$ curl https://raw.githubusercontent.com/intelsdi-x/snap/master/examples/tasks/mock-file.yaml > mock-file.yaml
$ snapctl task create -t mock-file.yaml
Using task manifest to create task
Task created
ID: 8b9babad-b3bc-4a16-9e06-1f35664a7679
Name: Task-8b9babad-b3bc-4a16-9e06-1f35664a7679
State: Running

This task generates mock data, "processes" it through a passthrough mechanism and then publishes it to a file. Now with a running task, you should be able to do two things:

See the data that is being published to the file:

$ tail -f /tmp/snap_published_mock_file.log
^C

Or tap into the data that Snap is collecting using the Task ID to watch the task (note: your Task ID will be different):

$ snapctl task watch 8b9babad-b3bc-4a16-9e06-1f35664a7679

If the Task ID already scrolled by, you can list it with:

$ snapctl task list

Nice work - you're all done with this example. You ran snapd manually, so stopping the daemon process will stop any running tasks and unload any plugins we loaded.

$ pkill snapd

Or you can continue to run more tasks using the loaded plugins (why not create a new Task Manifest that publishes mock data to another file?). Alternatively, you can stop any running tasks and unload any plugins you no longer wish to use manually:

$ snapctl task stop 8b9babad-b3bc-4a16-9e06-1f35664a7679
$ snapctl plugin unload processor:passthru:1
Plugin unloaded
Name: passthru
Version: 1
Type: processor
$ snapctl plugin unload publisher:file:3
Plugin unloaded
Name: file
Version: 3
Type: publisher
$ snapctl plugin unload collector:mock:1
Plugin unloaded
Name: mock
Version: 1
Type: collector

When you're ready to move on, walk through other uses of Snap available in the Examples folder.

Building Tasks

Documentation for building a task can be found here.

Plugin Catalog

All known plugins are tracked in the plugin catalog and are tagged as collectors, processors and publishers.

If you would like to write your own, read through Author a Plugin to get started. Let us know if you begin to write one by chatting with us on Gitter. When you finish, please open a Pull Request to add yours to the catalog!

Documentation

Documentation for Snap will be kept in this repository for now with an emphasis of filling out the docs/ directory. We would also like to link to external how-to blog posts as people write them. Read about contributing to the project for more details.

• snapd (snap agent)
• configuring snapd
• snapctl (Snap CLI)
• build and test
• REST API
• tasks
• plugin signing
• tribe

Here are blog posts related to Snap by the team:

• What if collecting data center telemetry was a snap? by @lynxbat
• My How-to for the Snap Telemetry Framework by @mjbrender
• Snap's first GUI is Grafana! by @mjbrender
• Adding a global configuration to Snap by @tjmcs1

Examples

More complex examples of using Snap Framework configuration, Task Manifest files and use cases are available under the Examples folder. There are also interesting examples of using Snap in every plugin repository. For the full list of plugins, review the Plugin Catalog.

Roadmap

We have a few known features we want to take on from here while we remain open for feedback after public release. They are:

• Authentication, authorization, and auditing (see issue #286)
• Workflow Routing (see issue #539)
• Windows support (see #671)
• Distributed Workflows (see #539 and #640)

If you would like to propose a feature, please open an Issue) that includes RFC in it (for request for comments).

Community Support

This repository is one of many projects in the Snap framework. Discuss your questions about snap by reaching out to us:

• Through GitHub Issues. Issues is our home for all needs: Q&A on everything - installation, request for events, integrations, bug issues, futures. Open up an Issue and know there's no wrong question for us.
• We also have a Gitter channel opened up on this repository that threads directly into our engineering team Slack (thanks to Sameroom).
• Submit a blog post on your use of Snap to our Medium.com publication

The full project lives here, at http://github.com/intelsdi-x/snap.

Contributing

We encourage contributions from the community. No improvement is too small. Snap needs:

• Feedback: try it and tell us about it through issues, blog posts or Twitter
• Contributors: We can always use more eyes on the core framework and its testing
• Integrations: Snap can collect from and publish to almost anything by authoring a plugin

To contribute to the Snap framework, see our CONTRIBUTING file. To give back to a specific plugin, open an issue on its repository.

Author a Plugin

The power of Snap comes from its open architecture. Add to the ecosystem by building your own plugins to collect, process or publish telemetry.

• The definitive how-to is in PLUGIN_AUTHORING.md
• Recommendations to make effective, well-designed plugins are in PLUGIN_BEST_PRACTICES.md

License

Snap is Open Source software released under the Apache 2.0 License.

Contributors

Initial Authors

All contributors are equally important to us, but we would like to thank the initial authors for helping make open sourcing snap possible.

Maintainers

Amongst the many awesome contributors, there are the maintainers. These maintainers may change over time, but they are all members of the @intelsdi-x/snap-maintainers team. This group will help you by:

• Committing to reviewing pull requests, issues, and addressing comments/questions as quickly as possible
• Acting as a point of contact for questions

Just tag @intelsdi-x/snap-maintainers if you need to get some attention on an issue. If at any time, you don't get a quick enough response, reach out to any of the following team members directly:

@andrzej-k	@candysmurf	@ConnorDoyle	@danielscottt	@geauxvirtual	@jcooklin
@lynxbat	@marcin-krolik	@mjbrender	@nqn	@tiffanyfj	@IzabellaRaulin

We're also looking to have maintainers from the community. Please let us know if you would like to become one by opening an Issue titled "interested in becoming a maintainer." We are currently working on a more official process.

Thank You

And thank you! Your contribution, through code and participation, is incredibly important to us.