README ¶
#+TITLE: Lambdabeat * Overview Lambdabeat allows you to ship AWS Lambda metrics (by way of the Cloudwatch API) to Elasticsearch. Here is an example of a document produced by Lambdabeat: #+BEGIN_SRC es POST http://localhost:9200/lambdabeat-2016.06.14/_search?pretty { "size": 1, "query": { "match_all": {} } } #+END_SRC #+begin_example { "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 79, "max_score" : 1.0, "hits" : [ { "_index" : "lambdabeat-2016.06.14", "_type" : "metric", "_id" : "AVVQbTk8fZl27BsUqkOv", "_score" : 1.0, "_source" : { "@timestamp" : "2016-06-14T01:03:00.000Z", "beat" : { "hostname" : "baamonde-ThinkPad-T460s", "name" : "baamonde-ThinkPad-T460s" }, "code-size" : 3564954, "description" : "my lambda function", "duration-average" : 2718.6, "duration-maximum" : 2718.6, "duration-minimum" : 2718.6, "duration-sample-count" : 1, "duration-sum" : 2718.6, "duration-unit" : "Milliseconds", "errors-average" : 0, "errors-maximum" : 0, "errors-minimum" : 0, "errors-sample-count" : 1, "errors-sum" : 0, "errors-unit" : "Count", "function" : "my-lambda-function", "handler" : "my-lambda-function.handler", "invocations-average" : 1, "invocations-maximum" : 1, "invocations-minimum" : 1, "invocations-sample-count" : 1, "invocations-sum" : 1, "invocations-unit" : "Count", "last-modified" : "2016-04-04T02:46:48.557+0000", "memory-size" : 128, "runtime" : "python2.7", "throttles-average" : 0, "throttles-maximum" : 0, "throttles-minimum" : 0, "throttles-sample-count" : 1, "throttles-sum" : 0, "throttles-unit" : "Count", "timeout" : 60, "type" : "metric", "version" : "$LATEST" } } ] } } #+end_example * Installation Binaries are available [[https://github.com/michaelbaamonde/lambdabeat/releases][here]]. Released packages contain: 1. The =lambdabeat= binary. 2. An example =lambdabeat.yml" configuration file. 3. The =lambdabeat.template.json= mapping template. * Configuration and Usage ** AWS: Credentials and Region Lambdabeat fetches data from the Cloudwatch API. As such, it does not need to be deployed anywhere in particular, but it does require valid AWS credentials to be specified either: 1. In an AWS configuration file located at =~/.aws/credentials.= 2. As environment variables (=AWS_ACCESS_KEY_ID= and =AWS_SECRET_ACCESS_KEY=.) See [[https://github.com/aws/aws-sdk-go#configuring-credentials][here]] for further details regarding credentials. Additionally, Lambdabeat requires that a default AWS region be set in the =lambdabeat.yml= configuration file. ** Functions Simply specify an array of Lambda functions for which you would like to retrieve metrics in =lambdabeat.yml= like so: #+BEGIN_EXAMPLE lambdabeat: functions: ["foo", "bar", "baz", "qux"] #+END_EXAMPLE ** Metrics Cloudwatch maintains data for these metrics with respect to Lambda: - =Invocations= - =Errors= - =Duration= - =Throttles= These are largely self-explanatory, but see [[http://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html][here]] for details and subtleties. Lambdabeat fetches and indexes all of these. ** Period and Interval Unlike many (really, most) other Beats, Lambdabeat's notion of a =period= does not necessarily determine the timestamps of the events being indexed. Rather, the Cloudwatch API provides data between two points in time at a given, user-specified interval. This is configured via the =interval= key in =lambdabeat.yml=. Separately, the =period= key specifies how often Lambdabeat itself should run. As such, the value of the =@timestamp= field for each event is *not* the time at which Lambdabeat fetched the data; it's the time for which Cloudwatch has returned data. At query time, this is an unimportant implementation detail. What matters is that the events indexed into Elasticsearch are timestamped in a meaningful matter, which they are. It is important, however, to understand the distinction at /configuration time/. Note that the value of =interval= *must* be a multiple of 60. Note also that it only really makes sense for =period= to be greater than or equal to =interval=. ** Backfilling data If you would like to index data that predates the first invocation of Lambdabeat, you may do so by passing a the option =-backfill-date= when invoking lambdabeat. This will fetch and index all data between the given =backfill-date= and the time at which Lambdabeat begins running. For example: #+BEGIN_SRC sh lambdabeat -e -v -backfill-date="2016-06-08T00:00:00.000Z" #+END_SRC ** Outputs Lambdabeat currently supports Elasticsearch as an output. Simply provide an array of hosts in =lambdabeat.yml= like so: #+BEGIN_EXAMPLE output: elasticsearch: hosts: ["http://localhost:9200"] #+END_EXAMPLE Lambdabeat introduces no novel infrastructure with respect to outputs. * Development *** Requirements - [[https://golang.org/dl/][Golang]] 1.6 - [[https://github.com/Masterminds/glide][Glide]] >= 0.10.0 *** Build To build the binary for Lambdabeat run the command below. This will generate a binary in the same directory with the name lambdabeat. #+BEGIN_EXAMPLE make #+END_EXAMPLE *** Run To run Lambdabeat with debugging output enabled, run: #+BEGIN_EXAMPLE ./lambdabeat -c lambdabeat.yml -e -d "*" #+END_EXAMPLE *** Package To produce a versioned tarball containing the lambdabeat binary, lambdabeat.yml, and lambdabeat index template, run: #+BEGIN_EXAMPLE make release #+END_EXAMPLE *** Cleanup To clean up the build directory and generated artifacts, run: #+BEGIN_EXAMPLE make clean #+END_EXAMPLE *** Clone To clone Lambdabeat from the git repository, run the following commands: #+BEGIN_EXAMPLE mkdir -p ${GOPATH}/github.com/michaelbaamonde cd ${GOPATH}/github.com/michaelbaamonde git clone https://github.com/michaelbaamonde/lambdabeat #+END_EXAMPLE For further development, check out the [[https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html][beat developer guide]].
Documentation ¶
There is no documentation for this package.
Click to show internal directories.
Click to hide internal directories.