dispatch-buffered

Documentation

Overview

Crawl an S3 directory of files containing WOF repos to index. These are assumed to have been written using the go-webhookd(-gocloud) package. Specifically the repo name will have been extracted using the https://github.com/whosonfirst/go-webhookd-github#github receiver and the https://github.com/whosonfirst/go-webhookd-github#githubrepo transformation and finally written to S3 using the https://github.com/whosonfirst/go-webhookd-gocloud dispatcher. This tool will invoke a Lambda function, which is not necessarily but otherwise assumed to be the one defined in cmd/launch-ecs-task which, in turn, will invoke an ECS task. The point being it's just a giant bucket brigade to pass around a repo name in order to prevent invoking a thundering herd of ECS tasks when a repo has a lot of small atomic commits (triggering GitHub receiver above).

For example:

$> ./bin/dispatch-buffered -dryrun \
	-bucket-uri 's3blob://sfomuseum-pending/?region=us-west-2&prefix=indexing/&credentials=session' \
	-lambda-uri 'lambda://WebhookdIndexingTask?dsn=region=us-west-2 credentials=session'