ObjInSync
Daemon to continuously synchronize a directory from remote object store to a local directory.
NOTE: it automatically ignores __pycache__
directory.
Usage
objinsync pull s3://bucket/keyprefix ./localdir
When running in daemon mode (without --once
flag), a healthcheck endpoint is
served at :8087/health
and a prometheus metrics endponit is served at
:8087/metrics
. You can use --status-addr
to override the binding address.
Objinsync also comes with builtin Sentry integraiton. To enable it, set the
SENTRY_DSN
environment variable.
Installation
Simply download the prebuilt binary from release page or use go get
command:
go get github.com/scribd/objinsync
Development
Run tests
make test
Run from source
AWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags