converter

README

RouteViews Archive Converter

Deploy to App Engine (Recommended)

App Engine has a much larger maximum timeout (24 hours) and can be integrated with Cloud Tasks.

To deploy, get into this directory and run gcloud app deploy.

Deploy to Cloud Run

1. Build the image from the root directory. (TODO: use docker-decompose.yaml)
   • In root directory,

   $   docker build -f cmd/converter/Dockerfile . -t us-docker.pkg.dev/public-routing-data-backup/cloudrun/rv-converter:latest
   

   • Don't forget to upload it to the registries:

   docker push
       us-docker.pkg.dev/public-routing-data-backup/cloudrun/rv-converter:latest`
   

2. Deploy the image by setting the output bucket BIGQUERY_BUCKET for converted updates.
   • Example:

   $   gcloud run deploy rv-converter \
       --image us-docker.pkg.dev/public-routing-data-backup/cloudrun/rv-converter:latest \
       --cpu 2 --memory 4Gi \
       --concurrency 2 \
       --update-env-vars BIGQUERY_BUCKET=routeviews-bigquery
   

3. [Only need once] Hook up a PubSub channel with the Cloud Run service through PubSub (see instructions).
   • Acknowledgement deadline is set to 300s to prevent too many retry messages.
4. [Only need once] Hook up a PubSub channel with the archive source bucket (see instructions).
5. [Only need once] Set up recurrent data transfer in BigQuery (see instructions)
6. [Only need once] Set up log-based alerts (TBD).