archive_upload_server

README

RouteViews Archive Server

Collects files to archive from remote service owners.

Deployment

Deployed to GCP CloudRun, from the project's Docker image store.

1. Build a current image from repository root:

$ docker build -f cmd/archive_upload_server/Dockerfile . \
               --tag us-docker.pkg.dev/public-routing-data-backup/cloudrun/rv-server:latest

1. Push the docker image to the registry:

$ docker push us-docker.pkg.dev/public-routing-data-backup/cloudrun/rv-server:latest

1. Have cloud run, run the job:

$ gcloud run deploy  rv-server \
          --image us-docker.pkg.dev/public-routing-data-backup/cloudrun/rv-server:latest \
          --use-http2 --no-allow-unauthenticated \
          --memory 1G

1. Reserve an IPv4/6 address for the load balancer (DO THIS ONCE) Names: cloud-backup-vip-v4 && cloud-backup-vip-v6

2. Setup certificates for the VIP/name mapping (DO THIS ONCE) Name: storage-archive.rarc.net

3. Setup loadbalancer config (DO THIS ONCE)

Documentation

Overview

Package main is the grpc server which accepts file storage requests,

Files are stored in cloud-storage, they may be converted and added to BigQuery tables as well, depending upon the request.

NOTE: Currently (12/2021) all files are stored to GCS, the conversion process

subscribes to a pubsub feed of buckets which are to be converted for BigQuery.