archive_upload_mass_upload

README

Cloud storage mass uploader

Compare an FTP archive to a cloud storage bucket, copy files which are either not included in the cloud storage bucket OR where the MD5 mismatches from the ftp archive to the cloud storage bucket.

Operations

Compile Code

$ go build -o mass_uploader mass_uploader.go

Create Service Account

Create a service account to be used in uploading and querying the cloud storage resources. Go to the [IAM -> ServiceAccounts] (https://console.cloud.google.com/iam-admin/serviceaccounts) page, create a new service account.

Create a new key for the service-account, download the JSON format key material. Store this key material securely on the machine which will be performing the synchronization.

Grant the service account the following roles/permissions:

• Cloud Run Invoker - configured in the CloudRun instance permissions.
• Storage Object Creator - configured in IAM permissions.
• Storage Object Viewer - configured in IAM permissions.
• Storage Object Creator - configured for the storage bucket
• Storage Object Viewer - configured for the storage bucket

Run the Upload Process

Run the program, provide the bucket and ftp archive as flags, provide the key file location through the environment variable: GOOGLE_APPLICATION_CREDENTIALS.

$ GOOGLE_APPLICATION_CREDENTIALS=<filesystem_path_to_key> mass_upload -bucket routeviews-archives -archive ftp://archive.routeviews.org/bgpdata

Review Logs

Review logged data for errors, address as required.

Documentation

Overview

Package mass_upload should upload content from a given ftp site to cloud storage. Verification of ftp site content against the cloud storage location before upload must be performed.

Basic flow is:

1. start at the top of an FTP site.
2. download each file in turn, walking the remote directory tree.
3. calculate the md5() checksum for each file downloaded.
4. validate that the checksum matches the cloud-storage object's MD5 value.
5. if there is a mis-match, upload the ftp content to cloud-storage.