handler

package module
v0.0.0-...-e8a84d0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 8, 2023 License: MIT, Apache-2.0 Imports: 8 Imported by: 0

README

basin-storage

Test License standard-readme compliant

Filecoin storage and indexing for Tableland Basin

Table of Contents

Background

Tableland Basin is a secure and verifiable open data platform. The Basin Storage replicates data to Filecoin. It works in conjunction with basin-cli and basin-provider.

🚧 Basin is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧

Development

Basin Storage leverages GCP Cloud Functions and Clould Storage to create a data archiving pipeline to Filecoin network. The File uploader function is triggered by an event from Cloud Storage.

This repository includes a development server that acts as an event trigger during development.

Running

Start the development server for testing Clould Functions locally. The required environment variables can be provided in uploader.env.yml and checker.env.yml.

make uploader-local
make checker-local

After the server is running, a mock cloud event can be triggered. It will execute the handler locally. In the following payload, we must have a real file path and a bucket for event simulation.

curl -X POST http://localhost:8080 \
-H "Content-Type: application/json" \
-H "ce-id: 1234567890" \
-H "ce-specversion: 1.0" \
-H "ce-type: google.cloud.storage.object.v1.finalized" \
-H "ce-time: 2020-08-08T00:11:44.895529672Z" \
-H "ce-source: //storage.googleapis.com/projects/_/buckets/tableland-entrypoint" \
-d '{
  "name": "feeds/2023-08-29/202308291525552525242120000000000-3ab461ed932d5f1c-1-2-00000000-employees-2.parquet",
  "bucket": "tableland-entrypoint",
  "contentType": "application/json",
  "metageneration": "1",
  "timeCreated": "2020-04-23T07:38:57.230Z",
  "updated": "2020-04-23T07:38:57.230Z"
}'

The checker function can be triggered by simply sending a POST request for example curl -XPOST localhost:8080.

Deploying Function

Deploy Uploader function
make uploader-deploy
Deploy Status Checker function
make checker-deploy

Run tests

make test

Contributing

PRs accepted.

Small note: If editing the README, please conform to the standard-readme specification.

License

MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func StatusChecker

func StatusChecker(w http.ResponseWriter, r *http.Request)

StatusChecker is the HTTP function that is called by the Functions Framework.

func Uploader

func Uploader(ctx context.Context, e event.Event) error

Uploader is the CloudEvent function that is called by the Functions Framework. It is triggered by a CloudEvent that is published by the GCS bucket. The CloudEvent contains the name of the bucket and the name of the file. The file is downloaded from GCS and uploaded to web3.storage.

Types

This section is empty.

Directories

Path Synopsis
pkg

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL