gcsds

README

GCS Datastore Implementation for IPFS

An implementation of the go-datastore interface backed by Google Cloud Storage, GCS. The implementation is based on go-ds-s3

Building binary

Build this plugin as an included plugin with kubo.

Clone kubo.

git clone https://github.com/ipfs/kubo

Get the GCS datastore plugin and add it to the preload list.

(cd kubo; go get github.com/ipfs-shipyard/go-ds-gcs/plugin)
echo "gcsds github.com/ipfs-shipyard/go-ds-gcs/plugin 0" >> kubo/plugin/loader/preload_list

Build kubo with the plugin

(cd kubo; make build)

(Optionally) install kubo onto the local system.

(cd kubo; make install)

Building Docker container

Follow the "Building binary" section first. Clone the GCS datastore plugin and build the Docker entrypoint.

git clone https://github.com/ipfs-shipyard/go-ds-gcs
(cd go-ds-gcs; go build docker/entrypoint.go)

Build docker and container.

docker build -f go-ds-gcs/docker/Dockerfile -t ipfs .

(Optional) tag and push image to repository. Replace repository with your GCR repository.

docker tag ipfs gcr.io/repository/ipfs
docker push gcr.io/repository/ipfs

Deployment: Docker

ipfs is the image name from the previous section, "Building docker container"

docker run -d -p 4001:4001 -p 5001:5001 -p 8080:8080 --name=ipfs ipfs

Or you can use the tagged image from the previous section.

docker run -d -p 4001:4001 -p 5001:5001 -p 8080:8080 --name=ipfs gcr.io/repository/ipfs

Deployment: Kubernetes (GKE)

1. Create a GKE cluster. The node pool must have write permission to the GCS bucket.
2. Replace repository with your repository name in kubernetes/ipfs.yml
3. Replace mynamespace with your namespace name in kubernetes/ipfs.yml
4. Create namespace in GKE.

kubectl create namespace mynamespace

1. Deploy to GKE.

kubectl apply -f kubernetes/ipfs.yml

1. IPFS can now be accessed at ipfs.mynamespace from other pods in the GKE cluster:

curl http://ipfs.mynamespace:8080/...

Deployment: Local

For a brand new ipfs instance (no data stored yet):

1. Install kubo/cmd/ipfs/ipfs
2. Initialize IPFS. Replace mybucket with your bucket name. Note that this depends on this Pull Request.

KUBO_GCS_BUCKET=mybucket ipfs init --profile gcsds

1. To run as a server, also run

ipfs config profile apply server
ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/8080
ipfs daemon

Configuration

The config file should include the following. Replace mybucket with your bucket name.

{
	"Datastore": {
		"Spec": {
			"mounts": [{
				"child": {
					"bucket": "mybucket",
					"cachesize": 40000,
					"prefix": "ipfs",
					"type": "gcsds",
					"workers": 100
				},
				"mountpoint": "/blocks",
				"prefix": "flatfs.datastore",
				"type": "measure"
			}]
		}
	}
}

Google Cloud credentials

Google Cloud credentials should automatically be provided when running in Google Compute Engine (GCE) or Google Kubernetes Engine (GKE). Note that for both GCE and GKE, the (node) VM needs to have write permission (scope) to GCS. For GKE, this is achieved by creating the node pool with the "Storage read/write" scope, which is "devstorage.read_write".

In other environments, you may have to provide credentials. One way is to use the GOOGLE_APPLICATION_CREDENTIALS environment variable. See this document for more details.

Contribute

Feel free to join in. All welcome. Open an issue!

This repository falls under the IPFS Code of Conduct and the Google Code of Conduct

License

Apache 2.0

Documentation

Index

• type Config
• type GCSDatastore
  • func NewGCSDatastore(cfg Config) (*GCSDatastore, error)
  • func (gd *GCSDatastore) Batch(_ context.Context) (ds.Batch, error)
  • func (gd *GCSDatastore) CheckBucket() error
  • func (gd *GCSDatastore) Close() error
  • func (gd *GCSDatastore) Delete(ctx context.Context, k ds.Key) error
  • func (gd *GCSDatastore) GCSPath(key string) string
  • func (gd *GCSDatastore) Get(ctx context.Context, k ds.Key) ([]byte, error)
  • func (gd *GCSDatastore) GetSize(ctx context.Context, k ds.Key) (size int, err error)
  • func (gd *GCSDatastore) Has(ctx context.Context, k ds.Key) (exists bool, err error)
  • func (gd *GCSDatastore) LoadMetadata() error
  • func (gd *GCSDatastore) Put(ctx context.Context, k ds.Key, value []byte) error
  • func (gd *GCSDatastore) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
  • func (gd *GCSDatastore) Sync(ctx context.Context, prefix ds.Key) error
• type Metadata
• type MetadataCache
  • func NewMetadataCache() *MetadataCache
  • func (md *MetadataCache) Delete(key string)
  • func (md *MetadataCache) Get(key string) (*Metadata, error)
  • func (md *MetadataCache) Has(key string) bool
  • func (md *MetadataCache) Iterator(prefix string, limit int) func() *Metadata
  • func (md *MetadataCache) Put(key string, size int64)
  • func (md *MetadataCache) Size() int

Types

type Config

type Config struct {
	Bucket         string
	Prefix         string
	Workers        int
	DataCacheItems int
}

type GCSDatastore

type GCSDatastore struct {
	Config
	// contains filtered or unexported fields
}

func NewGCSDatastore

func NewGCSDatastore(cfg Config) (*GCSDatastore, error)

func (*GCSDatastore) Batch

func (gd *GCSDatastore) Batch(_ context.Context) (ds.Batch, error)

func (*GCSDatastore) CheckBucket

func (gd *GCSDatastore) CheckBucket() error

CheckBucket checks that the GCS bucket exists and is accessible.

func (*GCSDatastore) Close

func (gd *GCSDatastore) Close() error

func (*GCSDatastore) Delete

func (gd *GCSDatastore) Delete(ctx context.Context, k ds.Key) error

func (*GCSDatastore) GCSPath

func (gd *GCSDatastore) GCSPath(key string) string

func (*GCSDatastore) Get

func (gd *GCSDatastore) Get(ctx context.Context, k ds.Key) ([]byte, error)

func (*GCSDatastore) GetSize

func (gd *GCSDatastore) GetSize(ctx context.Context, k ds.Key) (size int, err error)

func (*GCSDatastore) Has

func (gd *GCSDatastore) Has(ctx context.Context, k ds.Key) (exists bool, err error)

func (*GCSDatastore) LoadMetadata

func (gd *GCSDatastore) LoadMetadata() error

LoadMetadata pre-loads metadata for all objects in the ipfs prefix.

func (*GCSDatastore) Put

func (gd *GCSDatastore) Put(ctx context.Context, k ds.Key, value []byte) error

func (*GCSDatastore) Query

func (gd *GCSDatastore) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)

func (*GCSDatastore) Sync

func (gd *GCSDatastore) Sync(ctx context.Context, prefix ds.Key) error

type Metadata

type Metadata struct {
	Key string
	// Store object size as int64.
	// In practice, all IPFS objects are max 256kB.
	Size int64
}

type MetadataCache

type MetadataCache struct {
	// contains filtered or unexported fields
}

func NewMetadataCache

func NewMetadataCache() *MetadataCache

func (*MetadataCache) Delete

func (md *MetadataCache) Delete(key string)

func (*MetadataCache) Get

func (md *MetadataCache) Get(key string) (*Metadata, error)

func (*MetadataCache) Has

func (md *MetadataCache) Has(key string) bool

func (*MetadataCache) Iterator

func (md *MetadataCache) Iterator(prefix string, limit int) func() *Metadata

Offset not supported, for now.

func (*MetadataCache) Put

func (md *MetadataCache) Put(key string, size int64)

func (*MetadataCache) Size

func (md *MetadataCache) Size() int