filestore

README

Filestore - A microservice for storing files

This service acts as a file storage & retrieval system, intended to ease file management within a microservice environment. GRPC is used as the communication protocol between client and server, and the fileserver executable serves as both the client and the server.

It has a modular design regarding the "back end", where the file metadata is recorded. Currently two back ends are supported - Redis & Postgres.

You might find this useful if you want to be able to add and remove files with one service and use those files with another service - in my case (the reason I started this in the first place) there is an update service and a user-facing service.

There are two elements within the system: Volumes and Files. Files are added to volumes, and volumes have all the control over where the files get placed.

Volumes are also password protected. This system isn't intended to be internet facing, and only has a thin layer of security primarily intended to prevent accidental damage due to programmer error.

Use

First, make sure the config file correctly indicates where your default volume location should be, as in:

[vmgr]
dbengine = "redis" # or "postgres"

[vmgr.redis]
    addr = "localhost:6379"

[vmgr.paths]
    default = "/mnt/nasdrive/filestore"

From the commandline, the first thing you'd need to do is create a volume:

$ ./filestore nv art_assets -p SomekindaPassword
res:  {
    "name": "art_assets",
    "tags": [
        ""
    ],
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAzOTA0OTAsImp0aSI6IjE4Nzk5YzEzLTIzMDYtNGFhZC04MDM1LTU1MTlkMzMyNmI1ZCIsImlzcyI6ImZpbGVzdG9yZSIsInN1YiI6InRlc3R2b2wifQ.S9CZVS3XiQYiWjnO77TOIQk9FMyLQkqYFTnxqXWOpBs"
}

That token is important. It's a JWT that contains volume information, and is required for the put operation. You get an equivalent token by opening the volume, as in:

$ ./filestore open art_assets -p SomekindaPassword

resvd: {
    "name": "art_assets",
    "tags": [
        ""
    ],
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAzOTM0MjgsImp0aSI6Ijc4NzI2M2UzLTJjMDMtNDM1NS05YjVlLTkxYWM0ZGJjNzBlNCIsImlzcyI6ImZpbGVzdG9yZSIsInN1YiI6InRlc3R2b2wyIn0.VGy65nbMht1oNUM9Aheq8RSszz_Oj6cdLuAOSBW5_mU"
}

With the token, you can now upload a file via the put command:

./filestore put LICENSE -T eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAzOTM0MjgsImp0aSI6Ijc4NzI2M2UzLTJjMDMtNDM1NS05YjVlLTkxYWM0ZGJjNzBlNCIsImlzcyI6ImZpbGVzdG9yZSIsInN1YiI6InRlc3R2b2wyIn0.VGy65nbMht1oNUM9Aheq8RSszz_Oj6cdLuAOSBW5_mU"

The result is:

2020-05-24T00:48:24.640-07:00 [DEBUG] put.go:142: file state after put: {
    "ID": "ce8176d8-2407-44ed-a6f8-5b4f34c48745"
} took 158ms

That ID is now how you access the file:

./filestore get ce8176d8-2407-44ed-a6f8-5b4f34c48745 -T eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAzOTM0MjgsImp0aSI6Ijc4NzI2M2UzLTJjMDMtNDM1NS05YjVlLTkxYWM0ZGJjNzBlNCIsImlzcyI6ImZpbGVzdG9yZSIsInN1YiI6InRlc3R2b2wyIn0.VGy65nbMht1oNUM9Aheq8RSszz_Oj6cdLuAOSBW5_mU"

The result will be printed to stdout.

When creating a volume, you can specify the type of volume you want with a "type" attribute, as in:

./filestore nv specialstuff -a type=fast

The type parameter is how you choose amongst the paths specified in the config file. The demo config file has this:

[vmgr.paths]
    # The system will automatically add the name of the volume
    # to the end of the path, so it's ok to have the same path
    # listed multiple times.
    default = "/mnt/nasferatu/Dev/filestore"
    fast = "/mnt/nasferatu/Dev/filestore"

If you do not specify a type, the system will automatically choose the default entry. Because of this, the default entry is manditory. You may enter as many other types as you like, and use the -a type=<name> attribute to specify them when you are creating a volume.

Caveats

I originally designed the system to be able to accept tags (key:value) and attributes (list of strings) for a given file, in the spirit of Kubernetes. However, I've never actually used either, and will probably trim them from the design of the system.

GRPC is the underlying client/server communication protocol, and the fs.proto file may be found in cmd/service/filestore/fs.proto. There I have commented out some considered additions, but these will probably be removed unless a good reason to keep them is found.

The Postgres backend supports the tags and attributes, and the redis backend does not. If anyone is interested or can give me a good reason to keep them, I will add them to the redis backend. Otherwise they will probably be removed from the postgres backend in the future in the interest of KISS.

Origins

I wrote this service because I needed to be able to add the ability for one microservice to retrieve files and record them and allow another service to use them.

Kubernetes allows you to mount drives via PV & PVC, but only one service can use the volume at a time. I needed one service to be able to add files, and another service to be able to use those files.

My initial solution was to run a program within the container of the primary service, but this is completely unacceptable as a long term solution. Not only is it a hack, it cannot be integrated into the kubernetes cron-job system.

Documentation

Overview

Copyright © 2020 Colin Fox <greenenergy@gmail.com>

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.