cloudwatcher

README

Cloudwatcher

File system notification for Cloud platforms (and not) in Golang.

cloudwatcher is a file system notification library for cloud platforms (and not) in Go. Currently it implements the watchers for the following services:

• Amazon S3
• Google Drive
• Dropbox
• Git
• local filesystem (fsnotify/polling)

It is possible specify the directory to monitor and the polling time (how much often the watcher should check that directory), and every time a new file is added, removed or changed, an event is generated and sent to the Events channel.

Usage

package main

import (
	"fmt"
	"time"

	"github.com/Matrix86/cloudwatcher"
)

func main() {
    // the first parameter is the type of service to use: local, dropbox, gdrive or s3
	s, err := cloudwatcher.New("local", "/home/user/tests", time.Second)
	if err != nil {
		fmt.Printf("ERROR: %s", err)
		return
	}

	config := map[string]string{
		"disable_fsnotify": "false",
	}

	err = s.SetConfig(config)
	if err != nil {
		fmt.Printf("ERROR: %s", err)
		return
	}

	err = s.Start()
	defer s.Close()
	for {
		select {
		case v := <-s.GetEvents():
			fmt.Printf("EVENT: %s %s\n", v.Key, v.TypeString())

		case e := <-s.GetErrors():
			fmt.Printf("ERROR: %s\n", e)
		}
	}
}

The channel returned by GetEvents() function, will return an Event struct that contains the event type, the Key with the name of the file that generates the event and the object itself.

⚠ check the Event.Object field before use it...in some cases it could be nil (FileDelete event with fsnotify)

Amazon S3

The config of the S3 watcher is the following:

config := map[string]string{
    "bucket_name": "storage",
    "endpoint":   "s3-us-west-2.amazonaws.com",
    "access_key":  "user",
    "secret_key":  "secret",
    "token":      "",
    "region":     "us-west-2",
    "ssl_enabled": "true",
}

To use AWS IAM credentials or AWS file, it is possible to set one of the following vars to "true": aws_iam_credentials, aws_file. It is also possible to specify the IAM endpoint (aws_file_profile) or the path of the file to use (aws_file_profile), if not specified it will use the default endpoint and file ($HOME/.aws/credentials).

config := map[string]string{
    "aws_file_profile": "true",
}

An example can be found here.

💎 minio can be used for testing purposes

Google Drive

In order to use it you need to enable the drive API from here . After that you can specify the client-id and the client-secret on the config file. The logic to retrieve the token has to be handled outside the library and the token field should contain the json.

You can find an example in the examples directory.

config := map[string]string{
    "debug":         "true",
    "token":         Token,
    "client_id":     ClientId,
    "client_secret": ClientSecret,
}

Dropbox

First, you need to register a new app from the developer console. Use the client-id and the client-secret to retrieve the user token and use it on the token field of the config.

You can find an example in the examples directory.

config := map[string]string{
    "debug": "true",
    "token": Token,
}

Local filesystem

It is based on fsnotify library, so it is cross platform: Windows, Linux, BSD and macOS. It is not mandatory to call the SetConfig() function, and the polling time argument of cloudwatcher.New is not used.

Setting disable_fsnotify parameter on config to "true" the watcher doesn't use fsnotify and use the listing approach instead.

⚠ not set disable_fsnotify to "true" if you plan to use it on a big directory!!! It could increase the I/O on disk

Git

Git watcher has the following configurations:

Name	Description
debug	if "true" the debug mode is enabled (default "false")
monitor_type	it can be "file" or "repo" (default is "repo")
auth_type	authentication type to use: "none", "ssh", "http_token", "http_user_pass" (default "none")
ssh_pkey	path of the ssh private key (required if auth_type = "ssh")
ssh_pkey_password	password of the private key if set
http_token	token to use if auth_type = "http_token"
http_username	username of github account (auth_type = "http_user_pass")
http_password	password of github account (auth_type = "http_user_pass")
repo_url	url of the repository
repo_branch	branch to watch (if monitor_type is "repo" you can leave it empty to watch all the branches)
assemble_events	if "true" the events could contain one or more commit events (only if monitor_type = "repo")
temp_dir	temporary directory to use for clone the repo: if empty the tmp dir will be used

If monitor_type is set to "repo", the event channel will receive an event with the Object field filled with commits or tags. If assemble_events is "true" the Object field could contains one or more commits.

Documentation

Index

• Constants
• type Bool
  • func (b Bool) MarshalJSON() ([]byte, error)
  • func (b *Bool) UnmarshalJSON(s []byte) error
• type DropboxObject
• type DropboxWatcher
  • func (w *DropboxWatcher) Close()
  • func (w *DropboxWatcher) SetConfig(m map[string]string) error
  • func (w *DropboxWatcher) Start() error
• type Event
  • func (e *Event) TypeString() string
• type GDriveObject
• type GDriveWatcher
  • func (w *GDriveWatcher) Close()
  • func (w *GDriveWatcher) SetConfig(m map[string]string) error
  • func (w *GDriveWatcher) Start() error
• type GitCommit
• type GitObject
• type GitWatcher
  • func (w *GitWatcher) Close()
  • func (w *GitWatcher) SetConfig(m map[string]string) error
  • func (w *GitWatcher) Start() error
• type IMinio
• type LocalObject
• type LocalWatcher
  • func (w *LocalWatcher) Close()
  • func (w *LocalWatcher) SetConfig(m map[string]string) error
  • func (w *LocalWatcher) Start() error
• type Op
• type S3Object
• type S3Watcher
  • func (u *S3Watcher) Close()
  • func (u *S3Watcher) SetConfig(m map[string]string) error
  • func (u *S3Watcher) Start() error
• type Watcher
  • func New(serviceName string, dir string, interval time.Duration) (Watcher, error)
• type WatcherBase
  • func (w *WatcherBase) GetErrors() chan error
  • func (w *WatcherBase) GetEvents() chan Event

Constants

const (
	FileCreated = iota
	FileChanged
	FileDeleted
	TagsChanged
)

event's types

const (
	// Version need a comment?
	Version = "1.2.5"
)

Types

type Bool

type Bool bool

Bool is an alias for the standard boolean type with Unmarshal/Marshal functions

func (Bool) MarshalJSON

func (b Bool) MarshalJSON() ([]byte, error)

MarshalJSON allows the conversion from bool to string

func (*Bool) UnmarshalJSON

func (b *Bool) UnmarshalJSON(s []byte) error

UnmarshalJSON allows to unmarshal from string to bool

type DropboxObject

type DropboxObject struct {
	Key          string
	Size         int64
	LastModified time.Time
	Hash         string
}

DropboxObject is the object that contains the info of the file

type DropboxWatcher

type DropboxWatcher struct {
	WatcherBase
	// contains filtered or unexported fields
}

DropboxWatcher is the specialized watcher for Dropbox service

func (*DropboxWatcher) Close

func (w *DropboxWatcher) Close()

Close stop the polling process

func (*DropboxWatcher) SetConfig

func (w *DropboxWatcher) SetConfig(m map[string]string) error

SetConfig is used to configure the DropboxWatcher

func (*DropboxWatcher) Start

func (w *DropboxWatcher) Start() error

Start launches the polling process

type Event

type Event struct {
	Key    string      // Path of file
	Type   Op          // File operation
	Object interface{} // Object pointer
}

Event is the struct that contains the info about the changed file

func (*Event) TypeString

func (e *Event) TypeString() string

TypeString returns a text version of the event's type

type GDriveObject

type GDriveObject struct {
	ID           string
	Key          string
	Size         int64
	LastModified time.Time
	Hash         string
}

GDriveObject is the object that contains the info of the file

type GDriveWatcher

type GDriveWatcher struct {
	WatcherBase
	// contains filtered or unexported fields
}

GDriveWatcher is the specialized watcher for Google Drive service

func (*GDriveWatcher) Close

func (w *GDriveWatcher) Close()

Close stop the polling process

func (*GDriveWatcher) SetConfig

func (w *GDriveWatcher) SetConfig(m map[string]string) error

SetConfig is used to configure the GDriveWatcher

func (*GDriveWatcher) Start

func (w *GDriveWatcher) Start() error

Start launches the polling process

type GitCommit

type GitCommit struct {
	Hash        string
	Message     string
	Branch      string
	AuthorName  string
	AuthorEmail string
	Time        time.Time
}

GitCommit is the object that contains the info about the commit

type GitObject

type GitObject struct {
	Key      string
	Size     int64
	FileMode os.FileMode
	Hash     string
	Commits  []*GitCommit
}

GitObject is the object that contains the info of the file

type GitWatcher

type GitWatcher struct {
	WatcherBase
	// contains filtered or unexported fields
}

GitWatcher is the specialized watcher for Git service

func (*GitWatcher) Close

func (w *GitWatcher) Close()

Close stop the polling process

func (*GitWatcher) SetConfig

func (w *GitWatcher) SetConfig(m map[string]string) error

SetConfig is used to configure the GitWatcher

func (*GitWatcher) Start

func (w *GitWatcher) Start() error

Start launches the polling process

type IMinio

type IMinio interface {
	BucketExists(ctx context.Context, bucketName string) (bool, error)
	GetObjectTagging(ctx context.Context, bucketName, objectName string, opts minio.GetObjectTaggingOptions) (*tags.Tags, error)
	ListObjects(ctx context.Context, bucketName string, opts minio.ListObjectsOptions) <-chan minio.ObjectInfo
}

IMinio is an interface I used only for gomock

type LocalObject

type LocalObject struct {
	Key          string
	Size         int64
	LastModified time.Time
	FileMode     os.FileMode
}

LocalObject is the object that contains the info of the file

type LocalWatcher

type LocalWatcher struct {
	WatcherBase
	// contains filtered or unexported fields
}

LocalWatcher is the specialized watcher for the local FS

func (*LocalWatcher) Close

func (w *LocalWatcher) Close()

Close stop the polling process

func (*LocalWatcher) SetConfig

func (w *LocalWatcher) SetConfig(m map[string]string) error

SetConfig is used to configure the LocalWatcher

func (*LocalWatcher) Start

func (w *LocalWatcher) Start() error

Start launches the polling process

type Op

type Op uint32

Op defines the event's type

type S3Object

type S3Object struct {
	Key          string
	Etag         string
	Size         int64
	Tags         map[string]string
	LastModified time.Time
}

S3Object is the object that contains the info of the file

type S3Watcher

type S3Watcher struct {
	WatcherBase
	// contains filtered or unexported fields
}

S3Watcher is the specialized watcher for Amazon S3 service

func (*S3Watcher) Close

func (u *S3Watcher) Close()

Close stop the polling process

func (*S3Watcher) SetConfig

func (u *S3Watcher) SetConfig(m map[string]string) error

SetConfig is used to configure the S3Watcher

func (*S3Watcher) Start

func (u *S3Watcher) Start() error

Start launches the polling process

type Watcher

type Watcher interface {
	Start() error
	SetConfig(c map[string]string) error
	Close()
	GetEvents() chan Event
	GetErrors() chan error
}

Watcher has to be implemented by all the watchers

func New

func New(serviceName string, dir string, interval time.Duration) (Watcher, error)

New creates a new instance of a watcher

type WatcherBase

type WatcherBase struct {
	Events chan Event
	Errors chan error
	// contains filtered or unexported fields
}

WatcherBase is the struct included in all the specialized watchers

func (*WatcherBase) GetErrors

func (w *WatcherBase) GetErrors() chan error

GetErrors returns a chan of error

func (*WatcherBase) GetEvents

func (w *WatcherBase) GetEvents() chan Event

GetEvents returns a chan of Event