bucketscanner

package module
v0.1.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 11, 2018 License: Apache-2.0 Imports: 10 Imported by: 0

README

Bucket Scanner

pipeline status [coverage report] (https://cjbarker.gitlab.io/bucketscanner/test-coverage.html) Go Report Card GitLab license

Overview

Searching Cloud Storage Since 2017

Usage

The bucketscanner requires a cloud provider and action coupled with the bucket name(s).

Usage: ./bucketscanner-darwin-amd64 --cloud=CLOUD --action=ACTION [<flags>] <bucket-name>

Cloud command-line bucket (object) scanner.

Flags:
  --help               Show context-sensitive help (also try --help-long and --help-man).
  --version            Show application version.
  --cloud=CLOUD        Cloud provider to scan: aws, gcp, azure. Defaults to all.
  --action=ACTION      Scan action to invoke against bucket: (r)ead, (w)rite, all. Defaults to all.
  --throttle=THROTTLE  Time in milliseconds to throttle subsequent requests sent to a given provider.
  --download           Download bucket content(s).
  --output=OUTPUT      Download bucket content(s) destination directory. Defaults to current user's directory if
                       none passed.
  --JSON               Output results in JSON.
  --verbose            Verbose output messages. Defaults to quiet.

Args:
  <bucket-name>  Bucket(s) name(s) to scan. Does support comma separated for multiple buckets.

Example searching one bucket on AWS for read-access:

./bucketscanner --cloud=aws --action=read --JSON listing-test
{
    "files": [
        {
            "Body": null,
            "directory": true,
            "files": null,
            "name": "empty folder/",
            "size": 0
        },
        {
            "Body": null,
            "directory": true,
            "files": null,
            "name": "empty folder/empty folder/",
            "size": 0
        },
        {
            "Body": null,
            "directory": false,
            "files": null,
            "name": "index-bucketname.html",
            "size": 362
        },
        {
            "Body": null,
            "directory": false,
            "files": null,
            "name": "index-null.html",
            "size": 323
        },
        {
            "Body": null,
            "directory": false,
            "files": null,
            "name": "index-path.html",
            "size": 385
        },
        {
            "Body": null,
            "directory": false,
            "files": null,
            "name": "index-vh.html",
            "size": 385
        }
    ],
    "name": "listing-test",
    "noFiles": 6,
    "provider": "Amazon Simple Storage Service (S3)",
    "scanned": "2018-04-11T11:31:16.78290151-07:00",
    "state": 3,
    "totalSize": 1455,
    "uri": "https://listing-test.s3.amazonaws.com"
}

Developer

Bucketscanner supports multiple platform builds via GNU Make. It does assume and rely on Glide for GoLang package management including dependencies. Please ensure glide is installed and available in your path before continuing.

To build the binary and library you'll need to clone the repo, setup GoLang and run make.

The default make target builds both components command line binary and library (bucketscanner and libbucketscanner).

# Assumes GOPATH exists and golang installed with tools in path
# export PATH=${GOPATH}/bin:${PATH}

cd ${GOPATH}/src
mkdir -p gitlab.com/cjbarker/
cd gitlab.com/cjbarker
git clone git@gitlab.com:cjbarker/bucketscanner.git
cd bucketscanner
make

# Built Binary & Library
ls bin/
bucketscanner libbucketscanner

Continous Integration

GitLab's CI Pipelines handle the continuos integration (CI) and eventually will also handle the continuous deployment (CD) to cloud provider (TBA).

All management of CI/CD is handled via the .gitlab-ci.yml file. For more details on GitLab CI and job configuration consult: https://docs.gitlab.com/ce/ci/yaml/README.html

Any commit to a branch will trigger the CI. If you do not want the pipeline's job(s) to trigger you can add a skip to your git commit message.

git add <file>
git commit -m "[skip ci] will not trigger GitLab CI job"
git push

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	Version string
	Build   string
)

Variables to identify the build

Functions

This section is empty.

Types

type AwsScanner

type AwsScanner struct {
	// contains filtered or unexported fields
}

AwsScanner is struct for cloud scanner of Amazon Web Services

func (AwsScanner) GetProviderName

func (a AwsScanner) GetProviderName() (cloudProviderName string)

GetProviderName returns the given Cloud Provider's name for the scanner

func (AwsScanner) Read

func (a AwsScanner) Read(name string) (bucket *Bucket, err error)

Read establishes HTTP connection and reads the contents from the bucket

func (AwsScanner) Write

func (a AwsScanner) Write(name string) (isWritable bool, err error)

Write attempts to write a temporary file to a given bucket within AWS

type AzureScanner

type AzureScanner struct {
}

AzureScanner is struct for cloud scanner of Azure

func (AzureScanner) GetProviderName

func (a AzureScanner) GetProviderName() (cloudProviderName string)

GetProviderName returns the given Cloud Provider's name for the scanner

func (AzureScanner) Read

func (a AzureScanner) Read(name string) (bucket *Bucket, err error)

func (AzureScanner) Write

func (a AzureScanner) Write(name string) (isWritable bool, err error)

type Bucket

type Bucket struct {
	Provider string      `json:"provider"`
	Name     string      `json:"name"`
	Scanned  time.Time   `json:"scanned"`
	URI      string      `json:"uri"`
	State    BucketState `json:"state"`
	NoFiles  int64       `json:"noFiles"`

	TotalSize int64  `json:"totalSize"`
	Files     []file `json:"files"`
	// contains filtered or unexported fields
}

Bucket structure is the results of a given bucket including its meta-data

func (Bucket) Download

func (b Bucket) Download(destDir string) (archivePath *string, err error)

Download the contents of the bucket to a given destination directory

type BucketState

type BucketState int

BucketState denotes an integer defining its given state

const (
	Unknown     BucketState = iota
	Invalid                 // Bucket does not exists e.g. 404 Not Found
	Private                 // Bucket exists but is not accessible e.g. 403 Forbidden
	Public                  // Bucket exists and is available e.g. 200 OK
	RateLimited             // Unable to determine due to rate limiting e.g. 503 Slow Down
)

Bucket states

type Contents

type Contents struct {
	XMLName      xml.Name `xml:"Contents"`
	Key          string
	LastModified string
	Etag         string `xml:"ETag"`
	Size         int
	StorageClass string
}

Contents are the XML contents of the bucket (one per file)

type GcpScanner

type GcpScanner struct {
}

GcpScanner is struct for cloud scanner of Google Cloud Platform

func (GcpScanner) GetProviderName

func (g GcpScanner) GetProviderName() (cloudProviderName string)

GetProviderName returns the given Cloud Provider's name for the scanner

func (GcpScanner) Read

func (g GcpScanner) Read(name string) (bucket *Bucket, err error)

func (GcpScanner) Write

func (g GcpScanner) Write(name string) (isWritable bool, err error)

type ListBucketResult

type ListBucketResult struct {
	XMLName      xml.Name `xml:"ListBucketResult"`
	Name         string
	Prefix       string
	Marker       string
	MaxKeys      int
	IsTruncated  bool
	ContentsList []Contents `xml:"Contents"`
}

ListBucketResult is the analyzed results read from a given AWS bucket

type Scanner

type Scanner interface {
	Read(name string) (bucket *Bucket, err error)
	Write(name string) (isWritable bool, err error)
	GetProviderName() (cloudProviderName string)
}

Scanner interface declares functions for cloud provider scanner to implement

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL