s3bytes

package module
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 29, 2024 License: MIT Imports: 25 Imported by: 0

README

s3bytes

CI Go Report Card GitHub GitHub

s3bytes is a CLI tool that lists the size and number of objects in all buckets in one shot.

Features

  • High speed with concurrent processing
  • Various output formats
  • Simple processing using CloudWatch Metrics API
  • Simple command line flags

Options

List of possible values for flags as follows:

Option Description Allowed values Default value Environment Variable
--completion value -c value print completion scripts bash zsh pwsh - -
--profile value -p value set aws profile - - AWS_PROFILE
--log-level value -l value set log level debug info warn error info S3BYTES_LOG_LEVEL
--regions value1,value2... -r value1,value2... set target regions ap-south-1 eu-north-1 eu-west-3 eu-west-2 eu-west-1 ap-northeast-3 ap-northeast-2 ap-northeast-1 ca-central-1 sa-east-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2 All regions with no opt-in required -
--prefix value -P value set bucket name prefix - - -
--expression value -e value set filter expression for metric values > value >= value < value <= value == value != value - -
--metric-name value -m value set metric name of cloudwatch metrics BucketSizeBytes NumberOfObjects BucketSizeBytes -
--storage-type value -s value set storage type of s3 objects StandardStorage IntelligentTieringFAStorage IntelligentTieringIAStorage IntelligentTieringAAStorage IntelligentTieringAIAStorage IntelligentTieringDAAStorage StandardIAStorage StandardIASizeOverhead StandardIAObjectOverhead OneZoneIAStorage OneZoneIASizeOverhead ReducedRedundancyStorage GlacierIRSizeOverhead GlacierInstantRetrievalStorage GlacierStorage GlacierStagingStorage GlacierObjectOverhead GlacierS3ObjectOverhead DeepArchiveStorage DeepArchiveObjectOverhead DeepArchiveS3ObjectOverhead DeepArchiveStagingStorage AllStorageTypes StandardStorage -
--output value -o value set output type json text markdown backlog tsv text S3BYTES_OUTPUT_TYPE
--help -h show help - - -
--version -v print the version - - -

Output type

JSON format

[
  {
    "BucketName": "bucket0",
    "Region": "ap-northeast-1",
    "StorageType": "StandardStorage",
    "Bytes": 23373655,
    "ReadableBytes": "23 MiB"
  },
  {
    "BucketName": "bucket1",
    "Region": "ap-northeast-2",
    "StorageType": "StandardStorage",
    "Bytes": 134614,
    "ReadableBytes": "135 KiB"
  },
  {
    "BucketName": "bucket2",
    "Region": "us-east-1",
    "StorageType": "StandardStorage",
    "Bytes": 0,
    "ReadableBytes": "0 B"
  }
]

Text table format

$ s3bytes -o text
+------------+----------------+-----------------+----------+---------------+
| BucketName | Region         | StorageType     | Bytes    | ReadableBytes |
+------------+----------------+-----------------+----------+---------------+
| bucket0    | ap-northeast-1 | StandardStorage | 23373655 | 23 MiB        |
+------------+----------------+-----------------+----------+---------------+
| bucket1    | ap-northeast-2 | StandardStorage |   134614 | 135 KiB       |
+------------+----------------+-----------------+----------+---------------+
| bucket2    | us-east-1      | StandardStorage |        0 | 0 B           |
+------------+----------------+-----------------+----------+---------------+

Markdown table format

$ s3bytes -o markdown
| BucketName | Region         | StorageType     | Bytes    | ReadableBytes |
| ---------- | -------------- | --------------- | -------- | ------------- |
| bucket0    | ap-northeast-1 | StandardStorage | 23373655 | 23 MiB        |
| bucket1    | ap-northeast-2 | StandardStorage | 134614   | 135 KiB       |
| bucket2    | us-east-1      | StandardStorage | 0        | 0 B           |

Backlog table format

$ s3bytes -o backlog
| BucketName | Region         | StorageType     | Bytes    | ReadableBytes |h
| bucket0    | ap-northeast-1 | StandardStorage | 23373655 | 23 MiB        |
| bucket1    | ap-northeast-2 | StandardStorage |   134614 | 135 KiB       |
| bucket2    | us-east-1      | StandardStorage |        0 | 0 B           |

Installation

Install with homebrew

brew install nekrassov01/tap/s3bytes

Install with go

go install github.com/nekrassov01/s3bytes

Or download binary from releases

Shell completion

Supported Shells are as follows:

  • bash
  • zsh
  • pwsh
s3bytes -c bash|zsh|pwsh

Todo

  • Add code comment
  • Support directory buckets if possible

Author

nekrassov01

License

MIT

Documentation

Index

Constants

View Source 
const (
	// Version is the current version of s3bytes.
	Version string = "0.0.7"
)

Variables

This section is empty.

Functions

func CLI 

func CLI(ctx context.Context)

CLI is the entry point for the CLI.

func LoadAWSConfig 

func LoadAWSConfig(ctx context.Context, profile string, logger logging.Logger, logMode aws.ClientLogMode) (aws.Config, error)

LoadAWSConfig loads the AWS configuration.

Types

type CW 

type CW struct {
	*cloudwatch.Client
}

CW is a wrapper for the cloudwatch client.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

NewS3 is a constructor for the S3 client.

func NewClient 

func NewClient(cfg aws.Config) *Client

NewClient is a constructor for the Client.

func NewMockClient 

func NewMockClient(s3 IS3, cw ICW) *Client

NewMockClient is a constructor for the mock client.

type ICW 

type ICW interface {
	GetMetricData(ctx context.Context, params *cloudwatch.GetMetricDataInput, optFns ...func(*cloudwatch.Options)) (*cloudwatch.GetMetricDataOutput, error)
}

ICW is an interface for the cloudwatch client.

type IS3 

type IS3 interface {
	ListBuckets(ctx context.Context, params *s3.ListBucketsInput, optFns ...func(*s3.Options)) (*s3.ListBucketsOutput, error)
}

IS3 is an interface for the s3 client.

type Manager 

type Manager struct {
	*Client

	Buckets     []s3types.Bucket
	Batches     [][]cwtypes.MetricDataQuery
	Metrics     []Metric
	MetricName  MetricName
	StorageType StorageType
	MaxQueries  int
	Prefix      string // filter prefix for bucket names
	Region      string // current region state in process
	// contains filtered or unexported fields
}

Manager is a manager struct for the s3bytes package.

func NewManager 

func NewManager(ctx context.Context, client *Client, region, prefix, expr string, metricName MetricName, storageType StorageType) (*Manager, error)

NewManager creates a new manager.

func (*Manager) Debug 

func (man *Manager) Debug()

Debug prints a debug message.

func (*Manager) SetBuckets 

func (man *Manager) SetBuckets() error

SetBuckets sets the buckets.

func (*Manager) SetData 

func (man *Manager) SetData() error

SetData sets the metric data for the cloudwatch client.

func (*Manager) SetQueries 

func (man *Manager) SetQueries() error

SetQueries sets the metric data queries for the cloudwatch client. https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-dimensions.html

func (*Manager) String 

func (man *Manager) String() string

String returns a string representation of the manager.

type Metric 

type Metric interface {
	Label() string
	Value() float64
}

Metric is an interface for the metrics.

type MetricName 

type MetricName int

MetricName represents the metric name. 

const (
	// MetricNameBucketSizeBytes represents the bucket size in bytes.
	MetricNameBucketSizeBytes MetricName = iota

	// MetricNameNumberOfObjects represents the number of objects in the bucket.
	MetricNameNumberOfObjects
)

func ParseMetricName 

func ParseMetricName(s string) (MetricName, error)

ParseMetricName parses the metric name from the string representation.

func (MetricName) MarshalJSON 

func (t MetricName) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the metric name.

func (MetricName) String 

func (t MetricName) String() string

String returns the string representation of the metric name.

type MockCW 

type MockCW struct {
	GetMetricDataFunc func(ctx context.Context, params *cloudwatch.GetMetricDataInput, optFns ...func(*cloudwatch.Options)) (*cloudwatch.GetMetricDataOutput, error)
}

MockCW is a mock for the cloudwatch client.

func (*MockCW) GetMetricData 

func (m *MockCW) GetMetricData(ctx context.Context, params *cloudwatch.GetMetricDataInput, optFns ...func(*cloudwatch.Options)) (*cloudwatch.GetMetricDataOutput, error)

GetMetricData is a wrapper for the GetMetricData method.

type MockS3 

type MockS3 struct {
	ListBucketsFunc func(ctx context.Context, params *s3.ListBucketsInput, optFns ...func(*s3.Options)) (*s3.ListBucketsOutput, error)
}

MockS3 is a mock for the s3 client.

func (*MockS3) ListBuckets 

func (m *MockS3) ListBuckets(ctx context.Context, params *s3.ListBucketsInput, optFns ...func(*s3.Options)) (*s3.ListBucketsOutput, error)

ListBuckets is a wrapper for the ListBuckets method.

type ObjectMetric 

type ObjectMetric struct {
	BucketName  string
	Region      string
	StorageType StorageType
	Objects     float64
}

ObjectMetric is a struct for the object metric.

func (*ObjectMetric) Label added in v0.0.2 

func (t *ObjectMetric) Label() string

Label returns the label of the object metric.

func (*ObjectMetric) Value 

func (t *ObjectMetric) Value() float64

Value returns the value of the object metric.

type OutputType 

type OutputType int

OutputType represents the output type of the renderer. 

const (
	// OutputTypeJSON represents the JSON output type.
	OutputTypeJSON OutputType = iota

	// OutputTypeText represents the text output type.
	OutputTypeText

	// OutputTypeMarkdown represents the markdown output type.
	OutputTypeMarkdown

	// OutputTypeBacklog represents the backlog output type.
	OutputTypeBacklog

	// OutputTypeTSV represents the TSV output type.
	OutputTypeTSV
)

func ParseOutputType 

func ParseOutputType(s string) (OutputType, error)

ParseOutputType parses the output type from the string representation.

func (OutputType) MarshalJSON 

func (t OutputType) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the output type.

func (OutputType) String 

func (t OutputType) String() string

String returns the string representation of the output type.

type Renderer 

type Renderer struct {
	Metrics    []Metric
	MetricName MetricName
	OutputType OutputType
	// contains filtered or unexported fields
}

Renderer is a renderer struct for the s3bytes package.

func NewRenderer 

func NewRenderer(w io.Writer, metrics []Metric, metricName MetricName, outputType OutputType) *Renderer

NewRenderer creates a new renderer.

func (*Renderer) Render 

func (ren *Renderer) Render() error

Render renders the metrics.

func (*Renderer) String 

func (ren *Renderer) String() string

String returns a string representation of the renderer.

type S3 

type S3 struct {
	*s3.Client
}

S3 is a wrapper for the s3 client.

type SizeMetric 

type SizeMetric struct {
	BucketName    string
	Region        string
	StorageType   StorageType
	Bytes         float64
	ReadableBytes string // human readable bytes
}

SizeMetric is a struct for the size metric.

func (*SizeMetric) Label added in v0.0.2 

func (t *SizeMetric) Label() string

Label returns the label of the size metric.

func (*SizeMetric) Value 

func (t *SizeMetric) Value() float64

Value returns the value of the size metric.

type StorageType 

type StorageType int

StorageType represents the storage type. https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-dimensions.html#s3-cloudwatch-metrics 

const (
	StorageTypeStandardStorage StorageType = iota

	StorageTypeIntelligentTieringFAStorage
	StorageTypeIntelligentTieringIAStorage
	StorageTypeIntelligentTieringAAStorage
	StorageTypeIntelligentTieringAIAStorage
	StorageTypeIntelligentTieringDAAStorage

	StorageTypeStandardIAStorage
	StorageTypeStandardIASizeOverhead
	StorageTypeStandardIAObjectOverhead

	StorageTypeOneZoneIAStorage
	StorageTypeOneZoneIASizeOverhead

	StorageTypeReducedRedundancyStorage

	StorageTypeGlacierIRSizeOverhead
	StorageTypeGlacierInstantRetrievalStorage

	StorageTypeGlacierStorage
	StorageTypeGlacierStagingStorage
	StorageTypeGlacierObjectOverhead
	StorageTypeGlacierS3ObjectOverhead

	StorageTypeDeepArchiveStorage
	StorageTypeDeepArchiveObjectOverhead
	StorageTypeDeepArchiveS3ObjectOverhead
	StorageTypeDeepArchiveStagingStorage

	StorageTypeAllStorageTypes
)

func ParseStorageType 

func ParseStorageType(s string) (StorageType, error)

ParseStorageType parses the storage type from the string representation.

func (StorageType) MarshalJSON 

func (t StorageType) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the storage type.

func (StorageType) String 

func (t StorageType) String() string

String returns the string representation of the storage type.

Source Files

View all Source files

Directories

Path Synopsis
cmd
s3bytes

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.