util

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 14, 2023 License: GPL-3.0 Imports: 33 Imported by: 0

README

util

The code in this project is designed to be reusable across many different Go projects. It's designed to be completely general and can be reused without any adaptation.

This library calls panic in a number of places so you may want to capture both stdout and stderr like this:

go run ./fileserver/main.go 2>stderr.log 1>stdout.log

To run all tests:

go test $(go list ./... | grep -v manual)

For an overview of this package see https://pkg.go.dev/github.com/1f604/util

The AWS dependencies are only needed for the cloudflare stuff.

Documentation

Overview

Map with expiring entries For slightly better performance, replace map[string]string with map[int64]string. See https://www.komu.engineer/blogs/01/go-gc-maps Memory usage can be more than double what you actually store in it. Based on my own testing, storing 10 million 128 byte URLs will take around 3.6GB of RAM, so each 128 byte URL took around 360 bytes of RAM. Entries can only be inserted, they cannot be updated or deleted before they expire. Uses sync.Mutex to protect concurrent access. Adding, getting, and removing entries require obtaining the mutex first. TODO: Benchmark switching to use a RWMutex or a sync.Map for improved performance. I tested sync.Map, it apparently has no reserve feature? Bulk load is slow - 7.8 seconds. Heap-based implementation for performance and simplicity Benchmarks show that Remove_All_Expired takes 3 seconds to remove 10 million expired entries Benchmarks show that NewConcurrentExpiringMapFromSlice takes 3.5 seconds to load 10 million entries No requirement for entries to have same TTL duration No support for updating expiry time - though this functionality can be added later if necessary. Example use cases: 1. Expiring short URLs - short URL -> long URL map 2. Expiring pastebins - short URL -> file path map 3. Expiring tokens - token -> expiry time map Map will return an error for expired entries

It provides an API that has 3 methods: 1. Update map size rounded 2. Append new entry to log file 3. Delete expired log files As well as an "expiring" LogStructuredStorage where you can remove expired entries from old log files and rewrite them into new log files The reason this works is because it's okay to see the same entry multiple times since we'll just ignore it when we see the same entry again We ignore entries that expire earlier than the current entry we have, and we overwrite the current entry as soon as we see another entry that has a later expiration time

This file provides 2 structs: a "base" LogStructuredStorage for storing permanent data As well as an "expiring" LogStructuredStorage where you can remove expired entries from old log files and rewrite them into new log files The reason this works is because it's okay to see the same entry multiple times since we'll just ignore it when we see the same entry again We ignore entries that expire earlier than the current entry we have, and we overwrite the current entry as soon as we see another entry that has a later expiration time

Deletes the oldest files until the size of directory is back within limit

It provides an API that has 3 methods: 1. Update map size rounded 2. Append new entry to log file 3. Delete expired log files As well as an "expiring" LogStructuredStorage where you can remove expired entries from old log files and rewrite them into new log files The reason this works is because it's okay to see the same entry multiple times since we'll just ignore it when we see the same entry again We ignore entries that expire earlier than the current entry we have, and we overwrite the current entry as soon as we see another entry that has a later expiration time

This file provides 2 structs: a "base" LogStructuredStorage for storing permanent data As well as an "expiring" LogStructuredStorage where you can remove expired entries from old log files and rewrite them into new log files The reason this works is because it's okay to see the same entry multiple times since we'll just ignore it when we see the same entry again We ignore entries that expire earlier than the current entry we have, and we overwrite the current entry as soon as we see another entry that has a later expiration time

The purpose of this file is for storing the map size on disk. The purpose of the map size file is to make it faster to load entries from disk into map By storing the size of the map in the file, the next time on program on startup creates the map, the make() function can be given the correct size It doesn't matter if it's slightly too big or small since it won't affect the performance much. Better to be too big than too small to avoid resizing which is costly

This is a horrible hack to work around Go's lack of built-in support for type-safe enums True type safe enums are not possible in Go due to its lack of sum types This is a "best effort" workaround that will prevent some kinds of bugs but is not perfect

We use byte instead of rune because this alphabet contains only printable ASCII characters.

Index

Constants

View Source
const BASE53_ALPHABET_SIZE = 53
View Source
const XATTR_1F604_FILESERVER_CAN_BE_SERVED = "user.1f604.fileserver.can_be_served"

Variables

This section is empty.

Functions

func Assert_error_equals

func Assert_error_equals(t *testing.T, err error, expected string, skip_level int)

func Assert_no_error

func Assert_no_error(t *testing.T, err error, skip_level int)

func Assert_result_equals_bool

func Assert_result_equals_bool(t *testing.T, actual bool, err error, expected bool, skip_level int)

func Assert_result_equals_bytes

func Assert_result_equals_bytes(t *testing.T, actual []byte, err error, expected string, skip_level int)

func Assert_result_equals_interface

func Assert_result_equals_interface(t *testing.T, actual interface{}, err error, expected interface{}, skip_level int)

func Assert_result_equals_string_slice

func Assert_result_equals_string_slice(t *testing.T, actual []string, err error, expected []string, skip_level int)

func Assert_result_equals_time

func Assert_result_equals_time(t *testing.T, actual time.Time, err error, expected time.Time, line_number int)

func BuildStruct

func BuildStruct[T any]() *T

func Check_err

func Check_err(err error)

func Check_no_other_instances_running

func Check_no_other_instances_running(socket_addr string)

func Compute_String_Checksum added in v1.0.0

func Compute_String_Checksum(str string) string

func Convert_str_to_uint64 added in v1.0.0

func Convert_str_to_uint64(input_str string) uint64

func Convert_uint64_to_str added in v1.0.0

func Convert_uint64_to_str(bigendian_uint64 uint64, length int) string

func Copy_Slice_Into_150_Arr

func Copy_Slice_Into_150_Arr(slice []byte, arr [150]byte)

func Crypto_RandString added in v0.10.12

func Crypto_RandString(length int) string

Returns random string consisting of letters and numbers

func Crypto_Randint

func Crypto_Randint(max int) (int, error)

This function works, I've manually tested it. Returns integers from 0 up to AND NOT INCLUDING max

func Crypto_Random_Choice

func Crypto_Random_Choice[T any](arr *[]T) (T, error)

This function works, I've manually tested it.

func Divmod

func Divmod(numerator, denominator int) (int, int)

func Get_file_size

func Get_file_size(f *os.File) int64

this function assumes file pointer is valid. We could probably make this more efficient by calculating the file size in-process instead of making syscall each time.

func Getxattr

func Getxattr(path string, name string, data []byte) (int, error)

func Int64_to_string

func Int64_to_string(num int64) string

func IsSameType added in v1.0.0

func IsSameType(a, b interface{}) bool

This function works, I've manually tested it.

func LBSES_Get_bucket_filename added in v1.0.0

func LBSES_Get_bucket_filename(timestamp int64) string

func LBSES_Parse_bucket_filename_to_timestamp added in v1.0.0

func LBSES_Parse_bucket_filename_to_timestamp(filename string) (int64, error)

func LSPS_Parse_log_filename_to_number added in v1.0.0

func LSPS_Parse_log_filename_to_number(filename string) (int64, error)

func LoadStoredRecordsFromDisk added in v1.0.0

func LoadStoredRecordsFromDisk(params *LSRFD_Params) (ConcurrentMap, *MapSizeFileManager)

This is the one you want to use in production

func Power_Naive

func Power_Naive(a, b int) int

Naive algorithm, only suitable for small b.

func Power_Slow

func Power_Slow(a, b, m int) int

calculates a to the power of b mod m. If m is 0 then just returns a to the power of b. This function seems to create a memory leak, but it doesn't. Anyway, it's better to use custom power

func PrintMemUsage

func PrintMemUsage()

PrintMemUsage outputs the current, total and OS memory being used. As well as the number of garage collection cycles completed.

func PutEntry_Common added in v1.0.0

func PutEntry_Common(requested_length int, long_url string, timestamp int64, generate_strings_up_to int,
	slice_storage map[int]*RandomBag64, urlmap URLMap, b53m *Base53IDManager, log_storage LogStorage, map_size_persister *MapSizeFileManager) (string, error)

Shorten long URL into short URL and return the short URL and store the entry both in map and on disk

func ReplaceString

func ReplaceString(str string, replacement rune, index int) string

func Retryfunc

func Retryfunc(taskname string, dotask retrylib_task, expected_duration time.Duration, max_wait time.Duration)

func Retryproc

func Retryproc(procname string, expected_duration time.Duration, max_wait time.Duration)

func ReverseString

func ReverseString(s string) string

func RunFuncEveryXSeconds added in v1.0.0

func RunFuncEveryXSeconds(fn fn_type, run_interval_seconds int)

Does not tick shift - will run function precisely every X seconds even if function takes some time to run - as long as the function doesn't take too long of course.

Synchronous - next call cannot start until previous call has finished.

func Setxattr

func Setxattr(path string, name string, data []byte, flags int) error

func String_to_int64

func String_to_int64(s string) (int64, error)

func Validate_Timestamp_Common added in v1.0.0

func Validate_Timestamp_Common(timestamp_unix int64) error

Returns error if unix timestamp is before 2023 or after the year 20,000

Otherwise returns nil

func Write_Entry_To_File added in v1.0.0

func Write_Entry_To_File(key string, value string, timestamp int64, file_handle *os.File) error

IMPORTANT: This function DOES NOT close the file handle!!!

Types

type BandwidthMonitor

type BandwidthMonitor struct {
	// contains filtered or unexported fields
}

func (*BandwidthMonitor) GetTotalAllBytes

func (bm *BandwidthMonitor) GetTotalAllBytes() int64

func (*BandwidthMonitor) GetTotalTXBytes

func (bm *BandwidthMonitor) GetTotalTXBytes() int64

func (*BandwidthMonitor) RunThread

func (bm *BandwidthMonitor) RunThread(time_interval_secs int)

type Base53ErrorChecksumMismatch

type Base53ErrorChecksumMismatch struct{}

func (Base53ErrorChecksumMismatch) Error

type Base53ErrorIllegalCharacter

type Base53ErrorIllegalCharacter struct{}

func (Base53ErrorIllegalCharacter) Error

type Base53ErrorIllegalPair

type Base53ErrorIllegalPair struct{}

func (Base53ErrorIllegalPair) Error

func (e Base53ErrorIllegalPair) Error() string

type Base53ErrorStrWithoutCsumTooLong

type Base53ErrorStrWithoutCsumTooLong struct{}

func (Base53ErrorStrWithoutCsumTooLong) Error

type Base53ErrorStrWithoutCsumTooShort

type Base53ErrorStrWithoutCsumTooShort struct{}

Custom error types

func (Base53ErrorStrWithoutCsumTooShort) Error

type Base53ID

type Base53ID interface {
	GetStrWithoutCsum() string
	GetCsum() byte
	GetCombinedString() string
	Length() int
}

See https://stackoverflow.com/questions/57993809/how-to-hide-the-default-type-constructor-in-golang

type Base53IDManager

type Base53IDManager struct {
	// contains filtered or unexported fields
}

func NewBase53IDManager

func NewBase53IDManager() *Base53IDManager

pregenerate means strings up to n characters will be pre-generated and stored in RandomBags for fast PopRandom and Push later.

func (*Base53IDManager) B53_generate_all_Base53IDs

func (b53m *Base53IDManager) B53_generate_all_Base53IDs(n int) ([]Base53ID, error)

Generate all IDs of length n

func (*Base53IDManager) B53_generate_all_Base53IDs_int64

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_all_Base53IDs_int64_optimized

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64_optimized(n int, should_be_added_fn ShouldBase53IDBePlacedIntoSliceFn) ([]uint64, error)

Doesn't push it into slice if it's already in map.

func (*Base53IDManager) B53_generate_all_Base53IDs_int64_test

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64_test(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_next_Base53ID

func (b53m *Base53IDManager) B53_generate_next_Base53ID(old_id Base53ID) (Base53ID, error)

func (*Base53IDManager) B53_generate_random_Base53ID

func (b53m *Base53IDManager) B53_generate_random_Base53ID(n int) (Base53ID, error)

func (*Base53IDManager) Convert_uint64_to_Base53ID

func (b53m *Base53IDManager) Convert_uint64_to_Base53ID(bigendian_uint64 uint64, length int) (*_base53ID_impl, error)

func (*Base53IDManager) Convert_uint64_to_byte_array

func (b53m *Base53IDManager) Convert_uint64_to_byte_array(bigendian_uint64 uint64) []byte

func (*Base53IDManager) NewBase53ID

func (b53m *Base53IDManager) NewBase53ID(str_without_csum string, csum byte, remap bool) (*_base53ID_impl, error)

Construction is validation.

type CEMItem

type CEMItem struct {
	Key              string
	Value            string
	Expiry_time_unix int64
}

type CEMNonExistentKeyError added in v1.0.0

type CEMNonExistentKeyError struct{}

func (CEMNonExistentKeyError) Error added in v1.0.0

func (e CEMNonExistentKeyError) Error() string

type CEPUMParams added in v1.0.0

type CEPUMParams struct {
	Expiry_check_interval_seconds_ram  int
	Expiry_check_interval_seconds_disk int
	Extra_keeparound_seconds_ram       int64
	Extra_keeparound_seconds_disk      int64
	Bucket_interval                    int64
	Bucket_directory_path_absolute     string
	Size_file_path_absolute            string
	B53m                               *Base53IDManager
	Size_file_rounded_multiple         int64
	Generate_strings_up_to             int
}

type CPMNonExistentKeyError added in v1.0.0

type CPMNonExistentKeyError struct{}

func (CPMNonExistentKeyError) Error added in v1.0.0

func (e CPMNonExistentKeyError) Error() string

type CPPUMParams added in v1.0.0

type CPPUMParams struct {
	Log_directory_path_absolute string
	B53m                        *Base53IDManager
	Generate_strings_up_to      int
	Log_file_max_size_bytes     int64
	Size_file_rounded_multiple  int64
	Size_file_path_absolute     string
}

type ConcurrentExpiringMap

type ConcurrentExpiringMap struct {
	// contains filtered or unexported fields
}

keys are strings

func NewConcurrentExpiringMapFromSlice

func NewConcurrentExpiringMapFromSlice(expiry_callback ExpiryCallback, kv_pairs []CEMItem) *ConcurrentExpiringMap

batched mode for fast loading from disk Takes around 3.5s to load 10 million items, 300ms for loading 1 million items

func NewEmptyConcurrentExpiringMap

func NewEmptyConcurrentExpiringMap(expiry_callback ExpiryCallback) *ConcurrentExpiringMap

func (*ConcurrentExpiringMap) BeginConstruction added in v1.0.0

func (*ConcurrentExpiringMap) BeginConstruction(stored_map_length int64, expiry_callback ExpiryCallback) ConcurrentMap

This method properly constructs the object

func (*ConcurrentExpiringMap) ContinueConstruction added in v1.0.0

func (cem *ConcurrentExpiringMap) ContinueConstruction(key_str string, value_str string, expiry_time int64)

Caller must check that the key_str is not already in the map.

func (*ConcurrentExpiringMap) FinishConstruction added in v1.0.0

func (cem *ConcurrentExpiringMap) FinishConstruction()

func (*ConcurrentExpiringMap) GetAllItems added in v0.10.13

func (cem *ConcurrentExpiringMap) GetAllItems() map[string]ExpiringMapItem

func (*ConcurrentExpiringMap) Get_Entry

func (cem *ConcurrentExpiringMap) Get_Entry(key string) (MapItem, error)

func (*ConcurrentExpiringMap) NumItems

func (cem *ConcurrentExpiringMap) NumItems() int

func (*ConcurrentExpiringMap) Put_New_Entry

func (cem *ConcurrentExpiringMap) Put_New_Entry(key string, value string, expiry_time int64) error

Will only return an error if the key already exists.

func (*ConcurrentExpiringMap) Remove_All_Expired

func (cem *ConcurrentExpiringMap) Remove_All_Expired(extra_keeparound_seconds int64)

keep links around for extra_keeparound_seconds just to tell people that the link has expired this function will remove 10 million entries in 3 seconds

type ConcurrentExpiringPersistentURLMap added in v1.0.0

type ConcurrentExpiringPersistentURLMap struct {
	// contains filtered or unexported fields
}

func CreateConcurrentExpiringPersistentURLMapFromDisk added in v1.0.0

func CreateConcurrentExpiringPersistentURLMapFromDisk(cepum_params *CEPUMParams) *ConcurrentExpiringPersistentURLMap

This is the one you want to use in production

func (*ConcurrentExpiringPersistentURLMap) GetEntry added in v1.0.0

func (manager *ConcurrentExpiringPersistentURLMap) GetEntry(short_url string) (MapItem, error)

func (*ConcurrentExpiringPersistentURLMap) PrintInternalState added in v1.0.0

func (manager *ConcurrentExpiringPersistentURLMap) PrintInternalState()

func (*ConcurrentExpiringPersistentURLMap) PutEntry added in v1.0.0

func (manager *ConcurrentExpiringPersistentURLMap) PutEntry(requested_length int, long_url string, expiry_time int64) (string, error)

Shorten long URL into short URL and return the short URL and store the entry both in map and on disk

func (*ConcurrentExpiringPersistentURLMap) RemoveAllExpiredURLsFromDisk added in v1.0.0

func (manager *ConcurrentExpiringPersistentURLMap) RemoveAllExpiredURLsFromDisk()

Removed expired URLs from disk every x seconds

func (*ConcurrentExpiringPersistentURLMap) RemoveAllExpiredURLsFromRAM added in v1.0.0

func (manager *ConcurrentExpiringPersistentURLMap) RemoveAllExpiredURLsFromRAM()

Removed expired URLs from map in RAM every x seconds

type ConcurrentMap added in v1.0.0

type ConcurrentMap interface {
	Get_Entry(string) (MapItem, error)
	BeginConstruction(int64, ExpiryCallback) ConcurrentMap
	ContinueConstruction(string, string, int64)
	FinishConstruction()
	NumItems() int
}

type ConcurrentPermanentMap

type ConcurrentPermanentMap struct {
	// contains filtered or unexported fields
}

keys are strings

func NewEmptyConcurrentPermanentMap

func NewEmptyConcurrentPermanentMap() *ConcurrentPermanentMap

func (*ConcurrentPermanentMap) BeginConstruction added in v1.0.0

func (*ConcurrentPermanentMap) BeginConstruction(stored_map_length int64, expiry_callback ExpiryCallback) ConcurrentMap

You can call this on nil receiver

func (*ConcurrentPermanentMap) ContinueConstruction added in v1.0.0

func (cpm *ConcurrentPermanentMap) ContinueConstruction(key_str string, value_str string, expiry_time int64)

Caller must check that the key_str is not already in the map.

func (*ConcurrentPermanentMap) FinishConstruction added in v1.0.0

func (cpm *ConcurrentPermanentMap) FinishConstruction()

func (*ConcurrentPermanentMap) Get_Entry

func (cpm *ConcurrentPermanentMap) Get_Entry(key string) (MapItem, error)

func (*ConcurrentPermanentMap) NumItems

func (cpm *ConcurrentPermanentMap) NumItems() int

func (*ConcurrentPermanentMap) Put_New_Entry added in v1.0.0

func (cpm *ConcurrentPermanentMap) Put_New_Entry(key string, value string, _ int64) error

Returns an error if the entry already exists, otherwise returns nil.

type ConcurrentPersistentPermanentURLMap added in v1.0.0

type ConcurrentPersistentPermanentURLMap struct {
	// contains filtered or unexported fields
}

func CreateConcurrentPersistentPermanentURLMapFromDisk added in v1.0.0

func CreateConcurrentPersistentPermanentURLMapFromDisk(cppum_params *CPPUMParams) *ConcurrentPersistentPermanentURLMap

This is the one you want to use in production

func (*ConcurrentPersistentPermanentURLMap) GetEntry added in v1.0.0

func (manager *ConcurrentPersistentPermanentURLMap) GetEntry(short_url string) (MapItem, error)

func (*ConcurrentPersistentPermanentURLMap) PrintInternalState added in v1.0.0

func (manager *ConcurrentPersistentPermanentURLMap) PrintInternalState()

func (*ConcurrentPersistentPermanentURLMap) PutEntry added in v1.0.0

func (manager *ConcurrentPersistentPermanentURLMap) PutEntry(requested_length int, long_url string, _ int64) (string, error)

Shorten long URL into short URL and return the short URL and store the entry both in map and on disk

type CryptoRandomChoiceEmptySliceError

type CryptoRandomChoiceEmptySliceError struct{}

Custom error types

func (CryptoRandomChoiceEmptySliceError) Error

type ERROR

type ERROR struct {
	S string
}

func Error

func Error(s string) ERROR

type EXACT_MATCH_HANDLER_t

type EXACT_MATCH_HANDLER_t struct{}
var EXACT_MATCH_HANDLER EXACT_MATCH_HANDLER_t = EXACT_MATCH_HANDLER_t{}

type ExpiringHeapItem

type ExpiringHeapItem struct {
	// contains filtered or unexported fields
}

func (ExpiringHeapItem) String

func (p ExpiringHeapItem) String() string

type ExpiringHeapQueue

type ExpiringHeapQueue []*ExpiringHeapItem

============= All this stuff is just to implement the interface required by heap ===================

func (ExpiringHeapQueue) Len

func (pq ExpiringHeapQueue) Len() int

func (ExpiringHeapQueue) Less

func (pq ExpiringHeapQueue) Less(i, j int) bool

func (*ExpiringHeapQueue) Pop

func (pq *ExpiringHeapQueue) Pop() any

func (*ExpiringHeapQueue) Push

func (pq *ExpiringHeapQueue) Push(x any)

func (ExpiringHeapQueue) Swap

func (pq ExpiringHeapQueue) Swap(i, j int)

type ExpiringMapItem

type ExpiringMapItem struct {
	// contains filtered or unexported fields
}

func (*ExpiringMapItem) GetValue added in v1.0.0

func (emi *ExpiringMapItem) GetValue() string

func (*ExpiringMapItem) MapItemToString added in v1.0.0

func (emi *ExpiringMapItem) MapItemToString() string

func (ExpiringMapItem) String

func (p ExpiringMapItem) String() string

type ExpiryCallback

type ExpiryCallback func(string)

type FileEntry

type FileEntry struct {
	FilePath    string
	FileInfo    fs.FileInfo
	TimeCreated time.Time
}

type HandlerTypeEnum

type HandlerTypeEnum interface {
	// contains filtered or unexported methods
}

type KeyAlreadyExistsError

type KeyAlreadyExistsError struct{}

func (KeyAlreadyExistsError) Error

func (e KeyAlreadyExistsError) Error() string

type KeyExpiredError

type KeyExpiredError struct {
	// contains filtered or unexported fields
}

func (KeyExpiredError) Error

func (e KeyExpiredError) Error() string

type LONGEST_PREFIX_HANDLER_t

type LONGEST_PREFIX_HANDLER_t struct{}
var LONGEST_PREFIX_HANDLER LONGEST_PREFIX_HANDLER_t = LONGEST_PREFIX_HANDLER_t{}

type LSRFD_Params added in v1.0.0

type LSRFD_Params struct {
	B53m                        *Base53IDManager
	Log_directory_path_absolute string
	Size_file_path_absolute     string
	Entry_should_be_ignored_fn  func(int64) bool
	Lss                         LogStructuredStorage
	Expiry_callback             ExpiryCallback
	Slice_storage               map[int]*RandomBag64
	Nil_ptr                     ConcurrentMap
	Size_file_rounded_multiple  int64
	Generate_strings_up_to      int
}

type LogBucketStructuredExpiringStorage added in v1.0.0

type LogBucketStructuredExpiringStorage struct {
	// contains filtered or unexported fields
}

func NewLogBucketStructuredExpiringStorage added in v1.0.0

func NewLogBucketStructuredExpiringStorage(bucket_interval int64, bucket_directory_path_absolute string) *LogBucketStructuredExpiringStorage

the bucket interval is the all-important parameter that determines the number of buckets and when buckets will be deleted the bucket interval is in Unix time (seconds). it means that all entries between two time points will go into one bucket when that bucket expires, it will be deleted example: if bucket interval is 100, then all timestamps from 0 to 100 will go into one bucket, all timestamps 100 to 200 will go into next bucket and so on bucketing is done simply by the / (round-to-zero division) operation. expiry time will be divided by the bucket interval and placed into appropriate bucket (log file) e.g. if bucket interval is 100, then all timestamps from 86400 to 86500 will go into bucket 865 e.g. if bucket interval is 200, then all timestamps from 1200 to 1400 will go into bucket 7, all timestamps from 1400 to 1600 will go to bucket 8 and so on. e.g. if bucket interval is 200, then bucket 200 holds all timestamps 0-199, bucket 400 holds all timestamps 200-399, bucket 600 holds 400-599, and so on. bucket files are named "expires_before_18400" where the last number is a unix timestamp

func (*LogBucketStructuredExpiringStorage) AppendNewEntry added in v1.0.0

func (lbses *LogBucketStructuredExpiringStorage) AppendNewEntry(key string, value string, expiry_time int64) error

Adds a new entry to the log file

Also important: Make sure the input does not contain carriage return or newline.

func (*LogBucketStructuredExpiringStorage) DeleteExpiredLogFiles added in v1.0.0

func (lbses *LogBucketStructuredExpiringStorage) DeleteExpiredLogFiles(extra_keeparound_seconds_disk int64)

Delete expired buckets (log files) extra_keeparound_seconds_disk defines how long to keep around log files after they expired

func (*LogBucketStructuredExpiringStorage) ValidateLogFilename added in v1.0.0

func (*LogBucketStructuredExpiringStorage) ValidateLogFilename(filename string) error

Can be called with nil receiver.

type LogFileDeleter

type LogFileDeleter struct {
	AbsoluteDirectoryPath   string
	CurrentLogFileName      string
	DirectorySizeLimitBytes int64
}

func NewLogFileDeleter

func NewLogFileDeleter(directory_path string, size_limit int64, log_file_name string) *LogFileDeleter

func (*LogFileDeleter) Delete_Excess_Files

func (lfd *LogFileDeleter) Delete_Excess_Files()

func (*LogFileDeleter) RunThread

func (lfd *LogFileDeleter) RunThread(time_interval_secs int)

type LogStorage added in v1.0.0

type LogStorage interface {
	AppendNewEntry(string, string, int64) error
}

type LogStructuredPermanentStorage added in v1.0.0

type LogStructuredPermanentStorage struct {
	// contains filtered or unexported fields
}

func NewLogStructuredPermanentStorage added in v1.0.0

func NewLogStructuredPermanentStorage(log_file_max_size int64, log_directory_path_absolute string) *LogStructuredPermanentStorage

Works just like the log rotation library - once log file reaches the max size, create a new log file Except we don't need any clever naming scheme, just an increasing number will do, since we're going to read in every file on startup anyway The increasing number naming scheme is actually good for cloud backups since we can just send the highest numbered file every time

func (*LogStructuredPermanentStorage) AppendNewEntry added in v1.0.0

func (lsps *LogStructuredPermanentStorage) AppendNewEntry(key string, value string, generation_time_unix int64) error

Adds a new entry to the log file

Also important: Make sure the input does not contain carriage return or newline.

func (*LogStructuredPermanentStorage) ValidateLogFilename added in v1.0.0

func (*LogStructuredPermanentStorage) ValidateLogFilename(filename string) error

Can be called with nil receiver.

type LogStructuredStorage added in v1.0.0

type LogStructuredStorage interface {
	ValidateLogFilename(filename string) error
}

type MapItem added in v1.0.0

type MapItem interface {
	MapItemToString() string
	GetValue() string
}

func GetEntryCommon added in v1.0.0

func GetEntryCommon(cm ConcurrentMap, short_url string) (MapItem, error)

type MapItem2 added in v1.0.0

type MapItem2 struct {
	// contains filtered or unexported fields
}

type MapSizeFileManager added in v1.0.0

type MapSizeFileManager struct {
	// contains filtered or unexported fields
}

func NewMapSizeFileManager added in v1.0.0

func NewMapSizeFileManager(size_file_path_absolute string, size_multiple int64) *MapSizeFileManager

func (*MapSizeFileManager) UpdateMapSizeRounded added in v1.0.0

func (msfm *MapSizeFileManager) UpdateMapSizeRounded(actual_size int64)

rounds to the nearest size

You can call this function as many times as you like, since it only updates file if rounded size has changed.

type NewBase53IDParams

type NewBase53IDParams struct {
	Str_without_csum string
	Csum             byte
	Remap            bool
}

inspired by StripeIntentParams

type NonExistentKeyError

type NonExistentKeyError interface {
	NonExistentKeyError() string
}

type People added in v1.0.0

type People []MapItem2

Define a slice of Person structs

func (People) Len added in v1.0.0

func (p People) Len() int

Implement the Len method required by sort.Interface

func (People) Less added in v1.0.0

func (p People) Less(i, j int) bool

Implement the Less method required by sort.Interface

func (People) Swap added in v1.0.0

func (p People) Swap(i, j int)

Implement the Swap method required by sort.Interface

type PermanentMapItem added in v1.0.0

type PermanentMapItem struct {
	// contains filtered or unexported fields
}

func (*PermanentMapItem) GetValue added in v1.0.0

func (pmi *PermanentMapItem) GetValue() string

func (*PermanentMapItem) MapItemToString added in v1.0.0

func (pmi *PermanentMapItem) MapItemToString() string

type RESULT

type RESULT struct {
	S interface{}
}

func Result

func Result(s interface{}) RESULT

type RandomBag64

type RandomBag64 struct {
	// contains filtered or unexported fields
}

func CreateRandomBagFromSlice

func CreateRandomBagFromSlice(items []uint64) *RandomBag64

The RandomBag steals the slice that you pass to it. You should not use the slice anywhere afterwards.

func (*RandomBag64) PopRandom

func (rb *RandomBag64) PopRandom() (uint64, error)

Removes from array and swaps last element into it

Will only return an error if the bag is empty.

func (*RandomBag64) Push

func (rb *RandomBag64) Push(item uint64)

Push should always succeed

func (*RandomBag64) Size

func (rb *RandomBag64) Size() int

type RandomBagEmptyError

type RandomBagEmptyError struct{}

func (RandomBagEmptyError) Error

func (e RandomBagEmptyError) Error() string

type SafeTLSAutoCertManager

type SafeTLSAutoCertManager struct {
	// contains filtered or unexported fields
}

func NewSafeAutoCertManager

func NewSafeAutoCertManager(tls_email_address string, ssl_cache_dir string, hostnames_whitelist []string) *SafeTLSAutoCertManager

func (*SafeTLSAutoCertManager) GetSecureTLSConfig

func (m *SafeTLSAutoCertManager) GetSecureTLSConfig() *tls.Config

SecureTLSConfig creates a new secure TLS config suitable for net/http.Server servers, supporting HTTP/2 and the tls-alpn-01 ACME challenge type.

type ShouldBase53IDBePlacedIntoSliceFn added in v1.0.0

type ShouldBase53IDBePlacedIntoSliceFn func(string) bool

type URLMap added in v1.0.0

type URLMap interface {
	Put_New_Entry(string, string, int64) error
	NumItems() int
}

type ValidationResult

type ValidationResult struct {
	Success bool
	Message string
}

Directories

Path Synopsis
json_internals
Checks that JSON file does not contain any fields that are not in the struct
Checks that JSON file does not contain any fields that are not in the struct
This is a manual test for the log rotation deletion functionality You are supposed to run this file and then run `watch -n0.1 ls -alrt /tmp/logfiletest/` in a separate terminal to see that it's doing what it's supposed to do.
This is a manual test for the log rotation deletion functionality You are supposed to run this file and then run `watch -n0.1 ls -alrt /tmp/logfiletest/` in a separate terminal to see that it's doing what it's supposed to do.
Defends against directory traversal attacks
Defends against directory traversal attacks

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL