util

package module
v0.10.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 8, 2023 License: GPL-3.0 Imports: 27 Imported by: 0

README

util

The code in this project is designed to be reusable across many different Go projects. It's designed to be completely general and can be reused without any adaptation.

This library calls panic in a number of places so you may want to capture both stdout and stderr like this:

go run ./fileserver/main.go 2>stderr.log 1>stdout.log

To run all tests:

go test $(go list ./... | grep -v manual)

Documentation

Overview

Map with expiring entries For slightly better performance, replace map[string]string with map[int64]string. See https://www.komu.engineer/blogs/01/go-gc-maps Memory usage can be more than double what you actually store in it. Based on my own testing, storing 10 million 128 byte URLs will take around 3.6GB of RAM, so each 128 byte URL took around 360 bytes of RAM. Entries can only be inserted, they cannot be updated or deleted before they expire. Uses sync.Mutex to protect concurrent access. Adding, getting, and removing entries require obtaining the mutex first. TODO: Benchmark switching to use a RWMutex or a sync.Map for improved performance. I tested sync.Map, it apparently has no reserve feature? Bulk load is slow - 7.8 seconds. Heap-based implementation for performance and simplicity Benchmarks show that Remove_All_Expired takes 3 seconds to remove 10 million expired entries Benchmarks show that NewConcurrentExpiringMapFromSlice takes 3.5 seconds to load 10 million entries No requirement for entries to have same TTL duration No support for updating expiry time - though this functionality can be added later if necessary. Example use cases: 1. Expiring short URLs - short URL -> long URL map 2. Expiring pastebins - short URL -> file path map 3. Expiring tokens - token -> expiry time map Map will return an error for expired entries

Deletes the oldest files until the size of directory is back within limit

This is a horrible hack to work around Go's lack of built-in support for type-safe enums True type safe enums are not possible in Go due to its lack of sum types This is a "best effort" workaround that will prevent some kinds of bugs but is not perfect

We use byte instead of rune because this alphabet contains only printable ASCII characters.

Index

Constants

View Source
const BASE53_ALPHABET_SIZE = 53
View Source
const XATTR_1F604_FILESERVER_CAN_BE_SERVED = "user.1f604.fileserver.can_be_served"

Variables

This section is empty.

Functions

func Assert_error_equals

func Assert_error_equals(t *testing.T, err error, expected string, skip_level int)

func Assert_no_error

func Assert_no_error(t *testing.T, err error, skip_level int)

func Assert_result_equals_bool

func Assert_result_equals_bool(t *testing.T, actual bool, err error, expected bool, skip_level int)

func Assert_result_equals_bytes

func Assert_result_equals_bytes(t *testing.T, actual []byte, err error, expected string, skip_level int)

func Assert_result_equals_interface

func Assert_result_equals_interface(t *testing.T, actual interface{}, err error, expected interface{}, skip_level int)

func Assert_result_equals_string_slice

func Assert_result_equals_string_slice(t *testing.T, actual []string, err error, expected []string, skip_level int)

func Assert_result_equals_time

func Assert_result_equals_time(t *testing.T, actual time.Time, err error, expected time.Time, line_number int)

func BuildStruct

func BuildStruct[T any]() *T

func Check_err

func Check_err(err error)

func Check_no_other_instances_running

func Check_no_other_instances_running(socket_addr string)

func Copy_Slice_Into_150_Arr

func Copy_Slice_Into_150_Arr(slice []byte, arr [150]byte)

func Crypto_Randint

func Crypto_Randint(max int) (int, error)

This function works, I've manually tested it. Returns integers from 0 up to AND NOT INCLUDING max

func Crypto_Random_Choice

func Crypto_Random_Choice[T any](arr *[]T) (T, error)

This function works, I've manually tested it.

func Divmod

func Divmod(numerator, denominator int) (int, int)

func Get_file_size

func Get_file_size(f *os.File) int64

this function assumes file pointer is valid. We could probably make this more efficient by calculating the file size in-process instead of making syscall each time.

func Getxattr

func Getxattr(path string, name string, data []byte) (int, error)

func Int64_to_string

func Int64_to_string(num int64) string

func Power_Naive

func Power_Naive(a, b int) int

Naive algorithm, only suitable for small b.

func Power_Slow

func Power_Slow(a, b, m int) int

calculates a to the power of b mod m. If m is 0 then just returns a to the power of b. This function seems to create a memory leak, but it doesn't. Anyway, it's better to use custom power

func PrintMemUsage

func PrintMemUsage()

PrintMemUsage outputs the current, total and OS memory being used. As well as the number of garage collection cycles completed.

func ReplaceString

func ReplaceString(str string, replacement rune, index int) string

func Retryfunc

func Retryfunc(taskname string, dotask retrylib_task, expected_duration time.Duration, max_wait time.Duration)

func Retryproc

func Retryproc(procname string, expected_duration time.Duration, max_wait time.Duration)

func ReverseString

func ReverseString(s string) string

func Setxattr

func Setxattr(path string, name string, data []byte, flags int) error

func String_to_int64

func String_to_int64(s string) (int64, error)

Types

type BandwidthMonitor

type BandwidthMonitor struct {
	// contains filtered or unexported fields
}

func (*BandwidthMonitor) GetTotalAllBytes

func (bm *BandwidthMonitor) GetTotalAllBytes() int64

func (*BandwidthMonitor) GetTotalTXBytes

func (bm *BandwidthMonitor) GetTotalTXBytes() int64

func (*BandwidthMonitor) RunThread

func (bm *BandwidthMonitor) RunThread(time_interval_secs int)

type Base53ErrorChecksumMismatch

type Base53ErrorChecksumMismatch struct{}

func (Base53ErrorChecksumMismatch) Error

type Base53ErrorIllegalCharacter

type Base53ErrorIllegalCharacter struct{}

func (Base53ErrorIllegalCharacter) Error

type Base53ErrorIllegalPair

type Base53ErrorIllegalPair struct{}

func (Base53ErrorIllegalPair) Error

func (e Base53ErrorIllegalPair) Error() string

type Base53ErrorStrWithoutCsumTooLong

type Base53ErrorStrWithoutCsumTooLong struct{}

func (Base53ErrorStrWithoutCsumTooLong) Error

type Base53ErrorStrWithoutCsumTooShort

type Base53ErrorStrWithoutCsumTooShort struct{}

Custom error types

func (Base53ErrorStrWithoutCsumTooShort) Error

type Base53ID

type Base53ID interface {
	GetStrWithoutCsum() string
	GetCsum() byte
	GetCombinedString() string
	Length() int
}

See https://stackoverflow.com/questions/57993809/how-to-hide-the-default-type-constructor-in-golang

type Base53IDManager

type Base53IDManager struct {
	// contains filtered or unexported fields
}

func NewBase53IDManager

func NewBase53IDManager() *Base53IDManager

pregenerate means strings up to n characters will be pre-generated and stored in RandomBags for fast PopRandom and Push later.

func (*Base53IDManager) B53_generate_all_Base53IDs

func (b53m *Base53IDManager) B53_generate_all_Base53IDs(n int) ([]Base53ID, error)

Generate all IDs of length n

func (*Base53IDManager) B53_generate_all_Base53IDs_int64

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_all_Base53IDs_int64_optimized

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64_optimized(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_all_Base53IDs_int64_test

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64_test(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_next_Base53ID

func (b53m *Base53IDManager) B53_generate_next_Base53ID(old_id Base53ID) (Base53ID, error)

func (*Base53IDManager) B53_generate_random_Base53ID

func (b53m *Base53IDManager) B53_generate_random_Base53ID(n int) (Base53ID, error)

func (*Base53IDManager) Convert_uint64_to_Base53ID

func (b53m *Base53IDManager) Convert_uint64_to_Base53ID(bigendian_uint64 uint64, length int) (*_base53ID_impl, error)

func (*Base53IDManager) Convert_uint64_to_byte_array

func (b53m *Base53IDManager) Convert_uint64_to_byte_array(bigendian_uint64 uint64) []byte

func (*Base53IDManager) Convert_uint64_to_str

func (b53m *Base53IDManager) Convert_uint64_to_str(bigendian_uint64 uint64, length int) string

func (*Base53IDManager) NewBase53ID

func (b53m *Base53IDManager) NewBase53ID(str_without_csum string, csum byte, remap bool) (*_base53ID_impl, error)

Construction is validation.

type CEMItem

type CEMItem struct {
	Key              interface{}
	Value            interface{}
	Expiry_time_unix int64
}

type CPMItem

type CPMItem struct {
	Key   interface{}
	Value interface{}
}

type ConcurrentExpiringMap

type ConcurrentExpiringMap struct {
	// contains filtered or unexported fields
}

keys are strings

func NewConcurrentExpiringMapFromSlice

func NewConcurrentExpiringMapFromSlice(expiry_callback ExpiryCallback, kv_pairs []CEMItem) *ConcurrentExpiringMap

batched mode for fast loading from disk Takes around 3.5s to load 10 million items, 300ms for loading 1 million items

func NewEmptyConcurrentExpiringMap

func NewEmptyConcurrentExpiringMap(expiry_callback ExpiryCallback) *ConcurrentExpiringMap

func (*ConcurrentExpiringMap) Get_Entry

func (cem *ConcurrentExpiringMap) Get_Entry(key interface{}) (interface{}, error)

func (*ConcurrentExpiringMap) NumItems

func (cem *ConcurrentExpiringMap) NumItems() int

func (*ConcurrentExpiringMap) Put_New_Entry

func (cem *ConcurrentExpiringMap) Put_New_Entry(key interface{}, value interface{}, expiry_time int64) error

func (*ConcurrentExpiringMap) Remove_All_Expired

func (cem *ConcurrentExpiringMap) Remove_All_Expired(extra_keeparound_seconds int64)

keep links around for extra_keeparound_seconds just to tell people that the link has expired this function will remove 10 million entries in 3 seconds

type ConcurrentPermanentMap

type ConcurrentPermanentMap struct {
	// contains filtered or unexported fields
}

keys are strings

func NewConcurrentPermanentMapFromSlice

func NewConcurrentPermanentMapFromSlice(kv_pairs []CPMItem) *ConcurrentPermanentMap

func NewEmptyConcurrentPermanentMap

func NewEmptyConcurrentPermanentMap() *ConcurrentPermanentMap

func (*ConcurrentPermanentMap) Get_Entry

func (cpm *ConcurrentPermanentMap) Get_Entry(key interface{}) (interface{}, bool)

func (*ConcurrentPermanentMap) NumItems

func (cpm *ConcurrentPermanentMap) NumItems() int

func (*ConcurrentPermanentMap) Put_Entry

func (cpm *ConcurrentPermanentMap) Put_Entry(key interface{}, value interface{})

type CryptoRandomChoiceEmptySliceError

type CryptoRandomChoiceEmptySliceError struct{}

Custom error types

func (CryptoRandomChoiceEmptySliceError) Error

type ERROR

type ERROR struct {
	S string
}

func Error

func Error(s string) ERROR

type EXACT_MATCH_HANDLER_t

type EXACT_MATCH_HANDLER_t struct{}
var EXACT_MATCH_HANDLER EXACT_MATCH_HANDLER_t = EXACT_MATCH_HANDLER_t{}

type ExpiringHeapItem

type ExpiringHeapItem struct {
	// contains filtered or unexported fields
}

func (ExpiringHeapItem) String

func (p ExpiringHeapItem) String() string

type ExpiringHeapQueue

type ExpiringHeapQueue []*ExpiringHeapItem

============= All this stuff is just to implement the interface required by heap ===================

func (ExpiringHeapQueue) Len

func (pq ExpiringHeapQueue) Len() int

func (ExpiringHeapQueue) Less

func (pq ExpiringHeapQueue) Less(i, j int) bool

func (*ExpiringHeapQueue) Pop

func (pq *ExpiringHeapQueue) Pop() any

func (*ExpiringHeapQueue) Push

func (pq *ExpiringHeapQueue) Push(x any)

func (ExpiringHeapQueue) Swap

func (pq ExpiringHeapQueue) Swap(i, j int)

type ExpiringMapItem

type ExpiringMapItem struct {
	// contains filtered or unexported fields
}

func (ExpiringMapItem) String

func (p ExpiringMapItem) String() string

type ExpiryCallback

type ExpiryCallback func(interface{})

type FileEntry

type FileEntry struct {
	FilePath    string
	FileInfo    fs.FileInfo
	TimeCreated time.Time
}

type HandlerTypeEnum

type HandlerTypeEnum interface {
	// contains filtered or unexported methods
}

type KeyAlreadyExistsError

type KeyAlreadyExistsError struct{}

func (KeyAlreadyExistsError) Error

func (e KeyAlreadyExistsError) Error() string

type KeyExpiredError

type KeyExpiredError struct{}

func (KeyExpiredError) Error

func (e KeyExpiredError) Error() string

type LONGEST_PREFIX_HANDLER_t

type LONGEST_PREFIX_HANDLER_t struct{}
var LONGEST_PREFIX_HANDLER LONGEST_PREFIX_HANDLER_t = LONGEST_PREFIX_HANDLER_t{}

type LogFileDeleter

type LogFileDeleter struct {
	AbsoluteDirectoryPath   string
	CurrentLogFileName      string
	DirectorySizeLimitBytes int64
}

func NewLogFileDeleter

func NewLogFileDeleter(directory_path string, size_limit int64, log_file_name string) *LogFileDeleter

func (*LogFileDeleter) Delete_Excess_Files

func (lfd *LogFileDeleter) Delete_Excess_Files()

func (*LogFileDeleter) RunThread

func (lfd *LogFileDeleter) RunThread(time_interval_secs int)

type NewBase53IDParams

type NewBase53IDParams struct {
	Str_without_csum string
	Csum             byte
	Remap            bool
}

inspired by StripeIntentParams

type NonExistentKeyError

type NonExistentKeyError struct{}

func (NonExistentKeyError) Error

func (e NonExistentKeyError) Error() string

type RESULT

type RESULT struct {
	S interface{}
}

func Result

func Result(s interface{}) RESULT

type RandomBag64

type RandomBag64 struct {
	// contains filtered or unexported fields
}

func CreateRandomBagFromSlice

func CreateRandomBagFromSlice(items []uint64) *RandomBag64

The RandomBag steals the slice that you pass to it. You should not use the slice anywhere afterwards.

func (*RandomBag64) PopRandom

func (rb *RandomBag64) PopRandom() (uint64, error)

Removes from array and swaps last element into it

func (*RandomBag64) Push

func (rb *RandomBag64) Push(item uint64)

Push should always succeed

func (*RandomBag64) Size

func (rb *RandomBag64) Size() int

type RandomBagEmptyError

type RandomBagEmptyError struct{}

func (RandomBagEmptyError) Error

func (e RandomBagEmptyError) Error() string

type SafeTLSAutoCertManager

type SafeTLSAutoCertManager struct {
	// contains filtered or unexported fields
}

func NewSafeAutoCertManager

func NewSafeAutoCertManager(tls_email_address string, ssl_cache_dir string, hostnames_whitelist []string) *SafeTLSAutoCertManager

func (*SafeTLSAutoCertManager) GetSecureTLSConfig

func (m *SafeTLSAutoCertManager) GetSecureTLSConfig() *tls.Config

SecureTLSConfig creates a new secure TLS config suitable for net/http.Server servers, supporting HTTP/2 and the tls-alpn-01 ACME challenge type.

type ValidationResult

type ValidationResult struct {
	Success bool
	Message string
}

Directories

Path Synopsis
json_internals
Checks that JSON file does not contain any fields that are not in the struct
Checks that JSON file does not contain any fields that are not in the struct
This is a manual test for the log rotation deletion functionality You are supposed to run this file and then run `watch -n0.1 ls -alrt /tmp/logfiletest/` in a separate terminal to see that it's doing what it's supposed to do.
This is a manual test for the log rotation deletion functionality You are supposed to run this file and then run `watch -n0.1 ls -alrt /tmp/logfiletest/` in a separate terminal to see that it's doing what it's supposed to do.
Defends against directory traversal attacks
Defends against directory traversal attacks

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL