util

package module
v0.10.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 8, 2023 License: GPL-3.0 Imports: 27 Imported by: 0

README

util

The code in this project is designed to be reusable across many different Go projects. It's designed to be completely general and can be reused without any adaptation.

This library calls panic in a number of places so you may want to capture both stdout and stderr like this:

go run ./fileserver/main.go 2>stderr.log 1>stdout.log

To run all tests:

go test $(go list ./... | grep -v manual)

Documentation

Overview

Map with expiring entries For slightly better performance, replace map[string]string with map[int64]string. See https://www.komu.engineer/blogs/01/go-gc-maps Memory usage can be more than double what you actually store in it. Based on my own testing, storing 10 million 128 byte URLs will take around 3.6GB of RAM, so each 128 byte URL took around 360 bytes of RAM. Entries can only be inserted, they cannot be updated or deleted before they expire. Uses sync.Mutex to protect concurrent access. Adding, getting, and removing entries require obtaining the mutex first. TODO: Benchmark switching to use a RWMutex or a sync.Map for improved performance. I tested sync.Map, it apparently has no reserve feature? Bulk load is slow - 7.8 seconds. Heap-based implementation for performance and simplicity Benchmarks show that Remove_All_Expired takes 3 seconds to remove 10 million expired entries Benchmarks show that NewConcurrentExpiringMapFromSlice takes 3.5 seconds to load 10 million entries No requirement for entries to have same TTL duration No support for updating expiry time - though this functionality can be added later if necessary. Example use cases: 1. Expiring short URLs - short URL -> long URL map 2. Expiring pastebins - short URL -> file path map 3. Expiring tokens - token -> expiry time map Map will return an error for expired entries

Deletes the oldest files until the size of directory is back within limit

This is a horrible hack to work around Go's lack of built-in support for type-safe enums True type safe enums are not possible in Go due to its lack of sum types This is a "best effort" workaround that will prevent some kinds of bugs but is not perfect

We use byte instead of rune because this alphabet contains only printable ASCII characters.

Index

Constants

View Source 
const BASE53_ALPHABET_SIZE = 53
View Source 
const XATTR_1F604_FILESERVER_CAN_BE_SERVED = "user.1f604.fileserver.can_be_served"

Variables

This section is empty.

Functions

func Assert_error_equals 

func Assert_error_equals(t *testing.T, err error, expected string, skip_level int)

func Assert_no_error 

func Assert_no_error(t *testing.T, err error, skip_level int)

func Assert_result_equals_bool 

func Assert_result_equals_bool(t *testing.T, actual bool, err error, expected bool, skip_level int)

func Assert_result_equals_bytes 

func Assert_result_equals_bytes(t *testing.T, actual []byte, err error, expected string, skip_level int)

func Assert_result_equals_interface 

func Assert_result_equals_interface(t *testing.T, actual interface{}, err error, expected interface{}, skip_level int)

func Assert_result_equals_string_slice 

func Assert_result_equals_string_slice(t *testing.T, actual []string, err error, expected []string, skip_level int)

func Assert_result_equals_time 

func Assert_result_equals_time(t *testing.T, actual time.Time, err error, expected time.Time, line_number int)

func BuildStruct 

func BuildStruct[T any]() *T

func CheckTCPPort 

func CheckTCPPort(port int) net.Listener

https://rosettacode.org/wiki/Determine_if_only_one_instance_is_running#Port

func Check_err 

func Check_err(err error)

func Check_no_other_instances_running 

func Check_no_other_instances_running(socket_addr string)

func Copy_Slice_Into_150_Arr 

func Copy_Slice_Into_150_Arr(slice []byte, arr [150]byte)

func Crypto_Randint 

func Crypto_Randint(max int) (int, error)

This function works, I've manually tested it. Returns integers from 0 up to AND NOT INCLUDING max

func Crypto_Random_Choice 

func Crypto_Random_Choice[T any](arr *[]T) (T, error)

This function works, I've manually tested it.

func Divmod 

func Divmod(numerator, denominator int) (int, int)

func Get_file_size 

func Get_file_size(f *os.File) int64

this function assumes file pointer is valid. We could probably make this more efficient by calculating the file size in-process instead of making syscall each time.

func Getxattr 

func Getxattr(path string, name string, data []byte) (int, error)

func Int64_to_string 

func Int64_to_string(num int64) string

func Power_Naive 

func Power_Naive(a, b int) int

Naive algorithm, only suitable for small b.

func Power_Slow 

func Power_Slow(a, b, m int) int

calculates a to the power of b mod m. If m is 0 then just returns a to the power of b. This function seems to create a memory leak, but it doesn't. Anyway, it's better to use custom power

func PrintMemUsage 

func PrintMemUsage()

PrintMemUsage outputs the current, total and OS memory being used. As well as the number of garage collection cycles completed.

func ReplaceString 

func ReplaceString(str string, replacement rune, index int) string

func Retryfunc 

func Retryfunc(taskname string, dotask retrylib_task, expected_duration time.Duration, max_wait time.Duration)

func Retryproc 

func Retryproc(procname string, expected_duration time.Duration, max_wait time.Duration)

func ReverseString 

func ReverseString(s string) string

func SetLogFile 

func SetLogFile(filename string) *os.File

https://stackoverflow.com/questions/19965795/how-to-write-log-to-file

func Setxattr 

func Setxattr(path string, name string, data []byte, flags int) error

func String_to_int64 

func String_to_int64(s string) (int64, error)

func Use 

func Use(vals ...interface{})

https://stackoverflow.com/questions/21743841/how-to-avoid-annoying-error-declared-and-not-used

Types

type BandwidthMonitor 

type BandwidthMonitor struct {
	// contains filtered or unexported fields
}

func (*BandwidthMonitor) GetTotalAllBytes 

func (bm *BandwidthMonitor) GetTotalAllBytes() int64

func (*BandwidthMonitor) GetTotalTXBytes 

func (bm *BandwidthMonitor) GetTotalTXBytes() int64

func (*BandwidthMonitor) RunThread 

func (bm *BandwidthMonitor) RunThread(time_interval_secs int)

type Base53ErrorChecksumMismatch 

type Base53ErrorChecksumMismatch struct{}

func (Base53ErrorChecksumMismatch) Error 

func (e Base53ErrorChecksumMismatch) Error() string

type Base53ErrorIllegalCharacter 

type Base53ErrorIllegalCharacter struct{}

func (Base53ErrorIllegalCharacter) Error 

func (e Base53ErrorIllegalCharacter) Error() string

type Base53ErrorIllegalPair 

type Base53ErrorIllegalPair struct{}

func (Base53ErrorIllegalPair) Error 

func (e Base53ErrorIllegalPair) Error() string

type Base53ErrorStrWithoutCsumTooLong 

type Base53ErrorStrWithoutCsumTooLong struct{}

func (Base53ErrorStrWithoutCsumTooLong) Error 

func (e Base53ErrorStrWithoutCsumTooLong) Error() string

type Base53ErrorStrWithoutCsumTooShort 

type Base53ErrorStrWithoutCsumTooShort struct{}

Custom error types

func (Base53ErrorStrWithoutCsumTooShort) Error 

func (e Base53ErrorStrWithoutCsumTooShort) Error() string

type Base53ID 

type Base53ID interface {
	GetStrWithoutCsum() string
	GetCsum() byte
	GetCombinedString() string
	Length() int
}

See https://stackoverflow.com/questions/57993809/how-to-hide-the-default-type-constructor-in-golang

type Base53IDManager 

type Base53IDManager struct {
	// contains filtered or unexported fields
}

func NewBase53IDManager 

func NewBase53IDManager() *Base53IDManager

pregenerate means strings up to n characters will be pre-generated and stored in RandomBags for fast PopRandom and Push later.

func (*Base53IDManager) B53_generate_all_Base53IDs 

func (b53m *Base53IDManager) B53_generate_all_Base53IDs(n int) ([]Base53ID, error)

Generate all IDs of length n

func (*Base53IDManager) B53_generate_all_Base53IDs_int64 

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_all_Base53IDs_int64_optimized 

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64_optimized(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_all_Base53IDs_int64_test 

func (b53m *Base53IDManager) B53_generate_all_Base53IDs_int64_test(n int) ([]uint64, error)

func (*Base53IDManager) B53_generate_next_Base53ID 

func (b53m *Base53IDManager) B53_generate_next_Base53ID(old_id Base53ID) (Base53ID, error)

func (*Base53IDManager) B53_generate_random_Base53ID 

func (b53m *Base53IDManager) B53_generate_random_Base53ID(n int) (Base53ID, error)

func (*Base53IDManager) Convert_uint64_to_Base53ID 

func (b53m *Base53IDManager) Convert_uint64_to_Base53ID(bigendian_uint64 uint64, length int) (*_base53ID_impl, error)

func (*Base53IDManager) Convert_uint64_to_byte_array 

func (b53m *Base53IDManager) Convert_uint64_to_byte_array(bigendian_uint64 uint64) []byte

func (*Base53IDManager) Convert_uint64_to_str 

func (b53m *Base53IDManager) Convert_uint64_to_str(bigendian_uint64 uint64, length int) string

func (*Base53IDManager) NewBase53ID 

func (b53m *Base53IDManager) NewBase53ID(str_without_csum string, csum byte, remap bool) (*_base53ID_impl, error)

Construction is validation.

type CEMItem 

type CEMItem struct {
	Key              interface{}
	Value            interface{}
	Expiry_time_unix int64
}

type CPMItem 

type CPMItem struct {
	Key   interface{}
	Value interface{}
}

type ConcurrentExpiringMap 

type ConcurrentExpiringMap struct {
	// contains filtered or unexported fields
}

keys are strings

func NewConcurrentExpiringMapFromSlice 

func NewConcurrentExpiringMapFromSlice(expiry_callback ExpiryCallback, kv_pairs []CEMItem) *ConcurrentExpiringMap

batched mode for fast loading from disk Takes around 3.5s to load 10 million items, 300ms for loading 1 million items

func NewEmptyConcurrentExpiringMap 

func NewEmptyConcurrentExpiringMap(expiry_callback ExpiryCallback) *ConcurrentExpiringMap

func (*ConcurrentExpiringMap) Get_Entry 

func (cem *ConcurrentExpiringMap) Get_Entry(key interface{}) (interface{}, error)

func (*ConcurrentExpiringMap) NumItems 

func (cem *ConcurrentExpiringMap) NumItems() int

func (*ConcurrentExpiringMap) Put_New_Entry 

func (cem *ConcurrentExpiringMap) Put_New_Entry(key interface{}, value interface{}, expiry_time int64) error

func (*ConcurrentExpiringMap) Remove_All_Expired 

func (cem *ConcurrentExpiringMap) Remove_All_Expired(extra_keeparound_seconds int64)

keep links around for extra_keeparound_seconds just to tell people that the link has expired this function will remove 10 million entries in 3 seconds

type ConcurrentPermanentMap 

type ConcurrentPermanentMap struct {
	// contains filtered or unexported fields
}

keys are strings

func NewConcurrentPermanentMapFromSlice 

func NewConcurrentPermanentMapFromSlice(kv_pairs []CPMItem) *ConcurrentPermanentMap

func NewEmptyConcurrentPermanentMap 

func NewEmptyConcurrentPermanentMap() *ConcurrentPermanentMap

func (*ConcurrentPermanentMap) Get_Entry 

func (cpm *ConcurrentPermanentMap) Get_Entry(key interface{}) (interface{}, bool)

func (*ConcurrentPermanentMap) NumItems 

func (cpm *ConcurrentPermanentMap) NumItems() int

func (*ConcurrentPermanentMap) Put_Entry 

func (cpm *ConcurrentPermanentMap) Put_Entry(key interface{}, value interface{})

type CryptoRandomChoiceEmptySliceError 

type CryptoRandomChoiceEmptySliceError struct{}

Custom error types

func (CryptoRandomChoiceEmptySliceError) Error 

func (e CryptoRandomChoiceEmptySliceError) Error() string

type ERROR 

type ERROR struct {
	S string
}

func Error 

func Error(s string) ERROR

type EXACT_MATCH_HANDLER_t 

type EXACT_MATCH_HANDLER_t struct{}
var EXACT_MATCH_HANDLER EXACT_MATCH_HANDLER_t = EXACT_MATCH_HANDLER_t{}

type ExpiringHeapItem 

type ExpiringHeapItem struct {
	// contains filtered or unexported fields
}

func (ExpiringHeapItem) String 

func (p ExpiringHeapItem) String() string

type ExpiringHeapQueue 

type ExpiringHeapQueue []*ExpiringHeapItem

============= All this stuff is just to implement the interface required by heap ===================

func (ExpiringHeapQueue) Len 

func (pq ExpiringHeapQueue) Len() int

func (ExpiringHeapQueue) Less 

func (pq ExpiringHeapQueue) Less(i, j int) bool

func (*ExpiringHeapQueue) Pop 

func (pq *ExpiringHeapQueue) Pop() any

func (*ExpiringHeapQueue) Push 

func (pq *ExpiringHeapQueue) Push(x any)

func (ExpiringHeapQueue) Swap 

func (pq ExpiringHeapQueue) Swap(i, j int)

type ExpiringMapItem 

type ExpiringMapItem struct {
	// contains filtered or unexported fields
}

func (ExpiringMapItem) String 

func (p ExpiringMapItem) String() string

type ExpiryCallback 

type ExpiryCallback func(interface{})

type FileEntry 

type FileEntry struct {
	FilePath    string
	FileInfo    fs.FileInfo
	TimeCreated time.Time
}

type HandlerTypeEnum 

type HandlerTypeEnum interface {
	// contains filtered or unexported methods
}

type KeyAlreadyExistsError 

type KeyAlreadyExistsError struct{}

func (KeyAlreadyExistsError) Error 

func (e KeyAlreadyExistsError) Error() string

type KeyExpiredError 

type KeyExpiredError struct{}

func (KeyExpiredError) Error 

func (e KeyExpiredError) Error() string

type LONGEST_PREFIX_HANDLER_t 

type LONGEST_PREFIX_HANDLER_t struct{}
var LONGEST_PREFIX_HANDLER LONGEST_PREFIX_HANDLER_t = LONGEST_PREFIX_HANDLER_t{}

type LogFileDeleter 

type LogFileDeleter struct {
	AbsoluteDirectoryPath   string
	CurrentLogFileName      string
	DirectorySizeLimitBytes int64
}

func NewLogFileDeleter 

func NewLogFileDeleter(directory_path string, size_limit int64, log_file_name string) *LogFileDeleter

func (*LogFileDeleter) Delete_Excess_Files 

func (lfd *LogFileDeleter) Delete_Excess_Files()

func (*LogFileDeleter) RunThread 

func (lfd *LogFileDeleter) RunThread(time_interval_secs int)

type NewBase53IDParams 

type NewBase53IDParams struct {
	Str_without_csum string
	Csum             byte
	Remap            bool
}

inspired by StripeIntentParams

type NonExistentKeyError 

type NonExistentKeyError struct{}

func (NonExistentKeyError) Error 

func (e NonExistentKeyError) Error() string

type RESULT 

type RESULT struct {
	S interface{}
}

func Result 

func Result(s interface{}) RESULT

type RandomBag64 

type RandomBag64 struct {
	// contains filtered or unexported fields
}

func CreateRandomBagFromSlice 

func CreateRandomBagFromSlice(items []uint64) *RandomBag64

The RandomBag steals the slice that you pass to it. You should not use the slice anywhere afterwards.

func (*RandomBag64) PopRandom 

func (rb *RandomBag64) PopRandom() (uint64, error)

Removes from array and swaps last element into it

func (*RandomBag64) Push 

func (rb *RandomBag64) Push(item uint64)

Push should always succeed

func (*RandomBag64) Size 

func (rb *RandomBag64) Size() int

type RandomBagEmptyError 

type RandomBagEmptyError struct{}

func (RandomBagEmptyError) Error 

func (e RandomBagEmptyError) Error() string

type SafeTLSAutoCertManager 

type SafeTLSAutoCertManager struct {
	// contains filtered or unexported fields
}

func NewSafeAutoCertManager 

func NewSafeAutoCertManager(tls_email_address string, ssl_cache_dir string, hostnames_whitelist []string) *SafeTLSAutoCertManager

func (*SafeTLSAutoCertManager) GetSecureTLSConfig 

func (m *SafeTLSAutoCertManager) GetSecureTLSConfig() *tls.Config

SecureTLSConfig creates a new secure TLS config suitable for net/http.Server servers, supporting HTTP/2 and the tls-alpn-01 ACME challenge type.

type ValidationResult 

type ValidationResult struct {
	Success bool
	Message string
}

Source Files

View all Source files

Directories

Path Synopsis
json_internals
Checks that JSON file does not contain any fields that are not in the struct
 Checks that JSON file does not contain any fields that are not in the struct
logging_internals
This is a manual test for the log rotation deletion functionality You are supposed to run this file and then run `watch -n0.1 ls -alrt /tmp/logfiletest/` in a separate terminal to see that it's doing what it's supposed to do.
 This is a manual test for the log rotation deletion functionality You are supposed to run this file and then run `watch -n0.1 ls -alrt /tmp/logfiletest/` in a separate terminal to see that it's doing what it's supposed to do.
Defends against directory traversal attacks
 Defends against directory traversal attacks

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.