finder

Documentation

Overview

Package finder provides a `find`-like interface to file discovery.

Even with inotify, we need a find-based cleanup for two reasons:

1. If closed files exist in the directory when the program starts, there is no way to know whether they are open or not. So, if they are older than the max_file_age, we will assume that the files are closed and upload them.

2. There is a race condition in the notify library where it is possible to create a directory and then create a file in the directory before the recursive listener has been established. We work around this bug (and any other bugs) by having a "cleanup" job that unconditionally adds any files older than the max_file_age.

Index

• func FindForever(ctx context.Context, datatype string, directory filename.System, ...)

Functions

func FindForever

func FindForever(ctx context.Context, datatype string, directory filename.System, maxFileAge time.Duration, notificationChannel chan<- filename.System, times memoryless.Config)

FindForever repeatedly runs FindFiles until its context is canceled.

It randomizes the inter-`find` sleep time in an effort to avoid thundering herd problems after container restarts. We're not worried about overloading GCS, but without this we might end up running `find` for every experiment simultaneously forever, and this could periodically run the disk out of IOPs. We use the memoryless library to ensure that the inter-`find` time is the exponential distribution and that the time-distribution of `find` operations is therefore memoryless.