database

Documentation

Index

• Variables
• type DB
  • func New(ctx context.Context, creds database.DBCredentials, logger *logrus.Logger) (*DB, error)
  • func (db *DB) CancelStuckScans(ctx context.Context) (int64, error)
  • func (db *DB) Collection(name string, opts ...*options.CollectionOptions) *mongo.Collection
  • func (db *DB) FindOneSkylink(ctx context.Context, filter interface{}, opts ...*options.FindOneOptions) *mongo.SingleResult
  • func (db *DB) Ping(ctx context.Context) error
  • func (db *DB) Skylink(ctx context.Context, hash crypto.Hash) (*Skylink, error)
  • func (db *DB) SkylinkByID(ctx context.Context, id primitive.ObjectID) (*Skylink, error)
  • func (db *DB) SkylinkCreate(ctx context.Context, skylink *Skylink) error
  • func (db *DB) SkylinkSave(ctx context.Context, skylink *Skylink) error
  • func (db *DB) SweepAndLock(ctx context.Context) (*Skylink, error)
  • func (db *DB) UpdateOneSkylink(ctx context.Context, filter interface{}, update interface{}, ...) (*mongo.UpdateResult, error)
• type Skylink
  • func (s *Skylink) LoadString(skylink, portal string) error

Variables

var (
	// ScanTimeout defines how long we want to wait for a scan to finish
	// before giving up on it and returning the skylink back into the "new"
	// bucket, so the scan can be retried. This prevents scans from hanging
	// forever in case the scanning server crashed or otherwise failed to
	// either finish the scan or report its findings.
	ScanTimeout = build.Select(
		build.Var{
			Dev:      time.Minute,
			Testing:  10 * time.Second,
			Standard: time.Hour,
		},
	).(time.Duration)

	// ErrNoDocumentsFound is returned when a database operation completes
	// successfully but it doesn't find or affect any documents.
	ErrNoDocumentsFound = errors.New("no documents found")
	// ErrSkylinkExists is returned when we try to add a skylink to the database
	// and it already exists there.
	ErrSkylinkExists = errors.New("skylink already exists")

	// True is a helper value, so we can pass a *bool to MongoDB's methods.
	True = true
)

var (
	// ErrInvalidSkylink is the error returned when the passed skylink is
	// invalid.
	ErrInvalidSkylink = errors.New("invalid skylink")

	// SkylinkStatusNew is the status of the skylink when it's created.
	SkylinkStatusNew = "new"
	// SkylinkStatusScanning is the status of the skylink while it's being
	// scanned.
	SkylinkStatusScanning = "scanning"
	// SkylinkStatusUnreported is the status of the skylink after it has been
	// found to be malicious but before it has been successfully reported to the
	// blocker service. We only use this status if we fail to talk to blocker.
	SkylinkStatusUnreported = "unreported"
	// SkylinkStatusComplete is the status of the skylink after it's scanned.
	SkylinkStatusComplete = "complete"
)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB holds a connection to the database, as well as helpful shortcuts to collections and utilities.

func New

func New(ctx context.Context, creds database.DBCredentials, logger *logrus.Logger) (*DB, error)

New creates a new database connection.

func (*DB) CancelStuckScans

func (db *DB) CancelStuckScans(ctx context.Context) (int64, error)

CancelStuckScans resets the status of scans that have been going on for more than scanner.ScanTimeout. We assume that these scans have terminated unexpectedly without reporting their results (e.g. server crash).

func (*DB) Collection

func (db *DB) Collection(name string, opts ...*options.CollectionOptions) *mongo.Collection

Collection gets a handle for a collection with the given name configured with the given CollectionOptions.

func (*DB) FindOneSkylink

func (db *DB) FindOneSkylink(ctx context.Context, filter interface{}, opts ...*options.FindOneOptions) *mongo.SingleResult

FindOneSkylink executes a find command on Skylinks collection and returns a SingleResult for one document in the collection.

func (*DB) Ping

func (db *DB) Ping(ctx context.Context) error

Ping sends a ping command to verify that the client can connect to the DB and specifically to the primary.

func (*DB) Skylink

func (db *DB) Skylink(ctx context.Context, hash crypto.Hash) (*Skylink, error)

Skylink fetches the DB record that corresponds to the given skylink from the database.

func (*DB) SkylinkByID

func (db *DB) SkylinkByID(ctx context.Context, id primitive.ObjectID) (*Skylink, error)

SkylinkByID fetches the DB record that corresponds to the given skylink by its DB ID.

func (*DB) SkylinkCreate

func (db *DB) SkylinkCreate(ctx context.Context, skylink *Skylink) error

SkylinkCreate creates a new skylink. If the skylink already exists it does nothing.

func (*DB) SkylinkSave

func (db *DB) SkylinkSave(ctx context.Context, skylink *Skylink) error

SkylinkSave saves the given Skylink record to the database.

func (*DB) SweepAndLock

func (db *DB) SweepAndLock(ctx context.Context) (*Skylink, error)

SweepAndLock sweeps the database for new skylinks. It "locks" and returns the first one it encounters. The "locking" is done by updating the skylink's status from "new" to "scanning".

func (*DB) UpdateOneSkylink

func (db *DB) UpdateOneSkylink(ctx context.Context, filter interface{}, update interface{}, opts ...*options.UpdateOptions) (*mongo.UpdateResult, error)

UpdateOneSkylink executes an update command on the Skylinks collection to update at most one document in the collection.

type Skylink

type Skylink struct {
	ID                   primitive.ObjectID `bson:"_id,omitempty" json:"-"`
	Hash                 crypto.Hash        `bson:"hash" json:"hash"`
	Skylink              string             `bson:"skylink" json:"skylink"`
	Status               string             `bson:"status" json:"status"`
	Infected             bool               `bson:"infected" json:"infected"`
	InfectionDescription string             `bson:"infection_description" json:"infectionDescription"`
	ScannedAllContent    bool               `bson:"scanned_all_content" json:"scannedAllContent"`
	ScannedAllOffsets    bool               `bson:"scanned_all_offsets" json:"scannedAllOffsets"`
	Size                 uint64             `bson:"size" json:"size"`
	Timestamp            time.Time          `bson:"timestamp" json:"timestamp"`
}

Skylink represents a skylink in the queue and holds its scanning status.

ClamAV typically limits the amount of data it scans, e.g. it would only scan the first 16MiB of data in a given file. ScannedAllContent marks if we've managed to scan all the content or just its beginning.

Multiple skylinks can point to the same merkle root but use different offsets on the data. Since we're blocking the entire merkle root, we should be scanning all possible (for the size of the data) offsets. ScannedAllOffsets marks if we have done that or not.

Timestamp marks the last status change that happened to the record. It can be the time when it was created, locked for scanning, or scanned.

func (*Skylink) LoadString

func (s *Skylink) LoadString(skylink, portal string) error

LoadString parses a skylink from string and populates all required fields.