database

Documentation

Index

• type Database
  • func NewDatabase() *Database
  • func (db *Database) AddToQueue(url data.UrlQueue) error
  • func (db *Database) CanQueueUrl(url string) (bool, error)
  • func (db *Database) DeleteFromQueue(url data.UrlQueue) error
  • func (db *Database) GetCrawledPages(source string) ([]data.CrawledPage, error)
  • func (db *Database) GetQueue(source string) ([]data.UrlQueue, error)
  • func (db *Database) IndexProduct(product data.Product) error

Types

type Database

type Database struct {
	*mongo.Database
	AlgoliaIndex *search.Index
}

func NewDatabase

func NewDatabase() *Database

NewDatabase initializes a new instance of the Database struct.

Returns a pointer to the newly created Database.

func (*Database) AddToQueue

func (db *Database) AddToQueue(url data.UrlQueue) error

AddToQueue adds a URL to the queue in the Database.

It takes a parameter 'url' of type `data.UrlQueue` which represents the URL to be added.

func (*Database) CanQueueUrl

func (db *Database) CanQueueUrl(url string) (bool, error)

CanQueueUrl checks if a URL can be queued.

Parameters: - url: the URL to check.

Returns: - bool: true if the URL can be queued, false otherwise.

func (*Database) DeleteFromQueue

func (db *Database) DeleteFromQueue(url data.UrlQueue) error

DeleteFromQueue deletes a URL from the queue in the Database.

It takes a parameter `url` of type `data.UrlQueue`, which represents the URL to be deleted from the queue. This function does not return any value.

func (*Database) GetCrawledPages

func (db *Database) GetCrawledPages(source string) ([]data.CrawledPage, error)

GetCrawledPages retrieves crawled pages for a given source.

Parameters: - source: a string representing the source of the crawled pages. e.g. Jumia

Returns: - an array of data.CrawledPage representing the retrieved crawled pages.

func (*Database) GetQueue

func (db *Database) GetQueue(source string) ([]data.UrlQueue, error)

GetQueue retrieves a slice of data.UrlQueue from the Database. It randomly selects 10 URLs from the queue and returns them as a slice of data.UrlQueue.

func (*Database) IndexProduct

func (db *Database) IndexProduct(product data.Product) error

IndexProduct saves a product to the indexed_products collection in the database.

It takes a parameter `product` of type `data.Product`.