fetching

Documentation

Overview

Package fetching provides a HTTP Client to fetch syndication feeds from remote servers.

The Client performs HTTP conditional requests, and leverages the following HTTP headers; - ETag (response) / If-None-Match (request) - Last-Modified (response) / If-Modified-Since (request)

See: - https://www.rfc-editor.org/rfc/rfc9110 - HTTP Semantics - https://http.dev/conditional-requests - https://rednafi.com/misc/etag_and_http_caching/ - https://rachelbythebay.com/w/2022/03/07/get/ - https://rachelbythebay.com/w/2024/05/27/feed/ - https://stackoverflow.com/questions/824152/what-takes-precedence-the-etag-or-last-modified-http-header

Index

• Constants
• type Client
  • func NewClient(httpClient *http.Client, userAgent string) *Client
  • func (c *Client) Fetch(feedURL string, eTag string, lastModified time.Time) (FeedStatus, error)
• type FeedStatus

Constants

const (
	// The "ETag" field in a response provides the current entity tag for the
	// selected representation, as determined at the conclusion of handling the request.
	HeaderEntityTag string = "ETag"

	// The "If-None-Match" header field makes the request method conditional
	// on a recipient cache or origin server either not having any current representation
	// of the target resource, when the field value is "*", or having a selected
	// representation with an entity tag that does not match any of those listed
	// in the field value.
	HeaderIfNoneMatch string = "If-None-Match"

	// The "Last-Modified" header field in a response provides a timestamp indicating
	// the date and time at which the origin server believes the selected representation
	// was last modified, as determined at the conclusion of handling the request.
	HeaderLastModified string = "Last-Modified"

	// The "If-Modified-Since" header field makes a GET or HEAD request method
	// conditional on the selected representation's modification date being more recent
	// than the date provided in the field value. Transfer of the selected representation's
	// data is avoided if that data has not changed.
	HeaderIfModifiedSince string = "If-Modified-Since"
)

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

A Client performs outgoing HTTP requests to get remote feed data.

It leverages previously saved HTTP headers to perform HTTP conditional requests and benefit from remote server caching.

func NewClient

func NewClient(httpClient *http.Client, userAgent string) *Client

NewClient initializes and returns a Client.

func (*Client) Fetch

func (c *Client) Fetch(feedURL string, eTag string, lastModified time.Time) (FeedStatus, error)

Fetch performs a HTTP request to get feed data and parses it.

Adapted from gofeed.Parser.ParseURL with the following modifications: - User-Agent header - Use the value of the ETag header to set the If-None-Match header - Use the value of the Last-Modified header to set the If-Modified-Since header

type FeedStatus

type FeedStatus struct {
	StatusCode int

	ETag         string
	LastModified time.Time

	Hash uint64

	Feed *gofeed.Feed
}

FeedStatus represents the status of a remote feed after fetching.

The ETag and LastModified fields are populated from the headers sent by the remote server, and should be saved to leverage server caching in future requests.

If the remote server responded with `200 OK`, the Feed will be populated with data parsed from the response. We will also return the hash of the feed data, which can be used for an additional layer of caching (e.g. when remote feeds do not send ETag not Last-Modified headers).

If the remote server responded with '304 Not Modified', the Feed will be nil.