cache

README

Go HTTP Cache Transport

This package implements http transport functionality compatible with that of the Go standard library http.RoundTripper interface but which saves responses to disk locally based on very basic (at the moment) caching logic.

The simplest way to use is by overriding the http.DefaultTransport.

package main

import (
	"fmt"
	"net/http"
	"time"

	"git.sr.ht/~mariusor/cache"
)

const uri = "https://example.com"

func main() {
	// Use std library http.DefaultTransport, and save cached requests to /tmp
	http.DefaultTransport = cache.Shared(http.DefaultTransport, cache.FS("/tmp"))

	r, err := http.Get(uri)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Get %s: %s\n", uri, r.Status)
	fmt.Printf("From cache: %t\n", cache.FromStorage(r))
	if age := cache.Age(r); age > 0 {
		fmt.Printf("Age: %s\n", age)
	}
}

Documentation

Index

• Variables
• func Age(r *http.Response) time.Duration
• func FS(path string) fs
• func FromStorage(r *http.Response) bool
• func Mem(limit uintptr) mem
• type HeaderCacheControl
  • func (c HeaderCacheControl) String() string
• type Storage
• type Strategy
• type Transport
  • func ForDuration(maxAge time.Duration, t http.RoundTripper, st Storage) Transport
  • func Private(t http.RoundTripper, st Storage) Transport
  • func Shared(t http.RoundTripper, st Storage) Transport
  • func (t Transport) RoundTrip(req *http.Request) (*http.Response, error)

Examples

• ForDuration
• Private
• Shared

Variables

var (
	DefaultTransport http.RoundTripper = Transport{Base: defaultTransport, /* contains filtered or unexported fields */}
)

DefaultTransport is an implementation of http.RoundTripper that can be assigned to http.DefaultTransport in order to be used by http.DefaultClient. It uses disk persistence with a default cache duration of 1 hour.

Functions

func Age

func Age(r *http.Response) time.Duration

Age is a convenience function that returns the age of the stored response, by either reading it form the "Age" header or from our "cached-at=epoch millis" custom "Cache-Control" directive.

func FS

func FS(path string) fs

FS returns a cache Storage that persists data to the disk under a path.

func FromStorage

func FromStorage(r *http.Response) bool

FromStorage is a convenience function that returns true if the http.Header contains a Cache-Control entry marked as coming from storage.

func Mem

func Mem(limit uintptr) mem

Mem returns an in memory cache Storage with a cache limit.

Types

type HeaderCacheControl

type HeaderCacheControl struct {
	// contains filtered or unexported fields
}

HeaderCacheControl is a structured type for unpacking a Cache-Control header from an HTTP Response.

func (HeaderCacheControl) String

func (c HeaderCacheControl) String() string

type Storage

type Storage interface {
	Store(res *http.Response) error
	Load(r *http.Request) (*http.Response, error)
}

Storage is an interface representing the ability to store and retrieve multiple instances of http.Request to persistent storage.

var (

	// DefaultStorage saves cached requests directly on disk under the current user's cache directory
	// joined with the package path.
	DefaultStorage Storage = fs(filepath.Join(cacheDir, pkgPath))
)

type Strategy

type Strategy interface {
	// ShouldStore returns if the current response should be cached.
	// It uses the logic from RFC9111 Section 3.
	//
	//  A cache MUST NOT store a response to a request unless:
	// * the request method is understood by the cache;
	// * the response status code is final (see Section 15 of [RFC9110]);
	// * if the response status code is 206 or 304, or the must-understand cache directive
	// (see Section 5.2.2.3) is present: the cache understands the response status code;
	// * the no-store cache directive is not present in the response (see Section 5.2.2.5);
	// * if the cache is shared: the private response directive is either not present or allows
	// a shared cache to store a modified response; see Section 5.2.2.7);
	// * if the cache is shared: the Authorization header field is not present in the request
	// (see Section 11.6.2 of [RFC9110]) or a response directive is present that explicitly allows shared
	// caching (see Section 3.5); and
	// * the response contains at least one of the following:
	//   - a public response directive (see Section 5.2.2.9);
	//   - a private response directive, if the cache is not shared (see Section 5.2.2.7);
	//   - an Expires header field (see Section 5.3);
	//   - a max-age response directive (see Section 5.2.2.1);
	//   - if the cache is shared: an s-maxage response directive (see Section 5.2.2.10);
	//   - a cache extension that allows it to be cached (see Section 5.2.3);
	//   - or a status code that is defined as heuristically cacheable (see Section 4.2.2).
	//
	// https://datatracker.ietf.org/doc/html/rfc9111#section-3
	ShouldStore(*http.Response) bool

	// ValidCachedResponse tells us if the cached response can be served to the caller.
	// It uses the logic detailed in RFC9111 Section 4.
	//
	//  When presented with a request, a cache MUST NOT reuse a stored response unless:
	// * the presented target URI (Section 7.1 of [RFC9110]) and that of the stored response match, and
	// * the request method associated with the stored response allows it to be used for the presented request, and
	// * request header fields nominated by the stored response (if any) match those presented (see Section 4.1), and
	// * the stored response does not contain the no-cache directive (Section 5.2.2.4), unless it is successfully
	// validated (Section 4.3), and
	// * the stored response is one of the following:
	//   - fresh (see Section 4.2), or
	//   - allowed to be served stale (see Section 4.2.4), or
	//   - successfully validated (see Section 4.3).
	//
	// https://datatracker.ietf.org/doc/html/rfc9111#section-4
	ValidCachedResponse(*http.Response, *http.Request) bool

	// ShouldRevalidate returns if the transport should check with the origin server to see
	// if the cached response remains valid for this request.
	// It uses the heuristics detailed in RFC9111 Section 4.3, detailed below.
	//
	// 4.3. Validation
	// When a cache has one or more stored responses for a requested URI, but cannot serve any of them
	// (e.g., because they are not fresh, or one cannot be chosen; see Section 4.1),
	// it can use the conditional request mechanism (Section 13 of [RFC9110]) in the forwarded request
	// to give the next inbound server an opportunity to choose a valid stored response to use,
	// updating the stored metadata in the process, or to replace the stored response(s) with a new response.
	// This process is known as "validating" or "revalidating" the stored response.
	//
	// https://datatracker.ietf.org/doc/html/rfc9111#section-4.3
	// 4.3.1. Sending a Validation Request
	//
	// When generating a conditional request for validation, a cache either starts with a request it is attempting
	// to satisfy or -- if it is initiating the request independently -- synthesizes a request using a stored
	// response by copying the method, target URI, and request header fields identified by the Vary header field
	// (Section 4.1).
	// It then updates that request with one or more precondition header fields.
	// These contain validator metadata sourced from a stored response(s) that has the same URI.
	// Typically, this will include only the stored response(s) that has the same cache key,
	// although a cache is allowed to validate a response that it cannot choose with the request header fields
	// it is sending (see Section 4.1). The precondition header fields are then compared by recipients
	// to determine whether any stored response is equivalent to a current representation of the resource.
	// One such validator is the timestamp given in a Last-Modified header field (Section 8.8.2 of [RFC9110]),
	// which can be used in an If-Modified-Since header field for response validation,
	// or in an If-Unmodified-Since or If-Range header field for representation selection
	// (i.e., the client is referring specifically to a previously obtained representation with that timestamp).
	// Another validator is the entity tag given in an ETag field (Section 8.8.3 of [RFC9110]).
	// One or more entity tags, indicating one or more stored responses, can be used in an
	//  If-None-Match header field for response validation, or in an If-Match or If-Range
	// header field for representation selection (i.e., the client is referring specifically
	// to one or more previously obtained representations with the listed entity tags).
	// When generating a conditional request for validation, a cache:
	//  * MUST send the relevant entity tags (using If-Match, If-None-Match, or If-Range)
	//   if the entity tags were provided in the stored response(s) being validated.
	//  * SHOULD send the Last-Modified value (using If-Modified-Since)
	//   if the request is not for a subrange, a single stored response is being validated,
	//   and that response contains a Last-Modified value.
	//  * MAY send the Last-Modified value (using If-Unmodified-Since or If-Range)
	//   if the request is for a subrange, a single stored response is being validated,
	//   and that response contains only a Last-Modified value (not an entity tag).
	// In most cases, both validators are generated in cache validation requests,
	// even when entity tags are clearly superior, to allow old intermediaries that
	// do not understand entity tag preconditions to respond appropriately.
	//
	// https://datatracker.ietf.org/doc/html/rfc9111#section-4.3.2
	ShouldRevalidate(*http.Request, *http.Response) bool
}

Strategy presents the expected logic our Transport relies on in order to decide what to do with requests and responses.

type Transport

type Transport struct {
	Base http.RoundTripper
	// contains filtered or unexported fields
}

Transport implements an HTTP cache Implementation that holds requests and responses for reusing in subsequent requests. It can be either a shared cache, a private cache, or an indiscriminate cache. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#http_cache

func ForDuration

func ForDuration(maxAge time.Duration, t http.RoundTripper, st Storage) Transport

ForDuration creates an in-discriminant cache that "always" caches responses for the passed duration.

This type of cache is not RFC compliant and does not appear in the specification. It does not use any of the standard mechanisms for determining stored response freshness or if responses should not be cached. The default policy is to cache everything for the specified max age.

Example

server := testServer(forDurationHeaders)
var uri = server.URL

storage := Mem(KB)
http.DefaultTransport = ForDuration(time.Minute, http.DefaultTransport, storage)

r1, _ := http.Get(uri)

// Cleanup the cache file, so we can run the example multiple times
defer storage.Clean(r1.Request)

fmt.Printf("Get: %s\n", r1.Status)
fmt.Printf("From cache: %t\n", FromStorage(r1))

time.Sleep(500 * time.Millisecond)
// This time we should load it from cache
r2, _ := http.Get(uri)

fmt.Printf("Get: %s\n", r2.Status)
fmt.Printf("From cache: %t\n", FromStorage(r2))
fmt.Printf("Age: %s\n", Age(r2).Round(time.Second))

Output:

Get: 200 OK
From cache: false
Get: 200 OK
From cache: true
Age: 1s

func Private

func Private(t http.RoundTripper, st Storage) Transport

Private creates a private cache.

A private cache is one that exists in the client. It is also called local cache or browser cache. It can store and reuse personalized content for a single user.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#private_cache

Example

server := testServer(privateHeaders)
uri := server.URL

storage := Mem(KB)
http.DefaultTransport = Private(http.DefaultTransport, storage)

r1, _ := http.Get(uri)

// Cleanup the cache file, so we can run the example multiple times
defer storage.Clean(r1.Request)

fmt.Printf("Get: %s\n", r1.Status)
fmt.Printf("From cache: %t\n", FromStorage(r1))

time.Sleep(500 * time.Millisecond)

// This time we should load it from cache
r2, _ := http.Get(uri)

fmt.Printf("Get: %s\n", r2.Status)
fmt.Printf("From cache: %t\n", FromStorage(r2))
fmt.Printf("Age: %s\n", Age(r2).Round(time.Second))

Output:

Get: 200 OK
From cache: false
Get: 200 OK
From cache: true
Age: 1s

func Shared

func Shared(t http.RoundTripper, st Storage) Transport

Shared creates a shared, managed cache. It wraps an existing http.RoundTripper instance and stores the cached responses in st Storage.

A shared cache is one that exists between the origin server and clients (e.g. Proxy, CDN). It stores a single response and reuses it with multiple users — so developers should avoid storing personalized contents to be cached in the shared cache.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#shared_cache

Example

server := testServer(sharedHeaders)
var uri = server.URL

storage := Mem(KB)
http.DefaultTransport = Shared(http.DefaultTransport, storage)

r1, _ := http.Get(uri)

// Cleanup the cache file, so we can run the example multiple times
defer storage.Clean(r1.Request)

fmt.Printf("Get: %s\n", r1.Status)
fmt.Printf("From cache: %t\n", FromStorage(r1))

time.Sleep(500 * time.Millisecond)
// This time we should load it from cache
r2, _ := http.Get(uri)

fmt.Printf("Get: %s\n", r2.Status)
fmt.Printf("From cache: %t\n", FromStorage(r2))
fmt.Printf("Age: %s\n", Age(r2).Round(time.Second))

Output:

Get: 200 OK
From cache: false
Get: 200 OK
From cache: true
Age: 1s

func (Transport) RoundTrip

func (t Transport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip implements the http.RoundTripper interface.