importer

Documentation

Overview

Package importer imports content from third-party websites.

TODO(bradfitz): Finish this. Barely started.

Index

• Variables
• func Register(name string, fn Constructor)
• type Constructor
• type Host
  • func Create(name string, hl blobserver.Loader, baseURL string, cfg jsonconfig.Obj) (*Host, error)
  • func (h *Host) HTTPClient() *http.Client
  • func (h *Host) HTTPTransport() http.RoundTripper
  • func (h *Host) InitHandler(hl blobserver.FindHandlerByTyper) error
  • func (h *Host) NewObject() (*Object, error)
  • func (h *Host) ObjectFromRef(permanodeRef blob.Ref) (*Object, error)
  • func (h *Host) RootObject() (*Object, error)
  • func (h *Host) Search() *search.Handler
  • func (h *Host) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (h *Host) String() string
  • func (h *Host) Target() blobserver.StatReceiver
• type Importer
• type Interrupt
  • func (i Interrupt) ShouldStop() bool
• type Object
  • func (o *Object) Attr(attr string) string
  • func (o *Object) Attrs(attr string) []string
  • func (o *Object) ChildPathObject(path string) (*Object, error)
  • func (o *Object) PermanodeRef() blob.Ref
  • func (o *Object) SetAttr(key, value string) error
  • func (o *Object) SetAttrs(keyval ...string) error
• type ProgressMessage

Variables

var ErrInterrupted = errors.New("import interrupted by request")

ErrInterrupted should be returned by importers when an Interrupt fires.

Functions

func Register

func Register(name string, fn Constructor)

Types

type Constructor

type Constructor func(jsonconfig.Obj, *Host) (Importer, error)

Constructor is the function type that importers must register at init time.

type Host

type Host struct {
	BaseURL string
	// contains filtered or unexported fields
}

A Host is the environment hosting an importer.

func Create

func Create(name string, hl blobserver.Loader, baseURL string, cfg jsonconfig.Obj) (*Host, error)

func (*Host) HTTPClient

func (h *Host) HTTPClient() *http.Client

HTTPClient returns the HTTP client to use.

func (*Host) HTTPTransport

func (h *Host) HTTPTransport() http.RoundTripper

HTTPTransport returns the HTTP transport to use.

func (*Host) InitHandler

func (h *Host) InitHandler(hl blobserver.FindHandlerByTyper) error

func (*Host) NewObject

func (h *Host) NewObject() (*Object, error)

NewObject creates a new permanode and returns its Object wrapper.

func (*Host) ObjectFromRef

func (h *Host) ObjectFromRef(permanodeRef blob.Ref) (*Object, error)

ObjectFromRef returns the object given by the named permanode

func (*Host) RootObject

func (h *Host) RootObject() (*Object, error)

RootObject returns the root permanode for this importer account.

func (*Host) Search

func (h *Host) Search() *search.Handler

func (*Host) ServeHTTP

func (h *Host) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*Host) String

func (h *Host) String() string

func (*Host) Target

func (h *Host) Target() blobserver.StatReceiver

type Importer

type Importer interface {
	// Run runs a full or increment import.
	Run(Interrupt) error

	// Prefix returns the unique prefix for this importer.
	// It should be of the form "serviceType:username".
	// Further colons are added to form the names of planned
	// permanodes.
	Prefix() string

	// CanHandleURL returns whether a URL (such as one a user is
	// viewing in their browser and dragged onto Camlistore) is a
	// form recognized by this importer.  If so, its full metadata
	// and full data (e.g. unscaled image) can be fetched, rather
	// than just fetching the HTML of the URL.
	//
	// TODO: implement and use this. For now importers can return
	// stub these and return false/errors. They're unused.
	CanHandleURL(url string) bool
	ImportURL(url string) error

	ServeHTTP(w http.ResponseWriter, r *http.Request)
}

An Importer imports from a third-party site.

type Interrupt

type Interrupt <-chan struct{}

An Interrupt is passed to importers for them to monitor requests to stop importing. The channel is closed as a signal to stop.

func (Interrupt) ShouldStop

func (i Interrupt) ShouldStop() bool

ShouldStop returns whether the interrupt has fired. If so, importers should return ErrInterrupted.

type Object

type Object struct {
	// contains filtered or unexported fields
}

An Object is wrapper around a permanode that the importer uses to synchronize.

func (*Object) Attr

func (o *Object) Attr(attr string) string

Attr returns the object's attribute value for the provided attr, or the empty string if unset. To distinguish between unset, an empty string, or multiple attribute values, use Attrs.

func (*Object) Attrs

func (o *Object) Attrs(attr string) []string

Attrs returns the attribute values for the provided attr.

func (*Object) ChildPathObject

func (o *Object) ChildPathObject(path string) (*Object, error)

ChildPathObject returns (creating if necessary) the child object from the permanode o, given by the "camliPath:xxxx" attribute, where xxx is the provided path.

func (*Object) PermanodeRef

func (o *Object) PermanodeRef() blob.Ref

PermanodeRef returns the permanode that this object wraps.

func (*Object) SetAttr

func (o *Object) SetAttr(key, value string) error

SetAttr sets the attribute key to value.

func (*Object) SetAttrs

func (o *Object) SetAttrs(keyval ...string) error

SetAttrs sets multiple attributes. The provided keyval should be an even number of alternating key/value pairs to set.

type ProgressMessage

type ProgressMessage struct {
	ItemsDone, ItemsTotal int
	BytesDone, BytesTotal int64
}