wikibrief

README

wikibrief

Description

Package wikibrief is a golang package that provides utility functions for extracting information from Wikipedia dumps.

Installation

This package can be installed with the go get command:

go get github.com/negapedia/wikibrief

Documentation

API documentation can be found in the associated godoc reference

Documentation

Index

• Constants
• func FanOut(ctx context.Context, in <-chan EvolvingPage, n int) (out []<-chan EvolvingPage)
• func New(ctx context.Context, fail func(err error) error, tmpDir, lang string, ...) <-chan EvolvingPage
• type EvolvingPage
• type Revision

Constants

const AnonimousUserID uint32 = 0

AnonimousUserID is the UserID value assumed by revisions done by an anonimous user

Functions

func FanOut

func FanOut(ctx context.Context, in <-chan EvolvingPage, n int) (out []<-chan EvolvingPage)

FanOut creates n copies of the given EvolvingPage channel; if n < 1 panics.

func New

func New(ctx context.Context, fail func(err error) error, tmpDir, lang string, restrict bool) <-chan EvolvingPage

New digest the latest wikipedia dump of the specified language into the output channel. The revision channel of each page must be exhausted (or the context cancelled), doing otherwise may result in a deadlock. The ctx and fail together should behave in the same manner as if created with WithFail - https://godoc.org/github.com/ebonetti/ctxutils#WithFail The condition restrict restricts the digest to just one dump file, used for testing purposes.

Types

type EvolvingPage

type EvolvingPage struct {
	PageID          uint32
	Title, Abstract string
	TopicID         uint32
	Revisions       <-chan Revision
}

EvolvingPage represents a wikipedia page that is being edited. Revisions is closed when there are no more revisions. Revision channel must be exhausted (or the context cancelled), doing otherwise may result in a deadlock.

type Revision

type Revision struct {
	ID, UserID uint32
	IsBot      bool
	Text, SHA1 string
	IsRevert   uint32
	Timestamp  time.Time
}

Revision represents a revision of a page.