htmltable

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 27, 2022 License: MIT Imports: 9 Imported by: 6

README

HTML table data extractor for Go

GoDoc MIT license codecov build

htmltable enables structured data extraction from HTML tables and URLs and requires almost no external dependencies. Tested with Go 1.18.x and 1.19.x.

Installation

go get github.com/nfx/go-htmltable

Usage

You can retrieve a slice of header-annotated types using the NewSlice* contructors:

type Ticker struct {
    Symbol   string `header:"Symbol"`
    Security string `header:"Security"`
    CIK      string `header:"CIK"`
}

url := "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
out, _ := htmltable.NewSliceFromURL[Ticker](url)
fmt.Println(out[0].Symbol)
fmt.Println(out[0].Security)

// Output: 
// MMM
// 3M

An error would be thrown if there's no matching page with the specified columns:

page, _ := htmltable.NewFromURL("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies")
_, err := page.FindWithColumns("invalid", "column", "names")
fmt.Println(err)

// Output: 
// cannot find table with columns: invalid, column, names

And you can use more low-level API to work with extracted data:

page, _ := htmltable.NewFromString(`<body>
    <h1>foo</h2>
    <table>
        <tr><td>a</td><td>b</td></tr>
        <tr><td> 1 </td><td>2</td></tr>
        <tr><td>3  </td><td>4   </td></tr>
    </table>
    <h1>bar</h2>
    <table>
        <tr><th>b</th><th>c</th><th>d</th></tr>
        <tr><td>1</td><td>2</td><td>5</td></tr>
        <tr><td>3</td><td>4</td><td>6</td></tr>
    </table>
</body>`)

fmt.Printf("found %d tables\n", page.Len())
_ = page.Each2("c", "d", func(c, d string) error {
    fmt.Printf("c:%s d:%s\n", c, d)
    return nil
})

// Output: 
// found 2 tables
// c:2 d:5
// c:4 d:6

Complex tables with row and col spans are natively supported as well. You can annotate string, int, and bool fields. Any bool field value is true if it is equal in lowercase to one of yes, y, true, t.

Wikipedia, AMD AM4 chipsets

type AM4 struct {
    Model             string `header:"Model"`
    ReleaseDate       string `header:"Release date"`
    PCIeSupport       string `header:"PCIesupport[a]"`
    MultiGpuCrossFire bool   `header:"Multi-GPU CrossFire"`
    MultiGpuSLI       bool   `header:"Multi-GPU SLI"`
    USBSupport        string `header:"USBsupport[b]"`
    SATAPorts         int    `header:"Storage features SATAports"`
    RAID              string `header:"Storage features RAID"`
    AMDStoreMI        bool   `header:"Storage features AMD StoreMI"`
    Overclocking      string `header:"Processoroverclocking"`
    TDP               string `header:"TDP"`
    SupportExcavator  string `header:"CPU support[14] Excavator"`
    SupportZen        string `header:"CPU support[14] Zen"`
    SupportZenPlus    string `header:"CPU support[14] Zen+"`
    SupportZen2       string `header:"CPU support[14] Zen 2"`
    SupportZen3       string `header:"CPU support[14] Zen 3"`
    Architecture      string `header:"Architecture"`
}
am4Chipsets, _ := htmltable.NewSliceFromURL[AM4]("https://en.wikipedia.org/wiki/List_of_AMD_chipsets")
fmt.Println(am4Chipsets[2].Model)
fmt.Println(am4Chipsets[2].SupportZen2)

// Output:
// X370
// Varies[c]

And the last note: you're encouraged to plug your own structured logger:

htmltable.Logger = func(_ context.Context, msg string, fields ...any) {
    fmt.Printf("[INFO] %s %v\n", msg, fields)
}
htmltable.NewFromURL("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies")

// Output:
// [INFO] found table [columns [Symbol Security SEC filings GICSSector GICS Sub-Industry Headquarters Location Date first added CIK Founded] count 504]
// [INFO] found table [columns [Date Added Ticker Added Security Removed Ticker Removed Security Reason] count 308]

Inspiration

This library aims to be something like pandas.read_html or table_extract Rust crate, but more idiomatic for Go.

Documentation

Overview

htmltable enables structured data extraction from HTML tables and URLs

Index

Examples

Constants

This section is empty.

Variables

View Source 
var Logger func(_ context.Context, msg string, fields ...any)

Logger is a very simplistic structured logger, than should be overriden by integrations.

Functions

func NewSlice 

func NewSlice[T any](ctx context.Context, r io.Reader) ([]T, error)

NewSlice returns slice of annotated struct types from io.Reader

func NewSliceFromPage added in v0.4.0 

func NewSliceFromPage[T any](p *Page) ([]T, error)

NewSliceFromPage finds a table matching the slice and returns the slice

func NewSliceFromResponse 

func NewSliceFromResponse[T any](resp *http.Response) ([]T, error)

NewSliceFromString is same as NewSlice(context.Context, io.Reader), but takes just an http.Response

func NewSliceFromString 

func NewSliceFromString[T any](in string) ([]T, error)

NewSliceFromString is same as NewSlice(context.Context, io.Reader), but takes just a string.

func NewSliceFromURL 

func NewSliceFromURL[T any](url string) ([]T, error)

NewSliceFromString is same as NewSlice(context.Context, io.Reader), but takes just an URL.

Example (RowspansAndColspans) 
type AM4 struct {
	Model             string `header:"Model"`
	ReleaseDate       string `header:"Release date"`
	PCIeSupport       string `header:"PCIesupport[a]"`
	MultiGpuCrossFire bool   `header:"Multi-GPU CrossFire"`
	MultiGpuSLI       bool   `header:"Multi-GPU SLI"`
	USBSupport        string `header:"USBsupport[b]"`
	SATAPorts         int    `header:"Storage features SATAports"`
	RAID              string `header:"Storage features RAID"`
	AMDStoreMI        bool   `header:"Storage features AMD StoreMI"`
	Overclocking      string `header:"Processoroverclocking"`
	TDP               string `header:"TDP"`
	SupportExcavator  string `header:"CPU support Excavator"`
	SupportZen        string `header:"CPU support Zen"`
	SupportZenPlus    string `header:"CPU support Zen+"`
	SupportZen2       string `header:"CPU support Zen 2"`
	SupportZen3       string `header:"CPU support Zen 3"`
	Architecture      string `header:"Architecture"`
}
am4Chipsets, _ := htmltable.NewSliceFromURL[AM4]("https://en.wikipedia.org/wiki/List_of_AMD_chipsets")
fmt.Println(am4Chipsets[2].Model)
fmt.Println(am4Chipsets[2].SupportZen2)

Output:

X370
Varies[c]

Types

type Page added in v0.2.0 

type Page struct {
	Tables []*Table
	// contains filtered or unexported fields
}

Page is the container for all tables parseable

func New 

func New(ctx context.Context, r io.Reader) (*Page, error)

New returns an instance of the page with possibly more than one table

func NewFromResponse 

func NewFromResponse(resp *http.Response) (*Page, error)

NewFromResponse is same as New(ctx.Context, io.Reader), but from http.Response.

In case of failure, returns `ResponseError`, that could be further inspected.

func NewFromString 

func NewFromString(r string) (*Page, error)

NewFromString is same as New(ctx.Context, io.Reader), but from string

Example 
page, _ := htmltable.NewFromString(`<body>
		<h1>foo</h2>
		<table>
			<tr><td>a</td><td>b</td></tr>
			<tr><td> 1 </td><td>2</td></tr>
			<tr><td>3  </td><td>4   </td></tr>
		</table>
		<h1>bar</h2>
		<table>
			<tr><th>b</th><th>c</th><th>d</th></tr>
			<tr><td>1</td><td>2</td><td>5</td></tr>
			<tr><td>3</td><td>4</td><td>6</td></tr>
		</table>
	</body>`)

fmt.Printf("found %d tables\n", page.Len())
_ = page.Each2("c", "d", func(c, d string) error {
	fmt.Printf("c:%s d:%s\n", c, d)
	return nil
})

Output:

found 2 tables
c:2 d:5
c:4 d:6

func NewFromURL 

func NewFromURL(url string) (*Page, error)

NewFromURL is same as New(ctx.Context, io.Reader), but from URL.

In case of failure, returns `ResponseError`, that could be further inspected.

Example 
page, _ := htmltable.NewFromURL("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies")
_, err := page.FindWithColumns("invalid", "column", "names")
fmt.Println(err)

Output:

cannot find table with columns: invalid, column, names

func (*Page) Each added in v0.2.0 

func (p *Page) Each(a string, f func(a string) error) error

Each row would call func with the value of the table cell from the column specified in the first argument.

Returns an error if table has no matching column name.

func (*Page) Each2 added in v0.2.0 

func (p *Page) Each2(a, b string, f func(a, b string) error) error

Each2 will get two columns specified in the first two arguments and call the func with those values for every row in the table.

Returns an error if table has no matching column names.

func (*Page) Each3 added in v0.2.0 

func (p *Page) Each3(a, b, c string, f func(a, b, c string) error) error

Each3 will get three columns specified in the first three arguments and call the func with those values for every row in the table.

Returns an error if table has no matching column names.

func (*Page) FindWithColumns added in v0.2.0 

func (p *Page) FindWithColumns(columns ...string) (*Table, error)

FindWithColumns performs fuzzy matching of tables by given header column names

func (*Page) Len added in v0.2.0 

func (p *Page) Len() int

Len returns number of tables found on the page

type Table added in v0.2.0 

type Table struct {
	// Header holds names of headers
	Header []string

	// Rows holds slice of string slices
	Rows [][]string
}

Table is the low-level representation of raw header and rows.

Every cell string value is truncated of its whitespace.

func (*Table) String added in v0.2.0 

func (table *Table) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.