htmldate

package module
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2024 License: Apache-2.0 Imports: 18 Imported by: 3

README

Go-HtmlDate Go Reference

Go-HtmlDate is a Go package and command-line tool to extract the original and updated publication dates of web pages. This package is based on htmldate, a Python package by Adrien Barbaresi.

The structure of this package is arranged following the structure of original Python code. This way, both libraries should give similar performance and any improvements from the original can be ported easily.

Table of Contents

Features

  • Extracts original or updated publication date of web pages;
  • EXPERIMENTAL: Extracts original or updated publication time (and its timezone) as well;

Just like the original, Go-HtmlDate has two mode: fast and extensive. The differences are:

  • In fast mode, the HTML page is cleaned and precise patterns are targeted;
  • In extensive mode, Go-HtmlDate will also collects all potential dates and uses a disambiguation algorithm to determines the best one to use.

By default Go-HtmlDate will run in extensive mode. You can disabled it by setting SkipExtensiveSearch in options to true.

Status

This package is stable enough for use and up to date with the original htmldate v1.9.1 (commit 3e2a230). However, since time extraction is a brand new feature which doesn't exist in the original, use it with care. So far it works quite nicely on most news sites that I've tried, but it still needs more testing.

When time extraction is enabled, there are some behaviors that I'd like to note:

  • If time is not found or not specified in the web page, the time will be set into 00:00:00 (it will only returns the date).
  • If timezone is not found or not specified in the web page, the timezone will be set into time.UTC.

In future I hope we could improve the comparison script to check the accuracy for time extraction as well.

Usage as Go package

Run following command inside your Go project :

go get -u -v github.com/markusmobius/go-htmldate@master

Next, include it in your application :

import "github.com/markusmobius/go-htmldate"

Now you can use Trafilatura to extract date of a web page. For basic usage you can check the examples.

Usage as CLI Application

To use CLI, you need to build it from source. Make sure you use go >= 1.20 then run following commands :

go install github.com/markusmobius/go-htmldate/cmd/go-htmldate@master

Once installed, you can use it from your terminal:

$ go-htmldate -h
Extract publish date from a HTML file or url

Usage:
  go-htmldate [flags] [source]

Flags:
  -f, --format string       set custom date output format (default "2006-01-02")
  -h, --help                help for go-htmldate
      --ori                 extract original date instead of the the most recent one
      --skip-tls            skip X.509 (TLS) certificate verification
      --time                extract publish time as well
  -t, --timeout int         timeout for downloading web page in seconds (default 30)
  -u, --user-agent string   set custom user agent (default "Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0")
  -v, --verbose             enable log message

Performance

This library heavily uses regular expression for various purposes. Unfortunately, as commonly known, Go's regular expression is pretty slow. This is because:

  • The regex engine in other language usually implemented in C, while in Go it's implemented from scratch in Go language. As expected, C implementation is still faster than Go's.
  • Since Go is usually used for web service, its regex is designed to finish in time linear to the length of the input, which useful for protecting server from ReDoS attack. However, this comes with performance cost.

To solve this issue, we compile several important regexes into Go code using re2go. Thanks to this we are able to get a great speed without using cgo or external regex packages.

Comparison with Original

Here we compare the extraction performance between the fast and extensive mode. To reproduce this test, clone this repository then run:

go run scripts/comparison/*.go

For the test, we use 1,000 documents which taken from two sources:

  • 725 documents from BBAW collection by Adrien Barbaresi, Shiyang Chen, and Lukas Kozmus.
  • 275 documents from Data Culture Group from Northeastern University for additional worldwide news.

Here is the result when tested in my PC (Intel i7-8550U @ 4.000GHz, RAM 16 GB):

Package Precision Recall Accuracy F-Score Speed (s)
htmldate v1.9.1 fast 0.881 0.924 0.821 0.902 7.039
htmldate v1.9.1 extensive 0.868 0.993 0.863 0.926 11.507
go-htmldate fast 0.882 0.925 0.823 0.903 0.767
go-htmldate extensive 0.870 0.993 0.865 0.928 1.682

So, from the table above we can see that this port has a similar performance with the original htmldate but with better speed.

Additional Notes

Despite the impressive score above, there is a little caveat: the performance test is only used to measure the accuracy of publish date extraction, and not the modified date. This issue is occured in the original htmldate as well since we use the comparison data from there.

With that said, if you use this package for extracting the modified date, the performance might not be as good as the performance table above. However, it should be still good enough to use.

The weird thing is the default behavior for original htmldate is to extract the modified date instead of the original, so ideally the performance test is done for modified date as well. To be fair, collecting the modified date seems harder than collecting the original date though.

Acknowledgements

This package won't be exist without effort by Adrien Barbaresi, the author of the original Python package. He created htmldate as part of effort to build text databases for research. There are web pages for which neither the URL nor the server response provide a reliable way to find out when a document was published or modified. For more information:

@article{barbaresi-2020-htmldate,
  title = {{htmldate: A Python package to extract publication dates from web pages}},
  author = "Barbaresi, Adrien",
  journal = "Journal of Open Source Software",
  volume = 5,
  number = 51,
  pages = 2439,
  url = {https://doi.org/10.21105/joss.02439},
  publisher = {The Open Journal},
  year = 2020,
}

License

Like the original, go-htmldate is distributed under the Apache v2.0.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Options 

type Options struct {
	// ExtractTime specify if we want to extract publish time as well along the date. Still WIP.
	ExtractTime bool

	// UseOriginalDate specify whether to extract the original date (e.g. publication date) instead
	// of most recent one (e.g. last modified, updated time).
	UseOriginalDate bool

	// URL is the URL for the webpage.
	URL string

	// MinDate is the earliest acceptable date.
	MinDate time.Time

	// MaxDate is the latest acceptable date.
	MaxDate time.Time

	// EnableLog specify whether log should be enabled or not.
	EnableLog bool

	// SkipExtensiveSearch specify whether to skip pattern-based opportunistic text search or not
	// using the external `dateparser` library. Note: this extensive search might be quite slow,
	// so use as necessary.
	SkipExtensiveSearch bool

	// DeferUrlExtractor specify whether to use URL extractor only as backup to
	// prioritize full expressions.
	DeferUrlExtractor bool

	// DateParserConfig is configuration for the external `dateparser`. Only used extensive search
	// is enabled (`SkipExtensiveSearch=false`).
	DateParserConfig *dps.Configuration
}

Options is configuration for the extractor.

type Result 

type Result struct {
	// DateTime is the extracted date time.
	DateTime time.Time
	// HasTime specifies whether the result contains time or not.
	HasTime bool
	// HasTimezone specifies whether the result contains timezone or not.
	// Useful for differentiating UTC timezone or timezone not found.
	HasTimezone bool
	// SrcString is the source where the date and time extracted.
	SrcString string
}

Result is the result of date time extraction.

func FromDocument 

func FromDocument(doc *html.Node, opts Options) (Result, error)

FromDocument extract publish date from the specified html document.

func FromReader 

func FromReader(r io.Reader, opts Options) (Result, error)

FromReader extract publish date from the specified reader.

func (Result) Format 

func (r Result) Format(layout string) string

Format returns a textual representation of the time value formatted according to the specified layout. Wrapper for `Result.DateTime.Format`.

func (Result) IsZero 

func (r Result) IsZero() bool

IsZero reports whether the result is empty or not. Wrapper for `Result.DateTime.IsZero`.

Source Files

View all Source files

Directories

Path Synopsis
cmd
go-htmldate
internal
re2go
Code generated by re2c 3.1, DO NOT EDIT.
 Code generated by re2c 3.1, DO NOT EDIT.
selector
scripts
comparison

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.