skimmer

package module
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2023 License: MIT Imports: 19 Imported by: 0

README

skimmer

skimmer is a lightweight feed reader inspired by newsboat and yarnc. skimmer is very minimal and lacks features. That is skimmer's best feature. skimmer tries to do two things well.

  • fetch a list of URLs and download their items to an SQLite3 database
  • Display the contents of the SQLite3 database in reverse chronological order

That's it. No elaborate UI beyond what is easily accomplished using standard input, standard output and standard err.

skimmer needs to know what feeds to download and display. That is done by reading either a newsboat style url file or and OPML file. These are read and stored in an SQLite 3 database with the same base name of the file read but with a file extension of .skim. This allows you to easily maintain separate list of feeds to skim and potentially re-use the feed output.

Presently skimmer is focused on reading RSS 2, Atom and JSONfeeds. If this experiment evolves further than I hope to add support for txtxt as well as support for reading feeds from Gopher, Gemini and SFTP sites.

SYNOSIS

skimmer [OPTIONS] FILENAE [TIME_RANGE]

OPTIONS

-help : display a help page

-license : display license

-version : display version number and build hash

-fetch : Download items from the list of URLs

-limit N : Limit the display the N most recent items

-prune : The deletes items from the items table for the skimmer file provided. If a time range is provided then the items in the time range will be deleted. If a single time is provided everything older than that time is deleted. A time can be specified in several ways. An alias of "today" would remove all items older than today. If "now" is specified then all items older then the current time would be removed. Otherwise time can be specified as a date in YYYY-MM-DD format or timestamp YYYY-MM-DD HH:MM:SS format.

-i, -interactive : display an item and prompt for next action. e.g. (n)ext, (p)rev, (s)ave, (/)search, (d)elete

Examples

Fetch and read my newsboat feeds from .newsboat/urls. This will create a .newsboat/urls.skim.

skimmer .newsboat/urls

Fetch and read the feeds from my-news.opml. This will create a my-news.skim file.

skimmer my-news.opml

Get the latest items for the skimmer file "my-news.skim" Download some news to read later

skimmer -fetch my-news.skim

Read the last downloaded content from my-news.skim

Display the downloaded news

skimmer my-news.skim

Limit the number of items sent to the screen.

skimmer -display -limit 25 my-news.skim

Or my favorite is to run the output through Pandoc and page with less.

skimmer -display -limit 25 my-news.skim | \
    pandoc -f markdown -t plain | \
    less -R

Prune the items in the database older than today.

skimmer -prune my-news.skim today

Prune the items from the month of September 2023.

skimmer -prune my-news.skim \
    "2023-09-01 00:00:00" "2023-09-30 23:59:59"

Installation instructions

Installation From Source

Requirements

skimmer is an experimental. The precompiled binaries are not necessarily tested. To compile from source you need to have git, make, Pandoc SQLite3 and Go.

  • Git >= 2
  • Make >= 3.8 (GNU Make)
  • Pandoc > 3
  • SQLite3 > 3.4
  • Go >= 1.21.1
Steps to compile and install

Installation process I used to setup skimmer on a new machine.

git clone https://github.com/rsdoiel/skimmer
cd skimmer
make
make install

Acknowledgments

This experiment would not be possible with the authors of newsboat, SQLite3, Pandoc and the gofeed package.

Documentation

Index

Constants

View Source 
const (
	// Version number of release
	Version = "0.0.3"

	// ReleaseDate, the date version.go was generated
	ReleaseDate = "2023-10-09"

	// ReleaseHash, the Git hash when version.go was generated
	ReleaseHash = "bccbc71"

	LicenseText = `` /* 1025-byte string literal not displayed */

)

Variables

View Source 
var (
	// SQLCreateTables provides the statements that are use to create our tables
	// It has two percent s, first is feed list name, second is datetime scheme
	// was generated.
	SQLCreateTables = `` /* 612-byte string literal not displayed */

	// SQLResetChannels clear the channels talbe
	SQLResetChannels = `DELETE FROM channels;`

	// Update the channels in the skimmer file
	SQLUpdateChannel = `` /* 221-byte string literal not displayed */

	// Update a feed item in the items table
	SQLUpdateItem = `REPLACE INTO items (
link, title, description, updated, published, feedLabel)
VALUES (?, ?, ?, ?, ?, ?);`

	// Return link and title for Urls formatted output
	SQLChannelsAsUrls = `SELECT link, title FROM channels ORDER BY link;`

	// SQLItemCount returns a list of items in the items table
	SQLItemCount = `-- Count the items in the feed_items table.
SELECT COUNT(*) FROM items;`

	// SQLDisplayItems returns a list of items in decending chronological order.
	SQLDisplayItems = `` /* 233-byte string literal not displayed */

	SQLMarkItem = `UPDATE items SET status = ? WHERE link = ?;`

	SQLRelabelItem = `UPDATE items SET feedlabel = ? WHERE link = ?;`

	SQLTagItem = `UPDATE items SET tags = ? WHERE link = ?;`

	// SQLPruneItems will prune our items table for all items that have easier
	// a updated or publication date early than the timestamp provided.
	SQLPruneItems = `` /* 145-byte string literal not displayed */

)

Functions

func CheckWaitInterval 

func CheckWaitInterval(iTime time.Time, wait time.Duration) (time.Time, bool)

CheckWaitInterval checks to see if an interval of time has been met or exceeded. It returns the remaining time interval (possibly reset) and a boolean. The boolean is true when the time interval has been met or exceeded, false otherwise.

``` tot := len(something) // calculate the total number of items to process t0 := time.Now() iTime := time.Now() reportProgress := false 

for i, key := range records {
    // ... process stuff ...
    if iTime, reportProgress = CheckWaitInterval(rptTime, (30 * time.Second)); reportProgress {
        log.Printf("%s", ProgressETA(t0, i, tot))
    }
}

```

func ClearScreen added in v0.0.3 

func ClearScreen()

func FmtHelp 

func FmtHelp(src string, appName string, version string, releaseDate string, releaseHash string) string

FmtHelp lets you process a text block with simple curly brace markup.

func JSONMarshal added in v0.0.3 

func JSONMarshal(data interface{}) ([]byte, error)

JSONMarshal provides provide a custom json encoder to solve a an issue with HTML entities getting converted to UTF-8 code points by json.Marshal(), json.MarshalIndent().

func JSONMarshalIndent added in v0.0.3 

func JSONMarshalIndent(data interface{}, prefix string, indent string) ([]byte, error)

JSONMarshalIndent provides provide a custom json encoder to solve a an issue with HTML entities getting converted to UTF-8 code points by json.Marshal(), json.MarshalIndent().

func JSONUnmarshal added in v0.0.3 

func JSONUnmarshal(src []byte, data interface{}) error

JSONUnmarshal is a custom JSON decoder so we can treat numbers easier

func ParseURLList 

func ParseURLList(fName string, src []byte) (map[string]string, error)

ParseURLList takes a filename and byte slice source, parses the contents returning a map of urls to labels and an error value.

func ProgressETA 

func ProgressETA(t0 time.Time, i int, tot int) string

ProgressETA returns a string with the percentage processed and estimated time remaining. It requires the a counter of records processed, the total count of records and a time zero value.

``` tot := len(something) // calculate the total number of items to process t0 := time.Now() iTime := time.Now() reportProgress := false 

for i, key := range records {
    // ... process stuff ...
    if iTime, reportProgress = CheckWaitInterval(rptTime, (30 * time.Second)); reportProgress {
        log.Printf("%s", ProgressETA(t0, i, tot))
    }
}

```

func ProgressIPS 

func ProgressIPS(t0 time.Time, i int, timeUnit time.Duration) string

ProgressIPS returns a string with the elapsed time and increments per second. Takes a time zero, a counter and time unit. Returns a string with count, running time and increments per time unit. ``` t0 := time.Now() iTime := time.Now() reportProgress := false 

for i, key := range records {
    // ... process stuff ...
    if iTime, reportProgress = CheckWaitInterval(rptTime, (30 * time.Second)); reportProgress || i = 0 {
        log.Printf("%s", ProgressIPS(t0, i, time.Second))
    }
}

```

func SetupScreen added in v0.0.3 

func SetupScreen(out io.Writer)

Types

type Skimmer 

type Skimmer struct {
	// AppName holds the name of the application
	AppName string `json:"app_name,omitempty"`

	// DbName holds the path to the SQLite3 database
	DBName string `json:"db_name,omitempty"`

	// Fetch indicates that items need to be retrieved from the url list and stored in the database
	Fetch bool `json:"fetch,omitempty"`

	// Urls are the map of urls to labels to be fetched or read
	Urls map[string]string `json:"urls,omitempty"`

	// Limit contrains the number of items shown
	Limit int `json:"limit,omitempty"`

	// Prune contains the date to use to prune the database.
	Prune bool `json:"prune,omitempty"`

	// Interactive if true causes Run to display one item at a time with a minimal of input
	Interactive bool `json:"interactive,omitempty"`

	// AsURLs, output the skimmer feeds as a newsboat style url file
	AsURLs bool `json:"urls,omitempty"`

	// AsOPML, output the skimmer feeds as OPML
	AsOPML bool `json:"opml,omittempty"`
	// contains filtered or unexported fields
}

Skimmer is the application structure that holds configuration and ties the app to the runner for the cli.

func NewSkimmer 

func NewSkimmer(out io.Writer, eout io.Writer, appName string) (*Skimmer, error)

func (*Skimmer) ChannelsToUrls added in v0.0.3 

func (app *Skimmer) ChannelsToUrls(db *sql.DB) ([]byte, error)

ChannelsToUrls converts the current channels table to Urls formated output and refreshes app.Urls data structure.

func (*Skimmer) Download 

func (app *Skimmer) Download(db *sql.DB) error

Download the contents from app.Urls

func (*Skimmer) ItemCount 

func (app *Skimmer) ItemCount(db *sql.DB) (int, error)

ItemCount returns the total number items in the database.

func (*Skimmer) MarkItem added in v0.0.3 

func (app *Skimmer) MarkItem(db *sql.DB, link string, val string) error

func (*Skimmer) PruneItems 

func (app *Skimmer) PruneItems(db *sql.DB, startDT time.Time, endDT time.Time) error

PruneItems takes a timestamp and performs a row delete on the table for items that are older than the timestamp.

func (*Skimmer) ReadUrls 

func (app *Skimmer) ReadUrls(fName string) error

ReadUrls reads urls or OPML file provided and updates the feeds in the skimmer skimmer file.

Newsboat's url file format is `<URL><SPACE>"~<LABEL>"` one entry per line The hash mark, "#" at the start of the line indicates a comment line.

OPML is documented at http://opml.org

func (*Skimmer) RelabelItem added in v0.0.3 

func (app *Skimmer) RelabelItem(db *sql.DB, link string, label string) error

func (*Skimmer) ResetChannels added in v0.0.3 

func (app *Skimmer) ResetChannels(db *sql.DB) error

func (*Skimmer) Run 

func (app *Skimmer) Run(in io.Reader, out io.Writer, eout io.Writer, args []string) error

Run provides the runner for skimmer. It allows for testing of much of the cli functionality

func (*Skimmer) RunInteractive added in v0.0.3 

func (app *Skimmer) RunInteractive(db *sql.DB) error

RunInteractive provides a sliver of interactive UI, basically displaying an item then prompting for an action.

func (*Skimmer) Setup 

func (app *Skimmer) Setup(fPath string) error

Setup checks to see if anything needs to be setup (or fixed) for skimmer to run.

func (*Skimmer) TagItem added in v0.0.3 

func (app *Skimmer) TagItem(db *sql.DB, link string, tag string) error

func (*Skimmer) Write 

func (app *Skimmer) Write(db *sql.DB) error

Display the contents from database

Source Files

View all Source files

Directories

Path Synopsis
cmd
skimmer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.