rf

README

rss-feeds-go

A go utility package for handling RSS feeds.

Features

• Fetch RSS feeds from URLs
  • RSS feeds
  • Atom feeds
  • Manually
  • Periodically
• Cache fetched RSS feeds locally
  • In memory
  • In SQLite3 file
• Summarize RSS feed's content with Google Gemini API
  • Save summarized contents locally
    • In memory
    • In SQLite3 file
  • Transfer summarized contents to somewhere
• Convert cached feeds as RSS XML

Usages / Samples

Usage:

package main

import (
  "log"

  rf "github.com/meinside/rss-feeds-go"
)

func main() {
  if client, err := rf.NewClientWithDB(
    "abcdefghijklmnopqrstuvwxyz0123456789", // google ai api key
    []string{ // rss feeds' urls
      "https://hnrss.org/newest?points=50",
    },
    "./database.sqlite3", // sqlite3 db filepath
  ); err == nil {
    // TODO: do something with `client`
  }
}

Other sample applications are in ./samples/ directory.

Documentation

Index

• Constants
• func Prettify(v any) string
• func StandardizeJSON(b []byte) ([]byte, error)
• type CachedItem
• type Client
  • func NewClient(googleAIAPIKey string, feedsURLs []string) *Client
  • func NewClientWithDB(googleAIAPIKey string, feedsURLs []string, dbFilepath string) (client *Client, err error)
  • func (c *Client) DeleteOldCachedItems()
  • func (c *Client) FetchFeeds(ignoreAlreadyCached bool) (feeds []gf.RssFeed, err error)
  • func (c *Client) ListCachedItems(includeItemsMarkedAsRead bool) []CachedItem
  • func (c *Client) MarkCachedItemsAsRead(items []CachedItem)
  • func (c *Client) PublishXML(title, link, description, author, email string, items []CachedItem) (bytes []byte, err error)
  • func (c *Client) SetDesiredLanguage(lang string)
  • func (c *Client) SetGoogleAIModel(model string)
  • func (c *Client) SetSummarizeIntervalSeconds(seconds int)
  • func (c *Client) SetVerbose(v bool)
  • func (c *Client) SummarizeAndCacheFeeds(feeds []gf.RssFeed, urlScrapper ...*ssg.Scrapper) (err error)
• type FeedsItemsCache

Constants

const (
	ErrorPrefixSummaryFailedWithError = `Summary failed with error`
)

Functions

func Prettify

func Prettify(v any) string

Prettify prettifies given thing in JSON format.

func StandardizeJSON

func StandardizeJSON(b []byte) ([]byte, error)

StandardizeJSON standardizes given JSON (JWCC) bytes.

Types

type CachedItem

type CachedItem struct {
	gorm.Model

	Title       string
	Link        string // url to the original article
	Comments    string // url to the community comments
	GUID        string `gorm:"uniqueIndex"`
	Author      string
	PublishDate string
	Description string

	Summary      string
	MarkedAsRead bool `gorm:"index"`
}

CachedItem is a struct for a cached item

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client struct

func NewClient

func NewClient(googleAIAPIKey string, feedsURLs []string) *Client

NewClient returns a new client with memory cache.

func NewClientWithDB

func NewClientWithDB(googleAIAPIKey string, feedsURLs []string, dbFilepath string) (client *Client, err error)

NewClientWithDB returns a new client with SQLite DB cache.

func (*Client) DeleteOldCachedItems

func (c *Client) DeleteOldCachedItems()

DeleteOldCachedItems deletes old cached items.

func (*Client) FetchFeeds

func (c *Client) FetchFeeds(ignoreAlreadyCached bool) (feeds []gf.RssFeed, err error)

FetchFeeds fetches feeds.

func (*Client) ListCachedItems

func (c *Client) ListCachedItems(includeItemsMarkedAsRead bool) []CachedItem

ListCachedItems lists cached items.

func (*Client) MarkCachedItemsAsRead

func (c *Client) MarkCachedItemsAsRead(items []CachedItem)

MarkCachedItemsAsRead marks given cached items as read.

func (*Client) PublishXML

func (c *Client) PublishXML(title, link, description, author, email string, items []CachedItem) (bytes []byte, err error)

PublishXML returns XML bytes of given cached items.

func (*Client) SetDesiredLanguage

func (c *Client) SetDesiredLanguage(lang string)

SetDesiredLanguage sets the client's desired language for summaries.

func (*Client) SetGoogleAIModel

func (c *Client) SetGoogleAIModel(model string)

SetGoogleAIModel sets the client's Google AI model.

func (*Client) SetSummarizeIntervalSeconds

func (c *Client) SetSummarizeIntervalSeconds(seconds int)

SetSummarizeIntervalSeconds sets the client's summarize interval seconds.

func (*Client) SetVerbose

func (c *Client) SetVerbose(v bool)

SetVerbose sets the client's verbose mode.

func (*Client) SummarizeAndCacheFeeds

func (c *Client) SummarizeAndCacheFeeds(feeds []gf.RssFeed, urlScrapper ...*ssg.Scrapper) (err error)

SummarizeAndCacheFeeds summarizes given feeds items and caches them.

If summary fails, the original content prepended with the error message will be cached.

If there was an error with quota (HTTP 429), it will return immediately. (remaining feed items can be retried later)

type FeedsItemsCache

type FeedsItemsCache interface {
	Exists(guid string) bool
	Save(item feeds.RssItem, summary string)
	Fetch(guid string) *CachedItem
	MarkAsRead(guid string)
	List(includeItemsMarkedAsRead bool) []CachedItem
	DeleteOlderThan1Month()

	SetVerbose(v bool)
}

FeedsItemsCache is an interface of feeds items' cache