rf

README

rss-feeds-go

A go utility package for handling RSS feeds.

Features

• Fetch RSS feeds from URLs
  • RSS feeds
  • Atom feeds
  • Manually
  • Periodically
• Cache fetched RSS feeds locally
  • In memory
  • In SQLite3 file
• Summarize RSS feed's content with Google Gemini API
  • Save summarized contents locally
    • In memory
    • In SQLite3 file
  • Transfer summarized contents to somewhere
• Convert cached feeds as RSS XML

Usages / Samples

Usage:

package main

import (
  "log"

  rf "github.com/meinside/rss-feeds-go"
)

func main() {
  if client, err := rf.NewClientWithDB(
    "abcdefghijklmnopqrstuvwxyz0123456789", // google ai api key
    []string{ // rss feeds' urls
      "https://hnrss.org/newest?points=50",
    },
    "./database.sqlite3", // sqlite3 db filepath
  ); err == nil {
    // TODO: do something with `client`
  }
}

Other sample applications are in ./samples/ directory.

Documentation

Index

• func Prettify(v any) string
• func StandardizeJSON(b []byte) ([]byte, error)
• type CachedItem
• type Client
  • func NewClient(googleAIAPIKey string, feedsURLs []string) *Client
  • func NewClientWithDB(googleAIAPIKey string, feedsURLs []string, dbFilepath string) (client *Client, err error)
  • func (c *Client) DeleteOldCachedItems()
  • func (c *Client) FetchFeeds(ignoreAlreadyCached bool) (feeds []gf.RssFeed, err error)
  • func (c *Client) ListCachedItems(includeItemsMarkedAsRead bool) []CachedItem
  • func (c *Client) MarkCachedItemsAsRead(items []CachedItem)
  • func (c *Client) PublishXML(title, link, description, author, email string, items []CachedItem) (bytes []byte, err error)
  • func (c *Client) SetDesiredLanguage(lang string)
  • func (c *Client) SetGoogleAIModel(model string)
  • func (c *Client) SetSummarizeIntervalSeconds(seconds int)
  • func (c *Client) SetVerbose(v bool)
  • func (c *Client) SummarizeAndCacheFeeds(feeds []gf.RssFeed, urlScrapper ...*ssg.Scrapper) (err error)
• type FeedsItemsCache

Functions

func Prettify

func Prettify(v any) string

Prettify prettifies given thing in JSON format.

func StandardizeJSON

func StandardizeJSON(b []byte) ([]byte, error)

StandardizeJSON standardizes given JSON (JWCC) bytes.

Types

type CachedItem

type CachedItem struct {
	gorm.Model

	Title       string
	Link        string // url to the original article
	Comments    string // url to the community comments
	GUID        string `gorm:"uniqueIndex"`
	Author      string
	PublishDate string
	Description string

	Summary      string
	MarkedAsRead bool `gorm:"index"`
}

CachedItem is a struct for a cached item

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client struct

func NewClient

func NewClient(googleAIAPIKey string, feedsURLs []string) *Client

NewClient returns a new client with memory cache.

func NewClientWithDB

func NewClientWithDB(googleAIAPIKey string, feedsURLs []string, dbFilepath string) (client *Client, err error)

NewClientWithDB returns a new client with SQLite DB cache.

func (*Client) DeleteOldCachedItems

func (c *Client) DeleteOldCachedItems()

DeleteOldCachedItems deletes old cached items.

func (*Client) FetchFeeds

func (c *Client) FetchFeeds(ignoreAlreadyCached bool) (feeds []gf.RssFeed, err error)

FetchFeeds fetches feeds.

func (*Client) ListCachedItems

func (c *Client) ListCachedItems(includeItemsMarkedAsRead bool) []CachedItem

ListCachedItems lists cached items.

func (*Client) MarkCachedItemsAsRead

func (c *Client) MarkCachedItemsAsRead(items []CachedItem)

MarkCachedItemsAsRead marks given cached items as read.

func (*Client) PublishXML

func (c *Client) PublishXML(title, link, description, author, email string, items []CachedItem) (bytes []byte, err error)

PublishXML returns XML bytes of given cached items.

func (*Client) SetDesiredLanguage

func (c *Client) SetDesiredLanguage(lang string)

SetDesiredLanguage sets the client's desired language for summaries.

func (*Client) SetGoogleAIModel

func (c *Client) SetGoogleAIModel(model string)

SetGoogleAIModel sets the client's Google AI model.

func (*Client) SetSummarizeIntervalSeconds

func (c *Client) SetSummarizeIntervalSeconds(seconds int)

SetSummarizeIntervalSeconds sets the client's summarize interval seconds.

func (*Client) SetVerbose

func (c *Client) SetVerbose(v bool)

SetVerbose sets the client's verbose mode.

func (*Client) SummarizeAndCacheFeeds

func (c *Client) SummarizeAndCacheFeeds(feeds []gf.RssFeed, urlScrapper ...*ssg.Scrapper) (err error)

SummarizeAndCacheFeeds summarizes given feeds items and caches them.

type FeedsItemsCache

type FeedsItemsCache interface {
	Exists(guid string) bool
	Save(item feeds.RssItem, summary string)
	Fetch(guid string) *CachedItem
	MarkAsRead(guid string)
	List(includeItemsMarkedAsRead bool) []CachedItem
	DeleteOlderThan1Month()

	SetVerbose(v bool)
}

FeedsItemsCache is an interface of feeds items' cache