README
¶
YouTube Feeds
The feed plugin is complicated because we use the user's uploads
playlist, and that is sorted by actual upload date instead of publish date.
So say you have this channels videos:
- vid2 - uploaded 10pm - published 10pm
- vid1 - uploaded 6pm - published 6pm
Then a video was published (but uploaded a long time ago):
- vid2 - uploaded 10pm - published 10pm
- vid1 - uploaded 6pm - published 6pm
- vid3 - uploaded 5pm - published 11pm
vid3 was published after the latest vidoe but still appears at the bottom. this causes issues as we have no idea when to stop looking now. Currently YAGPDB handles this fine as long as it's not uploaded longer than 50 videos ago, in which case it may or may not catch it.
In the future I'll do a hybrid mode with search. Those super late published videos however will show up in discord super late. I cannot use search for 100% either because it costs 100 times for api quota to use, meaning it could be up to hours behind.
The feed plugin is complicated because we use the user's uploads
playlist, and that is sorted by actual upload date instead of publish date
so say you have this channels videos:
- vid2 - uploaded 10pm - published 10pm
- vid1 - uploaded 6pm - published 6pm
Then a video was published (but uploaded a long time ago)
- vid2 - uploaded 10pm - published 10pm
- vid1 - uploaded 6pm - published 6pm
- vid3 - uploaded 5pm - published 11pm
vid3 was published after the latest vidoe but still appears at the bottom. this causes issues as we have no idea when to stop looking now. Currently YAGPDB handles this fine as long as its not uploaded longer than 50 videos ago, in which case it may or may not catch it.
In the future i'll do a hybrid mode with search, those super late published videos however will show up in discord super late, i cannot use search for 100% either because it costs 100 times for api quota to use, meaning it could be up to hours behind.
Storage layout:
Postgres tables: youtube_guild_subs - postgres - guild_id - channel_id - youtube_channel
youtube_playlist_ids - channel_name PRIMARY - playlist_id
Redis keys:
youtube_subbed_channels
- sorted set
key is the channel name score is unix time in seconds when it was last checked
youtube_registered_websub_channels
- sorted set
key is the channel name score is unix time in seconds when it expires
At the start of a poll, it uses zrange/zrevrange to grab an amount of entries to process and if they do get processed it updates the score to the current unix time.
youtube_last_video_time:{channel}
- string
Holds the time of the last video in that channel we processed, all videos before this will be ignored.
youtube_last_video_id:{channel}
- string
Holds the last video id for a channel, it will stop processing videos when it hits this video.
youtube_push_registrations
- sorted set
Key is the channel id, value is the time it expires
youtube_currently_adding:{channelid}
- set, set when this channel is being added
Documentation
¶
Index ¶
- Constants
- Variables
- func BaseEditHandler(inner web.ControllerHandlerFunc) web.ControllerHandlerFunc
- func KeyLastVidID(channel string) string
- func KeyLastVidTime(channel string) string
- func MaxFeedsForContext(ctx context.Context) int
- func RegisterPlugin()
- type ChannelSubscription
- type ContextKey
- type CtxKey
- type Form
- type Link
- type LinkEntry
- type Plugin
- func (p *Plugin) AddFeed(guildID, discordChannelID int64, youtubeChannelID, youtubeUsername string, ...) (*ChannelSubscription, error)
- func (p *Plugin) HandleEdit(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)
- func (p *Plugin) HandleFeedUpdate(w http.ResponseWriter, r *http.Request)
- func (p *Plugin) HandleMQueueError(elem *mqueue.QueuedElement, err error)
- func (p *Plugin) HandleNew(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
- func (p *Plugin) HandleRemove(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)
- func (p *Plugin) HandleYoutube(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
- func (p *Plugin) InitWeb()
- func (p *Plugin) MaybeAddChannelWatch(lock bool, channel string) error
- func (p *Plugin) MaybeRemoveChannelWatch(channel string)
- func (p *Plugin) Name() string
- func (p *Plugin) PlaylistID(channelID string) (string, error)
- func (p *Plugin) SetupClient() error
- func (p *Plugin) StartFeed()
- func (p *Plugin) Status() (string, string)
- func (p *Plugin) StopFeed(wg *sync.WaitGroup)
- func (p *Plugin) ValidateSubscription(w http.ResponseWriter, r *http.Request, query url.Values)
- func (p *Plugin) WebSubSubscribe(ytChannelID string) error
- func (p *Plugin) WebSubUnsubscribe(ytChannelID string) error
- type XMLFeed
- type YoutubePlaylistID
Constants ¶
const ( MaxChannelsPerPoll = 30 PollInterval = time.Second * 10 WebSubCheckInterval = time.Second * 10 )
const ( RedisChannelsLockKey = "youtube_subbed_channel_lock" RedisKeyWebSubChannels = "youtube_registered_websub_channels" GoogleWebsubHub = "https://pubsubhubbub.appspot.com/subscribe" )
const ( GuildMaxFeeds = 50 GuildMaxFeedsPremium = 250 )
Variables ¶
var (
ErrIDNotFound = errors.New("ID not found")
)
var (
ErrNoChannel = errors.New("No channel with that id found")
)
var (
WebSubVerifyToken = os.Getenv("YAGPDB_YOUTUBE_VERIFY_TOKEN")
)
Functions ¶
func BaseEditHandler ¶
func BaseEditHandler(inner web.ControllerHandlerFunc) web.ControllerHandlerFunc
func KeyLastVidID ¶
func KeyLastVidTime ¶
func MaxFeedsForContext ¶ added in v1.6.0
func RegisterPlugin ¶
func RegisterPlugin()
Types ¶
type ChannelSubscription ¶
type ChannelSubscription struct { common.SmallModel GuildID string ChannelID string YoutubeChannelID string YoutubeChannelName string MentionEveryone bool }
func SubsForChannel ¶
func SubsForChannel(channel string) (result []*ChannelSubscription, err error)
func (*ChannelSubscription) TableName ¶
func (c *ChannelSubscription) TableName() string
type Plugin ¶
type Plugin struct { common.BasePlugin YTService *youtube.Service Stop chan *sync.WaitGroup }
func (*Plugin) HandleEdit ¶
func (p *Plugin) HandleEdit(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)
func (*Plugin) HandleFeedUpdate ¶
func (p *Plugin) HandleFeedUpdate(w http.ResponseWriter, r *http.Request)
func (*Plugin) HandleMQueueError ¶
func (p *Plugin) HandleMQueueError(elem *mqueue.QueuedElement, err error)
Remove feeds if they don't point to a proper channel
func (*Plugin) HandleNew ¶
func (p *Plugin) HandleNew(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
func (*Plugin) HandleRemove ¶
func (p *Plugin) HandleRemove(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)
func (*Plugin) HandleYoutube ¶
func (p *Plugin) HandleYoutube(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)
func (*Plugin) MaybeAddChannelWatch ¶
maybeAddChannelWatch adds a channel watch to redis, if there wasn't one before
func (*Plugin) MaybeRemoveChannelWatch ¶
maybeRemoveChannelWatch checks the channel for subs, if it has none then it removes it from the watchlist in redis.
func (*Plugin) SetupClient ¶
func (*Plugin) ValidateSubscription ¶
func (*Plugin) WebSubSubscribe ¶
func (*Plugin) WebSubUnsubscribe ¶
type XMLFeed ¶
type XMLFeed struct { Xmlns string `xml:"xmlns,attr"` Link []Link `xml:"link"` ChannelID string `xml:"entry>channelId"` Published string `xml:"entry>published"` VideoId string `xml:"entry>videoId"` Yt string `xml:"yt,attr"` LinkEntry Link `xml:"entry>link"` AuthorUri string `xml:"entry>author>uri"` AuthorName string `xml:"entry>author>name"` UpdatedEntry string `xml:"entry>updated"` Title string `xml:"title"` TitleEntry string `xml:"entry>title"` Id string `xml:"entry>id"` Updated string `xml:"updated"` }