youtube

package
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 23, 2018 License: MIT Imports: 26 Imported by: 0

README

YouTube Feeds

The feed plugin is complicated because we use the user's uploads playlist, and that is sorted by actual upload date instead of publish date.

So say you have this channels videos:

  1. vid2 - uploaded 10pm - published 10pm
  2. vid1 - uploaded 6pm - published 6pm

Then a video was published (but uploaded a long time ago):

  1. vid2 - uploaded 10pm - published 10pm
  2. vid1 - uploaded 6pm - published 6pm
  3. vid3 - uploaded 5pm - published 11pm

vid3 was published after the latest vidoe but still appears at the bottom. this causes issues as we have no idea when to stop looking now. Currently YAGPDB handles this fine as long as it's not uploaded longer than 50 videos ago, in which case it may or may not catch it.

In the future I'll do a hybrid mode with search. Those super late published videos however will show up in discord super late. I cannot use search for 100% either because it costs 100 times for api quota to use, meaning it could be up to hours behind.

The feed plugin is complicated because we use the user's uploads playlist, and that is sorted by actual upload date instead of publish date

so say you have this channels videos:

  1. vid2 - uploaded 10pm - published 10pm
  2. vid1 - uploaded 6pm - published 6pm

Then a video was published (but uploaded a long time ago)

  1. vid2 - uploaded 10pm - published 10pm
  2. vid1 - uploaded 6pm - published 6pm
  3. vid3 - uploaded 5pm - published 11pm

vid3 was published after the latest vidoe but still appears at the bottom. this causes issues as we have no idea when to stop looking now. Currently YAGPDB handles this fine as long as its not uploaded longer than 50 videos ago, in which case it may or may not catch it.

In the future i'll do a hybrid mode with search, those super late published videos however will show up in discord super late, i cannot use search for 100% either because it costs 100 times for api quota to use, meaning it could be up to hours behind.

Storage layout:

Postgres tables: youtube_guild_subs - postgres - guild_id - channel_id - youtube_channel

youtube_playlist_ids - channel_name PRIMARY - playlist_id

Redis keys:

youtube_subbed_channels - sorted set

key is the channel name score is unix time in seconds when it was last checked

youtube_registered_websub_channels - sorted set

key is the channel name score is unix time in seconds when it expires

At the start of a poll, it uses zrange/zrevrange to grab an amount of entries to process and if they do get processed it updates the score to the current unix time.

youtube_last_video_time:{channel} - string

Holds the time of the last video in that channel we processed, all videos before this will be ignored.

youtube_last_video_id:{channel} - string

Holds the last video id for a channel, it will stop processing videos when it hits this video.

youtube_push_registrations - sorted set

Key is the channel id, value is the time it expires

youtube_currently_adding:{channelid} - set, set when this channel is being added

Documentation

Index

Constants

View Source 
const (
	MaxChannelsPerPoll  = 30
	PollInterval        = time.Second * 10
	WebSubCheckInterval = time.Second * 10
)
View Source 
const (
	RedisChannelsLockKey = "youtube_subbed_channel_lock"

	RedisKeyWebSubChannels = "youtube_registered_websub_channels"
	GoogleWebsubHub        = "https://pubsubhubbub.appspot.com/subscribe"
)
View Source 
const (
	GuildMaxFeeds        = 50
	GuildMaxFeedsPremium = 250
)

Variables

View Source 
var (
	ErrIDNotFound = errors.New("ID not found")
)
View Source 
var (
	ErrNoChannel = errors.New("No channel with that id found")
)
View Source 
var (
	WebSubVerifyToken = os.Getenv("YAGPDB_YOUTUBE_VERIFY_TOKEN")
)

Functions

func BaseEditHandler 

func BaseEditHandler(inner web.ControllerHandlerFunc) web.ControllerHandlerFunc

func KeyLastVidID 

func KeyLastVidID(channel string) string

func KeyLastVidTime 

func KeyLastVidTime(channel string) string

func MaxFeedsForContext added in v1.6.0 

func MaxFeedsForContext(ctx context.Context) int

func RegisterPlugin 

func RegisterPlugin()

Types

type ChannelSubscription 

type ChannelSubscription struct {
	common.SmallModel
	GuildID            string
	ChannelID          string
	YoutubeChannelID   string
	YoutubeChannelName string
	MentionEveryone    bool
}

func SubsForChannel 

func SubsForChannel(channel string) (result []*ChannelSubscription, err error)

func (*ChannelSubscription) TableName 

func (c *ChannelSubscription) TableName() string

type ContextKey 

type ContextKey int
const (
	ContextKeySub ContextKey = iota
)

type CtxKey 

type CtxKey int
const (
	CurrentConfig CtxKey = iota
)

type Form 

type Form struct {
	YoutubeChannelID   string
	YoutubeChannelUser string
	DiscordChannel     int64 `valid:"channel,false`
	ID                 uint
	MentionEveryone    bool
}
type Link struct {
	Href string `xml:"href,attr"`
	Rel  string `xml:"rel,attr"`
}

type LinkEntry 

type LinkEntry struct {
	Href string `xml:"href,attr"`
	Rel  string `xml:"rel,attr"`
}

type Plugin 

type Plugin struct {
	common.BasePlugin
	YTService *youtube.Service
	Stop      chan *sync.WaitGroup
}

func (*Plugin) AddFeed 

func (p *Plugin) AddFeed(guildID, discordChannelID int64, youtubeChannelID, youtubeUsername string, mentionEveryone bool) (*ChannelSubscription, error)

func (*Plugin) HandleEdit 

func (p *Plugin) HandleEdit(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)

func (*Plugin) HandleFeedUpdate 

func (p *Plugin) HandleFeedUpdate(w http.ResponseWriter, r *http.Request)

func (*Plugin) HandleMQueueError 

func (p *Plugin) HandleMQueueError(elem *mqueue.QueuedElementNoKallax, err error)

Remove feeds if they don't point to a proper channel

func (*Plugin) HandleNew 

func (p *Plugin) HandleNew(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)

func (*Plugin) HandleRemove 

func (p *Plugin) HandleRemove(w http.ResponseWriter, r *http.Request) (templateData web.TemplateData, err error)

func (*Plugin) HandleYoutube 

func (p *Plugin) HandleYoutube(w http.ResponseWriter, r *http.Request) (web.TemplateData, error)

func (*Plugin) InitWeb 

func (p *Plugin) InitWeb()

func (*Plugin) MaybeAddChannelWatch 

func (p *Plugin) MaybeAddChannelWatch(lock bool, channel string) error

maybeAddChannelWatch adds a channel watch to redis, if there wasn't one before

func (*Plugin) MaybeRemoveChannelWatch 

func (p *Plugin) MaybeRemoveChannelWatch(channel string)

maybeRemoveChannelWatch checks the channel for subs, if it has none then it removes it from the watchlist in redis.

func (*Plugin) Name 

func (p *Plugin) Name() string

func (*Plugin) PlaylistID 

func (p *Plugin) PlaylistID(channelID string) (string, error)

func (*Plugin) SetupClient 

func (p *Plugin) SetupClient() error

func (*Plugin) StartFeed 

func (p *Plugin) StartFeed()

func (*Plugin) Status 

func (p *Plugin) Status() (string, string)

func (*Plugin) StopFeed 

func (p *Plugin) StopFeed(wg *sync.WaitGroup)

func (*Plugin) ValidateSubscription 

func (p *Plugin) ValidateSubscription(w http.ResponseWriter, r *http.Request, query url.Values)

func (*Plugin) WebSubSubscribe 

func (p *Plugin) WebSubSubscribe(ytChannelID string) error

func (*Plugin) WebSubUnsubscribe 

func (p *Plugin) WebSubUnsubscribe(ytChannelID string) error

type XMLFeed 

type XMLFeed struct {
	Xmlns        string `xml:"xmlns,attr"`
	Link         []Link `xml:"link"`
	ChannelID    string `xml:"entry>channelId"`
	Published    string `xml:"entry>published"`
	VideoId      string `xml:"entry>videoId"`
	Yt           string `xml:"yt,attr"`
	LinkEntry    Link   `xml:"entry>link"`
	AuthorUri    string `xml:"entry>author>uri"`
	AuthorName   string `xml:"entry>author>name"`
	UpdatedEntry string `xml:"entry>updated"`
	Title        string `xml:"title"`
	TitleEntry   string `xml:"entry>title"`
	Id           string `xml:"entry>id"`
	Updated      string `xml:"updated"`
}

type YoutubePlaylistID 

type YoutubePlaylistID struct {
	ChannelID  string `gorm:"primary_key"`
	CreatedAt  time.Time
	PlaylistID string
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.