godestone

README

godestone

Go scraper for The Lodestone.

Installation

Just add the package to your go.mod or run go get github.com/xivapi/godestone/v2.

Also be sure to install a data provider service to initialize the scraper.

Data providers

Package	Size	Description
bingode		A binary-packed data provider. Large and requires updates each patch, but works without relying on any websites besides The Lodestone.

Usage

Refer to the examples/ folder for basic usage.

Contributing

Make sure to checkout the submodules if you are changing CSS selector information.

Dependencies

• go-bindata

Repacking

To repack the submodules, just run generate.sh.

Documentation

Index

• type AchievementInfo
• type ActiveMemberRange
• type AllAchievementInfo
• type CWLS
• type CWLSMember
• type CWLSOptions
  • func (s *CWLSOptions) BuildURI(lang string) string
• type CWLSSearchResult
• type Character
• type CharacterOptions
  • func (s *CharacterOptions) BuildURI(scraper *Scraper, lang string) string
• type CharacterSearchOrder
• type CharacterSearchResult
• type ClassJob
• type ClassJobBozja
• type ClassJobEureka
• type ClassJobUnlockedState
• type CrestLayers
• type Estate
• type FreeCompany
• type FreeCompanyActiveState
• type FreeCompanyFocus
• type FreeCompanyFocusInfo
• type FreeCompanyHousingStatus
• type FreeCompanyMember
• type FreeCompanyOptions
  • func (s *FreeCompanyOptions) BuildURI(lang string) string
• type FreeCompanyRanking
• type FreeCompanyRecruitingState
• type FreeCompanyReputation
• type FreeCompanySearchOrder
• type FreeCompanySearchResult
• type FreeCompanySeekingInfo
• type GearItem
• type GearItemBuild
• type GearSet
• type GrandCompanyInfo
• type IconedNamedEntity
• type Linkshell
• type LinkshellMember
• type LinkshellOptions
  • func (s *LinkshellOptions) BuildURI(lang string) string
• type LinkshellSearchOrder
• type LinkshellSearchResult
• type Minion
• type Mount
• type PVPTeam
• type PVPTeamMember
• type PVPTeamOptions
  • func (s *PVPTeamOptions) BuildURI(lang string) string
• type PVPTeamSearchOrder
• type PVPTeamSearchResult
• type PageInfo
• type Scraper
  • func NewScraper(dataProvider provider.DataProvider, lang SiteLang) *Scraper
  • func (s *Scraper) FetchCWLS(id string) (*CWLS, error)
  • func (s *Scraper) FetchCharacter(id uint32) (*Character, error)
  • func (s *Scraper) FetchCharacterAchievements(id uint32) ([]*AchievementInfo, *AllAchievementInfo, error)
  • func (s *Scraper) FetchCharacterMinions(id uint32) ([]*Minion, error)
  • func (s *Scraper) FetchCharacterMounts(id uint32) ([]*Mount, error)
  • func (s *Scraper) FetchFreeCompany(id string) (*FreeCompany, error)
  • func (s *Scraper) FetchFreeCompanyMembers(id string) chan *FreeCompanyMember
  • func (s *Scraper) FetchLinkshell(id string) (*Linkshell, error)
  • func (s *Scraper) FetchPVPTeam(id string) (*PVPTeam, error)
  • func (s *Scraper) SearchCWLS(opts CWLSOptions) chan *CWLSSearchResult
  • func (s *Scraper) SearchCharacters(opts CharacterOptions) chan *CharacterSearchResult
  • func (s *Scraper) SearchFreeCompanies(opts FreeCompanyOptions) chan *FreeCompanySearchResult
  • func (s *Scraper) SearchLinkshells(opts LinkshellOptions) chan *LinkshellSearchResult
  • func (s *Scraper) SearchPVPTeams(opts PVPTeamOptions) chan *PVPTeamSearchResult
• type SearchLang
• type SiteLang
• type Title

Types

type AchievementInfo

type AchievementInfo struct {
	*models.NamedEntity

	Date time.Time
}

AchievementInfo represents information about a character's achievements.

type ActiveMemberRange

type ActiveMemberRange string

ActiveMemberRange represents the active member range filter of a search.

const (
	OneToTen         ActiveMemberRange = "1-10"
	ElevenToThirty   ActiveMemberRange = "11-30"
	ThirtyOneToFifty ActiveMemberRange = "31-50"
	FiftyOnePlus     ActiveMemberRange = "51-"
)

Active member range for searches.

type AllAchievementInfo

type AllAchievementInfo struct {
	Private                bool
	TotalAchievements      uint32
	TotalAchievementPoints uint32
}

AllAchievementInfo represents information about a character's achievements in aggregate.

type CWLS

type CWLS struct {
	Name      string
	DC        string
	ID        string
	ParseDate time.Time
	Members   []*CWLSMember
}

CWLS represents basic CWLS information.

type CWLSMember

type CWLSMember struct {
	Avatar            string
	ID                uint32
	Name              string
	LinkshellRank     string
	LinkshellRankIcon string
	Rank              gcrank.GCRank
	RankIcon          string
	World             string
	DC                string
}

CWLSMember represents information about a CWLS member.

type CWLSOptions

type CWLSOptions struct {
	Name                      string
	DC                        string
	Order                     LinkshellSearchOrder
	ActiveMembers             ActiveMemberRange
	CommunityFinderRecruiting bool
}

CWLSOptions defines extra search information that can help to narrow down a CWLS search.

func (*CWLSOptions) BuildURI

func (s *CWLSOptions) BuildURI(lang string) string

BuildURI returns a constructed URI for the provided search options.

type CWLSSearchResult

type CWLSSearchResult struct {
	Error error `json:"-"`

	Name          string
	ID            string
	DC            string
	ActiveMembers uint32
}

CWLSSearchResult represents basic CWLS information returned from a search.

type Character

type Character struct {
	ActiveClassJob    *ClassJob
	Avatar            string
	Bio               string
	ClassJobs         []*ClassJob
	ClassJobBozjan    *ClassJobBozja
	ClassJobElemental *ClassJobEureka
	DC                string
	FreeCompanyID     string
	FreeCompanyName   string
	GearSet           *GearSet
	Gender            gender.Gender
	GrandCompanyInfo  *GrandCompanyInfo
	GuardianDeity     *IconedNamedEntity
	ID                uint32
	Name              string
	Nameday           string
	ParseDate         time.Time
	Portrait          string
	PvPTeamID         string
	Race              *models.GenderedEntity
	Title             *Title
	Town              *IconedNamedEntity
	Tribe             *models.GenderedEntity
	World             string
}

Character represents the information available about a character on The Lodestone.

type CharacterOptions

type CharacterOptions struct {
	Name         string
	World        string
	DC           string
	SearchLang   SearchLang
	GrandCompany string
	Race         string
	Tribe        string
	Order        CharacterSearchOrder
}

CharacterOptions defines extra search information that can help to narrow down a search.

func (*CharacterOptions) BuildURI

func (s *CharacterOptions) BuildURI(
	scraper *Scraper,
	lang string,
) string

BuildURI returns a constructed URI for the provided search options.

type CharacterSearchOrder

type CharacterSearchOrder uint8

CharacterSearchOrder represents the search result ordering of a Lodestone character search.

const (
	OrderCharaNameAToZ CharacterSearchOrder = iota + 1
	OrderCharaNameZToA
	OrderCharaWorldAtoZ
	OrderCharaWorldZtoA
	OrderCharaLevelDescending
	OrderCharaLevelAscending
)

Search ordering for character searches.

type CharacterSearchResult

type CharacterSearchResult struct {
	Error error `json:"-"`

	Avatar   string
	ID       uint32
	Lang     string
	Name     string
	Rank     gcrank.GCRank
	RankIcon string
	World    string
	DC       string
}

CharacterSearchResult contains data from the character search page about a character.

type ClassJob

type ClassJob struct {
	ClassID       uint8
	ExpLevel      uint32
	ExpLevelMax   uint32
	ExpLevelTogo  uint32
	IsSpecialized bool
	JobID         uint8
	Level         uint8
	Name          string
	UnlockedState ClassJobUnlockedState
}

ClassJob represents class and job information.

type ClassJobBozja

type ClassJobBozja struct {
	Level  uint8
	Mettle uint32

	Name string
	// contains filtered or unexported fields
}

ClassJobBozja represents character progression data in the Bozjan Southern Front.

type ClassJobEureka

type ClassJobEureka struct {
	ExpLevel     uint32
	ExpLevelMax  uint32
	ExpLevelTogo uint32
	Level        uint8
	Name         string
}

ClassJobEureka represents character progression data in Eureka.

type ClassJobUnlockedState

type ClassJobUnlockedState struct {
	ID   uint8
	Name string
}

ClassJobUnlockedState represents the unlock state of a ClassJob

type CrestLayers

type CrestLayers struct {
	Bottom string
	Middle string
	Top    string
}

CrestLayers represents the layers of a community crest.

type Estate

type Estate struct {
	Greeting string
	Name     string
	Plot     string
}

Estate represents a housing area estate.

type FreeCompany

type FreeCompany struct {
	Active            FreeCompanyActiveState
	ActiveMemberCount uint32
	CrestLayers       *CrestLayers
	DC                string
	Estate            *Estate
	Focus             []*FreeCompanyFocusInfo
	Formed            time.Time
	GrandCompany      *models.NamedEntity
	ID                string
	Name              string
	ParseDate         time.Time
	Rank              uint8
	Ranking           *FreeCompanyRanking
	Recruitment       FreeCompanyRecruitingState
	Reputation        []*FreeCompanyReputation
	Seeking           []*FreeCompanySeekingInfo
	Slogan            string
	Tag               string
	World             string
}

FreeCompany represents all of the basic information about an FC.

type FreeCompanyActiveState

type FreeCompanyActiveState string

FreeCompanyActiveState is the active state of an FC.

const (
	FCActiveNotSpecified FreeCompanyActiveState = "Not specified"
	FCActiveWeekdaysOnly FreeCompanyActiveState = "Weekdays Only"
	FCActiveWeekendsOnly FreeCompanyActiveState = "Weekends Only"
	FCActiveAlways       FreeCompanyActiveState = "Always"
)

Active state for an FC.

type FreeCompanyFocus

type FreeCompanyFocus string

FreeCompanyFocus is an FC's focus.

const (
	FCFocusRolePlaying FreeCompanyFocus = "Role-playing"
	FCFocusLeveling    FreeCompanyFocus = "Leveling"
	FCFocusCasual      FreeCompanyFocus = "Casual"
	FCFocusHardcore    FreeCompanyFocus = "Hardcore"
	FCFocusDungeons    FreeCompanyFocus = "Dungeons"
	FCFocusGuildhests  FreeCompanyFocus = "Guildhests"
	FCFocusTrials      FreeCompanyFocus = "Trials"
	FCFocusRaids       FreeCompanyFocus = "Raids"
	FCFocusPvP         FreeCompanyFocus = "PvP"
)

Free Company Focus.

type FreeCompanyFocusInfo

type FreeCompanyFocusInfo struct {
	Icon   string
	Kind   FreeCompanyFocus
	Status bool
}

FreeCompanyFocusInfo represents a particular FC's intentions for a focus.

type FreeCompanyHousingStatus

type FreeCompanyHousingStatus uint8

FreeCompanyHousingStatus represents the housing status of a Free Company for the purpose of searches.

const (
	FCHousingAll FreeCompanyHousingStatus = iota
	FCHousingNoEstateOrPlot
	FCHousingPlotOnly
	FCHousingEstateBuilt
)

Housing status for Free Company searches.

type FreeCompanyMember

type FreeCompanyMember struct {
	Error error `json:"-"`

	Avatar   string
	ID       uint32
	Name     string
	Rank     gcrank.GCRank
	FCRank   string
	RankIcon string
	World    string
	DC       string
}

FreeCompanyMember represents information about a Free Company member.

type FreeCompanyOptions

type FreeCompanyOptions struct {
	Name                      string
	World                     string
	DC                        string
	ActiveTime                FreeCompanyActiveState
	Recruitment               FreeCompanyRecruitingState
	Order                     FreeCompanySearchOrder
	HousingStatus             FreeCompanyHousingStatus
	ActiveMembers             ActiveMemberRange
	CommunityFinderRecruiting bool
}

FreeCompanyOptions defines extra search information that can help to narrow down a Free Company search.

func (*FreeCompanyOptions) BuildURI

func (s *FreeCompanyOptions) BuildURI(lang string) string

BuildURI returns a constructed URI for the provided search options.

type FreeCompanyRanking

type FreeCompanyRanking struct {
	Monthly uint32
	Weekly  uint32
}

FreeCompanyRanking represents a particular FC's periodic rankings.

type FreeCompanyRecruitingState

type FreeCompanyRecruitingState string

FreeCompanyRecruitingState is the recruiting state of an FC.

const (
	FCRecruitmentClosed FreeCompanyRecruitingState = "Closed"
	FCRecruitmentOpen   FreeCompanyRecruitingState = "Open"
)

Recruiting state for an FC.

type FreeCompanyReputation

type FreeCompanyReputation struct {
	GrandCompany *models.NamedEntity
	Progress     uint8
	Rank         *models.NamedEntity
}

FreeCompanyReputation represents an FC's alignment with each Grand Company.

type FreeCompanySearchOrder

type FreeCompanySearchOrder uint8

FreeCompanySearchOrder represents the search result ordering of a Lodestone Free Company search.

const (
	OrderFCNameAToZ FreeCompanySearchOrder = iota + 1
	OrderFCNameZToA
	OrderFCMembershipHighToLow
	OrderFCMembershipLowToHigh
	OrderFCDateFoundedDescending
	OrderFCDateFoundedAscending
)

Search ordering for Free Company searches.

type FreeCompanySearchResult

type FreeCompanySearchResult struct {
	Error error `json:"-"`

	Active        FreeCompanyActiveState
	ActiveMembers uint32
	CrestLayers   *CrestLayers
	DC            string
	Estate        string
	Formed        time.Time
	GrandCompany  *models.NamedEntity
	ID            string
	Name          string
	Recruitment   FreeCompanyRecruitingState
	World         string
}

FreeCompanySearchResult represents all of the searchable information about an FC.

type FreeCompanySeekingInfo

type FreeCompanySeekingInfo struct {
	Icon   string
	Kind   role.Role
	Status bool
}

FreeCompanySeekingInfo represents a particular FC's intentions for a recruit roles.

type GearItem

type GearItem struct {
	*models.NamedEntity

	Creator string
	Dye     uint32
	HQ      bool
	Materia []uint32
	Mirage  uint32
}

GearItem represents information about a single gear item on a character.

type GearItemBuild

type GearItemBuild struct {
	Body        *GearItem
	Bracelets   *GearItem
	Earrings    *GearItem
	Feet        *GearItem
	Hands       *GearItem
	Head        *GearItem
	Legs        *GearItem
	MainHand    *GearItem
	Necklace    *GearItem
	OffHand     *GearItem
	Ring1       *GearItem
	Ring2       *GearItem
	SoulCrystal *GearItem
	Waist       *GearItem
}

GearItemBuild represents a full gearset on a character. All gear items can be nil.

type GearSet

type GearSet struct {
	Attributes map[baseparam.BaseParam]uint32
	ClassID    uint8
	Gear       *GearItemBuild
	GearKey    string
	JobID      uint8
	Level      uint8
}

GearSet represents the current gear information of a character.

type GrandCompanyInfo

type GrandCompanyInfo struct {
	GrandCompany *models.NamedEntity
	RankID       gcrank.GCRank
}

GrandCompanyInfo represents Grand Company information about a character.

type IconedNamedEntity

type IconedNamedEntity struct {
	*models.NamedEntity

	Icon string
}

IconedNamedEntity represents an instance of an object with an icon and a name.

type Linkshell

type Linkshell struct {
	Name      string
	ID        string
	ParseDate time.Time
	Members   []*LinkshellMember
}

Linkshell represents basic linkshell information.

type LinkshellMember

type LinkshellMember struct {
	Avatar            string
	ID                uint32
	Name              string
	LinkshellRank     string
	LinkshellRankIcon string
	Rank              gcrank.GCRank
	RankIcon          string
	World             string
	DC                string
}

LinkshellMember represents information about a linkshell member.

type LinkshellOptions

type LinkshellOptions struct {
	Name                      string
	World                     string
	DC                        string
	Order                     LinkshellSearchOrder
	ActiveMembers             ActiveMemberRange
	CommunityFinderRecruiting bool
}

LinkshellOptions defines extra search information that can help to narrow down a linkshell search.

func (*LinkshellOptions) BuildURI

func (s *LinkshellOptions) BuildURI(lang string) string

BuildURI returns a constructed URI for the provided search options.

type LinkshellSearchOrder

type LinkshellSearchOrder uint8

LinkshellSearchOrder represents the search result ordering of a Lodestone CWLS search.

const (
	OrderLinkshellNameAToZ LinkshellSearchOrder = iota + 1
	OrderLinkshellNameZToA
	OrderLinkshellMembershipHighToLow
	OrderLinkshellMembershipLowToHigh
)

Search ordering for linkshell and CWLS searches.

type LinkshellSearchResult

type LinkshellSearchResult struct {
	Error error `json:"-"`

	Name          string
	ID            string
	World         string
	DC            string
	ActiveMembers uint32
}

LinkshellSearchResult represents basic linkshell information returned from a search.

type Minion

type Minion struct {
	*IconedNamedEntity
}

Minion represents a minion.

type Mount

type Mount struct {
	*IconedNamedEntity
}

Mount represents a mount.

type PVPTeam

type PVPTeam struct {
	Name        string
	ID          string
	DC          string
	ParseDate   time.Time
	Formed      time.Time
	CrestLayers *CrestLayers
	Members     []*PVPTeamMember
}

PVPTeam represents information about a PVP team.

type PVPTeamMember

type PVPTeamMember struct {
	Avatar   string
	ID       uint32
	Name     string
	Matches  uint32
	Rank     gcrank.GCRank
	RankIcon string
	World    string
	DC       string
}

PVPTeamMember represents information about a PVP team member.

type PVPTeamOptions

type PVPTeamOptions struct {
	Name                      string
	DC                        string
	Order                     PVPTeamSearchOrder
	CommunityFinderRecruiting bool
}

PVPTeamOptions defines extra search information that can help to narrow down a PVP team search.

func (*PVPTeamOptions) BuildURI

func (s *PVPTeamOptions) BuildURI(lang string) string

BuildURI returns a constructed URI for the provided search options.

type PVPTeamSearchOrder

type PVPTeamSearchOrder uint8

PVPTeamSearchOrder represents the search result ordering of a Lodestone CWLS search.

const (
	OrderPVPTeamNameAToZ PVPTeamSearchOrder = iota + 1
	OrderPVPTeamNameZToA
)

Search ordering for PVP Team searches.

type PVPTeamSearchResult

type PVPTeamSearchResult struct {
	Error error `json:"-"`

	Name        string
	ID          string
	DC          string
	CrestLayers *CrestLayers
}

PVPTeamSearchResult represents basic PVP team information returned from a search.

type PageInfo

type PageInfo struct {
	CurrentPage int
	TotalPages  int
}

PageInfo represents pagination information in a search.

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper is the object through which interactions with The Lodestone are made.

func NewScraper

func NewScraper(dataProvider provider.DataProvider, lang SiteLang) *Scraper

NewScraper creates a new instance of the Scraper. Do note that all five language-versions of the website are on the same physical servers in Japan. Changing the language of the website will not meaningfully improve response times.

func (*Scraper) FetchCWLS

func (s *Scraper) FetchCWLS(id string) (*CWLS, error)

FetchCWLS returns CWLS information for the provided CWLS ID. The error is returned if the request fails.

func (*Scraper) FetchCharacter

func (s *Scraper) FetchCharacter(id uint32) (*Character, error)

FetchCharacter returns character information for the provided Lodestone ID. This function makes two requests: one to the base character profile, and another to the class and job page, returning an error if either request fails.

func (*Scraper) FetchCharacterAchievements

func (s *Scraper) FetchCharacterAchievements(id uint32) ([]*AchievementInfo, *AllAchievementInfo, error)

FetchCharacterAchievements returns unlocked achievement information for the provided Lodestone ID. The error is returned if the request fails with anything other than a 403. A 403 will not be raised when the character's achievements are private. Instead, the Private field on the AllAchievementInfo object will be set to true.

func (*Scraper) FetchCharacterMinions

func (s *Scraper) FetchCharacterMinions(id uint32) ([]*Minion, error)

FetchCharacterMinions returns unlocked minion information for the provided Lodestone ID. The error is returned if the request fails with anything other than a 404. A 404 can result from a character not existing, but it can also result from a character not having any minions.

func (*Scraper) FetchCharacterMounts

func (s *Scraper) FetchCharacterMounts(id uint32) ([]*Mount, error)

FetchCharacterMounts returns unlocked mount information for the provided Lodestone ID. The error is returned if the request fails with anything other than a 404. A 404 can result from a character not existing, but it can also result from a character not having any mounts.

func (*Scraper) FetchFreeCompany

func (s *Scraper) FetchFreeCompany(id string) (*FreeCompany, error)

FetchFreeCompany returns Free Company information for the provided Free Company ID. The error is returned if the request fails.

func (*Scraper) FetchFreeCompanyMembers

func (s *Scraper) FetchFreeCompanyMembers(id string) chan *FreeCompanyMember

FetchFreeCompanyMembers returns Free Company member information for the provided Free Company ID. The error is returned if the request fails.

func (*Scraper) FetchLinkshell

func (s *Scraper) FetchLinkshell(id string) (*Linkshell, error)

FetchLinkshell returns linkshell information for the provided linkshell ID. The error is returned if the request fails.

func (*Scraper) FetchPVPTeam

func (s *Scraper) FetchPVPTeam(id string) (*PVPTeam, error)

FetchPVPTeam returns PVP team information for the provided PVP team ID. The error is returned if the request fails.

func (*Scraper) SearchCWLS

func (s *Scraper) SearchCWLS(opts CWLSOptions) chan *CWLSSearchResult

SearchCWLS returns a channel of searchable crossworld linkshells. Please note that searches are notoriously poor, and often return exact matches far down in the results, or else return no search results when search results should be present. This library does one retry on each failure, but this is not a guarantee that all search results will be returned.

func (*Scraper) SearchCharacters

func (s *Scraper) SearchCharacters(opts CharacterOptions) chan *CharacterSearchResult

SearchCharacters returns a channel of searchable characters. Please note that searches are notoriously poor, and often return exact matches far down in the results, or else return no search results when search results should be present. This library does one retry on each failure, but this is not a guarantee that all search results will be returned.

func (*Scraper) SearchFreeCompanies

func (s *Scraper) SearchFreeCompanies(opts FreeCompanyOptions) chan *FreeCompanySearchResult

SearchFreeCompanies returns a channel of searchable Free Companies. Please note that searches are notoriously poor, and often return exact matches far down in the results, or else return no search results when search results should be present. This library does one retry on each failure, but this is not a guarantee that all search results will be returned.

func (*Scraper) SearchLinkshells

func (s *Scraper) SearchLinkshells(opts LinkshellOptions) chan *LinkshellSearchResult

SearchLinkshells returns a channel of searchable linkshells. Please note that searches are notoriously poor, and often return exact matches far down in the results, or else return no search results when search results should be present. This library does one retry on each failure, but this is not a guarantee that all search results will be returned.

func (*Scraper) SearchPVPTeams

func (s *Scraper) SearchPVPTeams(opts PVPTeamOptions) chan *PVPTeamSearchResult

SearchPVPTeams returns a channel of searchable PVP teams. Please note that searches are notoriously poor, and often return exact matches far down in the results, or else return no search results when search results should be present. This library does one retry on each failure, but this is not a guarantee that all search results will be returned.

type SearchLang

type SearchLang uint8

SearchLang represents character language for the purposes of search.

const (
	NoneLang SearchLang = 1 << iota
	SearchJA
	SearchEN
	SearchDE
	SearchFR
)

Language

type SiteLang

type SiteLang string

SiteLang represents the scraped website language.

const (
	JA SiteLang = "jp"
	EN SiteLang = "na"
	FR SiteLang = "fr"
	DE SiteLang = "de"
)

Lodestone website language. Do note that all five language-versions of the website (`eu` not listed) are on the same physical servers in Japan. Changing the language of the website will not meaningfully improve response times.

type Title

type Title struct {
	*models.TitleInternal
}

Title represents a character title.