Documentation ¶
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SetInterval ¶
SetInterval change Time interval to be used in Index.get
Types ¶
type Index ¶
Index is a structure of <sitemapindex>
func ParseIndex ¶
ParseIndex create Index data from text
type Sitemap ¶
Sitemap is a structure of <sitemap>
func ForceGet ¶ added in v0.3.0
ForceGet is fetch and parse sitemap.xml/sitemapindex.xml. The difference with the Get function is that it ignores some errors.
Errors to Ignore:
・When sitemapindex.xml contains a sitemap.xml URL that cannot be retrieved. ・When sitemapindex.xml contains a sitemap.xml that is empty ・When sitemapindex.xml contains a sitemap.xml that has format problems.
Errors not to Ignore:
・When sitemap.xml/sitemapindex.xml could not retrieved. ・When sitemap.xml/sitemapindex.xml is empty. ・When sitemap.xml/sitemapindex.xml has format problems.
If you want **not** to ignore some errors, use the Get function.
func Get ¶
Get is fetch and parse sitemap.xml/sitemapindex.xml
If sitemap.xml or sitemapindex.xml has some problems, This function return error.
・When sitemap.xml/sitemapindex.xml could not retrieved. ・When sitemap.xml/sitemapindex.xml is empty. ・When sitemap.xml/sitemapindex.xml has format problems. ・When sitemapindex.xml contains a sitemap.xml URL that cannot be retrieved. ・When sitemapindex.xml contains a sitemap.xml that is empty ・When sitemapindex.xml contains a sitemap.xml that has format problems.
If you want to ignore these errors, use the ForceGet function.
Example ¶
smap, err := Get("https://issueoverflow.com/sitemap.xml", nil) if err != nil { fmt.Println(err) } for _, URL := range smap.URL { fmt.Println(URL.Loc) }
Output:
Example (ChangeFetch) ¶
SetFetch(func(URL string, options interface{}) ([]byte, error) { req, err := http.NewRequest("GET", URL, nil) if err != nil { return []byte{}, err } // Set User-Agent req.Header.Set("User-Agent", "MyBot") // Set timeout timeout := time.Duration(10 * time.Second) client := http.Client{ Timeout: timeout, } // Fetch data res, err := client.Do(req) if err != nil { return []byte{}, err } defer res.Body.Close() body, err := io.ReadAll(res.Body) if err != nil { return []byte{}, err } return body, err }) smap, err := Get("https://issueoverflow.com/sitemap.xml", nil) if err != nil { fmt.Println(err) } for _, URL := range smap.URL { fmt.Println(URL.Loc) }
Output: