RobotsTxtParser

README

robotstxt-parser

A Robots.Txt Parser available as Golang Module

Documentation

Index

• type KV
• type RobotsTxt
  • func ParseTxt(txt string) (RobotsTxt, error)
• type UserAgentRule

Types

type KV

type KV struct {
	Key   string
	Value string
}

type RobotsTxt

type RobotsTxt struct {
	UserAgentRules []UserAgentRule
	Sitemaps       []string
}

func ParseTxt

func ParseTxt(txt string) (RobotsTxt, error)

type UserAgentRule

type UserAgentRule struct {
	UserAgents []string
	Allow      []string
	Disallow   []string
}