RobotsTxtParser

README

robotstxt-parser

A Robots.Txt Parser available as Golang Module

Features

• Different parsing of Robots.txt (from Ua to UA and Sitemap separate)

Documentation

Index

• type KV
• type RobotsTxt
  • func ParseTxt(txt string) (RobotsTxt, error)
• type UserAgentRule

Types

type KV

type KV struct {
	Key   string
	Value string
}

type RobotsTxt

type RobotsTxt struct {
	UserAgentRules []UserAgentRule `json:"user-agent-rules"`
	Sitemaps       []string        `json:"sitemaps"`
}

func ParseTxt

func ParseTxt(txt string) (RobotsTxt, error)

type UserAgentRule

type UserAgentRule struct {
	UserAgents []string `json:"user-agents"`
	Allow      []string `json:"allow"`
	Disallow   []string `json:"disallow"`
}