ambler

README

ambler

Something I started awhile ago but forgot about
- IngCr3at1on 2017

License

Copyright 2017 Just Another Organization

Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Documentation

Index

• Variables
• func FilterDuplicates(in []string) (out []string)
• func FilterPrefix(in []string, filterBy []string) (out []string)
• func FilterURLs(in []string) (out []string)
• func FindLinks(r io.Reader) (links []string)
• func GetDocumentFromWeb(_url string, timeout time.Duration) (io.Reader, error)
• type Page

Variables

var (
	// FilterByHTTP recommended default filter set.
	FilterByHTTP = []string{"http://", "https://"}
)

Functions

func FilterDuplicates

func FilterDuplicates(in []string) (out []string)

FilterDuplicates takes a slice of strings and filters out any duplicates.

func FilterPrefix

func FilterPrefix(in []string, filterBy []string) (out []string)

FilterPrefix takes an in slice of strings and a slice of prefix strings to filter by, returning only the strings which have a filter prefix.

func FilterURLs

func FilterURLs(in []string) (out []string)

FilterURLs takes an in slice of strings returning only valid URL strings.

func FindLinks

func FindLinks(r io.Reader) (links []string)

FindLinks takes a reader for an HTML document and returns all the links in string slice.

func GetDocumentFromWeb

func GetDocumentFromWeb(_url string, timeout time.Duration) (io.Reader, error)

GetDocumentFromWeb fetches a document from the web uses a URL string, it returns the document in the form of an FIXME:

Types

type Page

type Page struct {
	Doc io.Reader
	URL string
}

Page is a reader containing a document and URL string of where the document is located.