smex

README

smex

A blazing fast CLI application that processes sitemaps in Go Lang.

Smex is short for SiteMap EXtrator. It can handle various operations related to sitemaps with more being build over time. Smex has grown from the frustration of migrating multiple websites in the past and handling changes in URL semantics and massive amount of assets like images.

Smex supports:

• extraction of urls
• process local/remote sitemaps
• output to csv/json
• pattern matching on urls
• url status checking
• extraction of images
• extraction of video
• extraction of news
• support for sitemaps with multiple languages

Note: smex is not a sitemap validator and would not check the validity of sitemaps.

Usages

Extract

To extract information from sitemap

The following command extracts yoast's post sitemaps and prints to stdout smex extract https://yoast.com/post-sitemap.xml --remote --loc

You can also point the extract at a sitemap locally smex extract ~/Download/sitemap.xml

Check

To check the status of the pages

TODO

Help

To get help simply run smex without any commands and flags.

Installation

Using Go

go get -u github.com/hbish/smex

Binary

TODO

Documentation

TODO - once published For package documentation please check on pkg.go.dev.

Contribute

Getting the code

Clone the repo

git clone git@github.com:hbish/smex.git

Initialise local environment

make init

Running the source

go run github.com/hbish/smex [command] [flags...]

Running tests

make test

Make the changes, the linter is set up to run when you commit your code, if it passes feel free to submit a PR!

Last Thing

Smex is my first stab at building with Go. If you have any feedback, comments or notice any bugs, I'd be more than happy to consider them through the github issue tracker or better yet send me patches!

Documentation

Overview

Copyright © 2020 Ben Shi

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.