smex

README

smex

A blazing fast CLI application that processes sitemaps in Go Lang.

---

Smex is short for SiteMap EXtrator. It can handle various operations related to sitemaps with more being build over time. Smex has grown from the frustration of migrating multiple websites in the past and handling changes in URL semantics and massive amount of assets like images.

Smex supports:

• extraction of urls
• process local/remote sitemaps
• output to csv/json
• pattern matching on urls
• extraction of images
• extraction of video
• extraction of news
• basic auth to retrieve sitemap
• url status checking
• support sitemap indexes
• support for sitemaps with multiple languages

Note: smex is not a sitemap validator and would not check the validity of sitemaps against the xsd. It will try to parse the sitemaps on best effort.

Usages

Extract

To extract information from sitemap

The following command extracts only the urls from yoast's post sitemaps and prints to stdout

smex extract https://yoast.com/post-sitemap.xml --remote --loc

You can also perform the extraction at a sitemap locally

smex extract ~/Download/sitemap.xml

By default, smex write to stdout. Due to the limited space only important fields are displayed. If you want a full extract of the sitemap, then add --format or -f flag to write to a file, currently smex only support csv & json.

smex extract https://yoast.com/post-sitemap.xml --remote --format csv

To filter the URLs you can supply a valid regex pattern using --pattern or -p flag

smex extract https://yoast.com/post-sitemap.xml --remote --pattern ".*seo.*" --format csv

The --output or -o flag will change the filename (no extension required), this is defaulted to smex-output.(csv|json)

Check

To check the status of the pages

TODO: this feature has not yet being implemented

Help

To get help simply run smex without any commands and flags.

Installation

Using Go

go get -u github.com/hbish/smex

Using Homebrew

brew install hbish/taps/smex

Binary

Currently cross-compiled for:

• Mac (64 bit)
• Linux (32/64 bit)
• Windows (32/64 bit)

Latest versions can be downloaded via Releases.

Documentation

For package documentation please check on pkg.go.dev.

Contribute

Getting the code

Clone the repo

git clone git@github.com:hbish/smex.git

Initialise local environment and install commit hook

make init

Running the source

go run github.com/hbish/smex [command] [flags...]

Running tests

make test

Make the changes, the linter is set up to run when you commit your code, if it passes feel free to submit a PR!

Last Thing

Smex is my first stab at building with Go. If you have any feedback, comments or notice any bugs, I'd be more than happy to consider them through the github issue tracker or better yet send me a pull request!

License

Documentation

Overview

Copyright © 2020 Ben Shi

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.