shimakaze

README

Shimakaze

Shimakaze is Vtuber Wikia scraper and REST API.

Features

• Save vtuber details
  • Vtuber data
  • Vtuber channels data and videos
    • Youtube
    • Twitch
    • Bilibili
    • Niconico
• Save agency's vtuber list
• Auto update vtuber & agency data (cron)
• Interchangeable cache
  • no cache
  • inmemory
  • Redis
  • Memcache
• Interchangeable pubsub
  • NSQ
  • RabbitMQ
  • Redis
  • Google PubSub
• Swagger
• Docker
• Newrelic monitoring
  • HTTP
  • Cron
  • Database
  • Cache
  • Pubsub
  • External API

More will be coming soon...

Requirement

• Go
• MongoDB
• Youtube API key
• Twitch Client id & secret
• PubSub (NSQ/RabbitMQ/Redis/Google PubSub)
• (optional) Cache (Redis/Memcache)
• (optional) Docker & Docker Compose
• (optional) Newrelic license key

Installation

Without Docker & Docker Compose

1. Clone the repository.

git clone github.com/rl404/shimakaze

1. Rename .env.sample to .env and modify the values according to your setup.
2. Run. You need at least 2 consoles/terminals.

# Run the API.
make

# Run the consumer.
make consumer

1. localhost:45001 is ready (port may varies depend on your .env).

Other commands

# Update old vtuber data.
make cron-update

# Fill missing vtuber data.
make cron-fill

With Docker & Docker Compose

1. Clone the repository.

git clone github.com/rl404/shimakaze

1. Rename .env.sample to .env and modify the values according to your setup.
2. Get docker image.

# Pull existing image.
docker pull rl404/shimakaze

# Or build your own.
make docker-build

1. Run the container. You need at least 2 consoles/terminals.

# Run the API.
make docker-api

# Run the consumer.
make docker-consumer

1. localhost:45001 is ready (port may varies depend on your .env).

Other commands

# Update old vtuber data.
make docker-cron-update

# Fill missing vtuber data.
make docker-cron-fill

# Stop running containers.
make docker-stop

Environment Variables

Env	Default	Description
SHIMAKAZE_APP_ENV	dev	Environment type (dev/prod).
SHIMAKAZE_HTTP_PORT	45001	HTTP server port.
SHIMAKAZE_HTTP_READ_TIMEOUT	5s	HTTP read timeout.
SHIMAKAZE_HTTP_WRITE_TIMEOUT	5s	HTTP write timeout.
SHIMAKAZE_HTTP_GRACEFUL_TIMEOUT	10s	HTTP graceful timeout.
SHIMAKAZE_CACHE_DIALECT	inmemory	Cache type (nocache/redis/inmemory/memcache)
SHIMAKAZE_CACHE_ADDRESS		Cache address.
SHIMAKAZE_CACHE_PASSWORD		Cache password.
SHIMAKAZE_CACHE_TIME	24h	Cache time.
SHIMAKAZE_DB_ADDRESS	mongodb://localhost:27017	Database address with port.
SHIMAKAZE_DB_NAME	shimakaze	Database name.
SHIMAKAZE_DB_USER		Database username.
SHIMAKAZE_DB_PASSWORD		Database password.
SHIMAKAZE_PUBSUB_DIALECT	rabbitmq	Pubsub type (nsq/rabbitmq/redis/google)
SHIMAKAZE_PUBSUB_ADDRESS		Pubsub address (if you are using google, this will be your google project id).
SHIMAKAZE_PUBSUB_PASSWORD		Pubsub password (if you are using google, this will be the content of your google service account json).
SHIMAKAZE_CRON_UPDATE_LIMIT	10	Vtuber & agency count limit when updating old data.
SHIMAKAZE_CRON_FILL_LIMIT	10	Vtuber & agency count limit when filling missing data.
SHIMAKAZE_CRON_AGENCY_AGE	7	Age of old agency data (in days).
SHIMAKAZE_CRON_ACTIVE_AGE	1	Age of old active vtuber data (in days).
SHIMAKAZE_CRON_RETIRED_AGE	7	Age of old retired vtuber data (in days).
SHIMAKAZE_NEWRELIC_NAME	shimakaze	Newrelic application name.
SHIMAKAZE_NEWRELIC_LICENSE_KEY		Newrelic license key.
SHIMAKAZE_YOUTUBE_KEY		Youtube API key.
SHIMAKAZE_YOUTUBE_MAX_AGE	60	Age limit of youtube videos (in days).
SHIMAKAZE_TWITCH_CLIENT_ID		Twitch client id.
SHIMAKAZE_TWITCH_CLIENT_SECRET		Twitch client secret.
SHIMAKAZE_TWITCH_MAX_AGE	60	Age limit of twitch videos (in days).
SHIMAKAZE_BILIBILI_MAX_AGE	60	Age limit of bilibili videos (in days).
SHIMAKAZE_NICONICO_MAX_AGE	60	Age limit of niconico videos (in days).

Trivia

Shimakaze's name is taken from one of the fastest japanese destroyer. Also, exists in Kantai Collection games and manga.

Disclaimer

Shimakaze is meant for educational purpose and personal usage only. Please use it responsibly according to Wikia License.

All data belong to their respective copyrights owners, shimakaze does not have any affiliation with content providers.

License

MIT License

Copyright (c) 2023 Axel