rucio-dataset-monitoring

README

rucio-dataset-monitoring package

The rucio-dataset-monitoring package serves Rucio aggregated data, which is stored in MongoDB, using JQuery datatables.

Package development is in still progress. You may see test page from 🚀 here 🚀 , beware that some functionalities not there yet.

Introduction

Rucio dataset monitoring using aggregated Spark data

Main aim of the project is to show all Rucio dataset information in a web page with required functionalities

Used softwares

• DataTables: very popular JQuery library to show pretty tables with nice UI and search/query functionalities
• MongoDB: used to store following data in separate collections. Multiple MongoDB indexes are created to use full performance of it.
  • aggregated Rucio datasets results,
  • detailed dataset results,
  • short url hashId:request binding
  • data source timestamp
• JQuery/JS: to manipulate and customize DataTables, JQuery and JS used
• Go gin-gonic web framework is used to serve web pages and MongoDB query API calls

Main page functionalities

• Sort
• Detailed dataset in RSEs functionality: green "+" button
• Paging
• Count of search result
• Search using SearchBuilder conditions: "Add condition". Even though SB allows nested conditions, now it supports depth=1
• Buttons:copy, excel, PDF, column visibility
• Short URL: which is the advanced functionality of this service. Please see its documentation for more details.

Docs

• Deployment instructions
• Datatables example request
• Short URL implementation

Bug Report & Contribution

Please open GitHub issue

References

• https://github.com/dmwm/das2go
• https://dev.to/hackmamba/build-a-rest-api-with-golang-and-mongodb-gin-gonic-version-269m
• https://github.com/dmwm/CMSKubernetes/tree/master/docker/rucio-dataset-monitoring
• https://github.com/dmwm/CMSKubernetes/blob/master/kubernetes/monitoring/services/datasetmon
• https://github.com/gin-gonic/gin/issues/346

Special Thanks

• Special thanks to Valentin for his suggestions, reviews and guidance.
• Many thanks to Danilo Piparo for bringing up the idea of this project