RUCIO

 

Rucio is a project that provides services and associated libraries for allowing scientific collaborations to manage large volumes of data spread across facilities at multiple institutions and organisations.

Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities. Rucio is a highly scalable, and modular data management solution that can be used for the needs of different communities in the scientific domain like high-energy physics, astronomy, biology or anywhere else requiring large data storage and discovery.

 

Explore the technology specifics:

Rucio is a project that provides services and associated libraries for allowing scientific collaborations to manage large volumes of data spread across facilities at multiple institutions and organisations. Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities.

Rucio offers advanced features, is highly scalable, and modular. It is a data management solution that covers the needs of different communities in the scientific domain (e.g., HEP, astronomy, biology).

The project +

Having demonstrated very large scale data management capabilities, Don Quijote (DQ2), the ATLAS Distributed Data Management System used for HEP experiments at CERN had reached its limits in terms of scalability. The primary concerns were the requirement of a large number of support staff to operate. difficulty in interfacing with new technologies To address these very scaling requirements for HEP experiments, Rucio as a Distributed Data Management System, was developed. Drawing benefits from advances in Cloud & Big Data computations, it relies on a conceptual data model to ensure system stability. Dataflow autonomy and automation are the key design principles guiding the development of Rucio. To reduce the operational overheads of the support staff, it employs an automation framework and also accounts for newer use cases & user requirements of the LHC Run2 experiments.

What can Rucio do? + Standing on the shoulders of its predecessor, ATLAS, the capabilities of Rucio are currently leveraged for:

  • Storage of detector data, simulator data, and user data
  • Unified interfacing of heterogenous network & storage infrastructure
  • Support for newer protocols in Storage & Network using plugins
  • Data Recovery
  • Adaptive Replication

Architecture of Rucio +

Rucio is based on a distributed system architecture & can be sectioned into four major layers:

Clients

The clients layer consists of components such as the command line clients (CLI), Python clients, and the Javascript-based web user interface and configuration.

Server

The server layer serves the purpose of authentication & provides a common API for interaction with clients & other external application, as also the Web UI.

Core

This layer consists of all the Rucio-level abstractions that are explained at length in the Concepts section.

Daemons

The daemons layer takes care of all the asynchronous & continuous workflows in the background.

daemon rucio 1

 

In addition to the four main layers, we have the storage resources & transfer tools, as well as the underlying persistent layers. These are represented in the architecture diagram by the different queuing systems, transactional relational databases, & analytics storage on non-relational databases.

Storage & Transfer Tools layer

The Storage layer is responsible for the various interactions with different grid middleware tools & storage systems. The transfer tool interface is a feature of Rucio that enables the daemons to submit, query, and cancel transfers generically & independently from the actual transfer service being used. This layer is not a software that resides in a datacenter but is more representative of the various abstractions in a storage system. It can be configured dynamically & centrally to suite requirements.

Persistence Layer

This is the layer responsible for keeping all the data & application states for all daemons. Also known as the catalog, it requires a transactional database. The persistence layer supports many different types of transactions relational database management systems such as SQLite, MySQL, PostgreSQL, or Oracle 

persistance layer rucio 2

What can Rucio do? Standing on the shoulders of its predecessor, ATLAS, the capabilities of Rucio are currently leveraged for:

  • Storage of detector data, simulator data, and user data
  • Unified interfacing of heterogenous network & storage infrastructure
  • Support for newer protocols in Storage & Network using plugins
  • Data Recovery
  • Adaptive Replication

The technology can have many applications in:

  • Large data storage, discovery - any data federation
  • Moving data from public clouds to private clouds with a single API