Development of DKB ETL module in case of data conversion; Journal of Physics: Conference Series; Vol. 1015 : Information Technologies in Business and Industry (ITBI2018)
| Parent link: | Journal of Physics: Conference Series Vol. 1015 : Information Technologies in Business and Industry (ITBI2018).— 2018.— [032055, 5 p.] |
|---|---|
| Autor corporatiu: | |
| Altres autors: | , , , |
| Sumari: | Title screen Modern scientific experiments involve the producing of huge volumes of data that requires new approaches in data processing and storage. These data themselves, as well as their processing and storage, are accompanied by a valuable amount of additional information, called metadata, distributed over multiple informational systems and repositories, and having a complicated, heterogeneous structure. Gathering these metadata for experiments in the field of high energy nuclear physics (HENP) is a complex issue, requiring the quest for solutions outside the box. One of the tasks is to integrate metadata from different repositories into some kind of a central storage. During the integration process, metadata taken from original source repositories go through several processing steps: metadata aggregation, transformation according to the current data model and loading it to the general storage in a standardized form. The R&D project of ATLAS experiment on LHC, Data Knowledge Base, is aimed to provide fast and easy access to significant information about LHC experiments for the scientific community. The data integration subsystem, being developed for the DKB project, can be represented as a number of particular pipelines, arranging data flow from data sources to the main DKB storage. The data transformation process, represented by a single pipeline, can be considered as a number of successive data transformation steps, where each step is implemented as an individual program module. This article outlines the specifics of program modules, used in the dataflow, and describes one of the modules developed and integrated into the data integration subsystem of DKB. |
| Idioma: | anglès |
| Publicat: |
2018
|
| Col·lecció: | Mathematical simulation and data processing |
| Matèries: | |
| Accés en línia: | http://dx.doi.org/10.1088/1742-6596/1015/3/032055 http://earchive.tpu.ru/handle/11683/52924 |
| Format: | Electrònic Capítol de llibre |
| KOHA link: | https://koha.lib.tpu.ru/cgi-bin/koha/opac-detail.pl?biblionumber=659489 |
MARC
| LEADER | 00000nla2a2200000 4500 | ||
|---|---|---|---|
| 001 | 659489 | ||
| 005 | 20231223193243.0 | ||
| 035 | |a (RuTPU)RU\TPU\network\28061 | ||
| 035 | |a RU\TPU\network\28050 | ||
| 090 | |a 659489 | ||
| 100 | |a 20190221d2018 k y0engy50 ba | ||
| 101 | 0 | |a eng | |
| 105 | |a y z 100zy | ||
| 135 | |a vrcn ---uucaa | ||
| 181 | 0 | |a i | |
| 182 | 0 | |a b | |
| 200 | 1 | |a Development of DKB ETL module in case of data conversion |f A. Yu. Kaida, M. V. Golosova, M. A. Grigorjeva (Grigorieva), M. Yu. Gubin | |
| 203 | |a Text |c electronic | ||
| 225 | 1 | |a Mathematical simulation and data processing | |
| 300 | |a Title screen | ||
| 320 | |a [References: 11 tit.] | ||
| 330 | |a Modern scientific experiments involve the producing of huge volumes of data that requires new approaches in data processing and storage. These data themselves, as well as their processing and storage, are accompanied by a valuable amount of additional information, called metadata, distributed over multiple informational systems and repositories, and having a complicated, heterogeneous structure. Gathering these metadata for experiments in the field of high energy nuclear physics (HENP) is a complex issue, requiring the quest for solutions outside the box. One of the tasks is to integrate metadata from different repositories into some kind of a central storage. During the integration process, metadata taken from original source repositories go through several processing steps: metadata aggregation, transformation according to the current data model and loading it to the general storage in a standardized form. The R&D project of ATLAS experiment on LHC, Data Knowledge Base, is aimed to provide fast and easy access to significant information about LHC experiments for the scientific community. The data integration subsystem, being developed for the DKB project, can be represented as a number of particular pipelines, arranging data flow from data sources to the main DKB storage. The data transformation process, represented by a single pipeline, can be considered as a number of successive data transformation steps, where each step is implemented as an individual program module. This article outlines the specifics of program modules, used in the dataflow, and describes one of the modules developed and integrated into the data integration subsystem of DKB. | ||
| 461 | 1 | |0 (RuTPU)RU\TPU\network\3526 |t Journal of Physics: Conference Series | |
| 463 | 1 | |0 (RuTPU)RU\TPU\network\28043 |t Vol. 1015 : Information Technologies in Business and Industry (ITBI2018) |o International Conference, January 17-20, 2018, Tomsk, Russian Federation |o [proceedings] |f National Research Tomsk Polytechnic University (TPU) |v [032055, 5 p.] |d 2018 | |
| 610 | 1 | |a электронный ресурс | |
| 610 | 1 | |a труды учёных ТПУ | |
| 610 | 1 | |a модули | |
| 610 | 1 | |a большие данные | |
| 610 | 1 | |a обработка данных | |
| 610 | 1 | |a хранение | |
| 610 | 1 | |a методанные | |
| 610 | 1 | |a агрегация | |
| 610 | 1 | |a программные модули | |
| 701 | 1 | |a Kaida |b A. Yu. |c Specialist in the field of informatics and computer technology |c Programmer of Tomsk Polytechnic University |f 1995- |g Anastasia Yurievna |3 (RuTPU)RU\TPU\pers\45842 |9 22001 | |
| 701 | 1 | |a Golosova |b M. V. | |
| 701 | 1 | |a Grigorjeva (Grigorieva) |b M. A. |c specialist in the field of informatics and computer technology |c Researcher of Tomsk Polytechnic University |f 1983- |g Mariya Aleksandrovna |3 (RuTPU)RU\TPU\pers\38423 | |
| 701 | 1 | |a Gubin |b M. Yu. |c specialist in the field of informatics and computer technology |c Lead programmer of Tomsk Polytechnic University |f 1985- |g Maksim Yurjevich |3 (RuTPU)RU\TPU\pers\38455 | |
| 712 | 0 | 2 | |a Национальный исследовательский Томский политехнический университет |b Инженерная школа информационных технологий и робототехники |b Отделение информационных технологий |3 (RuTPU)RU\TPU\col\23515 |
| 801 | 1 | |a RU |b 63413507 |c 20150101 |g RCR | |
| 801 | 2 | |a RU |b 63413507 |c 20210204 |g RCR | |
| 856 | 4 | |u http://dx.doi.org/10.1088/1742-6596/1015/3/032055 | |
| 856 | 4 | |u http://earchive.tpu.ru/handle/11683/52924 | |
| 942 | |c CF | ||