Building analytical platform with Big Data solutions for log files of PanDA infrastructure; Journal of Physics: Conference Series; Vol. 1015 : Information Technologies in Business and Industry (ITBI2018)
| Parent link: | Journal of Physics: Conference Series Vol. 1015 : Information Technologies in Business and Industry (ITBI2018).— 2018.— [032003, 6 p.] |
|---|---|
| Autor corporatiu: | |
| Altres autors: | , , , , , |
| Sumari: | Title screen The paper describes the implementation of a high-performance system for the processing and analysis of log files for the PanDA infrastructure of the ATLAS experiment at the Large Hadron Collider (LHC), responsible for the workload management of order of 2M daily jobs across the Worldwide LHC Computing Grid. The solution is based on the ELK technology stack, which includes several components: Filebeat, Logstash, ElasticSearch (ES), and Kibana. Filebeat is used to collect data from logs. Logstash processes data and export to Elasticsearch. ES are responsible for centralized data storage. Accumulated data in ES can be viewed using a special software Kibana. These components were integrated with the PanDA infrastructure and replaced previous log processing systems for increased scalability and usability. The authors will describe all the components and their configuration tuning for the current tasks, the scale of the actual system and give several real-life examples of how this centralized log processing and storage service is used to showcase the advantages for daily operations. |
| Idioma: | anglès |
| Publicat: |
2018
|
| Col·lecció: | Mathematical simulation and data processing |
| Matèries: | |
| Accés en línia: | http://dx.doi.org/10.1088/1742-6596/1015/3/032003 http://earchive.tpu.ru/handle/11683/52921 |
| Format: | Electrònic Capítol de llibre |
| KOHA link: | https://koha.lib.tpu.ru/cgi-bin/koha/opac-detail.pl?biblionumber=659478 |
MARC
| LEADER | 00000nla2a2200000 4500 | ||
|---|---|---|---|
| 001 | 659478 | ||
| 005 | 20231101135058.0 | ||
| 035 | |a (RuTPU)RU\TPU\network\28050 | ||
| 035 | |a RU\TPU\network\27958 | ||
| 090 | |a 659478 | ||
| 100 | |a 20190221d2018 k y0engy50 ba | ||
| 101 | 0 | |a eng | |
| 105 | |a y z 100zy | ||
| 135 | |a vrcn ---uucaa | ||
| 181 | 0 | |a i | |
| 182 | 0 | |a b | |
| 200 | 1 | |a Building analytical platform with Big Data solutions for log files of PanDA infrastructure |f A. A. Alekseev [et al.] | |
| 203 | |a Text |c electronic | ||
| 225 | 1 | |a Mathematical simulation and data processing | |
| 300 | |a Title screen | ||
| 320 | |a [References: 6 tit.] | ||
| 330 | |a The paper describes the implementation of a high-performance system for the processing and analysis of log files for the PanDA infrastructure of the ATLAS experiment at the Large Hadron Collider (LHC), responsible for the workload management of order of 2M daily jobs across the Worldwide LHC Computing Grid. The solution is based on the ELK technology stack, which includes several components: Filebeat, Logstash, ElasticSearch (ES), and Kibana. Filebeat is used to collect data from logs. Logstash processes data and export to Elasticsearch. ES are responsible for centralized data storage. Accumulated data in ES can be viewed using a special software Kibana. These components were integrated with the PanDA infrastructure and replaced previous log processing systems for increased scalability and usability. The authors will describe all the components and their configuration tuning for the current tasks, the scale of the actual system and give several real-life examples of how this centralized log processing and storage service is used to showcase the advantages for daily operations. | ||
| 461 | 1 | |0 (RuTPU)RU\TPU\network\3526 |t Journal of Physics: Conference Series | |
| 463 | 1 | |0 (RuTPU)RU\TPU\network\28043 |t Vol. 1015 : Information Technologies in Business and Industry (ITBI2018) |o International Conference, January 17-20, 2018, Tomsk, Russian Federation |o [proceedings] |f National Research Tomsk Polytechnic University (TPU) |v [032003, 6 p.] |d 2018 | |
| 610 | 1 | |a электронный ресурс | |
| 610 | 1 | |a труды учёных ТПУ | |
| 610 | 1 | |a платформы | |
| 610 | 1 | |a Big Data | |
| 610 | 1 | |a лог-файлы | |
| 610 | 1 | |a инфраструктура | |
| 610 | 1 | |a высокопроизводительные системы | |
| 610 | 1 | |a программное обеспечение | |
| 610 | 1 | |a данные | |
| 610 | 1 | |a обработка | |
| 610 | 1 | |a хранение | |
| 701 | 1 | |a Alekseev |b A. A. |c specialist in the field of automatic control |c Engineer of Tomsk Polytechnic University, Postgraduate |f 1988- |g Aleksandr Aleksandrovich |2 stltpush |3 (RuTPU)RU\TPU\pers\37959 | |
| 701 | 1 | |a Barreiro Megino |b F. G. | |
| 701 | 1 | |a Klimentov |b A. A. | |
| 701 | 1 | |a Korchuganova |b T. A. | |
| 701 | 1 | |a Maendo |b T. | |
| 701 | 1 | |a Padolski |b S. V. | |
| 712 | 0 | 2 | |a Национальный исследовательский Томский политехнический университет |b Инженерная школа неразрушающего контроля и безопасности |b Центр промышленной томографии |b Научно-производственная лаборатория "Бетатронная томография крупногабаритных объектов" |h 7983 |2 stltpush |3 (RuTPU)RU\TPU\col\23717 |
| 801 | 1 | |a RU |b 63413507 |c 20150101 |g RCR | |
| 801 | 2 | |a RU |b 63413507 |c 20190228 |g RCR | |
| 856 | 4 | |u http://dx.doi.org/10.1088/1742-6596/1015/3/032003 | |
| 856 | 4 | |u http://earchive.tpu.ru/handle/11683/52921 | |
| 942 | |c CF | ||