|
Article information
2018 , Volume 23, ¹ 4, p.110-123
Potapov V.P., Popov S.E., Kostylev M.A.
The information and computational system for the massive parallel processing of radar data based on Apache Spark framework
The aim of the presented work is the development of an information computational system for processing radar images with the ability to visualize, configure and run algorithms for the main stages of processing interferometric data by the Persistent Scatterer method integrated with the MPP system (massive parallel processing) for high-performance monitoring of the Earth surface displacement of aerospace survey sites. As a result of the analysis of the different approaches used in the processing of radar data and the review of distributed computing technologies, a distributed information system based on the architecture of massively parallel execution of the Apache Hadoop ecosystem processes the streaming post-processing of radar images and the construction of a displacement map was proposed and implemented. A software implementation is presented in the form of a web portal based on ReactJS components, including automated downloading and updating of the Sentinel-1A radar image database using RESTful API technology. The innovation of suggested solution consists of the model of the interaction between developed processing modules based on the isolated execution context with HDFS data storage during the preparing procedure and the complete cycle for the processing of the Earth surface displacement. An integrated approach to the developing scalable front-end and back-end software complex components with the use of ReactJS, Redux and Apache Spark framework was used for the first time. Supporting of WPS specification makes it possible using almost any GIS, which works with this standard. The evaluation of a scientific and technological level of research shows high performance of the developed system while maintaining the results quality. In particular, the adapted and integrated ESA SNAP Toolbox returned identical arrays of processed interferometric data in the per-pixel comparison but the speed of the procedure is several times faster.
[full text]
Keywords: monitoring of Earth surface displacements, radar interferometry, systems with massively parallel execution of tasks, high-performance processing of spatial data
doi: 10.25743/ICT.2018.23.16507
Author(s):
Potapov Vadim Petrovich
Dr. , Professor
Position: Deputy director
Office: Federal Research Center for Information and Computational Technologies
Address: 650003, Russia, Kemerovo
Phone Office: (3842) 211400
E-mail: potapov@ict.sbras.ru
SPIN-code: 8947-1880Popov Semen Evgenievich
PhD.
Position: Senior Research Scientist
Office: Federal Research Center for Information and Computational Technologies
Address: 630090, Russia, Novosibirsk, Lavrentiev avenue, 6
Phone Office: (905)9692107
E-mail: popov@ict.sbras.ru
SPIN-code: 5627-9584Kostylev Mikhail Alexandrovich
Position: Student
Office: Institute of Computational Technologies SO RAN
Address: 630090, Russia, Novosibirsk, Lavrentiev avenue, 6
Phone Office: (3842) 211400
References:
1. Bondur, V.G., Savin, A.I. The concept for creating of environmental monitoring systems for environmental and natural resources purposes. Issledovanie Zemli iz kosmosa. 1992; (6):70-78. (In Russ.)
2. Kantemirov, Yu.I. Satellite synthetic aperture radar monitoring of ground and constructions displacements and deformations. "SOVZOND" Company experience. Scientific Journal of Science and Technology. 2013; 5(51):52-54 (In Russ.)
3. Sbas Tutorial. Available at: http://www.sarmap.ch/tutorials/sbas_tutorial_V_2_0.pdf (12.06.2018)
4. Sousaa, J. J., Hooperc, J.A., Hanssenc, R.F., Bastosd, L.C., Ruize, A.M. Persistent Scatterer InSAR: A comparison of methodologies based on a model of temporal deformation vs spatial correlation selection criteria. Remote Sensing of Environment. 2011; 115(10):2652-2663
5. Oparin, V.N., Kozyrev, A.A., Panin, V.I., Savchenko, S.N., Kasparyan, E.V., Rybin, V.V., Semenova, I.E., Kagan, M.M., Fedotova, Yu.V., Zhuravleva, O.G., Sashurin, A.D., Baryakh, A.A., Emanov, A.F., Leskova, E.V., Emanov, A.A. Destruktsiya zemnoy kory i protsessy samoorganizatsii v oblastyakh sil'nogo tekhnogennogo vozdeystviya [Earth's crust destruction and self-organization processes in highly industrial regions]. Novosibirsk: Izdatel’stvo SO RAN; 2012: 632. (In Russ.)
6. Adushkin, V.V., Oparin, V.N. From the alternating-sign explosion response of rocks to the pendulum waves in stressed geomedia. Part I. Journal of Mining Science. 2012; 48(2):203-222.
7. Adushkin, V.V., Oparin, V.N. From the alternating-sign explosion response of rocks to the pendulum waves in stressed geomedia. Part II. Journal of Mining Science. 2013; 49(2):175–209
8. Simmons, A.D., Kerekes, J.P., Raqueno, N.G. Hyperspectral monitoring of chemically sensitive plant sentinels. Proceeding SPIE 7457. Imaging Spectrometry XIV, 74570G, San Diego, CA. 2003: 45-51.
9. Lupyan, E.A., Savin, I.Yu., Bartalev, S.A., Tolpin V.A., Balashov I.V., Plotnikov D.E. Satellite monitoring service for vegetation ("Vega"). Sovremennye problemy distantsionnogo zondirovaniya Zemli iz kosmosa. 2011; 8(1):190-198. (In Russ)
10. Lavrova, O. Yu., Loupian, E.A., Mityagina, M.I., Uvarov, I.A., Bocharova, T.Yu. See the Sea - Multi-User Information System Ocean Processes Investigations Based on Satellite Remote Sensing Data. Bollettino di Geofisica teorica ed applicata. An International Journal of Earth Sciences. 2013; (54):146-147.
11. Gordeev, E.I., Girina, O.A., Lupyan, E.A., Kashnitsky, A.V., Uvarov, I.A., Efremov, V.Yu., Melnikov, D.V., Manevich, A.G., Sorokin, A.A., Verkhoturov, A.L., Romanova, I.M., Kramareva, L.S., Korolev, S.P. Studies of Kamchatka volcanic eruptions products using hyperspectral satellite data in VolSatView information system. Sovremennye problemy distantsionnogo zondirovaniya Zemli iz kosmosa. 2015; 12(1):113-128. (In Russ)
12. Lupyan, E.A., Bartalev, S.A., Ershov, D.V., Kotelnikov, R.V., Balashov, I.V., Burtsev, M.A., Egorov, V.A., Efremov, V.Yu., Zharko, V.O., Kovganko, K.A., Kolbudaev, P.A., Krasheninnikova, Yu.S., Proshin, A.A., Mazurov, A.A., Uvarov, I.A., Stytsenko, F.V., Sychugov, I.G., Fleitman, E.V., Khvostikov, S.A., Shulyak P.P. Satellite data processing management in Forest Fires Remote Monitoring Information System (ISDM-Rosleskhoz) of the Federal Agency for Forestry. Sovremennye problemy distantsionnogo zondirovaniya Zemli iz kosmosa. 2015; 12(5):222-250.
13.Takeuchi, S., Yamada, H. Monitoring of forest fire damage by using JERS-1 InSAR. Geoscience and Remote Sensing Symposium (IGARSS '02). Toronto, Ontario, Canada; 2002: 3290-3292.
14. Geohazardû Tep - Geobrowser. Available at: https://geohazards-tep.eo.esa.int/geobrowser/#!&context=InSAR_QL (31.05.2018)
15. McLean, S., Naftel, J., Williams, K. Microsoft .NET Remoting. 1st ed. Microsoft Press; 2003: 336.
16. Berman, F., Wolski, R. Application-Level Scheduling on Distributed Heterogeneous Networks. Supercomputing: Proceedings of the ACM/IEEE conference. Pittsburgh, Pennsylvania USA, May 25-28, 1996. IEEE Computer Society; 1996: 1-28.
17. Maheswaran, M., Ali, S. Dynamic Matching and Scheduling of a Class of Independent Tasks onto Heterogeneous Computing Systems. Journal of Parallel and Distributed Computing. 1999; 59(2):107-131.
18. Laszewski, G., Foster, I. Gawor, J. Laszewski, G. CoG Kits: A Bridge between Commodity Distributed Computing and High-Performance Grids. Proceedings of the ACM Java Grande 2000 Conference, San Francisco, CA, USA, June 3-5, 2000. ACM Press; 2000: 97-106.
19. Yang, T., Gerasoulis, A. DSC: Scheduling Parallel Tasks on an Unbounded Number of Processors. IEEE Transactions on Parallel and Distributed Systems. 1994; 5(9):951-967.
20. Qusay, H. M. Distributed Java Programming with RMI and CORBA. Oracle Technology Network. Available at: http://www.oracle.com/technetwork/articles/javase/rmi-corba-136641.html (14.06.2018)
21. Reyes-Ortiz, J.L., Oneto, L., Anguita, D. Big Data Analytics in the Cloud: Spark on Hadoop vs MPI/OpenMPI. INNS Conference on Big Data 2015: Conference proceedings. San Francisco, USA, 8-10 August, 2015. ACM Press; 2015: 121-130.
22. Mavridis, I., Karatza, H. Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. Journal of Systems and Software. 2017; (125):133-151.
23. Polato, I., Reginald, R., Goldman, A., Kon, F. A comprehensive view of Hadoop research - A systematic literature review. Journal of Network and Computer Applications. 2014; (46):1-25
24. Chen, Xu. Big Data Analytic Framework s for GIS (Amazon EC2, Hadoop, Spark). Comprehensive Geographic Information Systems. 2017; (1):148-152
25. Verbesselt, J. Big Data: Techniques and Technologies in Geoinformatics. International Journal of Applied Earth Observation and Geoinformation. 2015; 35 (Part B):368-369.
26. Yang, Ch., Yu Manzhu, Hu Fei, Jiang Yongyao, Li Yun. Utilizing Cloud Computing to address big geospatial data challenges. Computers, Environment and Urban Systems. 2017; 61(Part B):120-128.
27. Hadoop, Storm, Samza, Spark, and Flink: Big Data Frameworks Compared. Digital Ocean. Available at: https://www.digitalocean.com/community/tutorials/hadoop-storm-samza-spark-and-flink-big-data-frameworks-compared (12.05.2018)
28. Spark vs. Tez: What's the Difference? Xplenty. Available at: https://www.xplenty.com/blog/2015/01/apache-spark-vs-tez-comparison/ (12.06.2018)
29. Feature wise comparison between Apache Hadoop vs Spark vs Flink. The Server Side. Available at: http://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/Feature-wise-comparison-between-Apache-Hadoop-vs-Spark-vs-Flink (12.06.2018)
Bibliography link:
Potapov V.P., Popov S.E., Kostylev M.A. The information and computational system for the massive parallel processing of radar data based on Apache Spark framework // Computational technologies. 2018. V. 23. ¹ 4. P. 110-123
|
