Exploiting Multiple Social Media Sources and Multiple Modalities for Severe Weather Management: The Case Study of the Medicane Ianos

Abstract

Stathis G. Arapostathis*

The current paper explored aspects of social media datasets when those were treated for disaster management (DM). The aspects were accumulated into three basic components: I. simultaneous processing of multiple social media sources. ii. Processing of multiple modalities iii. Organizing and visualizing output effectively. The case study used was the Medicane Ianos, occurred during 2020 in the Central and East Mediterranean. Data were collected from 4 social media sources: Instagram, Flickr, X and YouTube (YT). After scraping the data were merged per modality, resulting to a dataset of 7,058 text strings, 2,949 photos and 168,150 video frames, extracted from 752 YT and Instagram videos.

Sequentially, the processing included four binary classifications of the text strings, in which the effectiveness of LSTM-RNNs and Transformers was assessed: I. Medicane identification, II. Consequences, III. Disaster Management (DM) Info, IV. Weather. Mapping the text strings was the third part of the process, and included location entity recognition (LER), geocoding, the use of conventional geoparsing methods and geographic information systems (GIS) analysis.

Next, photos related to the Medicane Ianos were identified from the 2,949 posted photos and 168,150 video frames. Three deep learning models: a VGG19, a ResNet101 and an EfficientNetB0 were fine-tuned on Deucalion dataset v1.0 (binary classification) for that purpose. Moreover a useful RSVI simple index was invented for measuring the related content in the posted videos.

Among the main findings of the research, was that LSTM-RNN was more effective in the majority of the text classification tasks. DM info was mostly extracted from X texts and less from Flickr captions. Moreover the ResNet101 model with train accuracy 0.95 and validation accuracy 0.93 performed better in both identifying images and video frames. Finally, according to the invented RSVI index, Instagram videos had greater accumulation of medicane related content, while the analyzed YT videos had more related volume.

HTML PDF