A multimodal hybrid deep learning approach for pest forecasting using time series and satellite images

Abstract

Accurate forecasting of agricultural pests is essential for optimizing crop protection and mitigating economic losses. This work introduces a multimodal hybrid methodology to forecast the population of the olive fruit fly, enabling one-week-ahead outbreak predictions. The methodology consists of integrating an embedding model based on a deep convolutional neural network and other machine learning models with the aim of processing data from satellite images and time series. Different machine learning algorithms such as random forest, extreme gradient boosting and a fully connected deep neural network have been evaluated to be used in the proposed multimodal hybrid methodology. Firstly, the deep convolutional neural network extracts spatial features from Sentinel-2 L2A satellite imagery, specifically vegetation indexes, and then these extracted features are fused with meteorological data to improve the accuracy of the predictions obtained by a machine learning model. Results using images and time series from four plots of olive groves in Andalusia (Spain) are reported and compared with a convolutional neural network, random forest, extreme gradient boosting and a fully connected deep learning model when used separately and without data fusion. Experimental results show that the multimodal hybrid approach outperforms standalone machine learning models, improving predictive capabilities and facilitating timely and informed decision-making in agricultural pest management. Additionally, our findings highlight the relevance of multi-source data fusion in forecasting tasks, reinforcing the potential of deep learning for real-world agricultural applications.