Estimating soil organic carbon using field VNIR-SWIR spectroscopy and existing soil spectral libraries: Mitigating heterogeneity, roughness and moisture effects

VNIR-SWIR spectra acquired in the field are inherently affected by uncontrolled conditions, such as variable illumination, surface roughness, and soil moisture. As a result, models trained on soil spectral libraries (SSLs), typically composed of dry, sieved samples analyzed in the lab, often fail when applied directly to field spectra. With this study we propose a routine to succeed with this desirable approach. We collected field spectra from 178 locations across seven countries under heterogeneous field conditions using different spectrometers. At each site, two surface smoothing intensities were compared. Two SSLs, LUCAS topsoil and GEO[sbnd]CRADLE, were used to train machine learning models for predicting soil organic carbon (SOC), later applied to the field spectra under different correction scenarios: with or without Internal Soil Standard (ISS) harmonization and External Parameter Orthogonalization (EPO) to mitigate the effects of soil moisture. Combining ISS and EPO enables SSL-based models to reliable predict SOC from field-acquired spectra, particularly when using the LUCAS SSL in combination with a spectrally localized approach to reduce training set size (R² = 0.70; RPD = 1.66). Model performances are consistent with previous laboratory-based studies despite the diverse field conditions. A refined workflow for SOC estimation using hybrid spectral data is proposed, consisting of three steps: i) Spectral acquisition on highly smoothed surfaces; ii) ISS harmonization to align spectra across from different instruments; iii) EPO correction to reduce non-systematic spectral variability due to masking factors such as moisture, enhancing spectral consistency under variable field conditions.