Quantifying feature importance for online content moderation

Accurately estimating how users respond to moderation interventions is key to designing effective and user-centred moderation strategies. This requires understanding which user characteristics are associated with different behavioural responses. We address this problem by analysing the informativeness of 753 socio-behavioural, linguistic, relational, and psychological features for predicting behavioural changes in 16.8K users affected by a large-scale moderation intervention on Reddit. We frame the task in terms of quantification, which is well-suited to estimating shifts in aggregate behaviour under distribution shift, and apply a greedy feature selection strategy to identify the most informative features and estimate their importance. Our results show that predictive performance varies substantially across tasks: changes in activity and toxicity can be estimated reliably, whereas changes in participation diversity are markedly harder to predict. We find that a small subset of features consistently improves performance across tasks, while many others are either task-specific or provide limited additional value. Importantly, models based on carefully selected features outperform both single feature groups and the full feature set, indicating that combining complementary signals is crucial for accurate estimation. Overall, our findings highlight the complexity and task-dependence of post-moderation user behaviour, suggesting that effective moderation strategies should be tailored not only to user characteristics but also to the specific behavioural outcomes of interest.