Inoculation vs Deplatforming

Evaluating Policy Synergies Against Online Extremism - An Agent-Based Simulation and Cost-Benefit Analysis

This post summarizes a short pilot behind a comprehensive project I am working on with Matteo Vergani and other colleagues at Deakin University. In this work, we developed a cost-benefit methodology to assess the peculiar merits of information inoculation as opposed to deplatforming for controlling extremism.

Evaluating Policy Synergies Against Online Extremism

Context

The research studied the emergence of extremist opinions within online communities. To do so we constructed a simulation model that mimics opinion formation within an anti-immigration Facebook group where users are susceptible to far-right extremism. We realistically calibrated the features of the artificial user base and opinion dynamics on empirical data from systematic reviews and empirical studies in the field of terrorism research thus ensuring a reliable replicable and evidence-based setting for our experiments.

Aims

The study aimed to assess the effectiveness of two distinct strategies adopted by policymakers around the world for mitigating the spread of online extremism: deplatforming (i.e. targeted surveillance and coercive removal of extremist users) and inoculation (i.e exposing the user base to weakened forms of extremist arguments to build group-wide resilience). Crucially our approach allowed us to evaluate the existence of synergies in the combined use of these policies as opposed to using only one policy.

Methodology

Following contemporary research on policy evaluation we base our assessment on realistic simulations of social dynamics. A similar approach is used for example for measuring the effectiveness of pharmaceutical or non-pharmaceutical interventions on the spread of COVID-19. To achieve robust predictions we tracked the evolution of opinion formation within the artificial group under each combination of policies through a time span of 365 days replicating each policy scenario for 100 times. As a result our study was conducted upon more than 4.4 million data points.

Figure 1. Radicalization dynamics for three policy mixes achieving equivalent end-state levels of radicalization through different configurations of inoculation and deplatforming: no inoculation/high deplatforming (left pane) scarce inoculation/medium deplatforming (centre pane) and high inoculation/moderate deplatforming (right pane). In each figure the black curve is the average radicalization level in the specific period and the grey area is the volatility of observed radicalization for that period across 100 simulations. Higher levels of inoculation stabilize the outcome of the simulations.
Figure 2. Pairs of cost-equivalent policy mixes (left one: no inoculation right one: max inoculation) under three target scenarios. Cost of monitoring is calibrated on a community of 1000 users using U.S. based Facebook contracting information. For each policy mix blue areas are the cost of monitoring whereas red areas are the total savings on monitoring caused by introducing the optimal level of inoculation.

Key Findings

  • Without inoculation deplatforming needs to be extensive and pervasive.
  • Inoculation is synergetic to deplatforming hence it can partially substitute it: inoculation enhances the success rate of coercive measures (i.e. stabilize the simulation outcomes) by structurally improving the societal resilience against extremism (see Figure 1).
  • A mixed approach that combines deplatforming with inoculation can abate extremism and at a lower cost than relying on deplatforming only (see Figure 2).

Implications

There is strong evidence in favour of Introducing inoculation policies on top of standard deplatforming strategies. This approach reduces the need for repressive measures which can elicit strong backlash and risk further radicalisation. Additionally inoculation provides transferrable skills and critical thinking that can help individuals develop resistance to other types of disinformation (e.g. health climate change). Importantly inoculation does not require the collaboration of tech companies and platforms.

Future Research

  • What types of anti-extremism inoculation and what population target would be most effective within the Victorian context?
  • What is the optimal inoculation frequency – and how many resources are needed – to prevent the spread of online extremism in the Victorian context?

Team: