Rating Disinformation Risk: The GDI approach to news sites

Rating Disinformation Risk: The GDI approach to news sites

The Global Disinformation Index (GDI) was founded with the objective of measuring the risk of disinformation – a rating system for the world’s media domains. Today we share with you an overview of the portion of our rating system that looks deeply into news sites and their risks to disinform.

The GDI ratings provide a multidimensional assessment of disinformation that goes beyond yes/no scores.

The ratings are seen as critical inputs for advertisers and ad tech companies to use in their brand safety decisions about where their ad spends end up on news sites. The GDI ratings can be used to better control the automatic advertising placement that provides vital funding to disinformation actors in it for the clicks.

We set out to build an index that was granular, methodologically sound, and based on best practices in index design.

The GDI rating system measures the risk of disinformation being present on a website, not whether a specific news domain is actually carrying disinformation.

The GDI risk assessment takes a ‘whole-of-site’ approach to understand this risk, rather than assigning ratings to individual articles.

The assessment is based on structural, contextual, operational and content signals that can be used to accurately flag disinformation. The GDI methodology report, available here, outlines all these aspects in more detail.

The index’s framework is based on a combination of 15 indicators that assess disinformation risk arising from these four areas.

Each pillar of the rating measures a unique disinformation risk factor:

  • The ‘structure’ pillar measures the risk based on site structural elements using AI and observable data. 
  • The ‘content’ pillar measures risk based on the narrative and presentational elements of a news site’s articles, using observable data.
  • The operations pillar measures risk factors related to organisational and journalistic standards and uses observable data. These are directly drawn from the Journalism Trust Initiative principles. 
  • The context pillar measures risks based on a news site’s past behaviour, drawing on perceptions-based data to understand risks over time. (see below)

The scoring scale for the index ranges from 0 (maximum risk of disinformation) to 100 (minimum risk of disinformation). A domain’s score on the index is the mean of the scores earned across all four of the pillars.  These scores are based on the following indicators per pillar.

Overview of Pillar Scores

‘Structure’ Pillar

  • 23 metadata signals.
  • Each signal scored on a binary scale – 0 (high) or 100 (low).

‘Content’ Pillar

  • Ten news articles assessed per domain.
  • Each article assessed on six indicators on ordinal scale of 0, 25, 50, 75, and 100.
  • Domain level score for each indicator is the average score across all ten articles assessed.
  • Final score for pillar is the average score across all six indicators – can range between 0 and 100.

‘Operations’ Pillar

  • Five indicators assessed at the domain level.
  • Each indicator scored on a 0 to 100 scale.
  • Final score for pillar is the average score across all five indicators.

‘Context’ Pillar

  • Four indicators.
  • Each indicator assessed by multiple experts on a five point Likert scale that is converted to 0 – 100 scale. 
  • Domain level score per indicator is the average score across all experts reviewing each domain.
  • Final pillar score is the average score across all four indicators.

Based on these scores, the index classifies sites into five distinct risk categories. These categories are mathematically set by using the mean of the scores and then taking the standard deviation from it.

The GDI index framework has been designed to ensure it is objective, statistically robust, and a realistic depiction of the disinformation risk encountered by news domains. The framework is grounded in and adheres to our core principles of neutrality, independence and transparency. 

The GDI framework also draws on the lessons we learned from doing a methodology landscape report in May that synthesised existing efforts to study and assess disinformation.

The GDI rating is one of the first multi-dimensional and scientific assessments of the disinformation risks faced by news domains worldwide. 

The GDI index framework and associated indicators will be further fine tuned based on data collected from media market assessments in 2020 and beyond. We also are engaging with a range of stakeholders (media sites, ad tech companies, civil society organisations, etc.) to get feedback and help us improve our methodology. It has been a productive process that we welcome.

All this effort is to ensure that GDI ratings become the go-to assessment of disinformation risks for news sites. Please join us in these efforts.

Close Menu