A Flawed Measure of Democracy

EVERY YEAR IN March, the V-Dem Institute, a thinktank that ‘measures democracy’ and its attributes, releases its annual report on the state of democracy in the world. In recent years, India has seen its rankings collapse to the point that many countries with a far more dubious record appear to be more democratic, at least going by the rankings and scores released in recent reports. In the 2024 report, India’s rank on V-Dem’s Liberal Democracy Index (LDI) is a lowly 104 out of 179 countries ranked. In terms of its status as an electoral democracy, El Salvador, Honduras and Sierra Leone rank better than India, never mind the observable condition of democracy in those countries. Are these measures worth the paper they are printed on?

The idea of measuring democracy and making distinctions between democracies themselves is of recent provenance. During the Cold War, the dividing line between democracy and non-democracy was simple enough. In a democracy, people had the freedom of expression, of association and of choosing their representatives. This was sufficient to demarcate countries where freedom prevailed and where it did not. There was the ‘Evil Empire’ or the USSR and its allies and the ‘Free World’ where

freedom prevailed. After the Cold War, this stark distinction did not

serve any ideological and political purposes. For example, how to make distinctions within the set of countries that professed to be democracies but would not adhere to ‘universal values’?

Making those finer distinctions required a conceptual re-orientation: democracy had to be ‘measured’ if one were to move away from the binary of democracy versus non-democracy. Nearly a quarter-century after the end of the Cold War in 1991, the Varieties of Democracy (or V-Dem) project was launched in 2014. Founded by the political scientist Staffan Lindberg at the University of Gothenburg in Sweden, the V-Dem Institute invented a series of indices that made ever finer distinctions, allowing for a sliding scale to classify democracies.

The project’s classifications were given a top billing in academia across the world until recently and have especially found favour in India’s liberal sector: elite academics, civil society activists, powerful law­yers and NGOs, all of whom espouse a particular type of politics.

But a closer look at the project reveals serious problems. For starters, and by Lindberg’s own admission in a recent paper, democracy is a contested concept. A recent paper co-authored by him noted, “Democracy is a contested concept, and indices measuring different notions of democracy capture different as­pects of political systems. For this and other reasons, including measurement error, democracy measures may disagree on lev­els of democracy and trends in democratic backsliding, both for individual countries and in the global aggregate.” (Conceptual and Measurement Issues in Assessing Democratic Backsliding in PS: Political Science and Politics, published online, 2024: 1-16). There are other problems as well, such as bias of experts who are consulted by V-Dem. But none of this humility is visible in the annual V-Dem reports where thriving democracies like India are described as “electoral autocracies” or even worse.

Methodologically, the problem is three-fold. One, what may be a democracy for one set of scholars may not be a democracy for another set. As a result, there is a divergence between what is a mere academic debate and the real world. Here, power equations enter the scene: who gets to decide whether a country is a democ­racy or a non-democracy? The people who live in a particular country or those ‘measuring’ it and their local collaborators? Until very recently, what V-Dem said was the gold standard in measurement of democracy: from multilateral agen­cies to powerful Western governments, what V-Dem said was correct. That is, until two scholars, Andrew Little and Anne Meng, pointed out serious flaws in the idea of measuring democracy and democratic backsliding.

The second problem is integral to the sliding scale measurement. V-Dem has a series of indices to classify and measure democratic behaviour on an annual basis. For example, there is the Liberal Democracy Index (LDI), a composite of 71 indicators that are captured in the Electoral Democracy Index (EDI) and the Liberal Component Index (LCI). There are other indices as well. But here’s the problem: in each democra­cy, one cannot assign equal weights to these components. Their relative importance is due to a combination of factors such as the innate preferences of the people of that country, those of its elite and its political history. To give a hypothetical example, imagine a country that is a monarchy but has a free press and compare it with a country that is a robust democracy with regu­lar, free and fair elections, but a less than perfect freedom of the press. How does one compare the two countries? Which one is more acceptable and which one less so? And on what grounds? The V-Dem answer would be a single rank and a score based on a combination of indicators. But beyond a point, this ‘measure­ment’ is meaningless as it does not reflect local preferences and institutional history. There are many such examples that can be imagined given the large set of combinations that are possible in the world’s polities.

The third problem—and one that V-Dem is at pains to deny— is the very process of measurement of democracy. In a prefatory note, the 2024 report notes that V-Dem has 4,200 country experts/ coders from 185 countries. The average time spent by a coder/ex­pert in a country is 39 years and the mean of the expert coder is 47 years. 73 per cent of the coders have a PhD. These are interesting statistics but one does not get to know the names and affiliations of individual coders/experts for each country from the report. V-Dem says that it takes care to crease out bias among experts. But the exact procedure by which that is done is not clear. In such a situation—and especially in the case of India—the danger of coders/experts being biased in a particular direction is almost certain: there is no way to explain India’s consistent poor rank­ings except for expert bias. Never mind the fact that after 1977, India has rarely missed holding an election on time except under extraordinary circumstances when the Election Commission of India was unable to do so. (Punjab in the 1980s and Jammu and Kashmir recently are examples of such conditions).

Somewhere these inconsistencies are too glaring to be ignored. On pages 60-61 of the report, the history of regimes of the world by country-year are plotted in neat graphs. V-Dem’s methodology of Episodes of Regime Transformation (ERT) allows the graphical representation of a country’s political his­tory over time. In its own words, the report says that, “the ERT method works by first identifying small annual changes (>0.01 on the 0-1 scale) in a country’s level of democracy measured by V-Dem’s Electoral Democracy Index (EDI). If they accumulate within several years to substantial (>0.1), it is an episode.”

In India’s case, this leads to incongruous results. In the years 1975 and 1976, India was an electoral autocracy while in 1977 it was in an “autocratic grey zone”. The same history of sorts has been repeated in the second decade of the 21st century: India was in a “democratic grey zone” from 2015 to 2016; the “autocratic grey zone” from 2017 to 2018 and it transformed into an electoral autocracy from 2019 onward. Anyone who has even a perfuncto­ry understanding of the Emergency will know that all freedoms were extinguished in that period and even the Supreme Court said that in Emergency there was no recourse to the law for a citi­zen (in the infamous ADM Jabalpur vs Shukla case). To hold that India is in the same zone now is to either betray gross ignorance of facts or, as is more likely, display extreme intellectual imbecil­ity. And yet, that is exactly what the V-Dem index ends up doing in an ‘objective’ manner by constructing an index that is made on the basis of expert judgement. Stronger words can be used to describe the V-Dem project but for now it is obvious that the idea behind it is not an intellectual exercise but an ideological one. The ruse is for everyone to see.

Andrew LittleAnne MengEuropeanIndian democracymeasurementstandardisationV-Dem