Why were the exit polls wrong? | Commentary

The story so far: Ahead of the June 4 election results, several exit polls are predicting that the Bharatiya Janata Party-led National Democratic Alliance will return to power, with the BJP alone set to win more than 300 seats. According to one pollster, Chanakya predicts 400 seats today In the case of NDA, it is plus or minus 15. Axis MyIndia said the NDA would win an average of 381 seats.As the results show, all of these polls were way off.

What was your prediction for the vote share?

CSDS-Loknitty’s post-poll projected the NDA to win 46% of the vote and the INDIA bloc (excluding the Left, the Trinamool Congress in West Bengal, and the AAP in Punjab) to win 35%. The margin of error was 3.08 percentage points. The results showed the NDA winning 292 seats (43.63% of the vote) and the INDIA bloc winning 205 seats (excluding the Trinamool Congress, which won 29 seats), with 37% of the vote. CSDS-Loknitty did not predict the number of seats for the coalition, but predicted that the NDA would return with a majority. The vote share was within the margin of error, but was 2.5 percentage points higher in favour of the NDA.

Axis My India predicted 47% for the NDA and 39% for the INDIA bloc, but the actual results showed that the predictions overestimated the NDA’s vote share beyond the margin of error. C-Voter predicted 353 seats for the NDA vs 383, with 45.3% of the vote and 38.9% for the BJP alone, which was 2.3 percentage points higher than the party’s actual vote share of 36.56%. The INDIA bloc figure was also about 2.4 percentage points lower than the actual figure. While the vote share was within the margin of error nationally, the seat counts were off by a wide margin in several states.

What are exit polls and how are they different from opinion polls?

Opinion polls are sample surveys in which a section of voters is randomly selected and interviewed about their choice of parties and candidates. These surveys can be conducted face-to-face or through devices like telephone polls. Exit polls ask voters about their choices outside the polling station, immediately after they have exercised their right to vote. Some pollsters prefer “post-poll surveys” which are conducted in the voter’s home after they have exercised their right to vote. The CSDS-Lokniti poll is a “post-poll survey”. Other surveys such as Axis MyIndia were “exit polls”.

Did the exit poll numbers get inaccurate because of the survey methodology?

For exit polls to be accurate, certain factors need to be kept in mind such as the sample size of the survey, the sample selection process, the method of conducting the survey, and weighting of the sample according to the population estimates.

The size of the sample must be representative, but as long as it is large enough to statistically predict the winner, it doesn’t matter how large the sample is. If the sample is selected randomly and the sample size is enough to accurately predict the likelihood of a particular candidate receiving more than 40-45% of the vote (as is typically the case in Indian elections), then even a representative sample of about 20,000+ respondents is enough to predict the winner in a country with a voting population of nearly 1 billion. Larger surveys with more than 20,000 respondents, or even hundreds of thousands of respondents, can be conducted, but the key to tracking the mood of the electorate is to get a good representative.

CSDS Lokniti’s sample size was a total of 19,663 across 23 states and 193 constituencies, while Axis MyIndia’s sample size was 582,574 across 543 constituencies. However, the former came up with vote share projections within the margin of error, while the latter did not.

To be properly representative, the sample should be selected randomly (to avoid bias) and also stratified (to ensure that no section of the population is overlooked).The most ideal way to select a random yet stratified sample is to use the electoral roll to identify the respondents.

Once sampling is done and the list of respondents is identified, it needs to be weighted based on the representativeness of sections of the population (proportion of women, Dalits, minorities, majority population, urban and rural voters). After a representative sample is prepared, the enumerators conduct the interviews. Ideally, face-to-face interviews, in the same language as the respondents, are more effective.

Because many respondents will be uncomfortable telling pollsters their voting choice, questions should be constructed in such a way that their voting choice can be ascertained thereby, or mechanisms should be provided in such a way that respondents can record their voting choice without openly revealing it to the pollster.

Do respondents reveal their choices?

Many respondents, especially those from marginalized groups, are likely to either not reveal their voting preferences or to require some level of trust with the pollster before revealing their choice. Pollsters who got it wrong in this election may have not sampled marginalized voters well enough, may not have trusted the pollster to reveal the right answer, or may have been misled by the respondents.

Once the information is collected, other steps must be followed. How should the pollster allocate the “undecided” options? For example, to the question “Who will you vote for?”, there might be answers like “Don’t know,” “No opinion,” “No answer,” “Don’t say.” Should the pollster omit them? Or should they weight them based on the proportion of the options that were decided? These are important considerations for the pollster.

What is the process after the survey is conducted?

Once the survey is done, the results need to be cross-checked with estimated demographic information. If out of a population of 100, there are 12 Dalit respondents and their choices are recorded and the actual percentage of Dalits in the population is 15, then a uniform weighting of 15 can be done based on the 12 respondents. But if only 39 women out of a population of 100 are interviewed, then extrapolating the opinions of the 39 to the opinions of 48 women (the actual estimate of the female population) is problematic as women do not vote as a single category. This could be one of the reasons why the Axis MyIndia poll got its estimates wrong in many states. The ratio of men to women in the poll sample was 69:31.

Most of the pollsters affiliated with TV stations used their surveys to project seat shares, but CSDS Lokniti did not. The conversion from vote share to seat share can be done in different ways. The most common way is to assess the change in vote share of a particular party since the last election in a state, or more precisely in a particular region, or even more precisely in a particular constituency if sample surveys allow. The change in support or opposition to an opponent can be the basis for whether an incumbent will be re-elected from a particular constituency or whether a party will be able to retain a certain number of seats in a region of the state or in the entire state.

Veteran election scholar and media personality Prannoy Roy points out in his book: verdictAccording to the Opposition Unity Index, co-authored by Dorab R. Sopariwala, some pollsters use past poll fluctuations and the “Opposition Unity Index” to gauge a particular candidate’s margin of victory and then project his seat share from the vote share.

There are other ways to assess this, but converting vote shares into seats also requires pollsters to be aware of the political dynamics of a particular state or region, and what happened in the most recent elections there.

For example, a party’s victory in the Assembly elections in a state creates a so-called “honeymoon effect” that affects the Lok Sabha elections if the Lok Sabha elections are held just a few months after the Assembly elections. In such cases, it is more appropriate to calculate party fate based on the Assembly elections in that state, rather than the previous Lok Sabha elections.

Not a single pollster affiliated with a major TV network was able to correctly predict the number of seats won, and because none of them have revealed what they consider their “secret sauce” conversion process, it’s hard to know why they got it wrong.

Is it hard to predict a close election?

It is clear that pollsters in India mostly predict the winners and seat share of elections with realistic figures when the election results are decisive. When the election results are close, like the current Lok Sabha elections, pollsters tend to give little accuracy in predicting the vote share and seat share.

Whether a polling agency has conducted a proper survey will be evident from its methodology: the sample size, the survey methodology, the representativeness of the sample, and the margin of error built in. If a survey does not reveal these things, it cannot be considered serious enough.