Opinion polls play prominent role in Kenyan politics. But what can these polls actually tell us? That depends on how the poll was carried out. In this blog, co-editor Nic Cheeseman argues that polling methods are frequently misunderstood, overlooked, or studiously ignored by the politicians and media outlets who quote them, and explores the consequences of this use and abuse.
Over the past twenty years opinion polls have gone from being a rarity in Africa to an established part of the political process. Nowhere has this transformation been more profound than in Kenya, where opinion polls are run more often and more professionally than almost anywhere else on the continent. But many commentators still misunderstand the methodology behind opinion polls and they continue to be used and abused in equal measure.
On Saturday 14 September, The Standard published an opinion poll on devolution and the campaign for a referendum as its front-page story. This is a critical issue of national importance, and deserves to be treated with the utmost care. Despite this, there were a number of limitations with the way that the poll was done. First, the sample was very small. Infotrak Harris only talked to 1,000 people, leaving a large margin of error of +/- 3%. The industry gold standard – as employed by the Afrobarometer and some other survey companies – is over 2,000, which gives a smaller margin of error of +/- 2%.
Second, the poll was conducted via “computer assisted telephone interviews”. There are a number of problems with this. The most obvious is that it excludes Kenyans too poor to own a mobile phone, and those whose phone is not registered. This group is smaller in Kenya than in many other African countries, but it is still a significant minority. Not just a small sample then, but also a potentially unrepresentative one.
Personally, I also have suspicions about how honest people are in phone polls. Of course, there are some sensitive questions that people might answer more honestly if they have the greater anonymity of speaking over the phone. But when it comes to the factual questions about respondents’ lives – which are critical if you want to actually understand why people hold the opinions that they do – phone interviews have the potential to be significantly less reliable. After all, it is harder to pretend that you are wealthy, or that you are not suffering as a result of a lack of public services, if someone is sitting in your front room.
I want to make it clear at this point that I am not suggesting that Infotrak Harris were at fault for doing a bad poll. I have no doubt that the blame lies not with Infotrak but elsewhere. My best guess would be that Infotrak had previously informed The Standard that a larger face-to-face poll would generate more reliable results, but that the paper went with the “quick and dirty” option because it was cheaper. I also suspect that The Standard’s editors were rushing because they had become aware that Ipsos-Synovate had already sold the results of a similar opinion poll to the Daily Nation, and didn’t want to be outdone.
The use and abuse of polls
Infotrak Harris can also not be blamed for how The Standard used their poll figures. The headline that ran with the poll numbers appeared to tell a simple story: “Why more Kenyans oppose referendum: New survey shows that voters see move by CORD as a waste of time, a gimmick by politicians to seek relevance and a scheme to derail Jubilee”. The tone of an accompanying online article, which started by pointing out the large majorities against a referendum in Central Province and the Rift Valley, made it sound as if there was little appetite in Kenya for a referendum.
But a closer inspection of the poll results told a different story. In actual fact, fewer Kenyans oppose the referendum because it is a “waste of money” (14.1%) than support it because it will “address insecurity” (15.2%) and may “reduce the cost of living” (17.7%). The Standard also failed to ask why so many people who oppose the Okoa Kenya referendum do not appear to know why (13%).
Despite the tone of the headlines, the poll also found that a surprisingly large minority of Kenyans (46%) were in favour of a referendum. With a margin of error of +/-3%, this means that the “pro-referendum” camp could be as high as 49%, and the “anti-referendum camp” could be as low as 51%. This is particularly noteworthy given that the graphic The Standard ran on its front page suggests that the newspaper counted the 1.4% of Kenyans who “refused to answer” as being opposed to the Okoa Kenya referendum, instead of removing them from the calculation as they should have done. Given that many commentators laughed at CORD when they first raised the idea of a referendum, this suggests a campaign that is gathering momentum, not one that has fallen at the first hurdle.
Moreover, while a narrow majority of Kenyans were against the referendum proposals put forward by CORD, the poll found that the Pesa Mashinani initiative put forward by Governors was more popular – although from the write up, it is not entirely clear exactly how many Kenyans would back this initiative. Taken together, this suggests that many Kenyans – perhaps even a majority – are sympathetic to the idea of a referendum, but are reluctant to support the CORD campaign for party political reasons.
Why did the editors of The Standard choose to frame the poll findings in this way? No one can say for sure, but it is hardly surprising that the paper was keen to pour water over Raila Odinga’s most recent initiative given the extent to which it has leaned towards the Jubilee Alliance since the last election. As a friend of mine quipped the last time I was in Nairobi, “if they lean any more, they’ll fall head over heels in love”.
Using polls to evaluate Kenyan politics
If representatives of The Standard were to reply, they would no doubt point out that I write a column for The Nation, and that I have commissioned a number of opinion polls from Ipsos-Synovate, so I can hardly claim to be an impartial observer. This is true, but I started writing on this subject back in 2005, before of any of this transpired. In an article written with Daniel Branch and published in the journal African Affairs, we noted two worrying aspects of the way that opinion polls were being used.
The first was the tendency for polls that were only conducted in urban areas to be reported as national findings. The obvious impact of this was to marginalize the opinions of rural dwellers. It also led commentators to spectacularly misunderstand political trends. For example, in July 2004, the Kenya Times ran the headline “Uhuru floors Kibaki in popularity rating”. The accompanying article predicted that Kibaki was about to be overtaken by a wave of support for Uhuru Kenyatta. But this was not the right lesson to take from the polls.
The results that the Kenya Times was referring to were urban only and so tended to include areas where Kenyatta was very popular, such as Nairobi, and to exclude some of Kibaki’s strongholds. Nationwide polls conducted around the same time placed Kenyatta second or third at best. Thankfully, the problem of urban only has largely gone away, although there is now the threat that it will be replaced by the extensive use of phone polls – which, like urban polls, are attractive to newspapers because they are cheaper.
We also pointed out the importance of thinking about how the timing of a poll can influence its findings, and the need to keep in mind how quickly public opinion can change. The political attitudes of Kenyans following the 2002 elections illustrate this point perfectly. In early 2003, Kenyans were the most optimistic people on earth. A remarkable 87% of urban residents thought that the next year would be better than the last.
The post-election euphoria didn’t last. By March 2014, the figure was down to 40%, as NaRC fell apart, undermining the dream that a rainbow alliance would transform Kenyan politics. Just a few months later, levels of optimism had fallen to just 29%. It is obvious that understanding how the political context in Kenya changed during this period is critical if we are to accurately assess the public mood in Kenya.
What is probably less obvious is that it is also matters for how we interpret every other finding in the opinion poll. More optimistic people are more likely to give positive answers to a range of different questions, from the level of corruption to how much they trust the government. An analysis of Kenya’s political institutions based on poll findings from March 2003 would therefore have reached radically different conclusions to one based on poll findings from just a few months later.
As with urban-only polls, awareness of this issue has improved significantly over the last few years. On the whole, the way that newspapers report on opinion polls has become much more sensitive to time and to context. The Daily Nation, for example, has a policy of not publishing opinion poll results after the period in which the poll was conducted. This is part of a wider trend in which newspapers have become more responsible in the way that they present survey results.
I remember being at an early press release of opinion poll findings by what was then called the Steadman Research Services, in which Tom Wolf, the undisputed talisman of Kenyan polling, implored journalists to use their newspapers to educate their readers about the methodology behind the polls. At the time, few did, but as opinion polls have become part of the political scene this has changed.
To be fair to the Standard, it has played a part in this improvement. For all of my concerns, the coverage run on the 14 September clearly stated the size of the poll and the margin of error. It also contained some insightful points by the head of Infotrak, Angela Ambitho, and respected academic Karuti Kanyinga.
Poll controversies
Of course, this does not mean that polls are always discussed in a responsible way. Not long ago the Standard published a commentary by Paul Kipkemei in which he attacked an opinion poll that found that 46% of Kenyans planned to vote for Raila Odinga. According to Kipkemei, this figure was “at best, a product of imagination”. Ultimately, Odinga secured almost 44% of valid votes, so the poll was not far off. This was not particularly surprising, because Kipkemei’s column was not so much a critique of the poll’s methodology, but of the fact that its findings did not fit with his own analysis of how the election would play out. Somewhat ironically, the headline chosen by the editors of The Standard was “Infotrack opinion poll results suspect, and may not be trusted in future”.
Such blanket critiques, which call into question not only specific opinion polls but the very principle of opinion polls, have unfortunately caused considerable confusion both within the media and the wider public. But one of the most interesting aspects of the way that opinion polling has developed in Kenya is that some of the people who criticize surveys most vocally in public rely on them most heavily in private. In late 2006, the boss of one of the main polling companies told me that seven MPs had approached his firm to run surveys in their constituencies ahead of the 2007 elections. This had surprised him, because four of them had previously taken to the media to attack unfavorable polls conducted by his company.
I asked him whether he thought the politicians had paid any attention to the poll findings. He replied “I can’t say for sure, but two dropped out of the race after finding out how far behind they were to a rival. One of them told me that he had thrown in the towel because ‘it hurts to swallow your pride, but it is cheaper than losing an election in Kenya’”.
Opinion polls: sometimes praised, sometimes rejected, rarely ignored.
