London. Last month was of remarkable significance for those of us concerned with public health nutrition on an international scale. This was because of the astonishing new data and judgements published by the Institute for Health Metrics and Evaluation (IHME) in The Lancet of 15 December, presented at London's Royal Society the previous day.

Global burden of disease

Hands on world health, disease, death

Hands on the globe – symbol for the Institute for Health Metrics and Evaluation, whose gigantic study was launched at London's Royal Society

The Institute for Health Metrics and Evaluation (1) was initially funded with $US 105 million by the Gates Foundation. It is based in Seattle within the University of Washington. It has as its mission statement 'accelerating global health progress through sound measurement and accountable science'. It reckons to have its hands on all relevant information concerning world health, as symbolised by the image above on its website. There is more below on the basis for 'sound measurement' and on accountability – like, to whom?

IHME director is Christopher Murray, a 1988 Harvard graduate, previously responsible for the Evidence and Information Cluster as a World Health Organization executive director. Announcing the opening of the institute he said: 'Very simply, health metrics is about figuring out what works and what doesn't work'. The idea, according to a media release at the time, is that 'Murray and his colleagues will be the main folks handing out report cards in the global health field' (2). At WHO he was co-responsible for the Global Burden of Disease (GBD) initiatives. With colleagues he has now developed the GBD concept and methodology into what we may term Big Science, outside the UN system.

The new IHME report is big stuff, no doubt. The findings so far are huge, taking up the whole of a 200+ page special triple issue of The Lancet (3). This is the first time in the 189-year history of The Lancet that an entire issue has been devoted to a single study. A total of 486 authors from over 300 institutions in 50 countries have so far participated. At the joint Lancet/IHME launch at the Royal Society in London, The Lancet editor Richard Horton, and Peter Piot, director of the London School of Hygiene and Tropical Medicine, likened the study, begun in 2007, to the sequencing of the human genome. Richard Horton said: 'Everyone concerned with health – health workers and policy makers, those working in technical agencies (across the UN system), development partners, civil society, and the research community – should use these latest findings to sharpen understanding of trends in disease, injury, and risk'.

On 20 December Bill Gates added in his blog: 'The study is important to the global health community, because it examines which diseases are preventing the world's poor from living healthy and productive lives, and the risk factors that contribute to an increase in those diseases. Only a handful of studies – like the GBD – make use of sophisticated analysis to quantify ill health and permit direct comparison of fatal and non-fatal diseases across time, countries, and populations'.

Some other commentators present at the Royal Society meeting were less confident, One comment was that the report at this stage is simply abstracts of all the work, analyses, assumptions and illuminating insights of the project.

Christopher Murray (left, below), with Alan Lopez (also formerly of WHO, now of the University of Queensland) one of the main authors, highlighted at the Royal Society launch that there are 15 outstanding unpublished meta-analyses. These are the basis for some of the now-published data and recommendations on 187 countries, none separately listed except in terms of crude death rates. Furthermore the nearly 500 collaborators are planning to publish at least 200 papers on their work. So prepare for an onslaught.

Christopher Murray of the Institute for Health Metrics and Evaluation
(left) and (right) Majid Ezzati of IHME and Imperial College, London

I had already had a preview of these results shown by Christopher Murray (above, left) and co-author Majid Ezzati of IHME and Imperial College London (above, right) in Riyadh in September (4). I had also had a clearer view of the analyses of the new global burden of disease figures when I was in Cairo at the beginning of December. Ala Alwan (left, below), the new Director of the WHO Eastern Mediterranean Regional Office with responsibility for 22 countries, has moved on from highlighting in Riyadh the problem of chronic non-communicable diseases (NCDs) in the Middle East and North Africa, to now starting to tackle throughout the region, first a salt reduction strategy, and then trans and saturated fat reduction policies.

Ala Alwan of the WHO Eastern Mediterranean Regional Office (at left) and (at right) Dariush Mozaffarian of the Harvard School of Public Health

In Cairo we had a teleconference input from Dariush Mozaffarian of the Harvard School of Public Health (right, above) from his Boston office, where he displayed the astonishing ranking of causes of death throughout the world, now printed in The Lancet and shown here in Figure 1. At that stage he could not immediately lay his hands on the actual data on the Middle East which Majid Ezzati had highlighted in his Riyadh lecture but the global conclusions were anyway astonishing.

---

Figure 1

The top 20 determinants of global deaths, 2010

Outcome of Institute of Health Metrics and Evaluation methodology

From Figure 2C in (5)

---

Dominance of diet, smoking and pollution

The conclusions above based on IHME methodology involved assessing 235 causes of death, 67 risk factors and clusters of risk factors in 21 regions of the world. These risks and their effects on different diseases, each considered separately, were estimated for 20 age groups, both sexes and 187 countries. These mammoth analyses apparently show that factors directly connected with diet amount to 7 of the top 20 causes of all deaths and disabilities throughout the world. These are, as you can see, in order of IHME importance, alcohol use, diet low in fruits, diet high in sodium, diet low in nuts and seeds, diet low in whole grains, diet low in vegetables, and diet low in seafood omega-3 fatty acids. Add iron deficiency and suboptimal breastfeeding, and the total is 9 out of 20. Then add high blood pressure, high body mass index, high fasting plasma glucose, high childhood underweight, and high total cholesterol, all of which are causally linked with diet, and the total is 14 out of 20.

There are gender differences in the ranking of factors. Females are more prone to putting on weight, and a high body mass index is also positioned as the third highest risk factor for women in the world (after high blood pressure and household air pollution).Women have less of a problem with alcohol but greater disability associated with iron deficiency than men, and girls are also more prone to being underweight in childhood. Nonetheless, the top 20 determinants of the specified causes of death are dominated by dietary factors, and also by smoking and air pollution. So these findings are apparently extremely important to public health nutrition professionals. But are they valid?

Strange judgements on fruits, nuts and seeds

The Lancet paper by Stephen Lim of IHME and many colleagues, part of the new series, which also includes Christopher Murray and Alan Lopez as main authors (5), sets out the basis and approach to defining the dietary and other risk factors. The conclusions are based on new analyses of a very large number of dietary surveys from across the world, using sophisticated statistical manipulations.

These will overwhelm the ordinary reader. Those with some knowledge who are more cautious or even suspicious may also be concerned. For example, on a global basis, diet low in fruit ranks fifth overall, with 5 million deaths attributed every year, and ranks fourth for men, after tobacco smoking, high blood pressure, and alcohol use. It is therefore positioned as more important than even high body mass index – or to be clear, overweight and obesity.

So where does the judgement on fruit come from? As apparent in The Lancet paper, the sources are as yet unpublished meta-analyses involving presumably collating dietary studies and possibly their associations, together with references to two published documents. These are a meta-analysis of the relationship between fruit and also vegetables and coronary heart disease, by Graham MacGregor and his colleagues (6) and analyses made by the World Cancer Research Fund/American Institute for Cancer Research in their 2007 report on dietary factors involved in the prevention of cancer (7).

The relationship between fruit or vegetable intake and coronary heart disease was based on analyses of reduction in risk in a total of 13 cohort studies which – as usual – were based on North American or European studies. They conclude that fruit and vegetables seemed to have separate effects in reducing cardiovascular risk. The data have some biological credibility, since potassium-rich fruit and vegetables have been shown to reduce blood pressure, and there may also be a modest reduction in cholesterol levels through particularly the non-cellulosic components of the non-starch polysaccharide content (8).

Figure 1 shows that the impact of fruit is dominated by its cardiovascular preventive role with only a very modest effect on prevention of cancer. This is just as well. Here I write as a member of the expert panel responsible for the WCRF/AICR report. The general impression members of the panel had at the time from the exhaustive and rigorous statistical analyses of the type of epidemiological literature reckoned to be most reliable, was that the importance of fruit and vegetables, and particularly of fruits, was considerably less than previously thought. Neither fruit nor vegetables rated a judgement of 'convincing' reduction of risk, although diets high in fruits and non-starchy vegetables were judged as probably preventive of cancers of the upper alimentary tract, and notably of the oral cavity and oesophagus.

Whatever the additional as-yet unpublished meta-analyses may show, the positioning of diet low in fruit as number 4 in the list, is surely strange. The remarkable impact of excess weight has been repeatedly highlighted in practically every analysis ever made, whether involving cohort studies, physiological experiments, detailed intervention trials, or long term analyses of mortality. So I was bound to be suspicious of the analytical framework. But the position of fruit is not the only startling judgement. Diet low in nuts and seeds is judged to be the 12th most important risk factor in the world for ill-health, disease and premature death, with 2 million deaths attributed every year. Really? This is startling. Although previous analyses and collations of data by WHO expert groups have considered some of the valuable nutrients contained in nuts such as omega-3 fatty acids, suddenly to identify out of the blue a new group of foods as in effect causally related to huge burdens of disease and premature mortality is astonishing. What is going on?

Can methodology be the problem?

The results of epidemiological analyses are to a large extent determined by methodology. This includes the assumptions made in framing the analyses and the relative weight given to different types of study analysed. This does not make the analyses and their results invalid, of course, but does make them always open to appraisal.

In the case of the IHME study with its Lancet results, special reliance is often placed on the types of study to which the WCRF/AICR panel initially gave prominence, which are prospective cohort studies. This is despite all the well-known problems of this type of study, such as ascertaining diets accurately and of being sure of reliable data on the dietary patterns of individuals over many years of follow-up. Also there may be hosts of interactive factors and confounding variables, some not identified as such, which are not always taken into account. Plus cohort studies, being very expensive, are almost all carried out in high-income contexts.

Concerned with issues such as these, WCRF/AICR panel members eventually took a broader view. We were supported in our somewhat jaundiced view of cohort studies by analyses showing that the combination of errors in dietary methodology and the self- reporting, or family member's reporting (often in practice under-reporting), of conditions such as diabetes and hypertension, even coronary heart disease, and the precise nature of a cancer, meant that apparently significant positive (or negative) results could emerge which actually were dud – a combination of errors in the data being used. If confounding factors are not rigorously eliminated, apparent levels of risk are liable to be exaggerated, and actual null effects seen as significant. Without robust and causally relevant biological markers, valid end-points, and meticulous analysis of confounding factors, it is unwise to jump from any apparently robust association of some aspect of diet with some apparent outcome in cohort studies, to proposing new nutrition and health policies, let alone putting such policies into practice.

The conclusions and judgements of the WCRF/AICR report took into account biological perspectives. The panel also examined physiological and short term intervention studies. These are often more robust and reliable than relatively superficial studies of vast numbers of people, which devotees of cohort studies all too readily believe are valid despite all the potential errors and the further dangers in sophisticated mathematical manipulations .

So I examined some of the methodology evident in the IHME report. I discovered that each factor is seen as totally relevant alone, without regard to any interactivity. Thus, in the top 20 risk factors, high blood pressure is isolated, and seen as far more important than diets high in sodium. The difference may reflect the effects of variations in total fat, vegetables and fruit intakes on blood pressure as well as the impact of salt, but the calculations also depend in part on what the authors perceive to be the optimum intakes of all these nutrients and foods. As another example, high body mass indices are positioned as being unrelated to blood pressure, even though the authors acknowledge that this is not true in reality!

Some of the methods and assumptions made are surprising. Data are not age-standardised as is now normal in UN and other authoritative work. The impact of disability is not discounted over time. Reliance is placed on lay people's responses to questionnaires about their degree of disability, and causes of death specified by relatives are used rather than medical reports with autopsy validations where possible.

These were just a few points that were immediately apparent. But did any of this really matter? In papers included in The Lancet special issue, Christopher Murray and his colleagues display brilliantly dazzling analyses, checking systems, data bases used, computations of confidence limits, heroic extrapolations, and mind-bending adjustments. I couldn't help feeling that only very competent mathematical statisticians could cope with these gyrations. Only those who are exceptionally qualified and conversant with the literature, particularly on how to estimate health statistics in poor countries with rudimentary disease analyses are likely to be seen as credible challengers to the IHME's multiple new approaches.

Statistics, validity, and the WHO view

Richard Horton of The Lancet (left), saw the results as comparable to those of the Human Genome Project. Richard Peto (right) was less confident

In this state of bewilderment at the IHME analyses and judgements, I listened in to the web-cast from the Royal Society. Was I alone? I had just plugged in when Richard Peto, the mathematically brilliant epidemiologist from Oxford who spent perhaps 40 years of his life working with Richard Doll on smoking and use of tobacco, was asked to give his views.

Well, it turns out that I am not alone. In quiet and polite tones Richard Peto said he was very impressed by the enormous amount of work done, but he clearly had severe doubts about the validity of the analyses, conclusions and judgements. He also indicated that he was exceptionally sceptical about the methodology chosen, and its implications for the relative importance of risk factors.

He highlighted the extreme difficulties he, Richard Doll and their colleagues had with their analyses of smoking, despite its relatively clear-cut nature and being able to have reasonable confidence in the reporting of smoking or non-smoking. By contrast, as is well known to us in the field, reporting of dietary intake, especially in studies involving very large numbers of people, is precarious.

In response, Majid Ezzati accepted these criticisms as reasonable and said there was need to keep working on the issues. By contrast, Christopher Murray in a comment no doubt meant to be amusing, indicated that one thing he had learned from the meeting was to beware of Richard Peto at the Royal Society!

Christine Kaseba-Sati took part in discussion. While being the First Lady of Zambia, she identified herself first as an obstetrician and gynaecologist. She felt that the IHME work, while positioned as global, had limited relevance to the world she knows. She was unimpressed by the crude data and the failure to look at actual health burdens in low-income countries. Thus, she doubted that the well-known but often unreported huge disabilities of women after delivery in Africa had been considered. Usually the physical and mental trauma that followed unsatisfactory childbirth is kept secret and therefore unreported. Hundreds of thousands if not millions of women require restorative surgery to cope with their fistulae and sometimes double incontinence following difficult childbirth. She also noted that very few contributors to the IHME study are Africans working in sub-Saharan Africa.

Dissent from Christine Kaseba-Sata (left) of Zambia, and distancing by Margaret Chan (right), director-general of the World Health Organization

On the grapevine I had heard that WHO had withdrawn from being associated with the study. However, Richard Horton stated that WHO welcomed the analyses and would hold a special meeting with UNICEF and others in Geneva in February to look at its policy significance. Colin Mathers, the key official at WHO headquarters in Geneva dealing with global burden estimates, was in the Royal Society audience, as pointed out by Richard Horton, but perhaps significantly and very surprisingly was not invited to contribute to the discussion and did not do so.

Richard Horton said that director-general of WHO Margaret Chan welcomed the report in her written piece for The Lancet special issue (9) But careful reading tells you a lot. She noted that many areas of the analyses 'differ substantially from analyses by WHO and other UN entities'. The IHME study' constitutes an important milestone that can be used as a basis for further work to address these differences, which are often based on choice of statistical techniques rather than realities on the ground'. In other words, being interpreted, don't trust the IHME study or its results.

The final part of the Royal Society meeting was concerned with whether the IMHE study and its judgements are going to change the world and lead to major policy innovations. It soon became obvious that many of us who are not engaged in or otherwise committed to the study, regard its data sets and presentation as essentially a series of thus-far locked black boxes. There were repeated calls for transparency and publication of the meta-analyses. It was also stated – quite rightly – that no country could trust data and judgements of the study until its governments and policy-makers knew for sure where these actually were derived from and how they were applied to its own situation and circumstances. Indeed, it would be best if all 187 countries had their own version presented vividly, with the basis of the analyses being made clear.

The need to take care

Academics associated with the IHMR project at the Royal Society seemed to have little feeling for the issues at national level especially in the global South, and emphasised the need for better health care. Richard Horton told everybody about the new United Nations call made just days before, setting out a new global standard, such that every country has universal health care as the norm. The immediate relevance of this idealistic but currently unrealistic call to the IHME study, was unclear. Perhaps it implied focus on secondary or even tertiary medical prevention – that is, treating the risk factors or the diseases, rather than going for the real causes of the diseases and focusing on primordial and primary prevention. .

So what now? The IMHE study is not 'the answer'. How useful its analyses and judgements may be, remains to be seen. Richard Peto implied that using the IHME work and results as a basis for policy-making would be naïve. He has told me in previous discussions that he regards obesity as intractable and far more difficult to address compared with his main professional interest of smoking.

In the final panel discussion at the Royal Society several people emphasised that a wholly different approach is needed to tackle the public health problems seemingly revealed by these findings. There is good reason to be concerned about the validity of the IHME techniques and results, although so far among recognised authorities, only Richard Peto seems to be challenging them.

There is though an even bigger issue here. It remains unclear how the results of any gargantuan study of the type produced by IHME, however awesome, can be a useful basis for policies and actions that could improve public health. The study is attempting to identify the major factors involved in death and disability. This is potentially really useful but is only the first step in policy-making even if the results are valid. In the case of IHME, its approach overlooks social and economic determinants of health and disease. Thus, poverty is a major basic driver of many of the identified risk factors, and itself needs to be faced and addressed. The feasibility of acting, the type of action needed, and the cost-effectiveness of different approaches also all need to be well understood, before trying to do business with government ministers of finance, trade or agriculture, who are always more influential than ministers of health.

Huge claims are being made, based on the evidence as accumulated and organised by the colossal and powerful IHME study. My conclusion, and that of others, is that caution is needed. Decades of careful analyses by WHO should not be discarded. All forms of evidence need to be taken into consideration in careful evaluations of the role of dietary patterns and food and nutrition.

It is also true though, that the IHME initiative, however obscure its scientific methodology and opaque its presentation, puts all public health nutrition professionals on the spot. The role of nutrition in public health is getting an ever higher profile. The opportunity to transform policy and initiate actions to reduce the diet-related health burden in both rich and poor countries has never been greater. Are we ready?

References

1. Institute for Health Metrics and Evaluation. Home page available at http://www.healthmetricsandevaluation.org/
2. Paulson T. $105 million Gates' gift helps start global health center. Seattle PI Reporter, 4 June 2007. http://www.seattlepi.com/local/article/105-million-Gates-gift-helps-start-global-1239473.php#ixzz2H3FMx2ja.
3. Lozano R, Naghavi W, Foreman K, et al. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010, The Lancet 2012, 380, 9859, 2095-2128.
4. James WPT. As I see it.Website of the World Public Health Nutrition Association, October 2012. Access pdf here.
5. Lim S, Vos T, Flaxman A, et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. The Lancet 2012; 380, 9859, 2224-2260.
6. He F, Nowson C, Lucas M, MacGregor G. Increased consumption of fruit and vegetables is related to a reduced risk of coronary heart disease. Journal of Human Hypertension 2007, 21:717-728.
7. World Cancer Research Fund/American Institute of Cancer Research. Food, Nutrition, Physical Activity, and the Prevention of Cancer: a Global Perspective. Washington DC: AICR,2007.
8. Appel L, Moore T, Obarzanek E, et al. A clinical trial of the effectsof dietary patterns on blood pressure. New England Journal of Medicine 1997; 336, 1117-1124.
9. Chan M. From new estimates to better data. The Lancet 2012, 380, 9859, 2054.