How significance tests are misused in climate science [PDF]

12 Nov 2010 - Guest post by Dr Maarten H. P. Ambaum from the Department of Meteorology, University of Reading, U.K. Clim

2 downloads 14 Views 608KB Size

Recommend Stories


How liberal are citizenship tests?
Goodbyes are only for those who love with their eyes. Because for those who love with heart and soul

PDF Review Tests of Significance
Don't watch the clock, do what it does. Keep Going. Sam Levenson

Forecasting without significance tests?
I tried to make sense of the Four Books, until love arrived, and it all became a single syllable. Yunus

how referendum campaign leaflets misused statistics
In the end only three things matter: how much you loved, how gently you lived, and how gracefully you

How many atoms are in chloride [PDF]
This tutorial introduces atoms in chemistry. This module explores Properties, sources and uses of the element strontium. Recall that the simple cubic lattice has large interstitial sites between each 8 atoms. Sodium chloride / ˌ s oʊ d i ə m ˈ k

PDF How Emotions Are Made
So many books, so little time. Frank Zappa

PdF How Emotions Are Made
Ask yourself: Do I believe that everything is meant to be, or do I think that things just tend to happen

PdF How Emotions Are Made
Ask yourself: If I could live anywhere in the world, where would I live? Next

PDF How Emotions Are Made
The only limits you see are the ones you impose on yourself. Dr. Wayne Dyer

HIV Tests Are Not HIV Tests
Almost everything will work again if you unplug it for a few minutes, including you. Anne Lamott

Idea Transcript


Look up a Term CLAM Bake

Climate Science Glossary Term Lookup Enter a term in the search box to find its definition.

Settings Use the controls in the far right panel to increase or decrease the number of terms automatically displayed (or to completely turn that feature off).

Term Lookup Term: Define

Settings Beginner

Intermediate

Advanced

No Definitions Definition Life: 20 seconds

All IPCC definitions taken from Climate Change 2007: The Physical Science Basis. Working Group I Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Annex I, Glossary, pp. 941-954. Cambridge University Press.

Home



Arguments



Software

Search...













Resources







Comments













The Consensus Project













Translations





About















Donate

How significance tests are misused in climate science





Climate's changed before It's the sun It's not bad There is no consensus

Posted on 12 November 2010 by Maarten Ambaum Guest post by Dr Maarten H. P. Ambaum from the Department of Meteorology, University of Reading, U.K. Climate science relies heavily on statistics to test hypotheses. For example, we may want to ask whether the global mean temperature has really risen over the past ten years. A standard answer is to calculate a temperature trend from data and then ask whether this temperature trend is “significantly” upward; many scientists would then use a so-called significance test to answer this question. But it turns out that this is precisely the wrong thing to do. This poor practice appears to be widespread. A new paper in the Journal of Climate reports that three quarters of papers in a randomly selected issue of the same journal used significance tests in this misleading way. It is fair to say, though, that most of the times, significance tests are only one part of the evidence provided.

It's cooling Models are unreliable Temp record is unreliable Animals and plants can adapt

The post by Alden Griffith on the 11th of August 2010 lucidly points to some of the problems with significance tests. Here we summarize the findings from the Journal of Climate paper, which explores how it is possible that significance tests are so widely misused and misrepresented in the mainstream climate science literature.

It hasn't warmed since 1998 Antarctica is gaining ice View All Arguments...



Username Password Keep me logged in New? Register here Forgot your password?

Latest Posts On its hundredth birthday in 1959, Edward Teller warned the oil industry about global warming 2017 SkS Weekly Climate Change & Global Warming Digest #52 2017 SkS Weekly Climate Change & Global Warming News Roundup #52 New research, December 18-24, 2017 From the eMail Bag: Carbon Isotopes, Part 2: The Delta Notation Fake news is a threat to humanity, but scientists may have a solution US government climate report looks at how the oceans are buffering climate change Some curious things about Svensmark et al. reference list 2017 SkS Weekly Climate Change & Global Warming Digest #51 2017 SkS Weekly Climate Change & Global Warming News Roundup #51 How blogs convey and distort scientific information about polar bears and Arctic sea ice New research, December 11-17, 2017 Analysis: How developing nations are driving record growth in solar power From the eMail Bag: Carbon Isotopes, Part 1: The Basics One Planet Summit: Finance Commitments Fire-Up Higher Momentum for Paris Climate Change Agreement Scientists have beaten down the best climate denial argument 2017 SkS Weekly Climate Change & Global Warming Digest #50 2017 SkS Weekly Climate Change & Global Warming News Roundup #50

Not unsurprisingly, preprints of the paper have enthusiastically been picked up by those on the sceptic side of the climate change debate. We better find out what is really happening here. Consider a scientist who is interested in measuring some effect and who does an experiment in the lab. Now consider the following thought process that the scientist goes through: 1. My measurement stands out from the noise. 2. So my measurement is not likely to be caused by noise. 3. It is therefore unlikely that what I am seeing is noise. 4. The measurement is therefore positive evidence that there is really something happening. 5. This provides evidence for my theory. This apparently innocuous train of thought contains a serious logical fallacy, and it appears at a spot where not many people notice it. To the surprise of most, the logical fallacy occurs between step 2 and step 3. Step 2 says that there is a low probability of finding our specific measurement if our system would just produce noise. Step 3 says that there is a low probability that the system just produces noise. These sound the same but they are entirely different. This can be compactly described using Bayesian statistics: Bayesian statistics relies heavily on conditional probabilities. We use notations such as p(M|N) to mean the probability that M is true if N is known to be true, that is, the probability of M, given N. Now say that M is the statement “I observe this effect” and N is the statement “My system just produces noise”. Step 2 in our thought experiment says that p(M|N) is low. Step 3 says that p(N|M) is low. As you can see, the conditionals are swapped; these probabilities are not the same. We call this the error of the transposed conditional. How about a significance test? A significance test in fact returns a value of p(M|N), the socalled p-value. In this context N is called the “null-hypothesis”. It returns the probability of observing an outcome (M: we observe an upward trend in the temperature record) given that the null-hypothesis is true (N: in reality there is no upward trend, there are just natural variations). The punchline is that we are not at all interested in this probability. We are interested in the probability p(N|M), the probability that the null hypothesis is true (N: there is no upward temperature trend, just natural variability) given that we observe a certain outcome (M: we observe some upward trend in the temperature record). Climate sceptics want to argue that p(N|M) is high (“Whatever your data show me, I still think there is no real trend; probably this is all just natural variability”), while many climate scientists have tried to argue that p(N|M) is low (“Look at the data: it is very unlikely that this is just natural variability”). Note that low p(N|M) means that the logical opposite of the null-hypothesis (not N: there really is an upward temperature trend) is likely to be true. Who is right? There are many independent reasons to believe that p(N|M) is low; standard physics for example. However many climate scientists have shot themselves in the foot by publishing low values of p(M|N) (in statistical parlance, low p(M|N) means a “statistically significant result”) and claiming that this is positive evidence that p(N|M) is low. Not so. We can make some progress though. Bayes' theorem shows how the two probabilities are related. The aforementioned paper shows in detail how this works. It also shows how significance tests can be used; typically to debunk false hypotheses. These aspects may be the subject of a further post.

So next time someone shows you a “statistically significant” result, do tell them: “I don't care how low your p-value is. Show me the physics and tell me the size of the effect. Then we can discuss whether your hypothesis makes sense.” Stop quibbling about meaningless statistical smoke and mirrors. Reference: M. H. P. Ambaum, 2010: Significance tests in climate science. J. Climate, 23, 5927-5932. doi:10.1175/2010jcli3746.1 0 0 Printable Version | Link to this page Comments

The US is penny wise and pound foolish on the climate

1 2 Next

Video: How not to panic about Global Warming

Comments 1 to 50 out of 84:

California's hellish fires: a visit from the Ghost of Christmas Future 2017 SkS Weekly Climate Change & Global Warming Digest #49 2017 SkS Weekly Climate Change & Global Warming News Roundup #49 The Carbon Brief Interview: Dr Katharine Hayhoe COP23 video: Three need-toknows from the UN climate talks in Bonn US government report finds steady and persistent global warming New research, November 27 December 3, 2017 The moral and intellectual bankruptcy of the Republican Party

The Consensus Project Website THE ESCALATOR

In the meantime, we need to live with the fact that “statistically significant” results are not necessarily in any relevant sense significant. This doesn't mean that those results are false or irrelevant. It just means that the significance test does not provide a way of quantifying the validity of some hypothesis.

New research, December 4-10, 2017 Research shows that certain facts can still change conservatives’ minds

97 : 00 : 00

1. Alexandre at 01:45 AM on 13 November, 2010 I was playing around with statistics a few weeks ago. It helps me understand Tamino :-) Then this claim below crossed my mind, just like Dr. Ambaum: In the meantime, we need to live with the fact that “statistically significant” results are not necessarily in any relevant sense significant. I think you could statistically correlate car sales and global warming, for instance, and it would mean nothing. It's the underlying physics AND the statistics that will give you the evidence - which is the case. 0 0 2. Daniel Bailey at 01:57 AM on 13 November, 2010 Re: Alexandre (1) To illustrate your point:

Archives Climate Hustle

Of course, some will argue that the recent surge in piracy is the cause of the "perceived flattening of the global temperature rise". Sigh. In life and statistics, some will see only what they expect to see. The Yooper 0 0 3. CBDunkerson at 02:06 AM on 13 November, 2010 Hoooray! We're saved. Daniel's chart 'proves' that global warming is caused by lack of pirates... but in the past several years piracy has been booming off the coast of Somalia! We should start seeing temperatures turn around now! :] 0 0 4. cynicus at 02:31 AM on 13 November, 2010 I recently saw an article in a journal that supported AGW but the numbers weren't significant at the p

Smile Life

When life gives you a hundred reasons to cry, show life that you have a thousand reasons to smile

Get in touch

© Copyright 2015 - 2024 PDFFOX.COM - All rights reserved.