Precognition and 5%

Many areas of scientific research such as psychology, medicine, and economics make heavy use of tests of statistical significance. A popular significance level is 5%; this means that the experiment results are less than 5% likely to have occurred by random chance. While these tests are arguably valid and useful, with the wide availability of automated experimentation and computerized data mining they become ever easier to unintentionally and intentionally game.

One well-known mistake is to omit necessary corrections when using the same data set to test multiple hypothesis. If a single hypothesis is 5% likely to be true by random chance, then by testing 14 hypothesis against the same data set it is over 50% likely [1 - (1 - .05)^14 = 0.51] to find at least one false positive result. Besides correction factors, a common solution is to have two separate data sets, where one set is used for data mining for possible hypothesis, and a second set is used to verify the result. On a large scale, however, poor hypothesis can still pass this second filter. This is especially true when the hypothesis themselves are being automatically generated and not based on any previously known plausible physical mechanisms.

A related problem is known as the "file drawer effect": positive results are published, while negative results remain in the "file drawer". This creates a serial version of the problem above, where if the same hypothesis is tested 14 times then by random chance it is over 50% likely to be confirmed at least once. The negative results are never published. There is a movement to publish negative results, but this is also problematic because the negative results may be due to recognized poor experimental procedure. These problems reduce the usefulness of meta-studies, since they are summarizing and aggregating the results of positive studies without having accurate information about how many other negative studies were never published.

These issues are becoming more well known, as problematic results in areas such as clinical drug trials are found. A useful "canary in the coal mine" for statistical techniques is parapsychology. While confirmation of abilities such as precognition (seeing the future) is possible, it is much more likely to indicate that statistical standards for research and publication in a field (in this case psychology) have fallen too low. Recently the paper "Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect" by Daryl J. Bem was accepted for publication in a prominent psychology journal. Critics claim the author has made numerous mistakes, including those discussed above. Bem found subjects could predict a future result correctly 53.1% of the time (where random chance was 50%). But as Wagenmakers, etc. have have pointed out a similar large-scale test has been running for a long time: casino roulette. In European roulette the house edge is 2.7% [36 to 1 odds against winning; 35 to 1 payout], so gamblers with Bem's 3.1% edge would have cleaned out the casinos already. It is suspicious that in most studies which do find psychic abilities it hovers at the edge of the statistical significance value chosen for the study.

Bem's response to critics is amusing. A major part of his defense is appeal to authority: the referees of his paper accepted it for publication in a prominent journal, and his statistical techniques are commonly accepted in psychology. He defers to another author (Radin) for defense of the historical record of parapsychological research. Radin cites meta-studies and makes an even more dubious appeal to authority: the U.S. patent office.

Poor Pluto

The demotion of Pluto from planet to dwarf planet (or planetoid) provides another example of the themes of my previous posts. Like the definition of "species", the definition of "planet" is vague, man-made, and changes over time. Scientific authorities also alter the membership of the category based on new information. These changes make many people (especially children) upset. I suspect changes to the planets also cause cognitive dissonance due to their ancient connections to astrology, mythology, and religion. The unchanging nature of heavenly objects was a bedrock belief: "as sure as the sun will rise." Yes, Pluto was not one of the original planets visible to the naked eye, but like the others on discovery it was given the name of a Roman (formerly Greek) god. Today people don't consciously remember how much of our daily language is based on past mythology. My favorite example (besides the planets) is the English names of the days of the week:

Sunday : Sun
Monday : Moon
Tuesday : Tyr (Norse god)
Wednesday : Woden / Odin (Norse god)
Thursday : Thor (Norse god)
Friday : Freya (Norse goddess)
Saturday : Saturn (Roman god)

Darwinism and Authority

American critics of evolutionary theory often prefer to use the term "Darwinism". In doing so they reveal the philosophical, psychological, and theological underpinnings of their world-view: the argument from authority

Appeal to authority is a fallacy of defective induction, where it is argued that a statement is correct because the statement is made by a person or source that is commonly regarded as authoritative.

Scientists don't believe the theory of evolution is correct because it was originally formulated by Charles Darwin, or because it was presented in his book "The Origin of Species by Means of Natural Selection".

Scientific presentations of new theories consist of two parts: the theory itself, and the gathered data which this theory explains. The availability of the gathered data means other scientists can check its validity and see whether it actually supports the proposed theory, or matches some other theory better. For example, the finches Darwin gathered in the Galapagos islands are actually still stored in the bird collection of the British Natural History Museum. Additionally, future scientists can gather additional data and perform new experiments to confirm, expand on, or refute Darwin's original theories. So far new gathered data has confirmed Darwin's basic theories, and vastly expanded on them (understandable, since Darwin was not even aware of Mendelian genetics, let alone DNA). But this new research is not being done to defend Darwin's legacy. Young scientists would be overjoyed to discover a radical new experimental finding or theory which overturns Darwin, and are feverishly looking for this. This very possibility of being wrong is what makes evolution a scientific theory.

An earlier post quoted Dawkins on how children are natural essentialists. I argue children are also, both by nature and nurture, believers in the argument from authority. Children want all statements to have invariant mappings to the categories "true" or "false". The vague category "as best we can tell right now, pending future information" is confusing and upsetting. Teachers and parents are also natural authority figures, and don't have the time (and often background) to explain the rationale behind all their instructions. Elementary memorization is useful, and a child which questioned everything would be impossible to teach. But for many students this attitude continues to their study of science, so Newton, and Einstein, and Darwin become unquestioned authority figures. They don't go back to see that these scientist's work consisted of the same type of presentation discussed above: theories with supporting evidence.

Critics of "Darwinism" are often actually modeling their critique on their early childhood religious upbringing. Early authority figures in their life (their parents) presented additional authority figures (church leaders) whose authority is ultimately derived from an unquestionable final authority figure (the religion's founder(s)). The founders words are contained in unquestionable texts (the revealed sacred documents), and all necessary truths can be obtained through detailed study of those texts. This technique of finding truths through interpretation of texts written by founders (and later texts written about texts written by founders) can arguably be useful in some fields such as law. But it is of only historical interest in science. The correctness of modern evolutionary theory is not bound up in the life of Darwin or the text of "The Origin of Species."

An amusing collision between intelligent design "science" and argument from religious authority is here. Dembski is at risk of being "Expelled"...