The Scientific Method of Investigation

The goal of this monograph is to help consumers better understand biomedical research, and to sort out science fact from fiction. It will help make sense of new research as distilled in the popular press or advertising claims for new products. It can be confusing to read reports that tend to be mutually contradictory. Yesterday’s dictum seems to be today’s derision. As you read this article, you may find yourself formulating questions to ask when reviewing research or advertising claims.

The scientific method of investigation is the process of arriving at theories to help us explain phenomena. Value of the theorem produced in the process is dependent upon the quality of the steps taken in producing that theorem.

• The first step is making careful observations.
• The second is asking a question based upon those observations.
• The third is formulating an hypothesis to attempt to answer the question.
• Fourth is experimentation to test that hypothesis.
• Fifth is to propound a theorem that interprets the tested hypothesis in light of the results of the experiments.

The theorem is then used as a test that is continually validated. If over time, the theory is not found to be useful, it may need to be revised or discarded.

Two illustrations may elucidate the importance of these steps. A few years ago a significant number of elderly people in the Georgian Republic were reported to have lived over 110 years; longevity was attributed to daily consumption of yogurt. This resulted in quite an interest by American consumers. It turned out that claims of longevity were exaggerated or could not be verified. This is an example of incautious observations. Another example illustrates misinformation that might have more serious repercussions.

Studies showed consumption of polyunsaturated vegetable oils results in lowering serum cholesterol levels (1). A plethora of advertising appeared touting the benefits of vegetable oils for preventing heart disease. Consumption of vegetable oils, margarine, and shortenings dramatically increased (2). Lost among the ballyhoo were studies addressing where the cholesterol went, or what side effects might occur in increased consumption of debulked, defibered vegetable oils. Research on polyunsaturated fatty acids (PUFA) a frequent component of vegetable oils has investigated free radical generation of oxidized LDL and atherosclerosis (3), and promotion of various cancers (4). The FDA stopped manufacturers from making some of the more outlandish claims, but the myths persist.

Even if careful observations are made, good questions asked, and sound hypotheses propounded it may still be possible to fall into the experimental pit. Laboratory or basic research is performed in animals or in vitro (literally, within glass), but not in humans. These experiments can often be tightly controlled; however, it is not always possible to extrapolate to human beings. Conducting experiments where people are subjected to some intervention and carefully observed, clinical trials, are not always possible. If it were possible to always perform clinical trials where all variables are carefully controlled, experimental results might be much easier to interpret. Due to ethical concerns and the long time to get results, it is often not possible. Humans are heir to the placebo effect, reporting errors, memory lapses, and the desire to please among other things. The placebo effect can account for up to 30% false positives in some studies. To help eliminate some of these effects, the best studies are performed double blind where neither the subject nor the experimenter knows who is receiving a real treatment or who receives a placebo. More often the test is performed single blind, where only the subjects do not know who received the placebo

Researchers often rely upon epidemiological studies. These may be prospective or retrospective (looking forward or backward respectively). In prospective (also called longitudinal ) studies, cohorts are used. A cohort is a defined population group followed prospectively. Participants are carefully selected and extensively questioned about such things as life styles. Over the course of the study, individuals are monitored for disease, and attempts are made to identify factors associated with the malady. In retrospective studies, or case control studies, people afflicted with a particular disease are carefully matched (by gender, age, etc.) to healthy controls. Researchers then attempt to identify factors that determined who got sick. These studies take less time and money to perform, but can often be difficult to interpret.

Neutrality of investigators is paramount. For example, it might be difficult to evaluate the effects of tobacco if investigation is being funded by a tobacco company. A more subtle influence to assess is the impact on research if the continued funding of research is dependent upon generation of statistically relevant results. Facing loss of funding for a project that the principal investigator may have dedicated years to, may be a difficult milieu in which to function without bias.

So far, we have touched upon observations, questions, hypotheses, and experimentation. Next is analysis of experimental data. It is this statistical analysis that may help to determine the validity of our hypothesis, and may lead us to a theorem.

After data is collected, statistical analysis can begin. Statistical analysis may be able to tell us if results are significant or merely the product of chance. Experimental design determines the type of analysis employed. Degree of certainty may be expressed in several ways; conventionally, a relationship is statistically significant if the probability of it occurring by chance is less than 5 out of 100. Not all experiments will lead us to conclusions. It may simply be that we discover inadequacy of our current methods to evaluate the problem. Ideally, the whole population should be studied, but this is usually not possible. Instead, representative samples must be taken. Sampling can be the downfall of experiments.

It is difficult to avoid bias and maintain randomness while keeping study populations small enough to handle. A prominent example of this problem may be seen in a recent study reported in a well respected journal (5) and widely quoted in the popular press. The study attempted to evaluate effects of fruits, vegetables and olive oil on breast cancer. It was conducted on 820 Greek women with breast cancer and 1,548 Greek women without breast cancer. The group who consumed the most vegetables had about 1/2 the risk, the group who ate the most fruit had about 1/3 less risk, and the group who consumed olive oil more than once per day had about 1/4 the risk of breast cancer. Unfortunately, scientists found several flaws with this study. The study population who consumed olive oil was only 99 women, mainly post menopausal who likely had consumed olive oil (in unknown quantities) for much longer than the year covered in the study. Information about diet came from one retrospective questionnaire in which the women were asked to estimate their nutritional intake for the past year and no questions were asked about the amount of olive oil consumed. Estimates of dietary intake are often erroneous and very difficult to do for a year in the past. Further, psychological investment for particular eating regimens were not controlled. Breast Cancer rates in the Mediterranean countries are quite a bit lower than in the USA; and, less animal fats, more fruits, vegetables, and monounsaturated olive oil are consumed. So, it is tempting when looking for a dietary connection to cancer, to look to these dietary parameters. Alas, linkage to any of these factors is not yet clear.

It is also important to consider the journal in which a study has been published. The best journals are peer reviewed. This is a process where experts in the field critically review the paper before it is accepted for publication. It is common for a paper to be revised in a way that takes into account the critique presented by the reviewers. And, a fair number of papers are rejected. One type of publication that may have excellent studies but are not peer reviewed, are symposium volumes. These are publications that publish work that has been presented at a conference, usually just as they were submitted. It is important to be careful when evaluating these articles.

Summary

To summarize, when evaluating claims, ask at least the following questions.

• Are claims based upon the scientific method of investigation?
• Is this a single study or are there related corroborating studies?
• Was the study conducted in vitro, in animals, or in humans?
• Was this a retrospective or a prospective study?
• Was evidence published in a refereed journal (i.e., one where the study was critically reviewed by specialists in the field prior to publication)?
• How large was the study population?
• Was the study population representative of the population as a whole or are the results likely to be valid only for people with specific characteristics?
• Was the study conducted for enough time for relevant effects to be observed?
• Can you identify any sources that might have biased the results?
• Is there some other reasonable explanation that might explain the results, or were other hypotheses possible but not tested?
• If this is an advertising claim, what research is referenced to support these claims?

One last example is illustrative of paradoxical results from apparently well-designed studies. You may wish to get copies of these two studies and try out your scientific sleuthing abilities. Recently, two very-well-designed studies on effects of hormone replacement therapy and cancer risk were published (6,7). Unfortunately, the results seem to be contradictory. The first study looked at almost 70,000 postmenopausal nurses, approximately 60% of whom took either estrogen alone (estrogen replacement therapy, ERT) or estrogen plus progestin (hormone replacement therapy, HRT). Women on HRT for more than 5 years had a 40% increase in breast cancer risk over the control group of women who did not take either ERT or HRT. Women on ERT had slightly less increase over control. The second study included 500 women with breast cancer and 500 without. They were followed for eight years and no increased risk of breast cancer was found in postmenopausal women taking ERT or HRT. Indeed, a slight decrease in risk was found in women taking HRT for greater than eight years. The answers to the apparent contradictory results may lie in the characteristics of the sample groups.

Copies of these articles are readily available. It can be stimulating to critically read studies such as these. It can be as entertaining as unraveling a mystery story, but more rewarding. The answers to these real life mysteries can add to your health and well being.

References

1. Wissler RW, Vesselinovitch D. Am J Cardiol Jul 20;52(2):2A-7A, 1983.
2. Rizek RL, et al. J Amer Oil Chem Soc 51:244, 1974.
3. Jialal I, Devaraj S. Clinical Chemistry 42(4):498-506, 1996.
4. National Research Council: Diet Nutrition and Cancer, Washington, National Academy Press, 1982.
5. Trichopoulou A, et al. J Natl Cancer Inst 18;87(2):110-, 1995.
6. Colditz GA, et al. N Eng J Med, Jun 15; 332(24)1589-93, 1995.
7. Standford JL, et al.. JAMA, July 12; 274(2)137-142, 1995.

   

Copyright© 1996-2002 by Medical Information Unlimited, Inc. All rights reserved.
Revised: May 07, 1999.