Distortion in reporting observational studies

Image by StartupStockPhotos from Pixabay

The distortion of research findings happens; it is a big problem; it is scientific mischief, Robert H Fletcher¹ and Bert Black reported in the Medical Law journal in 2007.

Sometimes, researchers do that consciously but not always; it can also happen unconsciously.

Spin in scientific reporting can result in profound negative implications, not only for our health but in the legal sector also.

Spin occurs when we interpret data because our interpretations are subjective. In this Tasnim Elmamoun‘s post, on the recognising spin in scientific literature, published on the PLOS SciComm Blogs, this discussion goes deeper.

Simply, La Harvey sums up what spin does in her writing to the Editor’s page of “Nature”: “Spin kills science”.

Therefore the contents of this post are particularly relevant for researchers, reviewers, journal editors, lawyers, and all those interested readers as well.

Specifically, the post discusses how spin occurred in reporting a much-debated observational study: Hormonal replacement therapy for coronary heart disease. Then, the post delves into a much deeper subject matter; declarative and descriptive verbs and when we can and cannot use these verbs.

Case study: Hormone Replacement Therapy for coronary heart disease

Background

In the early 80s, physicians began prescribing hormones to alleviate post-menopausal symptoms. During this time, some believed these hormones might also prevent coronary heart disease associated with the post-menopausal period. A group of interested researchers followed 1976 to 1980 a group of 121,964 female nurses who took hormonal tablets. They collected their responses to a mailed questionnaire periodically during the study period.

The researchers aimed to compare non-fatal and fatal heart attack rates between nurses who took and did not take the medications.

The findings of their study appeared in the New England Journal of Medicine in 1985.

Following is their abstract.

Look closely at the last sentence of the abstract. The study authors claim that their “data support the hypothesis that the postmenopausal use of estrogen reduces the risk of severe coronary heart disease”.

The keyword here is “reduces”.

Can they make that claim?

The experts say “NO”

They should not have used the verb, “reduces”.

Why?

Because they have adopted an observational study design. The observational study designs do not warrant using “declarative” verbs. We can use only “descriptive verbs”.

The observational study designs provide us information about either associations or correlations between the study variables, certainly not its causative relationships. If we want to find strong evidence for causative mechanisms we should adopt randomized controlled study designs. In other words, we can use active verbs such as “reduce” only with randomized study designs.

Misuse of verbs and verb tenses

Declarative verbs and descriptive verbs

It is worth digging a little further about declarative and descriptive verbs. I found a very useful paper about this subject. In 2017, two Austrian surgeons published a short paper citing examples of the differences between declarative and descriptive verbs. I am using the same example of verbs and verbal phrases here because it is pretty comprehensive and very useful, particularly for those whose English is not their primary language.

What are declarative verbs and verb phrases?

Some examples of declarative verbs and verbal phrases are shown below. The former ones declare some definitive action. The latter describes.

Declarative words and phrases	Descriptive words and phrases
Show	are in favour
Demonstrate	are in association with
Prove	believe
Establish	bring forward
Cause	carry, provide
Determine	find, report, suggest
Result in	get
Reduce	have
increase/decrease	highlight
	hold, underscore
	identify, think, underlie
	look, observe
	maintain, place

About the verb “show”

In that paper, the authors caution us when using the verb, ” show” – it can be used either in a descriptive or declarative sense – and they suggest instead using the following verbs when the verb, “show” is required to be used in its descriptive sense.

depict
exhibit
display
expose
reveal

Hedging verbs

We use another set of verbs called, “hedging verbs or verbal phrases”. The following are examples of those.

This may account for
seems
appears
suggest
could support
indicate

As suggested in the paper, we should not use those to imply or define causation.

How should we use the adjective, “significant”?

I had this puzzle at the beginning of my career. In research papers, the word “significant” is used with its literal meaning as well and when discussing statistical findings; we write our findings as either “statistically significant” or “statistically non-significant” based on the probability value (p-value). The paper suggests reserving the adjective the word “statistically” to avoid confusion. Instead, the authors suggest the use of words such as “substantial” and “substantive” instead of the adjective “significant”.

So, now how should we write a statistically significant or non-significant finding from an observational study design?

The authors of the above paper suggest that we should write sentences such as follows:

A statistically significant was found …..
A is associated with a statistically significant increase (or a decrease)…

“trend”

We can find this word in research papers and even in abstracts commonly when the p-value is more than 0.05. The experts advocate not to engage in this practice particularly not to “imply or claim statistical significance”.

Correlate and agreement

And, the above verbs too should be reserved to discuss statistics because the “correlate” is used with Pearson r and Spearman rank correlation statistical tests. Similarly, “X was in agreement with Y” should also be avoided because of Cohen’s “agreement” Kappa statistic.

Verb tense

The verb tense also matters; The above two editor surgeons claim that the past tense always is descriptive and the present tense is declarative. For example, X (intervention) decreased mortality” is descriptive and “X decreases mortality” is declarative and implies causality.

However, the year 2002 reversed it all. The JAMA published a paper based on the findings from a randomized controlled design. According to the paper, the authors stopped the study prematurely due to exceeding the expected breast cancer risk. And, they found higher coronary heart disease and stroke risks too. This debate is still going on. I do not intend to delve into this conflict. I aim to emphasize choosing the most appropriate verbs that reflect the strength of evidence, based on the study design.

Those who are interested can read the following blog post that explains more about this subject written by Hilda Bastian:https://absolutelymaybe.plos.org/2016/03/17/how-to-spot-research-spin-the-case-of-the-not-so-simple-abstract/#:~:text=Research%20spin%20is%20when%20findings,message%20you%20want%20to%20send.

Misuse of adjectives

In 2015, a research team published a paper about their findings on the usage of adjectives in 16,789 randomized controlled trial abstracts published in PubMed.

Among many, the most common adjectives and adjectival phrases found in RCTs either in the title or abstracts are as follows:

Well tolerated
Meaningful
Feasible
successful
usual
good

Is it wrong to use adjectives in the abstract?

The use of adjectives in science writing is a double-edged sword. It allows writers to generalize study findings. We should not use that based on the findings of one study.

In 2012, the Medical Publishing Insights and Practices Initiative (MPIP) recommended avoiding the use of adjectival phrases such as, “well-tolerated” and “generally safe”.

I am quoting below some adjective phrases in the context that they have highlighted for us to understand better usage.

“The data suggest that the drug ——- is well tolerated by the high-risk patients”
“clinically meaningful”
feasible
demonstrate good clinical efficacy and safety
successful treatment

The UpStreamBoat