Data quality issue: Chou_2004

[J535D165]

Various publications show poor automation performance for the Chou_2004 dataset (Cohen et al., 2006, Teijema et al., 2023). Teijema et al. (2023) show little automation, while Cohen et al. (2006) doesn't show any automation.

Chou R, Peterson K, Helfand M. Comparative efficacy and safety of skeletal muscle relaxants for spasticity and musculoskeletal conditions: a systematic review. J Pain Symptom Manage. 2004;28:140–75. https://doi.org/10.1016/j.jpainsymman.2004.05.002

Dataset

The Chou_2004 dataset is part of a machine learning dataset published by Cohen et al. (2006). The Cohen dataset was one of the first open-access machine learning datasets for evidence synthesis, and SYNERGY uses three datasets from this collection (Chou_2003, Chou_2004, Nelson_2002). In SYNERGY, Chou_2004 consists of 9 inclusions and 1630 records.

Original Cohen data https://dmice.ohsu.edu/cohenaa/epc-ir-data/epc-ir.clean.tsv shows the following inclusions:

SkeletalMuscleRelaxants	10	10770520	I	I
SkeletalMuscleRelaxants	16	10971659	I	I
SkeletalMuscleRelaxants	7005	11372804	I	I
SkeletalMuscleRelaxants	4781	12167135	I	I
SkeletalMuscleRelaxants	81	7964712		I	I
SkeletalMuscleRelaxants	86	8129762		I	I
SkeletalMuscleRelaxants	909	8290087		I	I
SkeletalMuscleRelaxants	83	8290095		I	I
SkeletalMuscleRelaxants	7001	9715216		I	I

The second column with integers is the PubMed identifier.

Data quality

The original Chou et al. (2004) paper states: "Ninety-eight reports presenting data for 101 randomized controlled trials of patients with spasticity (55 trials reported in 54 publications) or musculoskeletal conditions (46 trials reported in 44 publications) provided usable data and were included.". This does not correspond with the numbers found in the dataset.

Example 1

Let's consider the inclusions, starting with PMID 10770520. This publication can be found at https://pubmed.ncbi.nlm.nih.gov/10770520/ and has an APA formatted citation of

Orsnes, G. B., Sørensen, P. S., Larsen, T. K., & Ravnborg, M. (2000). Effect of baclofen on gait in spastic MS patients. Acta neurologica Scandinavica, 101(4), 244–248. https://doi.org/10.1034/j.1600-0404.2000.101004244x./

The Chou et al. (2004) paper doesn't reference to this paper. However, Chou et al (2004) refers to

Orsnes, G., Crone, C., Krarup, C., Petersen, N., & Nielsen, J. (2000). The effect of baclofen on the transmission in spinal pathways in spastic multiple sclerosis patients. Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology, 111(8), 1372–1379. https://doi.org/10.1016/s1388-2457(00)00352-7.

This article has PMID 10904217.

Both articles have different abstracts and authors. See the 2 abstracts:

Effect of baclofen on gait in spastic MS patients
G B Orsnes 1, P S Sørensen, T K Larsen, M Ravnborg
Affiliations Expand
PMID: 10770520 DOI: 10.1034/j.1600-0404.2000.101004244x./
Abstract
Objectives: To measure gait and postural stability by objective methods in spastic MS patients and to evaluate the effect of baclofen on gait and postural stability.

Patients and methods: Fourteen spastic MS patients were examined in a placebo controlled double-blind, cross-over trial of oral baclofen treatment. The gait was measured on a computerized treadmill and postural stability was measured on a computer assisted force-plate.

Results: Only insignificant improvements in the clinical measurements during baclofen treatment were found. At baseline gait was characterized by low speed, short steps and unsteadiness. Postural stability was severely impaired. During baclofen treatment only vertical unsteadiness of gait diminished significantly.

Discussion: We conclude that patients primarily with spasticity, concomitant with hampering or painful spasms and co-contractions should be offered treatment with baclofen. Only some will experience improvement of their gait disorders, when treated with baclofen.

The effect of baclofen on the transmission in spinal pathways in spastic multiple sclerosis patients
G Orsnes 1, C Crone, C Krarup, N Petersen, J Nielsen
Affiliations Expand
PMID: 10904217 DOI: 10.1016/s1388-2457(00)00352-7
Abstract
Objectives: To measure the effect of baclofen on the transmission in different spinal pathways to soleus motoneurones in spastic multiple sclerosis patients.

Methods: Baclofen was administered orally in 14 and intrathecally in 8 patients. H(max)/M(max), presynaptic inhibition by biceps femoris tendon tap of femoral nerve stimulation, depression of the soleus H-reflex following previous activation of the Ia afferents from the soleus muscle (i.e. postactivation depression), disynaptic reciprocal Ia inhibition of the soleus H-reflex and the number of backpropagating action potentials in primary afferents, which may be a sign of presynaptic inhibition, were examined.

Results: Baclofen depressed the soleus H(max)/M(max) ratio significantly following oral and intrathecal baclofen. None of the two tests of presynaptic inhibition, or the postactivation depression or the disynaptic reciprocal Ia inhibition of the soleus H-reflex were affected by baclofen administration. Also the action potentials of the primary afferents were unchanged during baclofen administration.

Conclusions: The antispastic effect of baclofen is not caused by an effect on the transmitter release from Ia afferents or on disynaptic reciprocal Ia inhibition. One possible explanation of the depression of the H-reflex by baclofen is suggested to be a direct depression of motoneuronal excitability.

Example 2

PMID 8290095 is publication:

Karp, B. I., Cole, R. A., Cohen, L. G., Grill, S., Lou, J. S., & Hallett, M. (1994). Long-term botulinum toxin treatment of focal hand dystonia. Neurology, 44(1), 70–76. https://doi.org/10.1212/wnl.44.1.70

This publication can't be found at all in the paper.

Summary

About half of the inclusions (5/9) can't be matched to a record directly.

Conclusion

There is a series of data issues with the Chou_2004 dataset. The original paper looks clean and well-documented but doesn't have accessible supplementary data (http://www.ohppr.state.or.us/hrc/PMPD_hrc.htm%35drugclass1). The issues might result from problems with the lookup of references via the PubMed API by Cohen at al. This has a significant impact on both the inclusions and exclusions. I expect that it will be hard for the SYNERGY team to resolve these issues. The dataset is published more than 20 years ago and reproduction and debugging can be hard.

I'm using SYNERGY. What should I do?

We'd like to keep the dataset in the current SYNERGY (v1) collection so that the results of machine learning studies remain comparable. It should have little effect on the overall result. For some tasks, it might be beneficial to exclude the dataset (e.g. parameter optimization studies).

In the upcoming SYNERGY Plus release, we will exclude the dataset from the collection.