Skip to main content

Validity and reliability of a pilot scale for assessment of multiple system atrophy symptoms

Abstract

Background

Multiple system atrophy (MSA) is a rare progressive neurodegenerative disorder for which brief yet sensitive scale is required in order for use in clinical trials and general screening. We previously compared several scales for the assessment of MSA symptoms and devised an eight-item pilot scale with large standardized response mean [handwriting, finger taps, transfers, standing with feet together, turning trunk, turning 360°, gait, body sway]. The aim of the present study is to investigate the validity and reliability of a simple pilot scale for assessment of multiple system atrophy symptoms.

Methods

Thirty-two patients with MSA (15 male/17 female; 20 cerebellar subtype [MSA-C]/12 parkinsonian subtype [MSA-P]) were prospectively registered between January 1, 2014 and February 28, 2015. Patients were evaluated by two independent raters using the Unified MSA Rating Scale (UMSARS), Scale for Assessment and Rating of Ataxia (SARA), and the pilot scale. Correlations between UMSARS, SARA, pilot scale scores, intraclass correlation coefficients (ICCs), and Cronbach’s alpha coefficients were calculated.

Results

Pilot scale scores significantly correlated with scores for UMSARS Parts I, II, and IV as well as with SARA scores. Intra-rater and inter-rater ICCs and Cronbach’s alpha coefficients remained high (> 0.94) for all measures.

Conclusion

The results of the present study indicate the validity and reliability of the eight-item pilot scale, particularly for the assessment of symptoms in patients with early state multiple system atrophy.

Background

Multiple system atrophy (MSA) is a rare progressive neurodegenerative disease characterized by autonomic dysfunction, Parkinsonism, and ataxia [1, 2]. MSA patients generally need wheelchairs in five years and die in ten years from disease onset. Though some underlying mechanisms of MSA have been revealed, such as the aggregation of α-synuclein to oligodendroglia, the complete pathogenesis of the disease remains to be elucidated [3]. As quantitative biomarkers for MSA have not yet been developed for use in clinical trials, clinicians must rely on evaluations of changes in symptoms. However, the usefulness of such evaluation varies according to the scale used, and the large numbers of patients required for MSA trials render redundant and unresponsive scales impractical. Therefore, a brief yet sensitive scale is desirable for clinical trials involving patients with MSA.

In a previous study, we compared the following five scales in their ability to assess symptoms of MSA [4]: Unified MSA Rating Scale (UMSARS) [5], Scale for the Assessment and Rating of Ataxia (SARA) [6], Berg Balance Scale (BBS) [7], MSA Health-Related Quality of Life scale (MSA-QoL) [8], and Scales for Outcomes in Parkinson’s Disease–Autonomic Questionnaire (SCOPA-AUT) [9]. We subsequently devised a simple pilot scale comprised of eight items representative of those exhibiting the largest standardized response means (handwriting, finger taps, transfers, standing with feet together, turning trunk, turning 360°, gait, and body sway) [4].

Our prior study revealed that the UMSARS Part II (motor examination), Part IV (global disability scale, SARA, and BBS are effective in evaluating MSA progression over 12 months, indicating their potential to assess rapid changes in MSA symptoms. Detailed item-by-item analyses suggested that the largest SRMs were obtained for the following items: handwriting, finger taps, transfers, standing with feet together, turning trunk, turning 360 degrees, gait, and body sway. Further analyses revealed that our eight-item semi-quantitative (total score = 36 points) pilot scale (Table 1) exhibited an SRM larger than those observed for the UMSARS Part II/Part IV, SARA, and BBS [4], suggesting that the pilot scale was most effective in detecting rapid changes in symptoms of MSA. In the present study, we aimed to investigate the validity and reliability of the pilot scale for the assessment of symptoms in patients with both cerebellar and parkinsonian subtypes of MSA.

Table 1 The items of the pilot scale

Methods

The present prospective observational study included hospitalized patients and outpatients receiving treatment in the Departments of Neurology at Hokkaido University Hospital and Obihiro Kosei Hospital between January 1, 2014 and February 28, 2015. Included patients had been diagnosed with probable or possible MSA per criteria defined in the 2008 consensus statement [10]. The present study was approved by the institutional review board of Hokkaido University Hospital. Written informed consent was obtained from all patients prior to their participation in the study. Those who declined to participate as well as those with severe cognitive impairments such as inability to understand explanations or to follow instructions in examination were excluded.

Previous reports utilizing both SARA and BBS were consulted in the design of the present study [11, 12]. Patients were separately evaluated by two independent neurologists. Patients first underwent evaluation by Rater 1 using the UMSARS, SARA, and pilot scale. Rater 2 evaluated patients using the pilot scale alone on the same day. Within one month, patients underwent re-evaluation by Rater 1 using the pilot scale. Each trial was performed blindly, under the same conditions, and in avoidance of acute phases in order to eliminate the influence of sudden changes in symptoms. No interventions were utilized in the present study, and patients were allowed to continue treatments (mainly drug and rehabilitation) already in progress. Amassed data were subjected to linkable anonymizing, following which statistical analyses were performed.

Statistical analysis

JMP® Pro Version 12.0.1 (SAS Institute Inc., Cary, NC, USA) was used for statistical analysis. Correlations between scores on the UMSARS, SARA, and the pilot scale were evaluated by Spearman’s rank coefficients. Inter-rater and intra-rater reliability for the pilot scale was assessed between Rater 1 and Rater 2. The total score as well as individual item scores for the pilot scale were analyzed based on Cronbach’s α coefficients and intraclass correlation coefficients (ICCs). Items with Cronbach’s α coefficients of more than 0.8 were considered to exhibit high internal consistency. ICCs were interpreted in conformity to the reference as slight (0.000 to 0.200), fair (0.201 to 0.400), moderate (0.401 to 0.600), substantial (0.601 to 0.800), or almost perfect (0.801 to 1.000) [13]. Mean values were presented along with standard deviations (SD).

Results

A total of 32 patients (15 male, 17 females; mean age: 63.4 ± 9.7 years old; range: 41 to 80 years old) were enrolled in the present study. Demographic information for included patients is presented in Table 2. Twenty patients had been diagnosed with MSA of the cerebellar subtype [MSA-C], while 12 patients had been diagnosed with MSA of the parkinsonian subtype [MSA-P]. The average time required for assessment was 16.4 ± 5.2 (range: 11–25) minutes for the UMSARS, 3.8 ± 1.0 (2–6) minutes for the SARA, and 5.0 ± 1.5 (2–7) minutes for the pilot scale (Fig. 1a).

Table 2 The characteristics of the study patients
Fig. 1
figure 1

Comparison of scales. aThe average time required for examination of UMSARS, SARA and the pilot scale. SARA and the pilot scale needed shorter time for examination than UMSARS. UMSARS, Unified Multiple System Atrophy Rating Scale; SARA, Scale for the Assessment and Rating of Ataxia. *: p < 0.05, Wilcoxon’s rank test. b. Correlation between scale scores. The scores on the pilot scale significantly correlated with those on UMSARS Parts I, II, IV and SARA. UMSARS, Unified Multiple System Atrophy Rating Scale; SARA, Scale for the Assessment and Rating of Ataxia. ρ: Spearman’s correlation coefficients

Total scores on each scale are presented in Table 2. Average scores for the first assessment were as follows: UMSARS Part I: 21.3/48, UMSARS Part II: 21.3/56, UMSARS Part IV: 3.0/5, SARA: 19.3/40, pilot scale: 20.8/36. There was no significant difference in the total score of the pilot scale between MSA-C and MSA-P (average total score of MSA-C: 21.2, MSA-P: 20.2). The same thing was also confirmed for each item’s score. Both total and individual item scores on the pilot scale significantly correlated with scores on UMSARS Parts I, II, and IV as well as SARA scores (Fig. 1b). Spearman’s correlation coefficients ρ were 0.8780–0.9392. No significant differences were observed between each assessment of the pilot scale (Wilcoxon’s rank test: p = 0.898 to 0.973). Table 3 depicts the distribution of scores assigned by Rater 1 during the first assessment. Scores for the second and third assessments showed similar tendencies. Many items had high item-total correlation coefficients (Spearman’s correlation coefficients: 0.525 to 0.937). ICCs and Cronbach’s α coefficients are presented in Table 4. Inter-rater and intra-rater ICCs and Cronbach’s α coefficients for total pilot scores were both greater than 0.9. Further, inter-rater and intra-rater ICC values over 0.6 (substantial) were obtained for almost all items on the pilot scale: Only item 2 exhibited a moderate inter-rater ICC. Cronbach’s α coefficients were greater than 0.9 for all items.

Table 3 Distribution of scores for rater 1 (first assessment) for the pilot scale (n = 32)
Table 4 Reliability of the pilot scale

Additionally, we considered prototype pilot scales consisting of five to seven items by excluding either a single item or a combination of three items (item 1: hand writing, item 2: finger taps, item 5: turning trunk) with relatively low inter-rater ICCs from the original pilot scale. Exclusion of such items maintained high total scores, intra-rater and inter-rater ICCs, and Cronbach’s α coefficients.

Discussion

Patients in the present study exhibited characteristics similar to those reported in previous studies of Asian/Japanese populations (Table 2) [14,15,16]. The distribution of UMSARS Part IV scores indicated that this study included relatively unbiased patients with mild to severe symptoms.

Scores on the pilot scale significantly correlated to scores obtained on the UMSARS and SARA (Fig. 1b), indicating the criterion-related validity of the pilot scale. The ability to administer this pilot scale in a short period of time further suggests its usefulness in the evaluation of MSA symptoms (Fig. 1a). In addition, ICC and Cronbach α coefficients remained high (Table 4), indicating high intra- and inter-rater reliability. Test-retest reliability and internal consistency were also high. When either one or three low inter-rater ICC items were excluded from pilot scale (Table 4), ICCs and Cronbach’s α coefficients remained relatively unchanged, indicating that a scale consisting only of items related to gait/standing is equally useful in assessing symptoms of MSA.

The present study possesses some limitation. Pilot scale items with low inter-rater ICC (handwriting, finger taps, turning trunk) exhibited ambiguity with respect to differentiating between scores. Further improvement in these areas of evaluation is required in order to more accurately assess changes in MSA symptoms. One such possibility involves combined assessment utilizing both the pilot scale and a gait accelerometer to record quantitative data. In addition, semi-quantitative scales such as that utilized in the present study often exhibit a ceiling effect [17]. This pilot scale also might show a ceiling effect among patients of advanced stage. Then, this pilot scale was not suitable for advanced MSA patients. On the other hand, in clinical trials, many participants would be early cases with mild to moderate symptoms, so influences of a ceiling effect were thought to be less likely. Further investigation regarding this point is required to more fully examine the effect of time course on the utility of the pilot scale. Additionally, this study included patients with mild to severe symptoms of MSA. And MSA-P patients were indeed relatively few. It is desirable that the reliability of this pilot scale would be presented in a larger cohort.

The SARA score of MSA-C group was similar to that of MSA-P group in this study. It should be noted that the score of SARA may be influenced by other symptoms such as Parkinsonism. The pilot scale of this study reflected symptoms of Parkinsonism and ataxia. It can be applied equally in both group without any modification. And the pilot scale showed larger standardized response mean than SARA and UMSARS [4]. It meant the pilot scale could sensitively capture symptom changes among MSA patients. The pilot scale was superior to SARA in terms of sensitivity even if it took some more time (5.0 ± 1.5 min) than SARA (3.8 ± 1.0 min). It is useful if it can suppress the deterioration of items with rapid symptomatic change (= items of this pilot scale) in clinical trials.

Conclusions

The results of the present study indicate that the eight-item pilot scale for the assessment of MSA symptoms is both valid and reliable and may be useful for evaluation of patients in the early stages of MSA. However, due to the limitations of the present study and small sample size, further research involving improved scales as well as larger patient populations is required.

Abbreviations

BBS:

Berg Balance Scale

ICC:

Intraclass correlation coefficient

MSA:

Multiple system atrophy

MSA-QoL:

MSA Health-Related Quality of Life scale

SARA:

Scale for the Assessment and Rating of Ataxia

SCOPA-AUT:

Scales for Outcomes in Parkinson’s Disease–Autonomic Questionnaire

UMSARS:

Unified Multiple System Atrophy Rating Scale

References

  1. Graham JG, Oppenheimer DR. Orthostatic hypotension and nicotine sensitivity in a case of multiple system atrophy. J Neurol Neurosurg Psychiatry. 1969;32:28–34.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Quinn N. Multiple system atrophy--the nature of the beast. J. Neurol. Neurosurg. Psychiatry. 1989;52(Suppl):78–89.

  3. Wakabayashi K, Yoshimoto M, Tsuji S, Takahashi H. Alpha-synuclein immunoreactivity in glial cytoplasmic inclusions in multiple system atrophy. Neurosci Lett. 1998;249:180–2.

    Article  CAS  PubMed  Google Scholar 

  4. Matsushima M, Yabe I, Oba K, et al. Comparison of different symptom assessment scales for multiple system atrophy. Cerebellum. 2016;15:190–200.

    Article  PubMed  Google Scholar 

  5. Wenning GK, Tison F, Seppi K, et al. Development and validation of the Unified multiple system atrophy rating scale (UMSARS). Mov Disord. 2004;19:1391–402.

    Article  PubMed  Google Scholar 

  6. Schmitz-Hubsch T, du Montcel ST, Baliko L, et al. Scale for the assessment and rating of ataxia: development of a new clinical scale. Neurology. 2006;66:1717–20.

    Article  CAS  PubMed  Google Scholar 

  7. Berg K. Measuring balance in the elderly: preliminary development of an instrument. Physiother Can. 1989;41:304–11.

    Article  Google Scholar 

  8. Schrag A, Selai C, Mathias C, et al. Measuring health-related quality of life in MSA: the MSA-QoL. Mov Disord. 2007;22:2332–8.

    Article  PubMed  Google Scholar 

  9. Visser M, Marinus J, Stiggelbout AM, Van Hilten JJ. Assessment of autonomic dysfunction in Parkinson’s disease: the SCOPA-AUT. Mov Disord. 2004;19:1306–12.

    Article  PubMed  Google Scholar 

  10. Gilman S, Wenning GK, Low PA, et al. Second consensus statement on the diagnosis of multiple system atrophy. Neurology. 2008;71:670–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Sato K, Yabe I, Soma H, et al. Reliability of the Japanese version of the scale for the assessment and rating of ataxia (SARA). Brain Nerve. 2009;61:591–5.

    PubMed  Google Scholar 

  12. Matsushima M, Yabe I, Uwatoko H, Shirai S, Hirotani M, Sasaki H. Reliability of the Japanese version of the Berg balance scale. Intern Med. 2014;53:1621–4.

    Article  PubMed  Google Scholar 

  13. Cohen J. Statistical power analysis for the behavioral sciences. 2nd edition: Hillsdale: Lawrence Erlbaum associates; 1988.

  14. Watanabe H, Saito Y, Terao S, et al. Progression and prognosis in multiple system atrophy: an analysis of 230 Japanese patients. Brain. 2002;125:1070–83.

    Article  PubMed  Google Scholar 

  15. Yabe I, Soma H, Takei A, Fujiki N, Yanagihara T, Sasaki H. MSA-C is the predominant clinical phenotype of MSA in Japan: analysis of 142 patients with probable MSA. J Neurol Sci. 2006;249:115–21.

    Article  PubMed  Google Scholar 

  16. Jamora RD, Gupta A, Tan AK, Tan LC. Clinical characteristics of patients with multiple system atrophy in Singapore. Ann Acad Med Singap. 2005;34:553–7.

    CAS  PubMed  Google Scholar 

  17. Blum L, Korner-Bitensky N. Usefulness of the Berg balance scale in stroke rehabilitation: a systematic review. Phys Ther. 2008;88:559–66.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

Financial Disclosure: Hidenao Sasaki received a Grant-in-Aid (No. 27280401) from Japan Agency for Medical Research and Development, a Grant-in-Aid from Takeda Science Foundation, and a Grant-in-Aid for the Research Committee for Ataxic Diseases of the Research on Measures for Intractable Diseases from the Ministry of Health, Labour and Welfare, Japan. The other authors have no financial disclosures to report.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

Authors

Contributions

Research project: A. Conception (MM, IY, HS), B. Organization (HS), C. Execution (MM, IY, IT, MH, TK, KH, HH). Statistical Analysis: A. Design (MM, IY), B. Execution (MM), C. Review and Critique (IY, HS). Manuscript: A. Writing of the first draft (MM), B. Review and Critique (IY, HS). All authors read and approved the final manuscript.

Corresponding author

Correspondence to Masaaki Matsushima.

Ethics declarations

The present study was approved by the institutional review board of Hokkaido University Hospital. Written informed consent was obtained from all patients prior to their participation in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Matsushima, M., Yabe, I., Takahashi, I. et al. Validity and reliability of a pilot scale for assessment of multiple system atrophy symptoms. cerebellum ataxias 4, 11 (2017). https://doi.org/10.1186/s40673-017-0067-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40673-017-0067-5

Keywords