Reliability, validity, and responsiveness of a smartphone-based manikin to support pain self-reporting
An article by Pain Reports featuring Ben James, uMotif Chief Design Officer & Co-founder.
Abstract
Introduction:
Many people worldwide suffer from chronic pain. Improving our knowledge on chronic pain prevalence and management requires methods to collect pain self-reports in large populations. Smartphone-based tools could aid data collection by allowing people to use their own device, but the measurement properties of such tools are largely unknown.
Objectives:
To assess the reliability, validity, and responsiveness of a smartphone-based manikin to support pain self-reporting.
Methods:
We recruited people with fibromyalgia, rheumatoid arthritis, and/or osteoarthritis and access to a smartphone and the internet. Data collection included the Global Pain Scale at baseline and follow-up, and 30 daily pain drawings completed on a 2-dimensional, gender-neutral manikin. After deriving participants’ pain extent from their manikin drawings, we evaluated convergent and discriminative validity, test–retest reliability, and responsiveness and assessed findings against internationally agreed criteria for good measurement properties.
Results:
We recruited 131 people; 104 were included in the full sample, submitting 2185 unique pain drawings. Manikin-derived pain extent had excellent test–retest reliability (intraclass correlation coefficient, 0.94), moderate convergent validity (ρ, 0.46), and an ability to distinguish fibromyalgia and osteoarthritis from rheumatoid arthritis (F statistics, 30.41 and 14.36, respectively; P < 0.001). Responsiveness was poor (ρ, 0.2; P, 0.06) and did not meet the respective criterion for good measurement properties.
Conclusion:
Our findings suggest that smartphone-based manikins can be a reliable and valid method for pain self-reporting, but that further research is warranted to explore, enhance, and confirm the ability of such manikins to detect a change in pain over time.
1. Introduction
Chronic pain is common and drives disability in people with musculoskeletal conditions, but the precise prevalence of chronic pain remains unknown. Chronic pain deteriorates people’s physical and mental health, which in turn causes disability that results in lower productivity and increased work absenteeism.4 Despite its high prevalence and impact, the management of pain continues to be suboptimal.6 To improve our knowledge on the prevalence of pain and how to manage it adequately, we need robust methods to measure pain in large populations. This will enable monitoring of pain and quantifying treatment effects, thereby creating opportunities for expediting pain research and improving pain (self-) management.
Digital pain manikins are increasingly common pain self-report tools,2 consisting of a human-shaped figure where people can record the location of their pain on a digital device. Manikins are intuitive and easy to complete, especially for reporting more complex pain patterns, such as pain that varies between locations. Previous research showed that digital pain manikins have good reliability and validity if deployed on a predefined device with fixed screen size.5,10 Although smartphone-based manikins would simplify pain data collection by allowing people to use their own device, the measurement properties of such manikins are less clear.1
Therefore, this study aimed to assess the reliability, validity, and responsiveness of a smartphone-based pain manikin to facilitate collection of pain self-report data to support self-management, clinical care, and research.
2. Methods
We designed this study in line with internationally established guidance for developing and evaluating health measurement instruments.8,11
2.1. Participant recruitment and eligibility
We recruited participants through 4 rheumatology departments in England (United Kingdom) and through patient social media groups. People were eligible if they were 18 years or older; had a clinical diagnosis of fibromyalgia, rheumatoid arthritis, and/or osteoarthritis; and had daily access to an Android device connected to the internet. People provided written informed consent for taking part. In line with methodological guidance,9 we aimed for a minimum sample size of 100.
2.2. Manchester Digital Pain Manikin and primary outcome
The pain self-report instrument under evaluation was the Manchester Digital Pain Manikin app, which we developed in collaboration with a technology partner (uMotif). It is a two-dimensional, gender-neutral manikin with a front and back view. Drawing involved selecting a pain intensity using the sliding scale at the bottom of the screen and shading painful areas directly on the manikin (Fig. 1). Users could zoom in by selecting prespecified body areas from a list.
In line with previous studies,5,10 we used manikin-derived pain extent as the primary outcome. This was defined as the proportion of the total body area affected by pain and calculated as the number of manikin squares shaded as painful divided by the total number of squares in the manikin grid (n = 14,238).
2.3. Data collection and analysis
We asked participants to collect the following data by completing online questionnaires and using the Manchester Digital Pain Manikin app:
- (1) Baseline: demographics and Global Pain Scale.7
- (2) During follow-up: a daily pain manikin drawing for 30 days; people were prompted every evening to submit a drawing; we instructed them to submit an empty drawing on pain-free days.
- (3) End of follow-up: Global Pain Scale.7
After completing follow-up, participants received a summary of their daily pain reports and a gift voucher to thank them for taking part. More details on the data collection procedures can be found elsewhere.3
All participants who completed their baseline questionnaire and submitted at least one manikin drawing were included in the full sample. Box 1 explains how we assessed the measurement properties of manikin-derived pain extent.8,11
3. Results
3.1. Participant characteristics
Of the 131 people consenting to take part, 104 were included in the full sample, submitting 2185 unique pain drawings; 160 (3%) drawings were empty, suggesting a pain-free day. The mean number of drawings across participants was 21 (SD, 7.7), with 91 people submitting 10 or more.3Table 1 describes participants’ baseline demographic and pain characteristics.
3.2. Measurement properties of Manchester Digital Pain Manikin–derived pain extent
3.2.1. Validity
We assessed convergent validity by looking at the association between participants’ mean pain extent across the follow-up period and their mean Global Pain Scale scores at baseline and follow-up. One person completed only one report, and an additional 9 had a missing end-of-follow-up questionnaire, resulting in a subsample of 94 (90%) participants included in the analysis. We found a Spearman correlation coefficient of 0.46 (P < 0.05), which met our criterion for good convergent validity.
For discriminative validity, we compared the mean (SD) pain extent for 4 subgroups of participants (within our full sample of total n = 104) based on their painful condition, namely those with only osteoarthritis (0.019 ± 0.020); only rheumatoid arthritis (0.048 ± 0.058); only fibromyalgia (0.121 ± 0.120); and more than one of these conditions (0.097 ± 0.080). The F statistics were 30.41 (P < 0.0001) and 14.36 (P < 0.001) for people with only fibromyalgia or osteoarthritis, respectively; these were not statistically significant for the other 2 subgroups.
3.2.2. Test–retest reliability
For assessing test–retest reliability, we included 100 (96%) participants in the analysis, after excluding 4 people who did not complete a pair of pain drawings on consecutive days. When comparing pain extent derived from drawings submitted on 2 consecutive days, the intraclass correlation coefficientagreement was 0.94 (95% confidence interval, 0.90–0.96), which indicates very good reliability.
3.2.3. Responsiveness to change
For assessing responsiveness, we included the same subsample of 94 (90%) as we did for analysing convergent validity. The mean change in pain extent between baseline and follow-up was an increase of 0.037 (ie, a deterioration in pain), whereas the mean change in Global Pain Scale score was a decrease of 0.745 (ie, an improvement in pain). The Spearman correlation coefficient for the association between the 2 changes was 0.2 (P, 0.06), indicating a weak, statistically not significant correlation that did not meet our criterion for good responsiveness.
4. Discussion
We found that smartphone-based pain manikins can be a reliable and valid instrument for measuring self-reported pain extent. This is in keeping with evaluations of a previous version of the Manchester Digital Pain Manikin12 and of digital manikins deployed on predefined devices with a fixed screen size.5,10 In addition, our manikin was able to distinguish fibromyalgia and osteoarthritis from rheumatoid arthritis but did not detect a change in pain scores over time. The latter warrants further exploration in studies with larger sample sizes, especially because this is the first study to assess responsiveness of smartphone-based manikins. Improvement and confirmation of responsiveness is also essential for digital manikins to facilitate monitoring of treatment response in clinical practice and outcome measurement in trials.
A systematic review on the adoption of digital pain manikins for research data collection2 showed that relatively few studies allowed study participants to bring their own device. Together with a systematic review of pain manikin apps,1 it called for efforts to strengthen the evidence base for good measurement properties of digital pain manikins, in particular those deployed on smartphones. Our study findings contribute towards this, with our future research plans including making the Manchester Digital Pain Manikin suitable for use on iPhones. Ultimately, this will facilitate smartphone-based manikins to become widely accepted as digital pain self-report tools in large population health and comparative effectiveness studies, while also being suitable for supporting clinical and self-management of pain.
Disclosures
B.J. is a cofounder and staff member of uMotif, the company that developed the Manchester Digital Pain Manikin. None of the other authors has any conflicts of interest to declare.
Acknowledgments
The authors thank all four local research nurse teams of rheumatology departments at NHS sites in Greater Manchester for supporting recruitment of eligible participants for this study. The authors thank all the social media groups’ moderators who helped us in promoting the study on relevant patient communities on Twitter, Facebook, WhatsApp, and other bespoke groups.
The University of Manchester’s Medical Research Council’s Confidence in Concept scheme 8 (grant number: MC_PC_19046) funded this study.
Ethical approval: The study received a favourable ethics opinion and Health Research Authority approval from the National Health Services (NHS) Westminster Research Ethics Committee in England, United Kingdom (21/PR/0342). In the United Kingdom, NHS Research Ethics Committee are the equivalent of Institutional Review Boards, and the authors confirmed that the ethics path in the sponsoring institution was fully implemented and approved.
Data availability: In line with written informed consent obtained from study participants, the data that support the findings of this study are not openly available but are available from the corresponding author upon reasonable request. Data are located in controlled access data storage at The University of Manchester, United Kingdom.
Guarantor: S.N.v.d.V. takes full responsibility for the article, including for the accuracy and appropriateness of the reference list.
Contributions: S.N.v.d.V. conceived the study. S.M.A. recruited participants and coordinated data collection. Z.Y. analysed the data. All authors were involved in interpreting the findings. S.N.v.d.V. drafted the manuscript. All authors reviewed the manuscript for important intellectual content and approved the final version of the manuscript.