Exploring the construct validity of the Patient Perception Measure - Osteopathy (PPM-O) using classical test theory and Rasch analysis

Background: Evaluation of patients’ experience of their osteopathic treatment has recently been investigated leading to the development of the Patient Perception Measure – Osteopathy (PPM-O). The aim of the study was to investigate the construct
of 12
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Similar Documents
  RESEARCH Open Access Exploring the construct validity of the PatientPerception Measure  –  Osteopathy (PPM-O) usingclassical test theory and Rasch analysis Jane Mulcahy 1 † and Brett Vaughan 1,2,3* † Abstract Background:  Evaluation of patients ’  experience of their osteopathic treatment has recently been investigatedleading to the development of the Patient Perception Measure  –  Osteopathy (PPM-O). The aim of the study was toinvestigate the construct validity of the PPM-O. Methods:  Patients presenting to osteopathy student-led teaching clinics at two Australian universities were askedto complete two questionnaires after their treatment: a demographic questionnaire and the PPM-O. Confirmatoryfactor analysis (CFA) and Rasch analysis were used to investigate the construct validity of the PPM-O. Results:  Data from the present study did not fit the  a-priori   6-domain structure in the CFA. Modifications to the6-domain model were then made based on the CFA results, and this analysis identified two factors: 1) Education & Information (9 items); and 2) Cognition & Fatigue (6 items). These two factors were Rasch analysed individually. Twoitems were removed from the Cognition & Fatigue factor during the analysis. The two factors independently wereunidimensional. Conclusions:  The study produced a 2-factor, 13-item questionnaire that assesses the patients ’  perception of theirosteopathic treatment using the items from a previous questionnaire. The results of the current study provideevidence for the construct validity of the PPM-O and the small number of items makes it feasible to implement intoboth clinical and research settings. Further research is now required to establish the measures ’  validity in a varietyof patient populations. Background A common concern of clinicians and clinical educatorsworking directly with patients is the significant variancein individual treatment efficacy of patients. The pro-posed causes for this variability in patients ’  experiencesof their health encounters are complex and multidimen-sional. Demographic factors such as: gender, age, socialgradient [1], education, ethnicity and geographic locationhave all been identified as factors that affect generalhealth and disease status [2], as well as access to, andutilisation of, treatments and health services [2]. Whilenuances of the clinician and the clinical environmentcontribute to aspects of treatment outcome such as pa-tient satisfaction [3], the patients ’  beliefs about theirhealth and wellbeing, their illness or disease and expec-tations of treatment would also appear to have a signifi-cant effect [4,5]. Patient experience and expectations Research investigating patients ’  experiences during, andas a result of, their treatment has tended to focus onthe: patient-therapist interaction [6,7], clinical environ- ment [8], satisfaction with treatment [9], and, efficacy of  treatment outcomes [7,10]. The patients ’  physical experi-ence of their treatment (i.e. sensations that the patientexperiences during or after their treatment) are seldomdescribed in manual therapy research. This aspect of thepatients ’  experiences of a treatment requires further ex-ploration to develop a more global picture of the patientexperience during and after their consultation. * Correspondence: † Equal contributors 1 Centre for Chronic Disease Prevention & Management, College of Health & Biomedicine, Victoria University, Melbourne, Australia 2 Institute of Sport, Exercise & Active Living, Victoria University, Melbourne,AustraliaFull list of author information is available at the end of the article CHIROPRACTIC & MANUAL THERAPIES © 2015 Mulcahy and Vaughan; licensee BioMed Central. This is an Open Access article distributed under the terms of theCreative Commons Attribution License (, which permits unrestricted use,distribution, and reproduction in any medium, provided the srcinal work is properly credited. The Creative Commons PublicDomain Dedication waiver ( ) applies to the data made available in thisarticle, unless otherwise stated. Mulcahy and Vaughan  Chiropractic & Manual Therapies  (2015) 23:6 DOI 10.1186/s12998-015-0055-x  Recently Cross et al. [4] used a qualitative approach toinvestigate patients ’  expectations of osteopathic treat-ment in private United Kingdom practices and con-cluded these expectations are primarily related to thepatient-therapist interaction. Further, patients identifiedprofessional expertise and customer service as expecta-tions of osteopathic treatment. Drawing on this work,Leach et al. [11] used a quantitative approach to identify patient expectations of their osteopathic care. The topthree aspects of care highlighted by patients in this study were the ability to ask questions of the practitioner, ac-tive listening and respect. Again, the focus was very much on the patient-therapist interaction. Although,these two studies provide valuable insights into what pa-tients expect from an osteopathic treatment, patients ’ cognitive, emotional and sensory responses to osteo-pathic treatment have not previously been established orincluded in commonly utilised patient reported out-comes measures (PROMs). Previous work by Mulcahy &Vaughan [12] investigated the patient-reported sensory experiences of Osteopathy in the Cranial Field (OCF)treatment. However these sensory experiences have notbeen validated in patients receiving general osteopathictreatment.The Patient Perception Measure  –  Osteopathy (PPM-O)was developed to enhance clinicians and clinical educatorsunderstanding of what patients perceive during osteo-pathic treatment. Items for inclusion in the PPM-O werebased on those used in a previous study to explore patientperception of OCF [12,13]. Confirmatory factor analysis The CFA was used to determine if the data fitted the 6domains identified by Mulcahy et al. [13]. CFA producesa variety of fit statistics indicating how well the data col-lected fits the proposed  a-priori  factor structure [14]. Arange of fit statistics should be generated because eachstatistic has different measurement properties [15,16]. The chi-square statistic is used to report the fit of thedata to the model and  p -values less then 0.05 indicate afit [17]. Whilst there is no agreement as to which type of fit statistics should be presented, in the current study the authors present a range of statistics to provide thereader with a more comprehensive representation of thedata fit. Fit statistics in the present study were in linewith those suggested by DiStefano & Hess [15] and in-cluded: the goodness of fit index (GFI), comparative fitindex (CFI), normed fit index (NFI), Tucker-Lewis index(TLI), root mean square residual (RMR) and the rootmean square error of approximation (RMSEA). The useof these fit statistics is also supported by other authors[17,18] and ensures that a range of global fit and relative fit indices are presented [15]. Rasch analysis Rasch analysis is part of the modern test theory (MTT)statistical technique group and is widely used in the de- velopment and analysis of questionnaires and measures.The approach was developed by Dutch mathematicianGeorge Rasch [19] and fit of the data to the Rasch modelis the desired outcome of the analysis [20]. Rasch ana-lysis is sample-independent compared to the sample-dependent analyses in classical test theory. In Raschanalysis the data is fitted to a mathematical model to de-termine if all respondents are responding to each itemin a manner dictated by the Rasch model. A range of sta-tistics related to the interaction between the questionnaireitems and the person responses (item-trait interaction) isgenerated. The item-trait statistics demonstrate the overallfit of the items and persons to the Rasch model [21]. Thisstatistic analyses how each item on the PPM-O relatesto all other items, and how each person is respondingto each item on the PPM-O. These statistics indicatehow the responses fit those expected by the Raschmodel. A Bonferonni-adjusted non-statistically signifi-cant chi-square indicates an overall fit of all personsand items to the Rasch model [21]. Rasch model itemand person fit is indicated by a fit residual standard de- viation (SD) of ± 1.5. Fit residual SDs outside of thisrange suggests that issues exist with the model fit of the items and/or persons. A Person Separation Index(PSI) is also generated to indicate the internalconsistency of the questionnaire being analysed and isinterpreted in the same way as Cronbach ’ s alpha [22].Fit of the individual items to the Rasch model is ana-lysed to ascertain whether misfitting items are impact-ing upon the overall model fit. Poor individual item fitis indicated by a fit residual of ± 2.5 and/or a statisti-cally significant chi-square probability [23]. In the caseof the PPM-O it may be that the probability of a pa-tient selecting a particular response on the Likert-typescale is not equal for all possible responses on thescale  –  this is referred to as a disordered threshold.The threshold is the point at which there is a 50%chance of a person selecting response 1 or response 2on a scale [21,23]. Where the threshold is disordered, respondents are selecting scale responses in a mannerthat is not consistent with the trait under investiga-tion. A disordered threshold may also result from per-sons answering the item having trouble differentiatingbetween the scale responses (i.e. likely, very likely,highly likely). It is possible to rescore the item to re-solve the threshold disorder [21,24]. The category  probability curves are used to ensure that each re-sponse on the scale is being used in an ordered man-ner. Each response option for the item should havethe highest probability of being selected at some pointalong the person location. Mulcahy and Vaughan  Chiropractic & Manual Therapies  (2015) 23:6 Page 2 of 12  Differential item functioning (DIF) is the investigationof how an item functions with respect to a person factorsuch as age or gender. In the present study, age, gender,satisfaction with life [12] and meaningful daily activity [12,25] were investigated to see if they had an impact on the way a person answers an item or items on the PPM-O. Each person factor is investigated separately to ascer-tain the impact of it on the fit of the data to the Raschmodel. Where an item demonstrates DIF (through astatistically significant Bonferroni adjusted chi-squareprobability), it can be removed or recalibrated (e.g. thoseunder 20 years of age can be split from those above20 years of age).Misfit of individual persons to the Rasch model is indi-cated by a fit residual of±2.5 [23]. A person is said tomisfit when their response to each of the item on aquestionnaire, in this case on items on the PPM-O, doesnot follow the prediction of the Rasch model for how that person should have responded to the item. Misfit-ting persons can impact on the Rasch model [21] andthey will often be removed from further analysis.The dimensionality of the measure is important be-cause this demonstrates whether it is measuring a singleunderlying construct [26]. Local dependency is wherethe response to one item dictates the response to an-other item [23,26] and this can inflate the PSI [27]. Where local dependency is identified (the PSI decreases)one of the correlating items will need to be deleted.Next, the dimensionality is assessed using a PrincipalComponents Analysis (PCA). The PCA is used to gener-ate the  ‘ Rasch factor ’  (factor 1) and display the positively and negatively loaded items. These items are then analysedusing a paired t-test to examine whether the positive andnegative loaded items are statistically significantly different[21]. Where no statistically significant difference exists,the questionnaire is thought to be unidimensional [21]. Study aim The aim of the present study is to explore the construct validity of the Patient Perception Measure - Osteopathy (PPM-O) using both confirmatory factor analysis andRasch analysis. Methods This study was approved by the Victoria University (VU,Melbourne, Australia) and Southern Cross University (SCU, Lismore, Australia) Human Research EthicsCommittees. Participants Patients attending the student-led osteopathy teachingclinics at VU and SCU were invited to participate in thestudy. At the conclusion of their treatment, patientswere invited to complete the PPM-O questionnaire by the reception staff. An Information to Participants sheetwas provided to each potential participant and consentto participate was implied by completing the question-naire. Completed PPM-O and demographic question-naires were placed in a secure box in the reception areaand collected by one of the authors weekly. Only theauthors had access to the collected data. Measure The Patient Perception Measure  –  Osteopathy (PPM-O)is based on the items from a previously developed 22-item questionnaire divided into 6 domains based onan  a-priori  theoretical structure [13]. The domainsidentified were Education & Information, Cognition &Fatigue, Effectiveness of Osteopathic Treatment, PerceivedEmotional Responses to Osteopathic Treatment, PerceivedPhysical Responses to Osteopathic Treatment, and Appli-cation of Osteopathic Principles.Participants were also asked to complete a single-pagedemographic questionnaire. Items on the demographicquestionnaire included age, gender, employment status,current medication usage and whether the participantsuffers, or suffered from, one of the seven major illnessesidentified by the Australian Institute of Health andWelfare [2]. Participants were also asked about 2 globalitems; satisfaction with life (SWL) and meaningfulness of daily activity (MDA) [25]. These global items were ratedon a Likert-type scale from 0 to 6, anchored at each end.The anchors for SWL were  ‘ not at all satisfied ’  (0) and ‘ extremely satisfied ’  (6), and the MDA anchors were  ‘ notat all meaningful ’  (0) and  ‘ extremely meaningful ’  (6).Higher scores on these global items indicated greatersatisfaction with life and meaningfulness of daily activity respectively. Elements of the demographic data wereused to examine the differential item function in theRasch analysis. Data analysis Data were entered into SPSS Version 21 (IBM Corp,USA) for analysis. The analysis of the PPM-O took placein two stages: 1) confirmatory factor analysis (CFA); and2) Rasch analysis. The CFA was conducted with AMOSVersion 21 (IBM Corp, USA) using the Maximum Like-lihood Method approach. The recommended fit statisticcut-off values for each analysis used in the present study are presented in Table 1. As the data were not normally distributed, a bootstrapping procedure was applied foreach of the two models (22 item PPM-O & 13-itemPPM-O), and 1000 iterations of the data were generated.Data were exported from SPSS to RUMM2030 [28] toperform the Rasch analysis using the Partial CreditModel [21] as the  ‘ distance ’  between the response cat-egories for each item were thought not to be equal. Thiswas confirmed with a statistically significant Likelihood Mulcahy and Vaughan  Chiropractic & Manual Therapies  (2015) 23:6 Page 3 of 12  Ratio Test (p <0.05). Graphical and numerical thresholdmaps, and graphical category probability curves wereproduced in addition to the statistical analysis. In thepresent study, the decision to remove items demonstrat-ing DIF was made  a priori  to make the PPM-O easy toadminister and interpret. Results One hundred and eighty four questionnaires were re-ceived however 32 (18%) contained incomplete data andwere subsequently removed from the CFA - 152 ques-tionnaires were analysed in the CFA. Data from all 184questionnaires were entered into RUMM for the Raschanalysis however one questionnaire did not containenough data to be able to analysed and was removed.One hundred and eighty three responses (n=183) wereanalysed in the Rasch analysis.The mean age of the respondents was 35.8 years(+/ −  15.1 years) and 60.5% (n = 92) were female. Em-ployment status was shared between employed (n = 63,41.4%) and students who were employed (n = 55,36.2%). Participants were generally satisfied with theirlife (4.03 +/ −  0.73) and found their daily activity mod-erately meaningful (3.96 +/ −  0.78). No participant indi-cated they were not satisfied with their life or thattheir daily activity was not meaningful (correspondingto a score of 0). Data were collected related to theseven major Australian illnesses [2], and prevalence of these disorders were: cardiovascular disease (n=9, 5.9%);cancer (n=3, 2.0%); mental health disorder (n=19,12.5%); diabetes (n=4, 2.6%); chronic respiratory com-plaint (n=13, 8.6%); and the combined arthritis andmusculoskeletal complaints (n=65, 42.8%).Descriptive statistics for the participant responses tothe PPM-O are presented in Table 2. Confirmatory factor analysis 1 Data were initially fitted to the  a-priori  6 domain struc-ture proposed by Mulcahy et al. [13]. The path diagramfor this model is presented in Figure 1 and the fit statis-tics are presented in Table 1. The data did not fit themodel as indicated by the statistically significant chi-square probability (p<0.001) however the GFI wasapproaching the recommended value. Rasch analysis 1 The data for the 22 item PPM-O did not fit the Raschmodel (  χ  2=171.95, df=44, p<0.0001). The PSI was0.783 indicating borderline internal consistency. Thestandard deviation fit residuals for both items (1.22) andpersons (0.82) were not greater than 1.5. A poor fit re-sidual (>2.5) was identified for item 18 and statistically significant  χ  2 values for items 2 and 15, indicating apoor fit of these items to the Rasch model. Disorderedthresholds were demonstrated for all items except 5 – 7,11 and 15. The completed questionnaire from one per-son did not contain enough data and was removed,therefore 183 responses were analysed. DIF was identi-fied for SWL and MDA at item 20 (I feel alone afterosteopathic treatment). Those participants with low SWL and MDA scores were more likely to endorse thisitem highly (agree or strongly agree). Assessment of di-mensionality indicated that the 22-item questionnairewas not unidimensional. PPM-O Modification Confirmatory factor analysis Given the lack of model fit in the first CFA and themultidimensional nature of the 22-item PPM-O, con-firmed through the initial Rasch analysis, the CFA modelwas modified to establish a multifactorial structure. Itemcovariances were analysed in order to modify the 22-item PPM-O. An item was removed if the covariancewith another item was greater than 10 or did not fit ontoa factor. Figure 2 demonstrates the correlation betweeneach of the 6 domains. Strong relationships were identi-fied between the Education, Effectiveness, Physical andOsteopathic Principles factors. The items in these Table 1 CFA fit statistics for the two versions of the PPM-O Statistic Recommended value 22-item PPM-O 13-item PPM-O χ 2 NA 357.23 130.46 χ 2 p-value <0.05 >0.0001 >0.0001df NA 194 64 χ 2/df < or = 2 1.84 2.04Goodness of fit index (GFI) > or = 0.9 0.828 0.879Comparative fit index (CFI) > or = 0.9 0.841 0.855Normed fit index (NFI) > or = 0.9 0.717 0.757 Tucker-Lewis index (TLI) > or = 0.9 0.811 0.824Root mean square residual (RMR) As close to 0 as possible 0.048 0.054Root mean square error of approximation (RMSEA) < or = 0.08 0.075 (CI 0.062-0.087) 0.083 (CI 0.062-0.103) Mulcahy and Vaughan  Chiropractic & Manual Therapies  (2015) 23:6 Page 4 of 12  domains were combined into a single factor calledEducation & Effectiveness. The items remaining in theCognition and Emotion factors were combined to formthe Cognition & Fatigue factor. This process produceda 2-factor, 15 item version of the PPM-O (Figure 2). Rasch analysis The revised 2-factor, 15-item PPM-O was Rasch ana-lysed. As two factors had been identified, they wereindependently analysed in order for each factor to fit theRasch model. Rasch analysis of the education & effectiveness factor  The Education & Effectiveness factor demonstrated fit tothe Rasch model (  χ  2=35.47, df=18, p=0.008). The PSIwas 0.763. The fit residual SD for items was 0.77 and0.95 for persons. None of the items demonstrated statis-tically significant chi-square probabilities or fit residualSDs. Disordered thresholds were observed for all itemsexcept 8, 9 and 16 (Additional file 1). There were 15misfitting persons (out of 183 responses) and none of the items demonstrated DIF for any of the person fac-tors. In order to achieve model fit, a number of modifi-cations were made. Items 2, 4, 5, 9, 11, and 14 wererescored (Additional file 1) and this resolved the disor-dering for all items. Eighteen misfitting persons were re-moved from the analysis - these persons were notsignificantly different from the analysed persons with re-gard to demographics. All items demonstrated orderedthresholds and there was no DIF for any item. Therewere no residual correlations. The PCA and subsequentpaired t-test of the positively and negatively loadingitems on the Rasch factor were statistically significantindicating the factor was unidimensional. With the itemrescoring, the possible total score for this factor is 39.The mean person-item distribution for this factor is 2.35(Figure 3). The item fit statistics are presented inTable 3. Table 2 Descriptive statistics for the 22-item Patient Perception Measure  –  Osteopathy (PPM-O) Item Response options Min Max Mean Std. Dev. 1. The way that my osteopath explains my osteopathic treatment is Poor, fair, good, very good, excellent 3 5 4.52 0.572. The way my osteopath answers all of my questions is Poor, fair, good, very good, excellent 3 5 4.59 0.543. My osteopath treats me with respect Never, rarely, sometimes, mostly, always 3 5 4.97 0.214. The instructions my osteopath gives me regarding my home exerciseprogram arePoor, fair, good, very good, excellent 1 5 4.30 0.715. Osteopathic treatment has helped my condition Never, rarely, sometimes, mostly, always 3 5 4.40 0.606. The way my management plan was explained to me was Poor, fair, good, very good, excellent 2 5 4.26 0.717. The osteopathic treatment I have received has improved my quality of life Never, rarely, sometimes, mostly, always 2 5 4.34 0.658. As a result of osteopathic treatment, my general health is Poor, fair, good, very good, excellent 2 5 3.89 0.729. During my treatment, the questions my osteopath asked were Poor, fair, good, very good, excellent 3 5 4.34 0.6310. After my osteopathic treatment I felt like my whole body was treatedrather than just one areaNever, rarely, sometimes, mostly, always 2 5 4.25 0.7911. Osteopaths at this clinic talk about the body ’ s ability to heal itself Never, rarely, sometimes, mostly, always 1 5 3.80 0.94 12. Osteopathic treatment makes me feel vague  Never, rarely, sometimes, mostly, always 1 12 2.15 1.24 13. I cannot focus on tasks after my osteopathic treatment   Never, rarely, sometimes, mostly, always 1 5 1.86 0.9214. I feel calmer after my osteopathic treatment Never, rarely, sometimes, mostly, always 1 5 4.28 0.7215. Osteopathic treatment makes no difference to my frame of mind Never, rarely, sometimes, mostly, always 1 5 2.14 1.1116. How helpful is osteopathic treatment in managing your condition Poor, fair, good, very good, excellent 2 5 4.22 0.69 17. I feel sad after osteopathic treatment   Never, rarely, sometimes, mostly, always 1 5 1.22 0.58 18. I feel tired after osteopathic treatment   Never, rarely, sometimes, mostly, always 1 5 2.48 1.04 19. I am anxious after osteopathic treatment   Never, rarely, sometimes, mostly, always 1 3 1.16 0.38 20. I feel alone after osteopathic treatment   Never, rarely, sometimes, mostly, always 1 4 1.11 0.3721. I feel less pain after osteopathic treatment Never, rarely, sometimes, mostly, always 1 5 4.05 0.81 22. I find it hard to concentrate after my osteopathic treatment   Never, rarely, sometimes, mostly, always 1 4 1.82 0.88 Note: negatively phrased items are in italics and require rescoring prior to analysis.Legend  –  response option scoring.Poor (1), fair (2), good (3), very good (4), excellent (5).Never (1), rarely (2), sometimes (3), mostly (4), always (5).NB these scores are reversed for negatively phrased items. Mulcahy and Vaughan  Chiropractic & Manual Therapies  (2015) 23:6 Page 5 of 12
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!