Education

Don't walk: Rasch to join the questionnaire trend!

Description
1. + Don’t walk: Rasch to join the questionnai re trend! Ritsumeik an University Keita Kikuchi Kanagaw a University J. W. Lake Fukuoka Women’s University 2. + Get…
Categories
Published
of 61
24
Categories
Published
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Share
Transcript
  • 1. + Don’t walk: Rasch to join the questionnai re trend! Ritsumeik an University Keita Kikuchi Kanagaw a University J. W. Lake Fukuoka Women’s University
  • 2. + Get ready to be Rasched… (This is supposed to be a joke. Laugh.) Structure of this workshop Ritsumeik an University Keita Kikuchi Kanagaw a University J. W. Lake Fukuoka Women’s University
  • 3. + Rasch concepts Terminology and explanations Structure of this workshop Ritsumeik an University
  • 4. + Structure of this workshop J. W. Fukuoka Women’s University Steps to create a questionnaire Lake Constructs, concepts, items, piloting, evaluation, revision…
  • 5. + e8 e9 e10 e11 Demotivation: An example The flaws of traditional FA and why Rasch can help questionnaire creation e1 e2 MET e3 ONE PRON e5 Structure of this workshop Keita Kikuchi Kanagaw a University Teacher Behavior GETA .69 .85 .75 e4 .69 EXPL .77 Environment AUD e14 TOPC e13 INTR e12 .67 .74 .76 .70 VID e15 e6 GRAM Experience of difficulties e7 VOCI VOC TEST SELF .72 .68 .66 .68 Lack of Interest NOP e22 NON e21 NOG e20 NOI e19 NOF e18 .82 .83 .78 .77 .81 NOTU .74 demotivation .74 .68 .69 d1 d2 d3 d4 .66 .64 FRN e16 .61 MST e17 .50
  • 6. + Matthew Apple Ritsumeikan University Department of Communication International Communication Program Rasch concepts
  • 7. + Rasch terms  Rasch log-odds (logits)  Rasch measures (logit scores)  Infit/Outfit (means sq. and z-score)  Item difficulty / endorsability  Person/item reliability / separation  Construct validity and unidimensionality (not strictly speaking Rasch, but…)  Rasch Principal components analysis (Rasch PCA)  Loadings  Contrasts and Residuals A probabilistic model
  • 8. + Logits (log-odds) The probability of a person correctly answering an item 50% of the time
  • 9. + Fit 0.75 to 1.3 logits 0.60 to 1.4 logits “Within 2 standard deviations of the mean” “Only Outfit z-scores of 3.0”
  • 10. + Separation The ratio of error-free variance and observed variance (Fisher, 1992) The number of groups distinguishable by the measurement instrument (Wilson, 2005; Wright, 1996)
  • 11. + Item map Persons and items on the same linear logit scale
  • 12. + The “Line” Items and persons on the same scale Item-person map (or) Wright map Ben Wright
  • 13. + The “Line” Items and persons on the same scale Item-person map (or) Wright map
  • 14. + How Rasch works for questionnaires  Likert-scale data  Likert-type category data  Questionnaires do not produce true interval but ordinal data  The steps in the “scale” can be conceived as thresholds (τ)  named after Thurston, originator Refer to Andrich (1977, 1978) of factor analysis)
  • 15. + How Rasch works for questionnaires Refer to Andrich (1977, 1978)
  • 16. + How Rasch works for questionnaires Refer to Andrich (1977, 1978) N 1 2 3 4 5
  • 17. + How Rasch works for questionnaires Refer to Andrich (1977, 1978) N 1 2 3 4 5 1 + 3 = 4
  • 18. + How Rasch works for questionnaires Refer to Andrich (1977, 1978) N 1 2 3 4 5 SD + N = A? 1 + 3 = 4?
  • 19. + J W Lake Fukuoka Women’s University Steps to create a questionnaire
  • 20. Steps in scale development: Issues to consider (Netemeyer, Bearden, Sharma, 2003) Step 1: Construct definition and content domain The importance of clear construct definition, content domain, and the role of theory. Construct dimensionality: unidimensional, multidimensional, or a higher-order construct? Determine the purpose of the scale: measurement or correlational analysis or model building
  • 21. Step 2: Generating and judging measurement items Theoretical assumptions about items (e.g., domain sampling) Generating potential items and determining the response format How many items as an initial pool Dichotomous vs. polytomous response formats Item wording issues  The focus on “content” validity in relation to theoretical dimensionality Item judging (expert and layperson) --- the focus on
  • 22. Step 3: designing and conducting studies to develop and refine the scale Pilot testing as an item-trimming procedure The use of several samples from relevant populations for scale development Designing the studies to test psychometric properties Initial item analyses via exploratory factor analyses (EFAs) Initial item analyses and internal consistency estimates Retaining items for the next studies EFA may be useful for correlational analysis or model building
  • 23. Step 4: Finalizing the scales The importance of several samples from relevant populations Designing the studies to test the various types of validity Item analysis via EFA The importance of EFA consistency from Step 3 to Step 4 Deriving an initial factor structure—dimensionality and theory Item analyses and confirmatory factor analyses (CFAs) Testing the theoretical factor structure and model
  • 24. + e8 e9 e10 e11 Keita Kikuchi Kanagawa University e1 e2 MET e3 ONE PRON e5 e6 Demotivatio Teacher Behavior n: An example GETA .69 .85 .75 e4 .69 EXPL .77 Environment AUD e14 TOPC e13 INTR e12 .67 .74 .76 .70 VID e15 GRAM Experience of difficulties e7 VOCI VOC TEST SELF .72 .68 .66 .68 Lack of Interest NOP e22 NON e21 NOG e20 NOI e19 NOF e18 .82 .83 .78 .77 .81 NOTU .74 demotivation .74 .68 .69 d1 d2 d3 d4 .66 .64 FRN e16 .61 MST e17 .50
  • 25. Example Study focused on specific external /internal forces that Japanese high school students may experience which might cause their motivation to be reduced or diminished  administered the questionnaire asking high school students to report what diminished their motivation to study in their high school days, which contained 40 Likert-scale questions (4-points) analyzed the quantitative data using a confirmatory factor analysis using Amos and Rasch PCA of the residuals using Winsteps.  If you’d like to read this process thoroughly, please locate Kikuchi (forthcoming).
  • 26. Demotivation Dörnyei (2001) Definition of demotivation  “specific external forces that reduce or diminish the motivational basis of a behavioral intention or an ongoing action” (p. 143). I expand this definition and explore demotivators including both internal and external forces.
  • 27. Previous Studies (Dörnyei, 1998)  Based on interviews with 50 secondary school students, he identified following as demotivators, the reason to get demotivated. 1. Teachers’ personalities, commitments, competence, teaching methods. 2. Inadequate school facilities (very big group, not the right level or frequent change of teachers). 3. Reduced self-confidence due to their experience of failure or success. 4. Negative attitude toward the foreign language studied. 5. Compulsory nature of the foreign language study. 6. Interference of another foreign language that pupils are studying. 7. Negative attitude toward the community of the foreign
  • 28. Previous Studies (Kojima, 2004, p.42) Languag e Level English Learning Demotivation Learning situation Amoun t of study .58 .86 .71 .46 .41 Gramm ar Readin g Self-confidenc e .89 personality Writing Learning method Teacher Change of teaching style Teaching approach Memori zing Vocab. Learner Level Listening problem Class atmospher e .77 .89 .89 .85 .90 .92 .80 .83 .85 GFI = 0.906 AGFI = 0.890 RMSEA = 0.052
  • 29. Previous Studies  Kikuchi (2009)  47 university students  open-ended questionnaires  reflection on high school days  Kikuchi and Sakai (2009)  112 university students  a 35-item questionnaire with a 5-point scale  Sakai and Kikuchi (2009)  676 high school students  a 35-item questionnaire with a 5-point scale
  • 30. Common demotivation factors  Sakai and Kikuchi(2009)  F1: Learning Contents and Materials,  F2: Teachers’ Competence and Teaching Styles  F3: Inadequate School Facilities [Classroom Environment]  F4: Lack of Intrinsic Motivation  F5: Test Scores [Experience of Inferiority]  Kikuchi and Sakai (2009)  F1: Course Books  F2: Inadequate School Facilities  F3: Test Scores  F4: Non-Communicative Methods  F5: Teachers’ Competence and Teaching Styles  Both studies used a principal axis factor analysis using the direct oblimin rotation
  • 31. Six Original Constructs  Teachers: Teachers’ attitude, teaching competence, language proficiency, personality, and teaching style  Characteristics of classes: Course contents and pace, focus on difficult grammar or vocabulary, monotonous and boring lessons, a focus on university entrance exams and the memorization of the language  Experiences of failure: Disappointment due to test scores, lack of acceptance by teachers and others, and feeling unable to memorize vocabulary and idioms.  Class environment: Attitude of classmates, compulsory nature of English study, friends’ attitudes, inactive classes, inappropriate level of the lessons, and inadequate use of school facilities such as not using audio-visual materials  Class materials: Not suitable or uninteresting materials (e.g., too many reference books and/or handouts)  Lack of interest: Sense of English used at schools is not
  • 32. Participants (N=1,266)
  • 33. Method Materials background questions a 40-item questionnaire  “We would like to study the situations of English study in high schools. The following statements are possible demotivating factors for English learning. To what extent are these statements true for you? Answer based on your experience.”  Questions are revised from what I used in my previous studies (Kikuchi and Sakai, in-press; Sakai and Kikuchi, 2009). Example of Items (1=Strongly disagree, 2=disagree, 3=agree, and 4=Strongly agree)  Teachers made one-way explanations too often.  The number of students in classes was large.  A great number of textbooks and supplementary readers were assigned.  I lost my understanding of the purpose of studying English.
  • 34. Results of EFA An Exploratory Factor Analysis 40 items principal axis factor analysis with a promax rotation procedure a four-factor solution teachers behaviors class environment experiences of difficulties Lack of interest Only 22 items left to be included in Confirmatory Factor Analysis.
  • 35. Table 2:Factor Analysis of Demotivation No. Item descriptions F 1 F 2 F 3 F 4 Factor 1: Experience of difficulties(α = .87) i16 There were too many vocabularies that I did not understand in reading. 0.81 0.03 -0.14 -0.11 i15 I had difficulty in memorizing words and phrases. 0.80 -0.13 -0.01 -0.08 i13 I got low scores on tests (such as mid-term and final examinations). 0.79 -0.17 0.06 -0.09 i8 I did not understand grammar even though I studied. 0.75 -0.02 -0.19 0.05 i39 I started not to understand the content of the class. 0.71 0.09 -0.09 0.08 i7 I could not do as well on tests as my friends. 0.62 -0.11 0.10 -0.09 i33 I got lost in how to self-study for English lessons. 0.60 -0.02 0.03 0.09 Factor 2: Teacher behavior(α = .84) i18 I thought that the approach that teacher used was not good. -0.05 0.93 -0.14 0.00 i5 Teachers' explanations were not easy to understand. 0.03 0.86 -0.22 0.02 i17 Teachers made one-way explanations too often. 0.06 0.78 -0.07 -0.02 i6 Teachers' pronunciation of English was poor. -0.10 0.73 0.02 -0.07 i34 I could not get along with teachers. -0.10 0.68 0.12 0.05 i31 The pace of lessons was not appropriate. 0.04 0.63 0.06 -0.03 Factor 3: Class environment(α = .85) i28 Audio materials (such as CDs and tapes) were not used. -0.08 -0.01 0.82 -0.06 i23 The Internet was not used. -0.12 -0.16 0.81 0.04 i27 Topics of the English passages used in lessons were old. 0.04 0.02 0.71 -0.11 i35 Visual materials (such as videos and DVDs) were not used. -0.03 0.05 0.69 0.04 i10 My friends did not like English. 0.02 -0.05 0.67 -0.07 i40 The number of students in classes was large. -0.03 -0.11 0.65 0.13 Factor 4: Lack of Interest(α = .90) i3 I lost my understanding of the purpose of studying English. -0.13 0.00 0.01 0.91 i2 I lost my goal to be a speaker of English. -0.08 -0.04 0.03 0.85 i26 I think that I will not use English in my future. 0.03 0.01 -0.10 0.81 i11 I don’t have specific goals for studying English. 0.04 -0.04 0.01 0.79 i24 I lost my interest in English. 0.12 -0.02 0.01 0.71
  • 36. Method Analysis Rasch PCA of the residuals/Confirmatory factor analysis of these six factors Conventional factor analysis confirmed only four factors! Rasch PCA factor analysis /Confirmatory factor analysis of these four factors were conducted once again…
  • 37. + Enter the Rasch With Matt
  • 38. + Category utility Measures the distance between thresholds among the Likert-type categories (“steps” of the scale)
  • 39. + Rasch PCA output Loading Measure Infit means squared Outfit means squared Principal components analysis
  • 40. + Item fit analysis Measure Standard error Infit Outfit means squared & z
  • 41. + e8 e9 e10 e11 Keita Kikuchi Kanagawa University e1 e2 MET e3 ONE PRON e5 e6 Demotivatio Teacher Behavior n, Part Deux: The Rasched GETA .69 .85 .75 e4 .69 EXPL .77 Environment AUD e14 TOPC e13 INTR e12 .67 .74 .76 .70 VID e15 GRAM Experience of difficulties e7 VOCI VOC TEST SELF .72 .68 .66 .68 Lack of Interest NOP e22 NON e21 NOG e20 NOI e19 NOF e18 .82 .83 .78 .77 .81 NOTU .74 demotivation .74 .68 .69 d1 d2 d3 d4 .66 .64 FRN e16 .61 MST e17 .50
  • 42. Results  Rating Scale Instrument Quality Criteria (based on Fisher, 2007) Criterion Poor Fair Good Very Good Excellent Item Model Fit Mean-Square < 0.33 - >3.0 0.34 - 2.9 0.5 - 2.0 0.71 - 1.4 0.77 - 1.3 Person and item measurement <.67 .67-.80 .81-.90 .91-.94 >.94 reliability Variance in data explained by measures <50% 50-60% 60-70% 70-80% >80% Unexplained variance in 1st contrast of PCA residuals >15% 10-15% 5-10% 3-5% <3%
  • 43. Table 1: Variance in measure explained by each demotivator construct. Six Demotivator Constructs Variance explained by measure Unexplained variance by measure Unexplained variance explained by 1st contrast Teachers 57.0% 43.0% 12.3% Characteristics of classes 45.4% 54.6% 9.7% Experiences failure 57.1% 42.9% 12.7% Class environment 40.4% 59.6% 13.3% Class materials 55.1% 44.9% 10.8% Lack of interest 62.1% 37.9% 12.8%
  • 44. Results –Rasch PCA- Loadings of No. Item description Logit Score Infit MNSQ Outfit MNSQ Contrasts Factor loadings 1. teachers(k=6, Rp=0.99, Gp=11.02) 15 Teachers shout or got angry 0.97 1.42 1.38 0.68 5 Teacher asked us to use accurate grammar 0.54 1.26 1.25 0.68 14 Teachers explanation not easy -0.85 0.79 0.85 -0.63 11 Teachers bad pronunciation 0.16 0.98 0.91 -0.45 40 Teachers’ bad teach method -0.31 0.75 0.72 -0.44 13 Teachers one-way explanation -0.51 0.83 0.84 -0.03 2. Characteristics of Classes (k=9, Rp=0.98, Gp=6.53) 10 Inappropriate pace of lesson -0.10 1.00 1.04 0.58 41 Monotonous class -0.22 0.98 1.04 0.58 1 Rare chance of communication -0.14 1.19 1.24 0.45 2 Focused on translation 0.06 0.81 0.83 -0.57 3 Focused on grammar -0.35 0.96 1.01 -0.56 42 Amount to study for mid-term/final tests -0.32 1.19 1.29 -0.20 6 Required memorizing passages in textbooks 0.26 0.94 0.95 -0.15 43 Amount of handout distributed 0.11 0.94 0.95 -0.13 4 Focused on college entrance exam. Prep. 0.69 0.90 0.89 -0.12
  • 45. No. Item description Logit Score Infit MNSQ Outfit MNSQ Factor loadings 3. Experiences of Failure (k=6, Rp=0.99, Gp=8.89) 27 Did not do well on tests compared w friends 0.64 1.04 1.06 0.76 8 low scores on school test 0.25 0.84 0.84 0.68 37 Did not understand grammar -0.15 1.07 1.07 -0.64 36 Did not understand class 0.23 1.01 1.01 -0.56 9 Get lost in self-study -0.26 1.05 1.07 -0.12 7 Could not memorize vocabulary and idiom -0.71 0.95 0.96 -0.05 4. Class Environment (k=6, Rp=0.98, Gp=7.40) 31 English being compulsory subject -0.73 1.08 1.20 0.82 26 Too many students in class 0.00 1.05 1.00 0.23 22 Video and DVDs not used -0.41 0.91 0.90 0.05 25 Audio not used 0.04 0.87 0.85 -0.59 29 Friends did not like English 0.67 0.97 1.03 -0.42 23 Internet not used 0.43 1.08 0.99 -0.31
  • 46. No. Item description Logit Score Infit MNSQ Outfit MNSQ Factor loadings 5. Class Materials (k=6, Rp=1.00, Gp=7.30) 16 Topics of Passages uninteresting 0.14 1.06 1.09 0.57 20 Topics of Passages old 1.42 0.96 0.95 0.56 35 Unclear answers to Questions 0.22 1.01 1.03 0.41 19 too much reading 0.44 0.97 1.00 0.19 44 Many difficult Vocabulary -1.21 0.98 1.00 -0.56 18 Sentences were difficult to read -0.97 0.98 1.03 -0.55 17 Passages too long -0.04 1.00 1.01 -0.47 6. Lack of Interest (k=5, Rp=0.98, Gp=6.87) 34 No goal for being a person who can use Eng 0.31 0.92 0.89 0.75 39 No need studying English 0.63 0.85 0.84 0.73 33 Lost interest in English -0.30 1.15 1.14 -0.56 32 Lost purpose of study English -0.59 1.06 1.06 -0.42 46 No use of English in the future -0.05 0.97 0.95 -0.31
  • 47. Figure 1: CFA of 6 factor models of demotivators Notes. GFI=800 CFI=808 RMSEA= .072 e6 i6 e5 i17 .76 teacher .82 e4 i18 .60 e3 i30.73 e1 i36 .56 e2 i34 .68 e7 i5 .74 .47 .60 .55 lesson e14 i4 e13 i19 e12 i20 e11 i21 .60 e10 i22 .58 e9 i25 e8 i29 .61 .59 e15 i31 .59 e16 i32 .46 environment i10 e17 i23 e18 i28 e19 i35 e20 i37 e21 i40 e22 .63 .67 .71 .70 .65 .62 material i1 e23 i9 e24 i12 e25 i14 e26 i16 e27 i27 e28 .53 .57 .50 .56 .59 .57 interest i2 e29 i3 e30 i11 e31 i24 e32 i26 e33 .82 .83 .77 .79 .81 failure i39 e35 i33 e36 i15 e37 i13 e38 i8 e39 i7 e40 .68 .74 .71 .67 .60 .67 i38 e41 .68 .79 .37 .27 .59 .54 .70 .56 .93 .79 .54 .88 .52 .73 .63 .57
  • 48. Figure 2. CFA of 4 factor models Notes. GFI=916 CFI=926 RMSEA= .062 e3 ONEW PRON Teacher Behavior e1 GETA e2 MET .69 .85 .75 e4 .69 e5 EXPL .77 Environment AUD e14 TOPC e13 INTR e12 .67 .74 .76 .70 VID e15 e6 GRAM e9 e10 Experience of difficulties e7 VOCI e8 VOC TEST SELFS .71 .67 .66 .68 Lack of Interest e11 NOP e22 NON e21 NOG e20 NOI e19 NOU e18 .82 .83 .79 .77 .81 NOTU .75 .67 .64 FRN e16 .61 MST e17 .46 .51 .44 .53 .23 .35
  • 49. e3 ONE PRON Teacher Behavior e1 GETA e2 MET .69 .85 .75 e4 .69 e5 EXPL .77 Environment AUD e14 TOPC e13 INTR e12 .67 .74 .76 .70 VID e15 e6 GRAM e9 e10 Experience of difficulties e7 VOCI e8 VOC TEST SELF .72 .68 .66 .68 Lack of Interest e11 NOP e22 NON e21 NOG e20 NOI e19 NOF e18 .82 .83 .78 .77 .81 NOTU .74 demotivation .74 .68 .69 d1 d2 d3 d4 .66 .64 FRN e16 .61 MST e17 .50 GFI=.911 NFI=.905 CFI=.920 RMSEA=.064 SRMR = .055 Figure 3. My tentative model of demotivation
  • 50. Results –Rasch PCA of four factor models- Four Demotivator Constructs Variance explained by measure Unexplained variance by measure Unexplained variance explained by 1st contrast Teachers 58.5% 41.5% 10.7% Experience of difficulties 58.0% 42.0% 12.2% Class environment 39.9% 60.1% 13.3% Lack of interest 54.8% 45.2% 15.1%
  • 51. This is how the poor factor was working…
  • 52. An activity for this workshop  Let’s try to make item bank of questionnaire items together for your practice. Topic is Demotivating factors in English education for communication in Japan. With your partner, think of constructs first and write items for each construct. How many constructs? How many items for each of them? Remember “the Line”! Please use the questionnaire items that you have in your handout about demotivating factors in English education in high school English classroom to generate your discussion.
  • 53. References  Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.  Dörnyei, Z. (2001). Teaching and researching motivation. Harlow: Longman.  Fisher, W. P. (2007). Rating Scale Instrument Quality Criteria. Retrieved November 25, 2007, from http://www.rasch.org/rmt/rmt211m.htm  Kikuchi, K. (2009). Student demotivation in Japanese hi
  • We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks
    SAVE OUR EARTH

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!

    x