Chapter 1 Introduction 1.1 Background of the study 1.2 Purpose of the study and research questions 1.3 Significance of the study 1.3.1 Theoretical significance 1.3.2 Methodological significance 1.3.3 Practical significance 1.4 A note on terms 1.4.1 Mock essays 1.4.2 Rater effects/variability/biases/errors 1.4.3 Rating accuracy 1.4.4 Scoring/score validity 1.4.5 Scoring/rating expertise 1.4.6 Cognitive strategy 1.4.7 Meta-cognitive strategy 1.4.8 Rating session 1.4.9 Feedback iteration 1.4.10 Counter-balanced design 1.4.11 The correlation coefficient 1.4.12 The mixed feedback 1.5 The structure of chapters Chapter 2 Literature review 2.1 Introduction 2.2 Rater variability as a serious threat to score validity in performance assessments 2.2.1 Models of performance assessment of writing 2.2.2 Factors that affect rater performance 2.2.3 Rater variability from the psychometric perspective 2.2.4 Rater variability from the hermeneutic perspective 2.2.5 Rater cognition studies 2.3 Rater training as a routine measure to control rater variability 2.3.1 Overview of procedures of rater training 2.3.2 Studies on rater training effectiveness 2.3.3 Critical issues in rater training 2.4 Rater feedback as an important part of training 2.4.1 Studies on rater feedback effectiveness 2.4.2 Some common problems 2.5 Methodological considerations 2.5.1 Content of the feedback 2.5.2 Delivery of the feedback 2.6 Summary Chapter 3 Method 3.1 Introduction 3.2 Research Design 3.2.1 The mixed methods design of this study 3.2.2 The quasi-experimental nature of this study 3.2.3 Context of the study 3.2.4 Materials 3.2.5 Participants 3.2.6 Instruments 3.2.7 Pilot study 3.2.8 Procedures 3.3 Data analysis 3.3.1 MFRM analysis of ratings 3.3.2 Analysis of degree of certainty 3.3.3 Verbal protocol analysis 3.3.4 The questionnaire of raters' perceptions of mixed'feedback 3.3.5 The questionnaire of raters' evaluation of feedback effectiveness 3.3.6 Interview with those who are negative with the feedback effectiveness 3.4 Summary Chapter 4 Results 4.1 Findings for Research Question 1 : The effect of the mixed feedback on rater variability 4.1.1 Findings for RQ1 （ 1 ） : The effect of the mixed feedback on rater severity 4.1.2 Findings for RQ1 （2） : The effect of the mixed feedback on rater internal consistency 4.1.3 Findings for RQ1 （3） : The effect of the mixed feedback on rater bias 4.1.4 Findings for RQ1 （4） : The effect of the mixed feedback on rater central tendency 4.1.5 Findings for RQ1 （5） : The effect of the mixed feedback on rater severity DRIFT 4.2 Findings for Research Question 2 : The extent to which EG raters can incorporate the mixed feedback into rating behavior when it is provided repeatedly over 3 iterations 4.2.1 First round results 4.2.2 Second round results 4.3 Findings for Research Question 3.The effect of the mixed feedback on raters' degree of certainty for their ratings 4.3.1 First round results 4.3.2 Second round results 4.4 Findings for Research Question 4.The effect of the mixed feedback on raters' decision.making 4.4.1 Findings for RQ4 （ 1 ） : The effect of the mixed feedback on raters' decision-making in terms of text features 4.4.2 Findings for RQ4 （2） : The effect of the mixed feedback on raters' decision-making in terms of information processing behaviors 4.5 Findings for Research Question 5 : Raters' perceptions of the mixed feedback ... 4.5.1 First round results 4.5.2 Second round results 4.6 Summary Chapter 5 Discussion 5.1 Discussion of Research Question 1 : What effect does the mixed feedback have on rater variability7 5.1.1 Methodological considerations 5.1.2 Interpretation of the results 5.2 Discussion of Research Question 2.To what extent can EG raters incorporate the mixed feedback into rating behavior when it is provided repeatedly over 3 iterations7 5.2.1 Methodological considerations 5.2.2 Interpretation of the results 5.3 Discussion of Research Question 3: What effect does the mixed feedback have on raters' degree of certainty for their ratings7 5.3.1 Methodological considerations 5.3.2 Interpretation of the results 5.4 Discussion of Research Question 4: What effect does the mixed feedback have on raters' decision-making? 5.4.1 Methodological considerations 5.4.2 Interpretation of the results 5.5 Discussion of Research Question 5 : What are raters' perceptions of such feedback? 5.5.1 Methodological considerations 5.5.2 Interpretation of the results 5.6 General discussion: Establish an internal link between findings 5.6.1 Linking findings of RQ1, RQ3 and RQ4 5.6.2 Linking findings of RQ1, RQ3, RQ4 and RQ5 5.7 Summary Chapter 6 Conclusions 6.1 Summary of findings 6.2 Implications of the study 6.2.1 Theoretical implications 6.2.2 Methodological implications 6.2.3 Practical implications 6.3 Limitations of the study and direction for future research References Appendices
鄂公網(wǎng)安備 42010302001612號 