A) Introduction
This is a write-up for the second inter-rater reliability analysis as part of the MHIN screening. The rationale for this analysis has been provided elsewhere. The screening process is described here - including a (preliminary) decision rule indicating in which case an intervention is screened “in” or “out”. As for the first round (results here), 6 mental health interventions have been screened by 8 raters (however currently only 6 as of 4th April). B) Key Findings and inferences (preliminary): In general:
Individual rating behaviour:
C) Discussion
→ altering the threshold to 8 (i.e., the CE-estimate of an intervention needs to be bigger than 8 to be screened “in”) gives a weighted proportion of agreement of 0.89 (round 1) and 0.65 (round 2). Out of 12 interventions, we would clearly screen “in” 3. → altering the threshold to 9 (i.e., the CE-estimate of an intervention needs to be bigger than 8 to be screened “in”) gives a weighted proportion of agreement of 0.89 (round 1) and 0.65 (round 2). Out of 12 interventions, we would clearly screen “in” 3. From these data, we can see that setting the threshold is very important for inter rater agreement as well as our overall sensitivity.
D) Recommendations (in a nutshell, up for debate):
|