Farrell et al: Evaluating validity evidence for 2 instruments developed to assess students' surgical skills in a simulated environment
Veterinary Surgery 5, 2022

🔍 Key Findings

39 of 40 checklist items for simulated OVH surgical assessment had good content validity (CVI = 0.81)
Only 1 of 6 items from the OSATS GRS (respect for tissue) met inclusion criteria (CVI = 0.80)
Checklist showed strong reliability (G-coefficient = 0.85) for moderate-stakes exams
Modified OSATS GRS showed acceptable reliability (G-coefficient = 0.79)
Two raters needed for acceptable reliability in high-stakes exams when using the checklist
Minimal interrater bias found; variance largely due to interaction among student, rater, and item
Digital recordings were a reliable method of evaluating surgical performance
Study supports using checklist over OSATS GRS for assessing preclinical students on simulated models

Simini Surgery Review Podcast

Thanks for the feedback!
‍We'll keep fine-tuning the articles vault.

Oops — didn’t go through.
‍Mind trying that again?

← Articles

Farrell et al: Evaluating validity evidence for 2 instruments developed to assess students' surgical skills in a simulated environment
Veterinary Surgery 5, 2022

🔍 Key Findings

39 of 40 checklist items for simulated OVH surgical assessment had good content validity (CVI = 0.81)
Only 1 of 6 items from the OSATS GRS (respect for tissue) met inclusion criteria (CVI = 0.80)
Checklist showed strong reliability (G-coefficient = 0.85) for moderate-stakes exams
Modified OSATS GRS showed acceptable reliability (G-coefficient = 0.79)
Two raters needed for acceptable reliability in high-stakes exams when using the checklist
Minimal interrater bias found; variance largely due to interaction among student, rater, and item
Digital recordings were a reliable method of evaluating surgical performance
Study supports using checklist over OSATS GRS for assessing preclinical students on simulated models

Simini Surgery Review Podcast

Know What Matters in the Literature - and Why

We distill peer-reviewed surgical studies into clinically relevant summaries and
exam-style questions, so you can make informed decisions with confidence.

Get Instant Access to Surgical Summaries

Free Access. No Spam. Just Smarter Surgical Learning

Multiple Choice Questions on this study

In Farrell 2022 et al., on checklist reliability in OVH simulation, how many OSATS GRS items met the content validity criteria?

A. All 6

B. None

C. 4

D. 1

E. 2

Answer: 1

Explanation: Only “respect for tissue” met content validity criteria; the other 5 OSATS GRS items were excluded.

In Farrell 2022 et al., on checklist reliability in OVH simulation, what was the generalizability (G) coefficient for the 40-item checklist instrument used in the study?

A. 0.92

B. 0.85

C. 0.76

D. 0.64

E. 0.79

Answer: 0.85

Explanation: The G-coefficient for the checklist was 0.85, indicating good reliability for moderate-stakes exams.

In Farrell 2022 et al., on checklist reliability in OVH simulation, what was the main benefit of using digital recordings for student performance assessment?

A. Increased student stress

B. Allowed for reuse of models

C. Improved model accuracy

D. Enabled blinded multi-rater evaluation

E. Reduced faculty workload

Answer: Enabled blinded multi-rater evaluation

Explanation: Digital recordings allowed multiple raters to evaluate performance without introducing bias.

In Farrell 2022 et al., on checklist reliability in OVH simulation, what was the recommended number of raters needed to achieve acceptable reliability (G ≥ 0.76) for high-stakes use of the checklist?

A. 1

B. 2

C. 3

D. 4

E. 5

Answer: 2

Explanation: Two raters per student were needed to achieve a G-coefficient of 0.76, which is considered acceptable for high-stakes exams.

In Farrell 2022 et al., on checklist reliability in OVH simulation, what percentage of checklist items were deemed essential by expert surgical educators?

A. 75%

B. 85%

C. 90%

D. 95%

E. 98%

Answer: 98%

Explanation: Thirty-nine of 40 checklist items (98%) were validated by expert panel review.

Access the full library of surgical summaries and exam-style questions.

Get Instant Access