Training activity information
Details
Evaluate the diagnostic performance of a medical device including:
- Sensitivity
- Specificity
- Diagnostic accuracy
Type
Developmental training activity (DTA)
Evidence requirements
Evidence the activity has been undertaken by the trainee.
Reflection on the activity at one or more time points after the event including learning from the activity and/or areas of the trainees practice for development.
An action plan to implement learning and/or to address skills or knowledge gaps identified.
Considerations
- Diagnosis of specific impairment
- Healthcare evaluation
- ROC/threshold adjustment
- Binomial regression where required
- Appropriateness of gold standard
Reflective practice guidance
The guidance below is provided to support reflection at different time points, providing you with questions to aid you to reflect for this training activity. They are provided for guidance and should not be considered as a mandatory checklist. Trainees should not be expected to provide answers to each of the guidance questions listed.
Before action
- What are sensitivity, specificity, and diagnostic accuracy, and why are they important for evaluating diagnostic devices?
- What data is needed to calculate these metrics?
- What is the medical device you will be evaluating?
- What is the gold standard for diagnosis of the condition it aims to detect?
- What data will you use for your evaluation (e.g., true positives, true negatives, false positives, false negatives)?
- What statistical methods will you use to calculate sensitivity, specificity, and accuracy?
- What insights do you hope to gain about the process of evaluating the clinical utility of a diagnostic device based on its performance characteristics?
- What do you already know about diagnostic testing and biostatistics?
- Will you define the criteria for a positive and negative test result according to the device and the gold standard?
- Will you gather or simulate a dataset of test results?
- Have you discussed your evaluation plan and analysis methods with your training officer or a statistician?
In action
- As you evaluate the performance, what data sources are you using and why are they appropriate for this evaluation?
- What statistical methods or calculations are you employing to determine sensitivity, specificity, and accuracy?
- Are these methods feeling intuitive or requiring more conscious application?
- During the analysis, are you identifying any limitations in the data or the device’s performance?
- How are these influencing your overall evaluation?
On action
- Summarise the data you used and your findings regarding the sensitivity, specificity, and diagnostic accuracy of the medical device.
- What did you learn about the concepts of sensitivity, specificity, and diagnostic accuracy?
- How did you calculate these metrics?
- What were the strengths and limitations of the device’s diagnostic performance based on your evaluation?
- Were there any unexpected results or challenges in obtaining or analysing the data?
- Why is evaluating diagnostic performance crucial for medical devices used in healthcare?
- What aspects of evaluating diagnostic performance could be further developed?
- How will you approach future evaluations of medical device performance?
- What are your next steps in learning more about statistical methods for diagnostic test evaluation?
- What resources (e.g., statistical software, epidemiological data) would be helpful?
Beyond action
- Have you evaluated the diagnostic performance of other medical devices since this DTA?
- Have you compared the sensitivity, specificity, and accuracy of the device you evaluated with other similar devices or published data?
- Have you considered the clinical implications of the diagnostic performance metrics you calculated?
- How has this experience influenced your understanding of the key metrics for evaluating diagnostic tests?
- Have you applied the skills of data analysis and interpretation in other evaluations or assessments?
- Has your appreciation for the importance of understanding diagnostic accuracy in clinical decision-making changed?
- What skills related to data analysis, statistical interpretation, and understanding of diagnostic testing did you develop?
- How has this experience informed your approach to evaluating and selecting diagnostic medical devices?
- What actions will you take to further develop your expertise in evaluating the diagnostic performance of medical technologies?
Relevant learning outcomes
| # | Outcome |
|---|---|
| # 7 |
Outcome
Use suitable tools for measurement and analysis. |
| # 8 |
Outcome
Plan and perform verification and validation of solutions. |