This vignette compares a PCL-5 symptom definition against the CAPS-5 diagnosis, and shows how to optimize a definition against the CAPS-5 diagnosis directly.
Why use the CAPS-5
The PCL-5 is completed by the patient; the CAPS-5 is administered by a clinician. Both rate the same 20 DSM-5-TR symptoms on the same 0 to 4 scale, so when both are available for the same participants the CAPS-5 diagnosis is the closest available reference to a clinical ground truth. That makes it the natural standard against which to judge a self-report definition.
The bundled paired data
The general-population dataset simulated_ptsd_genpop
ships both instruments for the same participants: the PCL-5 items
S1–S20 and paired CAPS-5 severity ratings
C1–C20, simulated so the two total scores
correlate about 0.8 (the level usually reported empirically). Because
both come from one data frame, the PCL-5 and CAPS-5 views describe the
same people in the same row order, which is exactly what a
paired-instrument comparison requires.
To analyse one instrument we standardise its 20 columns and park
everything else, including the other instrument’s columns, in
id_col, so rename_* sees exactly 20 item
columns. We use a 120-row subset for speed.
library(PTSDdiag)
data("simulated_ptsd_genpop")
gp <- simulated_ptsd_genpop[1:120, ]
demo <- c("patient_id", "age", "sex")
# PCL-5 view: rename S1..S20; carry the CAPS-5 columns through untouched
ptsd <- rename_ptsd_columns(gp, id_col = c(demo, paste0("C", 1:20)))
# CAPS-5 view: rename C1..C20; carry the PCL-5 columns through untouched
caps5 <- rename_caps5_columns(gp, id_col = c(demo, paste0("S", 1:20)))Computing the CAPS-5 diagnosis
create_caps5_diagnosis() applies the DSM-5-TR algorithm
to the CAPS-5 severity scores, using the same clusters and the same
presence threshold of 2 or higher as the PCL-5 diagnosis. It returns a
single logical column, PTSD_caps5.
caps5_dx <- create_caps5_diagnosis(caps5)
mean(caps5_dx$PTSD_caps5) * 100
#> [1] 21.66667Comparing the PCL-5 and CAPS-5 diagnoses
compare_diagnostic_systems() builds one summary table
that scores each diagnostic system against a chosen reference. Setting
reference = "caps5" makes the CAPS-5 the standard, so the
PCL-5 diagnosis and ICD-11 are evaluated against it. Each row reports
sensitivity, specificity, PPV, NPV, accuracy, and balanced accuracy
against the reference, together with the counts of false positives,
false negatives, and total misclassifications.
compare_diagnostic_systems(
ptsd,
caps5_data = caps5,
icd11 = TRUE,
reference = "caps5"
)
#> system n_diagnosed pct_diagnosed sensitivity specificity ppv
#> 1 DSM-5-TR (CAPS-5) 26 21.67 1.0000 1.0000 1.0000
#> 2 DSM-5-TR (PCL-5) 28 23.33 0.7308 0.9043 0.6786
#> 3 ICD-11 (PCL-5) 29 24.17 0.7308 0.8936 0.6552
#> npv n_false_negative pct_false_negative n_false_positive
#> 1 1.0000 0 0.00 0
#> 2 0.9239 7 5.83 9
#> 3 0.9231 7 5.83 10
#> pct_false_positive n_misclassified accuracy balanced_accuracy
#> 1 0.00 0 1.0000 1.0000
#> 2 7.50 16 0.8667 0.8175
#> 3 8.33 17 0.8583 0.8122Because the two instruments are correlated here rather than random, the self-report PCL-5 diagnosis agrees substantially with the clinician CAPS-5 reference. Sensitivity is the share of CAPS-5-positive participants the PCL-5 rule also calls positive, accuracy is the overall share classified the same way, and balanced accuracy averages performance in the CAPS-5-positive and CAPS-5-negative groups.
Optimizing against the CAPS-5 diagnosis
To make the CAPS-5 the optimization target rather than a row in a
comparison, pass the clinician diagnosis to
compare_optimizations() as a fixed criterion. The optimized
PCL-5 definitions are then reported alongside it, and the
symptom-frequency tools apply as in the Comparing diagnostic criteria
vignette.
comp <- compare_optimizations(
ptsd,
scenarios = list(
"4/6 Hierarchical" = list(n_symptoms = 6, n_required = 4, hierarchical = TRUE),
"4/6 Non-hierarchical" = list(n_symptoms = 6, n_required = 4, hierarchical = FALSE),
"CAPS-5 (reference)" = list(type = "fixed", criterion = caps5_dx$PTSD_caps5,
symptoms = 1:20)
),
n_top = 5,
score_by = "balanced_accuracy",
show_progress = FALSE
)
#> ℹ Generated 13685 valid cluster-constrained combinations
#> ℹ Evaluated 13685 combinations. Best: 2, 6, 7, 11, 16, 17 (26 additional tied)
#> ℹ Evaluated 38760 combinations. Best: 2, 5, 6, 7, 11, 12 (2 additional tied)
summarize_top_combinations(comp, top_n = 3, as_percent = TRUE)
#> Approach Rank Combination TP FN FP TN Sensitivity
#> 1 4/6 Hierarchical 1 symptom_2_6_7_11_16_17 23 5 0 92 82.14286
#> 2 4/6 Hierarchical 2 symptom_2_6_7_8_9_17 23 5 0 92 82.14286
#> 3 4/6 Hierarchical 3 symptom_2_6_7_8_10_17 23 5 0 92 82.14286
#> 4 4/6 Non-hierarchical 1 symptom_2_5_6_7_11_12 27 1 0 92 96.42857
#> 5 4/6 Non-hierarchical 2 symptom_2_5_6_7_11_20 27 1 0 92 96.42857
#> 6 4/6 Non-hierarchical 3 symptom_2_5_7_11_12_20 27 1 0 92 96.42857
#> 7 CAPS-5 (reference) 1 PTSD_CAPS.5..reference. 19 9 7 85 67.85714
#> Specificity PPV NPV Accuracy Balanced Accuracy
#> 1 100.0000 100.00000 94.84536 95.83333 91.07143
#> 2 100.0000 100.00000 94.84536 95.83333 91.07143
#> 3 100.0000 100.00000 94.84536 95.83333 91.07143
#> 4 100.0000 100.00000 98.92473 99.16667 98.21429
#> 5 100.0000 100.00000 98.92473 99.16667 98.21429
#> 6 100.0000 100.00000 98.92473 99.16667 98.21429
#> 7 92.3913 73.07692 90.42553 86.66667 80.12422Validating derived definitions against the CAPS-5 diagnosis
At a validation site the definitions arrive already derived (see Validating a shared definition across
sites), and the question becomes how each performs against the local
clinician diagnosis. evaluate_definitions() takes the
clinician standard through its reference argument: a
logical vector, a 0/1-coded column, or the name of a carry-through
column in the data. Participants without a clinician interview —
NA in the reference — are excluded automatically, with a
message reporting how many. A "Full 20-item PCL-5" ceiling
row is added by default: the complete PCL-5 diagnosis scored against the
same reference. That row separates the cost of dropping items from the
intrinsic PCL-5-vs-CAPS-5 disagreement, and no reduced definition can be
expected to beat it. Here we extract the top three combinations per
optimized rule from the comparison above and request the tidy table
layout.
definitions <- extract_definitions(comp, n = 3)
evaluate_definitions(ptsd, definitions,
reference = caps5_dx$PTSD_caps5, tidy = TRUE)
#> Approach Rank Combination TP FN FP TN Sensitivity
#> 1 4/6 Hierarchical 1 2, 6, 7, 11, 16, 17 17 9 6 88 0.6538462
#> 2 4/6 Hierarchical 2 2, 6, 7, 8, 9, 17 17 9 6 88 0.6538462
#> 3 4/6 Hierarchical 3 2, 6, 7, 8, 10, 17 17 9 6 88 0.6538462
#> 4 4/6 Non-hierarchical 1 2, 5, 6, 7, 11, 12 19 7 8 86 0.7307692
#> 5 4/6 Non-hierarchical 2 2, 5, 6, 7, 11, 20 19 7 8 86 0.7307692
#> 6 4/6 Non-hierarchical 3 2, 5, 7, 11, 12, 20 19 7 8 86 0.7307692
#> 7 Full 20-item PCL-5 1 <NA> 19 7 9 85 0.7307692
#> 8 ICD-11 1 2, 3, 6, 7, 17, 18 19 7 10 84 0.7307692
#> Specificity PPV NPV Accuracy Balanced Accuracy
#> 1 0.9361702 0.7391304 0.9072165 0.8750000 0.7950082
#> 2 0.9361702 0.7391304 0.9072165 0.8750000 0.7950082
#> 3 0.9361702 0.7391304 0.9072165 0.8750000 0.7950082
#> 4 0.9148936 0.7037037 0.9247312 0.8750000 0.8228314
#> 5 0.9148936 0.7037037 0.9247312 0.8750000 0.8228314
#> 6 0.9148936 0.7037037 0.9247312 0.8750000 0.8228314
#> 7 0.9042553 0.6785714 0.9239130 0.8666667 0.8175123
#> 8 0.8936170 0.6551724 0.9230769 0.8583333 0.8121931The Combination column is NA for the
ceiling row, which applies all 20 items rather than a subset. Comparing
each definition’s balanced accuracy with the ceiling shows how much of
the disagreement with the CAPS-5 is attributable to simplification and
how much the full instrument shares.
Interpreting agreement and disagreement
When the PCL-5 and CAPS-5 diagnoses agree closely, as they do in this paired sample, a simplified PCL-5 definition that reproduces the full PCL-5 diagnosis also largely reproduces the clinician diagnosis, so the choice of reference matters little. When the instruments diverge, the choice of reference matters, and reporting performance against both is the honest course; the fixed-criterion mechanism above makes that choice explicit.
See also
- Getting started for the single-cohort derivation workflow.
- Comparing diagnostic criteria for multi-rule comparison and the symptom-frequency heatmap.
- Validating abbreviated symptom definitions for internal and external validation.