To Örebro University

Background/Purpose: Reliability measures the consistency of an instrument’s assessments. Instruments intended for clinical and research use must exhibit high reliability. There is a need for studies that evaluate and compare the reliability of different instruments used to measure SLE disease activity. This study aims to estimate the intrarater and interrater reliability of 3 SLE disease activity measurement tools as assessed by lupus experts: the SLE Disease Activity Score (SLE-DAS), the SLE Disease Activity Index 2000 (SLEDAI-2K), and the Physician Global Assessment (PGA).[1,2]

Methods: A group of 19 lupus experts from 12 countries (Europe, North America, South America, and Asia) evaluated 24 clinical case vignettes of SLE covering a wide spectrum of organ manifestations and disease severity. All raters completed a training on scoring rules for SLE-DAS, SLEDAI-2K and PGA before assessing the clinical vignettes. Raters scored each clinical vignette with SLE-DAS, SLEDAI-2K, and PGA twice, with at least a 10-day interval between rounds. The clinical vignettes were randomly ordered and assessed through an online survey. Intrarater and interrater reliability were assessed using the Intraclass Correlation Coefficient (ICC) and reported with 95% CI. For this analysis, ICC estimates were derived from a two-way random effects model (single rater). All calculations were performed using the Stata Statistical Software (Release 17) with the kappaetc module. The Coefficient of Variation (CV) was also used as a measure of reliability.[3] The CV was calculated for each vignette, based on the 19 measurements from each rater, for SLE-DAS, SLEDAI-2K, and PGA. Then, for each disease activity measure, the mean of the CV values across the 24 vignettes was used as a summary measure of within-subject variability and is expressed as a percentage.

Results: The 24 clinical vignettes represented a wide variety of active SLE manifestations, including skin rash (20.8%), arthritis (12.5%), renal involvement (12.5%), thrombocytopenia (12.5%), cardiac/pulmonary involvement (12.5%), mucocutaneous vasculitis (8.3%), serositis (8.3%), and neuropsychiatric SLE (8.3%). Systemic vasculitis, myositis, alopecia, hemolytic anemia, and leukopenia were each present in 4.2% of the vignettes. Hypocomplementemia and/or positive anti-dsDNA were present in 75.0%. All the 19 lupus experts completed 2 rounds of assessment of the 24 clinical vignettes, totaling 912 case assessments. Scores ranged from 0.37 to 27.37 in SLE-DAS, 0 to 21 in SLEDAI-2K, and 0.0 to 3.0 in PGA. The interrater ICCs were 0.93, 0.91, and 0.74, and the intrarater ICCs were 0.94, 0.93, and 0.88 for SLE-DAS, SLEDAI-2K, and PGA, respectively. The CVs (first rating round) were 8.2%, 19.7%, and 41.1% for SLE-DAS, SLEDAI-2K, and PGA, respectively. The ICCs (95% CI) and CVs for SLE-DAS, SLEDAI-2K, and PGA are detailed in Table 1 and Table 2.

Conclusions: This study demonstrates that both SLE-DAS and SLEDAI-2K presented good to excellent interrater reliability, indicating strong consistency in scoring across different experts. Notably, SLE-DAS achieved excellent intrarater reliability, reflecting a high degree of stability in individual assessments between assessments at different times. Both SLEDAI-2K and PGA exhibited good to excellent intrarater reliability, while PGA showed moderate to good interrater reliability. Furthermore, SLE-DAS exhibited the lowest within-subject variability, as evidenced by its lower CV values compared to SLEDAI-2K and PGA.

References: [1.] Jesus D. Ann Rheum Dis 2019;78:365-71. [2.] Piga M. Lancet Rheumatol 2022;4:e441-9. [3.] Shechtman O. In: S.A.R. Doi, G.M. Williams (Eds.). Methods Clin Epidemiol 2013:39-49.