Optimizing Referral Pathways for Patients with Muskuloskeletal Pain through Data–Driven Referral Insights and AI Integration

PhD description

Elisabeth Lindrup Nielsen, MD Student.

Optimizing early stratification of joint pain referrals: identifying inflammatory and non-inflammatory arthritis (IA) using referral text, patient questionnaires, and artificial intelligence.

Close to one-third of visits to the general practitioner (GP) concern patients with musculoskeletal (MSK) complaints, but the path from here can be hard for the GP to determine, as different MSK conditions require care from different specialists. Often, individuals with conditions such as osteoarthritis (OA) or fibromyalgia are referred to rheumatologists, as their symptoms can mimic those of rheumatoid arthritis (RA) [1]. In reality, these patients would benefit more from physiotherapy or occupational therapy [2]. Such suboptimal referrals place unnecessary strain on the healthcare system and delay diagnosis and treatment for both inflammatory and non-inflammatory conditions, especially since the number of rheumatologists continues to decline [3, 4].

Efficient triage of patients with suspected inflammatory rheumatic disease (IRD) is crucial to prevent irreversible damage and disease progression, wherefore it is essential that all patients with MSK symptoms are promptly directed to the right healthcare professional to ensure timely and appropriate treatment [5, 6]. This includes rheumatologists, physiotherapists, chiropractors, and specialized nurses along with both orthopedic evaluations and pain management [7].

To achieve this, triage systems should ideally be implemented without adding extra workload for clinicians, especially given the anticipated shortage of rheumatology professionals in Europe. 

Rheumatologists play a central role in coordinating patient flow and may see selected patients together with physiotherapists to ensure accurate assessment and appropriate management. Combining structured triage processes with multidisciplinary collaboration can help optimize patient pathways, reduce unnecessary specialist visits, and enhance overall quality of care in IRD.

Building on this multidisciplinary foundation, technological solutions offer an opportunity to further improve triage quality. There is a need for a triage tool to support rheumatologists in assessing and prioritising referrals, helping to identify the most likely rheumatological condition and ensure appropriate, timely specialist management [8]. With the advent of artificial intelligence (AI), there are opportunities to integrate patient symptoms, referral texts, laboratory results, and radiology reports to differentiate IA from non-IA, supporting more efficient and evidence-based patient pathways for all patients with MSK symptoms [9].

Machine learning, Natural Language Processing (NLP) and Large Language Models (LLMs) have shown efficiency in improving artificial referral systems for patients with suspected IRD, both in terms of diagnostic performance and specificity, while maintaining high sensitivity [10] [11]. Incorporating additional laboratory parameters and allowing individual feature weighting may further improve accuracy and promote broader adoption [10] [11].

For the purposes of this phd-project, IA will be defined according to ICD-10 codes and categorized as Definite IA (M05, M07, M08, M45) and Early / Other IA (M06, M46, M35.3). Non-IA will include M15–M19, encompassing OA and related conditions. Patients who do not meet criteria for these categories will be classified as Unclassifiable / Other Chronic Joint Conditions, including those requiring long-term management without a definitive diagnosis [11]. This grouping reflects clinically relevant triage categories and is consistent with existing rheumatology referral and AI-based classification frameworks, and will be applied across all studies to facilitate comparison and AI model training [11].

Category	Description	ICD-10 Codes
Definite IA	Well-defined IA	M05, M07, M08, M45
Early / Other IA	Early or other IA	M06, M46, M35.3
Non-IA	OA and related non-IA joint conditions	M15 – M19
Unclassifiable / Other Chronic Joint Conditions	Patients not fitting above categories; may require long-term management	N/A

The overall aim is to investigate referral quality, diagnostic stratification, and patient pathways for individuals referred with arthralgia or suspected early IA to the Diagnostic Center in Silkeborg (SDC).

Retrospective Stratification (Study 1): To assess the diagnostic accuracy and consistency of rheumatologists when stratifying patients into likelihood categories of IA based solely on GP referral texts.

Prospective IASQ Cohort (Study 2): To evaluate the predictive value of the Inflammatory Arthritis Screening Questionnaire (IASQ) in distinguishing IA from non-IA among patients referred with joint pain.

LLM Substudy (Study 3): To explore whether LLMs can integrate referral text, laboratory results, and imaging data to predict IA vs. non-IA and suggest optimal diagnostic or treatment pathways.

Study 1 – Retrospective Stratification

Objective: Determine whether rheumatologists at SDC can reliably classify patients at the time of referral into high, moderate, and low likelihood of IA OR determine the correct of the four above mentioned categories.

Hypothesis: Rheumatologist-based stratification based on referral content shows limited concordance with final clinical diagnosis and effectively distinguishes IA from non-IA conditions.

Study 2 – Prospective IASQ Cohort

Objective: Validate the IASQ as a screening and triage tool for patients with MSK pain.

Hypothesis: IASQ responses can differentiate inflammatory from non-inflammatory symptom patterns and support appropriate triage decisions.

Study 3 – AI Substudy

Objective: Test whether a LLM trained on multimodal data can assist rheumatologists at SDC in stratifying referrals, distinguishing suspected early IA from non-IA OR the four categories, and guiding patients to the appropriate specialist for timely, targeted assessment.

Hypothesis: AI-based stratification will support rheumatologists at SDC in the initial triage of referrals by suggesting a tentative diagnosis, helping ensure patients are directed to the correct healthcare professional.

Study 1; Retrospective Stratification

Rationale: Early recognition of IA is essential for timely initiation of disease-modifying treatment and prevent irreversible joint damage [5, 6]. Many referrals are for non-inflammatory joint pain, resulting in unnecessary specialist assessments, and inefficient resource use. Evaluating rheumatologist stratification provides a benchmark for triage quality and informs structured screening tools or AI-based solutions, to effectively refer patients to the correct specialist.

Methods

Study Population Selection: The study population will include adults referred to SDC for suspected early IA or non-IA, as defined in the ICD-10 classification above. Exclusion criteria will be prior IA or referrals unrelated to joint symptoms.

Sample Size Selection: All referrals within one year will be considered, OR alternatively a selected number of referrals OR power calculation can be performed, or the number can be decided pragmatically. A minimum number of patients will be selected from over 1000 retrospective referrals. [TO BE DETERMINED]

Data Sources and Content: No clinical or follow-up data will be obtained for LLM training in this phase, as the aim is to determine the predictive information contained in referral texts, wherefore GP referral texts will be used as the main data source. OR ALSO: Information will be obtained from Electronic Medical Records (EMR): blood samples, imaging, dates of symptom recognition, referral, EMR-confirmed diagnosis (ICD-10 codes) and final treatment pathways (rheumatology, physiotherapy, orthopedics). Other relevant data or baseline characteristics can be included if appropriate [TO BE DETERMINED].

Data Management and Collection: A method for minimizing selection bias will be applied when choosing which referrals to include. [TO BE DETERMINED]. Data collection will be conducted by ELN (others?) using REDCap, with record-ID to anonymize data. Personally sensitive data will be stored in internal records (CPR numbers). (internal list?) [TO BE DETERMINED]

Analysis: Rheumatological assessment of referrals will be performed by three (or more) rheumatologists who independently and blinded review all referral texts (only what is in the referral) and assign their suggested diagnosis (ICD-10 codes) and treatment pathway. Their decisions will be compared, and intra- and inter-rater reliability will be calculated using Cohen’s kappa. Diagnostic accuracy metrics will include sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and/or Likelihood Ratios (LR+/LR-). Logistic regression and receiver operating characteristic (ROC) curve, including area under the curve (AUC), will be used to explore predictive referral features. Additionally, referrals will be grouped based on final diagnosis and treatment pathways to identify referral features that are characteristic of each diagnosis (or subcategory?). These features will be considered for implementation in the IASQ (Study 2). Other analyses can be added if relevant [TO BE DETERMINED]. Feature importance derived from referral characteristics will be analyzed to identify variables most predictive of IA vs. non-IA. These insights could inform subsequent AI modeling using explainable techniques such as SHAP (SHapley Additive exPlanations) to provide transparent insight into which referral features most strongly influence predictions of IA versus non-IA. These clinician-based classifications will serve as a reference standard for evaluating AI-assisted triage performance in Study 3.

Expected Outcomes: Baseline data for IASQ and LLM (study 2 and 3) (if we choose to do so), quantified intra- and inter-rater reliability, diagnostic accuracy metrics for referral-based rheumatological stratification, identification of referral features associated with IA or non-IA and diagnostic delay. Data will support LLM development.

Ethics: Institutional Review Board (IRB) approval will be obtained, with no informed consent required due to the retrospective design.

Feasibility and Workflow: Year 1 will include IRB approval, REDCap design, data collection, rheumatologist review of referrals, analysis, and script preparation. Year 2 will focus on preparation of data for integration in the LLM and Article 1 writing.

Study 2; Prospective IASQ Cohort

Rationale: Structured patient questionnaires can enhance triage accuracy over unstructured referral review. Prospective validation ensures symptom patterns reliably differentiate inflammatory from non-inflammatory conditions, improving prioritization, referrals, and efficient resource allocation.

Methods

Study Population Selection

The study population will include adult referrals to SDC with MSK pain (ICD-10?) over 12 months who also completed the IASQ. [TO BE DETERMINED]. Exclusion(ordblinde?)????

Sample Size Selection: All referrals with full completion of IASQ within one year, OR a pre-specified number (LLM friendly), OR the same count as in Study 1. Power calculation? [TO BE DETERMINED]

Data Sources and Content: Data sources will primarily be GP referral text and IASQ responses (either completed independently by the patient via e-Boks upon referral receipt or jointly with the GP or at SDC alone or with rheumatologist/nurse – depends on when patients sign consent) [TO BE DETERMINED]. Referral features identified in Study 1 as predictive of IA vs. non-IA will inform the IASQ design, ensuring that questionnaire items capture the most clinically relevant data for prospective validation. What could be deduced from a referral??? [TO BE DETERMINED]. Laboratory results (Erythrocyte sedimentation rate (ESR), C-reactive protein (CRP), rheumatoid factor (RF), anti-cyclic citrullinated peptide (anti-CCP)), Imaging (X-ray, magnetic resonance imaging (MRI)), EMR-confirmed diagnosis and final treatment pathways and baseline, demographic, and clinical characteristics.

Data Management and Collection: A method for minimizing selection bias will be applied when choosing which referrals to include. Personally sensitive data will be stored on the internal list (CPR numbers). Data collection will be conducted using IASQ and REDCap (EMR/referral texts) and record-ID will be used to anonymize data. (A standardized REDCap scheme similar to Study 1 will be created for those who review records (ELN, others). IASQ responses should/ will be coupled to referral data.

Analysis: Descriptive statistics for demographics and clinical characteristics. Predictive validity of IASQ: sensitivity, specificity, ROC, AUC. A Comparison between IASQ-based and referral-based stratification. Time to diagnosis and treatment initiation. Other relevant analyses can be added if appropriate [TO BE DETERMINED].

Expected Outcomes: Validation of IASQ as a triage tool, evidence for differentiating inflammatory from non-inflammatory joint conditions and quantification of diagnostic delay. Collected IASQ data, together with referral text and clinical results, will later serve as input for AI-based predictive modeling (Study 3).

Ethics: IRB approval and written informed consent will be obtained from all participants prior to data collection.

Feasibility and Workflow: Year 1: IRB approval, preliminary IASQ creation and validation, development of patient brochure to promote IASQ, and recruitment planning. Year 2: Recruitment and data collection over 12 months. Year 3: Analyses of IASQ predictive validity and outcomes, preparation of data for LLM/AI development (potentially also during Year 2), and article 2 writing.

Study 3; LLM Substudy

Rationale

AI-assisted triage can improve accuracy, reduce unnecessary referrals, and accelerate patient allocation. Combining structured and unstructured data may outperform physicians as reviewers or supplement their clinical decision-making.

Methods

Data Integration: First, to multimodal fusion Study 1 and Study 2 datasets (500-1000 patients), including referral texts, IASQ, lab results, imaging, final diagnoses, and treatment pathways.

Data Management: Data from Study 1 and Study 2 will be handled according to GDPR-compliant preprocessing and storage procedures.

Analysis and LLM Development: Rather than training a model from scratch, an existing transformer-based LLM (e.g., BERT or GPT architecture) will be fine-tuned on the collected dataset to optimize performance for rheumatology triage tasks. The LLM will be trained on integrated datasets from Studies 1 and 2, incorporating referral text, IASQ responses, laboratory results, imaging data, and final ICD-10-based diagnoses. Model interpretability will be assessed using SHAP to identify which features most strongly influence IA vs. non-IA predictions, supporting clinician decision-making. (AUC, precision, recall, and F1-score) [TO BE DETERMINED].

Expected Outcomes: Validated LLM for IA vs. non-IA prediction with insights into referral triage efficiency and potential improvements, along with identification of barriers and facilitators for LLM-assisted triage implementation, which will give evidence for integration of LLM with clinical pathways.

Ethics: IRB approval will cover both retrospective pseudonymized and prospectively collected data (or should we seek a new one?) [TO BE DETERMINED].

Feasibility Assessment and Workflow: An evaluation of workflow, interpretability, and clinician acceptance (can we integrate the LLM in a busy every day?) [TO BE DETERMINED]. The perspective will include how LLMs can be integrated in the clinic without replacing clinicians. Year 1 and 2: Preparation and education on how to design LLMs. Data collection during Study 1 and 2. Year 3: Model training with data, validation, and feasibility assessment, coordinated with clinical staff to evaluate workflow integration and reporting integration in everyday life. Article 3.

Diagnostic Centre Silkeborg

The Rheumatology Clinic at the Medical Diagnostic Center in Silkeborg (SDC) receives referrals for suspected diseases of the MSK system, including IA and non-IA and connective tissue disease. The clinic assesses all patients presenting with MSK pain, ensuring that each patient is directed to the right healthcare professional at the right time. This may include rheumatologists, physiotherapists, or chiropractors, with rheumatologists playing a central role in coordinating patient flow and occasionally seeing patients together with physiotherapists. Referrals undergo structured triage to identify the appropriate diagnostic pathway, wherefore clear and comprehensive referrals remain essential to allow allocation to the correct specialist unit or further diagnostic steps, minimizing diagnostic delays. Through optimized referral interpretation and early stratification, SDC aims to reduce patient delay and make accurate and fast definitive diagnosis, in order to initiate timely treatment, limit disease progression, and avoid unnecessary resource use (ref).

The Applicant???

The applicant has gained substantial research experience through previous studies on referral patterns and diagnostic processes in rheumatological diseases, including first-author publications and presentations at international conferences. This provides a strong foundation in data management, coding, and advanced statistical analysis. The project will be carried out in close collaboration with an experienced multidisciplinary team at the Department of Rheumatology, Aarhus University Hospital, and SDC, offering access to clinical data and methodological guidance, ensuring the necessary support and infrastructure for successful project execution.

Ethics and Data Governance

All studies will be conducted under IRB approval and in accordance with GDPR. Retrospective data will be handled under a waiver of consent, and all prospective participants will provide informed consent. Data will be securely with restricted access on???.

Integrated Project Timeline

Year 1

Study 1: IRB approval; REDCap design; retrospective data extraction and collection; rheumatologist review of referrals; data analysis and script preparation.

Study 2: IRB approval; preliminary IASQ creation and validation; development of patient brochure to promote IASQ; recruitment planning.

Study 3: IRB approval; Preparation and education on LLM/AI design.

Dissemination: Annual EULAR Conference.

Year 2

Study 1: Preparation of data for integration in the LLM; Article 1 writing and submission.

Study 2: Prospective cohort recruitment and IASQ administration over 12 months; ongoing data collection and analysis; Article 2 writing preparation. (to be determined, can we start this in year one)

Study 3: Continued LLM/AI design preparation and education; ongoing data collection from Study 1 and 2 for later fusion and integration in LLM.

Dissemination: Annual EULAR Conference.

Year 3

Study 2: Analyses of IASQ predictive validity and outcomes; preparation of data for LLM/AI development and Article 2 submission.

Study 3: AI model training, dataset integration, validation, and workflow feasibility assessment, coordinated with clinical staff to evaluate workflow and reporting integration in everyday clinical practice. Article 3 writing and submission.

Dissemination: Annual EULAR Conference.

Year 4

All studies: Final analyses, synthesis of findings, dissemination, and secure data archiving.

Dissemination: Annual EULAR Conference.

Year	Study 1	Study 2	Study 3	Dissemination
Year 1	IRB approval; REDCap design; retrospective data extraction and collection; rheumatologist review of referrals; data analysis and script preparation	IRB approval; preliminary IASQ creation and validation; development of patient brochure to promote IASQ; recruitment planning	Preparation and education on LLM/AI design	Annual EULAR Conference
Year 2	Preparation of data for integration in LLM; Article 1 writing and submission	Prospective cohort recruitment and IASQ administration over 12 months; ongoing data collection and analysis; preparation for Article 2 writing	Continued LLM/AI design preparation and education; ongoing use of data from Study 1 and Study 2 for integration into AI	Annual EULAR Conference
Year 3	–	Analyses of IASQ predictive validity and outcomes; preparation of data for LLM/AI development; Article 2	AI model training, dataset integration, validation, workflow feasibility assessment; coordination with clinical staff for workflow integration; Article 3	Annual EULAR Conference
Year 4	Final analyses, synthesis of findings, dissemination, secure data archiving	Final analyses, synthesis of findings, dissemination, secure data archiving	Final analyses, synthesis of findings, dissemination, secure data archiving	Annual EULAR Conference Final dissemination

Dissemination

Peer-reviewed, international journals (rheumatology, digital health, AI in medicine)

Presentations at EULAR, ACR, Danish rheumatology conferences

Clinical implementation: results shared with GPs, physiotherapists, and orthopedic colleagues to inform referral strategies

Research group

Elisabeth Lindrup Nielsen, (MD Student?), Department of Rheumatology, Aarhus University Hospital

Tue Wenzel Kragstrup, MD, PhD, Associate Professor, Main supervisor, Department of Rheumatology,

Aarhus University Hospital.

Line Thorndal Moll, MD, PhD, Main supervisor, Department of Rheumatology, Silkeborg Regional Hospital

Stine Daugaard Pedersen, MD, PhD student, Department of Rheumatology, Silkeborg Regional Hospital

Jesper Blegvad Nissen, MD, Department of Rheumatology, Silkeborg Regional Hospital

Peter Vedsted KIF

Charlotte fra KIF.

Johannes og Rachel som internationale. (Du kan skrive et udenlandsophold et af de to steder ind i PhD ansøgning til AU.)

1. van Delft, E., et al., Impact of triage by a rheumatologist on appropriateness of referrals from primary to secondary care: a cluster randomized trial. Scand J Rheumatol, 2023. 52(4): p. 403-411.2. Lynch, M.E., et al., A systematic review of the effect of waiting for treatment for chronic pain. Pain, 2008. 136(1-2): p. 97-116.3. Stack, R.J., et al., Delays between the onset of symptoms and first rheumatology consultation in patients with rheumatoid arthritis in the UK: an observational study. BMJ Open, 2019. 9(3): p. e024361.4. Krusche, M., et al., [Specialist training quo vadis?]. Z Rheumatol, 2019. 78(8): p. 692-697.5. van der Linden, M.P., et al., Long-term impact of delay in assessment of patients with early arthritis. Arthritis Rheum, 2010. 62(12): p. 3537-46.6. Fuchs, F., et al., Diagnostic delay stages and pre-diagnostic treatment in patients with suspected rheumatic diseases before special care consultation: results of a multicenter-based study. Rheumatol Int, 2023. 43(3): p. 495-502.7. Gormley, G.J., et al., Can diagnostic triage by general practitioners or rheumatology nurses improve the positive predictive value of referrals to early arthritis clinics? Rheumatology, 2003. 42(6): p. 763-768.8. Thoms, B.L., et al., Predictors of inflammatory arthritis among new rheumatology referrals: a cross-sectional study. Rheumatol Adv Pract, 2023. 7(3): p. rkad067.9. Wang, B., et al. Early detection of inflammatory arthritis to improve referrals using multimodal machine learning from blood testing, semi-structured and unstructured patient records. in Hawaii International Conference on System Sciences. 2023.10. Knitza, J., et al., Machine learning-based improvement of an online rheumatology referral and triage system. Front Med (Lausanne), 2022. 9: p. 954056.11. Maarseveen, T.D., et al., Improving musculoskeletal care with AI enhanced triage through data driven screening of referral letters. npj Digital Medicine, 2025. 8(1): p. 98.