Identifying clusters of people with similar patterns of Multiple Long-Term Conditions (MLTC) could help healthcare services to tailor care. In this population-based study, we developed a pipeline incorporating a DeBERTa language model to generate gender-specific clusters. Our model, EHR-DeBERTa, was pre-trained on longitudinal sequences of diagnoses, medications and test results from primary care electronic health records of 5.8 million patients in the UK. EHR-DeBERTa was used to generate patient embeddings for males and females separately, and clusters were identified by K-Means. Fifteen clusters were identified in females and seventeen in males, categorized into low disease burden, mental health, cardiometabolic, respiratory, and mixed diseases. Cardiometabolic and mental health conditions showed the strongest separation across clusters, with older patients in cardiometabolic clusters. Our approach demonstrates how LLMs can provide interpretable insights into disease patterns. Future work incorporating clinical outcomes could enhance risk prediction and support precision-medicine for people with MLTC.