EvalSense: a framework for domain-specific LLM (meta-)evaluation
File(s)
Published version
Author(s)
Dejl, Adam
Pearson, Jonathan
Type
Conference Paper
Abstract
Robust and comprehensive evaluation of large language models (LLMs) is essential for identifying effective LLM system configurations and mitigating risks associated with deploying LLMs in sensitive domains. However, traditional statistical metrics are poorly suited to open-ended generation tasks, leading to growing reliance on LLM-based evaluation methods. These methods, while often more flexible, introduce additional complexity: they depend on carefully chosen models, prompts, parameters, and evaluation strategies, making the evaluation process prone to misconfiguration and bias. In this work, we present EvalSense, a flexible, extensible framework for constructing domain-specific evaluation suites for LLMs. EvalSense provides out-of-the-box support for a broad range of model providers and evaluation strategies, and assists users in selecting and deploying suitable evaluation methods for their specific use-cases. This is achieved through two unique components: (1) an interactive guide aiding users in evaluation method selection and (2) automated meta-evaluation tools that assess the reliability of different evaluation approaches using perturbed data. We demonstrate the effectiveness of EvalSense in a case study involving the generation of clinical notes from unstructured doctor-patient dialogues, using a popular open dataset. All code, documentation, and assets associated with EvalSense are open-source and publicly available at https://github.com/nhsengland/evalsense.
Date Issued
2026-03-01
Date Acceptance
2026-03-01
Citation
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2026, 3, pp.480-491
Publisher
Association for Computational Linguistics
Start Page
480
End Page
491
Journal / Book Title
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)
Volume
3
Copyright Statement
©2026 Association for Computational Linguistics. This paper is licensed on a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/)
License URL
Source
19th Conference of the European Chapter of the Association for Computational Linguistics
Publication Status
Published
Start Date
2026-03-24
Finish Date
2026-03-29
Coverage Spatial
Rabat, Marocco