ResBench: a resource-aware benchmark for LLM-generated FPGA designs
File(s)
Published version
Author(s)
Guo, Ce
Zhao, Tong
Type
Conference Paper
Abstract
Field-Programmable Gate Arrays (FPGAs) are widely used in modern hardware design, yet writing Hardware Description Language (HDL) code for FPGA implementation remains a complex and time-consuming task. Large Language Models (LLMs) have emerged as a promising tool for HDL generation, but existing benchmarks for LLM-based code generation primarily focus on functional correctness while overlooking hardware resource usage. Furthermore, current benchmarks offer limited diversity and do not fully represent the wide range of real-world FPGA applications. To address these shortcomings, we introduce ResBench, the first resource-focused benchmark explicitly designed to distinguish between resource-optimized and inefficient LLM-generated HDL code. ResBench consists of 56 problems across 12 categories, covering applications from finite state machines to financial computing. Our open-source evaluation framework automatically tests LLMs by generating Verilog code, verifying correctness, and measuring resource usage. The experiments, which primarily analyze Lookup Table (LUT) usage, reveal significant differences among LLMs, demonstrating ResBench’s capability to identify models that generate more resource-optimized FPGA designs.
Date Issued
2025-05-01
Date Acceptance
2025-05-01
Citation
HEART '25: Proceedings of the 15th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies, 2025, pp.25-34
Publisher
ACM
Start Page
25
End Page
34
Journal / Book Title
HEART '25: Proceedings of the 15th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies
Copyright Statement
© 2025 Copyright held by the owner/author(s). This work is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/).
License URL
Identifier
10.1145/3728179.3728192
Source
HEART 2025: 15th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies
Subjects
CCS Concepts • Hardware → Board-and system-level test
Reconfigurable logic applications
Functional verification
Physical verification
• Software and its engineering → Source code generation Large Language Models (LLMs), Hardware Description Languages (HDLs), Verilog Code Generation, FPGA Resource Utilization, Automated Benchmarking, Empirical Evaluation of LLMs
Publication Status
Published
Start Date
2025-05-26
Finish Date
2025-05-28
Coverage Spatial
New York, NY, United States
Date Publish Online
2025-05-25