Long Short-Term Memory (LSTM) networks havebeen deployed in speech recognition, natural language processingand financial calculations in recent years, and are beginning to beused in systems where low latency and low power are required.To meet such requirements, we propose a stall-free hardwarearchitecture by reorganising the order of operations in an LSTMsystem and develop a unique blocking-batching strategy to reusethe LSTM weights fetched from external memory to optimise thebenefits of on-chip memory with a limited size for a large machinelearning model. Evaluation results show that our architecture canachieve up to 20.8 GOPS/W, which would be among the highestfor FPGA designs targeting LSTM systems with weights stored inoff-chip memory. Comparing to the state-of-the-art design usingoff-chip memory to store the weights, we achieve 1.65 timeshigher performance-per-watt efficiency and 1.60 times higherperformance-per-DSP efficiency. When compared with CPU andGPU implementation, our novel hardware architecture is 23.7and 1.3 times faster while consuming 208 and 19.2 times lowerenergy respectively, which shows that our approach contributesto high performance and low power FPGA-based LSTM systems.