Manually laying out the floor plan for buildings with highly-dense adjacency constraints at the early design stage is a labour-intensive problem. In recent decades, computer-based conventional search algorithms and evolutionary methods have been successfully developed to automatically generate various types of floor plans. However, there is relatively limited work focusing on problems with highly-dense adjacency constraints common in large scale floor plans such as hospitals and schools. This paper proposes an algorithm to generate the early-stage design of floor plans with highly-dense adjacency and non-adjacency constraints using reinforcement learning based on off-policy Monte-Carlo Tree Search. The results show the advantages of the proposed algorithm for the targeted problem of highly-dense adjacency constrained floor plan generation, which is more time-efficient, more lightweight to implement, and having a larger capacity than other approaches such as Evolution strategy and traditional on-policy search.