Reconfigurable intelligent surface (RIS) is expected to be a key technology in 6G to enhance wireless systems by efficiently and cost-effectively manipulating the propagation environment. In conventional RIS, each RIS element is independently controlled by a tunable load and it is disconnected from the other elements. Thus, conventional RIS is characterized by a diagonal scattering matrix, also known as a phase shift matrix, which has limited passive beamforming capabilities. To enhance the flexibility of RIS, beyond diagonal RIS (BD-RIS) has been introduced as a generalization of conventional RIS, in which the scattering matrix is not restricted to being diagonal. Motivated by the promising benefits unlocked by BD-RIS, in this thesis, we explore the fundamental performance limits of this technology with ideal hardware models and explore how hardware non-idealities impact the performance limits. First, we explore the optimization of the two main existing BD-RIS architectures, i.e., group- and fully-connected RISs. We derive a closed-form global optimal solution to optimize the scattering matrix of group- and fully-connected RISs and optimize the grouping strategy in group-connected RIS based on the channel statistics, showing the benefits of an optimized static grouping strategy. Second, we propose novel BD-RIS architectures to approach the best trade-off between achieved performance and BD-RIS circuit topology complexity. In particular, we propose forest- and tree-connected RISs as novel low-complex BD-RIS architectures and we derive the fundamental limits of the performance-complexity trade-off enabled by BD-RIS. Third, we model and optimize BD-RIS architectures in the presence of hardware impairments. Specifically, we consider BD-RIS with a practical discrete-value scattering matrix, with electromagnetic (EM) mutual coupling between the RIS elements, and with lossy interconnections. In the three technical parts of this thesis, we provide numerical results to corroborate our theoretical insights, confirming the significant superiority of BD-RIS over conventional RIS in EM wave manipulation.