We consider efficiency in the implementation of deep neural networks.Hardware accelerators are gaining interest as machine learning becomesone of the drivers of high-performance computing. In these accelerators,the directed graph describing a neural network can be implemented asa directed graph describing a Boolean circuit. We make this observationprecise, leading naturally to an understanding of practical neural networksas discrete functions, and show that so-calledbinarised neural networksarefunctionally complete. In general, our results suggest that it is valuableto considerBoolean circuits as neural networks, leading to the questionof which circuit topologies are promising. We argue that continuity iscentral to generalisation in learning, explore the interaction between datacoding, network topology, and node functionality for continuity, and posesome open questions for future research. As a first step to bridging thegap between continuous and Boolean views of neural network accelerators,we present some recent results from our work on LUTNet, a novel Field-Programmable Gate Array inference approach. Finally, we conclude withadditional possible fruitful avenues for research bridging the continuousand discrete views of neural networks.