Current computer-assisted surgical navigation systems mainly rely on optical markers screwed into the bone for anatomy tracking. The insertion of these percutaneous markers increases operating complexity and causes additional harm to the patient. A markerless tracking and registration algorithm has recently been proposed to avoid anatomical markers for knee surgery. The femur points were directly segmented from the recorded RGBD scene by a neural network and then registered to a pre-scanned femur model for the real-time pose. However, in a practical setup such a method can produce unreliable registration results, especially in rotation. Furthermore, its potential application in surgical navigation has not been demonstrated. In this paper, we first improved markerless registration accuracy by adopting a bounded-ICP (BICP) technique, where an estimate of the remote hip centre, acquired also in a markerless way, was employed to constrain distal femur alignment. Then, a proof-of-concept markerless navigation system was proposed to assist in typical knee drilling tasks. Two example setups for global anchoring were proposed and tested on a phantom leg. Our BICP-based markerless tracking and registration method has better angular accuracy and stability than the original method, bringing our straightforward, less invasive markerless navigation approach one step closer to clinical application. According to user tests, our proposed optically anchored navigation system achieves comparable accuracy with the state-of-the-art (3.64± 1.49 mm in position and 2.13±0.81° in orientation). Conversely, our visually anchored, optical tracker-free setup has a lower accuracy (5.86± 1.63 mm in position and 4.18±1.44° in orientation), but is more cost-effective and flexible in the operating room.