Simulations anticipate increased perturbation growth from non-uniform laser heating for magnetized direct-drive implosions. At the capsule pole, where the magnetic field is normal to the ablator surface, the field remains in the conduction zone and suppresses non-radial thermal conduction; in unmagnetized implosions this non-radial heat-flow is crucial in mitigating laser heating imbalances. Single-mode simulations show the magnetic field particularly amplifying short wavelength perturbations, whose behavior is dominated by thermal conduction. The most unstable wavelength can also become shorter. 3D multi-mode simulations of the capsule pole reinforce these findings, with increased perturbation growth anticipated across a wide range of scales. The results indicate that high-gain spherical direct-drive implosions require greater constraints on the laser heating uniformity when magnetized.