The methylammonium lead halides have become champion photoactive semiconductors for solar cell applications; however, issues still remain with respect to chemical instability and potential toxicity. Recently, the Cs2AgBiX6 (X = Cl, Br) double perovskite family has been synthesized and investigated as stable nontoxic replacements. We probe the chemical bonding, physical properties, and cation anti-site disorder of Cs2AgBiX6 and related compounds from first-principles. We demonstrate that the combination of Ag(I) and Bi(III) leads to the wide indirect band gaps with large carrier effective masses owing to a mismatch in angular momentum of the frontier atomic orbitals. The spectroscopically limited photovoltaic conversion efficiency is less than 10% for X = Cl or Br. This limitation can be overcome by replacing Ag with In or Tl; however, the resulting compounds are predicted to be unstable thermodynamically. The search for nontoxic bismuth perovskites must expand beyond the Cs2AgBiX6 motif.