We describe fast algorithms for approximating the connection coefficients between a family of orthogonal polynomials and another family with a polynomially or rationally modified measure. The connection coefficients are computed via infinite-dimensional banded matrix factorizations and may be used to compute the modified Jacobi matrices all in linear complexity with respect to the truncation degree. A family of orthogonal polynomials with modified classical weights is constructed that support banded differentiation matrices, enabling sparse spectral methods with modified classical orthogonal polynomials. We present several applications and numerical experiments using an open source implementation which make direct use of these results.