We present a stochastic model of active vesicular transport and its role in cell polarization, which takes into account positive feedback between membrane-bound signaling molecules and cytoskeletal filaments. In particular, we consider the cytoplasmic transport of vesicles on a two-dimensional cytoskeletal network, in which a vesicle containing signaling molecules can randomly switch between a diffusing state and a state of directed motion along a filament. Using a quasi-steady-state analysis, we show how the resulting stochastic hybrid system can be reduced to an advection-diffusion equation with anisotropic and space-dependent diffusivity. This equation couples to a reaction-diffusion equation for the membrane-bound transport of signaling molecules. We use linear stability analysis to derive conditions for the growth of a precursor pattern for cell polarization, and we show that the geometry of the cytoskeletal filaments plays a crucial role in determining whether the cell is capable of spontaneous cell polarization or only polarizes in response to an external chemical gradient. As previously found in a simpler deterministic model with uniform and isotropic diffusion, the former occurs if filaments are nucleated at sites on the cell membrane (cortical actin), whereas the latter applies if the filaments nucleate from organizing sites within the cytoplasm (microtubule asters). This is consistent with experimental data on cell polarization in two distinct biological systems, namely, budding yeast and neuronal growth cones. Our more biophysically detailed model of motor transport allows us to determine how the conditions for spontaneous cell polarization depend on motor parameters such as mean speed and the rate of unbinding from filament tracks.