Measurements of fragmentation functions for jets associated with an isolated photon are presented for the first time in pp and Pb-Pb collisions. The analysis uses data collected with the CMS detector at the CERN LHC at a nucleon-nucleon center-of-mass energy of 5.02 TeV. Fragmentation functions are obtained for jets with pjetT>30  GeV/c in events containing an isolated photon with pγT>60  GeV/c, using charged tracks with transverse momentum ptrkT>1  GeV/c in a cone around the jet axis. The association with an isolated photon constrains the initial pT and azimuthal angle of the parton whose shower produced the jet. For central Pb-Pb collisions, modifications of the jet fragmentation functions are observed when compared to those measured in pp collisions, while no significant differences are found in the 50% most peripheral collisions. Jets in central Pb-Pb events show an excess (depletion) of low (high) pT particles, with a transition around 3  GeV/c. This measurement shows for the first time the in-medium shower modifications of partons (quark dominated) with well-defined initial kinematics. It constitutes a new well-controlled reference for testing theoretical models of the parton passage through the quark-gluon plasma.