Phenolic recovery from agricultural byproducts has been highlighted due to their health-promoting bioactivities. However, uncontrolled discard of residues after extraction process would induce environmental pollution and bioresource waste. In this study, biorefining of phenolic-rich rapeseed meal (RSM) and its defatted sample (dRSM) was attempted by holistic utilization of phenolic extract and residue separately. Phenolic removal could significantly improve residues' Cr(VI) adsorption capacities by about 21%, which presented extended physical surface and more released functional groups. Moreover, simulating raw material by remixing 3% separated phenolic extracts or main component sinapic acid therein with corresponding residues further improved about 12% adsorption efficiencies. These indicated that the different present forms of phenolics had opposite effects on Cr(VI) removal. While natural conjugational form inhibited hosts' biosorption, free form had enhanced functions for either extract or residue. Four optimal adsorption parameters (pH, adsorbent dosage, contact time and initial Cr(VI) concentration), three kinetic (pseudo-first order, pseudo-second order and intra-particle diffusion) models and two isotherms (Langmuir and Freundlich) were used to reveal the adsorption process. The maximum Cr(VI) adsorption capacity on residues could reach about 100 mg/g, which was superior to that of most biosorbents derived from agricultural byproducts, even some biochar. Together with the residues' advantages with everlasting capacity after 3 adsorption-desorption cycles and excellent abilities for adsorbing multiple co-existed metal ions (Cr(VI), Cd(II), Cu(II), Pb(II), Ni(II) and Zn(II)), phenolic recovery was first proved to be a new and sustainable strategy for modifying biosorbents from agricultural byproducts with zero waste.