This paper links the nonequilibrium glassy relaxation behavior of otherwise athermal granular materials to those of thermally activated glasses. Thus, it demonstrates a much wider universality among complex glassy materials out of equilibrium. Our three-dimensional molecular dynamics simulations, fully incorporating friction and inelastic collisions, are designed to reproduce experimental behavior of tapped granular piles. A simple theory based on a dynamics of records explains why the typical phenomenology of annealing and aging after a quench should extend to such granular matter, activated by taps, beyond the more familiar realm of polymers, colloids, and magnetic materials that all exhibit thermal fluctuations.