There is considerable interest in the high-rate compaction of brittle granular materials such as sand. However, the vast majority of studies focus on a single granular system, limiting our ability to make comparisons between materials to discern how granular structure manifests as bulk material response. Here, three different silica sands with similar grain size and shape are studied: we compare a rough quarry sand, a smoother-grained sand, and a sandy loam. Quasi-static compaction and planar shock loading responses are compared, and recovered samples analyzed. The combination provides information regarding the interplay between granular properties, loading conditions, and material response. We show that the fundamental grain-scale behaviour depends on loading conditions: At low strain rates compaction behaviour is dominated by grain morphology, and in particular, smoothness and particle size distribution. Under shock loading, grain rearrangement and force chain effects are suppressed, and the nature of inter-granular contact points, modified by the presence of moisture or fines, is most important. Furthermore, grain fracture under shock loading is substantially reduced with increasing moisture content.