Gilbert-like damping in magnetization dynamics is commonly attributed to the interplay of the spin, the electron, and the phonon reservoirs. Spatial correlations within the spin reservoir itself, for example magnons, mediate damping as well. We show theoretically that temporal correlations within the spin reservoir cause a similar effect. We investigate the role of time retardation in the atomistic Landau-Lifshitz-Gilbert equation using two different retardation kernels. Although viscous damping is explicitly excluded, we find both analytically and numerically that damping and higher-order effects emerge due to time retardation. Thus, our results establish a mechanism for damping and inertia in magnetic systems.