At what age do old man balls kick in?

Or is it more of a gradual descent over a period of time?