The speaker is surrounded by endless talk and can't connect with any of it. The noise is "unfocused," a tumult that sounds like the symptom of a fever, as if the crowd itself were unwell. That fits the title's idea of madness, though the speaker seems more worn out and left behind than frightened.
When the speaker tries to listen to individual voices, each one sounds like a foreign language they can't speak. The people who do make sense to them are the silent ones, who might be lonely, thoughtful, or simply empty-minded. The speaker wonders whether these quiet people feel the same bewilderment about time, about how the past can slip from Spring into Winter so quickly.
That question gives the poem its ache. The mood is isolated and a little melancholy, with a sense of ageing or loss running underneath. The repeated "goes on forever" at the start and end, and the broken, scattered line breaks, make the noise feel unending and the speaker's search for kinship feel unresolved.