A man sitting alone in the quiet tries to shut out a memory that keeps stirring. At first he isn't sure he heard anything at all, and he brushes it off. But the poem's images show how persistent it is: the memory is a restless sleeper "tossing and turning" in a "cold and old dead bed," and even with the blinds closed, he admits it still burns.
The speaker insists he isn't lonely, at least not in the way someone might assume. He looks back often as he grows older, and his thoughts wander off, then return at night, when the silence lets them in. That silence is where he is now, and he can hear them whether he wants to or not.
The ending is a kind of release. He addresses a "you" and hands over every memory he has of them. The mood is weary and sore, but also quietly defiant, as if giving the past away might finally let him rest.