The speaker describes a quiet, transactional kind of intimacy. He takes the woman's wine but refuses her coffee, and the choice says a lot. Wine belongs to evenings and bodies, while coffee suggests sitting down and talking. By drinking one and turning down the other, he keeps the encounter physical and avoids anything closer.
The second stanza spells it out: "I am buying love / not an ear." Love here seems to be something he pays for, and he doesn't want to be listened to or to listen. The title, "Silence lover," fits this. He may be a lover of silence, or someone who loves in silence, and the poem leaves both readings open.
The last lines, "evening comes with few words / to my bed," bring a sparse, subdued mood. There is little warmth, but also little drama. The poem feels guarded and a little lonely, as if the speaker has arranged his life to avoid being known, and the price of that is quiet.