A woman asks a man why they cannot be together, and the poem is his answer. He begins by asking for one last embrace and a "innocent" kiss on the forehead, but he undercuts himself almost at once, admitting that her quick kisses on his cheek had a way of turning into something more. He takes the blame for that, saying the sin was "never in your giving, but my taking."
Then he explains why he cannot leave his wife. After so many years together, he says, it is hard to tell where one of them ends and the other begins. They share tears and memories, and leaving her would be like packing a bag to leave the universe. He once promised they would be buried side by side, and he means to keep it.
The mood is tender, sad, and a little desperate. The title fits: he keeps leaning toward the other woman and pulling back. His final plea, "please / Let me go," is unclear. He may be asking her to release him, or asking himself to be released from temptation.