The speaker spends the whole day longing for someone: wanting to hear their voice and feel their skin. The first three lines are simple and almost gentle, built on that repeated "to." Then the poem turns with a blunt question, "But what are you doing?" The answer stings. The person is with another woman, and treats her "like she's your best friend."
The last two lines carry the real hurt. "I thought, / I have been" is clipped and a little broken. The speaker seems to be saying that they believed they were the one who held that place. The line break after "I thought" makes it feel like a pause to swallow a lump in the throat.
The mood is quiet, raw, and wounded rather than angry. There are no accusations, only a small, painful realization of being replaced, or at least of not being as special as the speaker hoped. The shortness of the lines makes the feeling seem unguarded and honest.