Right away, the speaker clears up a misunderstanding. They walked away, but not because their love faded, and not because it was never there. The odd phrasing "or even never did" suggests the speaker knows the other person might doubt the love was real, and wants to shut that door too.
The real reason arrives in the last line, which is longer and heavier than the rest: the speaker couldn't stand the thought of not being loved back. So leaving becomes a kind of protection. It's easier to be the one who goes than the one who is slowly abandoned, even if that means giving up something precious.
The mood is quiet, sad, and a little confessional. There's no anger here, just a raw admission of fear. The short, plain lines feel like someone speaking carefully, maybe holding back tears, finally telling the truth. The poem suggests that love and insecurity can exist side by side, and that sometimes fear does the damage.