The speaker is addressing an ex-lover and, at first, sounds like someone delivering a curse. Again and again they say "you'll never" smile, laugh, love, be held or be kissed the way they were when "you were with me." The repeated phrasing makes it feel like a drumbeat, as if the speaker is trying to convince themselves as much as the other person. Even the claim that they don't need to be there to watch the other person fade "just a little everyday" sounds more wounded than cruel.
Then the tone shifts. "But with all that said" marks a turn, and the anger gives way to something sadder. The speaker insists they were never absent and never "miles away." The distance, they say, was in the other person's head. The final lines, with the closed door and the repeated "you just wouldn't let me in," leave a mood of hurt, regret and quiet helplessness.