The poem is a single sentence, and it reads like a quiet realization. The speaker looks back on the times they spent with another person and admits that, even then, they felt alone. The opening, "It says something," sounds almost casual, as if the speaker is shrugging, but the shrug hides a real hurt. They have clearly been turning this over in their mind and have reached a conclusion they can't ignore.
The title, "Harbor," adds an interesting tension. A harbor is usually a place of shelter and safety, somewhere you come in from the storm. Placing that word above a line about loneliness suggests that the person was supposed to be a safe place and wasn't. Judging by the wording, the speaker feels let down, though they don't shout about it.
The mood is flat, sad, and still. There's no anger and no dramatic scene, just a calm, painful honesty. The shortness makes the feeling land harder: loneliness inside a relationship can be more isolating than being on your own.