The speaker is looking back on a love that began in fifth grade and never really let go. The poem opens and closes with almost the same lines, "It pained me / To see you with other girls," and that repetition is the point: years have passed, the speaker has grown up, and the feeling hasn't moved an inch.
There's a quiet sadness in how the speaker describes the relationship. They tried to make this person "mine and only mine," which sounds a bit possessive, but the poem seems more wistful than controlling. The most telling moment is the gap in memory: the other person "slowly forgot" what the speaker "would not." One person moved on, and the other kept everything. The speaker still misses the eyes and smile, and longs for a look given only "every once in a while." The mood is lonely, tender, and a little stuck, like someone replaying the past because it's all they have.