The speaker recalls three men who came to her differently. Strephon kissed her in the spring, but his kiss seemed like a joke. Robin kissed her in the fall, but his kiss was only play. Colin did not kiss her at all—he simply looked at her. That look is the one she cannot get out of her mind.
It “haunts” her night and day, which suggests longing, regret, or a sense of unfinished feeling. The two actual kisses fade because they lacked seriousness. Colin’s gaze contains something the others’ kisses could not: a quiet intensity that feels more intimate. The poem’s mood is tender and slightly sorrowful, as if a wordless look has outlasted all spoken or physical gestures.