The speaker is always a step behind everyone else, and they say it isn't a choice: "I can't help it." What makes the feeling so tangled is the mix of "I hate it / And I love it." Being behind hurts, but it seems to offer something too, perhaps a quiet space of one's own, though the poem never says exactly what.
Other people move on quickly, forgetting the speaker and running far ahead, even past a house nobody invited them to. The speaker and "He" stay back together, hoping to be remembered and pulled into the conversation. The blunt line "But no" shuts that hope down.
The final stanza is bittersweet. Only when the speaker reaches the same destination will the others stop and remember them. The mood is lonely and resigned, but gently so. There's no anger, just a patient ache of wanting to be noticed, mixed with an acceptance of one's own slower pace.