The speaker can't let go of a relationship that is clearly ending. They hold onto memories, wonder if that makes them greedy, and try to repeat "the last season," as if the past could be replayed. Underneath is a simple wish: to be looked at, and wanted again.
The whole poem rests on one image, the dead leaf. Love is compared to leaves that drop and blow away, and every detail of autumn, such as the wind, the rain and the empty sky, becomes a sign of distance between two people. Even a touch feels fragile, as though it might crumble. The strangest line is "You're my fifth season," which suggests the other person exists outside the normal cycle, and the speaker still sees them as "green" when everything else is dying. That gap between hope and reality is the heart of the poem.
The mood is quiet, sad and resigned. By the end, the falling leaves feel like tears, and the final lines accept what the speaker has been resisting.