The speaker visits the Jardin du Palais Royal every morning and always sees the same figure: the leaf sweeper, who turns up to do his work because leaves keep falling, some days more than others. The line "there is no bad job" sounds cheerful, but it also reads as a bit of self-persuasion. The sweeper has a purpose, and the speaker, by contrast, finds "nothing waiting" for them there.
That gap is the heart of the poem. The speaker seems restless and a little lonely, drifting through a daily routine without a point. The odd line about "every Englishman" being a "you" hints at someone absent or longed for, though the poem doesn't say who. Envy creeps in too: the sweeper has a reason to look down, while the speaker has nothing to look for. The ending is flat and quietly resigned. The leaves fall "regardless of you," and the speaker could always become a leaf sweeper. The mood is wistful, deadpan and gently sad.