The speaker has a skeptical view of fame, and the poem makes its case in a few short lines. Fame, as the speaker sees it, is a trade that goes the wrong way: the more people know your name, the less of you there actually is. Having more leaves you being less. It's a neat reversal, and the plain, almost sing-song rhythm makes it sound like a hard-won piece of wisdom.
The last stanza turns to a surprising image. Specks of dust, tossed about by unreliable winds, are said to value life more than blazing stars do. Stars are the obvious picture of fame, bright and admired and far above everyone else, yet the speaker suggests they are somehow empty of the things that make life precious. Dust is small and unnoticed, but it holds on. The mood is quiet, wry and a little consoling. The speaker seems to be saying that being ordinary may be a richer way to live than shining.