The poem is a short, bitter portrait of a crowd the speaker clearly despises. The "speculators" of the title seem to be people who gamble or bet, and the "lords of chance" they serve have no use for ordinary pleasures. A pleasant day, which most of us would be glad of, means almost nothing to them.
The imagery is harsh and unflattering. The crowd looks like "lousy hair dressed in feathers": a mass of dirty, mangy stuff pretending to be fancy. Then the poem turns darker, with people "hanging one another's heads" and laughing at a wart or a red eye. The cruelty here is mocking and gleeful, and the laughter feels ugly rather than warm.
The last two lines land flatly. These people have "no beauty left to look upon / and no other mirth," so they only have each other's flaws to enjoy. The mood is disgusted and cold, with a sense of emptiness underneath the noise.