The speaker describes themselves as someone left outside, though the poem never sounds bitter about it. The "committee against anything distasteful" is a funny, pointed image: a group of people who police what is acceptable and decide what belongs. The speaker says they are a stranger to it, so they don't share its fussy certainty, and they aren't part of its circle.
The second image is gentler. An old hat lying at the end of town is something discarded, but also harmless and a little humble. It suggests the speaker sees themselves as set aside, yet not destroyed.
The last two lines turn the poem. "I believe in doubt" flips the usual idea of belief, setting uncertainty against the committee's confidence. Wanting "to be content" suggests a quiet peace with that position. The mood is dry, calm and wry, with a small note of defiance. Being an outsider seems to leave the speaker free to question things.