The speaker is tired of gossip. "Town-talk" suggests the endless chatter of a small community, and the poem opens with that weariness: the speaker is simply sick of it. What follows is a kind of retreat. The speaker hears "nothing hard," which could mean no harsh words reach them, or that the noise has softened into something easy to ignore.
The last two lines carry the main image. A "fuddle of wool" suggests a muffled, woolly confusion, the sort of mental fog that talk and fuss can produce. The speaker says their wits are never "overtaken" by it, so their mind stays clear even while the town goes on murmuring. The title, "Above the fray," supports this reading: the speaker has lifted themselves out of the quarrel.
The mood is calm, a little dry, and quietly confident. It is short and spare, and the brevity echoes the speaker's wish for quiet and clarity. There is also a hint of private satisfaction in staying unruffled.