Two short stanzas, and the poem is already full of tension. The first celebrates something a little naughty: the "joy of an oath," a curse that slips out when nobody else is listening. "Private discourse" suggests talk behind closed doors, perhaps with oneself or with someone close, where swearing feels freeing rather than rude.
The second stanza shifts the picture. "New chimneys falling / between my wife and me" is odd and a bit ambiguous. It could be read as a literal image of something collapsing in the space between two people, or as a way of describing quarrels, silences, or small collapses in a marriage. The word "new" is worth noticing: the chimneys haven't had time to age, so what falls is fresh and perhaps unexpected.
The mood is playful but uneasy. The speaker seems to enjoy cussing, yet the second image hints at a cost. The title, "Cussedness," points to stubbornness as much as swearing, and the poem leaves us wondering whether that stubbornness is funny, damaging, or both.