The speaker heads for the pub the way a sinner heads for church. That comparison is a little backwards, and that's the fun of it. Church is where you go to be forgiven, and here the noise and banter ("din and wit") are the place of worship. Going there feels like a ritual, even a confession, though what's being confessed is a weakness the speaker seems happy to keep.
Then the poem turns and speaks to someone directly: "Here, walk with me." It reads like an invitation, or maybe a plea for company. The last line, "Tell me we all like to lose," asks for reassurance. The speaker wants to hear that this self-defeat is shared and ordinary, not a private failing. The mood is wry and warm on the surface, with a loneliness underneath. It's a joke told slightly too lightly, and the speaker half knows it. The poem is short, so those few words carry both the charm and the ache.