The speaker is dragged back through years of painful memory, and the river is the way in. It runs "backward," so instead of carrying the speaker forward, it pulls toward the past. The odd phrase "that thin nation" suggests the river is a narrow strip of territory, something with borders and a kind of anger of its own, "all fury and revenge." The past seems to be a place the speaker can't help returning to, and it isn't a peaceful one.
The last two lines turn bitter. "Cruel men never suffer" sounds like a complaint about unfairness: those who hurt others walk away untouched. Then comes the surprising comparison to "the afternoon office," which is called barbarous. The poem seems to suggest that cruelty can live in ordinary, tidy places, not just in wild ones. The mood is angry, weary, and unsettled, with a feeling that old wrongs are still unresolved.