The poem sketches a remote seaside place where the harbor has fallen into ruin and the people who remain are old, quarrelsome, and hiding. The title, "In exile," frames all of this: these are people living away from where they belong, and the poem suggests they are cut off rather than simply retired to a quiet coast.
The speaker speaks as part of a group ("we are all old"), and the feeling is weary, bitter, and cornered. The line "must not be found" hints at a need to stay unseen, though the poem never says why. Their swearing at one another reads as frustration with confinement more than real hatred.
The ending turns to the sea, where "we see the dirtiness / of the road." The sea seems to work as a place of distance and reflection, from which the path that led them here looks grubby and disappointing. The mood is bleak, tired, and quietly resentful, with very little comfort offered.