The speaker describes Gray City Island, a cold, wet, stony place that sounds miserable. It is freezing in December and hardly warmer in August, and it has no passing prettiness to offer. Yet the stone holds up against time, "battling" it and "winning," which gives the island a stubborn kind of strength. The people who go there seem to be the ones who slip out of ordinary life: dreamers, the cursed, and a few who are neither.
The tone shifts as the poem goes on. The speaker first sounds like an outsider judging the place ("a horrible place to live"), then softens with "And yet / Perhaps / Some people enjoy it." The last lines are the most telling. The island may not be real at all, and the speaker admits, "I guess I'm just dreaming again." The mood is wistful and a little self-mocking, as if the speaker is both drawn to this bleak refuge and aware it may be only imagination.