Three short lines set up a quiet, painful contrast. Overhead, swans are passing over the house, which the title tells us is happening in migration time, the season when birds move on. Then the poem turns to someone else: "his great body in pain / refusing to go." Who "he" is isn't spelled out. It could be a sick or dying man, perhaps someone close to the speaker, and the word "great" suggests a large, once-strong body. That is an inference from the text, not something the poem confirms.
What the poem makes us feel is the gap between the birds' easy departure and the man's struggle. The swans leave as the season demands, while his body clings on, "refusing to go." The speaker doesn't explain or comment, which gives the poem a hushed, watchful mood. There is sadness here, and maybe a kind of grudging respect for that stubbornness. The ordinary movement of nature overhead makes the human suffering below feel even heavier.