The poem looks back on a series of secret meetings between two people, held night after night in a church. The church is "red with need," which suggests the place is charged with desire as much as devotion. The speaker and their lover seem to come together almost as a way of escaping mortality, to "free our hands of death," yet what they build together is doomed to "go bust." Nothing they make seems able to last.
The image of being "blessed as a swan in the palace" is a little puzzling. It could point to a moment of grace and beauty, though the swan also seems out of place, perhaps protected but not truly free. The speaker holds on to that feeling for as long as possible.
The ending is quiet and sad. Their "last words" are "washed out with the rain," so the farewell is lost, unheard or forgotten. The mood is tender, hungry, and wistful, as if the speaker is mourning something intense that vanished without a proper goodbye.