The speaker is apologizing for the end of a relationship, and the poem pictures that relationship as a castle. It was built slowly, "over years," and then wrecked in a moment. The speaker admits fault from the start ("this was not meant to happen") and keeps returning to the idea of mistakes, even saying they will never forget the ones they made.
The imagery turns gentle and sad as the poem goes on. Ivy climbs the walls, the stone crumbles, and plants take over, so the ruin looks quiet rather than violent. The two people become "new separate castles," each far away, which suggests they will carry on but apart. The ending shifts to a plea. The speaker begs the other person to be careful, since one wrong word or sudden movement can "kill everything beautiful." The mood is regretful and tender, with a note of warning. It reads like someone learning, too late, how fragile something precious was.