The speaker is grieving the end of a relationship, though the poem never says exactly what happened. Each time they cry, a memory of "us" seems to slip away, as if tears are a way of draining the past out of them. These aren't happy memories, either. The speaker wishes they had never had them, which suggests the closeness they shared now only causes pain.
The mood is quiet and tired, with a thin thread of hope. The speaker notes that the memories are "nearly all gone" and soon will be, and imagines smiling "without a care in the world." But that final line is worth pausing on. It sounds like relief, yet it also carries a hint of numbness, since being free of every memory means losing the good along with the bad. The lack of a full stop at the end leaves the poem feeling unfinished, as though the healing is still underway and the speaker isn't sure it will feel as good as they hope.