The title twists the old saying "absence makes the heart grow fonder," and the poem quietly plays against it. Here the speaker is happy about being apart from his wife for an afternoon. The word "absent" can be read two ways: she is away, or he is absent from her company, and either way the result is the same.
What stands out is the speaker's honesty about the feeling. He calls it "a mean joy," which suggests he knows it's a little guilty, even slightly selfish. He isn't saying he loves her less. The poem seems to admit that solitude can be a pleasure, even in a good marriage, and that it's hard to say so out loud.
The last line, "my hands full of quiet," makes silence something solid he can hold. The mood is calm, private, and a bit confessional, like a small secret shared with the reader.