The speaker describes the only moments when they feel able to smile: the times "you" aren't anywhere to be found. Those moments come when they're with someone else, especially a best friend, where they can relax, joke around and simply "be us." The poem never says who "you" is, which leaves it open. It could be a partner, a family member or someone else with power over the speaker's mood. What is clear is that this person's presence acts like a weight, and the speaker feels they can't be their real, playful self in front of them.
The mood is quiet, tense and a little sad. The short, repeated "the times when" lines work like a list of small escapes, which makes the speaker's happiness feel fragile and temporary. The last line shifts the tone, landing bluntly on the problem: the freedom disappears as soon as "you" show up.