The speaker spends a whole afternoon in a shed, and that simple fact becomes the poem's center. The shed is a small, plain, private place, and the speaker treats the time spent there as "my one habit." The word "one" matters. It suggests a life with few routines, or at least one that really belongs to the speaker, a thing they return to again and again.
Then comes the comparison: the habit is "like a hat / made for rough air." A hat is something you put on and carry with you, and one made for rough air is built for wind and bad weather. Read this way, the habit works as protection. Judging by the image, the shed afternoons may shield the speaker from whatever is turbulent outside, whether that's the world or their own restlessness.
The mood is quiet, a little wry, and content in a modest way. The poem doesn't explain what happens in the shed. It leaves us with the comfort of the habit itself.