The speaker sounds like someone taking stock of a life and not being sure what the tally adds up to. In the first lines, pleasure is something to be measured, and what the speaker finds is "thin things and fat", a mixed bag with no clear verdict. Then comes a quiet, slightly weary request: let desires leave me alone, since they are "too little for any use." Wanting something, the poem suggests, isn't worth much if the wanting is too small to act on.
The last stanza turns on itself. The speaker tells themselves not to blame "now" for "never", meaning the present moment shouldn't be punished for chances that have already slipped away. The "longest unpleasant afternoon" gives that idea a physical feel: time dragging, nothing happening, regret hanging in the air. The mood is dry, restrained and a little rueful. It isn't dramatic grief, more the flat ache of a dull hour in which the speaker tries, not very successfully, to be fair to the present.