A daydream has left something behind, and the speaker is still dealing with it. Someone arrives "this morning," seemingly out of that dream, carrying the feel of "blue fields" and "backward months," as if time itself has run in reverse. The sense is of memory and fantasy blurring into the present day.
The central image is "unseasonable sex," which suggests desire that has come at the wrong time, out of step with the season or the speaker's life. It is compared to "an ear on a birch," barbarously aflutter. The picture is odd and vivid: a leaf, perhaps, trembling on a tree, but also something listening, alert and exposed. The word "barbarously" gives the feeling a wild, unruly edge.
The last two lines turn to the speaker, who is guarded and a little rueful: "my fence," "my embers of order." Order here is nearly burnt out, only glowing, while the feeling outside is loud and alive. The mood is tender, restless, and tinged with wistfulness.