The speaker is travelling north to paint a locksmith's house on a remote place called Rhubarb Farm, but the trip is clearly about more than a job. This is where the speaker left "the memory of my first born": a bedroom, a dirty bib, an empty crib in the cold and dark. The poem never says outright what happened, but the imagery suggests a child's loss, and the title, "Elegy," points the same way.
The speaker feels weighed down and unsettled. The third stanza looks back on "many lives" in the house, none of them strong enough to drown out the world's noise or to stay and "settle in." The locks, the boarded windows and the crib "locked out from critical eyes" suggest a grief kept hidden. By the end, the speaker plans to pry boards from the windows, which hints at opening up painful memories. The mood is bleak and quiet, with a small sense of facing the past.