The speaker remembers a household shaken by loss and a new man's arrival. The horse comes back with an empty saddle, which suggests its rider is gone, and the poem implies this was the speaker's father. After the mother remarries, the stepfather shoots the horse. The poem states this flatly, without comment, and that plainness makes it hit harder.
The rest looks back at the horse's grief-like behavior. It lets only the speaker ride it, and it panics at loud noises. During storms it circles the pasture in a "frantic drumroll." Then the poem shifts: the speaker hears a heavy scraping, the Sunday roast being dragged across a table, from underneath, crouched among the chairs, hungry and watching those boots. The comparison is left unspoken, but the child seems to hear the horse's panic in the household itself, and to feel just as wary of the stepfather. The mood is tense, quiet, and fearful, with grief held in check.