The speaker pictures himself as a stone in a dish of peas: something hard, grey and out of place among soft green things. The title, "Hillbilly," suggests he sees himself as a rural outsider, and the line "I have come far" hints at a journey from somewhere else into this town. Now he is on display, and women look at him openly, "up and down," the way you might inspect something odd.
The poem plays on that phrase. The ladies look him up and down, and then some of them walk up and down the town, so the same motion shifts from a stare to a stroll. That echo seems to make the speaker pause, and the last line, "which makes me think," leaves the thought unfinished. The mood is wry and a little self-conscious. He feels awkward and noticed, but he also seems amused, watching the watchers and wondering what they make of him, and what he makes of them.