The title gives the clue: these are neuroses, the kind of irrational fears and discomforts that attach themselves to harmless things. The speaker looks at a rosebush and finds it cruel, then finds the rose itself frightening, and the comparison to "a ship in the Navy" makes the fear feel huge, grey and impersonal, though the link between the two is deliberately odd. Logic doesn't quite hold, and that's the point. Anxiety rarely explains itself.
The shame about "the timber in a book" is just as strange. A book is made of wood, and the speaker seems troubled by that hidden origin, as if paper were a small betrayal of trees. The last lines move from objects to thought itself: simply imagining a God turns the speaker's head into "a well of ice." The mood is uneasy, cold and quietly absurd. Everything, even beauty and belief, seems to carry a chill the speaker can't shake.