The speaker sits in a small, private pool of light, with a candle that seems almost foolish beside the moon. It feels like "a stranger," out of place in the bigger glow outside. Into this scene comes a woman who speaks Latin, and the speaker says plainly that he loves her books. Whether he loves her too is left open, and the poem seems to blur the two: her language and her books are how she appears to him.
The last lines are the most striking. Reading by "an inch of candle," he watches the images "burn" in a "sovereign, singing tongue." Latin becomes something alive, almost musical, and fire and language merge. The title, "Pentecostal," suggests the idea of tongues of flame, though the poem never spells that out. The mood is hushed, devoted, and a little awed. The speaker is humble and small, yet lit up by something much older than himself, and a short stretch of candle is enough for that.