This 1989 song by alternative rock band The Pixies plays while Rory and Jess are in the record store. Written and sung by the band’s frontman, Black Francis, it was released as a single from their 1988 album Doolittle. The song received a positive reaction from critics and went to #5 on the Billboard Modern Rock Tracks, and to #60 in the UK. It is considered one of the greatest songs ever.
Although the lyrics reference biblical numerology and environmental concerns, I think the song was chosen simply because it was a believable choice for a cool indie record store, and to indicate that Jess (a naughty little monkey!) is simply in heaven, being in a great record store with Rory, the girl he likes.
The Annotated Gilmore Girls 