1580s as a type of knitted cloth; 1842 as a breed of cattle; both from Jersey, one of the Channel Islands. Its name is said to be a corruption of Latin Caesarea, the Roman name for the island (or another near it), influenced by Old English ey "island;" but it is perhaps rather a Viking name (perhaps meaning "Geirr's island").
The meaning "woolen knitted close-fitting tunic," especially one worn during sporting events, is from 1845. In American English, short for New Jersey from 1758. Related: Jerseyman.