1837, contraction of and per se and, meaning "(the character) '&' by itself is 'and' " (a hybrid phrase, partly in Latin, partly in English). An earlier form of it was colloquial ampassy (1706). The distinction is to avoid confusion with & in such formations as &c., a once common way of writing etc. (the et in et cetera is Latin for "and"). The letters a, I, and o also formerly (15c.-16c.) were written a per se, etc., especially when standing alone as words.
The symbol is based on the Latin word et "and," and comes from an old Roman system of shorthand signs (ligatures) attested in Pompeiian graffiti, and not (as sometimes stated) from the Tironian Notes, which was a different form of shorthand, probably invented by Cicero's companion Marcus Tullius Tiro, which used a different symbol, something like a reversed capital gamma, to indicate et. This Tironian symbol was maintained by some medieval scribes, including Anglo-Saxon chroniclers, who sprinkled their works with a symbol like a numeral 7 to indicate the word and.
In old schoolbooks the ampersand was printed at the end of the alphabet and thus by 1880s the word ampersand had acquired a slang sense of "posterior, rear end, hindquarters."