1540s, "miserable person, wretch," from Latin miser (adj.) "unhappy, wretched, pitiable, in distress," a word for which "no acceptable PIE pedigree has been found" [de Vaan]. The oldest English sense now is obsolete; the main modern meaning of "money-hoarding person" ("one who in wealth conducts himself as one afflicted with poverty" - Century Dictionary) is recorded by 1560s, from the presumed unhappiness of such people. The older sense is preserved in miserable, misery, etc.
Besides general wretchedness, the Latin word connoted also "intense erotic love" (compare slang got it bad "deeply infatuated") and hence was a favorite word of Catullus. In Greek a miser was kyminopristes, literally "a cumin seed splitter." In Modern Greek, he might be called hekentabelones, literally "one who has sixty needles." The German word, filz, literally "felt," preserves the image of the felt slippers which the miser often wore in caricatures. Lettish mantrausis "miser" is literally "money-raker."