kind of thick, soft leather, 1570s, buffe leather "leather made of buffalo hide," from Middle French buffle "buffalo" (15c., via Italian, from Latin bufalus; see buffalo (n.)).
The color term "light brownish-yellow" (by 1788) comes from the hue of buff leather. Association of "hide" and "skin" led c. 1600 to the sense in in the buff "naked." Buff-colored uniforms of New York City volunteer firefighters since 1820s led to the meaning "enthusiast" (1903).
These men, together with a score or more of young boys who cherish ambitions to be firemen some day, make up the unofficial Fire Department of New York, and any one who imagines they are not a valuable branch of the service need only ask any firemen [sic] what he thinks of the Buffs to find out his mistake. The Buffs are men and boys whose love of fires, fire-fighting and firemen is a predominant characteristic, who simply cannot keep away from fires, no matter at what time of the day or night they occur, or how long they continue. [New York Sun, Feb. 4, 1903]