1715, "yellowish-gray; of the color of natural, undyed cloth," from the trade name for the color itself (1680s), which is from an earlier noun drab, drap meaning "thick, woolen cloth of a yellowish-gray color" (1540s), from Middle French drap "cloth, piece of cloth" (see drape (v.)). The figurative sense of "dull, not bright or colorful" is by 1880.
Apparently this word is not related to earlier noun drab "a dirty, untidy woman" (1510s), "a prostitute" (1520s), which might be from Irish drabog, Gaelic drabag "dirty woman," or perhaps it is connected with Dutch and Low German drabbe "dirt;" compare drabble. The notion seems to be of dabbling in the wet and mud.
The meaning "small, petty debt" (the sense in dribs and drabs) is by 1828, of uncertain connection to the other senses.