Middle English crome, crumme, from Old English cruma "fragment of bread or other food, a morsel, small fragment," from a West Germanic root of obscure origin (compare Middle Dutch crume, Dutch kruim, German Krume); perhaps from a PIE word for "small particle of bread" and cognate with Greek grumea "bag or chest for old clothes" (Beekes writes: "In origin, the word probably denoted small things of little value, later also the chest, etc.), Albanian grime.
The unetymological -b- appeared mid-15c., in part by analogy with words like dumb. Slang meaning "lousy person" is 1918, from crumb, U.S. slang for "body-louse" (1863), which were so called from resemblance.