15c. metathesis of Middle English drit, drytt "excrement, dung, feces, any foul or filthy substance," also "mud, earth," especially "loose earth" (c. 1300), from Old Norse drit, cognate with Old English dritan "to void excrement," from Proto-Germanic *dritan (source also of Dutch drijten, Old High German trizan).
Used abusively of persons from c. 1300; figurative of something worthless from early 14c. Meaning "gossip" first attested 1926 (in Hemingway).
As an adjective, "consisting or made of loose earth," by 1860. The dirt-bike is attested by 1970. Dirt-cheap "as cheap as dirt" is by 1766; dirt-poor "extremely poor" is by 1906. Dirt road, one not paved or macadamized, is attested by 1835, American English. Pay-dirt "earth containing gold" is by 1857, originally California miners' slang.
It is customary to speak of "the golden sands of California;" but a person who should believe that the gold is found in pure sand, would be far wrong. Usually, the pay-dirt is a very stiff clay, full of large gravel and stones. The depth of this pay-dirt varies. In a gully where the water is not more than five feet wide in the heaviest rain, the pay dirt will not usually be more than a foot deep. (etc.) [John S. Hittell, "Mining in the Pacific States of North America," San Francisco, 1861]