"small-scale, petty" 1853, American English, colloquial, in reference to towns; see one + horse (n.). Probably from earlier use in reference to a carriage, sleigh, plow, etc., "drawn by a single horse" (1750); also "possessing only one horse" (of a farmer); hence "petty, on a small scale, of limited capacity or resources; inferior."
Shortly afterwards I took a stroll over the town. It was what is generally denominated a "one horse town," and I would think a pretty small pony at that. Two stores, one grocery, a stable, and four dwellings made up the sum of its buildings. ["Daguerreotyping in the Back Woods," in Yankee Notions, March, 1855]