late 14c., "physically weak, enfeebled, crippled," from Old French impotent "powerless, weak, incapable of doing," from Latin imponentem (nominative impotens) "lacking control, powerless, feeble; lacking self-control," from assimilated form of in- "not, opposite of" (see in- (1)) + ponentem (nominative potens) "potent" (see potent).
Meaning "having no power to accomplish anything" is from mid-15c.; that of "completely lacking in sexual power" (of males) is from mid-15c. Middle English also had a native term for this: Cunt-beaten (mid-15c.). The figurative sense in Latin was "without self-control, headstrong, violent, ungovernable, lacking self-restraint," which sometimes is found in English (OED cites examples from Spenser, Massinger, Dryden, and Pope). Related: Impotently.