early 14c., "a piece of cloth," especially a rectangular piece, from Old French panel "piece of cloth, piece, saddle cushion" (Modern French panneau), from Vulgar Latin *pannellus, diminutive of Latin pannus "piece of cloth" (see pane).
Anglo-French legalese sense of "piece of parchment (cloth) listing the names of those summoned to serve upon a jury" led by late 14c. to the meaning "a jury selected for a trial." General sense of "persons called on to advise, judge, discuss," etc. is from 1570s. Sense of "more or less distinct part of the surface of a wall, door, etc." is recorded from c. 1600.
Panel-house (said to be from 1840s; popular from 1870s) was old slang for a disreputable place (typically a bordello) with panneled rooms. At least one panel could be slid back to allow for thefts from customers and other cheats. Hence panel-thief, panel-game, etc.
The requisites for a "panel house" in the proper sense, are,—a crafty, cunning street walker; a not less cunning and at the same time sturdy scoundrel—known in the slang of the business as a "Badger," and a room prepared specially for the purpose by having a small invisible opening, generally a noiselessly opening panel in the partition or entrance door, by which access to the place can be had from an adjoining room. These three requisites obtained, it becomes the duty of the panel-thief to find the fourth in any "greenhorn" that can be picked up on the streets and induced to come into the apartment. ["The Dark Side of New York Life and its Criminal Classes," 1873]