cardinal (n.) Look up cardinal at Dictionary.com
early 12c., "one of the ecclesiastical princes who constitute the sacred college" (short for cardinalis ecclesiae Romanae or episcopus cardinalis), from Latin cardinalis "principal, chief, essential" (see cardinal (adj.)).

Ecclesiastical use began for the presbyters of the chief (cardinal) churches of Rome. The North American songbird (Cardinalis virginianus) is attested from 1670s, so named for its resemblance to the cardinals in their red robes.
cardinal (adj.) Look up cardinal at Dictionary.com
"chief, pivotal," early 14c., from Latin cardinalis "principal, chief, essential," from cardo (genitive cardinis) "that on which something turns or depends; pole of the sky," originally "door hinge," which is of unknown origin. Related: Cardinally.

The cardinal points (1540s) are north, south, east, west. The cardinal sins (c. 1600) are too well known to require rehearsal. The cardinal virtues (c. 1300) were divided into natural (justice prudence, temperance, fortitude) and theological (faith, hope, charity). The natural ones were the original classical ones, which were amended by Christians. But typically in Middle English only the first four were counted as the cardinal virtues:
Of þe uour uirtues cardinales spekeþ moche þe yealde philosofes. ["Ayenbite of Inwyt," c. 1340]
By analogy of this, and cardinal points, cardinal winds, cardinal signs (four zodiacal signs marking the equinoxes and the solstices), the adjective in Middle English acquired an association with the number four.