3.2 Buses and Interfaces

The buses and interfaces are the fabric that connects the CPU to the peripherals that are part of the system. Each bus and interface has its own intricacies, and the level of support provided by Linux to the different buses and interfaces varies accordingly. The following is a rundown of the buses and interfaces found in embedded systems and a discussion of their support by Linux. Linux supports many other buses, such as SBus, NuBus, TurboChannel, and MCA, but these are workstation or server-centric.

3.2.1 ISA

The Industry Standard Architecture (ISA) bus was designed for and occupied the core of PC-AT architecture. It was odd even for its time, as it did not provide many of the facilities other buses offered, including ease of mapping into normal processor physical address space. Its simplicity, however, favored the proliferation of many devices for the PC, which, in turn, favored the use of PCs in embedded applications.

ISA devices are mostly accessed through the I/O port programming already available in the x86's instruction set. Therefore, the kernel does not need to do any work to enable device drivers to use the bus. Instead, the various device drivers access the appropriate I/O ports directly using the in/out assembly functions. Although the kernel provides support for Plug and Play (PNP) devices, this capability is of little use for embedded applications. Instead, embedded systems that do need to support hardware variations will be based on buses that support runtime hardware addition and removal, such as CompactPCI, PCMCIA, and USB. The kernel also supports Extended ISA (EISA) devices, but this bus has not been very popular and has been superseded by the PCI bus.

Information regarding the ISA bus can be found in many places. The PC Handbook and HelpPC mentioned above are good quick references for port numbers and their operation. ISA System Architecture by Anderson and Shanley (Addison Wesley) is an in-depth explanation of the operation of the ISA bus and related hardware. Also, Linux Device Drivers by Rubini and Corbet (O'Reilly) contains details about ISA programming in Linux.

3.2.2 PCI

The Peripheral Component Interconnect (PCI) bus, managed by the PCI Special Interest Group (PCI-SIG), is arguably the most popular bus currently available. Designed as a replacement for ISA, it is used in combination with many different architectures, including the PPC and the MIPS, to build different types of systems, including embedded devices.

Unlike ISA, PCI requires software support to enable it to be used by device drivers. The first part of this support is required to initialize and configure the PCI devices upon bootup. On PC systems, this is traditionally done by the BIOS. However, the kernel is capable of carrying out this task itself. If the BIOS has carried out the initialization, the kernel will browse the BIOS's table to retrieve the PCI information. In both cases, the kernel provides device drivers with an API to access information regarding the devices on the PCI bus and act on these devices. There are also a number of user tools for manipulating PCI devices. In short, the level of support for PCI in Linux is fairly complete and mature.

The Linux Device Drivers book mentioned above provides very good insight about PCI development in Linux and how the PCI bus operates in general. The PCI System Architecture book by Shanely and Anderson (Addison Wesley) gives in-depth information on the PCI bus for software developers. Of course, you can always get the official PCI specification from the PCI-SIG. Official specifications, however, tend to make for very dry reading material. Finally, there is the Linux PCI-HOWTO, available from the Linux Documentation Project (LDP) at http://www.tldp.org/, which discusses the caveats of using certain PCI devices with Linux and the support provided to PCI devices by Linux in general.

3.2.3 PCMCIA

Personal Computer Memory Card International Association (PCMCIA) is both the common name of a bus and the name of the organization that promotes and maintains related standards. Since the publication of the initial standard, which supported only 16-bit cards, other standards have been published, including the 32-bit CardBus and the USB CardBay specifications. When part of an embedded system, PCMCIA renders it flexible and easy to extend. On the iPAQ, for instance, it enables users to connect to a LAN using a wireless networking card. In other systems, it makes large permanent storage space available through the use of CompactFlash cards.

The extent of Linux support for PCMCIA can be confusing. First and foremost, there is the main Linux PCMCIA project, which is hosted on SourceForge at http://pcmcia-cs.sourceforge.net/ and is maintained by David Hinds. The package made available by this project supports a large number of cards, listed at http://pcmcia-cs.sourceforge.net/ftp/SUPPORTED.CARDS. Linux support for PCMCIA is quite mature for the i386 architecture and available in part for the PPC, but unfortunately, it's still in its infancy for other chips at the time of this writing. Apart from the package maintained by Hinds, the official kernel contains support for a portion of the PCMCIA cards supported by the Hinds' package. The developers' intent is to have the official kernel become the main source for PCMCIA support. Until then, the best choice is to use Hinds' distribution for production systems. It includes the necessary system tools to configure the automatic loading and unloading of the appropriate PCMCIA device drivers when a card is inserted or removed from a PCMCIA slot.

Apart from the official PC Card Standard available from the PCMCIA association itself, there are a number of books on PCMCIA. However, before investigating those works, you should read the Linux PCMCIA Programmer's Guide written by Hinds and available on the PCMCIA project's web site. This guide includes references to books that provide more information regarding PCMCIA.

3.2.4 PC/104

Although simple, the ISA bus is not well adapted to the rugged environments where embedded systems are deployed. The PC/104 form factor was introduced to address the shortcomings of ISA's mechanical specification. PC/104 provides a bus whose electrical signals are identical to those of the ISA bus, but with a different mechanical specification that is more adapted to embedded system development by providing ease of extensibility and ruggedness. Instead of using slots where cards are inserted, as in a regular ISA setup, PC/104 specifies the use of pin connectors. When PCI became popular, the PC/104+ specification was introduced to provide a PCI-signal-compatible bus as an addition to the PC/104 specification. Both PC/104 and PC/104+ are managed by the PC/104 Consortium, which includes more than a 100 member companies.

The PC/104 is identical to ISA and the PC/104+ is identical to both ISA and PCI from the signal perspective and, therefore, from the software's perspective. Therefore, Linux requires no special functionality to support these buses. However, this does not mean that Linux supports all PC/104 and PC/104+ devices. As with any other ISA or PCI device, you should seek exact information about Linux compatibility with the PC/104 device you are evaluating.

3.2.5 VME

The VME^[3] bus is largely based on Motorola's VERSA backplane bus, which was developed specifically for the 68000 in 1979. At the time, VERSA was competing with buses such as Multibus, STD, S-100, and Q-bus, although it is rarely used today. Striving to provide a new bus that would be microprocessor independent, Motorola, Mostek, and Signetics agreed that the three companies would support a new bus. This came to be the VME bus based on the VERSA's electrical signals and the Eurocard mechanical form factor. In the Eurocard form factor, VME boards are loaded vertically into a VME chassis and connected to its backplane using pin connectors, unlike common computer boards that use edge connectors inserted into slots. Since its introduction, the VME bus has become widely adopted as the preferred bus for building powerful and rugged computers. One factor that has helped the VME bus' popularity is that it is an open standard that isn't controlled by any single organization.

^[3] Although "officially" the letters VME mean nothing, it has been revealed by an engineer taking part in the discussions between the three companies that it is short for "VERSA Module Eurocard."

As the VME bus can accommodate multiple VME boards, each with its own CPU and OS, no central OS controls the bus. Instead, arbitration is used to permit a board to become bus master momentarily to conduct its operations. The job of Linux on a VME board is therefore to interact properly with its VME hardware interface to obtain the appropriate functionality.

There are currently two active Linux VME projects. The first aims at providing Motorola 68K-based boards with Linux support and can be found at http://www.sleepie.demon.co.uk/linuxvme/. Although the work of this project has since largely been integrated into the main kernel tree, the project's site is still the main resource for recent developments and news. The second project aims at providing Linux support for all VME boards, regardless of their CPU. It is called the VMELinux Project and can be found at http://www.vmelinux.org/. In each case, support is provided for each board individually. The supported boards are listed on each project's web site. So, when assessing whether your VME board is supported by Linux, you should look for support for the exact model you have. If the board you've chosen isn't supported yet, support for other board models will help provide you with examples on how to implement support for your VME board in Linux.

In addition to these two projects, a couple of software and hardware vendors provide Linux support for additional VME hardware within their own distributions. The kernel maintained by DENX Software Engineering and available from their web site using CVS, for example, provides support for various PPC-based boards not supported by the VMELinux project.

From the Linux perspective, the Linux VME HOWTO is available on the LDP's web site. The VMEbus International Trade Association (VITA) web site contains a number of recommended publications regarding the VME bus in general and the related standards. Missing from this list, though of interest, is John Black's The Systems Engineer's Handbook: A guide to building VMEbus and VXIbus systems.

3.2.6 CompactPCI

The CompactPCI specification was initiated by Ziatech and was developed by members of the PCI Industrial Computer Manufacturer's Group (PICMG), which oversees the specification and promotes the use of CompactPCI. The CompactPCI specification provides an open and versatile platform for high-performance, high-availability applications. Its success is largely based on the technical choices made by its designers. First, they chose to reuse the Eurocard form-factor popularized by VME. Second, they chose to make the bus PCI-compatible, hence enabling CompactPCI board manufacturers to reuse low-cost PCI chips already available in the mainstream market.

Technically, the CompactPCI bus is electrically identical to the PCI bus. Instead of using slot connections, as found in most workstations and servers, pin connectors are used to connect the vertically loaded CompactPCI boards to the CompactPCI backplane, much like VME. As with PCI, CompactPCI requires a single bus master,^[4] in contrast with VME, which could tolerate multiple bus masters, as explained earlier. Consequently, CompactPCI requires the permanent presence of a board in the system slot. It is this board that arbitrates the CompactPCI backplane, just as a PCI chipset would arbitrate a PCI bus in a workstation or a server.

^[4] The term "bus master" can mean different things in different contexts. In this particular instance, "bus master" designates the device that sets up and configures the PCI bus. There can be only one such device on a PCI bus, though more than one device on a PCI bus may actually be able to access the memory regions exported by other PCI devices.

In addition, the CompactPCI specification allows for the implementation of the Hot Swap specification, which describes methods and procedures for runtime insertion and removal of CompactPCI boards. This specification defines three levels of hot swapping. Each level implies a set of hardware and software capabilities. Here are the available levels and their requirements:

Basic hot swap

This hot swap level involves console intervention by the system operator. When a new card is inserted, she must manually inform the OS to power it up and then configure and inform the software of its presence. To remove a card, she must tell the OS that the board is about to be removed. The OS must then stop the tasks that are interacting with the board and inform the board to shut down.

Full hot swap

In contrast to basic hot swap, full hot swap does not require console intervention by the operator. Instead, the operator flips a microswitch attached to the card injector/ejector to notify the OS of the impending removal. The OS then performs the necessary operations to isolate the board and tell it to shut down. Finally, the OS lights an LED to notify the operator that the board can now be removed. On insertion, the OS carries out the inverse operations when it receives the appropriate insertion signal.

High Availability

In this level, CompactPCI boards are under complete software control. A hot swap controller software manages the state of all the boards in the system and can selectively reverse these individual boards according to the system's state. If a board fails, for example, the controller can shut it down and power up a duplicate board that is present within the same chassis for this very purpose. This hot swap level is called "High Availability," because it is mostly useful in what are known as high-availability applications,^[5] such as telecommunications, where downtime must be minimal.

^[5] To avoid any confusion, I will refer to this hot swap level as "High Availability hot swap level" and will continue to use the "high-availability" adjective to refer to applications and systems who need to provide a high level of availability, regardless of whether they use CompactPCI or implement the "High Availability hot swap level."

Linux accommodates the basic CompactPCI specification, through the PCI support it already provides. Support for dynamic insertion and removal of devices in Linux also exists in different forms. Primarily, Version 2.4 of the kernel includes the required kernel functionality. The associated user tools are available through the Linux Hotplugging project at http://linux-hotplug.sourceforge.net/.

That said, this level of support is insufficient to accommodate all the complexities of CompactPCI systems. In addition, there are few drivers within the main kernel tree for mainstream CompactPCI boards, although CompactPCI board manufacturers may provide Linux drivers. This caveat has led to the emergence of a number of commercial solutions that provide high-availability Linux solutions on CompactPCI, including Availix's HA Cluster and MontaVista's High Availability Framework. The ongoing High-Availability Linux Project, found at http://linux-ha.org/, aims at providing the open source components needed to build high-availability solutions using Linux. The project isn't restricted to a specific hardware configuration and is, therefore, not centered around CompactPCI.

In the future, we may see more open source software accommodating the various complexities of CompactPCI-based systems, both in terms of hot swap capabilities and in terms of software support for communication, resource monitoring, cluster management, and other software components found in high-availability systems. For now, however, if you want to use Linux for a CompactPCI-based high-availability application, you may need to work with one of the existing commercial solutions to obtain all the features described by the CompactPCI specification.

Documentation regarding Linux's hotplug capabilities, including how to write hotplug-aware drivers and how to configure the hotplug management tools, is available through the Linux Hotplugging project web site. The web site also includes a number of links to relevant information. Information regarding CompactPCI specifications can be purchased from PICMG.

3.2.7 Parallel Port

Although not a bus in the traditional sense, the parallel port found in many computers can be used to connect a wide range of peripherals, including hard drives, scanners, and even network adapters. Linux support for parallel port devices is extensive, both in terms of the drivers found in the kernel and the ones provided by supporting projects. There is no central authority or project, however, that directs Linux's support for parallel port devices, since the parallel port is a ubiquitous component of computer systems. Instead, there are good resources that describe which devices are supported. These include the Hardware Compatibility HOWTO found at the LDP and the Linux Parallel Port Home Page found at http://www.torque.net/linux-pp.html. It is worth noting that Linux supports the IEEE1284 standard that defines how parallel port communication with external devices is carried out.

As the parallel port can be used for many purposes besides attaching external devices, I will discuss parallel port programming when explaining the use of the parallel port as in an I/O interface in Section 3.3.

3.2.8 SCSI

The Small Computer Systems Interface (SCSI) was introduced by Shugart Associates and eventually evolved into a series of standards developed and maintained by a series of standard bodies, including ANSI, ITIC, NCITS, and T10. Although mainly thought of as a high-throughput interface for hard drives for high-end workstations and servers, SCSI is a general interface that can be used to connect various hardware peripherals. Only a small segment of embedded systems ever use SCSI devices, though. These systems are typically high-end embedded systems such as the CompactPCI-based high-availability systems discussed earlier. In those cases, a CompactPCI SCSI controller inserted in the CompactPCI backplane provides an interface to the SCSI devices.

If you consider using SCSI in an embedded system, note that although Linux supports a wide range of SCSI controllers and devices, many prominent kernel developers believe that the kernel's SCSI code requires major work or even a complete rewrite. This doesn't mean that you shouldn't use SCSI devices with Linux. It is only a warning so that you plan your project's future updates in light of such possible modifications to the kernel's SCSI layer. At the time of this writing, work on the SCSI code has not yet started. It is expected that such work would be undertaken during the 2.5 development series of the kernel. For now, the SCSI hardware supported by Linux can be found in the Hardware Compatibility HOWTO from the LDP. As with the parallel port, there is no single reference point containing all information regarding Linux's support for SCSI, since the SCSI interface is an established technology with a very large user base.

Discussion of the kernel's SCSI device drivers architecture can be found at http://www.torque.net/sg/, at http://www.andante.org/scsi.html, and in the Linux 2.4 SCSI subsystem HOWTO from the LDP. Information regarding SCSI programming in Linux is available in the Linux SCSI Programming HOWTO from LDP. This should be the starting point for the development of any SCSI driver for Linux, along with the Linux Device Drivers book by O'Reilly. For a broad discussion about SCSI, The Book of SCSI: I/O For The Millennium by Gary Field and Peter Ridge (No Starch Press) is a good start. As with other standards, there are always official standards documents provided by the standard bodies, but again, such documentation often makes for dry reading material.

3.2.9 USB

The Universal Serial Bus (USB) was developed and is maintained by a group of companies forming the USB Implementers Forum (USB-IF). Initially developed to replace such fragmented and slow connection interfaces as the parallel and serial ports traditionally used to connect peripherals to PCs, USB has rapidly established itself as the interface of choice for peripherals by providing inexpensive ease of use and high-speed throughput. Although mainly a mainstream device-oriented bus, USB is increasingly appearing in hardware used in embedded systems, such as SBCs and SoCs from several manufacturers.

USB devices are connected in a tree-like fashion. The root is called the root hub and is usually the main board to which all USB devices and nonroot hubs are connected. The root hub is in charge of all the devices connected to it, directly or through secondary hubs. A limitation of this is that computers cannot be linked in any form of networking using direct USB cabling.^[6]

^[6] Some manufacturers actually provide some form of host-to-host link via USB, but the standard was not intended to accommodate this type of setup. There are also USB network adapters, including Ethernet adapters, that can be used to connect the computers to a common network.

Support within Linux for behaving as a USB root hub^[7] is quite mature and extensive, comparing positively to the commercial OSes that support USB. Although most hardware vendors don't ship Linux drivers with their USB peripherals, many have helped Linux developers create USB drivers by providing hardware specifications. Also, as with other hardware components, many Linux drivers have been developed in spite of their manufacturers' unwillingness to provide the relevant specifications. The main component of Linux's support for USB is provided by the USB stack in the kernel. The kernel also includes drivers for the USB devices supported by Linux. User tools are also available to manage USB devices. The user tools and the complete list of supported devices is available through the Linux USB project web site at http://www.linux-usb.org/.

^[7] Whereby Linux is responsible for all USB devices connected to it.

Support within Linux for behaving as a USB device^[8] is limited in comparison to its support for behaving as a USB root hub. While some systems running Linux, such as the iPAQ, can already behave as devices, there is no general agreed-upon framework yet for adding USB device capabilities to the Linux kernel.

^[8] Whereby Linux is just another USB device connected to a USB root hub, which may or may not be running Linux.

Development of USB drivers is covered by the Programming Guide for Linux USB Device Drivers by Detlef Fliegl, available through the Linux USB project web site. The Linux Device Drivers book also provides guidelines on how to write Linux USB drivers. There are a number of books that discuss USB, which you can find by looking at the various online bookstores. However, the consensus among developers and online book critics seems to indicate that the best place to start, as well as the best reference, is the original USB specification available online from the USB-IF.

3.2.10 IEEE1394 (FireWire)

FireWire is a trademark owned by Apple for a technology they designed in the late 80s/early 90s. They later submitted their work to the IEEE and it formed the basis of what eventually became IEEE standard 1394. Much like USB, IEEE1394 enables devices to be connected using simple and inexpensive hardware interfaces. Because of their similarities, IEEE1394 and USB are often considered together. In terms of speed, however, it is clear that IEEE1394's architecture is much more capable than USB of accommodating throughput-demanding devices, such as digital cameras and external hard drives. Recent updates to the USB standard have reduced the gap, but IEEE1394 still has a clear advantage in regards to currently existing high-throughput devices and future capabilities. Although only a small number of embedded systems actually use IEEE1394, it is likely that the need for such a technology will increase with the demand in throughput.

In contrast to USB, IEEE1394 connections do not require a root node. Rather, connections can be made either in a daisy-chain fashion or using an IEEE1394 hub. Also, unlike SCSI, connections do not need any termination. It is also possible to connect two or more computers directly using an IEEE1394, which isn't possible with USB. To take advantage of this capability, there is even an RFC specifying how to implement IP over IEEE1394. This provides an inexpensive and high-speed network connection for IEEE1394-enabled computers.

Linux's support for IEEE1394 isn't as extensive as that provided by some commercial OSes, but it is mature enough to enable the practical, every day use of quite a number of IEEE1394 hardware devices. The kernel sources contain the code required to support IEEE1394, but the most up-to-date code for the IEEE1394 subsystem and the relevant user utilities can be found at the IEEE1394 for Linux project web site at http://www.linux1394.org/. The list of supported devices can be found in the compatibility section of the web site. The number and types of devices supported by Linux's IEEE1394 can only increase in the future.

Support for running an IP network over IEEE1394 in Linux is currently in its infancy. In due time, this may become a very efficient way of debugging embedded Linux systems because of the quantity of data that can be exchanged between the host and the target.

Documentation on how to use the IEEE1394 subsystem under Linux with supported hardware can be found on the IEEE1394 for Linux project web site. The web site also includes links to documentation regarding the various specifications surrounding IEEE1394. The main standard itself is available from the IEEE and is therefore expensive for a single individual to purchase. Although the standard will be a must for any extensive work on IEEE1394, the FireWire System Architecture book by Don Anderson (Addison Wesley) is a good place to start.

3.2.11 GPIB

The General-Purpose Interface Bus (GPIB) takes its roots in HP's HP-IB bus, which was born at the end of the 1960s and is still being used in engineering and scientific applications. In the process of maturing, GPIB became the IEEE488 standard and was revised as late as 1992. Many devices that are used for data acquisition and analysis are, in fact, equipped with a GPIB interface. With the advent of mainstream hardware in this field of application, many GPIB hardware adapters have been made available for such mainstream hardware and for PCs in particular.

GPIB devices are connected together using a shielded cable that may have stackable connectors at both ends. Connectors are "stackable" in the sense that a connector on one end of a cable has the appropriate hardware interface to allow for another connector to be attached to it, which itself allows another connector to be attached. If, for instance, a cable is used to connect a computer to device A, the connector attached to A can be used to attach the connector of another cable going from A to device B.

Though the kernel itself does not contain drivers for any GPIB adapter, there is a Linux GPIB project. The project has had a troubled history, however. It started as part of the Linux Lab Project^[9] found at http://www.linux-lab.org/. After some initial development and a few releases, which are still available at ftp://ftp.llp.fu-berlin.de/LINUX-LAB/IEEE488/, development stopped. The package remained unmaintained for a number of years, until Frank Mori Hess recently restarted the project in a new location, http://linux-gpib.sourceforge.net/, and updated the package to the 2.4.x kernel series. The package currently provides kernel drivers, a user-space library compatible with National Instrument's own GPIB library, and language bindings for Perl and Python. The package supports hardware from HP, Keithley, National Instruments, and other manufacturers. The complete list of supported hardware is included in the devices.txt file found in the package's sources and on the project's web site.

^[9] The Linux Lab Project is actually much more broad than GPIB. Its aim, as stated on their site, is to provide a comprehensive set of GPL software tools for all "Linux users dealing with automation, process control, engineering and scientific stuff."

Using this package, GPIB buses are visible from user space as /dev/gpib0, /dev/gpib1, and so on. Programming the bus to interface with the attached devices involves knowing their GPIB addresses. The /etc/gpib.conf file makes it easier to configure the addresses used by the attached devices. The file must be tailored to match your configuration. The installation and the operation of the package components are documented along with the GPIB library functions in the Linux-GPIB User's Guide included with the package.

3.2.12 I²C

Initially introduced by Philips to enable communication between components inside TV sets, the Inter-Integrated Circuit (I²C) bus can be found in many embedded devices of all sizes and purposes. As with other similar small-scale buses such as SPI^[10] and MicroWire, I²C is a simple serial bus that enables the exchange of limited amounts of data among the IC components of an embedded system. There is a broad range of I²C-capable devices on the market, including LCD drivers, EEPROMs, DSPs, and so on. Because of its simplicity and its hardware requirements, I²C can be implemented either in software or in hardware.

^[10] Though there is some SPI support in Linux, it is limited to a few boards. There is, in fact, no framework providing architecture-independent SPI support.

Connecting devices using I²C requires only two wires, one with a clock signal, serial clock (SCL), and the other with the actual data, serial data (SDA). All devices on an I²C bus are connected using the same wire pair. The device initiating a transaction on the bus becomes the bus master and communicates with slaves using an addressing scheme. Although I²C supports multiple masters, most implementations have only one master.

The main kernel tree includes support for I²C, a number of devices that use I²C, and the related System Management Bus (SMBus). Due to the heavy use of I²C by hardware monitoring sensor devices, the I²C support pages are hosted on the Linux hardware monitoring project web site at http://www2.lm-sensors.nu/~lm78/. The site includes a number of links, documentation, and the most recent I²C development code. Most importantly, it contains a list of the I²C devices supported along with the appropriate driver to use for each device.

Apart from the documentation included with the kernel about I²C and the links and documentation available on the hardware sensors web site, information regarding the bus and related specification can be obtained from Philips' web site at http://www.semiconductors.philips.com/buses/i2c/. Also of interest in understanding the bus, the protocol, and its applications is the I²C FAQ maintained by Vincent Himpe, found at http://www.ping.be/~ping0751/i2cfaq/i2cfaq.htm.