There are usually several ways to configure any given piece of network equipment to achieve the same basic result. These different configurations usually do not use resources in the same way, and they often do not have exactly the same performance characteristics. For example, an inefficient configuration of a router might mean that it has to do too much processing on each packet. This extra processing increases the amount of time that each packet spends inside the router and probably also increases the memory utilization of the router, as it has to buffer large numbers of packets.
A typical example of this increase happens when an engineer fails to use special features of the equipment. For example, in many routers the ability to make most common routing decisions is delegated to logic circuits supporting the interface card. This approach works well because it means that the CPU is free to coordinate the activities of the different cards. The result is vastly improved net throughput; however, the advantage can be completely lost if this engineer implements a CPU-bound process that examines the contents of every packet.
For example, the engineer might turn on a feature that prioritizes or routes packets based on their contents. This sort of feature usually requires the CPU to examine the packet. So the packet cannot be processed solely by the interface. The same CPU loading happens if the router is configured to rewrite the contents of the packets, as in an address-translation feature systematically.
This has several implications for network design. Locally, it is important to ensure that each device does only what it needs to do and that it does so by the most efficient method. Globally, it means that network designers have to be careful about what network functions are performed where.
The local issue is really a matter for the network engineer who should sit down with the manuals for the network hardware and find the most efficient way to implement the functions of this device. If special optimizations are available, such as Cisco's Fast Switching, then they should be used. Discussing the implementation details with the hardware vendor may also help, since the vendor should be aware of the features of the equipment.
The global issue, however, is for the network designer to resolve. A good example of this resolution comes up in prioritization and Quality of Service (QoS) implementation. The best place to decide on the priority of a packet is at the point where that packet enters the network. The entry-point router should examine the packet and mark its header with the appropriate priority value. Then, each subsequent device that handles the packet reads this priority value and treats the packet appropriately. The worst thing a designer can do is make every router look at the packet in detail and decide again what its priority should be.
Looking at a single byte in the header is easy for a router and can frequently be handled in highly optimized code in the hardware. However, looking at several bytes and making a decision about each packet takes a huge amount of extra resources. In most cases, it also increases the forwarding latency and reduces the number of packets that the router can handle.