Snapshot: Transition of Line Card of High-End Router

Here I would like to give a snapshot of development, transition and trends of high-end router ecosystem lightly. I will not go in detail on the product itself as it will produce a very lengthy and too technical document.

MSC-40G

When Cisco Systems released their ultimate carrier-class high-end router product, which was CRS-1 router, it came with one choice of line card (forwarding), which is MSC-40G. We know that in distributed router architecture, line card can be viewed as ‘autonomous system’ in terms of forwarding since they are the one who do the job mostly. They act as the real building block of the system and dictate the system real capacity. It is not uncommon that the price of line card is more expensive than the chassis itself.

The MSC-40G came with high forwarding capability. The key numbers are: 40G line rate forwarding capacity, 256 K queuing and 65 Mpps processing capabilities.  These figures allow transmission of 64B packet size line-rate @40G with many services turned-on. It also allows terminating thousands of subscribers each with multiple QoS queues.  

The early and subsequent adopters of Cisco CRS-1 are major Telco player and big ISP. The Telcos were experiencing the 3GPP R4 implementation (Nb over IP, split arch, etc) that requires the processing of relatively small IP packet with stringent High Availability requirements, while the ISPs  were looking for fulfillment of tremendous bandwidth demand. CRS-1 were the ultimate choices of the customer when they went for Cisco solution.

FP-40

Four years later, Cisco came with newer line card for CRS-1 families, which is FP-40 (Forwarding Processor-40). This line card comes with smaller processing capabilities compared to MSC-40. It has only 8 queue per port and 45 Mpps processing speed. With this number we will not be able to do 64B packet size @40G anymore like on MSC. There are other limitations on number of interfaces supported as well as licenses required to enable certain level of features and scale. And of course, the price justifies this decrease because it is only roughly 20% to 25% of equivalent MSC-40 counterpart for delivering 40G capacity per line card.

Original purpose of FP-40 was for internet peering and thin or light MPLS core and edge. We know that for Internet router, we don’t need many queues and also the internet traffic in average has bigger size (300 B and above where 800 Bytes is common). QoS implementation in Internet router is somewhat limited and in many occasion the big trunk is just only configured without any sub interface and no QoS at all. With much lower price tag, this line card justifies better to suite more competitive ISP environment.

The same things applicable for what Cisco refers to ‘thin core’. This supposed to be an IP/MPLS core network with fewer requirements on number of interfaces, queues and line rate capabilities. While the network can span big, but the requirements imposed are relatively small the network can be viewed as ‘thin core’.

Thus, with introduction of FP-40 Cisco improves its product competitiveness on Telco and Service Provider market and can match customer requirements better. Customer now also has better Capex/Opex when they shop this type of router. It is undeniable that many implementation of CRS-1 with MSC-40 line card, the utilization is low and the scalability ceiling is never or far away to be hit by network demand. Thus, waste of resources is avoided by putting the FP-40 instead of MSC-40 on places within network wherever applicable. The key here is wise or properly matching real customer requirements and improving Capex and Opex efficiency better.

FP-140 and MSC-140

Later, during the battle of 100G race, Cisco released newer product on CRS family that is called CRS-3 in 2010. This newer product has 3.5 bigger capacities per slot than its predecessor and has a lot of improvements, both in control plane and forwarding including power as well.

The new line card is called FP-140 and MSC-140. As the names implies, the cards have 140G bandwidth capacity. The notable difference is that Cisco keep 8 queues per port on FP-140 and 256 K queues on MSC-40 plus several license based features activation.  The price tag follows this parity where FP-140 is much cheaper than MSC-140.

One of the key things here related to our previous discussion is the changes in IP/MPLS design ‘theme’. The MSC families are now pushed to the edge and the remaining portion of the network can rely on FP line card. This means that, only high-density customer facing will use MSC, while any P-facing interfaces on PE, and all P interfaces can use FP (in most case). This is because even if we have many customer or CE connected on the edge (many sub interfaces) with multiple QoS on each of it (that can call for thousands of individual queues, thus requires MSC), the uplink will rely on much smaller queues requirement. For example, if the SP or Telco define 4 queues per sub interface or subscriber (best effort, data, control and voice/video) and it has let’s say 1000 customers (thus totaling 4000 queue required on the edge facing card), but when the packet is transport to the core it will be aggregated to only 4 queues. Thus, we can safely introduce FP based line card in the rest of the network including P router other than CE facing line card.

This redefinition of design approach gives more competitive network infrastructure for SP and Telcos. This combination improves Telco and SP spending better than before since they can wisely chose better and proper use of the FP and MSC combination of line card without easily overrun by future requirements. Some points of the network can be MSC based, FP-only based or proper mix among them.

Future Look

With tremendous growth of data, voice and video, the bigger capacity pipe is required on the network. The pressure for Service Providers and Telco to monetize their investment with lower cost of spending must be translated by any networking vendor with more innovative product. The proliferation of coming LTE and mobile broadband will determine the shape of high end networking product. The limit of higher-end line card (256 K queues’ and capabilities of 64 B line rate, etc) seems still high enough, then the battle to produce much lighter line card with lower cost will continue faster than opposite direction.

The Merging of the Two Worlds : Telco and ISP convergence

This is another stories on router line card again...

In the past, there are different approach used to design network for pure ISP and for Telco MPLS network. The ISP here means either pure Internet Service Provider or internet  part of big Telco network. While the Telco MPLS network means MPLS infrastructure for supporting GSM or CDMA operation.

The ISP usually requires less requirements compared to the Telco counterpart in terms of redundancy, packet processing capabilities, as well as pricing.
In terms of redundancy, for example, the ISP can tolerate two router in mated-pair redundant Routing Engine or Supervisor Engine. This means they just rely on box redundancy should one router fails, where router A can fail-over to Router B in case of failure or maintenance. Meanwhile, the Telco will push for both router (box) redundancy as well as Routing Engine / Supervisor Engine redundancy on each router. Thus, should one router engine fail, they can fail-over to its redundant Supervisor before going to fail-over to mated-pair box.

In terms of packet processing capability, the internet router (the ISP one), can sustain line rate processing at higher packet size (300B, 500B, 800B, etc) while the Telco tends to make it line rate at smaller packet size (64 B for example). This is due to average internet traffic size is bigger than average voice traffic.
Thus, when Telco start moving their GSM/CDMA traffic over IP, they tends to make perfect for everything with highest capability as possible.


The only part which ISP router demands more than Telco MPLS router may be on routing table size. Definitely ISP router needs to support full Internet routing tables while the average telco with 40M customers for example may only need to support 5000 entires of IGP in their routing table as requirement.

The other side to look at it is from financial or economic perspective. The Telco are CAPEX intensive environment. A moderate Telco with 50 Million subscriber can spend  more than one billion USD per year for all their telco needs (cell site, land acquisition, L1 infrastructure, radio, PS Core, CS core, etc). Out of that 1 Billion USD, may be less than 5 percents are allocated to IP part (the routers and switches). Thus, the impact of saving on redundancy and line card are not really significant and the money is there basically. Meanwhile, ISP business are quite budget sensitive, where the cost are much smaller as well the revenue and does not have the luxury of that Telco counterpart. Their cost is easier to be mapped, for example for router, link and access-distribution. Thus, saving in router will impact their P&L more than the Telco part.

ISP is pure IP since the beginning, while the Telco were TDM minded from the start. Thus, when the Telco were moving to IP, a lot of questions were raised whether the IP can provide TDM level of robustness and high availability. It is undeniable that a lot of resistance happened against the migration to IP, which pushed the Telco IP implementation to be as solid and as robust as possible to match the quality requirements.


The booming of Data and Video (non voice)

The proliferation of mobile broadband world-wide the push the data traffic to increase exponentially. This is fueled by many factor such as social networks, handset capacity, 3G services, etc. This phenomena leaves Telco in difficult situation s:
- the data (non voice) traffic increased significantly that requires more bandwidth and associated infrastructure
- but, the revenue from the data does not commensurate the traffic
While the Telco still rely on voice traffic as the major part of their revenue components, the Telco operator has no choice to the data part. Despite data revenue typically are still small, they have to cope with it like it or not. Thus, they are forced to optimize the investment  on infrastructure to support data and at the same time maximize the effort to monetize the data services.


The optimization of the data services now force the Telco to leave the luxurious router specification they used in the past. They now become as budgetary constraint as ISP when dealing with router investment. The correct responds from the vendor now is to create more affordable line card or model that will fit with this new requirements. The line card scale limit is pushed down and the price is becoming much more affordable now.

There are many approach by the telco to accommodate this  growth. One of the approach is to put investment on the growing part which is the data. In this way, the Telco split the network to accommodate the data and the network to accommodate the voice. While the data part comes with less stringent requirements, they can safely choose less powerful line card for this purpose.

In summary, there are transition happens inside the Telco networks to match their new growth and data services. This transition as if merge the ISP and Telco design in terms of line card requirements. We are not talking about the LTE yet, still 2G/3G networks but still this part design transition should be addressed properly to improve overall service offering to the customer.