The need for dedicated deep packet inspection (DPI) equipment has arisen out of the constant struggle between network managers and customers who use more than their fair (or authorized) share of network resources.
Some network managers have started blocking access to services and websites they felt users were not entitled to access (often including social sites or peer-to-peer (P2P) applications accessed via corporate networks, for example). Then the authors of these applications started using all sorts of obfuscating tricks and even payload encryption to get around the filters. The only (somewhat) reliable way to identify these stealthy applications, therefore, is through actual traffic flow analysis (or a thorough inspection of the first packets in the traffic flow). As the analyzing equipment has to look beyond the TCP headers, the technique is called deep packet inspection.
The past controversial uses of deep packet inspection to block P2P applications gave the technology a bad name, and its use might also be legally questionable (or even forbidden) under some regulations or laws. I believe the partisans of deep packet inspection banning did great damage to the majority of Internet users who never tried to squeeze the last drop out of the network. Now, even when they could benefit from deep packet inspection, they're not allowed to use it.
Just take a look at what Comcast had to do. After its initial DPI debacle, it chose to implement a much more fair scheme that does not use deep packet inspection. Whenever the network becomes congested, it limits heavy users (those who exceed 70% of their provisioned bandwidth within a 15-minute interval). Those users experience degradation of all services during network congestion, however, including any third-party real-time services they might be using (Skype, for example). As typical households have more than one workstation connected to a cable modem, kids downloading movies could put subscribers into the "heavy users" category and affect third-party voice services.
Carriers might decide to stay on the safe side, avoid the whole deep packet inspection debate and group users exclusively on the amount of traffic they generate. But if deep packet inspection is not banned in your jurisdiction, your competitors may surprise you by using it to their advantage. They may decide to identify P2P traffic and make it low priority, excluding it from traffic caps and thus endearing themselves to anyone using P2P applications. You probably can't afford that, so you should investigate the deep packet inspection technology, try to see how your services can benefit from it, and start equipment evaluation and pilot projects as soon as possible.
Back to part 1: When carriers need dedicated boxes for effective network traffic management.
About the author: Ivan Pepelnjak, CCIE No. 1354, is a 25-year veteran of the networking industry. He has more than 10 years of experience in designing, installing, troubleshooting and operating large service provider and enterprise WAN and LAN networks and is currently chief technology advisor at NIL Data Communications, focusing on advanced IP-based networks and Web technologies. His books include MPLS and VPN Architectures and EIGRP Network Design.Check out his IOS Hints blog.