Rise of Zero-Trust and SASE Shines New Spotlight on Deep Packet Inspection (DPI)
Guest blog post by Roy Chua, Founder and Principal of AvidThink
PART 1 OF 2
The security landscape continues to evolve. Driven by ongoing innovation in attack methods — from new payload delivery mechanisms for criminal ransomware to sophisticated phishing with personalized zero-day exploits designed by nation-states — public and private sector organizations struggle with their defenses. Meanwhile, insider compromise and exfiltration of intellectual property continue to rear their ugly head.
To mitigate risks, organizations are turning to a new breed of security solutions, including secure access services edge (SASE) and zero-trust network access (ZTNA) at the enterprise edge, and extended threat detection and response (XDR) for endpoints and networks. And at the heart of many of these new solutions is a trusted technology we all know — the deep packet inspection (DPI) engine — but evolved to meet unique needs.
At the heart of new security solutions is a trusted technology we all know — the deep packet inspection (DPI) engine — but evolved to meet unique needs.
Old DPI, Meet New DPI
For those of us who’ve been around the security block more than once, we remember the advent of DPI. Early inspection of network headers started with hardware network analyzers such as HP’s Protocol Analyzer and the Network General’s Sniffer (available as software later) in the mid-80s, used for debugging and troubleshooting networks. This was followed by open-source innovations for packet capture in tcpdump and libpcap, Vern Paxon’s Bro, a network security monitor (which became Zeek), Ethereal (renamed Wireshark) software network analyzer, and Martin Roesch’s open-source Snort intrusion detection system (IDS). Many of the ideas and software in these open-source systems were used by network security vendors to look beyond the typical IP 5-tuple (source and destination IP addresses, source and destination ports, and protocol type) for embedded threats.
While there are slightly varying definitions of DPI, most agree that DPI involves looking beyond layer 2/3/4 headers (IP/TCP/UDP/QUIC etc.), keeping limited state on a network flow, and decrypting the payload if necessary. For our post, we differentiate DPI from proxy-based approaches, which terminate and reissue a layer-7 application-aware session, inspecting data within the context of an application (e.g., HTTPS for web, SMTP for email); confusingly, though, DPI is sometimes used as a general term for any type of deep inspection by networking devices including application-layer proxies.
Regardless, DPI capabilities are critical to commercial intrusion detection and prevention systems (IDS/IPS), server-load balancers (SLBs)/application delivery controllers (ADCs), next-generation firewalls (NGFWs), and unified threat management (UTM) appliances. Early security vendors would adopt and modify open-source libraries or develop proprietary DPI methods to provide insight into each network packet traversing their devices. Vendors would tout their speed, efficiency, coverage, and accuracy. Over time, several pure-play DPI technology experts emerged. The benefits of specialization and scale economies drove security companies to license third-party DPI engines instead of continuing to invest in their own.
All essential functions of SSE and SASE require the ability to rapidly parse, identify, and classify network traffic flows.
DPI technology remains critical to security today, even with the migration to a new generation of enterprise security solutions helmed by SASE, ZTNA, and XDR. Security Service Edge (SSE), a key component of SASE, combines multiple network edge security functions. SSE capabilities include securing web traffic via a Secure Web Gateway (SWG), gating and policing access to cloud-based applications via a Cloud Access Security Broker (CASB) and protecting private applications and resources via ZTNA. All these essential functions and other elements of SSE and SASE require the ability to rapidly parse, identify, and classify network traffic flows. In addition, they depend on the ability to generate pertinent metadata around the identified application and session and possibly even perform payload inspection.
Some examples of how DPI features in each class of solution include:
- SWG — DPI is key to identifying the applications and the type of action (e.g., content upload, download) performed in any application to bring it under SWG protection and control.
- CASB — DPI can be used to discover shadow IT apps that should be brought under enterprise control. Similarly, DPI metadata can be used to build user behavior profiles to detect anomalous behavior.
- ZTNA — DPI is used to identify private applications that need to be controlled, as well as real-time signals on anomalous and malicious behavior.
- XDR — DPI can detect and control access for local protocols like NETBIOS/NETBIOS Naming Service and Samba (SMB/CIFS) for file access and sharing. Likewise, DPI can be used for understanding and de-encapsulating multiple layers of tunnels for endpoint traffic for deeper analysis.
The above is a small sample, and DPI features in many other security functions. For example, Data Leak Prevention (DLP) migrated from a standalone function to a feature in CASB, ZTNA, SWG and other edge security solutions requires looking deep into the application data. More comprehensive DPI libraries will tend to support DLP, often tied with a TLS-decryption module.
Even as DPI becomes essential to security functions, the landscape has evolved. Enterprise traffic is now at least two orders of magnitude higher than when DPI solutions first came on the scene. And with pervasive end-to-end encryption and TLS adoption, most traffic is encrypted. In addition, the number of distinct applications has exploded, and security inspection is migrating from enterprise premises to the cloud edge. These collective trends place additional demands on the new generation of DPI solutions.
Whether vendors choose to continue developing their in-house solutions or license third-party libraries, today’s DPI engines need to fulfill the following capabilities:
- Encryption handling — even when the data remains encrypted, next-gen DPI must provide accurate categorization. Not all deployments have the luxury of enabling TLS decryption through installing a cert on endpoints. Earlier SSL/TLS versions exposed information during the handshake that classifiers could use. Still, with the increasing adoption of TLS 1.3, network equipment that cannot act as a full TLS proxy has limited visibility even during the TLS handshake process. Using traffic patterns, DNS, and other techniques coupled with AI/ML, industry-leading DPI engines today can achieve application categorization accuracy rates of over 95% with encrypted traffic.
- Ongoing building of extensive application libraries — the number of applications continues to expand rapidly. Software-as-a-service provides low barriers to entry and offers continued innovations; the next Figma, Canva, Box, SalesForce, and Xero, are around the corner, and the DPI library needs to keep up. These applications’ traffic patterns, ports, and IP destinations must be updated.
- Convergence of operational technologies (OT) and IT — historically, OT and IT technologies were separated, with different physical infrastructures and OT using proprietary and unique protocols. However, organizations are converging the two infrastructures over standard ethernet-based wired and wireless connections with IP as the standard layer-3 protocol. In parallel, threats to OT infrastructure from attackers have increased, and security solutions require DPI technologies to identify, parse and categorize OT protocols like PROFINET, PROFIBUS, DeviceNet, ControlNet, and ModBus, or IoT protocols like MQTT.
- Evolution of AI/ML engine — as traffic patterns change and applications evolve, the trained AI/ML model needs to be updated to maintain high accuracy, and the engine needs to be continually trained to prevent drift.
- Ongoing update of evasion techniques — attackers continue to develop new evasion techniques to prevent DPI engines from detecting them. The DPI engine needs to adapt as new attacks are discovered by researchers or in the wild.
- Ongoing catalog update of VPNs and other anonymizing services — beyond application types, DPI engines often provide updated libraries of IPs for VPNs and anonymizing services, which allow DPI engines to report to the security applications whether a stream is running through a VPN, allowing the security application to determine how best to handle. For example, enterprise security departments might want to block applications accessed via unsafe anonymizers, suspect VPN services, or prevent access via Tor.
To read PART 2, click here.
