Similarities between Network Intelligence Technology and a relational database system

Network Intelligence Technology aims at providing full visibility on traffic flows; it is in many aspects very similar to a relational database system which allows users to query large sets of data generated and computed by information systems. Similarities between these technologies are not limited to the technical aspect, they also follow the same business trends, but 30 years apart!

Back in the 60s, a database was specifically designed for each application because it was such a complex and cutting edge technology that only specific systems could afford it. This complexity limited the implementations to large systems including specific hardware to support the database. Databases were reserved for very large organizations.

In the 70s, the INGRES project introduced major technical enhancements such as the relational database, which made data more actionable. This project was the foundation of the first commercial products such as Sybase and Informix which enabled developers to build systems using standard database building blocks including a DBMS engine and a query language. With the availability of COTS database technology, software vendors could create applications for any type of business requirement and not only for very large organizations. SMBs could use financial systems to track their sales, the local library could use a database system to manage its book portfolio. Today, thousands of applications benefit from these database building blocks, and even very large systems such as ERPs are designed with standard COTS database products such as Oracle. Using a COTS database is more rational from a cost point of view and also reduces the time to market. Today any software vendor building an application will choose a COTS database and will not even think of redeveloping its own BD system like in the 70s.

I see the EXACT same trend with Network Intelligence Technology. The first implementations mainly focused around Deep Packet Inspections (DPI) took place in routers and in specific appliances using custom hardware. Applications where limited to very specific tasks such as P2P blocking and every vendor would develop its own “in house” network flow analysis.

But the complexity of IP networks with a growing number of applications and protocols makes it very complex to get a 360° visibility over network traffic, which confines applications to very high-end solutions and technically advanced systems usually managed by network administrators. On the other hand, solution vendors in the business of billing network usage, optimizing networks or marketing network services have a growing need to understand more in detail the behavior of network and do not always have the skills to develop in-house network intelligence. The emergence of specialized vendors of network intelligence technology, whose mission is to provide building blocks for solution vendors and network equipment providers, can be compared to the emergence of the COTS database companies. Providing ready-to-use network intelligence components which can feed applications allows any developer to use the gigantic amount of information computed by or travelling over an IP network.

Network Intelligence Technology is a fast moving market with new protocols appearing and being updated frequently. Many solution vendors and network equipment manufacturers realize that their core business is not to focus on protocol technology. Instead of developing DPI or network intelligence in house, they can now source it from a Network Intelligence Technology specialist, and benefit from deep expertise, in the same was as they would select a commercial database product :-).

Jerome

Lawful Intercept: Another Application which Requires Network Intelligence

I have worked on IP networking technology for more than 15 years, and one thing is for sure: more and more applications require detailed network intelligence. Lawful Interception (LI) is one of them. 

While standards have been well-conceived to ensure proper technical implementation and facilitate investigations by LEAs, implementing LI at a telco creates a couple of important challenges:

1. Today, most telcos implement only basic LI capabilities, based on router info (OSI layers 3-4). But this approach does not take into account all the latest trends with Internet access everywhere and new applications such as IM, social networking, MMOGs, etc. Today, LEAs must be able to make a clear connection between virtual Web identities (logins on Gmail, FaceBook, LinkedIn, msn, Amazon, Entropia, etc.) and physical locations (at home, at work, at a WiFi hotspot, at a friend’s, on the iPhone, etc.) in order to pinpoint suspects; in addition, LEAs must also intercept communication (such as IM) embedded in non-telco applications like WoW…  

2. There is an incredible amount of data generated by apps like P2P, VoD, IPTV, etc. Here, the challenge is to reduce the storage requirements and speed up investigations by focusing only on person-to-person communication within the total traffic. Again, basic solutions are not able to extract the relevant information and therefore create unmanageable situations with huge amounts of data to store and lengthy post-processing.

For LI to be effective and serve its intended purpose, I believe we need a new approach based on passive LI probes which can be either a physical device or embedded software in routers. This is what we had in mind at Qosmos when developing our latest range of ixMachine probes specially designed for Lawful Interception. Hopefully, these probes will meet the challenges of LEAs and ensure that LI remains effective, even as technology evolves.

Jerome

Don’t procrastinate: process data on the fly!

We all know that there is an explosive growth in both fixed and mobile bandwidth. But not everyone might be aware of the new challenges created by the huge volume of data flowing on the networks: storage costs and lengthy post-processing,

As with other human activity, it is much more efficient to process information in real-time than to first store it and then go back process it. A bit similar to people who make purchasing decisions on a daily basis and know how to keep within their budget limits. They avoid the hassle and time to balance their checkbooks on evenings and week-ends… and the bad surprises!

With network intelligence technology, there is no place for procrastination: it processes data on the fly and focuses on key data only (such as communications metadata), which makes it very efficient. This approach reduces the total amount information which needs to be stored (100 time less!), provides actionable information immediately AND speeds up any subsequent post-processing (since the data has been nicely indexed). This real-time, network-based approach is well-suited for applications such as VoIP fraud detection or cyber defense, which required immediate action.

So, what is the difference between using data in motion vs. stored data?

Data in motion:

• Real-time processing of network traffic   
• Minimal storage and post-processing
• Can be used to get a dynamic understanding of relationships between pieces of information (think “video”)

Stored data:

• Post-processing of data from logs
• Requires extensive storage and post-processing
• Difficult to get a dynamic understanding of relationships between pieces of information (think “photo”)

Conclusion: use network intelligence technology to process data on the fly!

Jerome

Solving Unsolvable Problems: The Solution

Solution vendors are realizing that network intelligence requires advanced expertise and that there are clear advantages of sourcing the technology from specialists. The economic downturn accelerates the movement; vendors that come out as winners are those who stay focused on their core business.

Qosmos is dedicated to network intelligence, identifying and managing the evolving and growing number of network communication protocols as well as capturing traffic metadata or content. This real-time visibility enables enhanced security features, optimized technical performance and more precise usage in third-party solutions. Delivered as software development kits and hardware that integrate seamlessly, systems integrators, ISVs and NEP’s rely on Qosmos for expert network intelligence technology while maintaining complete control over their solutions. 

Qosmos users benefit in several ways:

1. Improve development efficiency

In high-tech, business success goes hand in hand with fast time to market, which in turn depends on development time. By using proven network intelligence toolkits, development teams can stay focused on bringing new solutions fast to market, able to adhere to more predictable development roadmaps in the face of the ever-changing network-based environment. Qosmos’ Software Development Kit includes fully documented developer tools, along with support and maintenance services designed to make solutions network-intelligent rapidly and efficiently. Qosmos efficiency is built-in to the technology itself: the same application building blocks can be integrated across different CPUs, NPUs and hardware platforms. In fact, the Qosmos network intelligence development kit can be ported to any type of modern hardware.

2. Focus on core competence

Sourcing an enabling technology externally allows solution vendors to concentrate all their efforts on the customer requirements of their solutions, instead of reinventing the wheel internally. In a perfect distribution of labor, network intelligence specialists such as Qosmos put all their resources on building the best network intelligence toolkits, providing unique expertise and pre-developed building blocks that may be integrated seamlessly and rapidly to empower third-party solutions. As an illustration of our technical know-how, we can extract network information by drilling down into 16 levels of protocol encapsulation.

3. Expand solution features

Finally, the use of Qosmos for specialized network intelligence provides more than efficient use of development resources, the depth and breadth of data and metadata extraction enables additional features that strengthen and expand the capabilities of the solutions.  For example, cyber security solutions built on network intelligence technology can detect abnormal network behavior that is invisible to standard, commercial “COTS” products. In another example, network intelligence technology can be used as a front-end for Lawful Intercept to process raw traffic and efficiently dispatch only the relevant data to an existing solution, even at very high data volumes.  Finally, sourcing network intelligence technology externally gives developers fast access to new protocols which can feed their solutions and expand their reach; Qosmos delivers a new batch of protocols each quarter.

Network Intelligence plays a key role in a network-based world; governments, operators and enterprises have a vital need to gather network intelligence from IP networks for protection, monetizing and optimizing purposes. There is an increasing need to understand network-based activity for a range of solutions such as lawful interception, cyber security, market research, network optimization, billing, and more.  Without detailed network intelligence, these solutions will not continue to function adequately and their vendors will face serious business issues.

Many of the winners will be companies who chose to boost their solutions with expert Network Intelligence Technology – from Qosmos.

Jerome

Solving Unsolvable Problems (Part 4): Managing highly complex technology

Since network intelligence goes beyond DPI in terms of the level of visibility, protocol management, attribute recognition, and information extraction, even companies who have incorporated DPI capabilities into their solutions will require a new level of expertise. For example, Qosmos engineers have developed a specific meta-programming language to build Webmail and HTTP protocol plug-ins. Specific techniques and tools must be developed for quality assurance and to make reverse engineering more efficient.  This is an order of magnitude remote from the business of someone who sells complete solutions. 

In most cases, a separate R&D organization must be created. But once committed, companies realize that development times are difficult to estimate, timelines are incompressible and that the skills are so specialized that it becomes nearly impossible to outsource parts of the development.

Jerome

Solving Unsolvable Problems (Part 3): Committing considerable resources, with uncertain returns

If you choose to tackle a family of protocols (e.g. Webmails), you have to develop network intelligence capabilities for the most important protocols in this family, otherwise your solution will ineffective or incomplete (think of traffic optimization or cyber security applications). In addition, the total number of applications and protocols increase continuously (e.g. 50,000 applications are now available for the iPhone), and very few protocols ever disappear…

For a company whose core business is not network intelligence technology, this translates into high costs of entry and ever-rising investments. To make things worse, end customers of turn-key solutions may not appreciate the importance of continuous protocol updates and the amount of work required to keep the solutions current. This means that a solution vendor could end up investing considerable resources for which and end customer does not perceive high value and therefore may not be ready to pay… 

Jerome

Solving Unsolvable Problems (Part 2)

As I described in my previous post, I believe that solution vendors are facing a crucial decision point and are realizing that developing network intelligence capability internally would create a number of “unsolvable problems”.

Unsolvable problem number 2: Not being able to use traditional product development and management methods

The high-tech industry typically uses a structured approach for product development and management, with most key activities aligned around go/no go decision points and defined time lines. These processes are built to ensure that new products are delivered on time, according to specifications and with the adequate quality.

However, companies who chose to develop network intelligence technology internally quickly discover that the usual methods cannot be used. Web protocols such as Webmails change continuously without notice, which means that development roadmaps cannot be easily controlled. Development teams must be quick to react to new protocol evolutions and use reverse engineering techniques to update their network intelligence software.

This way of working is counter-cultural for many high-tech companies and can even be incompatible with the rest of the organization. From a business standpoint, it can even be unsustainable and unprofitable - unless you make network intelligence technology your core business, like Qosmos;-)

Jerome