Network Intelligence: Coming Back to Qosmos

It is interesting to see the increasing interest for embedding network intelligence software into solutions. At Qosmos, we have been speaking to network equipment suppliers, ISVs and systems integrators for years. Which means that many of them have known us for years. However, during the initial discussions, many of them tell us that they have DPI skills internally, and while Qosmos technology is really impressive, they don’t need to source externally. “No problem”, we say, “but don’t hesitate to contact Qosmos if you change your mind”.
We now see more and more companies coming back to Qosmos.
Why do they come back to us? The reasons are simple:

• They find it increasing difficult to keep up with ever-changing protocols and applications
• They face challenges in scaling existing solution to network speeds beyond Gbps
• Resource constraints force them to focus all their energy on their core business (which is typically to build solutions, not enabling technology like DPI or Network Intelligence)

This is typical for new high-tech markets: initially, high-tech vendors will build everything in-house, because 1) it’s not too difficult and 2) there are no external suppliers. Think of databases: initially, all IT vendors built their own databases in-house (for example IBM DB2). Then vendors moved to source database technology from specialists like Sybase, Informix or Oracle. Same thing for micro-processors, which was initially developed internally by computer vendors, but is now sourced from specialists Intel and AMD.
There is now a similar trend with DPI and network intelligence technology: the market is shaping up for the benefit of everyone.
Welcome back : we are happy to work with you!

JT

Don’t procrastinate: process data on the fly!

We all know that there is an explosive growth in both fixed and mobile bandwidth. But not everyone might be aware of the new challenges created by the huge volume of data flowing on the networks: storage costs and lengthy post-processing,

As with other human activity, it is much more efficient to process information in real-time than to first store it and then go back process it. A bit similar to people who make purchasing decisions on a daily basis and know how to keep within their budget limits. They avoid the hassle and time to balance their checkbooks on evenings and week-ends… and the bad surprises!

With network intelligence technology, there is no place for procrastination: it processes data on the fly and focuses on key data only (such as communications metadata), which makes it very efficient. This approach reduces the total amount information which needs to be stored (100 time less!), provides actionable information immediately AND speeds up any subsequent post-processing (since the data has been nicely indexed). This real-time, network-based approach is well-suited for applications such as VoIP fraud detection or cyber defense, which required immediate action.

So, what is the difference between using data in motion vs. stored data?

Data in motion:

• Real-time processing of network traffic   
• Minimal storage and post-processing
• Can be used to get a dynamic understanding of relationships between pieces of information (think “video”)

Stored data:

• Post-processing of data from logs
• Requires extensive storage and post-processing
• Difficult to get a dynamic understanding of relationships between pieces of information (think “photo”)

Conclusion: use network intelligence technology to process data on the fly!

Jerome

Solving Unsolvable Problems (Part 4): Managing highly complex technology

Since network intelligence goes beyond DPI in terms of the level of visibility, protocol management, attribute recognition, and information extraction, even companies who have incorporated DPI capabilities into their solutions will require a new level of expertise. For example, Qosmos engineers have developed a specific meta-programming language to build Webmail and HTTP protocol plug-ins. Specific techniques and tools must be developed for quality assurance and to make reverse engineering more efficient.  This is an order of magnitude remote from the business of someone who sells complete solutions. 

In most cases, a separate R&D organization must be created. But once committed, companies realize that development times are difficult to estimate, timelines are incompressible and that the skills are so specialized that it becomes nearly impossible to outsource parts of the development.

Jerome

Solving Unsolvable Problems (Part 3): Committing considerable resources, with uncertain returns

If you choose to tackle a family of protocols (e.g. Webmails), you have to develop network intelligence capabilities for the most important protocols in this family, otherwise your solution will ineffective or incomplete (think of traffic optimization or cyber security applications). In addition, the total number of applications and protocols increase continuously (e.g. 50,000 applications are now available for the iPhone), and very few protocols ever disappear…

For a company whose core business is not network intelligence technology, this translates into high costs of entry and ever-rising investments. To make things worse, end customers of turn-key solutions may not appreciate the importance of continuous protocol updates and the amount of work required to keep the solutions current. This means that a solution vendor could end up investing considerable resources for which and end customer does not perceive high value and therefore may not be ready to pay… 

Jerome

Solving Unsolvable Problems (Part 2)

As I described in my previous post, I believe that solution vendors are facing a crucial decision point and are realizing that developing network intelligence capability internally would create a number of “unsolvable problems”.

Unsolvable problem number 2: Not being able to use traditional product development and management methods

The high-tech industry typically uses a structured approach for product development and management, with most key activities aligned around go/no go decision points and defined time lines. These processes are built to ensure that new products are delivered on time, according to specifications and with the adequate quality.

However, companies who chose to develop network intelligence technology internally quickly discover that the usual methods cannot be used. Web protocols such as Webmails change continuously without notice, which means that development roadmaps cannot be easily controlled. Development teams must be quick to react to new protocol evolutions and use reverse engineering techniques to update their network intelligence software.

This way of working is counter-cultural for many high-tech companies and can even be incompatible with the rest of the organization. From a business standpoint, it can even be unsustainable and unprofitable - unless you make network intelligence technology your core business, like Qosmos;-)

Jerome

Solving Unsolvable Problems (Part 1)

For the past several years, Deep Packet Inspection (DPI) has been used by developers to get basic traffic visibility for applications such as traffic shaping and network security.

We are now at a key juncture of the market, where many solution vendors need much more detailed visibility into network-based activity. For example, for market research applications it is not enough to recognize generic http traffic; it is necessary to extract metadata such as name of visited Website, page content, time spent on visit, basket share, referent, etc. The same is true for applications such as lawful intercept and government cyber security, which demand extremely accurate and fine-grained information on communication flows in order to map exact communication flows and identify threat patterns.

So the key question becomes: - should solution vendors develop complex network intelligence technology internally or should they source it from a specialist?

Solution vendors realize that developing network intelligence capability internally necessitates a step-function in terms of investment and strategy. It requires considerable internal R&D resources, detracts from core business and poses a number of additional problems in term of organizational efficiency, culture, and return on investment.

During the next couple of weeks, I will describe the “unsolvable problems” facing solution vendors and highlight the advantages of sourcing complex network intelligence technology from a specialist like Qosmos.

The first “Unsolvable Problem”: Having to continuously redevelop software

Developing network intelligence software to handle a particular protocol is only the tip of the iceberg: you also need to invest continuously to update your software so it can handle new versions of protocols. Some basic protocols are easy to manage since they are relatively stable: IMAP, SMTP, POP, HTTP, etc. However, there is a vast number of other protocols which are proprietary and evolve at a rapid pace (Webmails, P2P, social networking, gaming, etc.).

For example, the Livemail protocol changed several times during 2008. And each time a new version is released, most of the network intelligence software has to be redeveloped, without much reuse of previous development. This requires a particular culture and special methods: reverse engineering, custom-made tools, fast reaction to protocols changes, etc.

Jerome

BBWF09 - DPI Workshop / Monday, 07 September 10:00 – 12:30

Just a little heads-up to tell you that I will be a speaker at the upcoming Broadband World Forum which will be help in Paris 7-9 September. Il will participate in the workshop “Deep Packet Inspection: Technology, Promise & Controversy. What You Need to Know” on Monday, 07 September 10:00 – 12:30. It should be an interesting discussion, with views from solution suppliers, service providers and Network Intelligence Technology specialists (Qosmos).

For those of you who would like to see Qosmos Network Intelligence Technology in action, you are welcome to see a live demo on the exhibition floor: just come to our partner BreakingPoint’s Stand # 470.

I hope to see you there!

Jerome

Forbes article on Programming Innovation

Dan Woods just published an interesting article on Forbes.com called “Programming Innovation”. He describes how Qosmos technology enables a new kind of market research for our partner GfK.

Market research is of course only one example of how Qosmos Network Intelligence can be used. I like Dan’s way of describing our toolkit as a catalyst for innovation across a wide range of applications: “Now, tens of thousands of companies potentially have access to a new form of information derived from the network”.

Based on my daily discussions with customers and prospects, I can confirm that the trend is real – I can’t wait to see tens of thousands of companies benefiting from Qosmos Network Intelligence ;-)

Jerome

COTS not enough for government cyber security

I just read this interesting article by Nextgov, confirming that relying only on commercial products can increase vulnerabilities to cyber attacks: http://www.nextgov.com/nextgov/ng_20090618_3505.php. This is in line with my recent discussions with governments who use Qosmos technology to enhance their cyber security beyond COTS capabilities. Not only could COTS products contain hidden code; they often lack capabilities required by government security specialists for detecting certain threat patterns and/or they lack the flexibility to customize according to special government security policies.

Jerome

New Intelligence. A Smarter World

IBM’s Smarter Planet website “New Intelligence. Smarter thinking about how we collect, analyze and use information.” has the merit of placing network intelligence at the heart of a multitude of activities we would not automatically associate with developments in the cybersphere.

Instrumented. Interconnected. Intelligent
We would all agree that the information explosion has permanently changed the way we experience the world: everything – and everyone – is leaving digital tracks. For IBM this revolution implies three-fold change: moving from the "World Wide Web" to a "Web Wide World" means becoming instrumented, interconnected and intelligent. Instrumented , because "sensors” of all sorts are embedded everywhere and in all areas of activity: cars, appliances, cameras, roads, pipelines, medical instruments, livestock,etc.) and interconnected via internet. … But as IBM recognizes, in the information age being instrumented and interconnected isn’t enough. It's got to be intelligent.

“With computational power in things we wouldn't recognise as computers, any person, object, process, service, or organisation can become digitally aware, connected and smart.”
(IBM's ”A Smarter Planet” web site: http://asmarterplanet.com/)

It is not only computational power, however, that makes for an intelligent world. But what makes aninstrumented and interconnected world an intelligent one?

For IBM: “People, machines, processes… are increasingly connected to systems that process data, and benefit from advanced analytics capable of turning data into real insight, in real time.”

This is the criticalpoint where information can be either unweildy volumes of unmanageable and unusable data, or truly valuable information that is instantly available and ready to use.

IBM's initiativesto promote a Web Wide World that is instrumented, interconnected and intelligent, are exactly in phase with Qosmos' vision for Network Intelligence. Put most simply, it's what connects connectedness to uses, possibilities and needs. Asthe IBM website underlines: “it can be a daunting task for any enterprise to sift through massive amounts of data, extract information and transform it into actionable knowledge.”

In my next blog, I'll be looking some more into IBM's vision for a Smarter Planet, and how Qosmos network intelligence can contribute to it.

Visit the qosmos web site at: www.qosmos.com
IBM's ”A Smarter Planet” web site: http://asmarterplanet.com/

Jerome

The Convergence of Network Intelligence and Business Intelligence

Believe it or not, I am not the only one to blog on the topic of Network Intelligence (NI). Several analysts at Yankee Group have posted entries on this topic: see http://blogs.yankeegroup.com/category/network-intelligence. As a matter of fact, Yankee have been pioneering the concept of Network Intelligence in the telco space and I have had some good discussions with them. They just published an interesting report entitled “The Convergence of Network Intelligence and Business Intelligence”, with Jon Paisner as the main author, and co-authors David Vorhaus and Sheryl Kingstone. The document describes the similarities and synergies between Network Intelligence and Business Intelligence; I highly recommend it to all those interested in Network Intelligence technology as an enabler of smart pipe solutions. It also confirms the key role played by Network Intelligence-gathering technologies such as deep packet inspection and its various evolutions for protection, optimizing and monetizing network information.

Jerome

Breaking the high speed barrier

I have recently had several interesting discussions with solution vendors who are faced with the serious challenge of making their systems work at multi-Gbps line rates. To break this “high-speed barrier”, they can either 1) spend considerable time and resources redeveloping their products to work at higher speeds, or 2) optimize their solutions with the help of Network Intelligence.

The challenge is especially acute for vendors in the mobile data environment, where bandwidth is growing at an exponential rate. Vendors who sell (for example) mobile service quality monitoring need to continue offering the same level of functionality at speeds in excess of 1 Gbps. However, it is not as easy to do well at high speeds what they used to do well at lower speeds… Of course, some vendors may be tempted to reduce functionality in order to handle the increased bandwidth, but this is not an option : it is clearly not an acceptable solution for mobile operators and their clients.

So how can Network Intelligence accelerate existing mobile solutions?

As we all know, mobile data traffic is literally exploding, due to the combination of new iPhone-type handsets, new broadband infrastructure (3G, HSDPA, HSUPA) and attractive pricing schemes. At the same time, video streaming, P2P, and social networking applications consume a large percentage of this traffic, pushing bandwidth at the core of mobile networks to multi-Gbps. This creates a need to generate useful data at workable bandwidths, and only retain essential information (e.g. IP metadata or IPDRs) for lower priority services (P2P, etc.).

Example 1: TroubleShooting

In order for customer service to respond to customer complaints, mobile operators often need to check service quality for certain subscribers. In this case, a Qosmos probe can filter selected IMSIs and forward only relevant traffic to an existing troubleshooting solution, at manageable speeds. No change is necessary to the existing solution.

Example 2: Subscriber Knowledge

Mobile operators carry massive amounts of data traffic from services outside their walled-gardens. This traffic can represent more than 90% of total traffic (see this Webcast for a concrete example). To optimize networks and service offerings, MNOs require better network intelligence to answer questions like: what mobile devices are being used (PC/handheld)? Which applications drive data growth? What are user the behavior patterns? All these questions can be answered with Qosmos Network Intelligence technology.

Example 3: Lawful Intercept

Beyond mobile solutions, similar challenges exist in the area of Lawful Intercept, where many intercept probes have not been designed to handle very high bandwidth. Here again, Network Intelligence technology saves the day by processing the high-bandwidth raw traffic and carrying out optimized dispatching of information to an existing lawful intercept system that can continue to process efficiently.

In summary, Network Intelligence technology can be used to enable existing solutions to continue performing efficiently even at very high speeds! Even though total bandwidth exceeds several Gbps, the traffic selected for forwarding can be handled easily by existing applications. The benefit for solution vendors: by adding a layer of Network Intelligence, they can avoid costly development and continue to sell existing solutions even as bandwidth grows exponentially!

I should also point out that our approach is different from “load balancing” provided by some suppliers, since Network Intelligence implements smart traffic filtering based on deeply embedded criteria (e.g. IMSI, MIME type, email sender, etc.), and can generate CDRs even for discarded traffic.

Jerome

The Power of Metadata

Metadata is information about communication: who communicates with whom, how, when, and where. For example VoIP caller/called party, Email / Webmail sender / receiver / subject, IM contact list / status / sender / receiver, route update in routing protocol, etc.

With the exponential increase in IP communications, metadata has a great untapped potential for mapping communication patterns, especially for protection purposes as in the case of lawful intercept. It also solves the problem of ballooning storage requirements. In fact, the metadata approach may even be the ONLY way to handle lawful intercept in the future!

So the opportunity is to leverage metadata for intelligence gathering, for ex. to reconstruct links between people, to understand which virtual IDs the same person is using (starting from physical IP/IMSI) or to identify intentionally hidden information (“the Dark Web”).

However, IP communication metadata is not readily available on operator servers, or resides on third-party Web servers, outside the control of a given telco (think P2P,social networking, etc.). Therefore, metadata has to be extracted directly from the network.

So how can buyers of LI solutions leverage IP metadata?

1. Use the rich set of information available with IP metadata to build completely new types of intelligence solutions - as a complement to content-focused solutions. This gives a better understanding of potential threats (the macro-view)
2. Take advantage of more computer-based information processing 1.5 billion Internet users, thousands of Web applications… the number of analysts cannot increase at the same rate as the number of IP communications, the number of Web applications and the amount of content generated > need to think differently?!
   

The solution is to implement automatic detection of suspicious behavioral communication patterns and create real-time view of the threat situation based on continuous streaming of metadata and network information. There is also an opportunity to minimize storage and post-processing time by extracting significant information and structuring it as soon as each metadata is available

Long live metadata!

Jerome

Network Intelligence: a key trend in IT networking

I took advantage of the Christmas break to catch up on some reading, like this article on Forbes.com that a friend sent me. It was interesting to see that the author Dan Woods cleverly identified a key trend in IT networking: networks aren't just highways!

As Dan puts it, "This week the JargonSpy looks at the gradual transformation of the network as transport, to the network as a source of information. As the network comes alive as a center of intelligence, the way that many applications do their jobs will change for the better."

It is great to see that Network Intelligence is becoming a discussion topic even in mainstream business media!

Wishing everyone a happy new year,

Jerome

DPI or Information Extraction? - Making the difference

I just got back from the Broadband World Forum where I met a number of people to whom I had to describe the core competency of Qosmos. I realized that many people are not aware of the difference between DPI and Information eXtraction (IX for short).

Put simply, Deep Packet Inspection could be described in the following terms: it recognizes network traffic, it has the ability to act on the traffic, and vendors typically sell complete solutions for specific markets such as lawful intercept, traffic optimization, etc.

Information eXtraction – IX, on the other hand, is different: it actually extracts network information and makes complete sense out of it; ix doesn’t act on traffic flows. In the case of Qosmos and IX, the focus is on best-in-class network intelligence, for use in other people’s complete solutions. As a matter of fact, DPI specialists can even build their solutions based on an Information eXtraction engine like Qosmos.

This is Qosmos’s real differentiator, and if those we met at the BBWF were to take away one thing it should be this. Once potential users of information extraction understand this, the horizon is suddenly different. And a wealth of possibilities opens up – I’ll describe this in an upcoming post.

Jerome