Solving Unsolvable Problems (Part 4): Managing highly complex technology

Since network intelligence goes beyond DPI in terms of the level of visibility, protocol management, attribute recognition, and information extraction, even companies who have incorporated DPI capabilities into their solutions will require a new level of expertise. For example, Qosmos engineers have developed a specific meta-programming language to build Webmail and HTTP protocol plug-ins. Specific techniques and tools must be developed for quality assurance and to make reverse engineering more efficient.  This is an order of magnitude remote from the business of someone who sells complete solutions. 

In most cases, a separate R&D organization must be created. But once committed, companies realize that development times are difficult to estimate, timelines are incompressible and that the skills are so specialized that it becomes nearly impossible to outsource parts of the development.

Jerome

Solving Unsolvable Problems (Part 3): Committing considerable resources, with uncertain returns

If you choose to tackle a family of protocols (e.g. Webmails), you have to develop network intelligence capabilities for the most important protocols in this family, otherwise your solution will ineffective or incomplete (think of traffic optimization or cyber security applications). In addition, the total number of applications and protocols increase continuously (e.g. 50,000 applications are now available for the iPhone), and very few protocols ever disappear…

For a company whose core business is not network intelligence technology, this translates into high costs of entry and ever-rising investments. To make things worse, end customers of turn-key solutions may not appreciate the importance of continuous protocol updates and the amount of work required to keep the solutions current. This means that a solution vendor could end up investing considerable resources for which and end customer does not perceive high value and therefore may not be ready to pay… 

Jerome

Solving Unsolvable Problems (Part 2)

As I described in my previous post, I believe that solution vendors are facing a crucial decision point and are realizing that developing network intelligence capability internally would create a number of “unsolvable problems”.

Unsolvable problem number 2: Not being able to use traditional product development and management methods

The high-tech industry typically uses a structured approach for product development and management, with most key activities aligned around go/no go decision points and defined time lines. These processes are built to ensure that new products are delivered on time, according to specifications and with the adequate quality.

However, companies who chose to develop network intelligence technology internally quickly discover that the usual methods cannot be used. Web protocols such as Webmails change continuously without notice, which means that development roadmaps cannot be easily controlled. Development teams must be quick to react to new protocol evolutions and use reverse engineering techniques to update their network intelligence software.

This way of working is counter-cultural for many high-tech companies and can even be incompatible with the rest of the organization. From a business standpoint, it can even be unsustainable and unprofitable - unless you make network intelligence technology your core business, like Qosmos;-)

Jerome

Solving Unsolvable Problems (Part 1)

For the past several years, Deep Packet Inspection (DPI) has been used by developers to get basic traffic visibility for applications such as traffic shaping and network security.

We are now at a key juncture of the market, where many solution vendors need much more detailed visibility into network-based activity. For example, for market research applications it is not enough to recognize generic http traffic; it is necessary to extract metadata such as name of visited Website, page content, time spent on visit, basket share, referent, etc. The same is true for applications such as lawful intercept and government cyber security, which demand extremely accurate and fine-grained information on communication flows in order to map exact communication flows and identify threat patterns.

So the key question becomes: - should solution vendors develop complex network intelligence technology internally or should they source it from a specialist?

Solution vendors realize that developing network intelligence capability internally necessitates a step-function in terms of investment and strategy. It requires considerable internal R&D resources, detracts from core business and poses a number of additional problems in term of organizational efficiency, culture, and return on investment.

During the next couple of weeks, I will describe the “unsolvable problems” facing solution vendors and highlight the advantages of sourcing complex network intelligence technology from a specialist like Qosmos.

The first “Unsolvable Problem”: Having to continuously redevelop software

Developing network intelligence software to handle a particular protocol is only the tip of the iceberg: you also need to invest continuously to update your software so it can handle new versions of protocols. Some basic protocols are easy to manage since they are relatively stable: IMAP, SMTP, POP, HTTP, etc. However, there is a vast number of other protocols which are proprietary and evolve at a rapid pace (Webmails, P2P, social networking, gaming, etc.).

For example, the Livemail protocol changed several times during 2008. And each time a new version is released, most of the network intelligence software has to be redeveloped, without much reuse of previous development. This requires a particular culture and special methods: reverse engineering, custom-made tools, fast reaction to protocols changes, etc.

Jerome

BBWF09 - DPI Workshop / Monday, 07 September 10:00 – 12:30

Just a little heads-up to tell you that I will be a speaker at the upcoming Broadband World Forum which will be help in Paris 7-9 September. Il will participate in the workshop “Deep Packet Inspection: Technology, Promise & Controversy. What You Need to Know” on Monday, 07 September 10:00 – 12:30. It should be an interesting discussion, with views from solution suppliers, service providers and Network Intelligence Technology specialists (Qosmos).

For those of you who would like to see Qosmos Network Intelligence Technology in action, you are welcome to see a live demo on the exhibition floor: just come to our partner BreakingPoint’s Stand # 470.

I hope to see you there!

Jerome