Solving Unsolvable Problems (Part 2)

As I described in my previous post, I believe that solution vendors are facing a crucial decision point and are realizing that developing network intelligence capability internally would create a number of “unsolvable problems”.

Unsolvable problem number 2: Not being able to use traditional product development and management methods

The high-tech industry typically uses a structured approach for product development and management, with most key activities aligned around go/no go decision points and defined time lines. These processes are built to ensure that new products are delivered on time, according to specifications and with the adequate quality.

However, companies who chose to develop network intelligence technology internally quickly discover that the usual methods cannot be used. Web protocols such as Webmails change continuously without notice, which means that development roadmaps cannot be easily controlled. Development teams must be quick to react to new protocol evolutions and use reverse engineering techniques to update their network intelligence software.

This way of working is counter-cultural for many high-tech companies and can even be incompatible with the rest of the organization. From a business standpoint, it can even be unsustainable and unprofitable - unless you make network intelligence technology your core business, like Qosmos;-)

Jerome

Solving Unsolvable Problems (Part 1)

For the past several years, Deep Packet Inspection (DPI) has been used by developers to get basic traffic visibility for applications such as traffic shaping and network security.

We are now at a key juncture of the market, where many solution vendors need much more detailed visibility into network-based activity. For example, for market research applications it is not enough to recognize generic http traffic; it is necessary to extract metadata such as name of visited Website, page content, time spent on visit, basket share, referent, etc. The same is true for applications such as lawful intercept and government cyber security, which demand extremely accurate and fine-grained information on communication flows in order to map exact communication flows and identify threat patterns.

So the key question becomes: - should solution vendors develop complex network intelligence technology internally or should they source it from a specialist?

Solution vendors realize that developing network intelligence capability internally necessitates a step-function in terms of investment and strategy. It requires considerable internal R&D resources, detracts from core business and poses a number of additional problems in term of organizational efficiency, culture, and return on investment.

During the next couple of weeks, I will describe the “unsolvable problems” facing solution vendors and highlight the advantages of sourcing complex network intelligence technology from a specialist like Qosmos.

The first “Unsolvable Problem”: Having to continuously redevelop software

Developing network intelligence software to handle a particular protocol is only the tip of the iceberg: you also need to invest continuously to update your software so it can handle new versions of protocols. Some basic protocols are easy to manage since they are relatively stable: IMAP, SMTP, POP, HTTP, etc. However, there is a vast number of other protocols which are proprietary and evolve at a rapid pace (Webmails, P2P, social networking, gaming, etc.).

For example, the Livemail protocol changed several times during 2008. And each time a new version is released, most of the network intelligence software has to be redeveloped, without much reuse of previous development. This requires a particular culture and special methods: reverse engineering, custom-made tools, fast reaction to protocols changes, etc.

Jerome

BBWF09 - DPI Workshop / Monday, 07 September 10:00 – 12:30

Just a little heads-up to tell you that I will be a speaker at the upcoming Broadband World Forum which will be help in Paris 7-9 September. Il will participate in the workshop “Deep Packet Inspection: Technology, Promise & Controversy. What You Need to Know” on Monday, 07 September 10:00 – 12:30. It should be an interesting discussion, with views from solution suppliers, service providers and Network Intelligence Technology specialists (Qosmos).

For those of you who would like to see Qosmos Network Intelligence Technology in action, you are welcome to see a live demo on the exhibition floor: just come to our partner BreakingPoint’s Stand # 470.

I hope to see you there!

Jerome

Forbes article on Programming Innovation

Dan Woods just published an interesting article on Forbes.com called “Programming Innovation”. He describes how Qosmos technology enables a new kind of market research for our partner GfK.

Market research is of course only one example of how Qosmos Network Intelligence can be used. I like Dan’s way of describing our toolkit as a catalyst for innovation across a wide range of applications: “Now, tens of thousands of companies potentially have access to a new form of information derived from the network”.

Based on my daily discussions with customers and prospects, I can confirm that the trend is real – I can’t wait to see tens of thousands of companies benefiting from Qosmos Network Intelligence ;-)

Jerome

COTS not enough for government cyber security

I just read this interesting article by Nextgov, confirming that relying only on commercial products can increase vulnerabilities to cyber attacks: http://www.nextgov.com/nextgov/ng_20090618_3505.php. This is in line with my recent discussions with governments who use Qosmos technology to enhance their cyber security beyond COTS capabilities. Not only could COTS products contain hidden code; they often lack capabilities required by government security specialists for detecting certain threat patterns and/or they lack the flexibility to customize according to special government security policies.

Jerome

Scoble

I came across an interesting post today: http://scobleizer.com/2009/05/29/kara-is-wrong-about-2010web/. Scoble’s point is there’s no “web3.0” but there are several key trends that are making the web different today.

Scoble:

The things that are happening are NOT just Twitter and search. Here, let me recount again what is making up the 2010 Web:

1. Real Time. Google caught the Wave of that trend today BIG TIME.

2. Mobile. Google, again, caught that wave big time Wednesday when it handed Android phones to everyone at its IO conference.

3. Decentralized. Does Microsoft or Twitter demonstrate that trend? Not really well.

4. Pre-made blocks. I call this “copy-and-paste” programming. Google nailed it with its Web Elements (I’ll add a few of those next week).

5. Social. Oh, have you noticed how much more social the web is? The next two days I’m hanging out on an aircraft carrier with a few people who do social media for the Navy.

6. Smart. Wolfram Alpha opened a lot of people’s eyes to what is possible in new smart displays of information.

7. Hybrid infrastructure. At the Twitter Conference this week lots of people were talking about how they were using both traditional servers along with cloud-based approaches from Amazon and Rackspace to store, study, and process the sizeable datasets that are coming through Twitter, Facebook, and friendfeed.

How is this relevant for Network Intelligence Technology?

• Real Time: need to identify real-time data and metadata,
• Decentralized: need to both manage infrastructure and maintain visibility and control over communication when you can’t “be there”
• Pre-made blocks: this exactly what the Qosmos model - providing enabling technology to ISVs, SIs and developers. With ixEngine 5.0, we just announced broader protocol support and easier customization with the Protocol Plug-in API.
• Smart: on the broadest level, Qosmos technology makes the network itself more intelligent and more of a “participant” in the information delivery process. In ixEngine 5.0, the Sessionizer organizes data flows in logical and flexible ways so that people can make more sense of the communication flying around the web.

It is nice to see that Qosmos not only captures the wave of the evolving web, but our technology is actually enabling this evolution!

Jerome

NI (Network Intelligence) – the next BI (Business Intelligence)?

If you have been involved in data processing, this probably sounds familiar to you. Remember the 1980’s when the guys from the financial department had to ask database managers to program specific SQL queries into the DB2 accounting systems, and how difficult it was to get the right information, and how long it took? The result was that you had to wait for reports, or wait for program changes etc.

Why was it so difficult?

1) Programs and reports where hard-coded, build for only ONE type of use. Any other request was just impossible.

2) There was no way to access and use a database from outside the IT organization.

3) The financial controller could not program in SQL and the database manager did not know finance.

And this is why Business Intelligence has been so successful: it enabled non-programmers to access the information they need and create the report they want in a minute when it took weeks or months before… if even possible.

Now, people involved in the network world face the exactly same problem. Take the product manager from a Telco; he needs to gather detailed subscriber information to efficiently manage the portfolio of services and estimate future demand. The network can provide this information at a very high level of granularity, but how can our product manager get this information? He will go to the network administrator and ask him for the info. And the network administrator will go search for the answer inside the network equipment logs … and he will probably spend more time than he thought, ending up with partial results, because:

1) logs are complicated to use for anything else than troubleshooting, and they are not easily readable

2) logs only provide a partial view of events

3) logs have different formats when they come from different systems

4) how do you manage the huge amount of information???

And just like business intelligence opened up the world of databases to finance people, the network intelligence technology provides an actionable access to the enormous amount of information transiting through networks or computed by the network

1) Network intelligence probes are passive; they do not impact the production equipment in the network and can be access independently, without risk

2) Network Intelligence probes are not built for a specific use, and thus provide a simple access to all the data of the network, whatever the need

3) Network intelligence probes do not use logs, they scan the traffic in real time, providing the same data structure whatever the diversity of network equipments brand and models, making consolidation easy.

3) Network Intelligence probes allow users to select only the information they need, making data easily actionable

4) Network Intelligence probes feed reporting tools and enables non-programmers to access information, without going through the technical department.

The result: easy and fast access to the valuable information provided by traffic analysis. In the same way that business software solutions (e.g. ERP, CRM) today provide business intelligence (BI) capabilities, network solutions will provide Network Intelligence (NI) capabilities very soon!

Jerome