The Real Time Web

‘You can't ignore the real-time Web’ claimed Gartner Analyst James Lundy in his keynote address to the Collaborate 2.0 Summit in October 2009.

The web has always been close to real time. That was its attraction from the start. Digital was more flexible and faster to process than analogue communication. But for non geeks the Real Time Web has become fashionable. It's fashionable because of the phenomenal rise of Twitter. Twitter, now over three years old, showed everyone how fast information was spread across the web by social networks. Closely behind Twitter is Google’s Wave, a service for instant key-stroke-by-key-stroke communication and interaction.

Lundy points out that companies, particularly publicly traded and regulated ones, are concerned about real time services for one simple reason -- compliance, a requirement that companies keep track of communications related to company business.

But companies can't ignore the popularity of these services or their inevitable use, said Lundy. He recalled, for example, being in meeting with a Wall Street client who said instant messaging wasn't allowed at their firm.

"The minute those managers leave, we asked the other people in the room and they said, 'Absolutely, we still do it,' referring to instant messaging."

Brian Morrissey reported on Diet Coke’s initiatives in Real Time Web in AdWeek last November noting that

“Marketers including Burger King and Adidas are warming up to real-time Web content, mirroring a shift in digital media away from asynchronous communication and content delivery (e.g., the sending of e-mails and watching posted videos) towards instant feedback and interaction. Upping the ante for these marketers are real-time systems like Twitter and Facebook, which mix content delivery with communication, making something hours' old seem stale.

People, and notably companies, found they needed to be better informed and they needed to watch for mentions online and, urgently, Twitter as well as blogs and other social media.

But what do we mean by Real Time Web? Daniel Tenner described it well in his blog post:

“Real-time web” can mean any number of things, from “live updates without refreshing the page” to “see text as it’s typed”, but all those are technological rather than conceptual definition. At its core, the concept of “real-time web” must be about the immediacy of information flow. Something happens (whether it’s someone typing a message to you or Michael Jackson dying) and you find out about it immediately (or nearly so).

Monitoring the internet and specific content on the internet is not new. Organisation that offer such services include news monitoring by Google (Google Alerts), Technorati, CyberAlert and eWatch There are companies that exclusively focus on online/social media such as Radian6 and Scout Labs. They cover blogs, wikis, Twitter, social networks, bulletin boards and discussion lists. Meanwhile the traditional press clipping agencies such as Factiva, Moreover, Durrants and Cision still keep a wary eye on newspapers and magazines and re-digitise the content for computers to analyse.

Some of these vendors offer regular updates every day, some hourly and some, like Google Alerts in near real time.

There are other services that help organisations such as RSS and Atom feeds that poll web sites at regular (typically hourly) intervals. Then there are the real time services based on a simple, open, server-to-server ‘web-hook-based’ pubsub (publish/subscribe)’ protocol extension to Atom and RSS called the PubSubHubbub protocol that can get near-instant notifications when a topic (feed URL) is updated.

Real Time Web is available using such services. They are time consuming to set up and the client needs to know which sites to monitor in advance. So far only a few small feed readers have begun consuming these feeds; RSSCloud developer Dave Winer's own River2, a complex but customizable desktop feed reader, and LazyFeed, a simple but enjoyable feed-powered discovery engine, have turned on full support for real-time feeds.

Code named Wasabi from Netvibes is a widget service that will go into private beta later this week and will launch to the public at December's Le Web conference in Paris, where the theme of the event is the real-time web.

More contenders in this field are covered in a guest article in Mashable, the Social Media guide by Bernard Moon, who recognises a level of hype about the issue.

So what we find is a host of services covering a wide range of online and offline media.

Very few services are real time. They offer monitoring at intervals and where these services are swift they do not include all the channels out there.

There is one further flaw.

None of these services comprehensively monitors all the content that is publically available online.

There are so many channels for communication online that it is hard to watch them all. Some are, and will remain niche and almost insignificant. Others, though of little consequence in themselves feed the big beasts of the internet.

Much of the content is driven by bots and other automated services and there is still spam galore.

The service provided by Klea Global through its www.nextmention.com service resolves these two big issues. It monitors’ the web for everything and provides ten minute updates free and real time updates in its soon to be announces premium service.

Of course, this is by no means ideal because the many divergent channels from web sites to news to blogs, wikis, Twitter, social networks and all the rest are all jumbled up in the instant feed.

The service is more coherent on the Nextmention site which used a Bayesian bot to sort out the pages into media types and more developments in this direction are anticipated.

There are some other services that are worthy noting and which show how Real Time Web is driving a need for more and faster services. Topsy (http://topsy.com) is a real time search engine that stand out because it focused on real time links as opposed to real time content. So, when you perform a search at Topsy, instead of seeing what people are talking about on the real time web, you are to see what the most popular and prominent links are being shared on the real time web. You can even sort to see the most shared links over the past hour, day, week, or month. Meantime rumours have been swirling all over the web in regards to a partnership Yahoo is discussing with OneRiot. OneRiot (http://oneriot.com/) offers users a real time search engine which can be sorted based on web results and video results.

Meantime, People like Nova Sivack lead us to the problems this content and these services present. He writes in his blog Minding the Planet:

“In the next 10 years, The Stream is going to go through two big phases, focused on two problems, as it evolves:

1. Web Attention Deficit Disorder. The first problem with the real-time Web that is becoming increasingly evident is that it has a bad case of ADD. There is so much information streaming in from so many places at once that it's simply impossible to focus on anything for very long, and a lot of important things are missed in the chaos. The first generation of tools for the Stream are going to need to address this problem.
2. Web Intention Deficit Disorder. The second problem with the real-time Web will emerge after we have made some real headway in solving Web attention deficit disorder. This second problem is about how to get large numbers of people to focus their intention not just their attention. It's not just difficult to get people to notice something, it's even more difficult to get them to do something.”

This is where some of the thinking for the next phase of internet development is going on and how in a very short time one can imagine services that address both these problems with the Real Time Web..