All the PR data you can eat - without getting indigestion

In the next few months, I and some friends will be creating a new resource for PR practitioners, evaluation companies and academics (and a lot of other people too).

The idea is really quite simple.

If you could search online for all media (that includes all kinds of media from Twitter to the Telegraph) stories and comments about your organisation everyday (or hour, week, month, year etc) and put them in a database, spreadsheet, list or a web page that would be pretty average, but useful. If you could do the same about competitors, market sectors, business partners or individuals such as journalists, Wikipedians, FB friends that might be useful too. We are going to make this happen.

Each article (citation), will have a lot of information attached to it. We have some indication via Alexa, of  the age, sex and location of readers. We will also add all we know from Google Trends and Google Analytics that there is a lot more information that attaches to stories, blog posts and online comments. We will collect it  (and more) all and attach it to your story's URL.

Of course, it will be useful to be able to extract the text (without the HTML mark-up and advertisements)  and make it searchable, summarise-able and with lists of tags, mark-ups, semantic concepts (ranked in order of significance), Parts of Speech and hyper-links in and out. So we will add them also so you can use the information if you need to.

Now comes your part.

We will make these data available to you. The full set. Via an API, spreadsheet and a range of other formats so that you can download just what (that is, only the data and no more or less) you want. Yes, that's right, we will collect it, but you don't have to use it. So, if all you want is a news feed every day that's OK. If you want international trends by the hour, that's cool too. You can choose to smell the ocean or sip from the fire-hose.

Now you will be able to make your own news apps, evaluation apps and subject specific web news outlets.

You will be able to match media data with business data (enquiries, sales requests for information etc etc - anything you want). Making a list of Twitter users mentioning your brand will be as easy and that list of blogs mentioning your competitors or Slideshare mentions about your CSR programme. All the information will be ready for you to mine.

We will offer an API that any friendly programmer can use to make you anything you want.

We will offer some tools too. A de-duplication tool will be useful so that you can set up your own de-duplication parameters (all those re-tweets can be counted without having to edit every one) and a smart curation capability would, we guess, be helpful too.

Perhaps you have some ideas as to what you would like to include in our data set (mentions in Facebook ? - we're onto it).

If you are in-house, an agency or an academic PR, or evaluation company or research organisation and you want to be a beta tester, or early adopter, we will be delighted to talk you through what we have in mind to do.

In due course we shall be holding open events and application developer seminars but all that is for the future.

Right now we aim to offer a simple, though significant service.

Online Public Relations research tools

It has occurred to me that I have never shown the PR industry, and notably academic researchers, the technologies I, and my students and commercial partners have used to come to the conclusions we do.

I make them available to you here and now.

Some are quite old and have been superseded by better technologies and I am very happy to help researchers who want to use these tools in research activities that will give the PR industry better insights into the nature of online communication.

Semantic Web Experiments

We have been working on Latent Semantic Indexing for nearly a decade but now we are looking at a range of 

other ways the semantic web can offer practitioners insights.

This is an experiment that dynamically identifies an ontology. The objective here will be to allow the practitioner to drill down further and further to find out who is affected and involved with an entity in a web page (e.g. news story).

You can try it out for yourself her http://entitymap.appspot.com/

Reputation Wall

This is a development we have taken a very long way. It searches for pages about a search topic, opens up the web pages, normalises the texts, parses the texts of all the pages for semantic concepts (latent semantic indexing - we have our own software to do this) and then looks for the most powerful concepts month by month going back a year.

You can create your own 'Reputation Wall' here http://reputationwall.appspot.com

Track This Now

A media story or picture comes to prominance and you want to now where in the world it is popular right now. Well, here is the service that gives you an instant world and regional snap shot.

You can find your news of the moment here

http://www.trackthisnow.com/

Finding Semantic Concepts

This tool was used to discover relationships between people and organisations in a big research project. You can enter a lot of website URL's into it and it will return the 50 most significant semantic concepts in the corpora. 

 I find it is more manageable if you remove the URL's and then paste the words into a programme like TagCrowd to generate a semantic word map.

You can use this software from here http://www.netreputation.co.uk/summariser/getconcepts.php

Value Systems Analysis

This software levers the semantic analysis of pages and looks at bigger corpora. In this case current Google News, Blogs and natural search. The analysis shows values in bold in the texts. 

The software was developed as a series of software developments for academic research. In this case the  software was part of the development for building the values theory in PR. The outcome was presented at theBled symposium in 2009:

A proof of concept for automated discourse analysis in support of identification of relationship building in blogs. 
David Phillips, Bruno Amaral (DOC, 977 KB) 
A proof of concept for automated discourse analysis in support of identification of relationship building in blogs. 
David Phillips, Bruno Amaral (PPT, 1.5MB)

You can have a go here http://www.netreputation.co.uk/values/

Web Page Text Analysis

One of the hard things to do is to re-construct web pages to extract the text and then find the sematic concepts

and much more.

This tool is really clever because it shows the steps involved. You can extract the text on web pages with this tool too.

http://keeneuk.appspot.com/textanalyse

Video News

Finding the latest video is harder than you think. There are so many channels.

We thought that it would be a good idea to have them all in one place and this was the first part of developing a special type of search which you can see in NewsRokit.

You can play with the software here http://crowdmint.appspot.com/

Google Hourly Search to CSV

Everyone want to get a spreadsheet of the latest pages indexed by Google. This toy allows you to just the last

hour's worth of pages indexed by Google.

To try it yourself here is the URL http://search2csv.appspot.com/

Summariser

Did you want to make a quick summary of a web page?

This may help.

http://www.netreputation.co.uk/summariser/summari.php

Throughout, these experimental tools do not use word counts. The approach is always to use latent semantic indexing as the basis for experimentation.

Have fun with the technologies.

Semantic progress

Yesterday Philip Sheldrake gave a talk to the Chartered Institute of Public Relation Social Media gathering (anyone can come - it costs £10 and is at 5pm every Thursday) on the semantic web. It was excellent and you can access it here.

Among the things he showed us was the work of Philipp Heim (University of Stuttgart), Steffen Lohmann (Carlos III University of Madrid), Timo Stegemann (University of Duisburg-Essen).
They have taken existing structured data to allow you to go and find relationships between two entities (I chose to find the relationship between Nick Clegg and David Cameron on this page).

The work they have done is important and uses existing structured data sets.

At Klea Global my colleague Girish has been working towards a way of creating structured data sets using Natural Language Programming including LSI to build (RDF) structures on the fly from content derived from newspapers, Blogs, Facebook, Twitter etc.

He already has gone quite a long way and you can see an example of how it is possible to create this process with some very new and pretty smart tools. One of which is available for you to try here.

OK, so what is this for, and why is it relevant to Public Relations.

I guess the secret is in the second part of the name of our profession: relations.

Using these capabilities, we can find out all manner of relationships between two entities (subject - object). When, using the Semantic Web, these relationships make sense, all this data will be ever more powerful.

To get some idea of how much data, here is another 'toy' you can play with from Klea Global labs (and yes, I have started to put it all in one place at last): Track This Now. Using this free 'search and scope presence' software, you will see that an amazing amount of information is accumulating about your company, client, university etc.

Knowing how much there is, and knowing that most of what is said online about organisation does not come from the company or traditional media is only half the battle. There is so much accumulating out there that we are overwhelmed.

If only we could find out what the relationships were between all those tweets and press articles, we would have some chance of influencing them, building up huge SEO for clients and lots of other marvellous things. Worry not... salvation is at hand.

These are very early days for these developments to bear fruit for the PR industry but next year they will be quite astonishing. We already know how to do it and in less than a year will be doing it.

This is so exciting for our industry and my only regret is that we don't have a single university in the UK with a capability to do this sort of research.

Talking of Real Time Web

‘You can't ignore the real-time Web’ claimed Gartner Analyst James Lundy in his keynote address to the Collaborate 2.0 Summit in October 2009.

The web has always been close to real time. That was its attraction from the start. Digital was more flexible and faster to process than analogue communication. But for  non geeks the Real Time Web has very recently become fashionable. It fashionable because of the phenomenal rise of Twitter. Twitter, now over three years old, showed everyone how fast information was spread across the web by social networks. Closely behind Twitter is Google’s Wave, a service for instant key-stroke-by-key-stroke communication and interaction and then, of course, we no have real time search available fro Google which shows every new page it indexes almost as it happens.

Lundy points out that companies, particularly publicly traded and regulated ones, are concerned about real time services for one simple reason -- compliance. This is a requirement that companies keep track of communications related to company business.

But companies can't ignore the popularity of these services or their inevitable use, said Lundy. He recalled, for example, being in meeting with a Wall Street client who said instant messaging wasn't allowed at their firm.

"The minute those managers leave, we asked the other people in the room and they said, 'Absolutely, we still do it,' referring to instant messaging."

Brian Morrissey reported on Diet Coke’s initiatives in Real Time Web in AdWeek last November  noting that:

“Marketers including Burger King and Adidas are warming up to real-time Web content, mirroring a shift in digital media away from asynchronous communication and content delivery (e.g., the sending of e-mails and watching posted videos) towards instant feedback and interaction. Upping the ante for these marketers are real-time systems like Twitter and Facebook, which mix content delivery with communication, making something hours' old seem stale.

People, and notably companies, found they needed to be better informed and they needed to watch for mentions online and, urgently, Twitter as well as blogs and other social media.

But what do we mean by Real Time Web? Daniel Tenner described it well in his blog post:

“Real-time web” can mean any number of things, from “live updates without refreshing the page” to “see text as it’s typed”, but all those are technological rather than conceptual definition. At its core, the concept of “real-time web” must be about the immediacy of information flow. Something happens (whether it’s someone typing a message to you or Michael Jackson dying) and you find out about it immediately (or nearly so).

Monitoring the internet and specific content on the internet is not new. 

Organisation that offer such services include news monitoring by  Google (Google Alerts), Technorati, CyberAlert and eWatch.

There are companies that exclusively focus on online/social media such as Radian6 and Scout Labs. They cover blogs, wikis, Twitter, social networks, bulletin boards and discussion lists. 

Meanwhile the traditional press clipping agencies such as Factiva, Moreover, Durrants and Cision still keep a wary eye on newspapers and magazines and re-digitise the content for computers to analyse.

Some of these vendors offer regular updates every day, some hourly and some, like Google Alerts in near real time.

There are other services that help organisations such as RSS and Atom feeds that poll web sites at regular (typically hourly) intervals. Then there are the real time services based on a simple, open, server-to-server ‘web-hook-based’ pubsub (publish/subscribe)’ protocol extension to Atom and RSS called the PubSubHubbub protocol that can get near-instant notifications when a topic (feed URL) is updated.

Real Time Web is available using such services. They are time consuming to set up and the client needs to know which sites to monitor in advance. So far only a few small feed readers have begun consuming these feeds; RSSCloud developer Dave Winer's own River2, a complex but customizable desktop feed reader, and LazyFeed, a simple but enjoyable feed-powered discovery engine, have turned on full support for real-time feeds.

A number of services are now being introduces. S typical solution is  Wasabi from Netvibes is a widget service.

More contenders in this field are covered in a guest article in Mashable, the Social Media guide by Bernard Moon, who recognises a level of hype about the issue.

So what we find is a host of services covering a wide range of online and offline media. Very few services are really real time. They offer monitoring at intervals and where these services are swift they do not include all the channels out there.

There is one further flaw.

None of these services comprehensively monitors all the content that is publically available online.

There are so many channels for communication online that it is hard to watch them all. Some are, and will remain niche and almost insignificant. Others, though of little consequence in themselves, feed the big beasts of the internet.

Much of the content is driven by bots and other automated services and there is still spam galore.

The service provided by Klea Global through its www.nextmention.com service resolves these two big issues. It monitors’ the web for everything and provides ten minute updates free and real time updates in its soon to be announces premium service.

Of course, this is by no means ideal because the many divergent channels from web sites to news to blogs, wikis, Twitter, social networks and all the rest are all jumbled up in the instant feed.

The service is more coherent on the Nextmention site which used a Bayesian bot  to sort out the pages into media types and more developments in this direction are anticipated.

There are some other services that are worthy noting and which show how Real Time Web is driving a need for more and faster services.  Topsy (http://topsy.com)  is a real time search engine that stand out because it focused on real time links as opposed to real time content.   So, when you perform a search at Topsy, instead of seeing what people are talking about on the real time web, you are to see what the most popular and prominent links are being shared on the real time web.  You can even sort to see the most shared links over the past hour, day, week, or month.  

Meantime rumours have been swirling all over the web in regards to a partnership Yahoo is discussing with OneRiot.  OneRiot (http://oneriot.com/)  offers users a real time search engine which can be sorted based on web results and video results. 

Meantime, people like Nova Sivack lead us to the problems this content and these services present. He writes in his blog Minding the Planet:

“In the next 10 years, The Stream is going to go through two big phases, focused on two problems, as it evolves:

1. Web Attention Deficit Disorder. The first problem with the real-time Web that is becoming increasingly evident is that it has a bad case of ADD. There is so much information streaming in from so many places at once that it's simply impossible to focus on anything for very long, and a lot of important things are missed in the chaos. The first generation of tools for the Stream are going to need to address this problem.
2. Web Intention Deficit Disorder. The second problem with the real-time Web will emerge after we have made some real headway in solving Web attention deficit disorder. This second problem is about how to get large numbers of people to focus their intention not just their attention. It's not just difficult to get people to notice something, it's even more difficult to get them to do something.”

This is where some of the thinking for the next phase of internet development is going on and how in a very short time one can imagine services that address both these problems with the 

Real Time Web.

 What does all this mean to practitioners.

The key issues for the PR profession are not as easy. The need to be able to monitor the web real time is hard.

There are a lot of tools for monitoring the Fortunately, Publicasity has partnered with companies that have the necessary technology.

Being able to identify opportunities and dissonance between brands and the brand values held by consumers is the next big challenge.

The Publicasity digital team is already working on this with Lisbon University and Klea Global. Early examples of the Real Time technology research are available and Klea Global is closely associated with developments of the Real Time Reputation Wall.

We are now able to discover the way online communities understand corporate and consumer brands both when the brands are top of mind and in lifestyle situations.

This is a big advantage for Real Time Web interactions where marketers can respond to the changing consumer landscape as they evolve.

This is ground breaking capability that the account teams can bring to meet Real Time Web success.

A New Monitoring Tool - Real Time

Greg Cohn is the Director Strategy & Business Development at Yahoo! Today, he is making a big thing about Real Time Web and I am delighted.

The reason I am thrilled is that a company I am associated with, Klea Global, has launched a Real Time Monitoring service called NextMention. The basic service is free... as you would expect and a raft of commercial services are due out soon.

What is does is simple. It monitors everything. News, Blogs, Twitter, social networks, web sites, Google's Sidewiki and lots more and if your organisation or issue is mentioned it alerts you. The alert can be by instant messenger or email and for paid for services by SMS, Skype and lots of other communications channels and platforms.

For PR, monitoring clients 360 degrees of internet content is a big deal. You will be surprised at how much there is.

Modern Day PR Monitoring and Evaluation

Practitioners have monitored the environment affecting their clients forever. It’s what we do. Today we have more to monitor and we have to do it faster.

Most know how much of a challenge monitoring this is. Most have their ‘Google Alerts’, their blog and Twitter monitors and the daily updates from Linkedin groups. These are augmented with online media (web based publications) and media online (print publications with online content) and subscriptions to all manner of news services to supplement the daily John Humphreys pre-breakfast fest, newspaper, magazine, radio, TV and press clips.

The internet stream of consciousness seems endlessly oppressive because the practitioner needs to follow all the conversations while the users only follow one or two. It is, all too often, unmanageable and is, mostly, not very comprehensive.

Even with Tweetdeck and Feedreader going full blast, professional communication and relationship advisors are blithely ignorant of all but a fraction of web pages that mention their clients. Does, for example, PR Week see all of the citations that are published about it online at the rate of one every 90 seconds 24/7?

The truth is that after the news, blog, twitter, social networks and discussion list citations, the string of website references, comment in new channels and machine generated content is mostly factors larger. The client’s online web cloud grows every day. It is a competitive asset and creates a footprint for all to follow and affects the algorithms of search engines that make organisations searchable and famous. Klea Labs which is a new interest of mine has interesting capabilities such as its Web to IM service which provides real time monitoring of 'everything'.

For most organisations, more than half of online content appearing each day online is not monitored, measured or evaluated. In addition in an era of Real Time web, Twitter, is the nearest most organisations get to following the movers and shakers of internet reputation in real time.

Too much Too much, I can hear a whole profession cry. Yes, we do have to bring order to all this stuff and this is where there can be a happy marriage between PR and technology. All the content can be sorted into the different generics such that your Facebook content is not confused with your tweets.

Even when some practitioners get this information, is it enough in a digital age?

Far from it. In fact such a view of client publics would probably be misleading. The impression would be, as Colin Farrington once described it “ill-informed, rambling descriptions of the tedious details of life or half-baked comments on political, sporting or professional issues. They read like a mixture of the ramblings of the eponymous Pub Landlord and the first draft of a second rate newspaper column.”

But this is to take and overview of all the conversations of all the Pub Landlords and all columnists. Out of context they do seem banal. But once you are immersed in the community where these comments are made, they make sense and are about real people and the issues in their lives.

This means that monitoring is only part of the story. The content we read needs to be evaluated and evaluated in context.

As long ago as 2007, Read Write Web was discussing the importance of semantics to Google. It is semantics that allows us to make sense of content in context. For ten years I have been involved in semantic developments which provided the technology behind the relationship management research presented at the Bledcom PR conference this year. In PR, semantic analysis is a boon. It provides ways in which computers can mimic human needs. It is not able to completely second guess human understanding but it takes a lot of the hard work out of gaining actionable insights.

We have now come a long way from monitoring online content using tools like Google Alerts and RSS feeds to monitoring all web content in near real time then evaluating for actionable insights in context.

In 1995 it was quite hard to speak to a public relations audience to get understanding that the internet was going to change PR practice. Not many in the industry waited with bated breath for the findings of the CIPR/PRCA internet Commission in 2000. Few practitioners believed the world wide web was more than a fad. Only a minority agreed these developments would change our profession forever. Fourteen years and three online PR books later it still remains challenging.

Asking readers of PRWeek to move to a point where you can begin to believe that technologies will mediate in PR practice is a big ask but that is where I believe we are going.

More Stuff

While we are looking at our online toys, it may be an idea to look at some of the others we have been looking at such as some tracking software like TrackThisNow.

David teaching about blog post

This is really hard!!! I've taken over David's blog so that he can show me how its done. Very confused!!!! We've been talking about Delicious and now we're going to link to it.

Experimenting with web monitoring

In this era of social media management, we are often asked to monitor everything online.

The drawback is normally that we have to monitor the world to find out stuff that really only applies to a few counties or even just one. In addition, there are now so many different channels for communication, that we end up with some pretty complicated RSS fees and searches that it has become very time consuming and especially so when monitoring for multiple issues, brands or names.

There is another way. We can monitor everything that is newly indexed by search engines.

They limit what can be found by area, number of citations and a whole host of other limitations or simply do not index enough of the web each day to offer reliable data and, worse still the volumes can be very high.


Over the next few days, I will be doing tests on the pre-alpha edition of new software that, I hope will resolve most of these problems.

It is a programme that, theoretically finds new pages about a search term each day in individual countries.

The returns will, of course, include new entries in all manner of web pages but the experience we are gaining is helpful for future development.

There is a case for monitoring all new citations to provide comprehensive intelligence about a brand which can then be drawn into the online conversation.

Here are the first few returns for a test I did today for Yahoo and they give some idea as to where we have reached so far.

NKorea off US blacklist after nuke inspection deal - Yahoo! News After North Korea relented on nuclear inspection demands, the US on Saturday erased from a terrorism blacklist the communist country President Bush once ...
Yahoo Launches Analytics : ISEdb.COM As its latest attack to Google's supremacy in the search engine business, Yahoo! recently launched its own Web Analytics tool, a Web application enabling ...
Icahn Once Again Trying To Throw Yahoo's Yang Out On Ear? A reader insists that Carl Icahn is polling Yahoo's institutional investors to gauge support for another go at a palace coup (specifically, throwing Jerry ...
YouTube offers full-length CBS shows : Gina Hughes : Yahoo! Tech We've told you about Joost, Hulu, and Veoh, but all these sites may soon be forgotten now that YouTube has signed a deal with CBS and will start offering ...
Yahoo Developer Network at Future of Web Apps London (Yahoo ... This is an account from the perspective of us as exhibitors - there'll be coverage of the talks attended by Yahoos - Rajat Pandit to be exact - later on. ...
Yahoo seen as declining, while Microsoft looms : Technology ... SEATTLE - When Yahoo Inc. co-founder and CEO Jerry Yang spurned Microsoft Corp.' s rich buyout offer this spring, he promised that brighter days in Sunnyvale ...
Gamble fuels Burton's victory - NASCAR - Yahoo! Sports Opting to take fuel only on his final pit stop helped Jeff Burton win the Bank of America 500 and put himself back in title contention. - NASCAR news.


We can refine this quite a lot already but are the findings comprehensive.

I hope to find out over the next few hours.

Helpful ideas and suggestions will be welcome.

Agnostic evaluation of media content

Image via Wikipedia

This weekend, I have been playing with a sentiment engine and thinking about people's perspectives.

This one does not take sides. It is agnostic. It decides if a press story is positive or negative from a neutral perspective.

All you do is past text into a box and it analyses the content as positive or negative and shows the structure of sentences that leads to this conclusion.

You can try it here.

The returns it makes is an academic minefield. It challenges your thinking about truth.

It has a first cousin, that can add perspectives. For example, it can make similar decisions from the perspective of the actors (company, person brand etc) which is why it was invented but the returns with no perspective are very interesting.

Sometimes this programme gets it wrong but not often.

It is able to glean content that is negative but which contributes to the positive side of the article and you can see how it does it.

I have tried news articles, book chapters, reports and even client presentations to see where the sentiment lies.

So where is the beef?

This kind of development is useful for analysing sentiment of news articles, blogs and other content, which is its primary purpose but it also has applications in evaluating style and and bias all of which are very useful to the PR industry, regulators and watchers of political sentinemt on and off line.

Try it out and be challenged.