Signals intelligence journalism: using public information websites to source stories

Useful information is more widely and easily available than ever and the increasing amount of online data released by the government and others can help improve the originality of journalists’ work.

Look to VentnorBlog – the hyperlocal online effort based in the Isle of Wight which Journalism.co.uk commended during the Vestas protest coverage – for some inspiration.

[For those unfamiliar with the story, locals had been protesting against the closure of the wind turbine factory in front of national, local and hyperlocal media. Despite a long and well-publicised campaign in August 2009, Danish company Vestas has now pulled out of manufacturing on the Isle of Wight but protests and attacks by critics in the press continue. A national day of action to support redundant Vestas workers has been planned for Thursday, September 17.]

Last week, using the Area Ship Traffic Website, AIS, VB was able to report where two barges held by an agent – NEG  Micron Rotors – who used to own the Vestas’ factory were due to head. They would be used to move the blades from the factory, which are so huge that they can only travel away on the water on special vessels.

The correspondent who tipped off VentnorBlog knew that the wind turbine blades can only be transferred from the riverside to barge when it is high tide and across a public footpath so, using the information on the AIS site, concluded that the barges would be moved in a specific time slot.

As a result Vestas protesters asked supporters to join them at the Marine Gate on the River Medina. Of course VentnorBlog got down there to take some pictures.

Now let’s take that one step further: how can journalists tap into this kind of publicly available data to scoop stories?

Tony Hirst, Open University academic, Isle of Wight resident and prolific data masher, shared some thoughts with Journalism.co.uk. He said that we should look to signals intelligence for further inspiration: the interception and analysis of ‘signals’ emitted by whoever you are surveying. As military historians would be the first to tell you, they can be a very rich source of intelligence about others’ actions and intentions, he explained.

“A major component of SIGINT is COMINT, or Communications Intelligence, which focuses on the communications between parties of interest. Even if communications are encrypted, Traffic Analysis, or the study of who’s talking to whom, how frequently, at what time of day, or  - historically – in advance of what sort of action, can be used to learn about the intentions of others.”

And this is relevant to journalists, he added:

“For starters, data is information, or raw intelligence. The job of the analyst, or the data journalist, is to identify signals in that information in order to identify something of meaning – ‘intelligence’ about intentions, or ‘evidence’ for a particular storyline.

The VentnorBlog story, he said, describes how a ‘sharp-eyed follower of movements at the plant’ knew where two barges were headed and at what time – valuable journalistic information:

“Amid the mess of Solent shipping information was a meaningful signal relating to the Vestas story – the movement of the barge that takes wind turbine blades from the Vestas factory on the Isle of Wight to the mainland.”

Do you have suggestions for sources of ‘signals intelligence’ journalism? Or examples of where it has been done well?

Similar posts:

• VentnorBlog shows us high-quality hyperlocal reporting with the Vestas story
• Treasury reaches out to hyperlocal sites for Budget coverage
• Local news blogger refused entry to coroner’s court
• NUJ general secretary calls on coroner to allow blogger into court
• European Journalism Centre aggregation project tops 500 mark

ReadWriteWeb: Journalism needs data

As Zach Beauvais points out in his post for the ReadWriteWeb, it’s not new that facts are crucial to journalism.

“But as we move further into the 21st century, we will have to increasingly rely on ‘data’ to feed our stories, to the point that ‘data-driven reporting’ becomes second nature to journalists.”

“The shift from facts to data is subtle and makes perfect sense. You could that say data are facts, with the difference that they can be computed, analyzed, and made use of in a more abstract way, especially by a computer.”

Full post at this link…

Journalism.co.uk is extremely interested in the #datajourn discussion.

Computer-assisted reporting is also nothing new, the use of data in journalism is not particularly radical, but new developments in technology, mindset, and accessibility mean that data-sets will have a new place in the profession.

Join the conversation and please get in touch with your thoughts: judith@journalism.co.uk.

Similar posts:

• #DataJourn part 3: Useful and recent links looking at use of data in journalism
• #DataJourn part 1: a new conversation (please re-tweet)
• Online Journalism Scandinavia: Waiting for the CAR to arrive
• Editor&Publisher: New AP regional investigative teams will boost CAR and data journalism
• Jonathan Stray: A computational journalism reading list

#Tip of the day from Journalism.co.uk – importing tables into a spreadsheet

Figures: Confronted with a table of figures on a website or attachment? The latest version of Microsoft Excel lets you import a table quickly and easily into a spreadsheet so you can do your own calculations. Tipster: Laura Oliver.

To submit a tip to Journalism.co.uk, use this link – we will pay a fiver for the best ones published.

Similar posts:

• #Tip of the day from Journalism.co.uk – simple graphs for stats stories using Google Docs
• #Tip of the day from Journalism.co.uk – embedding tables
• Tip of the day from Journalism.co.uk – creating a slideshow
• #Tip of the day from Journalism.co.uk – number crunching tools
• #Tip of the day from Journalism.co.uk – tools for online maps

#datajourn: Simon Willison’s ‘hack day’ tools for non-developers

The Guardian’s second (internal) hack day is imminent; the development team, members of the tech department and even journalists get together to play and build.

Read about the first one here. Remember this effort by guest hacker, Matthew Somerville: http://charlian.dracos.co.uk/?

In preparation for the second, Simon Willison (@simonw), the lead developer behind the Guardian’s MPs’ expenses crowdsourcing application, has helpfully put together an (external) list of tools for non-developers: “sites, services and software that could be used for hacking without programming knowledge as a pre-requisite. “

Full list at this link…

Similar posts:

• UK Future of News gets local
• Nieman Journalism Lab: Four crowdsourcing lessons from the Guardian’s expenses experiment
• Future of News meet-ups in Brighton and Birmingham
• NewsInnovation videos from @newsmatters: featuring @kevglobal, @currybet, @markng, @simonw, @willperrin
• Guardian.co.uk: Crowd-sourced experiment – ‘Investigate your MP’s expenses’

#Tip of the day from Journalism.co.uk – tools for playing with data

Data: Great round-up from Nathan Yau, author of the Flowing Data blog, for the Guardian on tools for journalists looking to make better use of numbers and ways to visualise figures. Tipster: Laura Oliver.

To submit a tip to Journalism.co.uk, use this link – we will pay a fiver for the best ones published.

Similar posts:

• #Tip of the day from Journalism.co.uk – data inspiration for stories
• #Tip of the day from Journalism.co.uk – Data Miner’s tools for data journalism
• #Tip of the day from Journalism.co.uk – visualising data
• #Tip of the day from Journalism.co.uk – easy guide to map visualisations
• Tip of the day from Journalism.co.uk – Aviation resources for investigative journalists

Nieman Journalism Lab: Four crowdsourcing lessons from the Guardian’s expenses experiment

A great post from the Nieman Journalism Lab, offering a US perspective on the Guardian’s feat with expenses data. The title says it all really: ‘Four crowdsourcing lessons from the Guardian’s (spectacular) expenses-scandal experiment’.

Full post at this link…

Similar posts:

• Guardian.co.uk: Crowd-sourced experiment – ‘Investigate your MP’s expenses’
• Malcolm Coles: MPs’ expenses – the best of the web
• MPs’ travel expenses disparity highlighted by Guardian Open Platform projects
• Jon Bernstein: What MPs’ expenses tells us about the clash between new and old media
• Guardian launches crowdsourced investigation into Tony Blair’s finances

Telegraph.co.uk: Guide to the full MP expenses database

Telegraph.co.uk has now published a searchable database of all MPs’ expenses. It reports:

“The searchable database will include exclusive documentary evidence as well as detailed figures assembled over recent weeks as part of an exhaustive investigation into Parliament’s expenses claims system.”

“In the coming weeks, the database will be extended to include the uncensored documentation for the claims, including receipts and correspondence with the Parlimentary authorities, of all MPs.”

Full guide at this link…

MPs’ expenses database at this link…

More to follow from Journalism.co.uk on users’ experiences later today. How have you found using the data provided by the Commons, the Guardian and the Telegraph? Drop judith at journalism.co.uk a line, or via Twitter to @jtownend.

Similar posts:

• Let the expenses data war commence: Telegraph begins its document drip feed
• Malcolm Coles: MPs’ expenses – the best of the web
• Anger over army equipment motivated MPs’ expenses leak
• Telegraph to publish ‘unredacted’ expenses information… in print
• Guardian.co.uk: Crowd-sourced experiment – ‘Investigate your MP’s expenses’

Let the expenses data war commence: Telegraph begins its document drip feed

Andy Dickinson from the Department of Journalism at UCLAN sums up today’s announcement in this tweet: ‘Telegraph to drip-publish MP expenses online’.

[Update #1: Editor of Telegraph.co.uk, Marcus Warren, responded like this: 'Drip-publish? The whole cabinet at once....that's a minor flood, I think']

Yes, let the data war commence. The Guardian yesterday released its ‘major crowdsourcing tool’ as reported by Journalism.co.uk at this link. As described by one of its developers, Simon Willison, on his own blog, the Guardian is ‘crowdsourcing the analysis of the 700,000+ scanned [official] MP expenses documents’. It’s the Guardian’s ‘first live Django-powered application’. It’s also the first time the news site has hosted something on Amazon EC2, he says. Within 90 minutes of launch, 1700 users had ‘audited’ its data, reported the editor of Guardian.co.uk, Janine Gibson.

The Telegraph was keeping mum, save a few teasing tweets from Telegraph.co.uk editor Marcus Warren. A version of its ‘uncensored’ data was coming, but they would not say what and how much.

Now we know a bit more. As well as printing its data in a print supplement with Saturday’s newspaper they will gradually release the information online. As yet, copies of claim forms have been published using Issuu software, underneath each cabinet member’s name. See David Miliband’s 2005-6 expenses here, for example. From the Telegraph’s announcement:

• “Complete records of expense claims made by every Cabinet minister have been published by The Telegraph for the first time.”
• “In the coming weeks the expense claims of every MP, searchable by name and constituency, will be published on this website.”
• “There will be weekly releases region by region and a full schedule will be published on Tuesday.”
• “Tomorrow [Saturday], the Daily Telegraph will publish a comprehensive 68-page supplement setting out a summary of the claims of every sitting MP.”

Details of what’s included but not included in the official data at this link.  “Sensitive information, such as precise home addresses, phone numbers and bank account details, has been removed from the files by the Telegraph’s expenses investigation team,” the Telegraph reports.

So who is winning in the data wars? Here’s what Paul Bradshaw had to say earlier this morning:

“We may see more stories, we may see interesting mashups, and this will give The Guardian an edge over the newspaper that bought the unredacted data – The Telegraph. When – or if – they release their data online, you can only hope the two sets of data will be easy to merge.”

Update #2: Finally, Martin Belam’s post on open and closed journalism (published Thursday 18th) ended like this:

“I think the Telegraph’s bunkered attitude to their scoop, and their insistence that they alone determined what was ‘in the public interest’ from the documents is a marked contrast to the approach taken by The Guardian. The Telegraph are physically publishing a selection of their data on Saturday, but there is, as yet, no sign of it being made online in machine readable format.

“Both are news organisations passionately committed to what they do, and both have a strategy that they believe will deliver their digital future. As I say, I have a massive admiration for the scoop that The Telegraph pulled off, and I’m a strong believer in media plurality. As we endlessly debate ‘the future of news™’ I think both approaches have a role to play in our media landscape. I don’t expect this to be the last time we end up debating the pros and cons of the ‘closed’ and ‘open’ approaches to data driven journalism.”

It has provoked an interesting comment from Ian Douglas, the Telegraph’s head of digital production.

“I think you’re missing the fundamental difference in source material. No publisher would have released the completely unredacted scans for crowdsourced investigation, there was far too much on there that could never be considered as being in the public interest and could be damaging to private individuals (contact details of people who work for the MPs, for example, or suppliers). The Guardian, good as their project is, is working solely with government-approved information.”

“Perhaps you’ll change your mind when you see the cabinet expenses in full on the Telegraph website today [Friday], and other resources to come.”

Related Journalism.co.uk links:

• ‘Has the Telegraph failed by keeping expenses process and data to itself?’ [15/06/09]
• ‘Telegraph ‘didn’t tell any lies but was selective in its facts’, says Lib Dem Voice site editor’ [11/06/09]
• ‘Telegraph to publish ‘unredacted’ information… in print‘ [18/06/09]
• ‘Guardian launches ‘major crowd-sourcing experiment’ with MPs’ expenses application’ [18/06/09]

Similar posts:

• MPs’ expenses data will be officially released Thursday but how much will be edited out?
• Telegraph.co.uk: Guide to the full MP expenses database
• Telegraph to publish ‘unredacted’ expenses information… in print
• Malcolm Coles: MPs’ expenses – the best of the web
• Heather Brooke and Telegraph named in PSA Awards

Guardian.co.uk: Crowd-sourced experiment – ‘Investigate your MP’s expenses’

The Guardian has launched a new crowd-sourced experiment: ‘Investigate your MP’s expenses’. More to follow from Journalism.co.uk soon.

Extracts from the Guardian press release:

“The Guardian has today launched a major experiment in crowdsourcing following the publication of thousands of MPs’ receipts by the House of Commons.

“The Guardian has uploaded all of these documents to its own microsite, Investigate your MP’s expenses, allowing members of the public to interact with and analyse the data; an impossibility on the government’s website.

“For every document for every MP, users of the site will be able to: add narrative on individual expenses; highlight documents of interest; tell us how interesting that receipt is and provide a context for each receipt; help us by entering the relevant expenses figures and dates on each page.”

Similar posts:

• Nieman Journalism Lab: Four crowdsourcing lessons from the Guardian’s expenses experiment
• Tool of the week for journalists – DocumentCloud, to analyse documents as data
• News organisations should get ready for data, says Martin Moore
• Let the expenses data war commence: Telegraph begins its document drip feed
• MPs’ travel expenses disparity highlighted by Guardian Open Platform projects

OUseful: Gripes with Guardian’s DataStore #datajourn

Here are thoughts from Tony Hirst, one of the first adopters and success stories for the Guardian’s Open Platform, on what the OP’s DataStore is and is not doing, in terms of data curation (or gardening). He asks:

“Is the Guardian DataStore adding value to the data in the data store in an accessibility sense: by reducing the need for data mungers to have to process the data, so that it can be used in a plug’n'play way by the statisticians and the data visualisers, whether they’re professionals, amateurs or good old Jo Public?”

Hirst has a number of queries in regards to data quality and ‘misleading’ linking on the Guardian DataBlog. In a later comment, he wonders whether there is a ‘data style guide’ available yet.

If you’re not all that au fait with the data lingo, this post might be a bit indigestible, so we’ll follow with a translation in coming days.

Related on Journalism.co.uk: Q&A with Hirst, April 8, 2009.

Similar posts:

• MPs’ travel expenses disparity highlighted by Guardian Open Platform projects
• #jpod – Open data and journalism: How and why journalists should be digging for stories
• #Tip of the day from Journalism.co.uk – using spreadsheets for data stories
• Tool of the week for journalists: Data.gov.uk’s map-based search
• #DataJourn part 1: a new conversation (please re-tweet)