It was the information revolution discussed at a Silicon Valley Comes to Oxford event in 2003 that spurred a 23-year-old student to quit his MPhil in Medieval Literature at Oxford University and move to San Francisco.

Bob Goodson (University College, 2002) flew to Palo Alto at the invitation of PayPal co-founder Max Levchin, who took him out to lunch with co-founder Peter Thiel, the first outside investor in Facebook. They persuaded him to drop out – and he did, becoming the first employee at Yelp, now worth $1.5bn. Four years later he co-founded Quid with Sean Gourley (Balliol College, 2002).

“I realised we were going through an information revolution that we had not seen for 6,000 years and that was enough to modify my life plan,” explains Goodson, 32, at this year’s Silicon Valley Comes to Oxford event, held at the Said Business School. Quid now has seven clients, including Microsoft and Visa. They pay the company $1m per question to gain business insights from Quid’s data-driven analysis. Quid employs 44 people and has raised $17m from investors. Goodson says they “tease out the underlying intelligence from millions of documents” to, for example, predict emerging technology trends.

This is all based around a phenomenon known as big Big Data: a concept that, at its simplest, is about collecting, storing and analysing large volumes of data. In recent years it's gone from theory to practice, now made possible by the ubiquity and reduced cost of online storage and sensors. The explosion has been driven too by the number of devices that extract data from all of us – from mobile phones and wireless networking, to browser histories and social networking. Now, companies are scrabbling to make sense of all the data they can gather.

Although it has been compared to the gold rush, Goodson compares Big Data to oil mining. “It’s like finding a new oil field and everyone has to find out the best way to tap into it. It’s about aggregating insights,” he says. Michael Chui, Principal of McKinsey Global Institute, adds that “it is an emerging technology lots of venture capitalists are investing in.” Companies that make decisions based on data analysis will win in the marketplace in the future, Chui says.

But one of the challenges the sector faces is a shortage of analytical talent. By 2018, the United States will face a shortage of 150,000 actuaries and statisticians and of 1.5 million Big Data savvy business managers, financial analysts and engineers, says Chui. Add to that challenges such as violations of privacy, IT security, intellectual property and the headaches thrown up by managing often-erroneous unstructured data, and while the future is promising, it's far from straightforward.

That doesn't seem to be fazing the Oxonians working in the sector, though. “Consumers create a trail of activities – what they search and click on, location data and so on – known as data exhaust,” explains Tyler Bell (The Queen's College, 1993), Director of Product at Factual. At Factual all the data is chucked into what he describes as an “intelligent crucible where the logic happens”. Using online records his company maps millions of businesses in the US, in order to discover the kinds of people that frequent the particular areas and the types of businesses nearby. His clients, including mobile phone and financial services companies, lap up the results.

All of this may sound purely commercial, but elsewhere Big Data is being used to different ends. Washington-headquartered technology consulting firm Aristotle, for instance, aggregates data on voters from various public and commercial sources and puts together demographic profiles which are then used by political parties. Its technology was used in the recent US presidential elections and by the PDP Party in Tunisia in 2011– the first democratic elections held in the country following the Arab Spring uprising. It's also worth noting that the now-famous American statistician Nate Silver, who correctly predicted the results of the last US presidential elections, found notoriety using not intuition, but Big Data.

Elsewhere, it’s been used to coordinate the disaster response to Hurricane Sandy in New York, according to Lieutenant General Jeffery Talley (Keble College, 2010), chief of the US Army Reserve, as well as being used for earthquake predictions, disaster response, and disease prediction, too. Big Data, then, might be the Next Big Thing in the technology industry – but that doesn't means it's limited to corporations who want to know more about their customers. Instead, it's a vital new way of thinking, in a world where information pervades society like never before.