The 8 Biggest Risks of Big Data Projects

The phenomenon of “Big Data” exacerbates the tension between potential benefits and privacy risks by upping the ante on both sides of the equation. Any project can fail for any number of reasons: bad management, poor budget management, or just a lack of relevant skills. However, big data projects bring their own specific risks.

… we have a small to ask. Hundreds of people are turning to Godinterest for open, independent, quality devotionals every day, and readers in over 50 countries around the world have subscribed to us. In these perilous times, a truth-seeking ministry like Godinterest is essential. We have no shareholders or billionaire owners, meaning our writing is free from commercial and political influence – this makes us different. When it’s never been more important, support Godinterest from as little as $1 – it only takes a minute. If you can, please consider supporting us with a regular amount each month. Thank you.

Published in MakerRecommend0 recommendations

By George Brzozowski

I want to help you to see news events as starting points for constructive conversations. I seek to cut through the froth of the political spin cycle to underlying truths and values. I want to be so focused on progress that together we can provide a credible and constructive counter-narrative to the hopelessness-, anger-, and fear-inducing brand of discourse that is so pervasive in the news.

3 replies on “The 8 Biggest Risks of Big Data Projects”

From a data perspective, the the techniques of data mining, Bayesian and BAGGed regressions haven’t changed too much but the technology that allows you to run bandsaw analytics is much more prolific and outsource-able even over ther past 3 years. The issues that remain constant in these pushes are havnig sound theory-driven approaches to structuring the analyses and as stated good, clean data. Even inconsistent data can be logged out and refined, but reliability of the base connection with inputs and realities is noisy and stubborn obstacle. In the end, Big Data efforts often fail most frequently if the leader (or lead group) has insufficient analytics skillset to guide the business partners through the parameters, the WIIFM, purveying analysis quality and attach the capablity of the results back to filling a knowledge gap for the business. If those are done well, analysis leads to insight, insight leads to action. If not, then quite the contrary.

In a sense, security and privacy are the easy problems, where technology can shine. The more difficult upfront problems are knowing what you are collecting and that it is consistent and well defined, especially when coming from multiple sources.

Then it gets trickier. As the author says, “Big data analysis is prone to errors, inaccuracies and bias” – exacerbated by any weaknesses in data collection, which are often invisible to the analyst. Analysis creep can occur: As more data are available, more analysis is possible, leading more data requested, then more data to analyze . . .

I am particularly interested in predictive uses and making sure that the predictions can be reviewed for accuracy (and then revised to improve future prediction).

Interesting post (as usual!).

I’m always a fan of KISS (keep it simple, stupid). I think the ease of collecting data and information can lead to grand plans, but, like the article said, will implementation always follow through or yield desired results? And if it’s improperly analyzed, will it yield an incorrect conclusion?

Interestingly, people think of big data and privacy issues as a recent phenomenon. My father worked for a newspaper in advertising and market research. Back in the 80’s and 90’s (and probably even earlier) it was already quite common. There was one data aggregator who tracked and compiled lots of data on people from a wide variety of sources. If you sent in a card when you purchased a product – that went into the data base. If you called the free movieline (or other free information number – sports, horses,) number in the newspaper, it went in the database (your phone number was liked to you). If you subscribed to a certain magazine – it went into the database. If you bought a car, it went in the database (new vehicle purchases were available from counties). They knew the income demographics of your neighborhood from the census. He said it was incredible how much information was collected and aggregated about individuals. The difference was that it was probably better controlled and couldn’t be hacked (maybe physically stolen, though). It was primarily used by and sold to those who advertised to people.

Data can be good and revealing though. My wife works at a major hospital system. In the annual employee survey, cleaning people complained that they often didn’t have the materials they needed in their closets and had to go to a central repository and restock them themselves. Their badges also often didn’t give them access to areas they needed to clean (even non-sensitive areas) and they had to go get a security guard and tie up themselves and the security guard for up to 20 minutes. When they did a study and tallied up all the wasted time, they found it was a staggering amount and costly. The problems were resolved almost immediately – closets were always fully stocked and badges had all the access they needed. That wouldn’t be an argument for collecting data from people’s badges to track them every minute of every day; it would, though, be an argument for targeted data studies to investigate and evaluate a specific problem.

Leave a Reply

Your email address will not be published.

Translate »
%d bloggers like this: