Don’t delete big data, companies urged


Posted By : admin Comments are off
dont delete big data companies urged
Categories :#AnalyticsNews

Companies performing ad-hoc big data analytics operations have been reminded of the importance of keeping the data used in the process after it is completed.

Speaking at an IT Leaders Forum organised by, director of file, object storage and big data flash at IBM Alex Chen explained businesses may need to refer back to this information at a later date. This may be in order to meet regulatory requirements, or simply if people want to investigate what happened and why a particular decision was taken.

At the moment, many organisations are still in the early adoption stage when it comes to big data, which means they may be performing a large number of experimental and ad-hoc analyses as they learn how to bring this technology into their everyday operations.

Mr Chen said: "It's likely that someone in a line-of-business [in many organisations] has spinned-up a Hadoop cluster and called it their big data analytics engine. They find a bunch of x86 servers with storage, and run HDFS."

Many people tend to throw away this data after it has been processed in order to keep their system running efficiently. Mr Chen noted that even in these ad-hoc deployments, it is not terabytes, but petabytes of data that are being ingested, and the more data that has to be analysed, the longer it will take.

But while deleting this data may keep analytics processes running as fast as possible, it could mean businesses have no answers when they need to demonstrate what led them to their final decision.

"Performing analytics generates a lot more meta-data, too, and due to regulations or business requirements people may just want to see what happened and why they made certain decisions. So you will need to re-run the analytics that were run before," Mr Chen continued. "So you can't just throw away the data any more."

Many firms still lacking big data strategy


Posted By : admin Comments are off
big data strategy
Categories :#AnalyticsNews

Big data continues to be high on the agenda for many businesses, but despite the growing recognition of its importance, a large number of firms still do not have an effective plan in place for making the most of the technology.

This is among the key findings of a new survey conducted by DNV GL – Business Assurance and GFK Eurisko, which polled nearly 1,200 professionals from across Europe, Asia and the Americas. It revealed that many expect big data to play a significant role in future operations, but here is still a long way to go for many enterprises before this can be achieved.

More than three-quarters of respondents (76 per cent) predicted that investments in big data technology will be maintained or increased in the coming years, while two-thirds (65 per cent) are planning for an environment where big data is a key part of their operations.

But even though 52 per cent of professionals agreed that big data presents a clear business opportunity, only 23 per cent have a clear strategy in place for embracing the technology.

DNV GL noted that in order to make big data analytics a success, companies should treat it as a new journey, and make preparations and changes to their existing processes accordingly.

For example, 28 per cent of respondents say they have improved their information management procedures in order to make the adoption of advanced analytics tools as smooth as possible, while 25 per cent have implemented new technologies and methods for handling data.

However, fewer companies have worked on changing their day-to-day activities. Just 16 per cent have made efforts to change the culture or organisation to reflect a more data-driven approach, while 15 per cent have changed their business model.

"Big data is changing the game in a number of industries, representing new opportunities and challenges," said Luca Crisciotti, chief executive of DNV GL – Business Assurance. "I believe that companies that recognise and implement strategies and plans to leverage the information in their data pools have increased opportunities to become more efficient and meet their market and stakeholders better."

The survey found that all companies that have already adopted big data analytics report clear benefits from their efforts. For example, 23 per cent stated they have seen increased efficiency, 16 per cent reported better business decision making and 11 per cent witnessed financial savings. 

Meanwhile, 16 per cent stated their customer experience and engagement has improved as a result of big data, while nine per cent reported better relations with other stakeholders.

However, there are several factors that are still holding many firms back when it comes to adopting big data. Chief among these are a failure to develop an overall strategy and a lack of technical skills, both of which were named as issues by 24 per cent of respondents.

Therefore, getting the right personnel on board will be critical in making big data a success. These individuals need to understand the intricacies of big data analytics technologies, as well as take a leading role in preparing the business for the era of data.

Mr Crisciotti said: "The ability to use data to obtain actionable knowledge and insights is inevitable for companies that want to keep growing and profiting. The data analyst or scientist will be crucial in most organisations in the near future."

White House warns on big data risks


Posted By : admin Comments are off
White House warns about big data risks
Categories :#AnalyticsNews

A new report on the big data analytics sector from the White House has warned businesses that they must consider the ethical implications of their deployments and ensure that they are not discriminating against any individuals through their use of data.

The study, titled "Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights" noted that if used correctly, big data can be an invaluable tool in overcoming longstanding biases and revisiting traditional assumptions.

For instance, by stripping out information such as race, national origin, religion, sexual and gender orientation, and disability, big data solutions have the potential to prevent discriminatory harm when it comes to activities such as offering employment, access to finance or admission to universities. However, the report warned that if care is not taken with the implementation of these technologies, they could exacerbate any problems.

One of the big challenges is that despite what many people assume, big data is not necessarily impartial. It can be subject to a range of issues such as imperfect inputs, poor logic and the inherent biases of the programmer.

"Predictors of success can become barriers to entry; careful marketing can be rooted in stereotype. Without deliberate care, these innovations can easily hardwire discrimination, reinforce bias, and mask opportunity," the study stated.

For instance, poorly selected data, incomplete or outdated details and unintentional historical biases could all result in the wrong data being input into big data systems. Meanwhile, poorly-designed algorithms can also cause problems if they assume correlation equals causation, or if personalised recommendations use too narrow a criteria to infer a user's true preferences.

The report highlighted several case studies that illustrate how big data can be used to improve outcomes – as well as some of the pitfalls that need to be avoided.

For example, it noted that many people in the US have difficulty gaining access to finance because they have limited or non-existent credit files. This is an issue that particularly affects African-American and Latino individuals, who are nearly twice as likely to be 'credit invisible' than whites.

Big data presents a great opportunity to improve access to credit, as it can draw on many more sources of information in order to build a picture of an applicant. This may range from phone bills, previous addresses and tax records to less conventional sources, such as location data derived from use of cellphones, social media data and even how quickly an individual scrolls through a personal finance website.

However, it warned: "While such a tool might expand access to credit for those underserved by the traditional market, it could also function to reinforce disparities that already exist among those whose social networks are, like them, largely disconnected from everyday lending.

"If poorly implemented, algorithmic systems that utilise new scoring products to connect targeted marketing of credit opportunities with individual credit determinations could produce discriminatory harms." 

The report also included a number of recommendations for improving big data outcomes, such as increasing investments in research, improving training programmes, and developing clear standards for both the public and private sector.

"Big data is here to stay; the question is how it will be used: to advance civil rights and opportunity, or to undermine them," it added.

Retail banks turn to big data to regain customer trust


Posted By : admin Comments are off
Retail banks turn to big data to regain customer trust
Categories :#AnalyticsNews

For many retail banks, the task of regaining consumer trust in the wake of the financial crisis of 2008-09 will be a difficult and ongoing challenge. With the sector still viewed with suspicion by many people, presenting a more personal face and improving customer service levels will be a high priority.

It was noted by FusionExperience chief executive Steve Edkins in an article for ITProPortal that this has become even more important in today's connected era, where the internet and social media mean dissatisfied customers are able to quickly voice any complaints to a wide audience.

In order to improve their customer service and avoid such issues, many retail banks are therefore turning to big data to offer services tailored to individual customers.

According to a study from the Centre for Economics and Business Research (Cebr), more than four-fifths of retail banks (81 per cent) will have adopted big data analytics by 2020. As well as helping track key industry trends and allowing banks to proactively adapt their strategy, this will also have a key role to play in building profiles of individual customers.

This can be useful at every stage of the customer journey. Mr Edkins noted that initially, big data analytics can be used to more effectively evaluate risk and creditworthiness. Then, when it comes to retaining customers, offering specific deals and tailoring their services accordingly will go a long way towards making consumers feel valued.

However, financial institutions will face two key challenges when it comes to adding big data to their customer service activities. The first will be how they extract relevant information from the huge amount of data they collect – separating the signal from the noise in order to make informed decisions.

The second will be how they collate this data and turn it into a useable format in time to make a difference. Today's fast-paced world demands the ability to extract, analyse and act on insights gained from data quickly if a company wants to maintain a competitive advantage.

"It is no small feat for retail banks to ingratiate big data into their processes as it often requires a daunting technological overhaul," Mr Edkins said, adding that one of the biggest challenges for these firms is getting complex legacy systems in line with today's big data capabilities. These often result in key data being placed in silos, and make it difficult for businesses to get the information they need quickly.

"To rectify this, banks will need to make better use of growing data sets such as correspondence, loan facility letters, contracts and the diversity of customer interactions if they want to offer bespoke consumer products that will allow them to fend off their more agile competitors," he stated.

However, if retail banks can get this right and build a strong customer service culture centred around big data, the rewards on offer are significant. Cebr's data forecasts that effective analysis of data could add £240 billion to the UK's economy through improved efficiency and better understanding of the market and customer demands.

EU set to analyse big data use in mergers


Posted By : admin Comments are off
EU set to analyse big data use in mergers
Categories :#AnalyticsNews

The European Union (EU) is planning to take a more detailed look at whether the large amounts of consumer data carried by large internet companies violates competition rules as part of its analysis of proposed mergers.

After taking over as Europe's leading antitrust enforcer in 2014, Margrethe Vestager has enhanced the investigations into US web giants including Google and Amazon in order to determine whether this information should be regulated more strictly.

Ms Vestager noted that protecting consumer privacy transcends her agency's remit, adding that the power large internet companies wield should not make it too challenging for other companies to compete.

This move comes after the European Commission suggested that Google favoured its own shopping services in search results ahead of rivals and is considering potential sanctions against the search engine.

Speaking to a conference of top European and US entrepreneurs and investors, Ms Vestager explained: "If just a few companies control the data you need to satisfy customers and cut costs, then you can give them the power to just drive rivals out of the market.

"If we analyse a merger, if we have a suspicion or concern when it comes to antitrust, if it comes to data, of course we will look at it. If a company's use of data is so bad for competition that it outweighs the benefit, then you may have to step in to restore the level playing field."

Ms Vestager embraced efforts by businesses to develop standards for internet data protection and emphasised the advantages noticed in businesses analysing big data to identify consumer demand to offer personalised shopping and organise better transportation.

She noted that the EU is planning to publish a wide-ranging policy paper on online services by the middle of the year.

So you’ve got Hadoop – what are the next steps?


Posted By : Paul Groom Comments are off
Categories :Guides

Big data delivers benefits after 18 to 24 months

Big data delivers benefits after 18 to 24 months[/caption]Of all the technologies currently competing for attention in the big data analytics sector, one solution that no business can afford to ignore is Hadoop. Even though this platform is still in its relative infancy, projections for the future are highly optimistic.

Indeed, it was forecast by Forrester Research analyst Mike Gualtieri that as Hadoop continues to disrupt established ways of running analytics operations, it will become the only viable option for many users.

Speaking at the 2015 Hadoop Summit in San Jose, California, he said: “It’s a data operating system and a fundamental data platform that in the next couple of years 100 per cent of large companies will adopt.”

However, there’s a world of difference between adopting a solution and being able to make the most of it. While many companies may be driven to explore Hadoop as a result of the hype surrounding it, relatively few understand exactly how they will leverage the solution to improve their business once it is established.

More than just a storage solution

One of the biggest reasons why Hadoop deployments fail is because businesses do not use them to their full potential. In fact, in many cases, Hadoop is simply used as a cheap storage solution in which companies can dump all their data, without really considering what they do with this resource; the processing potential of Hadoop and its ecosystem is undervalued or mis-understood.

The fact that Hadoop offers a highly cost-effective way of storing data is only one of its key benefits, but it can lead to businesses failing to treat it as the powerful analytics platform it is capable of being. Coupled with the sometimes steep learning curve for the technology and its many components, it’s easy to see why companies fail to take full advantage of its potential.

The result of this is that instead of a useful ‘data lake’, where all of a business’ digital assets are easily available for continual analysis, companies end up with a ‘data attic’, in which lots of data is just parked and then forgotten about for many months. In these cases, by the time data scientists return to these attics, they will struggle to achieve timely value.

A clear plan

To avoid this, it’s vital that companies engage with their data as soon as possible. Even if they are not yet ready for running full analytics operations, encouraging users to pay close attention to the information they are inputting into their system has clear near-term benefits.

Therefore, it’s important that businesses don’t approach their Hadoop deployments with an attitude that sees them put all their data into the tools first, and figuring out what to do with it later. In order to be successful, a clear path to results will be needed, so users at all stages understand what the end-goal is and what steps will need to be taken along the way to achieve this.

If businesses don’t have such plans in place from the start, and instead treat their Hadoop as a data attic, then when they do eventually come back and look at big data analytics, they’re likely to find a sprawling mess of disparate data that requires a lot of work to convert it into useful insights.

Realising the true value

One of the best ways to prevent this is to ensure your business takes the time to assess its data estate for potential value right from the start. Instead of simply shovelling every scrap of raw data they collect into a Hadoop storage solution, companies need to effectively ‘triage’ their information base to determine whether or not elements will enhance or distract from the collective value.

Things to be considering at this stage include the quality of the data – how likely is it to be complete, clean and accurate – and how relevant you expect it to be for future use. It’s all too easy to just add data under the assumption that these are concerns to be thought about later, but doing this just creates potential clutter. To derive true value from your big data analytics, you need to plan carefully and appreciate that Hadoop needs to be much more than just a low-cost place to store your growing volumes of data.