A Computer Weekly buyer’s guide to business advantages of big data

a whitepaper from Computer Weekly

In this nine-page buyer’s guide to big data, Computer Weekly discovers how businesses can use the present explosion of data to their

advantage. Big data may appear to be just another bandwagon, but it is important, and needs to be addressed carefully and sensibly. The journey can be carried out at a measured pace, leveraging existing systems in conjunction with new systems. It just needs a strategic plan built from careful planning – and an eye to the long-term future.

Contents

How to make sense of the big data universe page 2

The prospect of analysing big data can appear daunting, so here Clive Longbottom looks at what data you need, where to find it and how to break it down into manageable, meaningful chunks

Make better decisions with big data page 4

Rob Toguri considers the issues CIOs need to address to ask direct questions of big data and get meaningful answers

Utilities rise to the smart meter data challenge page 6

The mass of information from smart meters is leading utility suppliers to reconsider how they use their data. Lindsay Clark reports

Data’s role in customer engagement page 8

As firms face an influx of structured and unstructured data, a combination of traditional business intelligence and big data approaches will provide the best results in managing and making use of the information, writes Sanchit Gogia

These articles were originally published in the Computer Weekly ezine.

1

buyer’s guide

CW Buyer’s guideBig data

© S

er

ge

j K

ha

cK

imu

llin

/Fo

toli

a.c

om

-2-

buyer’s guide

as is the way with IT, as soon as one bandwagon begins to be understood by the general public, another one

has to be rolled out. In this case, as cloud computing starts to become more of a reality, big data is rearing its head as – depending on the commen-tator – the next greatest opportunity or threat to the organisation.

As there was with cloud, there’s a lot of confusion out about big data. Many of the database vendors tried to

ployed to monitor their brands – pushed the idea that big data was moving towards the field of social networking. They said big data was all about using the wisdom of the crowd and identifying the sentiment of the masses.

But social networking has not usurped much that went before, so any solution still has to include all the information feeds such as e-mail, call recordings, customer relation-ship management (CRM) records, scanned documents and so on.

All the approaches cover some as-pect of big data, but they all miss the point as well. The best, simple defini-tion of big data comes down to vol-ume, velocity and variety.

The volume aspect of big data is

actually the one that is the least im-portant. Big data is not about peta-bytes of data – it can be down to rela-tively small volumes that need to be dealt with in a manner that requires a big-data approach.

However, for most organisations, big data will involve bringing togeth-er many different data and informa-tion sources which, by their nature, will tend to result in the overall amount of data under consideration being big. Therefore, volume is not something that is under the direct control of the organisation – what has to be considered is how the volume of data that ends up being analysed is minimised, (more on this later).

Again, the velocity aspect of big data may well be a moot point – everyone

play big data as purely having a lot of data in one or more databases. But that is not big data, it’s large data – a problem that can be handled with da-tabase federation, standard business intelligence and analytics.

Next, it was said to be a mix of data held in the organisation that needed to be brought together so decision makers could see everything the or-ganisation held around a specific topic to make better informed deci-sions – but only through whatever in-formation the organisation was al-ready aware of. So if the organisation wasn’t already aware of something, that was to be excluded from the re-sults – see the problem here?

Many technology companies – aided by the PR organisations em-

CW Buyer’s guideBig data

How to make sense of the big data universe

The prospect of analysing big data can appear daunting, so here Clive Longbottom looks at what data you need, where to find it and how to break it down into manageable, meaningful chunks

ima

ge

: hu

bb

leS

ite

.or

g

-3-

buyer’s guidewants results against their analysis of available data in as short a period as possible. However, everything is rela-tive – for example, every millisecond added to providing results to a finan-cial market trader can cost millions of pounds, whereas someone tracking variations in the global movement of tectonic plates may not be that wor-ried if results take a few seconds to come through.

The one aspect that really matters is the variety of the information. Big data is all about the mix of data and where it is held at any time. Here, for-mal databases under the organisa-tion’s direct control are only a very small component of the overall mix. There are all the office documents held as files across the organisation and you may need to include voice and video files as well.

Then there’s the information held in the value chain of suppliers and customers – information that is criti-cal to the process or service being provided, yet isn’t under the organi-sation’s direct control. Then, there may well be a requirement to include information from the various social networks out there – and whatever approach is taken has to be inclusive.

Inclusivity of data sourcesFor example, it is pointless construct-ing something that is Facebook-spe-cific, if most comments are appearing as hashtags in Twitter.

Further, it’s a waste of time writing multiple connectors to cover all of to-day’s social networks – remember MySpace, Bebo and Second Life? They were all the darlings of their day, but have faded to a withered existence as newer players have taken over.

Sites such as Pinterest are showing signs of major interest – yet this was also the case with Google+, which more resembles a Western desert than a viable, active social network, after just a short time.

Any social network solution has to be able to embrace new platforms at

minimal cost, so new networks that are just “spikes” on the continuum do not use up lots of money in creat-ing connectors specifically for them.

Even the largest organisations will have little control over anything be-yond a small percentage of the total available data. The two-edged sword of the internet raises its ugly head in that it does provide massive extra in-formation resources – but then again, it also includes a massive amount of dross that doesn’t add anything to the sum knowledge of an organisation.

So how are we to deal with this real big data challenge, without run-ning into Dilbert’s pointy-haired boss’s dictat, “Just run me off a copy of the internet”?

Storage and structureStorage needs must be fully consid-ered. EMC, NetApp and Dell are now talking about object, block and file storage, rather than focusing purely on high-performance database object storage to cover the various types of big data that needs to be controlled.

Other storage vendors, such as Nutanix, Coraid, Amplidata and FusionIO provide systems that focus on one aspect of big data, partnering where necessary to cover others.

The need for structure around semi- or unstructured data is leading to an explosion in interest in noSQL-based databases, such as Apache Cas-sandra, 10gen MongoDB, CouchDB and so on. Systems such as Apache’s Hadoop, (which enables a massively scaled-out platform for providing dis-tributed processing for large amounts of data), can use MapReduce, (the use of “chunking” data analysis into packets of work that can be dealt with in a parallel manner across a large resource pool), approaches to minimise the amount of information that needs to be dealt with.

What is being aimed for here is to take the seemingly infinite amount of available data and filter it down into manageable chunks. Standard inter-

net searches can feed into a Hadoop-based system, which can then act as a feed into either standard SQL-based database or into a noSQL-based one, depending on the type of information being dealt with.

Extra information can be added au-tomatically via rules engines or man-ually, as required, as metadata that adds to the value of the information stored. Once the information is held in a recognised form, it is then down to being able to apply the right form of data analysis against it to provide suitable feeds to the decision maker.

This is where the main problems still reside, but much work is being carried out. Unsurprisingly, a lot of this is coming from the incumbent business intelligence suppliers, such as SAS Institute, QlikTech, JasperSoft as well as those who have gained entry to the market through acquisi-tion such as IBM (Cognos, SPSS), SAP (Business Objects) and Oracle (Hyperion, Endeca).

The storage suppliers are also mak-ing plays in the space – EMC ac-

quired GreenPlum and Dell contin-ues to acquire companies that will help it create a more cohesive and complete big data approach.

Buyer dos and don’tsThe key for buyers is to treat big data as a journey. Set short- and medium-term targets of what is required and then put in place solutions that help to move towards these targets.

Don’t put in place anything that could result in a need for major fork-lift upgrades at a later date – embrace open standards, look for suppliers who espouse heterogeneity in storage systems and in tooling, as well as an approach that covers a hybrid mix of private and public clouds.

Don’t fall for any supplier who says that the world is moving to or from “standard” SQL-based databas-es – the move is to a mixed environ-ment of a Hadoop-style system paired with SQL and noSQL-based systems. Look for business analytics packages that enable links to be made to data sources of any kind that reside anywhere on the internet, and that can link into semi-structured systems such as social networking sites in a meaningful manner.

Big data may appear to be just an-other bandwagon at this stage – but it is important, and needs to be ad-dressed carefully and sensibly, rather than in a bull-in-a-china-shop man-ner that seems to be pushed by many vendors. The journey can be carried out at a measured pace, leveraging existing systems in conjunction with new systems. It just needs a strategic plan built from careful planning – and an eye to the long-term future. ■

clive longbottom is a director of analyst organisation Quocirca

ima

ge

: hu

bb

leS

ite

.or

g

The key for buyers is to treat big data as a journey. Set short

and medium-term targets of what is required

ima

ge

: hu

bb

leS

ite

.or

g

-4-

buyer’s guide

There is near-consensus across industries as to which big data sets are most valu-able. The Economist Intel-

ligence Unit conducted a survey, completed in February 2012, of 607 executives. Participants hailed from across the globe, with 38% based in Europe, 28% in North America, 25% in Asia-Pacific and the remainder coming from Latin America and the Middle East and Africa.

Fully 69% of survey respondents agree business activity data (for ex-ample, sales, purchases, costs) adds the greatest value to their organisa-tion. The only notable exception is

is a quick way to identify shifting views towards drugs and other healthcare products.

Social mediaOver 40% of respondents agree that using social media data for decision-making has become increasingly important, possibly because they have made organisations vulnerable to “brand damage”.

Social media are often used as an early warning system to alert firms when customers are turning against them. In December 2011 it took Veri-zon Wireless just one day to make the decision to withdraw a $2 “conveni-ence charge” for paying bills with a smartphone, following a social me-dia-led consumer backlash.

Customers used Twitter and other social media to express their anger at the charge. Verizon Wireless was prompt in responding to the outcry,

Make better decisions with big dataRob Toguri considers the issues CIOs need to address to ask direct questions of big data and get meaningful answers

possibly forestalling customer defec-tion to rival mobile operators.

But not all unstructured data is as easy to understand as social media. Indeed, 42% of survey respondents say unstructured content – which in-cludes audio, video, e-mails and web pages – is too difficult to interpret.

A possible reason for this is that to-day’s business intelligence tools are good at aggregating and analysing structured data while tools for un-structured data are predominantly targeted at providing access to indi-vidual documents (for example, search and content management). It may be a while before the more ad-vanced unstructured data tools – such as text analytics and sentiment analysis – which can aggregate and summarise unstructured content, be-come mass market.

This could be why 40% of re-spondents say they have too much

consumer goods and retail, where point-of-sale data is deemed the most important (cited by 71% of respond-ents). Retailers and consumer goods firms are arguably under more pres-sure than other industries to keep their prices competitive. With smart-phone apps such as RedLaser and Amazon’s Price Check, customers can scan a product’s barcode in-store and immediately find out if the prod-uct is available elsewhere for less.

Office documentation (e-mails, document stores, etc) is the second most valued data set overall, fa-voured by 32% of respondents. Of the other major industries in the sur-vey, only healthcare, pharmaceuti-cals and biotechnology differ on their second choice. Here social media are viewed as the second most valuable data set, possibly because reputation is vitally important in this sector and “sentiment analysis” of social media

CW Buyer’s guideBig data

-5-

buyer’s guide

unstructured data to support deci-sion-making, next to just 7% who say they have too much structured data.

Structured or unstructured, most executives feel they don’t have enough data to support their deci-sion-making. In fact, 40% of respond-ents overall believe the decisions they have made in the past three years would have been significantly better if they’d had all of the struc-tured and unstructured data they needed to make their decision.

And, despite the fact that respondents from the financial services and energy sectors are more likely than average to describe their firm as data-driven, they are also more likely than the average (46% from financial services, and 48% from energy) to feel they could have made better decisions if the needed data was to hand.

On first appearance, this may seem contradictory, given the surfeit of data and the difficulty organisations face in managing it, but Bill Ruh, vice-president of software at GE sees no contradiction.

“Because the problems we address are going to get more and more com-plex, we’re going to solve more com-plex problems as a result,” he says.

“What we find is the more data we have, the more we get innovation in those analytics and we begin to do things we didn’t think we could do.”

Decision automationFor Ruh, the journey to data fulfil-ment will be over when he can put a sensor on every component GE sells and monitor it in real time.

In this way, any aberrant behaviour can be immediately identified and ei-ther corrected through a control mechanism (decision automation) or through human intervention (deci-sion support).

“We’re trying to get to what we would call ‘zero unplanned outages’ on everything we sell,” says Ruh.

Across all industries and regions, most survey respondents concur there is scope for further decision au-tomation at their organisation. Over 60% of respondents dispute the proposition that most operational/tactical decisions that can be auto-mated, have been automated.

This view is fairly consistent across industries, although fewer healthcare and pharmaceuticals com-panies agree with the statement (52%) than manufacturing compa-nies (68%). (Respondents from the education sector also appear less cer-tain than peers elsewhere that there is much still to be automated.)

There is some regional variation, too. No more than 54% of executives in Asia-Pacific believe the job of au-tomation is incomplete, compared with 71% in Western Europe.

Ruh says: “One reason is that many of the environments we operate in are highly regulated, so we have to move at a speed that makes sense within the regulation,” he says.

“The second is because the sensors and the data weren’t really there to automate anything.”

Certainly decision-automation tools have evolved from simple “if then else” programmable statements (for example: “If credit rating = AAA, then approve loan, else reject”) to so-phisticated artificial intelligence pro-grams that learn from successes and failures. The more sophisticated the tools become, the more decisions that can be automated.

Decision automation, however, can introduce unnecessary rigidity into business processes. At times of high instability – such as the current eco-nomic climate – companies need to be nimble to adapt to changing con-

ditions. Hard-coded decisions can be costly and take time to change.

Respondents’ reports of the bene-fits of harnessing big data for deci-sion-making are many and varied. The road to these riches, however, is laced with potholes.

The biggest impediment to effec-tive decision-making using big data – cited by 56% of survey respondents – is “organisational silos”. This appears especially the case for larger organisations with annual revenue over $10bn, whose executives are more likely to cite silos as a problem (72%) than smaller firms with less than $500m in revenue (43%).

Mind the skills gapA big impediment to making better decisions with big data is the dearth of skilled staff to analyse it, an issue mentioned by 51% of respondents.

For consumer goods and retail firms, it is the single toughest obsta-cle, cited by two-thirds of respond-ents from those sectors.

Some experts Capgemini spoke to believe that, in terms of modelling, there will be a considerable shortage of specialists, especially in the ana-lyst domain.

According to Ruh of GE: “There is going to be a war for this kind of tal-ent in the next five years.”

Aside from a master’s degree or PhD in economics, mathematics, physics or other relevant field of sci-ence, analysts are expected to have in-depth domain knowledge, which usually takes years to acquire.

Interviewees for the report say the ideal analyst should have an ability to communicate complex ideas in a simple manner and should be cus-tomer-focused. Finding people with all of these abilities is never going to be easy and retaining them is going to be even harder, as the benefits of big

data become apparent to more firms. Technology companies recognise

the problem and are working with schools and universities to develop these much-needed skills. For exam-ple, SAS, a business analytics soft-ware firm based in Cary, North Caro-lina, developed Curriculum Pathways, a web-based tool for teach-ing data analytics to high school stu-dents. The course, aimed at science, technology, engineering and mathe-matics students, has been running for 12 years in the US and is used in 18,000 schools; it will be offered to UK schools, for free, from March 2012. SAS has also developed ad-vanced analytics courses with a num-ber of universities, including Centen-nial College, Canada, North Carolina State University and Saint Joseph’s University, Philadelphia, to provide the next generation of data analysts.

The time it takes to analyse large data sets is seen as another major im-pediment to more effective use of big data in decision-making.

“I think big data is going to stimu-late the need for more CPU power, because people are going to get very creative and they’re going to invent new algorithms, and we’re going to say ‘My God, everything’s slow again’,” says Ruh of GE.

“We are going to have to redo our compute and storage architectures, because they will not work where all this is going.”

The future of big dataAlex Pentland, director of the Human Dynamics Laboratory at MIT, says big data is turning the process of decision-making inside out. Instead of starting with a question or hypoth-esis, people mine data to see what patterns they can find. If the patterns reveal a business opportunity or a threat, then a decision is made about how to act on the information.

This is certainly true, but improve-ments in computing power and artifi-cial intelligence systems mean that asking direct questions of big data and getting an answer, in real time, is now a reality.

Although these systems are still very costly and not widely deployed, this research suggests that the appe-tite for real-time decision-making is huge. And when there is a business demand, it is only a matter of time before the need if fulfilled. ■

› rob toguri is vice-president of business information management at capgemini

› this is an edited excerpt. click here to read the full report

› click here to see the full the computer Weekly research library archive

-6-

buyer’s guide

In March 2011, the UK govern-ment fired the starting pistol in what is set to become a race to transform data management in

the utilities sector. The Department of Energy and Climate Change mapped out plans for the installation of 53 million smart meters in 30 million homes and businesses by 2020.

Smart meters offer households and businesses the chance to understand

that’s 17,500 meter readings a year. When we talk about ‘exponential’ growth of data, that is really expo-nential. And that is just considering the billing side,” Ravens says.

As such, utilities at first found scal-ing smart meter management systems difficult, he says. As not every meter message gets through to the back-end server, data validation can place a substantial I/O load on servers, Ra-vens says.

Forecasting energy usageThe ultimate goal of smart metering is to allow utility firms to forecast energy usage, to improve their per-formance on the settlement markets –

where money can be lost through in-accurate predictions – and to match supply and demand more closely.

The utilities are not new to analyt-ics and forecasting. They have been used on the operations and mainte-nance side of the industry for years.

Yorkshire Water, a water supplier and waste management firm which supplies around 1.4 billion litres of water each day, has been using ana-lytics to forecast flooding and pollu-tion in its network of 62,000 miles of water and sewerage mains.

Yorkshire Water senior IT profes-sional John Samson has helped im-plement the systems which enable water engineers to predict when

and reduce their energy usage in much greater detail than previously possible, when meter readings were taken once a quarter, or even annual-ly. They promise to help utility firms improve the accuracy of billing and cut visits to properties to read meters.

Given that utility firms have com-monly struggled with the accuracy of their customer and billing data, the prospect of mass smart metering could present a considerable chal-lenge, said Stuart Ravens, principal analyst, energy and sustainability technology with Ovum.

“At the moment we could have a meter reading every year. With smart meters, if they read every half hour,

CW Buyer’s guideBig data

Utilities rise to the smart meter data challengeThe mass of information from smart meters is leading utility suppliers to reconsider how they use their data, reports Lindsay Clark

-7-

buyer’s guide

Since 2007, the Isle of Wight, off England’s south coast, has set itself the goal of becoming carbon neutral and self-sufficient in energy by 2020.

Plans are afoot to harvest energy from the wind, tidal flow, the sun and various forms of waste. Analytics software is vital to the effort to sever the island’s depen-dence on energy from the mainland.

Working with IBM and energy utility SSE, Eco Island, the group behind the scheme, says it will be able to monitor the island’s energy usage at an appliance and household level. This data can be fed back to tenants or home owners, but is also aggregated using the Message Queue Telemetry Transport (MQTT) protocol developed by IBM, as well as Ethernet or wireless connections, as part of the scheme.

Using analytics software, the aim is to model the island’s energy consumption to predict peaks and troughs in demands. This can be combined with the likely supply from the various planned sources of renewable energy, in part using Met Office forecasts.

“In the framework of these algorithms, we will be able

to learn the pattern of the island’s behaviours, and hopefully be able to make predictive decisions,” says David Green, Eco Island chief executive and founder.

“We are trying to move demand side around so that it fits better with generation and supply.

Part of the problem with these new sources – such as photovoltaic or

wind – is that they tend to offer the most energy at inefficient times,” Green says.

By being able to forecast the demand cycle as well as the likely supply of energy, the island hopes to make most efficient use of hydrogen energy storage technologies and encourage

businesses and households to alter their consumption patterns. IBM plans to create an intelli-

gence operations centre using Websphere messaging servers, DB2

databases, Cognos analytics software and an Informix database.

“It’s a first-of-its-kind smart grid blueprint,” Green says. “This will allow the island to make smarter use of its energy by using advanced technology to harness renewables with the aim of energy self-sufficiency.”

Plans for self-sufficiency on the Isle of Wight by 2020damaging pollution is likely to take place in the sewerage overflow. Com-bined sewer overflows are designed to allow water to escape during times of heavy rainfall, when pollutants are diluted. Discharge during dry weath-er can be more serious.

Data mining tool IBM SPSS Model-er provides users with daily spread-sheets which forecast the overflow sites likely to develop blockages. Op-erators can then better target engi-neers’ visits.

“This is an extremely simple appli-cation and I think that is the strength of it – there is no complex user inter-face they have to worry about. They just pick up their spreadsheet and apply their own local knowledge and experience,” Samson says.

The system is now being devel-oped to incorporate Met Office rain-fall data in near real time, using an OSI Pi interface. This allows the sys-tem to build models which apply in wet and dry weather.

“We are going to turn the dry weather model into an all-weather model that will double its useful-ness,” Samson says.

Benefits of integrating dataOther examples of the successful applications of data analytics in the sector come from the customer-facing side of the business. EDF Energy has been using a SAS data management and predictive analytics platform to cut the number of customers leaving the utility firm (see panel, right).

While customer-facing and opera-tional data systems remain distinct, however, utility companies may fail to exploit the opportunities that smart metering can offer by integrat-ing these two sides of the business, says Roberta Bigliani, head of IDC Energy Insights for Europe.

“We could be in a situation where we are creating silos of data rather than making more consistent availa-bility of the data,” she says.

Meter data could help fraud detec-tion, predict maintenance require-ments and eventually lead to smart grids which respond intelligently to variations in supply and demand.

“To do this the data needs to be validated and translated into a meta-data model, to create something that is usable by multiple applications,” Bigliani says.

“IT people need to work with the line of business to define a master- data sort of approach and try to create a layer where all the data coming from meters or operational systems, are transformed into pieces of data that different applications can call.”

This is the Holy Grail for utilities. By offering tariffs which discourage consumption while supply is weak,

and releasing stored energy in antic-ipation of peaks in demand, utilities can avoid investing in production which matches only the maximum levels of demand. It is also part of the transition to alternative energy sources, such as wind, solar and tidal energy, which exhibit fluctua-

While the utility industry spotlight has been on the big data problems created by the mass roll-out of smart metering, EDF has found a use for analytics in the field of customer retention.

EDF Energy, which supplies gas and electricity to 5.5 million business and household customers in the UK, has set up a dedicated analytics team to try to reduce customer churn. Estimates from energy regulator Ofgen suggest 160,000 customers change gas or electricity supplier every week. The team’s primary focus has been churn modelling: evaluating the propensity for customers to leave the organisation.

EDF Energy implemented a SAS data management and predictive analytics platform. Customer databases were augmented with third-party datasets, including attitudinal and lifestyle data with demographics.

The overall aim is to target marketing at customers deemed to be at risk of leaving.

Clifford Budge, customer insight manager of B2C energy sourcing and customer supply at EDF Energy said: “If you were a supplier that suffered one million customer losses every year, with the top 25% of customers billing an average of £1,200 per year, that equates to a total risk from those customers alone of around £300m a year. Even if you can act on just 5% to 10% of that, you are talking about a significant saving.”

Case study: EDF deploys analytics to cut churn

tions in the energy they provide.Projects such as the Eco Island (see

panel, above) on the Isle of Wight strive to demonstrate that smart me-tering and analytics can become es-sential tools in forecasting variations. It is a core part of the effort to make the island self-sufficient in energy.

This project is a picture of where the utility sector could be heading, albeit on a smaller scale. The lessons learned from projects like this could provide important insight into the data challenges involved in making the leap to new sources of energy on a national or international level. ■

-8-

buyer’s guide

To serve a growing customer base and better manage the client experience across all customer touchpoints,

organisations are moving away from siloed transaction-oriented systems – such as enterprise resource plan-ning (ERP), customer relationship management (CRM) and dealer management systems – in favour of more integrated and socially aware systems.

Social media driversCompanies are taking advantage of social media’s growing user base, using platforms such as Facebook, LinkedIn and Twitter to engage with customers directly. Other examples of online communities include consumers reviewing products at

Data’s role in customer engagementAs firms face an influx of structured and unstructured data, a combination of traditional business intelligence and big data approaches will provide the best results in managing and making use of the information, writes Sanchit Gogia

mobile app stores and third-party merchant websites.

Firms such as AT&T, Carphone Warehouse, Domino’s, Procter & Gamble, Tesco and Unilever now regularly use a variety of these platforms to engage with their cus-tomers. Data from social media helps

This shift has two effects. Organisations need to manage the surge in the type and overall volume of data, and at the same time be able to analyse a large amount of com-plex data in real time. While the vol-ume of structured data from tradi-tional transactional business applications such as ERP and CRM continues to grow, CIOs are facing an onslaught of unstructured data from multiple sources such as social platforms, and semi-structured data from machine-to-machine (M2M) communication. This signifies the need for traditional business intelli-gence (BI) approaches to supple-ment big data approaches, even for basic structured transactions.

© S

er

ge

j K

ha

cK

imu

llin

/Fo

toli

a.c

om

Data from social media helps organisations undertake sentiment analysis on their consumers and better tailor their offerings

CW Buyer’s guideBig data

-9-

buyer’s guideorganisations undertake sentiment analysis on their consumers and better tailor their offerings.

Mobile informationBusinesses are increasingly using mobile devices not only to push information to consumers, but also to engage with them via services such as mobile banking and mobile wallets.

Other devices continuing to grow in usage include self-service kiosks and smartcards that automate basic processes such as checking in at air-ports and paying for parking. For ex-ample, one telecoms provider in south-east Asia regularly collects lo-cation data from its customers’ mo-bile handsets to better understand their behaviour and thereby improve its marketing campaigns.

Gathering sensor dataSources such as radio frequency iden-tification (RFID) and sensor networks, which are typically used in B2B en-vironments, transmit semi-structured data that firms must harness to make better business decisions.

Retailers are now able to use this data to gain insight into the demand for their products and better manage their inventory and supply chain. They can also use the data to influ-ence decisions around new product development to improve the custom-er experience.

A holistic approachAs organisations explore technolo-gies to complement their existing investments and support big data, it is critical to base decisions on the existing information management ar-chitecture and identify components that they can reuse and consolidate. Companies must view big data im-plementation as a business project, not an IT project.

Traditionally, IT-enabled business processes have largely been defined around the structured data streams of process-based apps such as CRM and ERP. But it is critical for organisations to change this to include external sources and to redefine the data ac-quisition process (sources and types).

Firms must start big data initiatives at the business process layer and work with business executives to better understand business events that will eventually define job and workflow management. Involving business leaders at this stage can also help IT leaders outline the key business-specific metrics and key performance indicators (KPIs) that they will ultimately use big data im-plementations to monitor.

Better customer data management helps reduce duplication in the data-base and improve the accuracy of in-

Better customer data management helps reduce duplication and improve the accuracy of individual profiles

dividuals’ profiles. While the process around managing customer data has traditionally been optimised for dis-crete attributes such as age, gender and name, big data is changing this drastically.

Companies now need to allow cus-tomer profiles to include behaviour-ally oriented data, such as social net-working information and mobile telephony details, and ensure that they can combine this with the tradi-tional attributes.

Firms must expect their integra-tion costs to grow in some scenarios, as the cost of adding new data can make traditional data integration methods too expensive.

Architectural considerationsTo run analytic models for big data, organisations must invest in technol-ogies that support massively parallel processing (MPP). It is important for senior decision-makers to pick and choose technologies that don’t require them to rip and replace the existing architecture, but that will complement current investments.

The ultimate goal should be a single platform for all data warehousing re-quirements, as opposed to disparate sets of processing units. But in the meantime, most firms will have to govern three key pieces of their data warehouse architecture: traditional enterprise data warehouses; data warehousing appliances; and big data processing capabilities. Analysing big data requires organisations to adopt new analytical models and ap-proaches that allow large-scale in-dexing of data entities and sup-port relationship analytics to better understand the relationships between these entities.

Big data sources such as location-based data, social networking infor-mation and telecoms user details can help organisations establish relation-ships among data entities. Real-time analysis of these big data sources can provide companies with insights val-uable for targeting and messaging.

However, the new real-time analyt-ics for big data have neither the luxury of historical data nor the time to allow for trending analysis, and only offer directional insights, so firms should expect slight degrees of variance. In-stead, more traditional sources of data that do not require real-time process-ing are more likely to offer accurate in-sights for the business.

It is important for organisations to define the right use case for both types of data – the value lies in har-nessing insights from both types of data to inform business decisions.

Big data headachesAs companies adopt new applica-tions and approaches to cater to non-traditional touchpoints, they are faced with an explosion of

information. This increase in data volumes poses a new set of chal-lenges around information manage-ment and architecture.

The technical challenges of managing, accessing, and analysing real-time data streams means that IT teams are often unable to respond quickly enough to new market dynamics and improve customer experiences. Conversations with senior decision-makers and a recent global survey of 60 Forrester clients with knowledge of or experience with big data confirm that the explosion of information, both from traditional data and big data, is not just about volume.

While respondents indicated that volume is the main reason for con-sidering big data solutions, they also indicated the velocity of change, the variety of data formats and structural variability as major concerns. ■

this is an extract from the report The big deal about big data for customer engagement (june 2012) by Sanchit gogia, senior analyst, vendor strategy, at Forrester research.

Sc

an

ra

il/F

ota

lia

.co

m