Practical analysis for investment professionals
25 January 2021

When to Use Big Data — and When Not To

“Big data” has been on the tip of everyone’s tongue for the past several years now, and for good reason. As digital devices and touchpoints proliferate, so too does the amount of data we each create. This information can be used to help us better understand clients and customers, make more effective decisions, and improve our business operations. But only if we can make sense of it all.

By choosing the right big data sources and applications, we can put our organizations at a competitive advantage. But to do that, we need to understand big data’s definition, capabilities, and implications.

Subscribe Button

Big data already has widespread applications. From Netflix recommendations to health care monitoring, it drives all types of predictive models that improve our daily lives. But the more we depend on it, the more we need to question how it shapes our lives and whether we should be relying on it so much. While progress is inevitable and something to embrace, big data’s contribution should not be measured by how many companies apply it, but by how much better off it makes society as a whole.

Defining Big Data and Its Relationship to Artificial Intelligence (AI)

Big data is more than just large datasets. It is defined by the three Vs of data management:

  • Volume: Big data is often measured in terabytes.
  • Variety: It can contain structurally different datasets, such as text, images, audio, etc.
  • Velocity: Big data must be processed quickly because of the increasing speed at which data is generated.

As the volume, variety, and velocity of data expands, it morphs into big data and becomes too much for humans to handle without assistance. So we leverage artificial intelligence (AI) and machine learning to help parse it. While the terms big data and AI are often used interchangeably and the two go hand-in-hand, they are, in fact, distinct.

“In many cases, it is simply no longer feasible to resolve every issue via human interaction or intervention due to the speed, scale or complexity of the data that needs to be observed, analyzed, and acted upon. Driven by AI-powered automation, machines can be imbued with the ‘intelligence’ to understand the situation at hand, assess a range of options based on available information, and then select the best action or response based on the probability of the best outcome.” — Ilan Sade

Simply put, big data powers AI with the fuel it needs to drive automation. But there are risks.

“However the tendency to add too much data in AI can cause the quality of the AI decision to suffer. So it is important to take the benefits from big data and analytics to prepare your data for AI and to ensure and measure the quality, but don’t get carried away by adding data or complexity to your AI projects. Most AI projects, which are mainly narrow artificial intelligence projects, do not require big data to provide its value. They just need a good quality of data and a big quantity of records.” — Christian Ehl

AI Pioneers in Investment Management

Realizing Big Data’s Business Potential

Properly applied, big data helps companies make more informed — and therefore better — business decisions.

“A few examples include the hyper-personalization of a retail experience, location sensors that help companies route shipments for greater efficiencies, more accurate and effective fraud detection, and even wearable technologies that provide detailed information about how workers are moving, lifting or their location to reduce injuries and increase safety.” — Melvin Greer

But this crucial competitive advantage is underused because so many companies struggle to sift through all the data and distinguish the signal from the noise.

Five principal challenges keep companies from realizing big data’s full potential, according to Greer:

  1. Resources: Not only are data scientists in short supply, the current pool also lacks diversity.
  2. Data aggregation: Data is constantly being created and it is a challenge to collect and sort it from all the disparate channels.
  3. Erroneous or missing data: Not all data is good or complete. Data scientists need to know how to separate the misleading from the accurate.
  4. Unfinished data: Cleaning data is time-consuming and can slow down processing. AI can help manage this.
  5. Truth seekers: We should not assume data analysis will yield a definitive answer. “Data science leads to the probability that something is correct,” Greer writes. “It’s a subtle but importance nuance.”

Addressing the first challenge is of paramount importance. The only way to solve the other issues is to first create the necessary human capital and provide them with the necessary tools.

Ad tile for Artificial Intelligence in Asset Management

The True Promise of Big Data

Data is a wonderful instrument, but it is not a cure-all. Indeed, “too much of a good thing” is a real phenomenon.

“In my years working with many businesses, I have indeed seen some companies that fell into the situation of not using data enough. However, these occurrences paled in comparison to the number of times I have seen the reverse issue: companies with an over-reliance on data to the point that it was detrimental. The idea that data is needed to make a good decision is a destructive one.” — Jacqueline Nolis

To illustrate her point, Nolis describes Coca-Cola’s introduction of Cherry Sprite. What motivated the decision? Data. People were adding cherry-flavored “shots” to Sprite at self-service soda dispensers. So score one for big data.

But as Nolis points out, the very similar-tasting Cherry 7UP already existed — and had since the 1980s. So the data team might have come up with the new flavor more efficiently simply by perusing the soft drink aisle at the local grocery store. The lesson: Too heavy a reliance on data can be a barrier to commonsense decision making.

Financial Analysts Journal Current Issue Tile

Big Data Applications: When and How

So how do we know when to put big data to work for our business? That decision needs to be made on a case-by-case basis according to the demands of each individual project. The following guidelines can help determine whether it is the right course:

  • Consider the desired outcome. If it’s to catch up with a competitor, investing in something the competitor has already done may not be a good use of resources. It might be better to let their example serve as guidance or inspiration and reserve big data analysis for more complicated projects.
  • If disruption is the goal, big data can be applied to test new ideas and hypotheses and maybe reveal other possibilities. But we need to beware of the downsides: Data can kill creativity.
  • If a business decision is urgent, the “data is still being analyzed” is not an excuse to delay it. Amid a PR crisis, for example, we won’t have the time to mine the available data for insights or guidance. We have to rely on our existing knowledge of the crisis and our customers and take immediate action.
Investment Professional of the Future report graphic

Of course, sometimes big data is not just useful but essential. Some scenarios call for big data applications:

  • To determine if a strategy is working as planned, only the data will tell the story. But before we measure whether success has been achieved, we first have to establish our metrics and define the business rules that determine what success looks like.
  • Big data can help process and create models out of vast amounts of information. So as a general rule, the larger and more data-intense the project, the greater the likelihood big data could be helpful.
Promotional tile for Cryptoassets: The Guide to Bitcoin, Blockchain, and Cryptocurrency for Investment Professionals

Big data might be the trendy topic in technology today, but it is more than a buzzword. Its potential to improve our businesses and our lives over the long term is real.

But that potential needs to be leveraged purposefully and in a targeted fashion. Big data is not the business equivalent of a wonder drug. We need to be mindful of where its applications can help and where they are superfluous or harmful.

Indeed, the full promise of big data can only be realized when it is guided by thoughtful human expertise.

If you liked this post, don’t forget to subscribe to the Enterprising Investor.

All posts are the opinion of the author. As such, they should not be construed as investment advice, nor do the opinions expressed necessarily reflect the views of CFA Institute or the author’s employer.

Image credit: ©Getty Images / Who_I_am

Professional Learning for CFA Institute Members

CFA Institute members are empowered to self-determine and self-report professional learning (PL) credits earned, including content on Enterprising Investor. Members can record credits easily using their online PL tracker.

About the Author(s)
Sameer S. Somal, CFA

Sameer Somal is the CEO of Blue Ocean Global Technology and co-founder of Girl Power Talk. He is a CFA Charterholder, a CFP® professional, and a Chartered Alternative Investment AnalystSM. Somal leads client engagements focused on digital transformation, risk management, and technology development. A testifying subject matter expert witness in economic damages, intellectual property, and internet defamation, he authors CLE programs with the Philadelphia Bar Foundation. Somal is a frequent speaker at private industry and public sector conferences, including engagements with the Federal Home Loan Bank, Global Digital Marketing Summit, IBM, New York State Bar Association, US Defense Leadership Forum, and US State Department’s Foreign Service Institute. He proudly serves on the Board of Directors of Future Business Leaders of America and Girl Power USA. Committed to building relationships, Sameer is an active member of the Abraham Lincoln Association, Academy of Legal Studies in Business, American Bar Association, American Marketing Association, Business Transition Council, International Trademark Association, and Society of International Business Fellows. A graduate of Georgetown University, he held leadership roles at Bank of America, Morgan Stanley, and Scotiabank. Somal is a CFA Institute 2022 Inspirational Leader Award recipient and was named an Iconic Leader by the Women Economic Forum.

Pablo A. Ruz Salmones

Pablo A. Ruz Salmones is the co-founder and CEO of Grupo Ya Quedó, a software development and artificial intelligence (AI) company headquartered in Mexico City. As a computer and business engineer, he leads new partnerships and enterprise client relationships at Grupo Ya Quedó in North America, Africa, and India. He also serves as director of marketing at Blue Ocean Global Technology. Ruz Salmones is a regular speaker at global conferences on topics ranging from scaling global businesses and e-commerce to the application and ethics of AI Ruz Salmones is an active member of Beta Gamma Sigma, the International Society of Business Leaders (ISoBL), the CCPM (Colegio de Contadores Públicos de México), and the Mexico City chapter organizer of Hackers/Founders. He holds an Ethical Leadership Certification from the NASBA Center for the Public Trust. Ruz Salmones is a published writer and technologist who recently developed a costing system for accurate assessment of data storage in cloud servers. He is a lifelong pianist and composer as well as a concert performer. Ruz Salmones is relentlessly committed to creating a world in which we all see everyone for what we are: human beings.

2 thoughts on “When to Use Big Data — and When Not To”

  1. Khaled says:

    Last time I hired Data Analytics he made infrastructure of data security for my business using different methods I really don’t remember names but I was not satisfied with his work or maybe the logics he made didn’t provide me what I wanted.

    Do you have suggestion related best data security method for small business online data?


  2. Monali Thakur says:

    Hey Dear Sameer and Pablo, I really want this kind of information. I visited many sites but can’t get specific knowledge, in your blog i get lots of information. Thanks Dear for this. I really appreciate your work.

Leave a Reply

Your email address will not be published. Required fields are marked *

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.