It is difficult to describe just how much data is generated by modern civilization. So difficult, in fact, that we have resorted to entirely new words to describe large amounts of information – such as the yottabyte, which is 10^24 bytes of information or 1 trillion terabytes.
Few industries have embraced this explosion of data like finance. Information on centuries worth of market movements along with up-to-the-second data from across the globe are now relied upon to provide insight into ever more complex investment tools.
But with all that data comes a lot of noise. Having the right strategies to figure out how to make the cream rise to the top is essential to modern financial analysis. It’s not just a spreadsheet and instinct that drive investment decisions – it’s complex, predictive models that filter out misleading information and outliers.
Big Data is on the Rise
For investors, portfolio managers, market strategists and industry analysts, information that previously trickled now comes via fire hose. And it’s not slowing down; it’s speeding up. IBM has said that 90% of all data in existence was produced in the past two years.
More information is better than less, but not all information is created equally and that can create problems for investors. Securities and Exchange Commission chief Mary Jo White commented: “We must continuously consider whether information overload is occurring as rules proliferate and as we contemplate what should and should not be required to be disclosed going forward.”
And while a good investor or portfolio manager might know good data from bad, the sheer volume now makes it nearly impossible for even a large team to sort through everything. The needles stand out in this haystack, but the haystack is growing at an exponential rate.
It’s hard to blame investors that have a bit of data fatigue. Just look at this Google Trends charts for mentions of “big data”:
This tells us that not only is there plenty of data out there, but plenty of people talking about it. It’s overwhelming, but fortunately has led to an influx of tools and expertise that not only make sense of the mountains of data available, but also conform to the needs of investors.
Dealing With Data Is NOT New
It has always been that way. The first shops to use advanced data techniques spent a large amount of capital on the people and infrastructure necessary to build a brand new system of financial analysis. This gave rise to intense competition for talent and technology that kept smaller players on the sidelines.
But like any new market, players have stepped up to fill these needs. As the techniques and technology became demystified and talent struck out on its own to meet demand for data analysis, firms have emerged to provide data analytics tools on a large scale.
Big Data Analytics
StatPro is one such group, specializing in providing data analysis that can be used to by an array of asset managers to analyse every part of a modern portfolio. The company is a prime example of how universal data analysis has become. With more than 300 clients in 25 countries, companies like StatPro now provide the tools and expertise to take advantage of mountains of data at a cost that is a small percentage of setting up one’s own data analysis department.
The importance of having the right tools to parcel out data should not be underestimated. For a baseball manager trying to figure out how his best left-handed reliever matches up against the middle of an opposing line-up, a bunch of statistics on the shooting percentage of college basketball point guards won’t help much.
That’s an outsized example, but with the sheer amounts of data available, just having that extra information can create a headache for investment managers attempting to provide meaningful analysis of a portfolio, which has created a niche for companies like StatPro.
Customization is Key
Every portfolio is different and no two portfolio managers have identical priorities. So when it comes to data, just having it provides about as much utility as access to your Bloomberg terminal. Certainly those basketball stats are not helping a baseball coach, but they are helping a basketball coach.
For asset managers, the same issue comes when trying to sort through data. A portfolio manager attempting to understand the risk profile of a certain investment needs different data from a hedge fund analysing how short-term interest rates will affect currency movements. Certainly some data will be useful to both parties – this is part of what makes data collection and organization so difficult. The bright lines between what is useful and useless are thinner than ever as the globalization of economics and markets removes barriers and creates almost real time reactions.
Sheer volume and organization are not the only barriers to effective data analysis, however. Once the data is there, someone has to do the math. Today’s financial analysis reaches far beyond price/equity ratios and moving averages to incorporate the most advanced modern mathematical techniques that are tailored to particular asset classes and investment strategies. Having the data is not enough. The best analytics use strategies to bring out details that are crucial to fully understanding one’s market position.
The Benefits of Big Data for Making Money
The upside for managers is that there has never been more expertise available than there is today. Increased demand for and competition among data analysis firms will continue to help provide the necessary tools and talent.
The golden age of data shows few signs of slowing down, but investors are in a tough position. Effective investment management demands the use of the large data sets currently available. But those data sets are growing larger at a rate that forces investors to stay on the cutting edge of data analytics.
That way the next yottabyte of data will, hopefully, be a piece of cake.
What is your thought on the future of big data for investing?