Big Data
“It’s not if, but when,” is a line I picked up – from Warren Buffet – from an annual report twenty years ago.
There are many who are frustrated that more data, at more granular levels, with 21st century valuation techniques are not available today. It is an old story from Enron days that has continued throughout my career.
There are those, like me, who do not see this as an issue. I see this as a major risk, not a minor one. I won’t say certain markets have a greater need than others. That provides an easy out for those few. The reality is that data needs are going to grow exponentially in the next decade.
Companies sitting on spreadsheets, great big ones, are going to be caught with a major catching up and a costly endeavor. That is because some companies are holding onto isolationist concepts that they hope will stay the same until they retire. I understand that thought. My reply, though, is what footprint are you going to leave? A carbon footprint or a footprint for everyone to follow, even decades after you retire to Phoenix?
I am not selling anything here either. I am an economic-based thinker and want to ensure we create optimal, risk-reward economic outcomes for trading and risk strategies and trading and risk software.
If your organization is behind on the basic systems, it is going to be harder to leap through the continuum of the 1990’s to the present day. It gets harder each day you don’t at least bring your data into a warehouse. And if the company has not experienced a software project in over five years, there will be fewer who have project experience. For example, in learning organizations, people who have experimented to full software process and system replacements will do much better in managing to get through projects; whereas, those who have little to no software project experience will not learn how to do projects in batch-mode, or play catch-up from 20th century systems, very well.
Some experience gained now goes a long way when you get deeper and deeper into understanding system needs, as you take those initial steps into buying and implementing software. That is why I do not care what state you are in today. I always ask, “What specific area is giving you the most headaches?”
The response, which I have said earlier, that I get from the triple-A trading personalities, is that they want any, all and automatically in the functionality or they don’t want it.
Iterations are necessary in growing any business. Big Data is going to be a major step for many companies. To some, I say bring your mountain climbing gear. I have some experience in hiking and climbing Colorado 14’ers. That is not a picnic, even for many experienced high altitude sporting challenges. Therefore, let’s get started climbing some small hills first.
In 1995, just out of grad school, I was one of the lucky few who had a Sun Microsystem computer on my desk. Yes, I was one of the big number crunching gurus tasked with analyzing correlations and volatilities for oil and gas markets.
With that luck, I also got assigned an ad-hoc request to create a monthly cash-flow of our trading books through its 27-year duration. However, the trading books were so large they wouldn’t fit onto an excel spreadsheet. I received a beta copy of Excel 5.0, which was the introduction of ‘subtotaling’ thankfully enough, but the data-set was still too large to download directly into 5.0, so IT had to divide the data into 5 or 6 separate downloads.
Though it was a sign, one of many, thankfully, I only stayed at that job for two years. Fast forward through my energy trading career, from risk to trading to buying, installing, and consulting on untold software projects, and the issue remains the same. ETRM systems still cannot calculate timely, nor with useful models (i.e. black-scholes is old school) to the satisfaction of the cutting-edge trading leaders.
Specialized bolt-on software to crunch hundreds of thousands of simulations or to process an MTM calculation with 30 million rows to any satisfaction – especially in this elusive idea of ‘real-time’ profit and risk results – has yet to finds its way into core ETRM systems. This has frustrated innumerable clients.
With the volatility of markets today, and with markets changing faster than ever, and that we are on the cusp of revolutionary changes in the Power markets, and the even greater need to do cash flow projects, tomorrow we will wake up to the fact that we need billions of rows of data. We’re calculating real-time correlations, volatilities, physical optimization, power storage possibilities, weather simulations, cash flow, risk, feedstock, and switching opportunity costs (to wind, solar, battery, oil, natural gas, hydro, or geothermal). A black-hole seems more easily understood than the challenges facing ETRM systems!
Are you ready to give up and succumb to a fate where we never catch up, not to the speed of light, but with the speed of change and Big Data times?
If you are still not convinced you need Big Data, look at the niche players who model sophisticated mathematical simulations using auto-regression, Geo-Brownian Motion modeling of spot and forward pricing, volatility and correlation, as well as weather data. Yes, simulating weather data. Take into account that companies are already segmenting consumer markets and behavioral sciences, human nature is already starting to be used to price consumer markets, especially in deregulated retail power and natural gas markets.
Today, though, we still have many utilities on spreadsheets just buying enough natural gas and/or power to cover their expected demand, and simply entering into spot markets to buy/sell the power and natural gas they need on any given day. That is simply the just-getting-by, old-school approach for with 20th-century thinking and application.
The question for you is: Where do you stand today? Are you satisfied with just getting by? What footprint are you trying to leave – a carbon one or a great legacy for others to follow?
Big Data is on our door step, ready to be unpacked. Front–2–Back believes there are ETRM software companies that are moving quickly into the Big Data arena. Others already see their own wake, and it may be too late. Said another way, ETRM systems have to get the next level, beyond basic transaction processing, without the old-school models, and take us into the 21st-century software needed to be successful in this ever-changing commodity and energy world.
Change is rapidly increasing, leading to greater volatility. Volatility equals risk. Risk equals reward – the free-market way. Get onto this wave and ride it into a winning market position!