Essbase futures at Oracle OpenWorld 2013

The Essbase blog post to end all Essbase blog posts

That phrase could mean two different things:

• I am going to share with you some of the really cool things that Oracle have announced at OpenWorld and so this is just a turn of phrase to show how excited I am by all of this.
• I am going to share with you some of the really cool things that Oracle have announced at OpenWorld but maybe didn’t want broadcast to world+dog and by doing so I will royally annoy Oracle product management, development, and probably areas of Oracle (hint:  legal) I’ve never heard of, get kicked out of the ACE program, and in despair I give up blogging.

Hopefully this post will fall more into the former than the latter but one never knows. In my possible future defense, I note that there are a bunch of Tweets out there that cover some of this very same stuff and there were people taking pictures at the session, so maybe it’s all okay. I hope.

So why bother?

I didn’t take any pictures (I got in trouble last year but happily my picture taking skills are pretty bad) this year although I saw many doing just that.

What I did take were notes. Somewhat alarmingly, I am actually not half bad (note, I did not say I was good) at typing thanks to my much-missed grandmother who taught high school business in Royal Oak, Michigan. And that meant when we visited her in the summer (Michigan is a great place for a kid) we got to play with her electric typewriter (yes, it was really cool in the 1970s and yes you now have an idea about how old I am) and all of the typing books publishers sent her as samples. And that meant that in those long ago days before video games, I learnt how to touch type. And that means you now understand why my blog posts are so long. Whew.

So with the caveat that I am on the outside looking in, and with the additional HUGE caveat that everything Oracle stated was under safe harbor, i.e., they are thinking about this stuff BUT NO PROMISES and what they present MAY NOT MAKE IT TO THE PRODUCT and the performance MAY NOT COME CLOSE TO WHAT IS PRESENTED I think I can now relate what Gabby Rubin and John Baker showed to the almost completely sold out room.

What’s new in 11.1.2.3

Yes, this isn’t exactly the future as 11.1.2.3 is here now, but Oracle related some DEVELOPMENT LABS numbers that I thought were pretty exciting. What does Oracle mean by DEVELOPMENT LABS numbers? DEVELOPMENT LABS numbers mean: in development code lines, using incredibly specific test cases, on servers of unknown capacity, such and such results were observed. What I take that to mean is that these numbers are a rough guide to performance but for goodness’ sake, do not take this as gospel – as always with Essbase, your performance may vary quite a bit and that variation may all be on the bad side so test, test, test. And don’t say I didn’t tell you so when it all goes pear shaped.

With that caveat, here are the big things in 11.1.2.3:

• New BSO functions, especially @CREATEBLOCK and run time Essbase Substitution Variables
• ASO dimension builds are improved (no percentage given), especially with shared members
• Better MDX performance
• XREF performance is hugely improved, sometimes coming very close to intracube performance. NB – These are DEVELOPMENT LABS numbers. Test, test, test.
• The JAPI has been improved but honestly my highly-touted note taking skills failed me here.

So the big one in 11.1.2.3 is the XREF performance. That means to me that all of those games we play in Planning-land to get XREFs declared as stored (and taking the cost to do the data moves) may no longer be required. And that we can reevaluate data export processes in all kinds of BSO databases to move data across databases – maybe file exports aren’t required any more.

What’s in the future for Essbase, post 11.1.2.3

A bit of background

Essbase is in kind of a unique position vis-à-vis other products – it straddles both EPM (my home) and Business Intelligence (my home, one day, maybe). In some respects this is fantastic for those of us who love Essbase because that means it gets funding. Of course there is a flip side that means that functions are sometimes for BI and don’t come over to EPM because it simply doesn’t make sense. At least Oracle thinks it doesn’t make sense where you or I might disagree.

An example of a BI-only function in Essbase is its Essbase Persistent Aggregates – these are totally dynamically generated Essbase databases that are optimized for performance against OBIEE.

And there’s another bit about Essbase that I think most of us aren’t aware of – Essbase is the analytic engine behind Oracle Fusion GL Balances, CRM Territory Management, and Projects Performance. These aren’t Essbase databases we’re supposed to access via Smart View and in fact may make our brains explode as we bring our EPM perspective to how a cube ought to look. Gabby made an analogy that stated that what we might think of as a four dimension model may resolve to 14 or 200 dimensions in Essbase. Yup, that sure isn’t Sample.Basic.

The next 12 months

Again, remember, this is all what Oracle product management would like to do. They don’t know if it will happen. I don’t know if it will happen. Under no circumstances make any business or technology decisions based on what I am going to write. You Have Been Warned.

With that yet again restated absolutely huge, did-you-pay-attention-this-means-you caveat, here’s what I at least think I heard.

Restrictions

Naming restrictions like member name length, reserved keywords, special characters – all of that is a thing of the past. Before you go crazy with joy, remember that when your relational source has crazy ascii characters they now are going to come into Essbase. I am thinking of the Biblical giveth with one hand, taketh away with the other. Maybe a better way to think about this is that flexibility can be a double edged sword.

This will be available first in JAPI and SQL data sources. The whole issue around uniqueness will be addressed at a later date.

Performance

Improved performance for queries, calculations, and API performance are all on the roadmap.

Performance (and I guess to some extent features) are going to have a bit of a split between Exalytics and everything else. When a terabyte or two or more of RAM along with 40 or 128 CPU cores are available to Essbase, Oracle know that they can do certain things. When there’s 32 gigabytes of RAM and four cores, Oracle realize that there is much Essbase simply cannot do. That difference in capacity is going to tell, especially with the continuing success of Exalytics. They’re not abandoning the non-Exalytics world but those of us in the “normal” world will just have to look over the wall and sigh in envy.

I mentioned above that Essbase is being driven by BI, EPM, and Fusion. You will see functions and performance come out that may only make sense in one of these environments. This is going to be weird for those of us who have used Essbase since the beginning but again just a fact of life.

Simplified sophistication

Oracle know that we Essbase geeks love to twirl the Essbase dials. This is particularly true in BSO but there is bit-twiddling that is available in ASO as well. That’s all well and good for those of us who enjoy it but their goal is to take the sophisticated tuning we geeks do and make it available to everyone while not losing that option to do the tweaking of the settings. I for one am very curious to see how this works out. Hopefully no more brain-dead Essbase databases will be the result.

Enhanced calc script language

The BSO calc language that we all know and love is viewed as a real product differentiator that makes Essbase the awesome thing it is. Note that this is not MDX. More on this in a bit.

As the BSO calc language is cool, awesome, and generally rocks, it will continue to be enhanced.

FIXPARALLEL

If you think about what needs parallelization in Essbase, aggregations are an obvious choice and of course it already exists. But what about all those calcs that use FIXes to “do stuff”? That will be, someday, available as a parallel process as well.

Beyond just member formulas and allocations, DATACOPY, DATAEXPORT, XREFs (this is beyond 11.1.2.3’s better performance), and XWRITE all will work within FIXPARALLEL.

FIXPARALLEL will work within FIX statements and vice versa. Btw, I wrote this when I heard it: “That is pretty damn cool.” And it is.

Additionally, you will be able to control task dimensions for a division of work that is not dependent on outline order and when specifying the parallelism you will be able to define the number of threads. Within those threads there will be new Essbase variables (a feature I use all the time today) that are initialized within each thread and for each task within a thread.

XRANGE

XRANGE will be, maybe, extended to many more (approximately 30) functions but I couldn’t type them all fast enough in my notes. So we’ll have to wait and see what happens.

Smart View Recorder Services

This is an internal utility that Oracle will provide through OTN so now there will be no support. Essbase Hackers aren’t afraid of that, are we?

Recorder Services is an Application Provider Services add on (I have no idea how it is installed) that can be used as a performance and data benchmarking tool. It records user activity in APS from Smart View and logs it to…disk? Tables? Sorry, I missed that part but I am guessing to disk.

That log can be analyzed or replayed. Imagine what you might do with that. <insert big grin>

MDX improvements

MDX is not the red headed stepchild that is beat on a regular basis. Nope, it’s the key to ASO Essbase and of course is a pretty awesome query language for ASO and BSO.

So what can we expect to see as improvements to MDX? Better optimization for attributes (Does this include varying attributes aka Slowly Changing Dimensions? Dunno.)

Optimized Aggregate command is coming. This is very commonly used in BI queries and there will be significant performance improvements for totals at multi-level hierarchies. Improvements will be based on query and dimension depth.

The big MDX improvement is the concept of Sub Selects. What this means is that in order for Essbase to support the huge Essbase cubes that BI generates, Sub Selects slice out or filter portions of the ASO cube and force operations to only apply to that subset of the overall database. This means that relative performance to the whole cube can be realized. What I take that to mean is that less data equals faster performance. When the required data set for a particular calculation isn’t the whole ginormous ASO database, this could be a very valuable function indeed. Note that relative performance will seem greater the larger the database in question – again this is a case of limiting the scope of the calculation.

Query parity

There is a big difference between data results from SQL versus Essbase. I think of this as the top down nature of SQL versus the bottom up nature of Essbase. By that I mean – for a given data set, if the Essbase outline doesn’t map to all of the fact members, data will get rejected (hello dataload.err). For those of you who read Developing Essbase Applications you will note that I wrote a chapter that forced metadata matching to fact to get round this very issue using Essbase, ODI, and SQL.

Oracle have taken a different approach with what they call the Renegade Member. Instead of forcing the metadata to match the fact, they create default members that get the aggregated results of all of the non-matching fact data. This is huge, to put it mildly as totals shall now always tie.

Of course you are still going to have to figure out what’s missing, but in fact there will be a Renegade Member error log that you can then use to build hierarchy.

You will be able to have one for each dimension if you wish (I can think of some dimensions where it would not be terribly likely) and the suggested good practice is to stick this Renegade Member at the second generation of the dimension.

The big one – Hybrid Essbase

I love BSO Essbase. I love ASO Esbsase. Sometimes BSO is the answer, sometimes ASO is the answer, sometimes it is a real stinker trying to bring the two of them together.

Wouldn’t it be nice to have a database that gave us the flexibility and power of BSO Essbase with the raw power of ASO? That would mean we could continue to do all of the cool block-y things we do in BSO and then not worry so much about what happens when BSO Essbase blows up when the database gets too big.

What the Hybrid Essbase engine does (Slight digression: ASO, BSO, so obviously CSO?) is give you a choice about aggregation. At the base is always the same level zero blocks that BSO Essbase has always supported. Above that, with Hybrid Essbase (HSO?) you have the choice of making some dimensions dynamic (or even all of them if you chose so) or some dimensions dynamic and others stored (so same BSO aggregations with blocks).

This is not going to be like ASO so no Multiple Hierarchies – the dynamic nature is either on or off on a dimension basis.

What is also not like ASO is there is no ASO bitmap but instead (as I sort of understand this) the very best Intelligent Calc (or something akin to it) the world has ever seen. Hybrid Essbase will understand what you are looking at an upper level and will sum the blocks dynamically and do that aggregation very quickly indeed.

You will still have the option of stored upper level members and that will act like a BSO stored dimension today.

Using DEVELOPMENT LABS NUMBERS, Hybrid Essbase pretty much matched ASO Essbase and in some instances beat ASO performance, at least in the case of one development test case. Who knows what the real world will bring but the numbers were very, very, very and did I mention very impressive. How do they do this? Beats me and yeah, when you don’t understand something well it seems like magic. :) I love magic shows.

Just to make this clear (and to reiterate why this is so exciting): Hybrid Essbase has BSO capabilities with ASO dynamic performance. I can hardly wait.

Beyond the next 12 months

Gabby also related what might come out in Essbase beyond the next 12 months but it was so much, so fast, and so tenuous that it simply isn’t worth listing.

Also, that is a teaser to get you to come to Kscope14. :)

And that’s the end

Hopefully just of this post, not my blogging career. The new features are just so exciting I thought I owed the Essbase world at least a peek into what is coming from Oracle, maybe.

I have to say that I wrote this post with some trepidation but I hope my frequent OMG DO NOT BELIEVE ANYTHING I WRITE statements will give you some idea of what’s coming for Essbase while not committing Oracle to much of anything.

Some of this (actually, a lot) was available at Kscope13 but it was super tenuous and Oracle specifically asked us not to write about these future features. The restrictions seem to be (mostly) off and I for one am hopeful that many if not all of these features actually ship, and soon. But I’m not holding my breath and neither should you.

Be seeing you.