Leveling the Playing Field for Rulebase Benchmarks

Greetings, Programs:

I have, in the past, published lots of benchmark results. Most were "fair" in my own eyes but recently Gary Riley has called my attention to the fact that they were not, in actuality, fair. For instance, some of them were done in the interpreted mode while some were done in the compiled mode. In the days of yore, I didn't have much of a choice since some of the rulebased vendors only had the compiled or the interpreted. Some would not allow external objects and others required external objects.

So, I have decided that we will start a whole new era of rulebase benchmarks. They will be broken down into several categories;

> Interpreted Rules

> Compiled Rules (usually some kind of OPS rulebase) for those that have such configurations and will include both inference mode and sequential mode

> Decision Table benchmark for Corticon, Visual Rules and other companies that use such tools.

Some companies will be able to do all three, such as ILOG (now IBM) and Fair Isaac. Others, such as Corticon and Visual Rules, will be able to do only the decision table version. OPSJ will be compiled only. I have determined that ALL of the three methods of using rules will HAVE to be able to use external objects, not internal objects. The reason for this is that "normal" business practices use external objects that have an API for calling or using these objects that adds abstraction to the objects - meaning that the objects can change, have things added to them, whatever and it should not affect in any way what happens with the old rules. New rules then can be written to access the old objects or the modified objects.

So, considering the first two categories, these would be conventional, OPS-type, IF-THEN-ELSE rules. What we don't want to do is resurrect the Miss Manners benchmark and probably not the Waltz benchmarks since both classes of benchmarks seem to rely on one rule firing over and over and over. Way too easy to cheat - and several closed-source companies have done so long ago.

But what if we came up with an NP-Complete problem that should be solved within a certain timeframe to just whether a rule engine can or can not be used within an viable commercial, business problem rule set AS WELL AS being used within the scientific and academic communities. I don't think that decision tables can do this kind of problem so we'll have to find a completely different benchmark for them, one that deals strictly with getting and reading the data, putting the data in the spreadsheet (OK, decision table) and then firing firing up the rule engine. Something like that. I'm open to suggestions so let me know if you can think of anything.

Meanwhile, congratulations again to Drools for best 2008 Open Source Rulebase/BRMS and to Fair Isaac Blaze Advisor for the best 2008 BRMS application. Well-deserved and well-earned.

SDG
jco