How Large is "Large" ??

Greetings, Programs:

WARNING: This is a (long) rambling, philosophic blog - but not a "rant" as in the past. But, if you are new to rules (or even if you have a couple of years experience) you might find it interesting.

OK, so how do mean when we use the terms "large" or "complex" when talking about a rulebase? First, let's consider the term "large" as compared to "medium" or "small." In the 80's the most advanced scientists and rule engineers wrote one of the most sophisticated medical diagnosis software (MYCIN) using a rulebase and it had 587 (or so) rules. Another, written in the late 70's and used through the 80's, was used for configuring large computer systems (REX) ad it had just over 1,000 rules. Today, most rulebase geeks, or most business analysts for that matter, think of "large" in terms of "tens-of-thousands" of rules.

But what does this mean exactly? 10K rules? 20K rules? 50K rules? A couple of years ago I asked the two major vendors what was the largest rulebased system that they had ever designed and successfully implemented. Both said that it was 20K+ rules and that most "enterprise" BRMS consisted of 10K rules or less. Today, even the "little" guys are implementing 10K+ rules by using a spreadsheet approach, called a Decision Table by the vendors, wherein each line in the spreadsheet is a rule. If the spreadsheet has 10 columns that can be true or false, then there are 2^10 decisions to be made, or about 1,024 rules. If you add one more column, they there would be 2,048 rows or rules. (By now you should have guessed that you need to keep the spreadsheets small in terms of the number of decision factors or columns.) The number of action columns in one of these spreadsheets is immaterial since the action for any one rule can be any number of things - we don't really care about that part right now.

Just by comparison, the Homeland Security Department has 500K rules now (running on an IBM mainframe) and it will probably grow to about 2M rules in the next few years. Most are VERY simple rules but the problem is the thousands and thousands of objects. If one rule matches to 10K objects and another rules matches to 20K objects then we have 30K cross-matches in those two rules alone.

A few years ago I had the privilege of dealing with one of the larger insurance companies in the UK who wanted to go into the health insurance business in a big way - they were already in the health insurance business but they were only the tenth largest or so (in the UK) in that part of their business. We looked at the number of spreadsheets that they were already writing and determined that they would have about one spreadsheet for each major complaint, one dealing with just back pain, one with just knee pain, one for just shoulder pain, all of which came under structural or skeletal problems. The knee pain spreadsheet had 512 rows, 19 CE columns and the compression came about ONLY because the vendor had just introduced a N/A or "Any Answer OK" cell for the spreadsheet. Without that, it would have thousands and thousands of rows to be considered.

So, was this a "big" system? In terms of size of the underlying rulebase, I would have say yes because it still was not finished when I left after six months and had grown to more than 50 spreadsheets of this type, some larger, some smaller. The underwriters were still laboring to get their heads around the things that they were saying and put them down on paper.

So, are the rules complex? Meaning, does the CE of each rule, or most of the rules, contain multiple forward-chaining type of propositional logic constraints? What is the average number of CEs in each rule? To answer this question I would suggest that we use the Conflict Resolution technique discussed in the Cooper and Wogrin book, "Rule-Based Programming in OPS5". The book is long since out of print but can still be purchased on-line at many locations in new or almost-new condition. For example, let's look at the following rule (done in modern English, not in the OPS5 syntax - see page 53 in the book for the original source code). If we weight a rule according to the following principles that they have given for specificity

Element Class Name = 1
Predicate Operator With a Constant = 1
Predicate Operator With a Bound Variable = 1
Disjunction = 1
Each Predicate or Disjunction WITHIN a Conjun = 1

So, using that logic for the following rule

IF there is any Student called student (1)
where
student.placed-in == null (1)
sex-is = student.sex
smoker = student.smoke
AND there is any Room called room (1)
where
number = room.room-num
capacity = room.max
vacancies > 0 and < room.max (2)
room.sex == sex-is (1)
room.smokers == smoker (1)

CE1a Element Class Name of Student = (1)
CE1b Predicate With a Constant of place-in being null = (1)
CE2a Element Class Name of Room = (1)
CE2d Each Test in Conjunction where > 0 and < max = (2)
CE2e Predicate With Constant checking for sex (1)
CE2f Predicate With Constant checking for smoker (1)

Accordingly, we can see that the specificity, which should be closely correlated to complexity, is 7 for this rule. [I have put the weighting for each line as a number in parenthesis.] Now, if we "weigh" each rule and divide by the number of rules we should be able to arrive at the overall complexity of any rule set or rulebase. There was a time when vendors not only used MEA (means ends analysis) or LEX for Conflict Resolution but they published how they did it. You won't find it in today's manuals. It seems that they basically use refraction, priority (specificity) and the order in which the rules were entered. BTW, the Cooper and Wogrin book covers in plain detail the examples of MEA and LEX. If only the vendors would put what was right before the convenience of saving time of development and fear of run-time numbers. Remember, with CLIPS, Drools and Jess you can still write your own Conflict Resolution - but that is another subject that I think that I have covered before. Would that you could get the "Big Boys" to actually do Conflict Resolution correctly. :-)

Well, if you enjoyed this, great. If not, well, it wasn't a total waste - at least you learned about another book to read. :-)

SDG
jco
(Corrected some typos on 29 July 08)

Leveling the Playing Field for Rulebase Benchmarks

Greetings, Programs:

I have, in the past, published lots of benchmark results. Most were "fair" in my own eyes but recently Gary Riley has called my attention to the fact that they were not, in actuality, fair. For instance, some of them were done in the interpreted mode while some were done in the compiled mode. In the days of yore, I didn't have much of a choice since some of the rulebased vendors only had the compiled or the interpreted. Some would not allow external objects and others required external objects.

So, I have decided that we will start a whole new era of rulebase benchmarks. They will be broken down into several categories;

> Interpreted Rules

> Compiled Rules (usually some kind of OPS rulebase) for those that have such configurations and will include both inference mode and sequential mode

> Decision Table benchmark for Corticon, Visual Rules and other companies that use such tools.

Some companies will be able to do all three, such as ILOG (now IBM) and Fair Isaac. Others, such as Corticon and Visual Rules, will be able to do only the decision table version. OPSJ will be compiled only. I have determined that ALL of the three methods of using rules will HAVE to be able to use external objects, not internal objects. The reason for this is that "normal" business practices use external objects that have an API for calling or using these objects that adds abstraction to the objects - meaning that the objects can change, have things added to them, whatever and it should not affect in any way what happens with the old rules. New rules then can be written to access the old objects or the modified objects.

So, considering the first two categories, these would be conventional, OPS-type, IF-THEN-ELSE rules. What we don't want to do is resurrect the Miss Manners benchmark and probably not the Waltz benchmarks since both classes of benchmarks seem to rely on one rule firing over and over and over. Way too easy to cheat - and several closed-source companies have done so long ago.

But what if we came up with an NP-Complete problem that should be solved within a certain timeframe to just whether a rule engine can or can not be used within an viable commercial, business problem rule set AS WELL AS being used within the scientific and academic communities. I don't think that decision tables can do this kind of problem so we'll have to find a completely different benchmark for them, one that deals strictly with getting and reading the data, putting the data in the spreadsheet (OK, decision table) and then firing firing up the rule engine. Something like that. I'm open to suggestions so let me know if you can think of anything.

Meanwhile, congratulations again to Drools for best 2008 Open Source Rulebase/BRMS and to Fair Isaac Blaze Advisor for the best 2008 BRMS application. Well-deserved and well-earned.

SDG
jco

Structure and Sensibility

Today's Rant:

Is there anyone out there who still remembers doing machine language? Assembly? Algol? FORTRAN, even? If so, consider yourself blessed. You had to learn how to write tight, precise code to fit into a small amount of memory, sometimes only a few Kilobytes, and squeeze out the most speed that you could. Not only that, but if you were running on the "big" IBM-360 or something of that nature, then the code had to be reviewed by your co-workers and then by your supervisor BEFORE ever running it - sometimes you even had to have a complete model of what you wanted to do before hand.

Today? No models. No reviews. Just a bunch of web-weenies slinging code hand-over-fist just to see what will happen. No design, no thought process. And, when he / she gets through with it the only question asked is, "Will it work?" If the (usually using the 20% testing covers 80% of the situations rule) test cases (let's just "assume" that there ARE test cases) produce the right result it probably will be rushed into production.

And then it happens. Somebody crashes into the system via a "little known bug" in the software. Or, because the testing was not extensive enough, the performance is crap. That's when the "fit hits the shan" and heads begin to roll. All because the business guys believed some young punk salesman (or an old silver-haired fox) who guaranteed them that it would work and perform blazingly fast. (My friend Yaakov Kohen wrote a blog on this a while back at http://yaakov2.mindpress.com on why rulebased and BRMS projects fail.)

"Those who will not learn from history will be forced to repeat it." (Was that Ben Franklin or someone else? Doesn't matter. Probably Descartes or Plato or some other philosopher from the past that our children today continue to ignore.) At point here is that we, the IT guys, the supposed "brains" of the projects, do NOT force the project managers to STOP and actually do the design work first. They (the PMs) come to the geeks and tell them, "We have to have 'something' to show for the past two weeks of work." And thinking is not something that you can show. Gathering data and rules is not, usually, something that you can show. BUT, if you make it understood up front that nothing tangible will be produced for the first 50% of the time allocated for the project (which is usually not enough time to do anything of any consequence) you still be get that same, tired, worn out line from the PM. "What do you have for me to show to the VP or CEO or some other person who controls the purse strings."

Remember when the VP was a geek? And he / she could not only understand what was happening but could ask really good, discerning questions about what was happening? And could keep the CEO off of your back until the time was right? Well, those guys are gone. Now all that we have are nit-wit, MBA bean counters who haven't the foggiest notion of anything that would help the company beyond the next two or three weeks. Long-term goals and benefits always give way to the short-term rewards that are usually accompanied by long-range disasters.

SDG
jco

Drools Day Camp

OK, they don't call it that, but the Drools team will be in Dallas, Texas, from October 15th to the October 21st of 2008 for seven days of Hard Core Drools. See http://blog.athico.com/2008/07/texas-october-rules-fest-and-drools.html for the "public invitation." They have sent out an invitation should you want to come and to my knowledge, it's totally free. The best part? It immediately precedes the Dallas Rules Group October Rules Fest (http://rulesfest.org) in Addison. The two events are close but they are not at the exact same location.

So, as Autumn draws to a close and Winter begins to beat on the door, come to Texas for some great food, (usually) pleasant weather, good music and not one but TWO great conferences. Mark Proctor has even found a hotel http://book.bestwestern.com/bestwestern/productInfo.do?propertyCode=44540 with decent rates (where they will be having the Drools meetings.) BTB, Addison, Texas, is in the Northern part of Dallas and has more restaurants per square mile than any other place in the world.