Rulebase Benchmarks 2009 Part 3

Greetings:

Once again we return to the world of Rulebase Benchmarks. Benchmarks for rulebased systems seem to have three camps: Those who love them and usually have good numbers to report. Those whose benchmarks are not remarkable one way or the other and are not worried about them; yet. Those who benchmarks are not very good and keep telling their users and/or customers, "Yes. That's OK for academics. But those benchmarks don't represent the 'real world' in which we have to live and work every day."

At October Rules Fest 2009 I asked the T3 (Thursday Think Tank) if they thought benchmarks were important. Dr. Forgy (who has the fastest rulebase engine on the planet Earth - called TECH) and myself (who has spent the past decade exploring benchmarks, their cause and effect) were ardently in favor of such things. FICO (Blaze Advisor) folks were kind of OK with the idea, especially since they use Dr. Forgy's engine in their rulebase. The rest of the group, whose performance was anywhere from just barely OK to abysmal, were not really against using benchmarks but it just wasn't something that they wanted to discuss in public.

So, here's my proposal: If anyone can produce a "real world" benchmark that can be run on any rulebase engine on any platform, PLEASE DO SO!! Otherwise, those of us who are in this for the long haul will continue to examine a rulebase from many different viewpoints but we shall NOT throw out tried and true tests that continually disclose fatal faults in fancy products that will not perform with large data sets combined with large rule sets. Such applications are large banks, large insurance companies, Homeland Security along with "real world" AI problems in rulebased forecasting, petro-chemical processing problems, electrical power grid production problems, shift scheduling for large plants and/or hospitals and anywhere else that will have the many objects - many patterns complex cross products.

Speaking of which, I guess it's about time to publish the 2009 Benchmarks. Our goal is to finish up in December some time so that at least we can say that we finished this year with a completed product. :-)

SDG

jco

Only Three More Days !!

Greetings:

If you want to attend the October Rules Fest AND you would like to avail yourself of the Early Bird Discount of $50, you should do so by midnight (PDT, GMT -8) THIS Friday night, July 31st. That's when it expires.

Also, remember that the first 100 attendees to sign up will get the best seating in the house. All attendees and speakers will have power outlets and WiFi available in the meeting room itself. But the first 100 or so will be two persons at a 6' (just under two meters) table.

I have said this so many times that I'm beginning to sound like a cracked record - if you're old enough to remember records made of vinyl that had a tendency to get caught in a groove and just repeat itself over and over and over... Anyway, ORF 2009 has absolutely the BEST Technical Speakers in the world. AND, these guys will be hanging around during the conference to chat with the attendees - especially during Pub Nights.

Who else can say that they bought a beer for Mark "Drools" Proctor, Charles "Dr. Rete" Forgy, Jason "The Jess Answer Man" Morris, Gary "CLIPS" Riley, John "EA" Zachman, Carole Ann Berlioz-Matignon, Carlos "Advisor" Seranno-Morales, Paul "Tibco" Vincent, Edson "CEP" Terilli, Daniel "TurboGeek" Brookshier, Rick "V&V" Hicks, Daniel "ANN" Levine, Hafedh "I wrote the books on JRules" Mili or Jacob "Open Rules" Feldman?? But, if you are at the conference and hang out on Pub Night at the Walt Garrison Room, YOU might !! :-)

The main theme of this year is just like last year: Have Fun! We are doing the one thing that we enjoy the most (outside of one or two indoor activities) and we should be having lots and lots of fun doing it. Last year was simply great. This year is even better. Better Hotel. Better location. Better presentations. I would say better speakers but some are the same as last year BUT we have many new speakers who will knock your socks off!! And the vendors who are the sponsors will have the very best technical personnel to answer your most perplexing problems.

This year, we will have redundancy on the video / audio recording so that we don't have gaps like last year. We learned our lesson about depending on a single source of technology. Greg Barton has promised the best recording and Chelanie Israel has promised to deliver a truly professional, well-polished product for those who attended the meeting. And, no, we do not have plans to sell the video to anyone. It's our free gift to those who attend the conference - and ONLY to them.

Sign up NOW and avoid the rush AND get an extra 10% along the way. Also, when you sign up, be SURE to sign up at the hotel for the special rates. If you have any trouble whatsoever, send me an email and let me know. [ jco@kbsc.com ]

SDG

jco

Rules or Code ????

Greetings:

Currently there is a bit of discussion on some of the developer groups as to, "Why not use scripting languages rather than a rulebase?" Good question and it deserves an answer.

One of the problems with rules is that those "outside" the rulebase community have little idea of what belongs in a rulebase and what does not. Nor do they have any idea of what constitutes a BRMS or a rulebase. So, before we run off and begin suggesting that folks use Java Script (or something equally mundane) to contain the rules, perhaps we should examine WHY we are developing a rulebase (and tacking a BRMS interface on top of it) in the first place. What follows will probably be elementary to most but maybe it will remind us of what it is that we are trying to accomplish.

A rulebase has been around since the late 1950's with Newell and Simon's Logic Theorist that preceded even the moniker of "AI". See https://www.rulelab.net/ for more discussion on this. A rulebase initially was to allow psychologists and programmers to try and solve a sticky problem when all of the rules were not known and so that there would not have to be a certain order of rule insertion into a program to be sure that one rule would not fire before another. They called this declarative programming and it is significantly different to all other languages such as BASIC, COBOL, Algol, FORTRAN, C, C++, Java and all of the scripting languages. They were trying to figure out how the brain works when is has a problem with conflicting rules and overriding rules and developed a system where the rulebase literally "chews" on the rules until there are no more rules to execute.

But, the question still remains, "What is a rule?" Or, as our business brethren put it, "What is a business rule?" More importantly, perhaps we should ask, "What is NOT a rule?" First, it is not data validation, even though I have used a rulebase (JRules) to do that very thing so that the business analysts could create, change and/or modify the rules without having to explain to the IT guys WHY this was true. (Shame on me for having been a party to such a venture but it paid the bills for 18 months.) I have a potential client now who has 10K+ so-called "rules" that are nothing more than data validation and they don't have any idea of what is a rule for a rulebase and what is a rule for Java/C++. No clue.

Another time that you should NOT use a rulebase is when you have to begin inserting more that three levels of priority. This alone indicates that the underlying problem is procedural in nature rather than declarative. I have seen so-called rulebased programs with as many a nine levels of priority. What this indicates is either ignorance on the part of the developer or lazy programming. This is much like the infamous "goto" statement in BASIC and other languages - yes, you CAN do it but it is NOT good programming. Just so with priority - use rule sets or rule flow (which I dearly hate but it seems to be a necessary evil) rather than so many levels of priority. (I prefer goal-oriented programming to rulesets but, in the end, it's pretty much the same thing.)

A BRMS, on the other hand, is nothing more than a cool interface and lots of toys to make the rulebase more appealing to the business guys. Remember them? The are the ones with the money who will buy our toys for us to solve their petty problems that could have been solved using any of the procedural languages. However, this does move 100% of the verification of the rulebase over into the business world where it belongs and allows the IT guys to go about their business of J2EE, database problems, web design and things of that nature.

But, and here's the crucial part: What is a valid case for using a rulebase? According to Girratano and Riley way back in 1998 (and they haven't changed much since then) the following are all guidelines for using a rulebase:

Can the problem be solved more effectively using conventional programming?
Is the domain well-bounded?
Is there a need and a desire for an expert system?
Is there at least one human expert who is willing to cooperate?
Can the expert explain his/her logic so that it is programmable?
Is the problem solution mainly heuristic or uncertain?

If these questions can not be answered, "Yes." (with the exception of the fist one) then probably you do not have a rulebase problem but rather a Java/C++/Scripting problem. If most can be answered, "Yes." then and one or two are "No." then you should carefully examine the potential hazards with using a rulebase.

You know, most businesses projects that I have had over the years, with a couple of notable exceptions, usually came straight out of the company manuals and NOT from domain experts. Who wrote the rules and WHY they wrote the rules was unknown for the most part. All the business guys usually know is that their competition is using a rulebased system for solving their problems and their business guys are getting to use a rulebase to do it so we have to do it as well. And the rationale that they use to justify such a solution is oftentimes amazing.

Whatever you do, have fun. You will have times when a client just insists on using a $1M+ rulebase to just keep track of rules that would be better suited to SQL or scripting. (Maybe their golfing buddy sells those things?) To keep your ethics in line, point out all of the obvious reasons for NOT using a rulebase and if the client insists on misusing a rulebase, take the money and run. :-)

SDG
jco

Leveling the Playing Field for Rulebase Benchmarks

Greetings, Programs:

I have, in the past, published lots of benchmark results. Most were "fair" in my own eyes but recently Gary Riley has called my attention to the fact that they were not, in actuality, fair. For instance, some of them were done in the interpreted mode while some were done in the compiled mode. In the days of yore, I didn't have much of a choice since some of the rulebased vendors only had the compiled or the interpreted. Some would not allow external objects and others required external objects.

So, I have decided that we will start a whole new era of rulebase benchmarks. They will be broken down into several categories;

> Interpreted Rules

> Compiled Rules (usually some kind of OPS rulebase) for those that have such configurations and will include both inference mode and sequential mode

> Decision Table benchmark for Corticon, Visual Rules and other companies that use such tools.

Some companies will be able to do all three, such as ILOG (now IBM) and Fair Isaac. Others, such as Corticon and Visual Rules, will be able to do only the decision table version. OPSJ will be compiled only. I have determined that ALL of the three methods of using rules will HAVE to be able to use external objects, not internal objects. The reason for this is that "normal" business practices use external objects that have an API for calling or using these objects that adds abstraction to the objects - meaning that the objects can change, have things added to them, whatever and it should not affect in any way what happens with the old rules. New rules then can be written to access the old objects or the modified objects.

So, considering the first two categories, these would be conventional, OPS-type, IF-THEN-ELSE rules. What we don't want to do is resurrect the Miss Manners benchmark and probably not the Waltz benchmarks since both classes of benchmarks seem to rely on one rule firing over and over and over. Way too easy to cheat - and several closed-source companies have done so long ago.

But what if we came up with an NP-Complete problem that should be solved within a certain timeframe to just whether a rule engine can or can not be used within an viable commercial, business problem rule set AS WELL AS being used within the scientific and academic communities. I don't think that decision tables can do this kind of problem so we'll have to find a completely different benchmark for them, one that deals strictly with getting and reading the data, putting the data in the spreadsheet (OK, decision table) and then firing firing up the rule engine. Something like that. I'm open to suggestions so let me know if you can think of anything.

Meanwhile, congratulations again to Drools for best 2008 Open Source Rulebase/BRMS and to Fair Isaac Blaze Advisor for the best 2008 BRMS application. Well-deserved and well-earned.

SDG
jco

Senior?

OK, what does it mean to be a Senior anything? In the normal use of the word, a Senior Citizen is someone over 50. That's the only qualification - age. However, in the technical world, usually it means 10+ (sometimes 15 or 20) years of experience AND someone who is at the top of their profession. In the Java world, a Senior Consultant is "normally" someone who knows OO design, most of the Java API, is usually (not always) certified at one level or the other, and has been a team lead on at least one enterprise project using Java and/or J2EE and/or Spring, etc.

But what about the rulebase world? Where do we draw the line between a "Senior Consultant" and an "Architect" - or is there any difference? What follows here are just my thoughts and not anywhere referenced - so feel free to reference this blog as authoritative. Since I'm over 40 (OK, over 50) and have 23 years experience in IT (yes, since 1984) and another 10+ years prior to that with experience in Electrical Engineering - so I have the longevity if nothing else. Does that alone make me a "Senior" anything? Nope. Not even the 15 or 20 published articles and white papers would server to "officially" make me a Senior consultant without considering their context, audience, etc.

General Patton once commented that he had a mule that had been through two world wars and many battles, but the mule was still a mule. Time alone is not sufficient. The "Senior" rulebase consultant must have been recognized by his/her peers (in the industry, not just with in the company) as someone with exceptional expertise in something to do with rules and/or a rulebased system. And, that person should have experience with MORE than just one particular rulebase tool or one vendor. In addition, that person should have experience with presentations, demonstrations, talks, and, mot of all, at lease ONE SUCCESSFUL enterprise design work that has held up for several years. Installing a rulebase system is easy - any Java monkey can mimic what has been learned in a one-week class and install a system. BUT that system should still be in place, albeit with modifications, two or three years later.

So, what can we, as the industry "leaders" do to correct this all-too-obvious problem? We can demand (probably without results but we don't know until we try) to have a certification board across all vendors that would be vendor-agnostic that would verify and validate a candidate. A written exam is a rather silly way to do this since we have proven over and over (Microsoft and Sun are the latest victims of this craze) that written exams mean nothing. I have seen non-programmers study for and take the Sun Java Programmer certification and pass the first time. I have also seen experienced Java programmers, who did not study for the exam, fail.

Some of the best consultants in the rulebase industry have not one single written exam proclamation on their wall to proclaim their proclivity and productivity in a given field. Except, maybe, for a Masters Degree or a Ph.D. - but even that is not usually on the wall. No, what we need is to have agreement among the techies of all of the vendors (something close to Nirvana) that would be sufficient to satisfy all. I'm thinking of something like a Master's degree thesis or a Ph.D. thesis level of a white paper submitted to the panel - not some project paper but something that requires thinking and thought application.

So, here's my suggestion: Take this idea to the President or CEO of your company and explain that a "Senior" position means nothing today. But, if we could all get agreement on the meaning of that title and validate that person's claims to being a "Senior Consultant" then the company would be able to charge basically twice what they have been getting in the past for the services of that person. Customer would benefit because they KNOW that the person on their project has been examined (not tested - examined) and found to be one of the best in the world. This would mean that a potential rulebase project would have a 90% or better chance of success rather than the 40% or less that we are getting today. Now, how much is a 50% probability of success worth to a client or to a vendor? Let's have some fun and find out.