Conflict Resolution in Rules - Part 1

Greetings:

One more time - and this will be the first part of several editions on the subject. Later blogs will consider the details of the various Conflict Resolutions (CR) systems, when to use which one, and why each one is important. So, since CR has been such a critical part of a rulebased systems in the past, why is it that most rulebased systems being written and used today use either priority (salience) or the order of the rules as they were entered into the system to resolve conflict resolution? The answer is simple: It's easier to code the engine that way. Only a few (4 at last count) of today's rulebase engines still allow you to have a system that can allow you to depend on a true conflict resolution. Dr. Forgy discussed this with John McDermott way back in the 1980's in an article ("Production System Conflict Resolution Strategies; Pattern-Directed Inference Systems") that emphasized the importance of CR. Later, they wrote that MEA (Means-Ends-Analysis) was a slightly better system of CR than was LEX (Lexicographical) and WHY we can not use conflict resolution as a center of intelligence. It's a way to think about rules and how they should be enforced, not the rules themselves.

So why have most vendors moved away from MEA? Ease of programming AND because most Java programmers who are pretending to be rulebased consultants or engine programmers have no understanding of CR nor its importance to rules in general. So, why IS CR so blooming important? Aren't the rules just put into a big bucket enough to do the job? Two-part answer: (1) For a small job with less than a few thousand rules, you probably won't see any difference; usually because the "architect" has been able to arrange the rules and objects such that the CR has little or not effect on the performance nor the answer. (2) However, for a large project (10K rules or more along with several thousand objects) the process of "thinking" about the rules becomes paramount.

Think of it this way: If your rulebase is concerned with only one project, one problem (an insurance policy approval comes to mind) then all of the rules are, for all practical purposes, focused on solving a single problem. However, what if you have to think about a lot of things at one time, much like the human mind has to deal with many problems at one time and all of them in different time slots and conflicting time slots and many shades of priorities? Conflict Resolution is best discussed in some of the text books listed at http://www.kbsc.com/aiBooks.html because the academics (and this is NOT a derogatory term) don't have to support the idea of CR when actually writing a rulebased engine.

Here is a proposition: If you are using priority of the rules ONLY as the method of CR, then you may as well write the system in straight-up Java. It would, after all, be far easier to understand, to write and to put into operation. If you are using the order of entry as the means of resolving the CR, then why NOT just use a straight-up CASE statement or a huge IF-THEN system of Java clauses. After all, isn't that the kind of technique that is being taught in most rulebased classes?

Ah, because both the CASE statement nor the IF-THEN clause are monotonic - meaning that you can go through the rules one time and then quit. But, really now; implementing nonmonotonicity in either clause is just a matter of bookkeeping a huge WHILE clause (or maybe several while clauses) of some kind. So then, again, why use CR to resolve which rule to fire next?

The answer is that we are trying to insert some kind of intelligence into the system. CR of rules in the human brain is one of its distinguishing features in comparison to computers. Most of the CR in the human brain is done through the neural network of the brain and is not a cut-and-dried process as you might think at first. When we lose the ability to directly affect the thought process of the brain, we become stumbling idiots incapable of the simplest tasks. When we refuse to us CR for rules, we move from an intelligent rulebase to an idiot rulebase. It has been shown that as we remove the CR from the rules we do two things: (1) We make them far more fragile and (2) we make rule maintenance a nightmare.

Please, Rulebase Designers: Stop and think. Push back to the project manager who says that CR is not important because your competitor has dropped it and nobody noticed it. (Which might be true.) But, unfortunately, if the developers at that company did notice, there probably is very little that they could do other than complain and since the developers are not the ones who pay the bills, they wouldn't be noticed back at the vendor level. But what leaving CR out of the rulebase mix has done is to move us further and further away from what was, at one time, one of the defining factors of intelligent rules and a step up from the straight up IF-THEN rules of COBOL, C, C++ and/or Java.

Yes, I know that someone who actually lived back then will say that the only reason that they did this was to increase performance. Well, partially, that's true. BUT, that is not the main reason.

Next time: What is MEA and LEX and how do they differ both from each other and from common paradigms of CR. Thanks for listening...

SDG

jco

Warm Sunday Afternoons

Greetings:

Imagine for a moment that it's a warm, sunny Sunday afternoon in a city other than home. Your're sitting here in a strange hotel room trying to think of what you should have ready for tomorrow and for the all-day meetings all of next week on various topics, sub-topics and sub-sub-topics that will, eventually, decide what your product and your fate will become for the next few years. On the one hand, it's exciting, exhilarating and makes you feel more alive than anything else. On the other, there's the fear and dread of failure to live up to what you think you can do, what others think you can do to keep your established place in the market place and, if possible, to improve it; whether by leaps and bounds or even just bit by bit.

So, try as you might, you seek to think creatively without being obviously silly and, as my musician son would put it, without being "cheesy"; without so obviously trying to grab the attention of the market place with cheap tricks and shallow reasoning. Whatever you do it simply has to be solid and something that the market place needs and can't do without. It has to be something that is substantial and which, at the same time, won't create too radical a change in the way things are being done presently otherwise you will have a completely new product.

So, here we go - I'm asking for your help: What do you think that the "market place" of rulebased systems needs most? What is it that we can do for a product that it needs to be an even better product than it is today? Let's assume that you already have the finest Product Managers in the business, some of the most talented developers (engineers) that any company could want, and the God Father of Rulebased Systems himself as the chief scientists working on the problems and design analysis. With all of that, why would you have to ask for more help? Because it is the technical public, the business analysts, the USERS of the systems who can help a company define what a rulebased system should be and what it should and should not do.

[Back to first person] From our Ivory Towers, we can not predict what will be a "hit" and what will be a "miss" in this business. We have to get down in the trenches where those who work with our product or any other product day after day and find out what they want and, even more importantly, why they want it. It isn't enough to have one or more users to say that they want an engine that will solve a particular problem in a certain manner and give the appropriate answers, we have to know WHY that problem should be on the immediate need list and HOW that solution will make business easier for users.

Please feel free to either comment or contact me directly.

SDG

jco

Why Should I Care About Un-enforced Standards?

Greetings:

During these days of smaller and smaller budgets, way too many times, vendors, especially the smaller vendors, don't care about standards unless they directly affect sales today or next month. OK, maybe if they affect sales this year. But, here's the problem: Belonging to a standards committee is, many times, a defensive measure (to keep your product from being written out of the standards) or an overt attempt to directly influence the committee into accepting that YOUR standards as the only (or maybe just one of two) standard. A third, and unusually rare problem, is that the vendor is seen as doing one of the above when, all the time, their intentions are totally altruistic and in the interest of the industry as a whole - meaning that they are one of the "good guys".

And that is what we should be doing: Writing standards that help our industry to establish definitions so that we can have a common language so that when we talk with customers we can say that we adhere to standard XYZ and the customer know that when they say "blatherskater" it has the same meaning to both of us as well as any competitor that adheres to that standard.

But what if we adhere to our industry standards, what if we are concerned about our industry and we (all of us) are trying our best to be sure that we all agree on standards. Now, what if one of us should fail to adhere to one of those standards and produce a product that claims to adhere to that standard and the product does not adhere to that standard. What happens then? If there is no governing body to reinforce some kind of sanction, the standard is valueless. If the standards are not enforced by a governing body that can enforce some kind of economic sanction, or even a legal sanction, when there are violations of those accepted industry standards, then the standards have no value.

Back to the original question: Why should I care if they are not being enforced? Because we, all of us, have a duty and an obligation (especially the "Thought Leaders" such as FICO and IBM) to "do the right thing" even when no one is looking nor checking. The servant of the most value to the master is the one who does what is right and proper even when the master will never know nor find out. That is what is known as a "trusted servant." And we, the leaders of the industry, must do what is right and proper with respect to standards and ethics even when we know that we don't have to those things and that there is no retribution when we get caught doing the wrong thing. We, the "thought leaders" of our various industries, must be "Trusted Servants" of the industry.

All that being said, we MUST follow our own standards upon which we all will have agreed even when it means that we will take some kind of financial hit. The hardest thing of all is combating the argument, "But if we do this thing (that we know is wrong) it will cost us more money and if we don't do it no one will find out and even if they do find out, so WHAT? There's nothing that they can do about it." (Yes, I worked in sales for a year or so and heard all of that kind of thing.)

Hopefully, this won't be my last blog on this subject. Maybe next time I'll go after JSR-94 and what it means to have a standard that can be observed but means absolutely nothing to anyone whether you adhere to it or not. Maybe...

SDG

jco

Blaze Advisor

Greetings:

Well, we're all working our tails off over here trying to get Blaze Advisor 6.9 out the door this quarter (more exact dates will come later) and start ramping up for Blaze Advisor 7.0 I've seen the new and improved 6.9 and it looks really good. For starters, the GUI part has been improved and a lot lof the suggestions from the users have been incororated into it. Advisor 7.0 will be even better, but 6.9 still has a LOT of really good stuff coming out.

I did work with Yaakov at KBSC to do some benchmarks for a client the other day to see how many trivial rows of a five column Decision Table we could put into a BA 6.8 Decision Table using Vista 64-bit O/S, 12GB of RAM and Java 6 JDK 64-bit. A lot of stuff was running in the background but still it installed and compiled a single table with 250,000+ rows. A single table (NOT the sane way to design a table) but someone had asked the question so we decided to just see what we could do. Certainly I hope that no one ever tries to do this in real life but at least we know what can be done.

Which brings up the question of proper rulebase architecture, something often forgotten. Just as you would not ask a beginner to design you database (you would get a certified DBA) then, in the same line of logic, you need a real RuleBase Architect to design and help maintain an enterprise rulebase. Going to a one-week (or three week) school certainly will get you started, but you need a professional if you have more the, say, 2K or 3K rules and you're running with a commercial database and a commercial server and any commercial anything. You MUST have the right person for the big jobs, and usuall that's one of the vendor's guys from the plant who has the contacts to reach out and touch for help in really sticky problems.

Well, enough for now. Check back later and I'll try to have more goodies for you. OR, a better idea, just sign up for the RSS feed and read them whenever they are posted.

SDG
jco

Collaboration and Participation

Rules Fest 2010 Conference - Feb 15th

Greetings:

Jason Morris is the leader of the new and improved Rules Fest 2010 - formerly known as October Rules Fest. This is known as REALLY short notice but I've blogged almost everywhere else and forgotten this one. The conference begins at 10:00 CST, 2:00 p.m. London time, 3:00 p.m. Paris and Munich, 8:00 a.m. in San Francisco.

Just go out to http://www.dimdim.com and register for the jcmorris-mts conference.

SDG

jco

October Rules Fest 2010 Update

Greetings:

FYI - Because I took a job with a vendor (FICO, the #1 Rulebase/BRMS in the world) as a Product Manager, Jason Morris of Morris Technical Services is now the Honcho of ORF. His # 2 guy (for now) is Mark Proctor and I think # 3 is either Rolando Hernandez or Jacob Feldman.

AND, the name might change - stay tuned for further developments. Film at 11. :-)

Seriously, Jason is going to need lots and lots of help; both time and money, from anyone who wants to be part of the group. Those of us who are vendors can help but I don't think that we should have an "official" position in the group. And, furthermore, if you want to be a REAL part of the group, then you will become a partner of the group, meaning that if the event loses money you lose money, if it makes money you make money. Vendor employees would be greatly encouraged to become partners. Perhaps there would be a partner list:

Diamond Partner: $10K / annum

Platinum Partner: $5K / annum

Gold Partner: $2K / annum

Partner: $1K / annum

And the partners could be listed on the web page itself. This would be similar to Art Groups where some of the more financially-gifted persons would be able to help in a more substantial way. Partners could be individuals or companies. There might even be certain benefits to being higher-level partners. That's for Jason to figure out, not me.

However, it has been a GREAT run for the past three years. Truly, I have enjoyed it and I would like to mention those who have made it possible - and I know I will leave someone out:

• Dr. Charles Forgy and Gary Riley: These two gave the conference the credibility that made everything else possible.
• Greg Barton (my strong right arm and without whom ORF would NOT have been possible.)
• Rolando Hernandez (2008 would not have happened without him and Ile doing the web pages and the brochures.)
• Pete Carapetyn for helping get the 2008 conference off the ground.
• Carole Ann Berlioz-Matignon, formerly of FICO / Fair Isaac and now an independent consultant. She devoted time and energy and, most importantly, lots and lots of money (Diamond Sponsor both years) to the event. Without her, well, we would have been way, WAY in the hole.
• Carlos Seranno-Morales, also formerly of FICO / Fair Isaac and now an independent consultant. His support came in the way of money (YES!) and a couple of really great presentations.
• David Kim of Visual Rules, 2008 Gold Sponsor.
• Mark Proctor, Edson Tirelli, Chris Verlaelen and others from Drools in both 2008 and 2009. Their Drools boot camps (both years) accounted for almost 25% of the attendees and almost all of the European attendees.
• Academic Speaker such as Dr. Leon Kappelman (University of North Texas), Dr. Gopal Gupta (University of Texas at Dallas) and Dr. Daniel Levine (University of Texas at Arlington). For the 2010 conference (by whatever name) I'm hoping that Jason will get them to play an even greater part in the conference but already they are the academic foundation of ORF.
• Larry Terrill who laid out the foundation of the Rete Algorithm for all beginners in a way that managers, students, academics, ANYONE could understand. Totally fantastic job!
• Daniel Selman of ILOG who gave a great presentation in 2008 and, even after the conference, helped financially to make up some of the losses. Who else would contribute to a conference as a sponsor AFTER the conference was over???? What a guy!!
• Dr. Rick Hicks (Texas A&M) who keeps reminding us of the Validation and Verification process for ANY rulebased process. I, for one, certainly hope that he returns every year to keep reminding us of the hazards and pitfalls of any technical project.
• Thomas Cooper (Distinguished Guest 2009) and one of the God Fathers of Rulebased Systems for his enlightening talks and insightful questions. His presence at the Pub Nights was a blessing as well.
• Paul Vincent (Tibco), Edson Tirelli (Drools), Charles Young (Solid Soft) and Adam Mollenkopf (FedEx) for their insights into Complex Event Processing. Hopefully 2010 will have even more on this subject.
• David Holtz and Luke Voss for reminding us of our AI heritage and bringing the conference back home whenever we tended to stray from our roots. Both guys are scary brilliant and I hope that they return each and every year.
• Finally, the ground rod of the group, Jason Morris, who started the Jess Boot Camp last year and has agreed to pick up the reins and make sure that ORF will always be the real conference BY developers and FOR developers.

Thanks to all of you and I know that I left someone out - probably several someones. :-) Regardless, it's been a great run, lots of fun, and I will be back next time as a speaker. So, if you can find it in your heart to contribute just $1K toward the next conference, I urge you to contact Jason and make your contribution. We, the developers, the geeks, the trolls under the bridge that keep the wheels of progress turning, should be able to make this happen each and every year. Please help Jason get this together and make 2010 even better and bigger than 2008 or 2009 ever dared to hope to be.

SDG

jco