Benchmarks - AGAIN !!!

This is in reference to Daniel Selman's posting on Meta-Policies where he quietly snuck in some references to benchmarks that were totally out of whack.

I turned all of my benchmarking for InfoWorld over to Steve Nunez back in late December 2006. BUT, as I recall, I did run a BUNCH of tests - over and over and over again - before I released all of it. The last published results still can be found at http://www.kbsc.com/Performance2006.xls - and for JRules "optimized" versus JBoss. Daniel shows Waltz (I can only guess that it's Waltz 50 or something like that) for JRules and Drools as

JRules: 2.65 seconds
Drools: 170.578 seconds

WRONG!!! I did these on earlier versions and the speed hasn't improved dramatically since that time. What I got way back then on a WinXP at 2.0 GHz was

JRules: 3.703
Drools: 14.187
Blaze Advisor: 2.266
OPSJ: 0.703

Drools 4.x is much improved but I have NOT re-run the benchmarks. Mark Proctor has and (I actually trust his results) reports that they have just about halved the previous times.

BTW, it should also be noted that, to my knowledge and not a fact stated from ILOG, that the "optimized" version of JRules is nothing more than compiled sequential rules - which should give great results. But compiled sequential loses the versatility of straight Rete rules where we can add rules wherever we like. If you look at my old spreadsheets, the straight Rete version of JRules puked (ran out of memory) on the 6.1.1 version. I would hope that the later versions are better but I seriously doubt it.

Blaze Advisor has several methods of running as well: Rete, Rete 2 (or Rete III) or Compiled Sequential. Compiled Sequential will "usually" run just a bit faster than Rete III in most commercial applications. BUT, compiled sequential, in my own humble opinion, is a violation of most rulebased principles. I mean, after all, why not do it in Java or C++ or even VB??? Sorry, I'm from the old school where a rulebase is part of the AI environment. Assembly (or plain-jane C) will usually run faster than most anything else.

So, if speed were the ONLY consideration, throw out the rulebase and use Assembly. And hard code it on a chip - bypass the OS completely. BUT, if you want intelligence, versatility Yaakov Kohen and not as any kind of representative of any company.

DS has never me a MetaPolicy he didn't like

Daniel Selman wrote a neat article on Meta: MetaPolicies, MetaRule, MetaData, etc. See http://blogs.ilog.com/brms/author/dselman/ for the whole thing.

I rather like the idea that he proposes in "over-riding" rules and things of that nature. I would think, however, that it might be easier to structure the rules in some kind of architecture using a common ruleset such that you don't have to override rules. Perhaps all of the common rules in one ruleset, then another ruleset that selects the appropriate attribue for each State (USA thing) or Country or whatever so that the proper rate is applied. It probably is more a matter of personal preference but two things I don't really like: Overriding rules and Inheritance of rules. Sooner or later, you get into trouble with which one does what and when.

Everything is new again

OK, FINALLY got the autority to post to Javarules at Google again. And, yes, I like the name and I think I'll keep it. Lots of things have happened since 2006. I've accepted a position with Fair Isaac - yes, I sold out to the dark side of corporate life. I've worked on about 8 or 9 projects since then.

Also, started to do some work on using statistics formerly used in MYCIN to help with "forecasting" in Insurance Underwriting. Meaning, how to "predict" that a person will be a good risk or bad risk from certain data.

Also, doing some work on a book with the Forgenator - a "how to" book on BRMS.

Finally, getting started on doing some work on parallel rulebase systems, a subject that has been around for 20 years now but no one has really done anyting with it in the past.

So, now that Google has finally straightened out my problems with access to my blog, I'll be back. :-)