This is in reference to Daniel Selman's posting on Meta-Policies where he quietly snuck in some references to benchmarks that were totally out of whack.
I turned all of my benchmarking for InfoWorld over to Steve Nunez back in late December 2006. BUT, as I recall, I did run a BUNCH of tests - over and over and over again - before I released all of it. The last published results still can be found at http://www.kbsc.com/Performance2006.xls - and for JRules "optimized" versus JBoss. Daniel shows Waltz (I can only guess that it's Waltz 50 or something like that) for JRules and Drools as
JRules: 2.65 seconds
Drools: 170.578 seconds
WRONG!!! I did these on earlier versions and the speed hasn't improved dramatically since that time. What I got way back then on a WinXP at 2.0 GHz was
JRules: 3.703
Drools: 14.187
Blaze Advisor: 2.266
OPSJ: 0.703
Drools 4.x is much improved but I have NOT re-run the benchmarks. Mark Proctor has and (I actually trust his results) reports that they have just about halved the previous times.
BTW, it should also be noted that, to my knowledge and not a fact stated from ILOG, that the "optimized" version of JRules is nothing more than compiled sequential rules - which should give great results. But compiled sequential loses the versatility of straight Rete rules where we can add rules wherever we like. If you look at my old spreadsheets, the straight Rete version of JRules puked (ran out of memory) on the 6.1.1 version. I would hope that the later versions are better but I seriously doubt it.
Blaze Advisor has several methods of running as well: Rete, Rete 2 (or Rete III) or Compiled Sequential. Compiled Sequential will "usually" run just a bit faster than Rete III in most commercial applications. BUT, compiled sequential, in my own humble opinion, is a violation of most rulebased principles. I mean, after all, why not do it in Java or C++ or even VB??? Sorry, I'm from the old school where a rulebase is part of the AI environment. Assembly (or plain-jane C) will usually run faster than most anything else.
So, if speed were the ONLY consideration, throw out the rulebase and use Assembly. And hard code it on a chip - bypass the OS completely. BUT, if you want intelligence, versatility Yaakov Kohen and not as any kind of representative of any company.
Posts
Who stole my spring??
-
After a nice 20C degrees day yesterday, I woke up this morning to this:
10 years ago
8 comments:
The ilog manual is now online, so you can checkout the docs they have for the various rule configurations of the optimised mode.
optimize method
Optimizes the rules. After the optimization is finished, the ruleset is locked such a way that an exception is raised when a modification method of the ruleset is called.The optimization consists of determining whether a dynamic or a static agenda is needed for the ruleset, whether an iterated or a Rete rule is chosen.
optimise method
The optimization consists of determining whether a dynamic or a static agenda is needed for the ruleset, whether an iterated or a RetePlus rule is chosen, and guessing hashers.
iterated
The best results of this mode occur when the ruleset is highly incremental. A large number of modifications (insertion, update, retraction) are applied on the working memory while the rules are executed.
static agenda
This mode delays the rule instance creation when it is needed to just before the rule is executed. Contrary to the default RetePlus, there is no rule instances list. This strategy may reduce the the memory and improve performance when rule execution leads to removal of some rule instances of the agenda.
JIT
e JIT technology in this instance translates the condition part of each rule to Java bytecode. The bytecode is then used to evaluate the condition tests.
...
When this property is set to true, the rule engine uses dynamic rule compilation in its internal algorithm to evaluate rules. In terms of performance, the JIT feature enables an execution speed similar to that of compiled rules.
One thing I discovered is the rules Daniel used retracts the seating fact in the find seating rule. That results a 10x improvement in performance, so it's not Miranker's original manners benchmark. I think the benchmark is still valid, but it is clearly not a stress test like the original version.
I've posted a comment to JRules forum and asked if the test is running in sequential or RETE mode. My guess is sequential. Hopefully I will get a response on the JRules forum.
http://forums.ilog.com/index.php?topic=53.0
peter
ilog sequential mode can't do first order logic, no 'not'. It's running with their "optimised" mode, which I've detailed, as much as you can, from their manual in my first comment.
According to Changhai Ke the Business Rules Product Architect, it's not sequential. The version of manners that ships with JRules does not switch the context, which means it is a much easier test and doesn't thrash the agenda. The results for the version that ships with JRules and the version up on Daniel's blog provide examples of how JRules performs, but it's not directly comparable to the original manners benchmark.
Using the original manners rules, I discovered that JRules 6.6.1 goes into an infinite loop, so there is a bug in 6.6.1. Until that bug is fixed, I don't think a direct comparison between JRules and other rule engines using the original manners rules isn't feasible. Using the modified version on Daniel's blog, JESS, CLIPS and Drools all provide comparable performance.
my bias opinion for what it's worth. probably a half pence.
Mark, I don't understand why you say JRules sequential mode does not support "not". This is not true. Does the documentation need to be clarified?
None of these benchmarks (which are all based on inference) were using sequential mode, they were using RETE Plus.
We are still looking at Peter's comments on the Manners results and will post to the forum soon.
Sincerely,
Daniel Selman
http://blogs.ilog.com/brms/author/dselman/
Daniel,
I haven't seen a manual since the 4.6 days (until you put 6.6 on the public web), where it use to say explicitly that JRules' sequential mode does not support first order logic, i.e. 'not' and 'exists' unless used with 'in' or 'from'. I don't have this manual any more, but i've asked someone to dig up the quote for me from sequential's limitations page:
"You use the IRL constructs not, exists, and collect in a technical rule, and they are not followed by an in or a from relation enumerator."
Looking online now these limitations are no longer documented.
However an ILog JRules client recently had these limitation confirmed from your support department, and passed this information on to me:
"Just received definite answer from ILOG tech support about JRules' limitations of sequential processing (11 days). They admitted that a chapter of the limitations is missing from JRules 6.6's documentation; and there are little changes in this area since version 5 - the only improvement is that ?instance is now available for all modes."
Just to re-confirm I asked someone else to put in a single rule with just a 'not' condition in it and execute in sequential mode, they got the following error:
"Exception in thread "main" java.lang.RuntimeException: ilog.rules.engine.sequential.IlrJitterException: Sequential mode, "rule Test_01.XXX", not condition not supported"
If people are doing something wrong, or missunderstanding something, please let us know :)
I am confused. Did Daniel mean JRules supports "not" in sequential mode? Here is an excerpt about JRules'limitations in sequential mode from JRules 5.1 documentation:
"ILOG Rule Language (IRL) Limitations
.....
not, exists, collect conditions without an enumerator
The not, exists, and collect conditions with an enumerator (from or in) are translated to Java bytecode. When an enumerator is specified, the set of objects is available as a value tied to the condition.
..."
And here is the response I received from ILOG Tech support:
"I have confirmed with our development team, all the limitations listed from JRules document 5.1 are still present in JRules 6 except the ?instance variable. The lack of documentation is now registered as JRules Doc Bug Report JRULES-13670."
Regards,
Ming Jin
Post a Comment