Benchmarks 2009

Greetings:

As is my wont (don't you just love Old English?) I will be running my annual "First Quarter - No Quarter Given" benchmarks beginning in January of 2009. Right now I'm limited to existing benchmarks of Waltz-50, WaltzDB-16 (almost all Rete-engine vendors) and some of them for the WaltzDB-200, a new version of WaltzDB-16 but using 200 variations rather than 16 - that should prove interesting.

Also, the platforms will be limited to Mac OS X running on a Dual G5 with 4GB of RAM and a Core2Duo running with 3GB of RAM. I have beefed up the Windows-32bit XP (Dual Threaded single CPU) machine to 3GB of RAM just to be able to run certain software that is incompatible with either Mac (Unix) or Windows Vitria-64-bit OS on i7 CPU.

IF, and let me emphasize the IF, I can get the time I will have benchmarks for Decision Table (only?) engines with a 10K and 100K Telecom Benchmark that will do nothing more than show off processing power of single-row data validation. So far, the DT vendors have not been very helpful in coming up with a benchmark of their own this past year so I pretty much label all of the a 5 in terms of performance - meaning that it's neither good nor bad; pretty much an unknown. Mostly because my editor won't all me to give them a zero. :-)

Also, we might do some of what Peter Lin and/or Mark Proctor suggested in the way of "micro-benchmarks" that would remove any level of cheating. If we throw in Gary Riley's Sudoku and/or the 64-Queens problem, we'll have something else that is not actually business related but will give some indication of engine performance.

The benchmarks will be

1. Waltz-50
2. WaltzDB-16
3. WaltzDB-200
4. 10K Telecom
5. 100K Telecom
6. MicroBenchmarks
7. Sudoku
8. 64-Queens

The classes of vendors will be

1. Rete-based engines, internal objects (CLIPS (?), JRules, Advisor, Jess, Drools, etc.)
2. Rete-based engines, external objects (CLIPS, JRules, Advisor, Jess, Drools, etc.)
2. Compiled Java Engines (Visual Rules, OPSJ, JRules, Advisor, Drools, etc.)
3. Sequential Compiled Java Engines (Visual Rules, JRules, Advisor, Drools(?), et al)
4. Decision Table Vendors (Corticon, Visual Rules, Haley Office System, VisiRules, etc. but could include JRules, Advisor and Drools)

Folks, that's a LOT of work for one little old Texas boy unless I can find someone independent to help AND if I can get some help from the vendors writing these benchmarks to be checked by myself and any independent help that I can get. If you want to help (and thereby ensure your name be placed with the other immortals of rulebase benchmarking) send me your name and we'll get you started.

Remember, to help with the overall project, you MUST be independent and NOT working for any of the vendors that are being tested. (You can be working on any vendors project as long as you are being paid by the client and NOT by the vendor.) To help with the project from a vendor point of view, all I need is the code for all of the tests in the appropriate syntax for that vendor. I (we) still have to read it and verify that nobody cheated but that should be really helpful and will be duly noted in the tables that will be published.

Maybe, just maybe, (no agreement yet) InfoWorld or some other equally high-visibility journal will be willing to publish these benchmarks in the form of an article of some kind. Otherwise, it will be just another blog on benchmarks. :-)

SDG
jco

NON-Documentation

Greetings:

Did you ever download a software package and start reading the Readme.txt or ReadMeFirst.html file to see how to install everything and then find out that it had not been changed recently? Or that you downloaded the Mac OS X version, installed the Mac OS X version only to find out that all of the instructions were for either Windows or Linux? OK, so Linux should be close, right? If it works, yes. If it doesn't work, wrong!

I just downloaded ILOG JRules Studio 6.7.2 Evaluation package. The instructions clearly state
=======================
To run Rule Studio do one of the following:

In Windows, use the shortcut: Start > All Programs > ILOG > ILOG JRules > ILOG Rule Studio for Java.
On Linux, start Eclipse using the executable in /studio
=======================

Nope. Not on Mac. There are some .app files there but they don't run on the Mac - maybe on Solaris or Linux, I wouldn't know since I usually do everything on a Mac or, sometimes, Windows. So, you have to find the build.xml file, read through it, and see if you can modify it so that you can make it run on the Mac properly.

And, remember all of those older files that you had to run under JRules? Forget it. You may as well rewrite all of that stuff using the "New and Improved" Eclipse interface. (Personally, I like writing my rules with a text editor but even the old BAL was kind of nice.)

Have you read through the Drools documentation recently? I really like Mark Proctor, Edson Terelli and the rest of the Drools guys - and I like the product - but they write what HAS to be the worst documentation in the industry. BUT, now that they are moving into "Big Boy" territory of BRMS tools (Advisor, jRules, etc.) Red Hat needs to hire a team of "real" Technical Writers to come in and help with writing proper documentation. I certainly hope that they do something soon or the only people using Drools will be the ones who can use the rulebase engine only and forget any of the ancillary bells and whistles that they have spent the past eight to twelve months adding to Drools.

I guess that the best documentation that I have seen is (forgive me for plugging an ex-employer) is the Fair Isaac stuff. Good docs that (usually) agree with the current version and is easy to use. Visual Rules docs are probably a close second. I've never had the privilege of using Haley Expert Rules so I can't say about them. I did look at Haley Office Rules but wasn't overly impressed with their documentation. ILOG JRules (the purchased version) documents used to be on a par with FIC but recently they have tried to go the "El Cheapo" route and skimp on that end of the product.

Anyway, before you DO happen to download the Studio, go to the chat room and see all of the problems that they are having with this "freebie" that they are putting out for six-month evaluation. THEN, if you still want it, at least you have been forewarned.

SDG
jco

Bringing RuleBased Forecasting (RBF) to 2009

Greetings:

We have started an Expert Systems Consulting Group over at http://ExSCG.blogspot.com and, for now, the leadoff topic is that of RuleBased Forecasting. If you would like to work on this, drop by the blog and check in. If you want to blog on that link, let me know and I'll add you to the list of bloggers there. If you want to work on the project, drop me a email and I'll see how we can distribute the load so that all can be involved. Don't worry, there's lots to do but we have a few weeks to get started. :-)

SDG
jco

Center for the Advancement of the Enterprise Architecture Profession

Greetings:

I ran across this today because one the members of another blog had joined and put that organization down as something to which he belonged. I went over and read over the charter and the pledge. I would that everyone would pledge to abide by the moral and ethical conventions that are espoused there. I fear, however, that some my have joined just to be joining - much like some folks just join the church because their spouse joined - a desire to fit with their contemporaries.

Regardless, check it out. If you think that you can live with the pledge, I encourage you to agree to abide by the by laws and the pledge agreement.

http://caeap.org/default.aspx

SDG
jco

Name Changes

Greetings:

As a follow-up to the previous blog, Rolando Hernandez has changed his blog from BRxG to an even higher level, Agility Alliance ( AA) that encompasses BRxG. He has put up a framework for lots of different groups for other to blog. I encourage you to visit there but don't forget the Expert Systems Consulting Group. One (AA) has more of a business focus and slant while the other (ExSCG) is more of a technical gathering without the pictures and Google Ads. Both have a purpose in life and, I suppose, that AA will blossom with members while ExSCG will grow more slowly but should be far more technical. It just depends on what you want out of life. :-)

SDG
jco

Too Many Blogs?

Greetings:

A friend of mine was hesitant to start up another blog on BRMS since I have two (this one, which is my "personal" blog and Expert Systems Consulting blog at http://exscg.blogspot.com/ which encourages other experienced consultants to blog there on common subjects.) My immediate answer is NO! But, what we could do is to promote all of the other blogs on AI, Rulebased Systems, Rulebased Forecasting, Neural Nets, Constraint Programming, Complex Event Processing, specific vendor blogs, etc. What this will do is to make everyone aware that there is more than one way to skin a cat - or any other animal.

But here's the thing - most blogs allow you to post links to other blogs and some, like BlogSpot, allow you to include a short snipped of the latest blog and when the last blog there was written. If done in this manner then the reader can determine which one to visit. After all, we don't have time to go hither, thither and yon to visit blogs that we have either seen before, that are two or three years old or that don't pique our interest.

So, WHY should anyone blog on the ExSCB above if they have their own blog? Because, unlike my private blog that accepts only comments on my own blog, this will allow that reader to respond with a blog that is "front and center" and either start another thread or comment that will catch you attention. If you read my private blog, you don't get to see some of the really astute comments that others have made.

So, while I like for readers to follow what is happening on my blog, I really encourage readers to visit (what I hope will be) a much better blog that takes in the opinions of everyone. Blogs might replace meetings, which is what Rolando is trying to do with BRxG blog at http://brexperts.ning.com/ by putting in slides, video, several SIGs, and is focused more on the Business side of rulebased systems; the CxO guys and Technical Managers.

SDG
jco