Once again we return to the world of Rulebase Benchmarks. Benchmarks for rulebased systems seem to have three camps: Those who love them and usually have good numbers to report. Those whose benchmarks are not remarkable one way or the other and are not worried about them; yet. Those who benchmarks are not very good and keep telling their users and/or customers, "Yes. That's OK for academics. But those benchmarks don't represent the 'real world' in which we have to live and work every day."
At October Rules Fest 2009 I asked the T3 (Thursday Think Tank) if they thought benchmarks were important. Dr. Forgy (who has the fastest rulebase engine on the planet Earth - called TECH) and myself (who has spent the past decade exploring benchmarks, their cause and effect) were ardently in favor of such things. FICO (Blaze Advisor) folks were kind of OK with the idea, especially since they use Dr. Forgy's engine in their rulebase. The rest of the group, whose performance was anywhere from just barely OK to abysmal, were not really against using benchmarks but it just wasn't something that they wanted to discuss in public.
So, here's my proposal: If anyone can produce a "real world" benchmark that can be run on any rulebase engine on any platform, PLEASE DO SO!! Otherwise, those of us who are in this for the long haul will continue to examine a rulebase from many different viewpoints but we shall NOT throw out tried and true tests that continually disclose fatal faults in fancy products that will not perform with large data sets combined with large rule sets. Such applications are large banks, large insurance companies, Homeland Security along with "real world" AI problems in rulebased forecasting, petro-chemical processing problems, electrical power grid production problems, shift scheduling for large plants and/or hospitals and anywhere else that will have the many objects - many patterns complex cross products.
Speaking of which, I guess it's about time to publish the 2009 Benchmarks. Our goal is to finish up in December some time so that at least we can say that we finished this year with a completed product. :-)
SDG
jco
Posts
