Scoring / rating engines - advice and examples?

Question

I need to create a flexible (and preferably dynamic) scoring engine, much like a credit scoring or premium calculating system. Does anyone with practical experience of creating a scoring engine have any advice, examples or suggested patterns?

I already know about:

• Rete Algorithm
• FICO
• The open source rules engines listed here

Thanks!

Edit: To provide a little more detail.. Ok, so I have had a look around and I think a rules engine is what I am after, it's more flexible and rules can be used to achieve pretty much anything. However, the material I can find on the web is highly abstract - the Rete algorithm, nodes, forward chaining and so on. I really need practical, architectural advice. So for example, how would you tackle these problems:

• Assume the rules engine itself is generic and agnostic of the context in which it is being used, so it is 'pluggable'. Now, in order to use it you have to feed specific and identifiable data items in and match those items with conditions and rules. So how would you go about solving this conundrum?
• How would you handle the situation where one rule updates a data item which invalidates other previously assessed rules?

Answer 1

A scoring engine has to actually score something - with your FICO score, it's the risk that you'll default on new credit. Essentially, there are a handful of steps:

1. Collect mountains of data
2. Decide what you want to know about the data - what do you want to predict?
3. Mine the data for a relevant formula that predicts accurately
4. Implement the formula in code in a flexible way

Assuming you're asking about step 4, and not about an earlier step in the data mining process, here are some ideas:

• If your formula is simple, you can just code it and allow access to edit the co-efficients (Ax + By + C, as an example, where A, B and C are stored in a database somewhere and are easily updated, and where x and y are some data from your user/customer).
• If you want something highly dynamic, in that you might dramatically change the formula later, a rules engine is one choice, though I'd still lean towards a custom-coded solution that can be easily switched out. Maybe even a DLL with a single function in it that does your calculation, and can be easily replaced if the formula is changed significantly.

If you add some more details, or if you're not quite at step 4 in your mining process, I can suggest some steps for you to take or some reading that might help.

Answer 2

Generally speaking, people who go about developing a scoring system have a good idea about how they might do the scoring by hand, and the start of the process is to code up the naive algorithm suggested by their experience. Sometimes, people then get fancy, using concepts such as genetic algorithms. Thinking about things like FICO scoring, another part of the work tends to be the collection of data, both transactional data (purchase patterns, earning patterns) and 'result' data (defaults or other issues with credit). By analyzing patterns and results, the scoring tools develop a score for users, which is then used as a predictor of future behaviour (is this person with a FICO score of x likely to cause me problems if I extend them credit?).

I don't think there is a general solution for these analyses. Someone has to have an insight into how to do the analysis and correlation, and can then use computers to define and refine the insight.