Do Credit Scores Work?

We’ve all heard the horror stories.  I closed all my accounts and my credit scores dropped! I paid that collection off two years ago!  Why is more score still suffering for it?  Why won’t my score go up? Do these credit score things work?

Rest assured that lenders are asking the same questions about credit scores.  After all, they want some assurance that the tools they’re using are actually doing what they’re designed to do.  In fact, they track performance pretty closely because their business depends on it.  You’ve heard that a FICO score of 700 in past is more like a 680 today.  Well, how do they know that?

I’ll spare you all the statistical speak and try and boil it down for you.  Essentially you want two things, a large snapshot data set of consumers with scores at a particular point in time. They call this an observation data set.  Then, you want a second data set containing indications of how those consumers paid over a period following the original score data – usually 12 months. This is called performance data and consumers are usually classified as “good” or “bad” based on how they paid.

Don’t get hung up on the labels.  They could just as easily call them “A” and “B”.  (They don’t really know or need to know who the consumers are.  All the data comes depersonalized.  In other words, no names, addresses, SS#, account numbers or anything that can identify the consumer.)  Then, the consumers are ranked by score and the performance is compared to the score to see how well the score predicted the outcome.  Keep in mind that scores are designed to work in the aggregate.  Individual consumers are not analyzed but groups of consumers within score ranges.  So for example if there were 1,000 consumers in the 680 to 700 score range and 967 of them were “good” and 33 of them were “bad” you’d say that consumers in that score range have 30 to 1 odds of being “bad” or that you observed roughly 30 “goods” for every “bad”.  Typically this would be compared to the odds that the score was designed to have when it was developed.  If they’re roughly the same from one score range to the next then they know that the scores are operating as designed.  There are a myriad of other statistical tests that are used as well but not that can be explained in this small space.

But what if they find that the odds have changed over time?  Well they have a couple of options.  They can either adjust their expectations (a 700 now has similar odds to a 680 before) alter their acceptance strategies or they go back to the model builders and demand they rebuild the models.  The former, while inconvenient is usually preferable to the latter.  A model rebuild can take months or years and may not yield any better results.  Since scores are designed to relatively rank order consumers, simply adjusting strategies usually does the trick.  In don’t want to understate the case.  It’s still a significant problem, but it’s the quickest way to get back to reliable results.

During the financial crisis there was much debate over whether credit scores were working.  The answer varied depending on how you define “working” and different lenders had different results.  The short answer is that yes the scores do what they’re and lenders spend significant resources to ensure that they do.  But scores attempt to model a very diverse and dynamic population.  There will always be outliers – individuals that don’t fit the data.  That doesn’t mean they’re not working.

John Ulzheimer is the President of Consumer Education at SmartCredit.com, the credit blogger for Mint.com, and a Contributor for the National Foundation for Credit Counseling.  He is an expert on credit reporting, credit scoring and identity theft. Formerly of FICO, Equifax and Credit.com, John is the only recognized credit expert who actually comes from the credit industry.  Follow him on Twitter here.

Categorised in: Credit Cards, Credit Report, Credit Score, Improving Credit, Money & Identity

This post was written by John Ulzheimer