Menu Home

How to Pick an Optimal Utility Threshold Using the ROC Plot

Nina Zumel just completed an excellent short sequence of articles on picking optimal utility thresholds to convert a continuous model score for a classification problem into a deployable classification rule.

This is very compatible with our advice to prefer continuous scoring models, and probability models in particular, to “hard” classification rules during model development for classification problems. In fact her point is the benefit: by delaying the converstion of a contiuous score to a classification rule you may be able to use better more business oriented criteria to make the conversion. And you may be able to change the rule when utilities ore prevelances change.

For completeness we are sharing an example of how you reproduce her calculation on the ROC plot itself. This is a bit of a “more standard way of doing things”, but I think in fact didactically inferior to her method of working more directly in terms of utilities. So to see how to work with the ROC plot, please check out our new note here.

Unnamed chunk 9 1

Figure from: “ROC optimization”

Categories: data science Expository Writing Opinion Pragmatic Data Science Pragmatic Machine Learning Statistics Tutorials

Tagged as:

jmount

Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.

%d bloggers like this: