I’ve added a worked
R example of the non-convexity, with respect to model parameters, of square loss of a sigmoid-derived prediction here.
The undesirable property is: such a graph says that a parameter value of
b = -1 and
b = -0.25 have similar losses, but parameters values in-between are worse. This might seem paradoxical, but it is an artifiact of the loss-function – not an actual property of the data or model. The same note shows the deviance loss has the desirable convex property: interpolations of good parameter values are also good.
Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.