Model fitting for classification (3): are Least Squares a sensible choice? Probably not so...

This video discusses the applicability (well, rather the NON-applicability, at least in a general setting) of least squares in classification; The video is a continuation of the videos • Model fitting for clas... and • Model fitting for clas... .
Although "least squares" for fitting a function f(x,\theta) to training data {-1,+1} might be reasonable as a first option to start with the problem, it suffers from several drawbacks:
-- The cost is `symmetric', but some problems fare better with an asymmetric goal to minimize,
-- The only thing that really matters in the fit is the sign of $f(x,\theta)$ not its value,
-- Probability distributions (or logarithms thereof) with binary results (Bernouilly, or binomial if we count repetitions) are not proportional to 'squared error' as that actually is the logarithm of the normal distribution in least squares for fitting continuous data (under certain assumptions such as additive normal measurement noise).
Therefore, a 'naive' solution of applying 'linear regression' or, in general, optimizing a least-squares index for binary data, may not have the desired performance or may not have a formally adequate statistical interpretation.
________________
PDF/code/notes at: personales.upv...
#machinelearning #statistics #regressionanalysis
_________________
Antonio Sala
Full collection of videos at: personales.upv...