In this notebook, I’ll give high level intuition on convexity of cross-entropy loss and non-convexity of squared loss for Logistic regression.

import numpy as np
import matplotlib.pyplot as plt

def sigm(z):
    return 1./(1+np.exp(-z))

def sq_sigm(z):
    return 1./(1+np.exp(-z))**2

X = np.array([-1, 1])
y = np.array([0, 1])

First, let us create a simple dataset

plt.scatter(X, y,c=y, s=100)

<matplotlib.collections.PathCollection at 0x7f950da22910>

We will make a very simple assumption, there is no intercept term and the model is only \(x\theta\)

def y_hat(x, theta):
    return sigm(x*theta)

Let us write the squared loss and cross entropy loss functions

def cost_sq(X, theta):
    y_pred = y_hat(X, theta)
    y_true = y
    return (y_pred - y_true)@(y_pred - y_true)

def cost_cross(X, theta):
    y_pred = y_hat(X, theta)
    y_true = y
    return (-y*np.log(y_pred) - (1-y)*np.log(1-y_pred)).sum()

Let us plot these two losses over theta varying from -3.2 to 3.2

theta_lin = np.linspace(-3.2, 3.2, 100)
costs_square_vals = np.zeros_like(theta_lin)
costs_cross_vals = np.zeros_like(theta_lin)


for i, th in enumerate(theta_lin):
    costs_square_vals[i] = cost_sq(X, th)
    costs_cross_vals[i] = cost_cross(X, th)


plt.plot(theta_lin, costs_square_vals, label="Squared")
plt.plot(theta_lin, costs_cross_vals, label="Cross-Entropy")
plt.legend()

<matplotlib.legend.Legend at 0x7f950daa7730>

Let us look at the squared loss curve in more detail. Clearly, if we draw a line joining two points, some part of the curve is above and some part of the curve is below the line.

plt.plot(theta_lin, costs_square_vals, label="Squared")

plt.plot([-1, 3], [cost_sq(X, -1), cost_sq(X, 3)], color='k')

[<matplotlib.lines.Line2D at 0x7f950db99d30>]

We can also understand the same from a formulae perspective

Let us first discuss cross-entropy loss. For our data it is:

\(\hat{y}_0 = \frac{1}{1+e^{\theta}}\)

\(\hat{y}_1 = \frac{1}{1+e^{-\theta}}\)

Loss_cross = \(-1*\log(1-\hat{y}_0) -1*\log(\hat{y}_1) \because y_0 = 0, y_1 = 1 \)

From https://www.sarthaks.com/354259/expand-log-1-e-x-in-ascending-powers-of-x-up-to-the-term-containing-x-4 we can see that \(\log(1+e^{x})\) contains \(x + x^2 +x^4\) like terms, thus this loss is convex

x2 = np.linspace(-10, 10, 100)
plt.plot(x2, sigm(x2))
plt.plot(x2, sq_sigm(x2))

[<matplotlib.lines.Line2D at 0x7f950d7f8730>]