Homework 2

1. (2 points)

You are given a tiny dataset as shown in the figure below. There are two categories of 

data labeled as y=-1 (denoted with cycles) and y=+1 (denoted with diamonds). You are 

required to use SVM to train a binary classifier. Please explain your solution with both 

theoretical derivations and optimization algorithms. (Hint: recall key concepts of SVM 

such as the objective function, dual problem, kernel and the SMO optimization

algorithm. How to put them together to solve your problem).

2.  (3 points)

Given  a  dataset  � = {�(ℓ), �(ℓ)}ℓ$% & ,  we  want  to  train the  following  multi-layer 

feedforward network for multi-class classification:

where

�!(ℓ) = #�!%�%(ℓ) &%'( �'(ℓ) = tanh(�'(ℓ)) �((ℓ) = 1�('�'(ℓ) )'$*

Suppose the activation function of each hidden unit is the hyperbolic tangent ()

function: 

tanh(�) = +!,+"! +!-+"! ,

and the softmax function is applied to the outputs so that yi approximates the posterior 

probability P (Ci | x)： �((ℓ) = exp (�((ℓ)) ∑ exp (�.(ℓ) . )

We train the neural networks by minimizing the cross-entropy loss:

�(�, �|�) = −11�.(ℓ)

log �.(ℓ) /.$% &ℓ$%

We use the simple gradient descent for learning the network weights.

(a) Derive the weight update rule for the second-layer weights vih. 

(b) Derive the weight update rule for the first-layer weights whj. 

Hint: modify the derivation in class which uses the sigmoid activation function.