Skip to content

Caffe:Api:ReLULayer

Rectified Linear caffe layer class.

ReLUParameter

negative_slope [default 0]
specifies whether to leak the negative part by multiplying it with the slope value rather than setting it to 0.

Protobuf example 

layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
  relu_param {
    negative_slope: 1
  }
}

Math

  • 방정식 (Forward)

\(f(x) = \left \{ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ x & \mbox{for} & x \ge 0\end{array} \right.\)

  • 미분 (Backward)

\(f'(x) = \left \{ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ 1 & \mbox{for} & x \ge 0\end{array} \right.\)

C++

  • 방정식 (Forward) 
// n: Negative slope
top[i] = max(bottom[i], 0) + n * min(bottom[i], 0);
  • 미분 (Backward) 
// n: Negative slope
bottom_diff[i] = top_diff[i] * ((bottom[i] > 0 ? 1 : 0) + n * (bottom[i] <= 0 ? 1 : 0));

Graph

Graph

Python source code

Matplotlib-caffe-relu.png 
import matplotlib.pyplot as plt

plt.figure(1)
plt.subplots_adjust(hspace=0.4)

plt.subplot(321)
plt.plot([-2, -1, 0, 0, 1, 2], [-2, -1, 0, 0, 1, 2])
plt.grid(True)
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.title('n=1 forward')

plt.subplot(322)
plt.plot([-2, -1, 0, 0, 1, 2], [-1, -1, -1, 1, 1, 1])
plt.grid(True)
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.title('n=1 backward')

plt.subplot(323)
plt.plot([-2, -1, 0, 0, 1, 2], [0, 0, 0, 0, 1, 2])
plt.grid(True)
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.title('n=0 forward')

plt.subplot(324)
plt.plot([-2, -1, 0, 0, 1, 2], [0, 0, 0, 1, 1, 1])
plt.grid(True)
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.title('n=0 backward')

plt.subplot(325)
plt.plot([-2, -1, 0, 0, 1, 2], [2, 1, 0, 0, 1, 2])
plt.grid(True)
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.title('n=-1 forward')

plt.subplot(326)
plt.plot([-2, -1, 0, 0, 1, 2], [1, 1, 1, 1, 1, 1])
plt.grid(True)
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.title('n=-1 backward')

plt.show()

See also

Favorite site