Janek Kozicki wrote:
My conculusion is that maybe I'm wrong to say that it is the shape only (although I belive one of the papers I read said exactly that). But then - why people are not using a simple polymonial for that?
Right after I posted, I thought "why not use a rational approximation", or better a polynomial approximation: the derivatives of the latter are of course trivial. I did a quick Google and found http://lib.tkk.fi/Diss/2005/isbn9512275279/article8.pdf#search=%22sigmoid%20... Which is basically about implementing the sigmoid in hardware, but Eq15 suggests that: 1 - 0.5 * (1 - (0.25 * x)^2) is a viable approximation. If ldexp is faster than a multiply then the multiplication by powers of 2 could be replaced with ldexp.
Oh, I know why.... its integral is this:
2*atan((2*x-1)/sqrt(7))/sqrt(7)
(this is a sigmoid function)
I belive that backpropagation will work correctly when the derivatives/integrals are exact.
Neural networks use both: the sigmoid and it's derivative (a bell curve), and they both need to be fast.
Oh, and I found another reason right here in wikipedia :)
" A reason for sigmoid popularity in neural networks is because the sigmoid function satisfies this property:
sigmoid_derivative(x) = sigmoid(x)*(1-sigmoid(x)) "
So I was wrong saying that it's only the shape that is important :)
So, are you going to add those?
- sigmoid, - sigmoid_derivative, - double_sigmoid, - double_sigmoid_derivative
The derivative of double sigmoid can be done as a connection of two sigmoid_derivatives :)
No, I believe it's outside the scope of the current library: providing support for the statistical code. Or to put it another way: there's too much else to do!
Another question: how about Fast Fourier Transform? is it by definition already out of the scope of your library, or maybe not?
Yes, definitely outside the scope, and also patent protected I believe. John.