Neal Becker wrote:

This looks really good. After just a short look, 2 points:

It's great to see that the functions section shows the algorithms used and gives an indication of accuracy. I didn't see this in the distributions though.

Yep, the lack of that info for the distributions has been vaguely nagging at the back of my brain. The implementations are mostly trivially in terms of other functions, so it'll probably just be "accuracy: see incomplete beta" or whatever.

wish list - there is a central chi-square. I could use also non-central chi-square.

I was afraid someone was going to ask for that :-) Paul and I have been deliberately avoiding the non-central distributions ('cos they're hard!), but if anyone has any references / implementation pointers I'll at least take a look and see if it's feasible. Thanks for the comments, John.

Janek Kozicki wrote:

John Maddock said: (by the date of Wed, 4 Oct 2006 18:43:00 +0100)

...

Mathematical Special Functions

how about sigmoid, double sigmoid and their derivatives? Used for neural networks.

Those are new ones to me.

Just a suggestion. They are so simple, that perhaps you can assume that the user can write them himself (as it was done in the past).

Unless you know some optimized way to implement them....

I suspect you will never improve on a call to exp() ? Of course if you want 1 - sigmoid(x) then you can rearrange things to avoid cancellation error. But I don't know how important such things are. One of the problems we have is that there are a virtually unlimited number of special functions, not least all those in TR1. So it's a case of trying to draw the line in the right place, but do feel free to lobby for your favorites :-) John.

John Maddock said: (by the date of Fri, 6 Oct 2006 10:04:56 +0100)

Janek Kozicki wrote:

...

...

Mathematical Special Functions how about sigmoid, double sigmoid and their derivatives? Used for neural networks. Those are new ones to me.

yes, actually there is a wide family of sigmoid functions, they all have a similar shape. However there is one function called "sigmoid" and unless otherwise specified what sigmoid function is used, this one is assumed: sigmoid(x)=1/(1+exp(-x)) Theodore, please note that: (tanh(x/2)+1)/2 = 1/(1+exp(-x)) It's the same function, only written in different way. First parameter to tweak here, is the steep: sigmoid(x,s)=1/(1+exp(-x/s)) But there are other sigmoid functions. Double sigmoid is this: double_sigmoid(x,d,s) = sign(x-d)(1-exp(-((x-d)/s)^2)) d - function centre s - steep factor Actually it's all here: http://en.wikipedia.org/wiki/Sigmoid_function http://en.wikipedia.org/wiki/Gaussian_curve Both sigmoid functions and bell curves (e.g. gaussian) are useful in neural networks. But also it is necessary to have their first derivative. The most popular learning algorithm (called backpropagation) uses a gradient descent in n-dimensional space, and there the derivatives are necessary. Please note that it's the function shape that is important, not its exact vaules at any point. So if you know a 'faster' function that has similar shape and has a first derivative I'd like to know about it too! -- Janek Kozicki |

John Maddock said: (by the date of Sun, 8 Oct 2006 16:12:02 +0100)

Janek Kozicki wrote:

...

Please note that it's the function shape that is important, not its exact vaules at any point. So if you know a 'faster' function that has similar shape and has a first derivative I'd like to know about it too!

I guess if x is small, then you could use a truncated tailor series, and either evaluate it as a polynomial, or maybe even just as:

1/(2 - x + x*x)

but other than that, I'd be surprised if there's anything much faster than exp()?

(to readers: this function is a bell curve.]) hmm.. you are right. My knowledge of neural networks is limited, I haven't had a dedicated course, so it's only what I managed to learn myself from articles I have found. And I learned only enough to get my job done, and have some fun with it altogether. My conculusion is that maybe I'm wrong to say that it is the shape only (although I belive one of the papers I read said exactly that). But then - why people are not using a simple polymonial for that? Oh, I know why.... its integral is this: 2*atan((2*x-1)/sqrt(7))/sqrt(7) (this is a sigmoid function) I belive that backpropagation will work correctly when the derivatives/integrals are exact. Neural networks use both: the sigmoid and it's derivative (a bell curve), and they both need to be fast. Oh, and I found another reason right here in wikipedia :) " A reason for sigmoid popularity in neural networks is because the sigmoid function satisfies this property: sigmoid_derivative(x) = sigmoid(x)*(1-sigmoid(x)) " So I was wrong saying that it's only the shape that is important :) So, are you going to add those? - sigmoid, - sigmoid_derivative, - double_sigmoid, - double_sigmoid_derivative The derivative of double sigmoid can be done as a connection of two sigmoid_derivatives :) Another question: how about Fast Fourier Transform? is it by definition already out of the scope of your library, or maybe not? Or maybe it's already there but I haven't found it? :) -- Janek Kozicki |

John Maddock said: (by the date of Sun, 8 Oct 2006 18:39:07 +0100)

...

So, are you going to add those?

- sigmoid, - sigmoid_derivative, - double_sigmoid, - double_sigmoid_derivative

The derivative of double sigmoid can be done as a connection of two sigmoid_derivatives :)

No, I believe it's outside the scope of the current library: providing support for the statistical code.

Or to put it another way: there's too much else to do!

uhh... so maybe rename component "Mathematical Special Functions" into "Statistical Special Functions" ? I think it was natural for me to search for my favorites inside a component named like that... -- Janek Kozicki |

"John Maddock" <john@johnmaddock.co.uk> wrote:

...

Another question: how about Fast Fourier Transform? is it by definition already out of the scope of your library, or maybe not?

Yes, definitely outside the scope, and also patent protected I believe.

FFT is way too old to be patent protected. There are a number of free, high quality implementations. Cheers, Walter Landry wlandry@ucsd.edu

Topher Cooper wrote:

At 01:43 PM 10/4/2006, you wrote:

...

Documentation and source downloads are available online at www.johnmaddock.co.uk/toolkit

Regards,

John Maddock.

I've just started reading the documentation but I want to point out that phrases like:

"we conclude that there is no significant difference, and accept the null hypothesis"

are likely to interfere with any statistician taking the package seriously (unjustly, I think -- the statistics may be weak but you've obviously worked hard at the numerics, which is what you are supplying).

One never, ever "accepts the null hypothesis." One collects evidence and on that basis you either reject the null hypothesis or fail to reject it. The point is that you don't ever really have evidence *for* the null hypothesis, only a lack of evidence against it. It is quite a different thing to say "any difference in the means in this test is statistically insignificant" than to say "This test gives me an objective reason to believe that the difference in the means is exactly 0 (i.e., the null hypothesis) rather than, say, 1.0E-23 (which is as much a part of the alternate hypothesis as is 1.0E+23)". Or in other words, the lack of evidence of a difference should not be taken as evidence of a lack of difference.

Ah, very good point. Thanks for raising this, looks like another editing session is needed, I'm fairly sure there will be other areas where we've got the terminology wrong as well. John.