"Calum Grant" <calum@visula.org> wrote
BTW, the file you provided is very interesting. I played with it yesturday, but so far my results are pretty slow (no wonder -- I have to consider more than 400,000,000 possible combinations). One of possible reasons is the lack of co-processor on my PC (quite a bit of math is done), but I also found some RTL inefficiencies (as I said, we haven't spend a lot of time on the optimization yet). I will do some more research and then provide the results. Will probably take a few days to a week.
You could instead sort on the square of the distance - saving you one square root per iteration, and since you don't need square root, you can just work with integers. I'm just intrigued how you are going to fit 400,000,000 items into an index in memory!!
I never wanted to do this -- I index the resulting groupby, not the crossproduct. But you are right I can do something more intellegent -- this particular task does allow this. Regards, Arkadiy