Excellent discussion - and the application area (cross-correlation of sensor network data) is actually quite a bit more popular than one might think! Avalanche/landslide detection, earthquakes, bridge monitoring, industrial settings...
On a tangent, if those cross-correlations are big-ish a frequency domain (FFT) correlation could be more efficient, assuming that hasn't already been tried. This would probably save storage memory as well, though I am not 100% sure. If you could get the sensors to send the FFTs directly that could be even better!
This also has echoes of a previous HN discussion on space filling curves [0], which led to the purchase of this book [1]. I still haven't dug into it yet, but maybe these are useful references for others who are interested.
Original author here. Thanks for the spacial data structures link, I'll look into it.
The correlations are done in the frequency domain. And, although I didn't look into it, I suspect they have pretty good algorithms to compute the correlations themselves. At least they have very smart people working on that.
On a tangent, if those cross-correlations are big-ish a frequency domain (FFT) correlation could be more efficient, assuming that hasn't already been tried. This would probably save storage memory as well, though I am not 100% sure. If you could get the sensors to send the FFTs directly that could be even better!
This also has echoes of a previous HN discussion on space filling curves [0], which led to the purchase of this book [1]. I still haven't dug into it yet, but maybe these are useful references for others who are interested.
[0] https://news.ycombinator.com/item?id=7480857
[1] http://www.amazon.com/Space-Filling-Curves-Introduction-Appl...