You may want to have a look at these as they have similar goals to yours:

http://openwebindex.eu/en/

https://deusu.org

Sourcecode for deusu.org is on: https://github.com/MichaelSchoebel/DeuSu

OpenWebIndex is only in the idea stage as far as I know. Deusu has been running for over a year and already has about 1.2 billion pages in their index.