1. I replied to the parent comment here; our answer to the scale problem is to recognize that people doing web scraping are as decentralized as the police. Our goal is to empower people who have questions about the police to answer them.
2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere.
3. This is a big, big question. Right now, the answer is dependent on the use case. Rather than trying to make the world's biggest database, we're going to respond to community needs and build this kind of thing as it comes up.
4. https://measuresforjustice.org/ is doing something like this! We're interested in creating incentives for police departments to make their data more accessible and transparent.
Not to be too rude/negative/mean - but it seems like a big concern to me that a "police data" project that's existed for three years, has a couple employees, has some funding, etc - doesn't really seem to have any police data. If I wanted to write a scraper to gather documents off of police websites and run it by myself and store my results locally - I could. What does your project add?
What I would expect to see is something like:
1. Here's what data we want from each police department each day. Here's what value you should use to indicate that data is not available.
2. Here's a list of police departments. Write a scraper. If it passes tests to show it's generating valid data, and code review, we will run the scraper in a daily basis storing the output in this database.
It's ok, this feedback is how we understand what our work looks like to other people, and how we improve!
We've only had paid staff for about 6 weeks. We've run several experiments and started from scratch a few times over the years. We've been slowly inching the project forward in our spare time; I was the only volunteer for much of that time, and I can't even code!
1. Sounds great, we're building something like this.
2. Even making a list of police departments is a big challenge. We've made a good start but have work to do.
2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere.
3. This is a big, big question. Right now, the answer is dependent on the use case. Rather than trying to make the world's biggest database, we're going to respond to community needs and build this kind of thing as it comes up.
4. https://measuresforjustice.org/ is doing something like this! We're interested in creating incentives for police departments to make their data more accessible and transparent.