Yeah, before I pivoted to trying to flip journals, I spent a year exploring crowd sourcing with an eye on improving peer review. After building a beta and collecting a bunch of user feedback, my conclusion is that academics on the whole aren't ready to crowd source. Journal editors are still necessary facilitators and community organizers. So that lead to exploring flips.
However, I think there's a lot that software can do to nudge towards better peer review. And once we have journals using a platform we can build lots of experimental features and make them easy to use and adopt to work towards improving it.
I've kept crowd sourcing preprint review in the platform - though I removed the reputation system since UX research suggested it was an active deterrent to people using the platform - to enable continued experimentation with it. And the platform makes it easy for preprint review to flow naturally into journal review and for the two to live comfortably alongside each other. The idea being that this should help enable people to experiment with preprint review with out having to take a risk by giving up journal publishing.
And the platform has crowdsourced post-publication review as well.
My thought is that if we can get the journals using the platform, that will get authors and reviewers in the platform and since preprint and post-publish review are really easy to do in the platform that will drastically increase the usage of both forms of review. Then folks can do metascience on all of the above and compare the three forms to see which is most effective. Hopefully that can then spur movement to better review.
I also want to do work to ensure all the artifacts (data, supplementary material, etc) of the paper live alongside it and are easily accessed during review. And work to better encourage, rewards, and recognize replications. I think there's a lot we can explore once we have a large portion of the scholarly community using a single platform.
However, I think there's a lot that software can do to nudge towards better peer review. And once we have journals using a platform we can build lots of experimental features and make them easy to use and adopt to work towards improving it.
I've kept crowd sourcing preprint review in the platform - though I removed the reputation system since UX research suggested it was an active deterrent to people using the platform - to enable continued experimentation with it. And the platform makes it easy for preprint review to flow naturally into journal review and for the two to live comfortably alongside each other. The idea being that this should help enable people to experiment with preprint review with out having to take a risk by giving up journal publishing.
And the platform has crowdsourced post-publication review as well.
My thought is that if we can get the journals using the platform, that will get authors and reviewers in the platform and since preprint and post-publish review are really easy to do in the platform that will drastically increase the usage of both forms of review. Then folks can do metascience on all of the above and compare the three forms to see which is most effective. Hopefully that can then spur movement to better review.
I also want to do work to ensure all the artifacts (data, supplementary material, etc) of the paper live alongside it and are easily accessed during review. And work to better encourage, rewards, and recognize replications. I think there's a lot we can explore once we have a large portion of the scholarly community using a single platform.
The trick is getting there.