I don't think LLMs poor judgement are a replacement for actual scientists yet. Still we should design a scientific system to be 'machine compatible', because it will allow exactly to measure how 'good' these AI scientists are, and profit from it when it gets good, unlike the publication system.

To everyone that says Goodhart's law is reason to not measure scientific progress. If you blind yourself, you accept to rely on feelings, reputation, and social power dynamics. Measures allows to ground decisions in reality. Building good measurement tools is what science is suppose to do.

AI in science is not the problem, just like any other tool. Incentives given by considering papers as unit of work and journals and citations as measure of quality are the direct cause of the current enshitification of science. AI just amplified the means to follow incentives blindly.

Filtering academic works is not only futile, it also drives bad incentives. Whenever there is a filter, a KPI becomes maximising the number of works that passes the filter. Thus quantity over quality, strain on the filter, and you end up exactly in our current peer-review situation.