Why metrics don’t measure the quality of science, and why they corrupt it

This was my contribution to a webinar about "impact", It was ornganised by NFAIS. The slides can be dowloaded at nfais.membercl...
After agreeing to talk, I discovered that NFAIS is a commercial organisation that charged $195 to participate (what?!): www.nfais.org/i...
But they did supply an mp4 after the event, and this is my bit, including the Q & A.
The whole event can be seen at fpdl.vimeocdn....

[@kaleida31]

I found this presentation generally on the spot and really not that provocative from a ’metric point of view. What I found most interesting was the focus on individual cases and the fact that researchers that are recognised as successful (in terms of prizes or general recognition) would not have met simple (and stupid) baseline counts for performance as demanded by both internal and external forces in academia.
But most (if not all) bibliometricians are aware and of the shortcomings of the H-index, JIF, the skewness of citation distributions and other simple measures and would never use them in evaluative practice (or when they do, in very controlled and modest forms ((really!)). Instead it is more likely that it is lazy administrators or cheap leaders in academia that would carry out simple DIY-bibliometrics for the detriment of scholarly practice in the local setting in the short run and as a potential negative consequence in research in general.

[@DavidColquhoun1]

I certainly agree that the main culprits are senior academics, who are sufficiently innumerate to take these numbers seriously.
But bibliometricians must also take some of the blame for propagating silly metrics. And so must the corporations who sell them All universities have to do is to stop buying them and they'd go away.

[@beyder1]

thanks for talking about this important topic. as junior faculty it can sometimes feel like I am a slave to the impact factors... and it really hurts curiosity-driven science. but since metrics are not going to go away, is(are) there metrics that is (are) meaningful?

[@DavidColquhoun1]

I know of no way to measure the quality of research without reading it. In fact, as I point out in the talk, even reading it may well not allow one to evaluate the eventual importance of the work. You may need to wait for 20 or more years to evaluate that.
Scientists should be the first people to admit that they don't know, that they can't measure something in a satisfactory way. The fact that so many don't admit it is just a sign of the corruption of science.

[@beyder1]

Thanks for the reply - totally humbled :)
it must be frustrating though to be in the quantification business and not have an ability to quantify something. but as you see here in US healthcare, if the stakeholders don't do someone else will...