We talk a lot about metrics. And when you do that, there’s always the risk what you’re measuring or why will become unclear. So this is worth repeating, as was reminded in a nice conversation with Anurag Acharya of Google Scholar (thanks Anurag!).

Metrics are no good by themselves. They are, however, quite useful when they inform filters. In fact, by definition filtering requires measurement or assessment of some sort. If we find new relevant things to measure, we can make new filters along those dimensions. That’s what we’re excited about, not measuring for it’s own sake.

These filters can mediate search for literature. They can also filter other things, like job applicants or or grant applications. But they’re all based on some kind of measurement. And expanding our set of relevant features (and perhaps a machine-learning context is more useful here than the mechanical filter metaphor) is likely to improve the validity and responsiveness of all sorts of scholarly assessment.

The big question, of course, is whether altmetrics like tweets, mendeley, and so on are actually relevant features. We can’t now prove that one way or another, although we’re working on it. I do know that they’re relevant sometimes, and I have the suspicion that they will become more relevant as more scholars move their professional networks online (another assumption, but i think a safe one).

And of course, measuring and filtering are only half the game. You also have to aggregate, to the pull the conversation together. Back when citation was the only visible edge in the network, we used ISI et al. to do this. Of course the underlying network was always richer than that, but the citation graph was the best trace we had. But now the the underlying processes—conversations, reads, saves, etc—are becoming visible as well, and there’s  even more value in pulling together these latent, disconnected conversations. But that’s another post 🙂

With most big companies you could hand over your source code and your business plan and they still would not be a threat to you.

–Paul Graham

We’ve been thinking a lot lately about how TI can make money to support infrastructure and future development. We’re still unsure what kind of organisational structure will best serve the goals we’ve got for TI (a conventional startup, a B Corp, a non-profit foundation, or something else?). But we’re increasingly sure that there’s a clear business model to support whatever we come up with, built around selling high-volume access to the data we collect.

In the short term, TI is nicely positioned  at the intersection of social-media analytics (Gnip, Radian6, etc) and scholarly impact (ISI, Scopus, etc). The great things about this confluence from a market perspective are:

• There are a bunch of special academic-specific sources (insulating us from competition from Gnip et al).
• The institutional inertia of big players like the ISI and Scopus makes them unlikely competitors in the short- to medium-term. They see this stuff as toys. Once they don’t, it’ll take time to develop relationships with many providers, relationships we’ve been building for the last year.
• As budgets tighten, there’s a growing clamor for more and better metrics to support  funding applications.  There’s growing dissatisfaction with the IF, but still a culture comfortable with (addicted to?) making decisions based on popularity in social sharing networks (which the literature, at its core, is).

Of course there are challenges here; the IF is a very well-established brand, and citations are the coin of the realm. But there’s also a growing sense that the Web opens exciting possibilities for scholcomm that we scholars are letting slip away.

The scholarly impact market—which ultimately drives the whole scholarly enterprise—is based on 1960s data sources and 1960s technology. It’s ripe for disruption. And selling this new data with a low-risk SaaS model is the perfect way to get that disruption started.

Over the long term, sources like TI can be the infrastructure upon which all of science is built.  Future researchers won’t read journals, but rather customised feeds, filtered using data gleaned from their social networks. As a provider of that data, total-impact sits atop a powerful revenue stream, as well helping to push science into the future.