What might you measure, if not GitHub stars?
Since posting a rant about measuring, and boasting about, GitHub stars, I’ve had a couple of conversations with people about what you might want to measure instead.
It’s a complex question, actually. And here it’s true, it matters whether you’re evaluating business health as a mature business or evaluating the potential of a project for monetization.
In the later example, I think instead of GitHub stars things like:
Community engagement, even just the number of messages on Discord / Slack. I even think raw number of messages is more important than sentiment analysis, because if someone takes time to post a message saying they think the project is crap, that means the might think the project is crap but they care enough about the problem space to be frustrated and to make the comment.
Actual active users. A project/product like Scarf is probably needed to get reasonably accurate info about this.
Activation rate. What percentage of people who download the project go through the setup process and experience whatever magic the project has to deliver, even if it’s only once?
This assumes a certain relationship between project and product, however — one in which the project is an awareness play / marketing, or one in which the value the project brings is in shortening the feedback loop and developing a better project/product. What if the primary value of the project to the company is that you’re selling to companies that value transparency? In that case, all the metrics above are pretty irrelevant.
If you already have a mature open source business… let’s face it, you’re tracking leads and sales conversations and revenue. What you should be tracking in terms of your open source project? It kind of depends on what you see as the business value of the project — is it for lead gen? If so, how many commercial leads say they found you through the project? Is it for feedback? If so, how much feedback are you getting from your open source community? Is it for awareness/adoption? If so… how many users / downloads do you have?
Open source companies are really diverse. It’s annoying to say it, but the right things to measure depend on a lot of factors, and you can’t really evaluate if something is good or bad without some context.