Once the feature goes in, it has to stay there until we all die…

— this is one of “jokes” told by Jared Spool at “Mind the Product” conference, at his speech “Building a winning product and UX strategy from the Kano Model” (full joke starts at 15m 10s):

2 weeks ago I’ve added 2 features to pairwise testing tool: “hotkeys” and “exclude pairs”:

Added hotkeys and “exclude pairs” support to online pairwise tool

I felt fine about “exclude pairs” feature, but was not sure about “hotkeys”. To track features usage I’ve logged requests for clicks and hotkeys and for using exclude pairs and not. So, I’ve got some stats to make the decisions based on data, not on my feelings.

“Hotkeys” feature usage stats

Only 8 times of 300 calls users used hotkeys. Other 292 times they used mouse button clicks. According to rough statistical significance calculator (made by Alexey Eroshenko) I found, that such “conversion rate” for this “hotkeys” feature with 95% probability lays between 1 and 4.5%:

And this value is in the boundaries of “Innovators” (less than 2.5%) and maybe “Early adopters” (less than 13.5%) of “Diffusion of innovation curve”:

Image credit: Wikipedia

This curve says, that “Early majority” and “Late majority” will never use the feature until its usage reaches 16% of market penetration.

That’s why feature either needs better advertisement, explanation, placement, etc. — to pass that 16% threshold. Or it’s better to remove it at all.

“Exclude pairs” feature usage stats

50 times for 329 calls of the app users included something into “exclude pairs” field. Which means, that with 95% probability feature usage (conversion rate, adoption) lays between 11 and 19% — which covers “Early adopters” easily and maybe even passes that 16% threshold:

Conclusion

So, I decided to remove “hotkeys” feature rather than to push it further. This decision will simplify instructions, make button “Generate pairs” more clean and I hope will make overall UX better because of it. It’s not a big deal to click on a button, especially because the tool is used once in several weeks or months.

I like this concept of using “Engineers” and “Data” as the sources of ideas for innovations, rather than “Customers” and “Stakeholders” (starts at 5m 55s):

— so I used “Engineering” (tracing feature usage in logs + statistics science) and “Data” to evaluate new features and to drive the product evolution further. Not by feelings and guesses, but by the numbers behind UX.

If you noticed, that I’ve made any mistakes in my assumptions or conclusions, I’d be glad to read about them in the comments below!

Thanks!