ABBY

Andy Goldschmidt spent four years working as a data scientist, with a stint at Jimdo, a website builder company, where he ran countless A/B tests. He noticed a persistent problem: while the concept of A/B testing was simple, execution was messy, and documentation was almost nonexistent. "Even though the concept of an A/B test is simple, it's not that easy to do it right," he recalls. Teams across Jimdo were losing valuable institutional knowledge because test results weren't being properly recorded or shared. Andy built an internal tool to solve this—one that could automatically evaluate tests and document results. His teammates loved it; suddenly they didn't need to manually evaluate tests anymore. Instead, they just fired a SQL query and got structured documentation that the whole team could reference and learn from. With his CEO's blessing, Andy decided to spin this tool into a standalone product: ABBY.io.

Andy spent nine months building ABBY, combining his skills as a hybrid data scientist and programmer. The timeline was grueling—he worked on this as a side project with only "a handful of hours per week" to invest. He handled web design, backend development, promotion, and everything else himself. The turning point came after an exhausting coding session when he made an impulsive decision: "Someday after an exhausting coding session, I decided to pull the trigger in a 'f*ck it, ship it' moment." ABBY launched as a free-to-use service, which helped with initial conversions, but there was no pre-launch strategy or marketing plan in place.

Andy's initial growth attempts were modest. He posted ABBY to Twitter, Hacker News, and relevant subreddits, but the traffic was underwhelming. He also ran a Google AdWords campaign using free credits, but keywords related to A/B testing were expensive, so the campaign didn't generate significant clicks. The real breakthrough came unexpectedly: someone hunted ABBY on ProductHunt, which drove about 20,000 visitors and generated roughly 100 sign-ups. While the conversion rate was low due to ProductHunt's diverse audience and ABBY's niche positioning, Andy considered 100 sign-ups a validation success—enough proof that the concept had merit.

The fatal flaw became clear quickly: most people didn't understand why they needed documentation for A/B tests. "To make such a service successful, I would have needed to educate the users, and that's not something you can do before you offer any value to them," Andy explains. ABBY had no immediate value proposition—users would sign up, not see immediate benefits, and abandon the product. Beyond this core problem, Andy faced brutal competition. Google Analytics offered basic A/B testing built-in, Optimizely dominated the serious testing market with end-to-end solutions, and free tools were available for anyone who wanted to roll their own implementation. Andy's biggest mistakes stemmed from his approach: he was unprepared for B2B growth dynamics, had no launch strategy, and tried doing everything himself with limited hours. He spent months building in an echo chamber without outside feedback, only to discover that nobody actually wanted what he'd built.

ABBY ultimately failed. Andy moved on to his current role as a data scientist at AdTriba, a marketing attribution tool. However, the experience was transformative. He learned that validation must come first—talking to potential customers and pre-selling before building anything substantial. "Build it, and they will come" is a fallacy, he now understands. His key takeaway: start small, find people who might be customers, ask if they'd pay for the service, and only then build. "I spent months building a software no one used in the end," he reflects. "That is a depressing experience that you should avoid." The lessons learned shaped his approach to future projects: faster validation, detailed go-to-market planning before launch, and the discipline to focus on marketing as core to success, not an afterthought.