What I have learned about Idea Validation

I have ideas. Lots of them. My head is literally overflowing with ideas. We all know, ideas are cheap and execution is everything. I have been spending alot of time recently trying to figure out how to validate individual ideas to give them the best chance of successful execution. All that thinking has been boiled down to the 7 points below.

Full disclosure before we get too far: I have not (yet) made it down the path of enlightenment to a glorious exit or turned a product into a wild river of revenue. But I have had some success getting funded and wanted to share what I have learned so far about idea validation in the hopes that you might gain something from my trials and errors.

<aside>I have said this before, but it is worth repeating. If you are not pitching your idea(s) to everyone and anyone that has a pulse and will listen to you, you will not arrive at a validation or rejection of your idea.</aside>

One final note: Just checking the box for each of the steps below is not necessarily enough. You also want to monitor the quality of the check. Receiving $$ from someone with exceptional deal flow like the Vancouver 3 (Boris, Mike and Jason) or getting those precious $11,000 from Y Combinator is on a different scale than getting a cheque from someone whose main claim to fame is a whole bunch of zeros at the end of their bank balance.

Strive to get your idea in front of people that understand your space, have been pitched to hundreds of times and will ask you the uncomfortable questions that further your understanding of your hypothesis. I enjoy talking to people that bring insight.

I arrived at this checklist by pitching dozens of ideas to hundreds of people. After the list, are 4 ideas I have utilized this methodology with.

1. Have you validated your market with face-to-face Customer Development?

One mistake people consistently make despite volumes of people pointing this error out, is failing to get out of their cave/bubble/tower/office to talk to potential customers about their actual problems. Are you going to revolutionize the Job Market (as is my plan) but have not talked to dozens of recruiters and HR managers about their pains? Turns out I still needed to learn this lesson.

When I shared my idea about incorporating social data into the recruiting process with Andres Glusman (Co-Organizer of the NY Lean Startup Meetup, Meetup Strategy Guy) the first question he asked was whether I had talked to anyone in the field about their pains. My bumbling and blushing response said it all. I should have known better.

The first step to Customer Development is to set up informational interviews and figure out that there is a pain to solve in the niche you are targeting and what your potential customers are currently doing to solve that pain. Similar to doing usability testing, I like to do customer development interviews in pairs. That way one of you can ask questions and keep things moving while the other is free from distraction and can actively listen to the responses. Find 10 potential customers to talk to and make it happen. This is literally ground zero, you need to do this before moving on to any other task on this list.

2. Have you collected data from real target customers?

For a recent idea I had about building an app on top of Meetup’s API (more on this below), I sent out this survey to 150 warm contacts and 3 Meetup groups I help organize (total around 700 members). I also posted the survey on the Meetup forums. Over 100 people filled out survey responses which will help shape the MVP of this product.

I might even argue that if you can not get 100 people to fill out a survey about your potential product you do not have a deep enough understanding of your market niche to be pushing forward. Keep your survey short (5 Qs < 1 min) there is no room for fluff, make sure every question word of your survey is necessary. Ideally, your survey should validate the market pain plus collect info about a minimum feature set.

3. What does the Adwords/Unbounce combo say?

This is a deadly combination to help validate the secondary market (ie. people you do not have direct contact with) plus collect email addresses to benefit your product at launch. Grab a $75 Dollar Credit for Google AdWords, sign up for a free account at Unbounce and listen to what the internet says about your idea.

Is your click through rate abysmally low (well under 1%)? Time to test a few different pains in the niche before you start development. An Adwords CTR of 2-4% is really solid, I have not been using Unbounce long enough to give you an indication of a successful conversion percentage. If you are an Unbounce veteran, can you please post some numbers in the comments?

A negative result here is not a deal breaker. Sometimes people do not know what they want until you can put it in front of them. Or their friends start using it. As The Job-onian One says : “You can’t just ask customers what they want and then try to give that to them. By the time you get it built, they’ll want something new”.

In addition, the Unbounce/Adwords combo can be a great tag team to discover pains in your niche you might not have previously been aware of, just make sure you listen to the data.

4. Can you get someone to dev your product idea for free?

My developer friends are going to hate me for this… but having someone dev your initial prototype for free is a key validation point.

There is nothing developers hate more than greasy business people trying to get them working on their project while promising nothing more than future considerations. Today’s developers are wanted like 2003 Brad Pitt at an over-40s Divorcee Luncheon. Any half decent developer can move to the Valley, immediately start collecting 100K/year and get treated like a rock star. Give me one good reason why should they work on your idea for free?

The answer is to put yourself in a position where people want to collaborate with you. Jason Freedman wrote an absolutely fantastic post on How to Find Earn a Technical Co-Founder. If you are an ‘Idea Person’ looking for dev talent, this is a must read.

If you can convince someone to turn down Rock Star status and join you on the Startup (aka. Ramen) diet, they must believe in the product just as much as you do. Your initial team needs to have this level of buy-in if you are going to have any chance of success.

What if you can’t find anyone to join you in the poor house? Similar to customer development, don’t stop at the first ‘no’. But if you are repeatedly getting rejected or sense a wishy-washy level of commitment from your partner(s), it is time to reshape the plot of the project and question whether you have the right people in the right roles.

There is an incredible Paul Graham post on meta patterns he has found in successful founding teams at Y Combinator. One of the key points of success is a pre-existing friendship. Look around your immediate social circle for someone to partner up with and dev your MVP without laying out cash. If there is no one there, consider growing your social circle before burning your savings on development costs.

5. Do you have a unique competitive advantage?

What would prevent someone else from doing this exact product? And the answer better not be “nothing”. There are two options:

You have key relationships that will help with distribution channels. aka. You can get a jumpstart on customer acquisition. eg. Zuckerberg’s connections at Harvard
You have unique domain knowledge or expertise that you can turn into a piece of technology that is not easily replicated. eg. Google Search Algorithms.

Cue Captain Obvious: ”For best results, have both!”

Update: Tom noticed I was a little light for this section and suggested this great read on: Real Unfair Advantages by A Smart Bear

6. Can you get someone to give you money?

Warning – Achtung! - ¡Atención, Por Favor! Not all money is equal.

There are a variety of benefits and drawbacks to taking money from Friends and Family, Angels, VCs or Government Programs. You need to consider the terms, but perhaps just as importantly what doors can that person or group open for you. IRAP funding can open channels into other Government departments and formal international ties through DFAIT (for us Canadians). However, if that does not match your market, you may want to look elsewhere.

A local incubator like Grow Lab, has strong ties to the Valley and its band of VCs. A well-connected local Angel like Mike Edwards can open doors throughout the North American Tech Community. Mike hooked me up with Meetup Co-Founder Matt Meeker, someone who has an incredible understanding of the space I am trying to enter and a person I may have struggled to connect with on my own. If you are going to take money, try to align it to access distribution channels or contacts in your target market.

Another note, to convince a savvy investor to cut you a cheque, you are going to have to show you understand and have validated your market. Dozens of people like you turn to them for money each month, make sure you have a compelling argument why should they put your name on the next cheque they write.

7. Would you use your product yourself?

To me, this is the most important point. Look yourself in the mirror and give yourself a healthy dose of brutal honesty. If you are not solving a pain in your life, it is going to be really hard for you to get out of bed each and every day and dedicate yourself to pushing your product forward.

Are you prepared to do that for your idea? The path of the entrepreneur is not easy, there are pitfalls and challenges at every turn. You will need to have a sheer, blazing and uncompromising belief in your product… and yourself.

Be honest, does your idea give you that level of confidence? If you have a shaky response now, just think what happens when your lead dev quits days before launch or a key partner bails on you to sign with the competition. Almost every story in Jessica Livingston’s Founders at Work has a The-World-Just-Fell-Out-From-Beneath-Us Moment. Shit will happen. When it does, are you going to pack it in or will you have the audacity, confidence and will power to stay the course?

Finally, the Ideas:

The 7 points above do not guarantee success, but they have helped guide my idea validations. Lets move on and apply this framework to four ideas I have recently been involved. These are all works in progress, I will endeavor to update the details as progress is made.

B-Liner

Update: This died. There is nothing to see here, move along.

Idea: Foursquare for public transport. Users would be automatically checked in to specific bus routes, frequent riders would become ‘Kings’ of that route and unlock badges and rewards like “Carbon Saver – You Rode 30kms on the bus and saved X number of trees”. Initial market is university students on the 99 B-Line (for out of towers, the 99 is the most heavily used bus route in Vancouver). The goal is to get more people using public transport.

Possible revenue generation: 1. Ultra localized deals. If someone gets on and off at a specific bus stop everyday, the businesses in that immediate area would be interested in acquiring that customer. 2. With enough users, you could get up-to-the-minute traffic flow data and sell that back into GPS companies.

1. Customer Development - I conducted customer interviews while riding the bus. In general, the response was positive. However, when we pitched the idea to people that had a good understanding of the location-based space and check-in models the response was much less receptive. Dot in particular was a solid dose of reality. She had previously worked at a Valley Accelerator that worked extensively with teams working on location-based checkin apps. Her first hand knowledge that this space was:

It is near impossible to gain traction
People were not ‘checking in’ with significant volume

Her feedback helped us make the decision to pause development until we had a better understanding of our potential users.

2. Collect Data- Not yet conducted

3. Adwords/Unbounce – Not yet conducted

4. Get someone to work on your product for free - This product idea has appeal as a social cause. Two extremely talented people: Dave Emmett (UI/UX) and Vasili Sviridov (Backend) stepped up to contribute.

5. Do you have a unique competitive advantage? No. There is no key relationship or unique domain experience here. The three of us take the bus, but so do hundreds of thousands of other people. The technology itself can be replicated by any of thousands of local developers. We could potentially build relationships with Translink, but there are no preexisting competitive advantages, they need to be built.

6. Get someone to give you money – Not yet conducted. This is the kind of initiative that could receive support from local, provincial or federal governments. A key distribution channel would be Translink. Establishing a partnership there would be key to success of the project. I think you would be hard-pressed to raise investment on this product without widespread adoption.

7. Would you use it yourself? I had to be honest with myself, although I felt I would use this app I was not sure I could sell it to the level that was required to make it a blazing success. To me, this is a passion or dabbler side-project, not something that I could hustle full-time.

Result: This project is on standby. It was clear this was a good team and we did not want to siphon off energy to push forward on a project that wasn’t fully baked. I still think there is something in this space, and that gamification of public transport could increase ridership and have a positive social effect on the city. I am also looking forward to working on *something* with Vasili and Dave in the future. I think we all want to make sure whatever we choose to work on has the best chance of success.

WeAreVanTech

Idea: A blatant rip-off of WeAreNYTech.com. Create a story showcase of local talent in the tech industry as a focal point for people new to the city or local companies looking to connect with local talent. The differentiator to this site is that you actually conduct interviews with every person posted to the site. This gives you unique and interesting content that cannot be found elsewhere (great for Longtail Search of individual people).

Possible Revenue Generation: WeAreNYTech shows the way. They released a job board that had 37 job posts on its first day. This is a fairly recent idea, hence not a lot of data below. I will update as progress happens.

1. Customer Development - Not yet conducted

2. Collect Data - Not yet conducted

3. Adwords/Unbounce - Not yet conducted

4. Get someone to work on your product for free - I sent out this tweet and almost immediately had responses from two top-notch developers (and people): @ujm and @harph. That is a good sign.

5. Do you have a unique competitive advantage? Maybe. Successful execution of this project is going to be data dependant. Can we get lots of local tech people to take a couple minutes for the interview to help us build a great data set? That is the kind of networking challenge I am well suited for and really enjoy.

6. Get someone to give you money? Not yet conducted. I could see government funding for part or all of this project as it showcases all of our local talent. That will be the first avenue explored if this moves forward.

7. Would you use it yourself? A talent map of the people in the local tech community is something that I would absolutely utilize. There is also a pain point there that you are solving for others who are new to the community or are looking to fill a specific role inside their organization.

Result: I am going to meet with the people who created WeAreNYTech. Depending on the outcome of that we will move back into the top of the customer development cycle. But its great to know it sparked an interest amongst several key people in the community.

Wicked Awesome Meetup App

Idea: A service that gives you information about other Meetup attendees. Target Market: Meetup Power Users and Organizers

1. Customer Development - I began pitching this idea to dozens of people at various Meetups over the past several weeks and the response was mostly positive. People at meetups are potential users of this product, so it was a nice fit.

2. Collect Data - Survey conducted with over a hundred responses. These results will shape the core feature set of an MVP product.

3. Adwords/Unbounce - I have not used AdWords yet to test this idea, but the Unbounce landing page is converting at nearly 50% with highly qualified traffic from personal contacts and Meetup forum visitors.

4. Get someone to work on your product for free - I am going to build the initial prototype myself using jQuery Mobile. I have had offers from several developers to contribute to the project and will move with that once I hit the upper limit of my current skill set.

As a partial aside: Nitobi’s CEO Andre Charland recently blogged about Dreamweaver 5.5 support for jQuery Mobile and PhoneGap. That means you can use a WYSIWYG editor to write code once and release native apps for iOS, Android, Blackberry etc. This will dramatically reduce the barriers to entry in the Native App market and allow people like me to put products in the app store by ourselves. Kind of scary…

5. Do you have a unique competitive advantage? I have several key relationships with Meetup that could potentially help with distribution or provide really solid feedback. In addition as an organizer of 5 meetup groups, I have access to a pool of direct feedback and potential beta users and have a great understanding of my target audience.

6. Get someone to give you money - I am holding off on this until I have a better understanding of the market and an MVP product in place with a solid base of users.

7. Would you use it yourself? - I am solving several of my own pains as a power user of Meetup. This one is a big ‘Yes’. More importantly, I can see myself selling this product and enjoying it.

Result: The response to this product idea has been positive. It shows promise amongst Meetup power users, especially organizers. More importantly, organizers are already accustomed to paying for Meetup. There are plenty of details to work out, but this is an idea I will push forward with.

TouchKeyword

Idea: A one-touch mobile tool that would make keyword research accessible for everyone. Pain I was trying to solve is the high cost of hiring someone to perform your keyword research ($80-100/hour) or the time costs associated with learning to do it yourself. Target market: Mom and Pops with their own personal website or e-shop. Secondary market: Professional Search Marketers.

1. Customer Development - I started a Meetup group around Search Engine Marketing. I ran two sessions about Keyword Research that each had about a dozen attendees. The crowd was mostly noobs, which matched with the initial target market. I showed them some of the current tools (Market Samurai, AdWords Keyword Tool etc.) and then watched where they stumbled (too much data, too much thinking). I then partnered with The Rad Vlad and we built a tool that alleviated those stumbling points.

2. Collect Data - As part of the ACETECH program, I conducted 10 interviews with potential customers as data points. However, I did not collect any survey data.

3. Adwords/Unbounce - I ran an AdWords campaign to test pricing strategies. The click through rates were under 0.1%, which is scary low. The TouchKeyword Unbounce landing page is currently converting at about a 10% clip. Those numbers are both low, however we need to invest some more time to test whether it is a poor job on my part with the campaigns or the market is actually not there.

4. Get someone to work on your product for free - Radu and I worked on TouchKeyword evenings and weekends for 3 weeks (total dev time ~40 hours) and entered it in Sencha’s Mobile App Development contest. We split $2000 in prize money.

5. Do you have a unique competitive advantage? – I have domain experience in Search enabling the backend algorithms and have several key distribution channels as a result of my volunteer efforts with IIMA.

6. Get someone to give you money – I was fortunate enough to get $15,000 in funding from IRAP to continue this project.

7. Would you use it yourself? – Doing keyword research was a part of my job. A part of my job that I did not enjoy. I was stoked to have a tool that fully automated that time-consuming task. It solved a pain of mine and I would use it.

Result: This project is ongoing. There has been a positive response from people in the search community. We are currently testing whether our initial target market is valid before continuing to develop a feature set.