Placeholder Image

Subtitles section Play video

  • So this is going to be the first of four lectures on how to build a great product.

  • I apologize because none of the people here today can speak to that topic.

  • >> [LAUGH] >> [LAUGH] My name is Michael.

  • I'm the CEO of YC and help run the batch and

  • program as most of you understand YC to be.

  • This here is Steve who's the co-founder and CEO of Reddit.

  • And there's Emmet who's cofounder and CEO of Twitch.

  • Can you guys introduce yourselves in a couple of seconds?

  • >> You just did but I can do it again.

  • >> [LAUGH] >> And Steve, yeah co-founder and

  • CEO of Reddit.

  • I study computer science.

  • I graduated from UVA in 2005.

  • I was at Reddit for five years, I left for five years.

  • I've been back for about a year and a half.

  • >> My name is Emmet I graduated from Yale computer science in 2005.

  • I started my first company right out of college.

  • We made a bad copy of Google calendar before Google calendar existed.

  • And- >> Google calendar

  • >> Yeah, that's right, and

  • then we started Justin TV which is the company that one turn and

  • so I've been doing, basically continuously employed

  • working on that since 2006 >> And

  • then can you just, each of you guys say whatever the KPI you guys track for

  • the business and approximately what it is?

  • Just to give folks a sense of why you guys might be

  • experts at building great products.

  • >> [COUGH] One of our more shareable ones is monthly active users and

  • we hit 300 million last month.

  • >> Nice. >> [LAUGH]

  • >> That's a good number.

  • >> Our primary metric that we care about is a metric called minutes watched,

  • the number of minutes of video that are consumed every month on Twitch.

  • And we just crossed 30 billion minutes.

  • This year for the first timers.

  • This is very exciting.

  • >> Total? >> No, per month.

  • >> Jesus.

  • >> [INAUDIBLE] >> So

  • that's the main thing we look at rather than uniques.

  • >> So, what we're going to do for

  • you guys today is I have about 12 questions I'm going to ask these guys.

  • We're going to see how many of them we get through.

  • And then we're going to leave the last ten minutes for

  • questions about start up stuff, YC, or whatever else you'd like to talk about.

  • My goal is to kind of focus this advice on early, early stage.

  • So, these guys do a lot of stuff now that really

  • is completely irrelevant to you >> But hopefully they'll be talking to you

  • about the things they did, bad or good, when they were starting.

  • So, I want to start with one of the major tenants of

  • YC is how you talking to your users.

  • And we repeat that a bunch of times, and then we don't really go into any detail.

  • So, can you guys talk a little bit about how you talk to talk to your users?

  • Maybe the good and maybe some of the mistakes you've made as well.

  • >> Sure, so, I, most of the advice I give today will have two perspectives,

  • one at Reddit and one at Hipmunk.

  • because the companies are completely different and had different paths and

  • I think, kind of the ground rules for this discussion,

  • the first piece of advice I would give is every company is really different,

  • and there are lots of different paths you can take.

  • And what works in one place, very likely won't work in another.

  • So at Reddit, I mean the product is talking to users, right?

  • That was built in.

  • In the early days we didn't have comments and we didn't have communities.

  • So actually, we got a lot of emails, was our,

  • for the first probably six months was the way we communicated to our users.

  • But when we added the comment feature,

  • there was basically no option but to talk to our users.

  • And the challenge then becomes which users do you want to listen to?

  • And what are they actually saying?

  • And what do they actually mean?

  • Because they're often saying one thing and they mean something else.

  • At Hipmunk, because it was selling plane tickets and hotel reservations.

  • It was completely different.

  • We didn't really have a good forum for connecting with our users.

  • We used a product called Olark which was basically this imbedded Java Script

  • chat thing.

  • And that was indispensable, we still use it.

  • It was users, if they got stuck, they could ask us for help.

  • If they wanted to give us compliments or criticisms, they could do that.

  • And we can often, both over email at Reddit and over Olark at Hipmunk,

  • turn Like emotional users, angry users will often become your most loyal

  • users if you can flip them around, there's like this absolute value of emotion.

  • And so, if you can find those angry users that are having a bad time and

  • treat them well they will likely.

  • Quickly turn into some of your most loyal supporters.

  • So, we found that whole cycle to be very valuable.

  • >> The interesting thing for me, between Twitch and Justin TV,

  • which are two very similar product right there.

  • With live video products with chat.

  • So for Justin TV, we were our own users.

  • We were building a product for us to build a TV show.

  • And after that,

  • we really sort of declined to actually speak to users in any significant way.

  • And, unsurprisingly, all of the good ideas we had were things we came up with

  • when we were building it for ourselves.

  • Which sort of brings me to my first point, of talking to users, is it's possible,

  • especially very early, to build a product yourself.

  • And, if you truly are building something that is for

  • you to use, that can be super effective.

  • You don't necessarily need to talk to talk to anyone else.

  • Eventually, you'll need to, because you'll learn about people other than yourself but

  • for the first three months it's to develop something, just something that you want.

  • But you have to actually really want it.

  • It can't be, you can't be like lying to yourself about that,

  • it can't be something you think is cool.

  • It has to be something you, yourself actually desperately want to use everyday.

  • And then we had went through a long period in the desert where we were

  • wondering about.

  • Not talking to users, not really using our own product, and

  • not building anything of value.

  • And the thing we did with Twitch that I thought was really good for

  • production users is we talked to them in person.

  • I had done some amount of surveys and talking to users over email.

  • And I found that getting people on, at least on Skype, or if not,

  • literally in person changed my depth of understanding of who they were.

  • The other thing I would say about about it is that we focused down really hard on

  • which of our users we cared about.

  • And we decided it was the broadcasters and

  • not only was it the broadcasters, it was the successful broadcasters.

  • We didn't want to talk to people who had two viewers.

  • We wanted to talk to people who had 200 views.

  • And so, we went and talked to a bunch of broadcasters on the platform who

  • decided they were the important ones.

  • The other thing we did that was I think really valuable is we decided it wasn't

  • just people who were currently using our service it was people who we wanted to use

  • our service.

  • And so, we talked to a bunch of people who were streaming on other platforms rather

  • than just already on Twitch.

  • Because the people who have chosen to not use your product,

  • especially if they have deliberately looked at your product,

  • decided it sucked and went somewhere else, are like,

  • some of the best people to talk to, because they know what's wrong with it.

  • And so, I actually think the most valuable things I learned, especially early on,

  • were from users who had tried Justin TV, they had tried UStream.

  • They tried own 3D, they tried YouTube, and

  • they had a coherent opinion as to why ours was not the one they chose.

  • And that was super valuable,

  • because they really pointed at the details of where we were wrong.

  • >> And often those details are smaller than you think.

  • >> It's like memorably one of the biggest issues was we were

  • not listed on the Team Liquid Starcraft fan website.

  • We lost maybe 3,000 broadcasters over that.

  • We had to email them and ask them to re list us,

  • because they just didn't have us there.

  • Tiny issue, but like really mattered to our users.

  • >> It's easy to run a company for years and then be like,

  • I should have fixed this stupid one day fix if you don't talk to your users.

  • >> So a lot of people talk about MVPs, minimum viable products.

  • How many of you have heard that term, MVPs?

  • So one of the challenges I have is when folks like you apply to YC and

  • get into YC even though you know the idea that you should be releasing something

  • that is crappy, you don't do it.

  • And as a result, users slow down.

  • And oftentimes, that prevents you from getting into YC or

  • prevents you from building and talking to customers.

  • And so, I think that what's interesting is that just TV and

  • Reddit are pretty good examples of crappy MVP's.

  • So, can you guys describe how the products worked in the beginning or

  • maybe how they only kind of worked?

  • >> Yeah, we started, I think I wrote my first line of code on June 3rd or

  • 4th, 2005 for Reddit and we launched on June 22nd.

  • I didn't even launch it.

  • Paul Graham linked to us from his blog without telling me.

  • >> [LAUGH] >> And from then,

  • it was on and the good thing about that for

  • us in particular is we didn't actually have a vision for Reddit for

  • better or for worse which would affect us in a lot of different ways going forward.

  • But as soon as we launched, that's when we started to find the path and we just would

  • kind of follow the users everyday as what do we think is best for the users?

  • What do we think is best for us?

  • And just build towards that and

  • we've built a lot of loyalty through those actions and

  • that would not have happened, if we were sitting in our apartment not launching.

  • Because if we launched and then for the next six months, we just added a ton of

  • new features, only about 25% of which lasted more than a day or two, but

  • there was stuff we though was a great idea.

  • Like we had these categories, I remember in July at some point.

  • I made Alexis my co-founder stay up all night one night categorizing every link

  • that'd ever been submitted to Reddit.

  • And the next morning, I was like we lost.

  • It's not working turn it off, but we could've probably, if we had launched,

  • we could have had that feature until November and then launched and

  • then wondered why it didn't work.

  • >> I think that Amazon actually has a really good saying about mineral and

  • viral products.

  • They like to call them, what they're aiming for

  • a minimum remarkable product which I think gets more at what you're trying to

  • accomplish which is a viral product bring into mind this whole ecosystem of support

  • like a viable organism that's got all the things it needs to live and

  • you don't actually want that.

  • Like if you're launching something that's actually self-sufficient, that's viable in

  • a this thing could live on its own sense, you definitely waited too long.

  • This thing is going to be on live support when you launch it and

  • you are the life support system.

  • What you actually want is you want something about

  • the product when you launch it to be actually good.

  • You want to get to the part where you've built something that maybe at least one

  • person in the world finds remarkably useful or remarkably good and you don't

  • know really if you've gotten there until you put it in front of someone,

  • which is why you need to launch it so quickly.

  • I think it's really important to remember Dropbox, for example,

  • one of the big YC successes did not launch an immediate product, because they were

  • building a backup system and backup systems aren't allowed to lose your data.

  • [LAUGH] And so, the last thing you want to do if you're building backup software is

  • go rush something out that's going to lose your customer's data,

  • because you will never get that trust back.

  • And so, the minimum remarkable product for Dropbox was a video.

  • Basically, saying, look at this awesome thing we were going to build and

  • getting all the senate for a newsletter and that worked, because the idea like

  • the visual idea of what Dropbox is going to do was super compelling.

  • That would not have worked for just in TV.

  • We really need to get something out in front of customers Immediately.

  • Most importantly, because we all need to push the show in this idea we are going to

  • build the show and we basically did the minimum possible work

  • may be a little less than that to get our show on the internet.

  • And when we launch it, it like could only have one channel.

  • The way that broadcasting worked, the only way we could

  • actually get it to work consistently is we had a Mac Mini

  • running Parallels which is a virtualization environment for

  • Windows XP where we had this piece of a broadcasting

  • software called, it had got bought by Adobe later.

  • Onflicks?

  • OnLot?

  • I can't remember.

  • Anyway, that we were automating with Windows shell scripting

  • to pick up the right camera input and then auto reboot if it went down,

  • because it broke all the time because it was really buggy software and

  • that extremely jury-rigged thing worked.

  • It was like put a video stream on the internet and it was not a scale.

  • We could not scale that up to thousands of streams.

  • It was impossible, but we got something that at least was compelling to one person

  • onto the internet, that was compelling to us and that was a critical step for us.

  • because before, we did that.

  • We basically weren't learning anything.

  • >> At HitMonk, we again, started in June 2010 and

  • I'd given my co-founder, basically yhis ultimatum of what our MVP was.

  • We have to have legit data, because we could scrape anybody.

  • We needed legit data, real data.

  • And when a customer buys a flight, we get revenue.

  • And I'll say, if we can do that in three months,

  • we will continue with this business.

  • And if we can't launch that in three months, we're calling it.

  • And that for me, that MVP, that was that minimum useful thing.

  • Like if we had a customer who was giving us money,

  • that meant we did something right.

  • And unfortunately, we actually achieved that in three months, so

  • I worked on it for five years.

  • >> [LAUGH] >> But

  • I think it's good, sometimes to set yourself deadlines and specific

  • milestones to keep you focused and keep you pointed in the right direction.

  • >> So another thing that comes up a lot and this is actually one of

  • the things that hits the, when a company gets into YC is we will often ask them,

  • what are they doing for analytics?

  • What are they tracking?

  • And I think that this is another one of those MVP like things, everyone knows they

  • should be doing it and it's the first thing that gets caught off of any list.

  • So, can you guys talk about how an early stage startup

  • should think about measuring?

  • What analytics tools should they be using?

  • And generally, like how you incorporate tracking into building a product?

  • >> I have two very different answers, so reality is somewhere in between.

  • At Reddit, we just didn't.

  • We didn't know our traffic for years.

  • We didn't measure anything.

  • We did the product development by intuition for a very long time.

  • The company ended up working out.

  • I'm still now quite sure how many users we have and it's a huge,

  • huge pain in the **** now.

  • >> How many users you have?

  • When you get to 300 million, it's like- >> [LAUGH]

  • >> No, in all seriousness,

  • we have no idea how many unique people use the site every month.

  • >> It's probably somewhere between 250 and 320.

  • Yeah.

  • >> Anyway, at Hipmunk we were actually really good about it from day one.

  • And we were really because we didn't have that kind of instant feedback with

  • the users we actually got really disciplined about testing, and analytics,

  • and measuring, and watching.

  • And that company lasted honestly,

  • quite a bit longer than I think it would have without doing that.

  • And so if I can do it again,

  • there are a lot of corners you can cut when you're building your MVP.

  • You can create all kinds of tech sins and corporate sins and I don't know,

  • be completely dysfunctional, your users don't care.

  • But data debt can haunt you.

  • because you can't make up missing data, and

  • eventually your intuition's going to fail you.

  • So I would think very carefully about what's the minimum viable data.

  • You don't even have to look at it.

  • Just log it somewhere so it's there when you need it.

  • >> To reinforce that point, having historical baselines for

  • important user actions, you will thank yourself later.

  • Every time I've ever been like, right, how many people are using that feature?

  • And we don't know, and we can't find out.

  • And now even if we find out how much it is now, we don't know if it's up a lot, or

  • down a lot.

  • And we have no idea what the trends are.

  • It's awful and you can't make a decision for

  • another three to six months as you build that baseline.

  • Actually I got the best advice about this from Sue Hale, who runs Mixpanel,

  • when he was convincing us to run Mixpanel for the first time.

  • Which, by the way, didn't work, because Mixpanel at the time was only six month

  • old, and it broke when we tried to everything.

  • But, we went back to it- >> Thanks for

  • convincing me to use Mixpanel before you came to that conclusion.

  • [LAUGH] >> You're welcome.

  • >> It works better now.

  • >> It works a lot better now, we went back to it, we're still on Mixpanel today.

  • We use a bunch of other stuff to, because we're a little too big for just Mixpanel.

  • But Mixpanel, the advice Sue Hale gave me that was,

  • I think, more important that the software we were using.

  • Mixpanel in specific, even though I do think it's good software was,

  • pick your top five to seven most important user actions, whatever those are, and

  • just log those.

  • You don't need to log everything, you really don't.

  • Most of the things people are going to do with your product aren't important.

  • There's maybe five things people can do.

  • Like for Reddit, Reddit might be vote on something, comment on something,

  • submit a story, load a page, I don't know some short list of things.

  • And those are important and you can kind of just ignore everything else.

  • If they're not instrumented who cares.

  • And so we instrumented, I watched a minute of video,

  • I sent a chat message, I followed a stream,

  • and I bought a subscription.

  • And that turned out to just give, those four actions gave us a huge amount of

  • insight into what was going on with our customers, that we hadn't had before.

  • >> This is one of those areas where, if I could go back and give myself just advice

  • for 30 minutes, that would dramatically affect the trajectory of the company,

  • it would be around this topic.

  • And so if I were in your shoes, I would just take half a day, and

  • just read best practices for collecting events and storing them.

  • >> But and also, I know another thing I'd recommend is use a third party service.

  • Use, collect logs, by all means, put them into your own system.

  • because I guarantee you,

  • you'll want to do something a third party service doesn't allow at some point and

  • you'll be super annoyed if your data's only in the third party service.

  • So collect logs or whatever, but just in the beginning,

  • you don't have time to mess around with building analytics tools.

  • So just use Mixpanel or use Google Analytics or use whatever.

  • Anything, they're all better than what you need right now.

  • Later on you'll need something more complicated.

  • You don't need it today.

  • >> So one of the challenges that happens usually after a company,

  • in the early stages, starts talking to their customers.

  • Is they realize maybe there's a big disconnect between what they thought their

  • customers wanted and what they actually want.

  • And at YC this usually comes to us in office hours in terms of,

  • we call the big redesign.

  • So everything about our site's incorrect.

  • We just basically need to rebuild it from scratch.

  • Or rebuild huge components of it from scratch in order to start serving our

  • customers.

  • Needless to say, this is a very, how do I phrase this?

  • Distracting and oftentimes not fruitful exercise.

  • But how do you see that type of challenge of, man, we got this feedback and

  • we think we need to be going in a different direction now?

  • But how do you deal with that kind of challenge?

  • >> That's a big one.

  • And I think you need to be clear whether you're talking about tech or the product.

  • I'll repeat what I said before, is your users don't care about your tech.

  • So if you can move forward, leave that for another day.

  • When you're talking about redesigning your products.

  • You're probably in trouble,

  • because whatever habits led you to build a product your customers don't like.

  • If you haven't addressed those habits,

  • you're probably going to build another product your customers don't like.

  • So I would make sure that, because I've been through lots of rewrites.

  • At Hipmunk we rewrote out flight and hotels probably half a dozen times each.

  • At Reddit, we re-wrote the tech a bunch of times.

  • And we're redoing the UI and

  • UX now with better habits and better data.

  • And so you want to make sure that you know something new.

  • But honestly, if you're in that situation, I would also go through

  • another mental exercise of okay, what's the new MVP, right?

  • It comes up in tech a lot, right?

  • The second system syndrome.

  • The same thing happens in the product where you're like, all right,

  • I've got this vision.

  • I'm going to fix all of the products, all of the mistakes at once, and

  • it's going to be perfect.

  • That product never ships.

  • It never ships.

  • So can you iterate your way there, can you just start from scratch with an MVP and

  • build it back up?

  • Those are much more likely to actually get you somewhere.

  • But they're also harder to do, and requires a little bit more discipline.

  • >> So we've also gone through a bunch of redesigns.

  • Big redesigns where you have to rebuild the whole product.

  • They've been mixed actually.

  • We've had some that were super successful.

  • Twitch did a big redesign of our mobile app, maybe three years ago,

  • from the first version that had basically been a cloned version of

  • the Justin.tv app to a completely new version.

  • It's basically the version you can use today.

  • And that increased engagement by 35, 40% per customer when we launched it.

  • It's huge, changes that make that big of an impact are actually really rare.

  • >> Usually data errors.

  • >> Yeah, yeah, [LAUGH] yeah, they're usually data errors.

  • And in this case it wasn't.

  • In this case, it actually was a huge increase in usage.

  • And so absolutely, redesigns can make a big difference.

  • But we weren't doing the big Twitch, we didn't do that redesign because we'd

  • discovered we couldn't move forward on our existing product.

  • Since we got some insights is the what what things should

  • be presented at the top level of a nav.

  • And we re-did the navigation to present those things at the top level instead of

  • burying them three clicks deep.

  • Three touches deep, I should say.

  • Most of the time when I see people doing a big redesign it's because they fall into

  • this trap that engineers and project managers often fall into.

  • Which is, we have to fix this eight things.

  • Let's just fix them all at once it'll be easier.

  • That's wrong, you should just fix them each one at a time individually.

  • If you have eight things that are wrong with your product that's awesome,

  • you have this huge great list of things.

  • Just fix them one at a time, fix everything one at a time.

  • Crank them out quickly,

  • you'll get there way faster one change at a time than you will in a big rewrite.

  • That'd be great for

  • a really early stage startup is almost equivalent to pivoting the company.

  • And we generally advise against it.

  • >> It's this weird sort of programmer product procrastination.

  • If I don't know what to do,

  • therefore I'm going to invent this big project to work on.

  • >> [LAUGH] >> We don't need to use that library.

  • I'm going to build it myself.

  • >> [LAUGH] >> Very common, use the damn library.

  • >> [LAUGH] >> So I'd like to talk

  • a little bit about decision making.

  • So maybe after you talk to your users but before you decide what to build, usually

  • there has to be some type of consensus building process either function or

  • dysfunctional between the co-founders.

  • And what are some tips or tricks on how to come to a consensus?

  • And then maybe more importantly,

  • how do you deal with the contrarian, how do you deal with

  • having to make a decision where you know there's going to be a disagreement?

  • >> I think the first thing you said is actually super important.

  • It's something I want to reinforce.

  • You talk to your users first, and then you have the ideas about the product.

  • Almost everyone does it in the opposite order.

  • It's called product validation.

  • If you find yourself thinking, I'm going to go talk to my users to validate my

  • product idea, you've gone horribly off the rails.

  • You do it in the other order, you don't talk to users to validate your product

  • ideas, you talk to your users to have your product ideas.

  • And so I would just have a rule in general for

  • people who want to have a voice in product design,

  • is that they better talk to the users and look at the data too.

  • If you haven't talked to users and

  • you haven't looked at data, you don't get to have an opinion about the product.

  • I'm sorry, the person who's actually done the work gets to have the opinion.

  • You can have ideas but they get to make the call.

  • But in case you wind up where both people really have talked to the users.

  • And both people really have looked at the data.

  • And people still disagree as to what we should build.

  • Which by the way is actually much more rare.

  • Once both people have actually talked to users and

  • actually looked at the data, it's usually relatively easy to get to consensus.

  • But it will come up where people still disagree.

  • I've found it's better when you just have someone who's in charge.

  • And at the end of the day, you have someone who's the CEO, and

  • everyone agrees this person's judgement will be trusted.

  • And you let them make the call.

  • Don't avoid conflict.

  • Talk about it. Argue about it,

  • but then you have someone who just gets to make the call, because anything else

  • leads to just decision making that's far too slow for a startup.

  • >> Yeah, maybe it's because I'm usually the one in charge,

  • I just feel like I don't have a lot of disagreements.

  • >> [LAUGH] >> [LAUGH]

  • >> There are-

  • >> Meet the CEO.

  • >> Yeah.

  • Largely, I find the situation I'm in, I say one of two things a lot,

  • which is I don't want to argue about it if we can just test it,

  • and I'm very willing to be surprised on this.

  • I'm probably right, but

  • this isn't getting into my top three important things to worry about today.

  • So let's just see.

  • It's easy to do that when you have more resources, to let's just see.

  • But I think about, like Ciss and I over the years.

  • And Adam and I over the years at Hipmunk, I think as it panned out,

  • we've very rarely had these philosophical differences over with the product,

  • sometimes it was timing.

  • Should we build this thing now?

  • What's the prioritization of these things?

  • But we always agreed on general mechanics of what we were doing and

  • why we were doing it.

  • And so the strategy might change but the details, we're very rarely not emotional.

  • So those are getting emotional,

  • I think there's probably something else structurally wrong.

  • You might be swimming upstream.

  • That happens a lot.

  • The arguments have gotten that rid over the last year like when we try to promote,

  • we do cross promotion from mobile web to the app, it just doesn't feel great.

  • It works.

  • It works and we like the numbers going up but

  • it's not really with the user's best intention in mind all the time.

  • Sometimes it drifts into this like it's good for the company but

  • maybe not for the users.

  • And when you're in those situations,

  • that's usually when the disagreements come out.

  • And you're not really arguing about the future anymore.

  • You're arguing about short term or long term, or best for the business or best for

  • the users.

  • And then when you beat at it, and beat at it, and

  • beat at it, eventually you come to a solution that actually feels right, right.

  • That meets your intuition, you don't feel like you're fighting the technology,

  • you're not fighting the users anymore.

  • Those are usually situations I look out for.

  • >> Sir, I want to get to you guys telling product story but

  • I want to cover one topic first which is that lets take people passed lunch.

  • Well now you need to have some type of process around talking to users,

  • looking at data, making a decision, building a product, releasing a product.

  • Checking your success and repeating, product development cycle.

  • What are some advice you can give to early stage startups around how to set that up,

  • what that cadence should be, maybe some pitfalls to avoid?

  • >> The first, just to take one little step back is, I see a lot

  • of startups, they get too emotionally high when the numbers are going up.

  • And they're not investing in this sort of process.

  • Because growth masks all problems.

  • And then at some point that kind of stops and

  • you don't have any of these mannerisms.

  • I also see startups get a little too low when things not going right at first.

  • And they kind of psych themselves out.

  • So you want to live in that in between, right?

  • If the data's really good, well, check the data.

  • Right, if the data's really bad, check the data.

  • [LAUGH] >> [LAUGH]

  • >> Okay, so then in terms

  • of the actual process, first of all it's going to be different at every company and

  • it's going to change, as you scale.

  • I think it's just most important to have a way of working,

  • to have if you just want to start out really, really specific of.

  • The way I like to think about it is where do we want to be in a year?

  • Okay, let's draw a straight line from that to here,

  • what do we need to be doing today to get there in a year?

  • So that's why I like our hypothesis.

  • Do we need to validate anything about that?

  • Probably a few things here or there.

  • Do that, go to check out.

  • Yes, and then it's just like straight line.

  • And I like to work on two week cycles.

  • Check in every week on basically that kind of thing.

  • Where do we want to be in a year?

  • Are we still on track, are we still doing the right thing?

  • Does it still make sense?

  • Are our assumptions still holding to be true?

  • If not, pop it up a stack.

  • And then you give people whatever it is.

  • If you're early startup, you give them 13 days to work.

  • And if you're a more mature startup, you give them nine days to work.

  • [LAUGH] >> [LAUGH]

  • >> And then keep going on at that.

  • I found that that's worked really well for me.

  • >> One thing I found actually is that if your goal of talking to users

  • shouldn't be course corrections on a week to week basis.

  • I did a whole crap on a talking users at the very start of

  • trying to figure out what the users wanted, how to think about them.

  • But then actually, we didn't because we had other reasons to go talk to users,

  • but we could have gone six months without talking to a user,

  • because actually we knew what users wanted.

  • They wanted to make money on the service.

  • Streamers wanted to be able to make money, they wanted to reach more viewers and

  • they wanted positive social feedback for what they were doing.

  • And you kind of just check anything you were doing,

  • is this going to deliver this to our streamers, yes or no?

  • And if so how much?

  • Great, you don't have to talk to anybody.

  • You already know what they want.

  • And you need to go back and revisit it and sort of talk to users

  • periodically because you need to understand more of the nuances.

  • The things about your service that are bugging them [INAUDIBLE] doing something

  • really cool.

  • But the goal of talking to users is not to get them to tell you what features to

  • build because users are really bad at that.

  • They actually have no idea what features to build.

  • [INAUDIBLE] users is to get to understand them really well.

  • And users don't change that fast.

  • So I personally prefer more in-depth time With my customers,

  • talking to them for a while.

  • And then, depending on the size of the thing we need to do or we're trying to

  • move, we might go just work and build and look at metrics for the next six months.

  • And not actually talk to users that much for a while.

  • Because if you know that what people want is to make more money,

  • you can spend six months optimizing the sell through rate on an object or

  • on how much advertising you have.

  • And you just can have faith that it's going to be good and you will be rewarded.

  • >> I think the classic advice that's very difficult to follow is,

  • you need to have the courage of your convictions.

  • So, you decide what you're going to build, it might take six months to build it.

  • And at Reddit we just kind of do it in front of everybody and

  • get flamed by users the whole time.

  • Which is fine, I'm used to it, but I have to kind of tell our product people like,

  • just keep pushing, keep pushing.

  • Just ship a little bit every week, and then in six months, they'll get it.

  • And at the same time,

  • the completely contradictory advice is you have to know when you're wrong.

  • You have to be able to identify when you're wrong.

  • You have to be able to say, all of you people are wrong, and I'm right.

  • And then at some point you have to decide, wait no, you're right.

  • >> [LAUGH] >> And if you can walk that line, you need

  • to have this like good combination of high ego and high humility and balance those.

  • But I always try to just kind of run this process in my head.

  • Wait, am I wrong?

  • Nope, they're still wrong.

  • >> [LAUGH] >> As much as I said don't talk to users,

  • you don't have to talk to users every week, it's totally unnecessary.

  • You do need to look at your numbers every week, you really need to know.

  • And you don't need to look at them every day, it's actually like pointless and

  • almost just ego gratification to look at your numbers every day.

  • But you should look at your numbers every single week, and

  • you should confront, are they up or down and ask yourself why.

  • And the holy grail of product design is for your numbers to move and

  • for you to truly understand why they moved.

  • We still struggle with that.

  • Like, there have been points where I feel like I understood it, and

  • then it slips away again as the business gets more complicated.

  • But you're always trying to be able to explain fully down to like atomic ground

  • truth, we are up 20% this week because we got this many more new devices in.

  • And our retention rate is this on those devices, and

  • this is the conversation rates.

  • And if you do the math and you can explain it all the way down to this new feature

  • moved this conversation rate on this page, by this amount, and therefore we are up.

  • And if you can really do that, it's super powerful.

  • It's also really hard.

  • But you'll never get there if you don't build with the intention of looking at it

  • every week.

  • >> So in the last five minutes before we open for questions,

  • can you guys walk through a feature that is visible on your sites now?

  • And I'll let you choose, that's either bad or good.

  • And talk about the process of how it came to be.

  • >> I have a lot.

  • >> [LAUGH] >> I'll talk about

  • one of my favorite ones.

  • This is from the early days of Reddit.

  • So the early days of Reddit, it was just links.

  • Everything was external.

  • And eventually we added comments, probably about six months in.

  • The first comment was, this is the end of Reddit.

  • >> [LAUGH] >> The first of,

  • that would be the first comment on every product we've ever released.

  • And users started doing this curious thing where they would, so

  • on any post you could go, you could click on the link or

  • you could click on the comments page to go to the comments for that page.

  • Our IDs for posts are just a counter that's in the URL.

  • So users would submit the URL that would represent the comment page for

  • the URL they're submitting, right?

  • So that would just increment, they would take the newest post and

  • increment by one and submit that URL.

  • So that you had a post that would link to its own comments page.

  • And it was really cool, so we called those self posts.

  • And users were doing this, and I remember watching them do it thinking,

  • this is really cool.

  • I would have never thought to do this.

  • Sometimes they'd guess wrong, right?

  • And so they'd submit a post to somebody else's comment page.

  • >> [LAUGH] >> Which was funny.

  • And so because our user base at the time was fairly technical, the way we made this

  • work is when you're submitting a link, you would just type in in the URL area, self.

  • And it would automatically do that little circular thing for you.

  • And that was the beginning of self posts on Reddit.

  • Self posts are now 60% of our content.

  • >> [LAUGH] >> I don't know, a few million a day.

  • And we would've never built that feature if we hadn't been watching our users.

  • I don't think I, I mean I'm explaining it now, but if I explained

  • that feature without it existing I don't know if it would make a ton of sense.

  • And users, most of them didn't get it at first either.

  • They were just like, comments are destroying the site and

  • there's not even a link here, it's just comments.

  • And it just kind of goes to show, I think the best product ideas are the ones

  • where your users doing it anyway, and you just grease it a little bit.

  • That's when I talk about before where you're either swimming upstream, or

  • just everything just feels right.

  • That was a classic example of it just felt right.

  • And it completely changed the site.

  • >> I think my favorite story here is an advertising product actually,

  • which is funny.

  • So all of our users told us really really clearly, we want to

  • show advertising and make money off of our streams.

  • It was a clear desire, everybody wanted it.

  • And they also told us their least favorite thing about

  • our competitors' products was that they would show these ads over the stream, and

  • it would be disruptive to their user's experience, and they hated it.

  • And it was very interesting getting these two pieces of feedback because

  • they both really wanted us to put ads on.

  • And sometimes, literally, sometimes different people,

  • sometimes it was literally the same people.

  • Their biggest complaint about the opposing service was,

  • competitive service is annoying my viewers with all these ads.

  • And this was like, this is a hard nut to crack.

  • Because you kind of need to solve for both,

  • you're not allowed to just give up as a product manager and pick one.

  • And I'm really proud of our solution, which was, we gave them a button.

  • We inverted it, and

  • we gave the streamer the ability to hit a button to run ads on their own channel.

  • And, like it's funny because normally you'd think

  • of that as something that will, if you're uploading content to like Facebook,

  • you would never press a button on that to show more ads to your friends.

  • But because we were cutting them in on the revenue,

  • they were actually heavily incentivized to hit this button.

  • And we basically got this paid workforce

  • of people placing advertising in the downtime of the stream.

  • And I'm really proud of that clever solution and

  • I think it points at something that I think is a really important principle of

  • product design in general.

  • Which is, that you want to just try inverting all of your assumptions.

  • We got stuck on that advertising problem for

  • a long time because we didn't want to make it disruptive to the customer experience.

  • And we couldn't figure out how to automate putting it on the stream in a good way.

  • And at the same time we really wanted to show ads on the video stream.

  • And we had

  • this implicit assumption in our heads that we had to figure out where to put the ad.

  • And once we gave that assumption up,

  • it was much more clear what to do going forward.

  • And I think that's a, and it's one of the most, I don't know,

  • I'm just really proud of that product today because it was one of the things

  • that made our broadcasters super happy.

  • And no one, like you can't complain, you hit the button.

  • >> How many times is that button clicked every day now?

  • You know, that's a really good question, I don't know the numbers off hand, a lot,

  • it's clicked very often.

  • People really like that button because it's a,

  • click the button make $15 button and like, you know.

  • >> [LAUGH] >> But

  • they're also aware of the trade off.

  • Which is that their viewers don't like it when they hit the button too much.

  • And so by putting that in their court, they get to make the optimal trade-off for

  • their stream and their situation.

  • >> All right, those are my questions, I want to hook it up to you guys.

  • Happy to talk about product, happy to talk about YC stuff,

  • whatever you'd like to chat about, go ahead.

  • >> What about the process of discovering you've had product market fit.

  • Getting past that launch pathway and discovering there's something sustainable,

  • beyond this real core group of users that are really committed to you?

  • >> Just to repeat the question, how did you know when you had product market fit?

  • So at Reddit, this is the nice scenario, we were growing.

  • Despite not knowing anything about our business,

  • knowing little about our users, not really knowing how to run a business,

  • not having a ton of product sense, being down a lot.

  • We were growing.

  • So that's the nice scenario, that's a good one.

  • At Hipmunk, our natural state was zero users.

  • I'm not sure we ever got product market fit.

  • The users who came to Hipmunk liked it, but

  • we had to fight for every single one of them.

  • It's possible to do that, but we were, again, I think that entire company,

  • we were swimming upstream.

  • And I remember the day we kind of realized that.

  • We ran the survey of like, why do you use Hipmunk?

  • And we put a lot of effort into the product,

  • we thought it was a really good product.

  • Legitimately you can find a flight a lot faster on us than our competitors.

  • 35% of people said it's because of the logo, and

  • 50% of the people said because we had lower prices, which we didn't.

  • >> [LAUGH] >> So, [LAUGH] I mean it's like,

  • we made it.

  • The company ran for five years.

  • And I ended up selling, which is fine, but yeah.

  • I think we were really fighting that one.

  • >> I think this is a huge misconception, by the way.

  • Most of the companies that apply to YC don't have product market fit.

  • Most of the companies on demo day don't have product market fit.

  • Most companies die before product market fit.

  • A lot of people talk about product market fit as this organic step in the process.

  • You raise angel, then you get an office, and

  • you hire some people, and you have product market fit.

  • And it's just like, you would be surprised at how few companies.

  • Product market fit is typically described as, you are overwhelmed with demand for

  • your product.

  • >> It's really obvious when you've hit product market fit.

  • So your product is bad, you can point at seven things that are broken about it.

  • And you have all these features you know you need to build.

  • And weirdly, it's growing really fast anyways.

  • It's really obvious when you have product market fit,

  • your product is growing like crazy.

  • And you are spending, you don't get to fix any of the things that are broken or

  • bad about your product,

  • because you're spending all of your time fixing it as it scales.

  • And you are, so the metaphor I use for startups is kind of Sisyphean,

  • you're pushing this boulder up a hill.

  • And you're pushing the boulder and

  • you're pushing the boulder, you're pushing the boulder.

  • Product market fit is when you crest the hill.

  • And your job doesn't get easier, but it does change a lot, because now you

  • are chasing the boulder as it tries to roll away from you down the other side.

  • And now you're springing flat out, trying to catch up, which is hard in its own way.

  • But it feels totally different, because as whereas before you have this, God,

  • where every inch is like, as Steve was describing with Hipmunk,

  • every inch is you're pushing this boulder and you're fighting for every inch.

  • Now it's like, crap, if we don't keep up with this, it's going to run away from us.

  • And that's a totally different feeling, and you'll know.

  • >> And here's the problem.

  • Eventually you'll catch the boulder on the way down, and then you're like ****,

  • I'm actually pushing it uphill again.

  • >> [LAUGH] Yep, absolutely.

  • >> [LAUGH] >> That's what happens.

  • >> [LAUGH] >> And I guess it's like asking,

  • how do you know if you're in love?

  • Trust me, you know.

  • >> It's funny, because I think one of the major problems that YC companies have is,

  • how do they stay small and lean and don't spend a lot of money.

  • Don't have a management overhead,

  • don't have a lot of bills, until they reach product market.fit.

  • One of the typical mistakes is confusing an angel round or

  • a couple million dollars in the bank for product market fit.

  • It's extremely common.

  • All right, other questions?

  • >> How do you recognize your first group of user?

  • >> How do you recognize your first groups of users?

  • >> Reddit was easy, they were just like Alexis and I.

  • Emmet was kind of talking about the intuition phase of building products.

  • We were in that for a long time.

  • Our users, they read the same Paul Graham blog that we did.

  • They were programmers like I was, so it was really, really easy.

  • Hipmunk, we were actually completely wrong about who our users were.

  • And I don't know if we ever quite solved this problem.

  • They solved it through acquisition.

  • Which is, we built a product for ourselves, like tech savvy

  • people who travel a fair amount and buy their own travel.

  • But in reality, the only way to make money in travel is to sell to business

  • travelers, because they're the only one that travels multiple times a year.

  • And business travelers don't get use things like Hipmunk or Kayak, right?

  • They use American Express and Concur.

  • So it took us actually a couple years to synthesize that, and

  • we were just like, ****.

  • >> [LAUGH] >> I think there's two kinds of startups.

  • There are startups that are building things for themselves,

  • where your customer is easy to identify.

  • It's you and people like you.

  • And there are startups building things for third party group users.

  • Where you should arrive at which customers are your customers through

  • an analytic mode, right?

  • Through like thinking about the problem.

  • So Twitch was an analytic start up,

  • because I was a huge fan of watching game streaming.

  • But actually I wasn't much of a streamer, I've never been much of a streamer.

  • And I basically dug in on Justin TV and realized that 80% of

  • the minutes watched were coming from our top 200 streamers.

  • And I was like they're the most important people.

  • We lose them, we lose everything, and if we can get 200 more of them,

  • we'll be twice as big.

  • And so it was sort of identified analytically that

  • that was the right group of users, and those are the right customers.

  • And I think that you should just know which one you are.

  • And the most important thing there is, if it's not for you, if you're not

  • legitimately building something for you and for you to use, and you aren't a big

  • group of people in this case, you should absolutely use the analytic mode.

  • I don't know, it's different for every single company.

  • But think hard about it, I don't know.

  • >> But if it's not for you, your intuition is lying to you.

  • >> Yeah, yes, yes.

  • >> Every time, and you have to supplement with data or talking to users or whatever.

  • >> So shockingly, we were 100% on time.

  • If everyone can thank these guys for coming out today, appreciate it.

  • >> [APPLAUSE]

So this is going to be the first of four lectures on how to build a great product.

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it