Hello.

Plan that wait.

But back then, back singing will happen.

Happening will happen happening will happen again and again because you and I will always be back.

Then you will always be back.

Then if there were some amazing for south inside of time to take us back to wear and hang each moment up like pictures on the walls inside a 1,000,000,000 tiny friends so that we could see it all, all it would look like will happen happening with happen, happening happen again and again.

Be back then you know that we'll always be back, then you and will always be back.

Then that's why.

And I will always be best friend.

I just knew I play the ukulele and I also like games.

And so there's many things you might know about me or want to know about me.

These are the things that are pertinent to this talk.

Um, games like lots of games, board games, card games, video games, like a lot of you.

I also program, Um and then lastly, I really like mischief.

Um, I like because I think of it as a jump sounding board jumping board, whatever that term is into, like, creative problem solving, creative thinking on innovation.

I like Mr so much that I named my son Loki.

Um, this is my secretary, uh, father shot of my my son, who I'm so proud of.

Host.

So one of the games I've been playing this game called Set.

Um, I was playing this game with my friends.

Susan, I want to come and go over all the rules with you first.

Um, So it's a card game.

Where, by the way, the New York Times has a set game that you can play.

It's kind of fun.

So you could go check that out.

Um, said is a card game.

It's a matching card game.

You're trying to make, uh, sets of three cards each.

You All these cards have, um, a color number, pattern and shape.

I talk.

You listen really quickly for color blind people.

Um, so amongst these attributes, what you're going for is a set of three cards where the attributes, the they're all the same are all different.

Um, and so here this is This is an example of a set where the number is all different.

12 and three the patterns, actually, all the same, they're all striped.

Their shapes are all different, and the colors are all different.

Um, so this is a very fun game until you play with somebody like my friend Susan.

Um, who I want to give you A I don't know.

I don't have music.

Music on my thing.

Maybe there's musics really quickly.

That's correct.

Try again.

Uh, no music.

So this is what playing set against Susan is like, um, it's where she just starts taking cards.

And I'm just observing and telling her Hey, good job.

Um, but wow.

Great.

Great.

Take their.

Um Oh, you almost had that one.

Oh.

Okay.

Uh, wait.

Do I see something?

No, no, no.

Um, so it's trust me.

It is a fun game.

Um, but until you play against somebody like Susan So how does that make me feel, Azad, protagonist of this talk.

Um, how does my protagonist too satisfied with their life?

I can't be Susan.

It's set.

What would it take for my protagonist?

Me to find satisfaction.

This is their goal.

Beat Susan, it's set.

Uh, what are my productions?

Biggest years and character flaws.

We'll probably three things first is that I don't really want to put the effort into getting good at this game so I could sit down and just play alone a lot until I get really good at this game because it's tricking your brain into kind of seeing these sets as quickly as possible.

But I am pretty lazy and so not interested in that.

Um, I'm also impatient.

I don't take the time to do that.

Um, I'd rather do it.

Use it for something else.

And last of all, the mother of all character flaws.

Hubris.

Um, I'm pretty sure I I was reading about Starcraft being beat by deep mind and off ago in Alpha Zero.

And like, I can do this, I can solve set with my computer.

Um, and I can finally beat Susan this way.

Um and so I started thinking about how this problem this forms.

And so for a human, it's interesting, because if you think the problem in two parts, the set making is the hardest part for human, Uh, when you you can quickly see these cars and you know what they are.

But you can't make the sets very quickly out of them for a computer.

It's actually the opposite.

The set making of super easy.

Um, And the vision part, that's the hard part.

So I can tell we're gonna computervision the crap out of this problem.

All right, here's the plan.

Um, we I want this around my phone, so I'm gonna use the cam on my phone, get into a browser on canvas.

Um, and then I get pixels from those pixels.

We're gonna identify a bunch of cards in the thing.

Um, from there, we want to extract out every card, um, and then put each card.

We won't identify the color number of shading in shape, and then eventually, we're gonna force himself in the south of here.

All right, so let's get into it.

Um, so this is my test image for this for this thing?

Um, as you can see a bunch of cards.

Um, getting out the things that you kind of care about are these ones this get images.

Data is what we care about here.

We get out this big, huge number array, so this looks kind of foreboding and creepy.

Um, it's got one No, 11 million pixels in it.

um So let's go over what this image data object is really quickly.

Um, you can think of images as a series of pixels.

Uh, here we have not six pixels, six huge pixels.

Um, and then each pixel is itself basically four numbers.

There's a red green blue on Transparency Alfa Complainant, RGB A.

So that's four numbers times six pixels.

That's 24 numbers.

SATs are 24 numbers that we see here.

Um, and then it's a flat array, so it's one dimensional.

But if you know that with that, you can, you know, basically where rapid.

Um And so this array corresponds with this picture up here.

And if I change this, that that's that's transparency.

So it goes away, I'm gonna bring it back.

I can make it black.

I could make this one red, so you can see how those things up.

So we're gonna be using this image data object a lot throughout this talk.

So, uh, gotta get familiar with it.

So first up is we want to threshold out.

We want to simplify this picture.

Whole bunch.

Um, well, basically changed in tow, straight, black and white.

Um, and that's to make our job a lot easier later on to do this.

This is called threshold ing.

Ultimately, what I'm doing here is I'm converting every RGB value into a gray.

And if that gray is over below a certain value, either side of the white 55 or black zero, um, itself it changed this slider.

You can kind of see how it affects the image.

Um, and then on entertaining thing.

That note here is that you'll notice that this is not a 10 year transform from RGB gray.

Um, and that really quickly is if you look at these green and blue values here.

So this is actually the same amount of green and blue light hitting your eye, but because of how your retinas all jacked up, um, you perceive the green a lot more than the blue.

Um, And so you think green is a lot lighter than blue?

And so that's take into account with this formula here.

Um, all right, so we have a black and white image.

Now we want to find those cards again.

Um, to do that, we go to N.

P.

M.

And just find some random package that finds finds, But what this package does it It's called Maur Neighborhood tracing.

Which sounds scary because it has a name on the front of it.

Um, use neighborhood in the context that makes no sense whatsoever.

Um, but you can think of this as, say, you're an ant.

Um, and you really like black pixels and your exploratory ants.

So you just out for a walk and your baby methodical.

So you just scanning through all of this black.

Um, And then you also hit this white picks on you.

Well, what is this?

This is new.

Uh, let's figure this thing out.

So you kind of back up, you go to the side.

You kind of headed again.

What picks a look at you back up.

You decide.

You do that a whole bunch until you're kind of like making a way around this whole thing until you're all the way back to the beginning again.

Um and so if you've paid attention like late doubles down, used, basically made and trace the path around this card, Um, so that's more neighborhood chasing.

Um, you'll notice that I then so I take the threshold image running through that I get a bunch of contours.

Um, and then I end up filtering based on area here.

Um, and so if I kind of change some of these values, which now you'll notice that I'm twiddling values, a lot of these values have to do with, um, the source image.

So I'm kind of like picking numbers out of the air based on kind of the parameters of what I'm assuming that's gonna be important later on.

Um, so here I've, like, made the area threshold go away, lowers basically put out smaller shapes, but I actually don't want those.

So go back to here and only get 12 of them.

Um, some Bactine marginal 12 cards and see where we we went to.

Now pull out each card to do this, we need to simplify that path a little further.

Um, instead of that path is like a whole bunch of dots around that card.

Ultimately, what I want is this rectangle ish thing are quadrilateral that consists of only four verte sees to do that.

Um, there's probably a cool name for this thing, but I made up some ran the way to do this.

Um, what I end up doing is basically removing pixels.

So I want to remove her done pixels a longer line.

Um, and so I just keep removing them until it down before the way I choose which pixel to remove Next is basically the one that result in the minimum least area change which result in a pretty good rectangle this point.

Um, Now I want to pull out every card.

So this is now where I'm going from here to here.

Um, so this is called perspective transform, which again?

Go to N P.

M.

You just search for a perfect chance for you get this thing.

Um, this is basically linear algebra.

I'm not playing this one, but then your algebra matrix math you can pull out, transformed from a perspective in tow.

This perspective.

All right, we got every card.

That's kind of, um oh, so we need to correct for the lighting in the room.

Basically, you'll notice here that this is I took this in like a yellowish little room.

Sometimes you get warm lighting.

Sometimes you get cool lighting.

Ultimately, one kind of correct for that.

So we want the back of the cards that basically be as like to 55 to 5 to five white.

Um, to do that, we can assume that our white point.

So we need a white point.

We can assume that the background on the card is white.

Um, and so what I've done is just average the top and bottom borders of every card.

Um, and call it that my white point on.

And then using that, I can shift every pixel in the image in RGB space, uh, into basically where the backgrounds of averaging out to be more white than the rest of the card and all the, um, all of the other colors in the slightly frightening Also, which is actually good.

Okay, so recapping where we are, we got our image.

We identified some cards we extract with cards, and now we want to go for all of the colors and all the things in them.

So here we have the spectrum of the record.

So I've gone through and basically just grabbed every pixel, mapped it to a que, um and then throwing out all the white.

So you can tell this is actually pretty pretty indicative of kind of all the cards I ran through.

It's pretty easy to tell that there's purple card, a red card, a green card.

This weird one here is kind of interesting.

So that one was actually this, uh, one here.

We can see some rain building going on here, which is still fine.

Um, there's a lot of strong signal here.

Still in the purple, so really fine.

Um, next one is number.

So for number again, I just throw contouring out again for each card.