20. Speculative Parallelism & Leiserchess - VoiceTube: Learn English through videos!

Subtitles section Play video

The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
CHARLES E. LEISERSON: OK, let's get started.
On Thursday, we are really privileged to have
Jon Bentley, who is one of the masters of performance
engineering, come and give us a guest lecture.
In 1982, he wrote this wonderful little book
from which we adapted the Bentley
rules that you folks have seen.
And he's also famous for things like kd-trees.
Who's seen kd-trees. trees?
Yeah, so he invented kd-trees.
And who's ever used the master method of recurrence solving?
He invented the master method of recurrence solving.
And so he's just a wonderful, wonderful fellow,
with lots and lots of insights, and just a superior performance
engineer.
And he's going to give a guest lecture on Thursday.
And so I would encourage you to bring your friends.
Bring friends maybe who dropped the class,
and others who if any of you have UROPS, other people
in your research group, whether they're graduate students,
they're all welcome to come.
Because this is really one of the opportunities
you get to actually meet one of the legends of the field,
if you will.
And you'll see he's very approachable.
He's very approachable.
So anyway, my advertisement for Jon Bentley.
Good.
So let's start.
I want to talk about speculative parallelism for a little bit,
because that's kind of what you need to make the code go fast.
So if you look at code like alpha beta,
it turns out that it's inherently serial.
That is, if you try to do things in parallel,
then you missed the cutoffs.
And if you missed the cutoffs, then you
start doing work that you wouldn't necessarily
need to do.
And so the only way to sort of make this type of code run
fast is to use speculation.
Which is to say, that you're going
to guess that you can do stuff in parallel
and it'll be worthwhile, and then, occasionally, you're
going to be wrong.
But the trick is, well, how do you minimize the wasted effort
when things go wrong?
So let's just take a look at a very simple example
of speculative parallelism, which is thresholding a sum.
So here we have a little program that is returning a boolean.
It takes as input an array of size n.
So thresholding a sum-- we have an array
of length n and a limit.
And these are all unsigned integers, say.
And I start out the sum at 0, and what I'm going to do
is add up all the elements of the array.
And if it exceeds the limit, then I'm going to return.
And so how might I optimize this code a little bit?
Yeah.
AUDIENCE: Split up [INAUDIBLE] four processes, [INAUDIBLE]
split up an array [INAUDIBLE].
CHARLES E. LEISERSON: No, let's say on processor.
On one processor, what might you do?
AUDIENCE: In the middle of the loop,
you can check [INAUDIBLE].
CHARLES E. LEISERSON: Yeah, once you
check so that once you would exceed it,
why keep adding the other numbers?
So here's a code that does that-- quit early
if the partial product ever exceeds the threshold.
This isn't necessarily an optimization.
Because notice that now in the optimization,
not only do I have a memory reference and an addition,
I also now have an unpredictable branch.
Actually, it's predictable branch--
predictable branch.
But it's still one more step, because it's always
going to predict that it's not exceeding until it actually
does exceed.
So that'll be pretty predictable.
But still, I've added something into the loop
so that maybe slower.
How might I mitigate that problem?
So I don't want to add too much into the inner loop.
Yeah.
AUDIENCE: It could have an inner loop
that goes [INAUDIBLE] them.
CHARLES E. LEISERSON: Yeah.
So basically, I can do what's called strip mining.
I replace the loop of n iterations with a loop of,
let's say, n over four iterations,
with an inner loop of four iterations.
And every fourth iteration, I check
to see whether or not I've exceeded,
so that the cost of the check is going to be relatively
small at that point.
So are there ways of making this sort of go fast.
So now, we want to make this operate in parallel.
And the problem when I operate in parallel
is that as I'm adding stuff up, I want to do it.
So I'm going to do this here with a reducer.
And so, basically, I'm going to add up
the values in the reducer.
But now, I'd like to do the same thing
of being able to quit early.
And so the question is, well, how
can we parallelize a short-circuited loop?
And so, the way I'm going to do this-- and this
is sort of by hand here, but it's
going to give you-- so actually, as you
know, underneath the loop is really a divide
and conquer loop.
And so I could write it as a parallel loop.
And now, it becomes, I think, a little bit clearer
how I could, in this case, what I'm
going to do is return the value of the sum.
And now the question is, well, how
can I quit early and save the work if I exceed the threshold?
Understanding that when I'm executing
this and something like Cilk, it's
going to tend to go down one path.
And often, it's going to be a serial execution.
So I'd like if it's possible.
So here's one way of doing it, which
is that I add an abort flag to the code, which
is going to say whether I should keep going
or whether I've actually exceeded at that point.
And so I'm going to use that recursively.
And now, I take a look, for each time through the loop--
or through the divide and conquer--
to see whether the sum is greater than a limit.
And I haven't you know already aborted the flag,
then I set the abort flag to be true.
Why do I bother testing the abort flag
before I set it to true?
So notice that setting the abort flag is a race,
isn't it-- a determinancy race.
Because-- great.
Thank you.
It's because I have the stuff operating in parallel
is all trying to set the abort flag whenever something aborts.
I can have two guys who are in parallel setting
the abort flag.
But if they're setting it, they're
setting it to the same value.
So it's a benign race, assuming your memory model is such
that you can actually set the values.
So what happens here when--
why do I why do you bother checking this?
So what happens when several guys in parallel
want to write to the same variable?
This is quiz 1 stuff.
Yeah.
AUDIENCE: Cache bouncing.
CHARLES E. LEISERSON: Yeah.
You can have it bouncing along the cache.
It will serialize it.
Because if they're all trying to write,
then they have to get exclusive access for it
to write and modify it.
And so that happens-- boom, boom, boom-- one at a time.
And so by checking it first, it can be in a shared state,
and then one guy clobbers it, and then
it will update all the other ones.
So it makes it so we don't continually
have a true sharing race.
And then in checking to see if it exceeds,
we basically just check to see--
we basically call this function. we set the abort flag to false,
and then we return the sum of all the values.
And if it aborts, then it just returns early.
So it doesn't have to keep going through the computation.
And, of course, once again, you can coarsen
the leaves and so forth.
So this is nondeterministic code, which we should never
write unless we have to.
Because that's the only way of getting performance.
And you have to make sure you, of course, reset the abort flag
after the use.
And then, actually, do you need a memory fence
here on the abort flag?
Do you need to set--
where would you put an abort flag
to make sure that the value was--
yeah.
AUDIENCE: Maybe do it by setting default [INAUDIBLE]..
CHARLES E. LEISERSON: Yeah.
So what would you have to do?
AUDIENCE: Put a [INAUDIBLE].
CHARLES E. LEISERSON: OK.
So indeed, it turns out you don't need a memory fence here.
And the reason is because the code is correct,
whether it's the initial value false
or whether it's become true.
So it doesn't matter when it becomes true.
And so there's an issue of, if you put in the fence,
then you know that it becomes true earlier,
rather than waiting for the computation.
But that may actually slow things down,
because memory fences are expensive.
But because the transition is always
from false to true during the execution,
you don't actually need a memory fence there
in order to make sure that you've got the correct value.
Does that makes sense?
So you don't need a memory fence.
So that's a classic instance of speculative parallelism.
It occurs when our program spawns
some parallel work that might not be performed
in a serial execution.
So you're performing it, anticipating
that you're probably going to need to do it.
And then, typically, what you want to do
is have some way of backing out if you discover that you
didn't need to do it that way.
Have some way of making sure that you don't do any more.
So basic rule of speculation is, don't spawn speculative work
unless there's little other opportunity for parallelism
and there's a good chance it will be needed.
So one of the things I've seen, in research papers, no less,
is people who say, oh, I'm going to have a calculation where
I'm trying to find, say, the minimum of a bunch of values.
And so I'm going to spawn off n things, each of which
is looking for the minimum.
As soon as the minimum one comes back,
I'm going to retract all the other computations.
And I'm going to get super linear speed up that way.
Because in the serial execution, I
might have done the longer one first.
And maybe the minimum is not the first one or whatever.
And so they claim super linear speedup
by doing speculative parallelism.
But that's not really a good example,
because there was a better serial code.
If that was really a good way of doing it,
they should have been doing what's
called dovetailing, which is doing
a little bit of this computation,
a little bit of this, a little bit of this,
a little bit of this, et cetera, and then going back.
That would have been a better algorithm for which you
would then use it.
And the risk they have is that they're spawning off
n things, of which most of them may not be needed,
and now they don't get any speed up,
even though they've just used a lot more work.
And that often is a bad choice.
So usually, you don't want to speculative work
unless there's little other opportunity
and there's a good chance it'll be needed.
My experience is, what kind of chance do you need?
You need to have certainly less than--
for parallelism, if the chance that you're going to need
the work-- if you have p processors,
if the chance that it could be not needed is less than--
actually, I have a theorem at the end which
all which I've I'll refer you to,
because it's a little bit hard to say,
because I didn't put on the slide.
But there's a final slide for this,
which gives a nice little theorem about when
you should do this.
So now, let's talk about parallel alpha-beta search,
because that's kind of what you're
doing with your principal variation search.
So if you remember the analysis done by Knuth and Morris,
they basically showed that for a game
tree with the branching factor b and depth d,
and alpha-beta search with moves searched
in best-first order examines it exactly
b to the ceiling of d over 2 plus b to the floor of d
over 2 minus 1 nodes at ply d.
So that's basically square rooting the amount of work
that you would need if you did a full depth ply.
That's with alpha-beta.
So naive algorithm looks at b to the d nodes at depth d.
And so for the same work, the search depth
has effectively doubled, or the work is square rooted.
So that we solved last time in Helen's lecture.
So the key optimization here is pruning the game tree.
And as I mentioned at the beginning of this,
when you prune the game tree, you've
got what's effectively a serial thing.
And if you let something go ahead,
you may be working on something that would be pruned.
So then how does the parallelism really help you there?
You're just wasting processors that could be better
spent perhaps elsewhere.
Except, where else can you spend it?
Because there's no other parallelism in the code.
So we want to find some solution.
So the solution comes from an idea
from a very nice paper by Burkhard Monien
and some of his students in Germany.
And they made the observation that in a best-ordered tree,
the degree of every node is either 1 or maximal.
This is not their observation, this
is actually the observation in the Knuth paper.
So when you have a best-ordered tree,
if you can get all the moves ordered in their true order,
then it turns out that either you're
exploring all the children or you are refuting something
and you get a cutoff from the very first one
that you look at.
And so in this case, for example, it turns out,
on the principal variation, those are all full-width--
sorry, you have to explore all the children.
And then from there on it alternates.
One level, you have just a single child
that needs to be explored.
And when you explore it, if it's best-ordered,
you will get a beta cutoff for the others.
And then it alternates, and then it's full-width.
And so their idea was to say, well,
if the first child fails to generate a beta cutoff,
speculate that in fact you have the best one,
and the remaining children can be searched in parallel
without wasting any work.
So if it fails, you're going to say,
oh, I'm going to speculate that this
is in fact a full-width thing.
Now, in practice, you don't necessarily
get things best-ordered, but there
are a bunch of heuristics in the code we've given you
in chess code to make it so that you tend to do things
in the proper order.
The most important of those is, if you've seen the movie before
and it's in the transposition table,
you use the move that you've seen before, even
if it's to a shallower depth.
You guess that that's going to be their best move still,
even if you search deeper.
And that's pretty good.
And so they call that the young siblings weight algorithm.
They actually called it the young brothers weight,
but in the modern era, we call it young siblings weight.
And we abort subcomputations that proved to be unnecessary.
So you're going to start out searching,
but then you want to get rid of things
that you discover, oops, I did get a cutoff from searching
from one of these things.
I don't need to do this.
Let's not have it keep spawning and generating
work and some other thing, let's abort it.
And here's the idea for the abort mechanism.
So what you do is--
the abort can occur at any given node.
You get a beta cutoff--
you want to abort all the children that wouldn't
have been searched anyway.
But they may have already been spawned off.
So they have a record in each node that tells you
whether or not you've aborted.
And what you do is you just simply climb up the tree
to see, periodically, whether or not
you have an ancestor who is aborted.
If you have an ancestor that's aborted,
it says I'm aborting the side computations,
then you say, oh, I'm done, and I just have to return.
And so you check that on a regular basis.
So do people follow what that mechanism is?
And so, that's basically what you're
going to be implementing for the parallelization,
is this thing of climbing up.
You're going to guess, after you've searched one child,
that it's good.
Hey, actually some people say, why don't you
check two before you search the others,
so that you're even more sure that you
don't have a cutoff, because neither of the first two
aborted.
Because, in practice, of course, the game tree
is not best-ordered.
And so you're going to waste some work.
But the idea is, you're going to pull up the tree
to see whether or not-- and, of course,
you don't want to pull on every evaluation that you do,
you want to just pull frequently.
So you may have a counter in there
that says, OK, every 10th time, I'm going to pull up the tree.
You put a voodoo parameter there that says
how often it makes sense to actually check, because there's
cost to going up and checking versus exploring
more of the tree.
And so, I have an example here.
And I think this is where I'm going to cut this short.
So there's an example, which I suggest you take a look at.
I want to, as I say, spend time doing Q&A, and talking
about the other parts of the program,
and give you folks some good ideas for the thing.
So this is sort of the theory thing,
and then I have some slides that will explain this
with examples in more depth.
Because I don't expect that everybody
got every detail of this.
So I have some examples and so forth in here
that I'm going to let you guys look at on your own.
Now, if you parallelization the spawning
off loop of the younger siblings,
you will get a code with races.
And the reason you get races is because there are several--
the are three data structures--
that the search is employing that are kind of global.
The first is the transposition table--
looking things up to see whether or not you've seen them before.
That's a major optimization.
You don't want to get rid of the transposition table.
And so you have a choice with the transposition table.
What are you going to do with it?
Are you going to lock it so that the races become--
at least your data-- race-free and updated atomically.
Or you could replicate it.
For example, you could have a worker local copy of the data
structure, and each worker that is working on it
only accesses their own local copies.
Or you can make a copy when things are stolen
or you can make just one where you maintain it locally.
Or you can decide that, well, even if there's a race there,
the odds that it's going to affect how I play in the game
are not that high, so I'll run with the race.
Because any other solution is going to be more expensive.
And then maybe you get funny answers.
So one thing for that kind of data structure,
let me just recommend that you have
some way of turning of the data structure
so that you can actually do the parallel search
and get deterministic, repeatable results.
So even though it may be aborting
and some things may not be aborting,
you want to have some way of executing.
And so, for example, you want to be
able to turn off even the speculation in your code,
so you can test all the other things
in your code that don't depend on the speculation.
Because if you have something that's not deterministic,
as I say, it's a nightmare to debug.
So you're going to do the evaluation function.
Hey, if you're testing it, turn off the speculation.
You should be able to find out whether you've got bugs
in your evaluation function.
There's no point in discovering you of a bug,
and it's like, well, where did it come from or whatever.
So you want to have things you're turning out.
And also, want to have ways of turning off,
for example, access to the transposition table.
Yes, I'm going to speculate, but no, I'm
not going to access the transposition
table in parallel, because I may get
a race on the entry in the transposition table.
The hint that I will give you for that
is that in the code for our program
that took second prize in the world computer chess
championship many years ago--
I think it was in 1999--
where we actually almost won the tournament,
and we lost in the playoff.
We would have won the tournament,
except that our programmer suggested a rule
change at the beginning of the tournament
that everybody agreed to, and then we
were on the short end of that.
Otherwise, we would have been world champions.
And that was the last competition
against computers that Deep Blue, the IBM computer that
beat Kasparov, the human [INAUDIBLE],,
just a few months later.
They performed it.
And they placed third in the tournament.
We tied for first.
Our only loss was to Deep Blue.
And we were running on an 1,824 node supercomputer
at Sandia National Labs, a computer that probably is today
less powerful than this.
But it was very big type of computation.
They basically said, if there's a tie,
they're going to base who wins on strength of opponents.
So you take a look at your opponent's records,
and if you had stronger opponents,
you'll win against somebody.
And we said, you know, if there's
a tie just between two programs, we
should really have them face-off against each other
as an extra round.
And everybody said, yeah, that's a good idea.
And then, wouldn't you know, at the end,
we had the strongest opponents, and we
were tied with another program called
Fritz-- an excellent program.
And we were outsearching Fritz in the playoff game.
And then we saw a move that afterwards,
when we were able to do offline searches deeper,
we were outsearching them by like 2 ply.
But then there's a move that looks
like it's a good move for the depth we were looking at it--
doesn't look like a good move if you
search much deeper, or frankly, if you search much shallower.
It was one of these things where it just
looked like a good move for the ply
we happened to be searching.
Because there's no guarantee--
because you can't see the end of the game--
that if you search deeper you're actually
going to beat somebody else.
Because there's the horizon effect of you just
simply can't see what's coming up in the future.
You can only see sort of on average.
So we made this bad move.
We almost recovered.
We almost tied.
Had we tied, we would have been world champions,
because we had the stronger--
we won the tiebreaker.
And we weren't able to recover from the error.
And so we took second prize.
It was like--
[LAUGHTER]
Anyway, in that program, we decided
that we were just going to let there
be a race on the transposition table.
And the reason was, we calculated,
what are the odds that the race actually
affects the value that you would actually pick that value?
And if it affected the value, what
are the odds that it affects the move that you're going to make?
And if it affects the move you're going to make,
what are the odds it affects the game?
And if it affects the game, what are the odds that affects
your result in the tournament?
And when you've figured all these things out,
it was like, eh, we would slow the program down more
by putting in, for example, locking--
because that would slow down every access--
than we would if we just ran the thing.
Because one of the things that happens in alpha-beta
is if you get an extreme value, usually those are lopped off.
Because if it's such a good move for you,
your opponent's not going to let you play it.
So if you ended up with a score that was extreme
because it was based on a value that you were racing on,
usually it's going to not even propagate up
to the root of the tree.
So anyway, we just ran naked, so to speak.
So there are two other data structures
you going to have to worry about to make decision about.
Another one is the killer data structure.
So the killer data structure is a heuristic
that says, at a given depth of code,
you tend to see the same moves that are good.
And a move that is good for one thing at a given depth,
tends to be good for something else.
So for example, it may be that you play bishop takes queen.
So you've won the other players queen.
And then their response is to do something irrelevant.
Well, now you mate them.
Well, there's a lot of other moves
where you could make them on that move, for which they're
playing things on the board that are irrelevant.
and.
So the same type of move tends to be a killer--
always, you're able to, at that depth in that position,
make the same move.
And if they don't address the issue,
that's always a good move to check.
And so that ends up being ordered near the front.
So the killer table keeps track of that.
And I think, in our code, we have two killers.
Is that right, Helen?
I think it's set up to allow you to do for up to four killers,
but we only do two in the code that we gave you.
And you can enable it and see whether four killers--
but that's a shared data structure.
And so one of the questions is, should you
be keeping a local copy of that or should you
be you using a global one that they all share and updating it?
The third one is the best move table,
which is used to order all the low order thing.
So the first thing that's most important
is, did you get a move out of the transposition table?
That tends to be quite accurate.
And the second is, did you get a move out of the killer table?
And finally, it's, statistically,
how good are these moves?
Have you seen these a lot before?
If you've seen them a lot before,
that's how the other moves get ordered.
And that's kept in the best move table.
And once again, that's a shared table.
In the search, you're going to have to figure out,
do you want to do a thread worker local copy,
or do you want to synchronize it,
or how are you going to manage that data structure?
And the answer, I would say, is different
compared to what you do with a transposition table.
The transposition table, generally, it's
not worth locking or whatever.
And it is good to share.
Because if you have seen that move before,
that saved you a huge amount of computation
to be able to look it up and not do it.
So those are some of the tips for parallelization, which
we'll get to in the beta 2.
But now, I want to talk about, for most
of the rest of the time, some of the other things
that you can do with the code you've currently got.
And let me take these in some order,
and then we can ask questions.
So opening-- who's contemplating doing an opening book?
Anybody?
For beta 1?
Are you going to do an opening book for beta 1 or beta 2?
For beta 1, let me see.
OK.
Beta 2?
Who wants to do an opening book better 2?
Final?
OK.
Yeah, OK.
So the idea here is to precompute best moves
at the beginning of the game.
So, well, we know what the starting position is.
So why don't I spend $1,000 or whatever on Amazon,
and compute things to some ungodly ply,
and see what the best moves are at the beginning of the game,
and put that into a book so I don't have to search those?
So that's the idea.
I think, to begin with, there are lower hanging fruit
than the opening book.
If you look at it, you're going to be an opening
book, if you're lucky, for six or eight moves.
And the games tend to last-- have
you looked to see how long games tend to last?
What is it?
AUDIENCE: [INAUDIBLE]
CHARLES E. LEISERSON: No, most games don't go 4,096.
We don't let them go that long anyway.
So did anybody take a look, statistically,
to see how long games are?
I think they tend to be like 40 or 50 moves.
I mean, this year is different from other years,
because we have different rules.
But I think it's like 40 or 50 moves.
So you're not spending--
you're doing something that will help you in 10% of the game.
Whereas there are other places you
could do it which are going to help you in the whole game.
Nevertheless, we've had teams that
did a great job on opening books and clobbered people
by having a fantastic opening book.
It's actually cheaper to keep separate opening books
for each side than to keep one opening book for both sides.
And in this game, it's actually fairly easy
to keep a symmetry thing and basically
have one opening book that works for the side on move.
And that allows you to store.
You don't have to store--
if I make a given move, if I have a given position,
I only need, in principle, to store
one answer for that position.
Whereas, then, my opponent may make any number
of moves for which then I only need to know one move-- what's
my best move in that position.
So you can see that you have, once again,
this property, that on my move, I only
need to have one move stored.
My opponent's moves, I need to have all the moves stored.
And then my move, I have one move store.
And that basically means that you're effectively
going twice the depth, as if you kept all my moves,
and then all of the other opponents
moves, and then all of those.
Because when I do that, it's like, well,
why do I need to know the best move.
So it's better to keep them separate.
When you build the opening book, you'll store a lot less.
You'll have a lot smaller storage.
You'll be able to store a lot more moves if you do that.
But I would say, not necessarily the first thing
I would go to optimize.
But it's also the case that this is
a place where one of the teams built a fabulous opening
book one year.
And one of the things about the opening book
is it allows you also to sort of threshold
and say, in a given move, I don't just
have to have one move.
Let me have three moves and pick randomly among them,
so that I'm not predictable.
What the team did that used the opening book is they
observed that everybody was searching from the beginning.
So they were all finding exactly the same moves
at the beginning of the game.
And so they knew exactly what the path
was that all the other programs were going to follow.
And so they could make the best move on their move.
But if you have something that's got some randomness to it--
and there is a jitter in there.
Who found the randomness--
the place we insert randomness in the code?
So I think Zach had a comment on Piazza about that.
So there's a place in the code where
we dither the amount so that you will get unpredictable results.
And that way, you won't always play the same move, just
by changing things.
Because a hundredth of a pawn value is not very big.
And who cares whether or not we have--
these heuristics aren't measuring things
so closely that we care about a hundredth of a pawn.
So you can dither things and have
one move be better than another and not really affect
the quality of the playing.
So it is a good idea to do something
to not be so predictable.
But to build a full opening book, that's
a lot of work for beta 1.
For beta 2 and the final, yeah, you'll get to that point.
The next thing is iterative deepening.
Do people understand how this part of the search code works?
Yeah?
No?
Let me go over it.
So you can imagine, say, I'm going to search to depth d.
And in fact, instead of just searching depth d,
we do a depth 1 search.
If you look at the code there, there's a depth 1 search.
And then we do a depth 2 search.
Then we do a depth 3 search.
Then we do a depth 4 search.
And we keep doing that until our time control expires.
Why is that a good strategy, generally?
Well, first of all, the amount of work with each depth
is growing exponentially.
So you're only spending, in principle,
a constant factor more work than you would if you searched
to depth d right off the bat.
But the important thing is that you
can keep move ordering information
as you search deeper.
And the move ordering information,
remember, we want our searchers to use best-first move ordering
so that we get all the cutoffs we can possibly
get for pruning.
And so by searching it easier, actually you
end up with better move ordering.
Because when you find that same position in the transposition
table, you say, oh, well.
I didn't search it as deep as I need to search it now.
So I can't use the value that's in the transposition
table for the position, because maybe I've searched something
that's ply 8 in the transposition table,
but I've got it searched to ply 9--
not good enough a search.
But the value-- the move that you've found at 8--
that's probably a pretty good first move--
pretty good guess at best move.
So by doing iterative deepening, you actually
get the move ordering for all those intermediate positions
really really, really strongly.
Because that transposition table is the very best information
you've got for what the your best move is.
And also, as I mentioned, is a good mechanism
for time control.
Is that clear?
So that's why you bother.
It's one of these things that looks like it's redundant.
Why are you searching-- if you're
going to search to depth d, why search to depth d minus 1?
You're going to that as part of depth d.
Well, no, because you're going to get move ordering
information that's going to be really valuable when
you search to depth d minus 1.
So when you search to depth d, you've
got much better pruning than if you just
went straight to depth d and had no information
about the move ordering.
Does that makes sense?
Endgame database-- so here's the idea.
If there's a few enough pieces on the board,
you can precompute the outcomes and store them in a database.
So for example, the most common database to come up with
is the king versus king database.
So if you do King versus King, that
means that it's not that you expect the game will end up
as king versus king, it's that you're
going to encounter that in the search.
It'd be nice to know who has the win if you're King versus King.
Now, the code actually plays optimally for king versus king.
The code we gave you will play the king versus king perfectly
well.
And, in fact, there's just two heuristics
that are responsible for that.
One is called k aggressive heuristic,
which basically makes you move towards your opponent.
And the other one is the k face heuristic,
which makes you point towards your opponent.
And those two things turn out to do a pretty good job of when
there's one there's two kings and playing,
they go through this little endgame.
Did everybody do that by hand, by the way?
Did you play king versus king?
A really good idea if you want to understand the game
is just go and do a king versus king version of the game.
Get an eight by eight chessboard and just put down
two tokens kings that have orientations, and then just
you and a friend, just try to see if one can mate the other.
And you'll see that it goes through a very ritualistic type
of behavior which ends up with one of the Kings being mated.
So it's nice to know which one it's going
to be so if you encounter that.
But here's the question, how many games
actually make it to an endgame?
Once again, you have to look at what the odds are there.
Because if games are ending in the middle game,
there may not be any need to search
all the way to the endgame.
So that's something that you need to weigh.
Now, if you do an endgame database,
it doesn't suffice to just store win, loss, or draw
for a position.
And the reason is because you can get into a situation
where you've got a winning position,
and then you say, OK, well, let me move to another winning
position, and another winning position,
and another winning position.
Oh, and I'm back at my original winning position.
And I just keep going around in a circle,
always in a winning position, never moving towards the win.
And that, of course, will end up in a draw.
So you need to know which direction do you
go for the winning position.
So the typical thing you do there is you
keep the distance to mate to avoid cycling.
So instead of keeping just a boolean value,
you say, here's my distance to winning.
My distance to winning is 6.
Now, when I'm searching, I don't want
to go a distance to winning that 7 or 6,
I want to go to distance to winning
that's 5 when I make my move.
So it's very important to make sure
that you maintain the distance in order to avoid cycling.
Quiescence search-- we talk about quiescence,
which is to make sure that when you're doing the evaluation
you're not in the middle of an exchange.
So you zap their pawn, they're going to zap your pawn,
or maybe even zap your King.
And you don't want evaluate in the middle there.
And so the idea of quiescence is that when
you're done with the regular search,
you just play off moves that involve captures,
and then you evaluate the position.
So that's quieting the position.
Make sense?
Another heuristic there is null-move pruning.
In most positions, there's always something
better to do than nothing.
So sitting still is usually not as
good as actually making a move.
And so, the idea is, suppose that I can forfeit my move,
and search a shallower depth, and I still
have the best position--
then I still am winning.
And it generates a cutoff.
Then why bother exploring any other moves?
I probably am in a really good position here.
OK And if I don't manage to get a cutoff from the null-move,
then I do the full depth search.
So the place this is dangerous is in Zugzwang.
In chess, it's called Zugzwang situations.
So Zugzwang is a situation where everything is fine,
but if you have to make a move, you lose.
So usually, having the advantage of a move,
that's called the initiative in chess, and that's a good thing.
You want the initiative.
You want to have the extra move over your opponent.
But there are some situations where if you move, you lose.
And so you want to make sure you don't
have a Zugzwang situation.
I don't actually know if there's a Zugzwang
situation in Leiserchess.
So if somebody comes up with one,
I'd be interested to see that, where if I had to move,
I lose--
but if I sit there.
So in chess, what they do, is in the end game--
that's when Zugzwangs come up--
they turn off the null-move pruning.
So is this clear what null-move does for you?
What's going on there?
I see some people who are quizzical.
Anybody want to ask a question?
I see some people with sort of quizzical looks on their faces.
So the idea is for null-move, my position
is so good, that even if I do nothing, I still win.
So I don't want to search anymore in this part of it,
because I get a beta cutoff, you're
not going to let me make this move--
the move that got me here.
Yeah, question?
AUDIENCE: That was it.
CHARLES E. LEISERSON: OK.
That was it.
OK, good.
My move is so good, I can do nothing, and I still win.
And so why bother?
Let me verify that without necessarily
doing as deep a search.
So null-move is fairly cheap to do,
and it often results, in a lot of positions, for cutoff.
And if I don't get a cutoff, then I just
do the normal search.
This is a pretty complicated piece of code, isn't it?
Pretty complicated.
There are other situations.
We mentioned killers.
There's also move extensions.
So typical move extensions that people look at
is you grant an extra ply in chess
if the king is in check, or for certain captures,
or if you have a forced move.
So suppose you have a move and it's
the only move you can make, then don't count that as ply.
So you can search deeper along lines
where there are forced moves.
That's what they do in chess.
In Leiserchess, not quite so clear what you do--
which ones you would do extensions for.
Because it's rare that you have just one
move in Leiserchess, compared to an in chess.
By force move, it's like, if you don't do this,
you're going to get captured, or mated, or what have you.
So that may come up in Leiserchess.
But anyway, it's something to think about.
So the transposition table, we talk
about this as a search tree, but it's really a dag,
because I can get to the same position by transposing moves.
This guy does a, this guy does b, this guy just c,
this guy does d.
It's the same thing by doing a, d, c, b.
I transpose those two moves, and I get
to exactly the same position.
And so I'd like not to search that position again
if I've seen it.
And that's what the transposition does.
There's a quality score that is in the transposition table that
tells you how good to move you've made.
And the quality is essentially the depth
that you had to search in order to establish
the value that stored in the transposition table.
And so you don't want to use something of too low quality
when you're searching.
If you have to search the depth d,
something of quality d minus 1 is not good enough.
Because, otherwise, you'll not be searching the full tree.
But something typically that is deeper--
if I'm looking at depth d and I find something
in the transposition table that's d plus 1 or d plus 2,
that's great to use.
That just gave me an even deeper view
of what's behind that move.
And so if you look at the logic there,
you'll see that that's how they do it.
It's very tricky.
One of the things that's tricky is
when you find a mate, how you store that in the transposition
table.
And you'll see, there's special code
for handling mate positions.
And the reason is because what you're
interested in doing when you find a mate is
knowing your distance to mate.
But not from that position--
the distance to mate from the root of your search.
So, for example, let's say this is the root of my search,
and I search down to a given point.
And now, I do a lookup in the transposition table,
and I discover that there's a mate in 5 here.
And this, let's say I've searched 9 ply or something.
Well, if I store that this is a mate in 5,
and I store the value for mate in 5,
then if it makes it up to the top here,
it thinks there's a mate and 5.
But there isn't a mate in 5, there's a mate in 14.
So when you look it up and you discover
there's a mate and 5 in the table,
you have to do a little bit of a calculation to translate that
into being a mate in 14 as the value
that's going to be used here, rather than a mate in 5.
Does that makes sense?
So you'll see, that's the logic that's in there for dealing
with mates.
So a mate basically takes a very large number--
way larger than all the material on the board--
and then it subtracts what your number of ply
is to get to mate.
So some big number--
I don't know, 32,000 or something--
minus the depth to mate is how you represent a mate.
So it's some very, very big number
and then minus the depth.
And that way, you can make sure that you're
going for a mate in 13 instead of a mate in 14,
for example, preferring that you take the shorter path.
Makes sense?
So that's one of the things to look at in there.
And there's also a lot of stuff in there.
The transposition table has--
we talked about caching--
it's actually a k-way associative cache.
And so you can vary k to see what's
the best choice of how many entries you should have.
The more entries you have, the longer
it'll take you to search them.
On the other hand, the more likely
it is that good move stay in the cache.
So you can decide what's the right trade-off there.
So there's a good tuning optimization there.
Questions?
Any questions about transposition table?
Transposition table is a major optimization.
So Zobrist hashing-- so most of these, Helen
talked about at some level.
But I wanted to give a chance to have people ask questions.
Nobody's asking questions.
I came here for Q&A. All you're getting is A.
[LAUGHTER]
So let me explain Zobrist hashing again a little bit.
So Zobrist hashing-- very clever idea.
So we have our board with a bunch of pieces on it.
That's probably the wrong number of squares, right?
That's seven by six.
OK, who cares.
So the idea is that what I'm going to do
is have a table of random numbers
that I'm going to compute in advance.
And this is going to be indexed by my row, my column, my piece
type, and my orientation.
And every different combination of row, column, type,
and orientation corresponds to a different random number
in the table.
So we have some sort of random number there.
And what my hash function is is it's
the XOR of all of the values of all the pieces
that are on this table with their orientations.
OK so if I have a king here, that's a white king--
I guess I need to know not just what
the type is, I also need to know whether it's white or black--
so the side.
I think, actually, that gets encoded somehow.
But in any case, I think maybe in the type
we actually keep whether it's a white pawn or black pawn.
Yeah, it's in the type, I think, is the way we actually
implemented that.
So there's white king, black king, white pawn, black pawn,
or space.
So if I have a king there, that corresponds to a certain value.
If I end up with a pawn here--
let's say, a white pawn--
then that will be another one, and I XOR those together.
And I take the black king and I XOR
his position is, and the black pawn, XOR that in.
So I XOR all these four values.
That's the hash function that I use
to do things like look up things in the transposition table.
Now, if I had to compute this every time
I changed the position, that's if I have a lot of pieces--
how many pieces I've got, 7, 8, 16 pieces?
Each side has seven pawns and a King.
So 16 pieces-- I have to do 16 XORs in order
to compute my hash function.
So Zobrist hashing is really clever.
It takes advantage of that XOR trick
that I taught you in the first--
no, it wasn't the first, it was in the second lecture?
Yeah, the bit tricks lecture.
And that is that XOR is its own inverse.
So if I want to remove a piece-- let's
say I'm going to move this pawn from here to here.
So I remove this piece.
What do I do to my hash function to remove a piece?
I look up the value for that pawn in the hash table
and I XOR that into my hash for the table.
And I now have a hash for those three pieces left.
Does that make sense?
So I have my hash function, which
is a hash of the four things.
I'm not sure you guys can see very well there.
And I simply XOR it with the hash of the pawn
that I removed in this case.
And now, I moved it to here.
So what do I do?
I look up that one and I XOR that value in here
for the new position--
the new pawn position--
and that gives me, now, the hash for the new table
with the pawn in that position.
So any move that I make, I can basically
update my hash function with only two XORs.
And XORs are very fast.
They're one instruction.
And they can do lots of XORs and stuff.
So this is actually a very, very cheap thing to do--
a very cheap thing.
So that Zobrist hashing, that you can keep these things up
to date.
And that's something we implemented for you.
That's an optimization that was a freebie.
We could have given you the hash function like this
and had you implement that, but this
is one of the ones we gave you as a freebie for optimization.
You're not all saying, thank you very much, Professor Leiserson?
[LAUGHTER]
No, in some sense I took away an opportunity for optimization,
didn't I, there?
And so in the transposition table,
there are records for the Zobrist key, the score,
the move, the quality, also a bound type,
whether it's upper, lower, or exact.
Because when I return something from alpha-beta,
I only know a bound on it, if it's greater than alpha or less
than beta.
And in some sense, the age of how old is this move.
Because as things get older, I also
want to age them out of the table.
There are several aging things there.
And you'll see the best move table also has
an aging process, whereas every time it updates the values
and gives a new value, it ages all the other values
so that they gradually disappear and aren't relevant.
One of the ones that people get confused about
is the Late Move Reductions--
so-called LMR.
This is the situation where I'm going to do, let's say,
my parallel search of all my moves.
And the question is--
once again, you're trying to prune everything you can.
And so the idea is, which are the moves that are more
important to search deeper?
The ones near the beginning of your move list,
if it's sorted in best-first order?
Or the ones towards the end of your move list?
So where is it most important to search deeply?
For things that you think are possibly
the best move or the ones that you think are the worst moves?
Yeah.
AUDIENCE: Search for the best move.
CHARLES E. LEISERSON: Yeah, it makes sense
to search the best moves, right?
So the idea is, well, if something
is way down on the move ordering list, why search it as deeply?
It's probably not as good a move.
And so, let me search it more shallowly?
I probably don't lose much of an opportunity
to discover that that's actually is indeed a bad move.
There's a reason that got it ordered down there.
And so that's a late move reduction.
So with a good move ordering, a beta cutoff
will either occur right away or not at all.
So you search the first few moves normally,
and then you start reducing the depth for moves.
I believe, in our code, we have two numbers where
we reduce by depth 1 after a certain number of moves
and reduce by depth 2 after a certain number of other moves.
Those are things that you can tune.
I wouldn't tune them very much.
I don't think.
And once again, I could probably be wrong here,
and someone will discover, oh, if you tune it like this,
it's way better.
But that's the idea of the late move reductions.
Probably one of the most important things to think about
is the representation of the board.
Right now, we represent the board as it is.
That's a terrible representation.
It's very time-consuming.
There's sort of two major ways you can do it.
Oh, I didn't put the other one here.
Well, anyway, one of them is bitboards.
Here, you use a 64-bit to represent, for example,
where all the pawns are on the 64 squares of the board.
And then you can use POPCOUNT and other bit tricks
to do move generation and to implement other chess concepts.
So if you're looking to see what are
the possible places a pawn can move, and you want to say,
can they move right?
And let's say it's stored in row major order,
you can just do a right shift by 1,
and that tells you where all the places
that those pawns could move.
And now, you can just pick them off one bit
at a time to generate your move list.
And then you can do it that way.
If you're going to move up a thing,
well, then you're actually doing a shift by or down.
You're doing shift by how much?
By eight, to move up or down, if you're
storing things in row major.
That makes sense, right?
So if it's 8 by 8, and you're keeping a bit for each thing,
then if I want to generate where is this one?
If I shift this whole thing stored in row major order
by 8, if I shift it right, it basically puts it there.
So I'm moving by 7, 8, and 9, that gives you--
and then shifting it by 1 or minus 1
gives you this, or left by 1.
And then, similarly, you can do by shifting
by 7, 8, or 9 that way.
And I can generate all the possible moves.
So that's one way of doing move generation is using bitboard.
And there are a lot of things, for example, that you can
do with bitboards in parallel.
Because you can say, did I make a capture here?
Or let me use a bitboard to represent where the laser goes.
Did I make a move that is going to affect the path of laser?
Because one of the major optimizations
you can do is in the evaluations in dealing with the laser.
Because you're spending a lot of time
stroking out laser positions.
What's the point of doing that--
if you made a move of something and it
didn't affect where the laser goes why bother doing it out?
You can just cache what the value of the laser is.
And there are a lot more good stuff on the chess programming
wiki.
So, yeah, question?
AUDIENCE: How do you [INAUDIBLE] a right shift when
the [INAUDIBLE] if the shift by 8,
and it's clear when it falls off [INAUDIBLE]..
CHARLES E. LEISERSON: Yeah.
You got be careful there, right?
Because if I do a shift to the right,
for example, what'll I do?
I'll just do a mask to eliminate all the things that
got wrapped around.
So two instructions or whatever.
Yeah, details.
Yes.
Details are good though.
Good question.
Good question.
Whose timed their program?
Where are the opportunities that you
see for performance engineering for a first pass?
What are the expensive things?
What do you have as an expensive thing?
AUDIENCE: Add laser paths--
[INAUDIBLE]
CHARLES E. LEISERSON: Sorry the?
AUDIENCE: Marking the laser path.
CHARLES E. LEISERSON: Marking laser path, yep.
Good.
What else is time expensive?
AUDIENCE: Laser coverage.
[INAUDIBLE]
CHARLES E. LEISERSON: Yeah, laser coverage.
Boy, that is really expensive.
That is really expensive.
So where else is expensive?
What else is expensive?
So I would definitely go after l cover.
That's a huge one to go after.
One of the things, by the way, if you are making your changes
to the code and you're going to change the representation,
leave the old representation there.
Take it out at the end.
Add the new representation and put
in assertions that say that things are equivalent
or whatever.
But don't get rid of the old stuff,
because you'll end up with broken code.
And definitely, used things like perft
to tell you whether or not you made any change If you touch
anything with move generation.
So where else is expensive?
Yeah.
AUDIENCE: Can you explain the laser [INAUDIBLE]??
CHARLES E. LEISERSON: Sure.
How much detail do you want?
How it actually works?
AUDIENCE: [INAUDIBLE]
CHARLES E. LEISERSON: OK.
So what is supposed to do is figure out how safe--
the idea is, I want my laser to get closer to your king,
and I want your laser not to be close to my king.
And if I can move into positions where
my laser is closer to your king but your laser doesn't
get closer to my King, that's a good sort of thing.
But when we say get the laser close, what
happens when I've got, say--
let me do it this way--
a position like this.
So here's the--
OK.
Suppose I look at this position.
How close does the laser get?
Let's say I'm black here, and I look at the laser.
Well, the path of laser is it bounces there and goes across.
So I didn't get very close to the king there,
but I'm one move away from getting it pretty close.
Because if I just move this guy out of the way,
now I've got it really quite close.
So if I compare that to, let's say, a situation
where instead of the pawns are there,
let's say a pawn is here.
Now, the laser is actually closer to the King
than it was in the first position,
but it's a much worse position.
The first one was much better when
I had the pawns here and here, because I was simply
one move away from getting it really close.
And so if you use just the direct laser thing--
and we did some tests on this--
this turns out to be not very--
it doesn't guide the program very well
on getting into situations where my laser is
getting close to your king.
Does that make sense?
So then we said, well, how should we
measure how close the laser gets?
So what we said is, well, let's take a look
at all the possible moves from here of one move
and then look to see how close we get things.
And the way we did that is we said--
actually, Helen and I worked on this really hard,
because this is a new heuristic that we have not
used in previous years.
And it works great, it's just slow as anything.
But it works well, so you have to evaluate,
is it worth doing something if it's really slow?
It may be that you'd do better to use a simpler heuristic
and get deeper search than it is spending
a lot of time evaluating.
But anyway, we gave you one, that if you can make it
go fast, should be really good.
So the idea is that what we do is
we look at all the different paths from moving any one piece
and look at the paths of the laser.
And what we do is we go we count 1 for every position
that we go away--
2, 3, 4, 5, 6, 7, et cetera.
We actually add an extra value if we bounce off
an opponent's--
how much do we add, Helen, do we add 1 or 2 if we bounce off
an opponent?
AUDIENCE: 2.
CHARLES E. LEISERSON: I think, 2.
Yeah.
So if this is an opponent's pawn,
we would basically go from 3 to 5, 6, 7, 8, 9.
And we basically do that for all the moves.
And the way we combine them is we
take the minimum number that I can get there.
What's the what's the cheapest way I can get to every square
that the laser goes?
So we take all the different moves, we stroke them out,
and we take the minimum.
So if I have another path, let's say
by turning the king it will go this way,
and there's a you know there's another one here
that's my pawn, then I may get there in 1, 2, 3, 4, 5, 6 7.
And so then this would become a value of 7 here.
so that's the first thing is we basically get
a map of how close are things.
And the second thing we do is say, well,
how valuable is it to have these to be near the particular king?
And then what we ended up with is something where
if I look at the distance that I am away--
let's say, this one, for example, is one row
and one column away, this is 0 row 0 column--
we use, I think, it's 1 over the row plus 1,
times 1 over the column plus 1, as a multiplier
to weight how good it is that we've been close to the king.
And we invert these values so that it's better
to have a smaller number there, and then we add them up.
We don't quite add them up.
No, I'm sorry.
What we do is we look at these things
as a fraction of my shortest path distance.
Sorry, that's what we did.
Yes, I'm sorry.
That was a previous heuristic.
So here, for example-- let's say this is the 9--
the shortest way of getting there is 1, 2, 3 4, 5, 6 7, 8.
So this would actually, when we do the conversion,
would give a value of 8/9 for being in that square.
So we didn't get it.
Whereas this one, the shortest path distance
is 7 because of this path, and you can get there in 7.
This would have a value of 1.
So this is better.
And so we do that for all the squares.
So we get a fraction of how much do we do.
And then we wait it by something like this,
which falls away quickly.
But it's important, in heuristics like this,
to have a smooth path, so that you can get things closer.
If I made all these things be 0, it
wouldn't know that it could move up and get a little bit better
in the second or third order digit to get closer.
And so then we add it all up, and that
ends up being a number, we discovered,
that's sort of in the range of--
what did we figure the range was there?
It was up to like 4, or so, right?
Something like 4 would be the maximum value it would be.
And then we said, OK, then we have
this magic constant multiplier that you can play with,
that says, OK, let's represent that as a fraction of a pawn.
How much is that worth?
And we multiplied by that value.
So that's what we're doing.
So that it makes it so that we do
tend to move our laser closer, we
subtract how close the opponent can get from us,
and then we say that's the evaluation.
Now, if you have a better way of determining whether a laser is
getting close than this one, that's cheaper to compute,
that's good.
For first pass, I would just simply try
to make this go fast.
Because for many of the moves that you're going to look at,
it's going to be moving this pawn to there.
Nothing changed.
There's no point in stroking this out, and calculating
all the minimum, et cetera, because it
didn't touch the laser path.
Things that are going to touch a laser path
are things that you move on or off the path.
So why bother?
And so if there's a clever way of caching it so that you
can do things, that's good too.
Any questions about other things that could be done?
So another place, I'll tell you, that's a good idea to look at
is--
especially once you get this heuristic top rate a little bit
faster--
is the sorting of moves is actually pretty expensive.
Once you figure out, here all the moves--
there are a lot of moves--
now you go through and you do a sort.
And if you think about it, in a best move order tree, sometimes
you only explore one of those.
So why'd you bother sorting all of them?
On the other hand, sometimes you do need to sort all of them.
So how you optimize that sort so that you
don't waste time sorting stuff that you're not
going to actually ever explore, that's
another opportunity for optimization.
Yeah, question?
AUDIENCE: How do you sort the moves [INAUDIBLE]??
CHARLES E. LEISERSON: The moves are sorted by these things
like--
there's a sort key in there that represents
a whole bunch of things, like whether it was a killer.
If it's a killer or it's a transposition table value,
it's a very big value.
If it's a statistical thing that the history table says,
historically, this has been a pretty good move,
then you get another value.
And then exchanges get ordered and so forth.
So there's a bunch of things that
are where you end up with information,
and then it sorts those things.
So that's actually kind of complicated logic
in terms of what the actual values that are used there.
But the idea is, here all the moves, now
let's figure out what's our best guess as to what they are.
Most of the good moves are coming out
of the transposition table.
But sometimes, the transition table it says it's
not a very good move, and there may be a better move.
So the quality of what you get out of transposition table
is good, but that doesn't mean that that's always
the best move.
AUDIENCE: [INAUDIBLE]
CHARLES E. LEISERSON: Yeah.
Any other questions about what should be worked on?
Let me just make sure I--
OK, go ahead.
AUDIENCE: Where does that sorting happen?
I've never [INAUDIBLE].
CHARLES E. LEISERSON: Where does the sorting happen?
That happens, I think, at the move generation.
Is that where it is?
Is it stored in the move generation?
AUDIENCE: In search.
CHARLES E. LEISERSON: Or is it in search?
I think it's in search.
I think you're right.
Search calls move generation and then sorts it.
I think that's right.
Yeah, question?
AUDIENCE: So one of the things we
did that we were providing [INAUDIBLE],,
we have a very tiny board implementation.
CHARLES E. LEISERSON: A very tiny board implementation.
AUDIENCE: Yeah, [INAUDIBLE].
You can do a lot of matrix on it as well.
But it's like maybe [INAUDIBLE] replace it everywhere.
CHARLES E. LEISERSON: Yes.
AUDIENCE: So what's a good strategy?
CHARLES E. LEISERSON: Well, as I say, keep the old one around.
AUDIENCE: [INAUDIBLE] still use [INAUDIBLE]..
CHARLES E. LEISERSON: Well, you use the old one
while you're still getting the new one right.
It's also good to be able to go from old representation
to new representation and having conversion functions.
But, yeah, making representation changes, painful, painful.
And boy, if there was one tool I would
love that would automate stuff like that, that would be it,
to be able to change representations of things
and still have things go well.
I should have mentioned, by the way,
the other representation besides bitboard.
Another one is just to keep a list of the pieces
and their positions.
Because that's going to be smaller than keeping
this great big board.
And do you keep this list sorted or do you not keep it sorted?
But the advantage of that is, particularly
as the game goes on, that list gets shorter and shorter.
And so if you're doing less in manipulating the board
position, that's generally a good thing.
AUDIENCE: So for the board reputation,
my recommendation is to refactor all the board
axes into a function.
And then you would still use the old representation
in the function.
Then you can validate that you refactor everything correctly.
And then you can easily change the board representation
to whatever you want and keep changing it
without having to do it a lot of refactor after that.
So find all the board axes and refactor that to a function
call before you modify it.
CHARLES E. LEISERSON: Yeah.
Because the compiler will inline that stuff.
So putting in a function call is not necessarily a bad idea.
And if it doesn't, you can put it in macro,
and it'll be effectively the same thing.
So great idea.
Great idea.
Yeah, question?
AUDIENCE: Anything else that wasn't
there that you would highly recommend for us to look at?
CHARLES E. LEISERSON: Testing, testing, testing.
If you can test and know that when you make a change
it's correct and that you're not--
that's probably the number one hole that people go into
is they don't test adequately.
So having good test infrastructure is good.
Yeah?
AUDIENCE: Some questions about codes counts--
I saw on Piazza there was [INAUDIBLE]
if the beta changed the [INAUDIBLE] count will
stay the same.
CHARLES E. LEISERSON: As long as you're
doing it deterministically, yes?
AUDIENCE: Yeah.
But even the references limitation
is nondeterministic serially for [INAUDIBLE]..
CHARLES E. LEISERSON: That's because there's
a randomness thing.
There's a little variable, and you can set the variable
to be not random.
And then it will be.
AUDIENCE: Is that a part of the set option RNG?
CHARLES E. LEISERSON: Yes.
AUDIENCE: When that's set, it's still nondeterministic.
CHARLES E. LEISERSON: Run serially?
AUDIENCE: Yeah.
I checked the-- is there anything else that
should be done?
Or should it just be that seed option?
CHARLES E. LEISERSON: I think that's--
AUDIENCE: [INAUDIBLE]
CHARLES E. LEISERSON: [INAUDIBLE],,
can give to her the mic?
AUDIENCE: Yeah.
AUDIENCE: [INAUDIBLE].
Yeah.
Did you try the things that [INAUDIBLE] include
[INAUDIBLE] like [INAUDIBLE].
AUDIENCE: Yeah.
I got out of the opening book and I set the RNG option.
AUDIENCE: OK Let me test it again.
Because when I tested it , it was deterministic.
So [INAUDIBLE].
AUDIENCE: OK.
AUDIENCE: [INAUDIBLE]
CHARLES E. LEISERSON: Yeah.
It shouldn't be-- if you run serially,
the only things that should be doing things is
if you're doing--
it should be deterministic if you turn off the random number
generator.
So as I say, that's put in so that it
will behave nonpredictably.
But that's exactly the kind of thing
you want to find right at the beginning.
So that's exactly the thing is find out
all the ways of making it deterministic,
and that would be really important for beta 2.
So Thursday, we have Jon Bentley, legend of the field,
opportunity to meet a celebrity.
Bring friends.
He's going to give a great lecture.