Placeholder Image

Subtitles section Play video

  • Hello World.

  • This is CS 50 Podcast.

  • My name is David Malin.

  • My name is Brian You, and it's been a while since our last episode, but that's only because Brian and I insist that this whole team have been rather busy here in Cambridge and in New Haven, teaching the fall version of CS 50 on campus at Harvard and Yale.

  • But we're back now, and we thought we'd begin the coming year.

  • But the discussion of how the past fall went indeed, the story really begins back in June of 2019 the start of summer here in Cambridge, when we gather the team together for so called Summer 50 a retreat where we all go away together for a few days to discuss the coming summer's projects, coming fall's changes and really the coming year's curriculum for CS 50 CS 50 x, CS 50 AP, and the theme that we decided to focus on back in June of this past year was a chapel auntie to move forward, to evolve the course, to change and make a really thoughtful reformulation of what it is we wanted to do and what it is we wanted to teach in the class.

  • And that led to a whole list of changes that we actually announced toward the talent of this past summer by a C s fifties medium Blawg titled A Job.

  • Volante is Well, it's kind of amazing looking through this list, just like how many changes there were that we were trying to put into effect this year.

  • Like it feels like the course definitely changes from year to year.

  • But this year, especially, it feels like there were a lot of new parts to the class and things that were just doing differently.

  • Yeah, absolutely.

  • This is probably in the 13 years that I, at least have been helming the class, probably the most significant overhaul of the curses, structure and curriculum on design.

  • Really?

  • So we thought we'd share some of the thinking behind what those changes were.

  • Us do a bit of debrief as to how we think things went and highlight not only the successes we think, but really some failures, things that we thought were a good idea in June of 2019 but didn't really pan out as we'd hoped.

  • So lectures for years have they been one of the defining characteristics of the course in so far as they're designed to introduce the week's concepts, particularly those concepts that students need for application on the courses, problem sets or programming assignments.

  • And in recent years, it's worth noting that attendance here in Cambridge in person on campus really has been declining.

  • And this is not in and of itself problematic.

  • And indeed, we, the class and I philosophically have very much been in favor of what we call here on campus.

  • Simultaneous enrollment, the ability for students to enroll not only in CS 50 but maybe even in some other class on campus that meets at the same or overlapping time, the implication of which is that they have to physically choose to goto one class or the other.

  • Insofar, Osias fifties materials have long been filmed, and we placed such emphasis on the quality of the digital materials we have been comfortable in telling students go to the other class and watch CS 50 in person.

  • However, this is, of course, had the side effect quite causally off.

  • Relatively few students attending the courses lectures in person, even after just a few weeks of the class case in point, I think in fall 2018 we had maybe 152 100 students out of nearly 800 physically attending by only the third or so week of class.

  • And I don't think that's really specific to CS 50.

  • Like I feel like all across Harvard and probably universities more generator like If there's a way for students to watch lecture from the comfort of their home without showing up, there's gonna be much fewer people that actually end up showing up to the lectures.

  • Yeah, that's quite fair.

  • Certainly, in computer science and other fields here on campus, there's a long standing tradition of filming classes and that it certainly has value.

  • And so far, students can review material.

  • If they miss material, they can catch up.

  • But indeed, I do think we see students walking literally with their feet out elsewhere.

  • Well, I'm curious.

  • When you were in college, were any of your classes filmed and did you go to class or were you more of the watch?

  • The video?

  • Oh, I was super uptight and always the one going to class.

  • I don't know if I ever missed.

  • I can think of a class who's sections I missed.

  • I'll admit that I realized halfway during the term that an economics class I was taking actually had sections which somehow I forgot about.

  • But I want all of my CS lectures and sections, and at the time, if you missed a class back in, what 1995 to 1999 you could go to the library for some classes and asked to borrow the VHS tape, which you then watch on a machine there.

  • Although I did take a networking class at Harvard, I think in probably 1997 taught by Professor HT Keung, who's still in the faculty here in computer science.

  • And he was, to my knowledge, the first professor to stream his courses lectures online.

  • So it wasn't live streaming, but it used is a technology Microsoft Net show, I think.

  • Itwas so.

  • This is well before we had the sort of YouTube and empty fours and all streaming technologies of today, and we actually in a tiny window that was probably like 100 and 60 pixels wide, were able to watch lectures from home if we wanted.

  • We made a lot of progress and screaming now, but in any case.

  • This is something that we have been encouraging or certainly allowing and endorsing.

  • But I do feel in recent years and certainly last year when we saw these numbers, that too many students, I think we're sort of following in the footsteps of their classmates without necessarily making an informed decision themselves as to what would be best for them to learn the material.

  • I think too many first year's freshmen who had just arrived on campus, who had spent the past four years presumably in high school, going to classes and now we're suddenly stopping because they saw more upperclassmen doing the same was probably not the ideal informed decision, and ultimately it was just having a ripple effect, I dare say, like CS 50 has such an operation behind it in terms of so many teaching fellows and so many sections and so many office hours that if a student does not go to a lecture and then for whatever reason, ends up not watching the lecture live or on demand later on in the evening or even the next day, it means they show up at our on campus sections, led by the teaching fellows unprepared for that section, and then they dive into the problem set and they're unprepared for the courses material.

  • And then they come to office hours and then ask us questions for which they should have been prepared had they been prepared for section and prior to that in person in lecture.

  • So it really had an operational cost, I dare say, which then cost other students.

  • Ah, in so far as we had fewer resource is to answer their questions.

  • Yeah, and I really noticed that I think that really improved this year.

  • It felt like in section in office hours, like students came to them prepared, having been to lecture, whereas normally a lot of students would wait until later in the weekly cycle in orderto actually watch the lecture and begin to engage with the materials.

  • That was just more time to be able to think about things, to take the pace of the problems.

  • That's a little more comfortably just to make the week a little more manageable.

  • I'm really going to hear that because that, of course, was our hope that we would actually have this ripple effect in a positive way.

  • On subsequent support structures and oh my God, it's so liberating as a teacher to be able to say something or repeat something for a student and be able to assume that they were there the first time.

  • So you can draw contextual references and put things that the context where is in years past there was maybe a 25% chance that a student in talking with them had been to the most recent lecture.

  • So I dare say this change that we rolled out, which was, quite simply, to effectively require attendance at lecture.

  • Ah, the course had some nine lectures in total in person.

  • This year, an expectation was that students attend at least the 1st 8 of those which is not all that unreasonable.

  • There a couple of hours a week, it's only eight lectures, and so that felt like a reasonable compromise, even though philosophically, I dare say rarely and more libertarian, and that students should be free to do what they want.

  • But I do think it was a net positive to try to help students help themselves by effectively requiring attendance.

  • But even that keeping track of who was that attendance was non trivial.

  • O you were instrumental in solving that.

  • Yeah, we spent weeks trying to figure out, like, how do you take attendance for a class this big?

  • Like a lot of other classes, you can do that like, just call off all the names and mark check marks.

  • But that's not gonna work in a class with 800 students.

  • So we had to go through a lot of different possible options.

  • All the students here have I d cards and you can think about different ways of trying toe swiped them or try to scan them.

  • But even swiping them is gonna take a long time.

  • If you only have one swipe for, like, 800 different people, all of whom arrive in the five minutes before class is supposed to begin of, right?

  • Exactly.

  • And you wanna make sure that doesn't have a huge bottleneck in a long line.

  • So we eventually settled on a system where we have these, like four scanners, the type of scanners that you might see like at a grocery store checkout when they're scanning the barcodes on various items with laser scanner is getting the barcodes on the I D cards on.

  • We had four of them a different entrances to the lecture hall on that tended to work pretty well.

  • I think they're a couple of lines that built up right as lecture is starting, but generally it worked pretty quickly and we were able to get people inside.

  • Well, I'm so glad it worked out because it's a slippery slope.

  • We had talked about writing our own custom software and how you go about attaching some kind of scanner to a tiny device, whether it's an iPod or an iPhone or maybe a raspberry pi.

  • And they're just felt like it would be fraught with technical support challenges if something went wrong.

  • You're sort of dead in the water, but these things, they're pretty dummy proof.

  • They're meant to be used.

  • Ah, we saw for, like, warehousing applications, someone going up and down an aisle, presumably taking inventory.

  • Yeah, I think that was that's intended use case, and it works pretty well when you're trying to scan a lot of things very quickly.

  • So for our use case, I think it's pretty good choice.

  • Yeah, and so to incentivize students to come because it's not a road requirement that you come to lecture but it's an expectation we actually did for the first time.

  • Really, though we saw some prior incarnations of this idea on experimental form in past years, um essentially had a participation portion of students grade where you get it all if you meet all expectations coming to section, coming to lectures as well.

  • But it's amazing.

  • It's amazing to me that not only asking students to come to lecture but incentivizing it as with some fairly small numbers of points toward one's final grade, it worked wonders.

  • We went from like 200 somewhat students to 650 consistently week after week.

  • I was amazed every time you guys took attendance or reported back after the scanners were done, it being used like it was around 6 50 every week.

  • So doesn't feel any different when you're lecturing in front of 600 people of the post to a much smaller big time, feel more pressure when they're that many people.

  • Oh yeah, no, I We have photographic evidence.

  • I sweat every lecture this year.

  • Where is in the past?

  • I actually wonder.

  • I used to blame the temperature in Cambridge and the lack of strong air conditioning in the building, but I think it's my nerves, my heart rate.

  • And I should probably check my apple watches history of my heart rate during that 3 to 5 PM slot on Mondays.

  • Um, to see if that partly explains, but yeah, no, I was much more nervous.

  • You know as well as anyone that how much more I and we prepared before each lecture because you just have so many more eyes on you.

  • Uh, and it was good pressure in a way.

  • And the energy is just so much better.

  • We had many more volunteers.

  • There was more audience participation, both in terms of questions and answers.

  • And when you have the occasional applause or laughter, it's sort of infectious and a good way, and it actually does motivate, I think, the experience of coming to a lecture So we didn't want people there just for the sake of being there.

  • We wanted them have a hopefully an inspiring or at least an eye opening experience, and I think you can pull that off like eight times a semester.

  • I I would not want us to do what we did some 13 years ago and have three lectures a week so that we now have some 30 classes that students have to attend in person.

  • Beyond the sort of up front cost of setting up for and striking our TV set up for each of those lectures.

  • I just don't think you need 30 times together to sort of feel that shared community experience, like eight or thereabouts feels reasonable.

  • Yeah, I think so.

  • And I student.

  • It's nice because you didn't get Tau get to know each other based on the people that are sitting next to, and they can have conversations about the material, which you just can't do as well.

  • If you just at your desk alone watching the electricity Well, especially for first years, theoretically, maybe even rub shoulders with more people that you meet and then stay in touch with more than you would if you were just in the library or at home watching on your laptop.

  • Now we should concede that we also changed the day of lectures this past year.

  • Past couple of years we've been using Fridays, or at least last year we did, and it was Friday mornings of all things I think was 99 a M officially to 11 45 that we typically started a little later closer to 9 45 so that we still had a two hour window.

  • And we know statistically from CS 50 Zone Data and Harvard College.

  • His own data Friday is hands down the least popular day for classes, both among students and reportedly among faculty.

  • It's hard enough to get students awake in the morning, let alone Friday mornings.

  • I'd let alone David and the team, but but we steered toward Friday because of exactly this.

  • It was wide open on Sanders Theater schedule, which meant we had the ability to book the room the entire day, which is great for CS Refuse Production Team because they Scott Dr 6 a.m. for a 9 a.m. or 9:45 a.m. Class just to set up the A vehicle.

  • It mint in technology and test everything, so there's a huge amount of set up, So just having that runway was liberating.

  • Plus, theoretically and tragically, it didn't work out this way.

  • You'd think more students would be free on Friday mornings.

  • 9 a.m. because statistically they don't have conflicts or other classes and there probably aren't extracurriculars going on at that hour, but there is a lot of sleeping or non attending going on at that hour to Yeah, a lot of Steven's just tried to, like Free up your Friday like it was a goal among many of my friends senior year to, like try and arrange your schedule So you didn't have to go anywhere on Friday, and people tried to get the view, but, yeah, they were very successful at that inn.

  • Follow 2018.

  • So in short, I think this was a net positive, I dare say.

  • I think the lectures and hopefully the audience, is the students.

  • Engagement with them was so much more positive, an experiential this year than in past years.

  • Hopefully, we really did set students up for more success each week in so far as they now had gone to a lecture and therefore went to section prepared and then went to office hours or didn't go to office hours because they didn't necessarily have a CZ many questions and ultimately succeeded.

  • The problem sets because they had all of those building blocks ready, ready for them and without jumping ahead too much.

  • I think it's I would like to think, although there's many other factors that we should discuss some day CSF.

  • These grades ultimately skewed higher this year on the courses problem sets, and we think this might be for a few reasons.

  • But I would like to think that one of the reasons was that we actually set students up for a better educational experience simply by incentivizing motivating, expecting that they take advantage of the resource is that frankly, have always been there.

  • But we're perhaps being underutilized.

  • Yeah, that would be nice.

  • I hope so.

  • I certainly hope that's the case.

  • Me, too.

  • Me too.

  • Well, this wasn't the only change, certainly.

  • And we hypothesized that by moving see his fifties lectures to Monday afternoons and not having sections until Tuesday's and Wednesday's.

  • That allowed us Monday evenings to even provide students with an opportunity for review.

  • And this is a term we keep deploying in different forms for several years.

  • But this year, Super Section took a new and improved form.

  • Yeah, so the goal of the super sections, which were kind of new this year we're to be an opportunity for students after a lecture after they've had a little bit of time to think about things, have some dinner to come back and ask questions about that day's material because after the lecture, the problems that's immediately released that theoretically students could begin to work on it.

  • But often times in a in a long lecture, where a lot of topics are covered, not all students are gonna get a chance to ask all of the questions that they might have, especially in a classes Biggest Yes, 50.

  • So one of the hopes I think of the super sections was to just be a chance for students to sign in.

  • Virtually in a virtual classroom, you zoom, which is like a video conferencing tool to do it, and students could sign in.

  • And then I would lead these weekly super sections where students could ask questions.

  • I would show some additional examples based on the lecture material, just to give students some more opportunities for Q and A and some more opportunities Toe really get a firmer understanding in that day's lecture material in preparation for that week's quiz, but also in preparation for that week's problems that overall, I feel like they were very helpful for the students who were there like they got their questions answered.

  • And there were always a lot of questions that students did have.

  • They were super well attended.

  • I'd say maybe we had anywhere from 15 to 30 people on any given week.

  • Um, I think part of that is probably just you've already been to a two hour lecture on Monday, and now that's another hour Plus, on a Monday evening.

  • Maybe it was overlapping with some people's dinner, other extracurricular commitments.

  • But I do think that on the whole for students who attended, it was helpful.

  • And we made the recordings available to students afterwards.

  • So even if you weren't able to attend in person, you could review them later.

  • And in essence, students did that for preparation for the exam from that type of thing.

  • Yeah, and I do think I, as you know, was such a fan of the idea of trying out zoom or video conferencing more generally for this, because if you decrease the friction for students being able to access something like this right, they don't on campus have to walk 15 minutes to get there.

  • They don't have to walk 30 minutes after, especially if then we have one or a few questions you would think that many students wouldn't be availing themselves of resource is if there's two high of ah operational cost to them or too much friction between them and the resource.

  • But I was disappointed honestly, like it is great for the 15 30 or so students that attended, and it was often a recurring group of students.

  • We saw some familiar faces many weeks, but I don't know.

  • I mean out of 800 students on campus, 100 plus through Harvard Extension School is 15 to 30 out of 900 ish.

  • A success or a failure.

  • Do you think for something like this?

  • I think it's probably too small to really call it a success that it was a fair amount of time investment that the production staff had to be there in order to make it work and opposed to produce the video and make it available online on DSO.

  • I'm not sure now.

  • Granted, part of it may be that there were just other avenues that students had to ask questions.

  • It's not one of our intended changes that we originally reported here.

  • But we started working with a new question and answer software tool called Ed this fall for the first time that just, I think, with a much nicer interface for students to be ableto quickly, ask questions and get feedback from the staff.

  • And I feel like a lot of students ended up.

  • Using that tool is a way of asking questions, too.

  • So maybe that was another avenue that students found the opportunity to ask questions.

  • Would you be inclined to have us Nick's them the coming year?

  • Do you think it's a good question?

  • I think right now I would probably lean against doing the super sections, at least in the same form that we did them this past fall if it didn't seem like there were nearly high enough impact.

  • But the super sections have evolved a little bit over time.

  • Like two years ago.

  • They were in person, but they weren't every week, and they were just on the weeks when we thought the students would have needed them and then attendant with a little bit higher s.

  • So maybe I wouldn't keep them in their current form.

  • But maybe there's some changes we can make.

  • And I'd be curious to hear what, like feedback from students is later about them.

  • Yeah, absolutely.

  • And we did a survey students throughout the semester, so we have some good data there.

  • But this is on a concept super section so larger than life sections aware.

  • We've tried for years to get these off the ground successfully, and they always have been the sort of middling success.

  • And I remember some 10 years ago when super sections were only in person.

  • I think by the end of the year we had one student attending at the end, but, you know, fun fact she went on to be our head teaching fellow for three years.

  • So isa wonderful recruiting mechanism.

  • It seems to see who's really, really passionate about the material.

  • And another one of the goals of the Super section, at least this year, was because it was the first year that we had weekly quizzes, which I do want to ask you about.

  • The super sections were designed to be an opportunity to help get some question and answer, get understanding in the material in order to allow students to be ableto work on this quiz is So I did want to ask you about about those quizzes like, what was the motivation in your mind behind them?

  • And how do you feel like that?

  • Went around a really good question.

  • So Society for many years has had a tradition of either quizzes or tests or which were really just midterms in some form, and the format of them has evolved over time.

  • But in recent years they've taken the form of sort of thought based, short answers, questions that aren't really regurgitated of recent material from the class.

  • They really are meant to pluck up, pick up some idea and take it further than class time allowed for or, frankly, just introduced some new topic altogether.

  • And so this year's quizzes, which were weekly Google form based, small assessments two or three questions that we gave to students after each lecture do the next morning, the idea being we wanted them to submit these quizzes after they had attended or maybe watch lecture.

  • We wanted them to submit them before they attended sections so that they were there by with higher probability arriving, prepared for section and awesome.

  • Honestly, we also wanted to use them as a distribution vector for the types of questions that we didn't really have much time for in other Windows during the term.

  • We only had time in terms of the academic calendar this year, for one test was in some years we've had two such tests or two quizzes or the like, and we wanted to be able to integrate some kind of short answer questions throughout the course that don't necessarily belong in a coding based problem set.

  • These really are different types of assessments, and they're not even just assessments.

  • They were meant to be also instructive in some form.

  • So they ended up, I think, by design, and I was complicity in this probably being a little more evaluative of the previous material.

  • I think we tended to cherry pick relatively accessible topics, in large part because we knew students had on Lee just seen the material they hadn't yet had an opportunity to review necessarily or think about it or applied in the problems that so I don't think I was.

  • I was in love with the types of questions we asked, but as a mechanism for incentivizing preparation for section, I do think they were successful.

  • I mean, students were putting in the time, and if there was any design flaw besides the wth each ideal questions I wish we had been able to ask was really just the time frame.

  • I think if a student finishes CS 55 p m, even if they're there in person, then they go out to dinner than maybe they have a commitment or they certainly might have work in other classes.

  • Not much room in the evening toe actually fit it in before the deadline.

  • But of course, our section started on Tuesdays around noon time, and so we wanted them to submit in advance.

  • So I think my takeaway is they're worth trying.

  • I think they were sufficiently successful, but I think we need to give students a slightly wider window of time.

  • Yeah, it's tricky just how I wish there was more time in the week to be able to, like, have an extra day or something where we could in orderto make room for this type of thing because I do think it was a good thing that the sections were earlier in the past.

  • Last year you mentioned that the lectures were on Friday and cause that was at the end of the week, it meant realistically, most sections didn't happen until Monday, which meant we sort of lost that weekend of time.

  • This year, I thought it was actually really big improvement.

  • That right after a lecture on Monday Section started on Tuesday, which meant very early on in this weekly cycle of lecture than section than problems that students were able to see some more examples, get some more practice and then really feel like they had the ability to go forth and actually start working on the problems that earlier than they would have been able to last year.

  • And I think that was a really good thing.

  • But it also meant less time for the quiz.

  • We really need an extra day, at least between Monday and Tuesday.

  • So if anyone out there is listening, we need in a day a week and then sees 50 would be it would be better Well, so Sections two.

  • We have taken varying approaches to over the past few years, and this year we, uh, continue to expect attendants, which was something that we rolled out a couple of years ago and We also tried to make them more hands on because of recurring bit of feedback from students is really that they don't have much time or much opportunity between lectures and P sets to really apply what was in lecture, but then is expected of them in pieces.

  • So how successful do you think this was over all but the team?

  • Yeah, this is a really common piece of feedback.

  • So every semester after 50 is over, I go through and read our cue evaluations.

  • Accu evaluations air Harvard's student evaluations of the courses where students answer a whole bunch of questions about things they liked about the class, things they didn't like.

  • And we go through and we read this every semester to see, like, What is the general feedback?

  • And a big piece of feedback about the sections was like, I really like my section leader, but it feels like Section is dressed, a rehash of lecture feel like we're going over the same topics from Lecture on, and there were a couple of reasons why that might have been the case.

  • One is that last year a lot of students came to section having not watched the lecture.

  • And so, as a result of that, Maura affection ended up having to be like Let's go through the lecture material again to make sure that no one feels totally lost during the course of the section.

  • So I think one improvement this year is that because we could assume with higher probability that students were actually at section, that the sections were able to be a little bit more interesting.

  • We didn't have to rehash what happened in lecture, and we could actually go forward a little bit.

  • But another common piece of feedback, with students saying that they wished they had more opportunity in section toe actually try things hands on.

  • But it felt like they understood things conceptually in lecture, but then struggled when it came to working on the problems that to be able to know how to actually write the code to implement what they had in their mind.

  • And I think that's a common struggle in computer science.

  • You can understand a concept or an algorithm in theory, and conceptually and when some one draws a diagram about it.

  • But when it comes time to actually start writing the code like that can introduce new challenges, and it can be tricky in ways that you might not have expected before.

  • So one of my big goals with sections and what I was trying to work with, a lot of the teaching fellows about was making sections very hands on to be ableto talk about topics but really to give students opportunities to work through practice problems, to try approaching problems in different ways to let them work in pairs together as they tried to write code to solve smaller problems not as big as a problem set, but still related concepts that were going to come up on the problem set just to give students practice with taking a problem, formulating an idea in their head and actually writing the code to be able to solve.

  • And I think that ended up working really, really well that for a lot of students for whom CS 50 is their first exposure to computer science and programming, just that little bit of extra practice with someone their toe help guide you and Thio to point you in the right direction.

  • If you find yourself lost, you're struggling that that adds a little bit of confidence that you feel like, Oh, you can solve these smaller problems and now this larger problems that maybe is a little bit less intimidating.

  • Well, I'm really glad to hear, because this is something we've been trying for years, and it's such a simple thing.

  • The fact that students air more prepared is theoretically for class.

  • That that makes a difference.

  • But hopefully we're getting even the finer points just right now, too.

  • Yeah, and I feel like in just talking to students, I seem Thio anecdotally, enjoy their sections and feel like they were getting things out of it.

  • Our course evaluations for this semester haven't come out yet for the college.

  • I think they're supposed to come out next week or so, but I'll be curious to read those and you'll see what all the feedback is about sections and lectures and all the other changes that have been happening with the mister.

  • Yeah, absolutely.

  • Well, in another big one, too, was really the third of the primary curricular support structures for students known as office hours, which for us take the form of one on one opportunities for help with the courses problems that and over the years.

  • We've offered these in different forms, usually several days or evenings of the week.

  • We have reserve some locations on campus, whether it's a computer lab or dining hall or library for students to drop in with their laptops and ask questions of the courses staff.

  • And we usually have multiple members of the staff teaching fellows and course assistance there to answer questions.

  • But you know, for a CZ, long as I've been teaching CS 50 and frankly, probably taking CS 50 back in the day, have office hours been one of the greatest challenges of scale with a class of CS 50 size.

  • Because even though we throw as many resource is as we can, as best we can, it invariably there's a line.

  • There's a weight, and it's significant sometimes.

  • I mean, we have quantitative data over the years where some your students were waiting at least an hour just to get their question answered.

  • And if you like in this to the real world, where maybe you go to a store and you're waiting for help from someone, er you call someone on the phone or open a chat window to get technical support.

  • I mean, that's the same kind of experience in these air students in an academic environment, just trying to ask questions and clear up some confusions.

  • It's been really hard to keep up with the load.

  • So this year we tried to rethink these fundamentally, and we have tried for so many years to gather students in one big place like a dining hall were or library and a lot of members of the staff, sometimes 10 or 20 or even 30 plus members of the staff all at once.

  • But even then, we've never really achieved an optimal ratio of staff to students.

  • And if you think about it, even if you have as good of a ratio, seemingly good is like 1251 teaching fellow to five students.

  • Even if that teaching spelt fellow spends on Lee 10 12 minutes with each student, it's gonna take them an hour just to get back around to the first student again.

  • So that's the sort of sense of the problem here.

  • So we rolled out tutorials and tutorials for us were opportunities to for students to sign up for office hours quite simply by appointment, and we capped attendance at these tutorials at five, maybe six students, the idea of which was, even though that's not necessarily on a deal ratio.

  • We remove students from the stressors of from the pressure's up from the chaos of these larger spaces that, frankly, were all too conducive in a bad way to stress and rising stress levels.

  • And we have spread them out on campus in offices and conference rooms and dorm rooms and dining halls.

  • And what do you think?

  • How did they go?

  • I think they were an improvement, and the word you used before let the chaos.

  • I think that was a good word to describe what office hours were like in past years that a student shows upto office hours, and they're just so many other students that are all trying to get help at the same time.

  • And there's a limited number of staff that air themselves stress because they're trying to balance between everyone.

  • There was a staff member who I took the class last year, and when I was talking to her about what her office hours experience was like, we were talking about officers this year, she said to me last year, it felt like going to office hours was like going toe war.

  • You would like show up and you would have to, like, fight against all these of the students in orderto get a couple minutes of attention from one of the staff members on Dhe that just wasn't healthy for anyone.

  • Nothing was healthy for the students of the staff members know I used to be terrified of Thursday nights when problems that's were due on Fridays.

  • Yeah, and I think the tutorials just they feel so much calmer, like the ratio is 1 to 5.

  • And we eventually had to change it to 1 to 60 because demand for them was so high, which I think is a testament to students valuing the help that they got at these tutorials.

  • But going to a room where there's just four or five other students just feels way healthier than going to a room where there's 100 people all trying to get help at the same time.

  • That was a small group of people.

  • That was certainly our hope that you could.

  • The fact that the teaching father was just a few feet away and not a few tables away just meant that you knew sort of socially that they're gonna get to you shortly, even if it is maybe 10 or so minutes, it just doesn't feel the same.

  • And yet, tragically, I mean, there was motivation for our larger scale office hours, so to speak, especially in the dining halls in office hours one.

  • We originally hoped that they would just be much more social and much more interesting than the basement of Harvard Science Center, where they once were back in yesteryear.

  • But to there's just a nick on Ammi of scale, right, with a 126 ratios in some corner of campus, there's arguably in inefficiency.

  • And if there's some quiet time or little time, that teaching fellow is just kind of standing there waiting.

  • They're not actually helping anyone, whereas in the library or dining hall, if there were student at the other table, that TF could wander off and go help them.

  • So I do think in theory, the larger scale office hours were a better allocation of resource is, but it came with a lot of negative externality.

  • Yeah, and there was also the benefit of having many staff members in the same place, like one common thing.

  • It was struggle with the tutorials.

  • There's one staff member with five students, and as a result of that, if a staff member is struggling with something in the larger officers, where there's a lot of staff there, the staff member can turn to another TF and ask them for their input or multiple lives could be on the same piece of code to try and help things together.

  • It was, I think, a greater challenge for the staff to lead these tutorials.

  • I think they're harder to leave than office hours because there's a lot more independence involved in them that you need to be able to solve problems on your own because there isn't always going to be a whole bunch of other staff that you can turn Thio at that particular time.

  • Well, we did try to mitigate that by co locating some of the tutorial so that in the same room or same building, there were two groups of 5 to 6 students and one or two t s.

  • Do you have a sense of whether that, indeed helped by just having to smart people in the same room instead of just one.

  • I think that does help, and I think as much as possible trying to pair people up.

  • Um is useful for being able to bounce ideas off someone else and to do a little bit of that efficient allocation of resources where if there's a lull in, one area, staff member can go to another area in order to help students there, too.

  • But it's not quite the same as having a lot of staff in the same place and being able to brainstorm things together because they're definitely issues that air.

  • Tricky enough that I remember like standing around a computer with two or three other staff members on a particularly tricky bug or something that none of us could quite figure out for a little bit of time until we really thought about it together on DSO.

  • I think they're definitely trade offs, but I think on the whole these tutorials have been much healthier than the office hours and just a better experience for students.

  • Yeah, no, this one's a keeper.

  • In my mind, this is one of the best changes for sure, and yet I think it's only fair that we admit we were a little nervous at best, terrified at worst that these were gonna blow up on us in that we knew that even though we have a teaching team of some 80 teaching fellows and course assistance for our 800 some odd students, there was not going to be enough slots during a week for everyone to sign up for a tutorial if they want.

  • We were kind of banking on the fact that numerically, based on the number of people we wanted to have maximum later tutorial and based on the number of tutorials maximally that we wanted to expect of our teaching staff for each week.

  • So the thing balance, their other responsibilities and their other classes We couldn't handle 100% uptake of tutorials.

  • We could only handle roughly 50%.

  • But we, um we eked by and I think you noted that we increased the capacity from 5 to 6.

  • Not a huge problem, but that definitely speaks to I do think the uptake in the the success of them.

  • Yeah, I'm looking at the numbers here and just we recorded the attendance at all of these tutorials so we could look back at the end of the semester and see, like how many people were actually showing up to tutorials as compared to office hours.

  • And in week one, it looks like about 200 students signed up for office for tutorials.

  • Rather, in the second week it was like 380 or so in the third week of his 506 150.

  • So they became more and more popular week after week after week, and so are around the midpoint of the semester.

  • I was definitely starting to get a little nervous about how many sign ups we were having, and we only have so many staff that can only work so many hours.

  • And there's only so much help Weaken provided these tutorials.

  • But ultimately, I think it ended up working out well, well.

  • And thanks to you, we had a special tool for this, too, because we needed some mechanism by which students could sign up for these tutorial.

  • So Brian kindly whipped up a Web based tool that students log into, and then they click a button next to the day in time and location that they want to sign up for, and it also had the benefit of capping attendance of that Onley.

  • Five or six people could sign up for a given tutorial.

  • But there's interesting sociological things that we discussed overtime, too, because it was not uncommon for students just to be no shows or to email us at the last minute and say they couldn't make it.

  • And this was actually a hard problem, because here we had a scarce resource that we were trying desperately to inshore was being officially allocated to as many students as possible.

  • And the fact that a student might sign up for one of, say, five or one of six slots and then not show or cancel at the last minute such that there's no time for someone else to take.

  • It was really frustrating, and it was hard to sort of deal with, and students were often apologetic.

  • But I'm not sure some of them appreciated that this was a resource you're effectively taking away from someone else because we were at capacity, certainly mid semester.

  • Yeah, I don't know if it was.

  • It's totally apparent of students just how it's scarce.

  • The resource necessarily Waas that if you're signing up for something, where there is no cat.

  • Maybe you feel like, Oh, it doesn't matter if I don't show up That's fair.

  • But yeah, with the tutorials that it is different.

  • And it was always unfortunate when Student didn't show up in another student probably wanted that time, and they were only available at that particular hour and they weren't able to attend to the result well, and here I dish really was an opportunity for us.

  • Ah, teachable moment, and what we actually tried doing and software helped with this was we prevented the student who was a no show and who did not give us enough time to reschedule someone else.

  • We prevented them from signing up from a future tutorial until they took a moment to email us the courses heads, uh, just to sort of ask that that hold be lifted, the idea being to sort of send a social message and also, hopefully an educational one that like hey, okay, something must have come up.

  • Life happens.

  • That's fine, but you need to handle this differently, and I think it would be interesting to look retrospectively at the data to see just how frequently was the same person a no show and to see if we actually had some positive impact here on expectations.

  • Yeah, I haven't looked at that data, but I have a feeling like I can't remember too many multiple no shows where we had thio remove the ability to sign in for tutorials multiple times for the same person.

  • I can take another look at the data, but that'd be an interesting question that I'd like to get the answer to.

  • Well, shall we move on to one of our biggest failures?

  • All right, go for it.

  • So I was such a fan in June of 2019 and for the months after that of what we called code reviews.

  • Ultimately, the idea being not only in academia but the real world, sitting down with someone or digitally corresponding with someone about the quality of your code and getting typed or verbal feedback about what you could have done better or differently or otherwise with your code.

  • We've in CS 50 have, of course, experimented with different forms of feedback over the years.

  • Correctness, which is now automated style, which is now automated and design which historically was done by humans.

  • But this year we deliberately eliminated the quote unquote design feedback from student scores, in large part because one analysis of past data that we've collected saying Fall 2018 suggested that the numbers we were assigning two students design scores really weren't influencing their final grades as a result.

  • But the cost operationally too, providing that design feedback was huge.

  • Of the 12 plus hours a week that one of our teaching fellows might work on CS 50 and work with their students, they were spending upwards of six or Maur hours alone on grading.

  • This is an isolated activity for many of them.

  • It's heads down in from the laptop, spending hours on end, providing feedback to students.

  • And then we knew, moreover, quantitatively for Mom, that students were barely engaging with this con with this feedback and sometimes weren't even looking at it, which really called into question.

  • So while undoubtedly I do think there is value to providing very precise feedback when it comes to the design of some students code, what could they do better and why?

  • The reality is, I think we made the right call reallocating the teaching fellows.

  • Time to Maur tutorials or more office hours or more in human time.

  • A human time in person, not in human time is was probably a net positive, given a finite number of hours in the day.

  • Yeah, I think so.

  • And I remember the frustration of like having to do hours and hours of grading and then students not reading any of that feedback that I spent a whole lot of time working on.

  • It just felt like an inefficient use of my time.

  • If I'm spending all this time and students aren't actually reading any of that feedback, and so this year, fall 2019 we introduced code reviews.

  • The idea was that alongside the courses tutorials for which students could sign up, you could also, via the Web site, sign up for one or more code reviews for feedback from a live human via videoconferencing to make things easy and in announce to get feedback on your most recent submission of the problems that the idea being to still provide on an opt in basis as much designed feedback of students would like on their code.

  • But Maur Taylor to the students who actually did want it.

  • Unfortunately, roughly, how many students would you say wanted it.

  • I think we got maybe two or three sign ups in any given week out of a class of 800 plus students.

  • Yeah, this was a big fail.

  • I think it was a wonderful idea in spirit, but I can only surmise that conflicting demands on students time the fact that this is optional, the fact that this is opt in maybe the fact that it was by design on video was just a bit strange, even though that meant they could pop in, spend a few times a few minutes talking with the staff member and pop out was theoretically a compelling thing.

  • But in practice, it just flopped.

  • Yeah, I think part of it is that it was totally optional and much like the super sections were optional.

  • When it's optional, there's just much less engagement and the other piece of it, I think if I just had to guess, I don't have any data to back this up.

  • But I think it's just a matter of the context, switching that when a week ends, a lot of students you goto another lecture.

  • You want to focus on the next week and the code reviews are the only part of the week that really aren't focused on that week.

  • They're focused on the previous week.

  • It's looking back at the work you did last week, which is a valuable thing to do, to learn how to do things better.

  • But in an academic semester that just moves so quickly week after week after week, not only in CS 50 but in other classes I think it's a challenge to get students toe opt into engaging with stuff in the past.

  • Yeah, I agree.

  • And here, too, if anyone's list thing, if we could have twice as many weeks, we could then have an on weekend an off week because truly, I think pedagogic Lee, that would be the ideal.

  • After submitting some work, you have an evaluative process, a feedback loop, even an opportunity, maybe for students to work on further and resubmit.

  • The previous week's problem, said So is to just get better at it and then actually take into account actively that feedback, as opposed to just passively looking at or hearing it so we'll try again.

  • But rest in peace code reviews.

  • They won't return next year in this form, shall we turn finally to perhaps the other biggest change this year, which was the courses notion of tracks in which you and a few others were instrumental.

  • Yeah, so this was a really big change to the real structure of the curriculum of the course.

  • Um, the course for a long time has had the last couple of weeks be focused on Web programming where students learn HTML to build Web pages and they used Python in order to build Web applications.

  • And we had a little bit of JavaScript and CSS on top of that as well.

  • And one of the effects of that we found with that the lot of students final projects When it came time for the Sea of 50 Fair, we're all very, very similar.

  • There were a lot of Web applications that all looked fairly similar, and a couple of students would take it upon themselves to try to teach themselves how to do something different howto do something with hardware or howto build on IOS out, for example.

  • But it definitely wasn't a lot, and we thought that maybe part of that was just the fact that because we teach how to build Web applications that we sort of inadvertently push students towards building Web applications for their final project, even though there are a lot of other options that students could pursue as interesting things that they might find fun to build on their own or in a group.

  • So the tracks were a change we made to the curriculum this year where we actually gave students a choice of what they want to do with the last two weeks of their semester.

  • Instead of just pushing everyone into Web programming, we said you could pick what you would like to study for the last two weeks.

  • You could pick Web programming.

  • You can pick mobile app development with either IOS or for Android.

  • You think Java or you could pick game development if that was something of interest to you and we had separate problem sets for each of those individual tracks, we had lecture videos, so I led the Web programming videos.

  • Tommy McWilliam, a former had teaching fellow for the course, led the mobile application tracks for IOS and Android on Colton Ogden, who runs through the Games course at Harvard Extension School.

  • In on an ex he led the game development track.

  • So we all prepared lecture videos.

  • We all prepared projects and let students pick what they wanted to do with their last two weeks.

  • But how would you assess the end result?

  • The end result.

  • So there were it was a good idea in theory.

  • I mean, one thing we definitely found was that a lot of students just skewed towards Web programming.

  • Web programming was like, I think, 80% plus that you didn t 7 37%.

  • Yeah, that sounds about right ended up choosing what programming there was a smaller percentage for IOS and games.

  • And then I think there was like 1% of students ended up pursuing Android in part.

  • Maybe because I think just based on statistics at Harvard, almost everyone 9% of students have iPhones.

  • 99% of students have iPhones.

  • Android is, for whatever reason, not super popular around here.

  • No, but to be fair in the Ed X version of the class in the open courseware version of the class for which these tracks are now available as of January 1st, I do think we'll see a lot more uptake of the android track to be fair.

  • Yeah, I I definitely agree with that.

  • And I think it's a good thing that they exist for students who want to pursue them.

  • But it was interesting that thing's skewed so heavily towards Web programming this past year on.

  • Maybe that was just students from prior years that were encouraging this year students to do Web programming cause that's what they had seen before.

  • But I'm not entirely sure all the factors that might have attributed Thio that skew.

  • Yeah, and I'm not sure.

  • I'll admit that we should keep these next year.

  • Um, I mean 87% toward

Hello World.

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it