With the current craze surrounding MOOCs, some of the players in the open courseware arena seem to get very little attention. One of the underdogs is the Saylor Foundation, which pursues a unique strategy by curating content from around the Internet, adding original content, bundling it all up into individual modules, and offering examinations. All this happens in collaboration with academics from a wide range of institutions. Currently, the Saylor Foundation offers virtual “degrees” in about a dozen academic disciplines.

I’m not so much interested in Communications or Art History, but I was curious about their offerings in computer science. The Saylor Foundation replicates a basic computer science degree on their website, drawing from a wide range of sources, including MIT OCW and Stanford, but also Khan Academy and, if needed, the occasional blog post or two. I decided to dip my toes into the water and went for their CS101: Introduction to Computer Science I. I have some background in this field, so it was a breeze to go through the material.

At first I was a skeptical about the general approach of the Saylor Foundation since they draw from many sources, seemingly of uneven quality, but I found the material surprisingly engaging. It’s certainly more motivating than going through some of the introductory college-level textbooks I have seen. The main source for CS101 was an interactive tutorial on computer science designed by Bradley Kjell of Central Connecticut State University. Given that many MOOCs are offered by elite universities, some readers may now want to dismiss Saylor, but this would be misguided. CS101 is an introductory class, and you don’t need an academic superstar to teach it.

Kjell’s tutorial is very well-laid out and gradually introduces concepts. Some textbooks have the audacity to tell you to just accept a concept or language feature as fact if you don’t understand it yet, and refer you to a future chapter in which everything would be cleared up. Didactically, I find this to be pretty questionable approach. On the other hand, Kjell does a very thorough job. The only legitimate complaint would be that he might move a bit too slow at times. Thankfully, he isn’t very verbose, so it is easy to skim parts. On the other hand, if you think the pacing is just right, then you can also listen to audio recordings of the material. I preferred reading, but he does have a voice that is very pleasant to listen to.

The presentation may have been a bit archaic, but the content is good. There were even some unexpected highlights like a digression on machine language where students were asked to construct a while loop using machine instructions. The toothbrush has to be instructed to move bristles until it is switched off:

Yes, this is artificial, but it’s a pretty neat exercise nonetheless.

Bradley Kjell’s material focussed on Java. However, Saylor’s CS101 introduces Python early-on to illustrate how concepts are implemented in a different language. I thought this was a very good idea as it broadens your horizon. Normally, introductory courses only focus on one language.

Overall, the content started from first principles, and programming language constructs were introduced very gently. Therefore, CS101 would be a good course for someone with little previous knowledge as some of the introductory CS MOOCs can move rather quickly. You won’t just be exposed to the basic building blocks of computer programs like data types, operators, control structures, or input/output mechanism. There is also the typical brief coverage of fundamental hardware and software concepts. Perhaps surprisingly, software engineering methodologies are very briefly discussed as well, with a brief overview of the waterfall model and its stages.

The only downside might be that there are no interactive programming exercises, and that the (static) programming exercises themselves are not very numerous. However, DrJava is your friend and will serve you well. You probably want to supplement this course with a few Java practice exercises on CodingBat, though.

The final exam is randomly generated. You have to answer 50 multiple-choice questions. If you get at least 70 % right you’ll earn a certificate. If you don’t pass, you can try again after two weeks. I thought that in the final exam the do/while loop and switch statements were too prominent. Sure, they might come in handy at times, but many programmers view them as superfluous language constructs and often ignore them altogether. This is just a minor quibble, though.

Overall, Saylor’s CS101 provides a gentle introduction to computer science. Even though the course doesn’t offer interactive exercises, the format worked surprisingly well. The final exam is a useful feature, too, and it helps distinguish Saylor from other OCW sites.

Jennifer Widom’s Introduction to Databases has an impressive pedigree, having been one of the three inaugural Stanford MOOCs that caused the avalanche of online courses we can all enjoy today. Furthermore, this course has had many homes, tracing part of the MOOC revolution. Originally, it was a stand-alone class offered by Stanford in autumn 2011. The content was later on added to Coursera and offered in self-service mode, i.e. without exam and involvement of TAs in the forums, before it was offered again this January on Stanford’s Class2Go platform. Less than a week ago, though, it was announced that Class2Go would be folded into MIT and Harvard’s EdX platform. On a side note, the other two inaugural Stanford MOOCs were Andrew Ng’s Machine Learning, and Introduction to Artificial Intelligence by Peter Norvig and Sebastian Thrun. Andrew Ng became co-founder of Coursera, and Sebastian Thrun launched Udacity. Looking back, it’s quite amazing how much has happened in the course of little more than a year.

Unlike machine learning or artificial intelligence, knowledge in databases is likely to be relevant for a large part of software engineers since databases are ubiquitous. I think that a course on databases is one of the most “useful” courses one could take as part of a typical computer science degree. Such knowledge is presumably deemed too practical in some departments, which might explain why an intro to databases is often an elective instead of a mandatory course.

I took an introductory database course last semester at university, and frequently referred to some of the videos of Jennifer Widom’s class, which was offered in self-service mode on Coursera back then. I found her explanations very lucid. My lecturers were overly fond of theoretical explanations, without demonstrating how the concept are used in practice. In contrast, Jennifer Widom follows a didactically much more sensible approach by showing examples first and explaining various pros and cons before eventually covering theoretical definitions. Her explanations on referential integrity and the various so-called normal forms were all very clear. She even helped me understand multi-valued dependencies, which I initially found difficult to grasp. However, that kind of knowledge will presumably be less relevant in practice.

I originally didn’t intend to go through the entire course, since I have already taken a similar one at a brick-and-mortar institution, but shortly before the course started, I learnt that Introduction to Databases was moved to Stanford’s new platform Class2Go. I was simply curious how their platform worked and had a look around. What motivated me to work through the material was simply the great amount of flexibility it offered. Even though the content was spread over nine or ten weeks, everything was available from the very beginning. There were deadlines, but you had the option of working well-ahead of schedule. The only other fixed dates were two two-day periods for the mid-term and final exam.

I wasn’t going to invest a lot of time in this course, but wanted to explore areas I hadn’t been exposed to yet, or only cursory, such as JSON, and more advanced SQL topics like views or triggers. I was thus pleased to notice that the course had two different tracks. If you wanted to earn a statement of accomplishment, you had to score 75% from a weighted score of exams, interactive exercises and quizzes, in addition to scoring 75% of so-called challenge-level problems. I was content with the regular statement of completion that was awarded for a score of at least 50%. Thus, I skipped most of the challenge-level exercises. The course itself offers a smörgåsbord of database-related topics. The curriculum even encouraged designing your own learning path, and the conditions for the statement of completion were such that you could easily skip a few topics and still “pass” the course.

I very much enjoyed learning about JSON, which made me even more skeptical about XML. Another highlight was recursion in SQL, but this topic might see little application in the real world. I’ll just list all topics of the curriculum so that you’ll get an idea of the breadth and depth it offered:

– Relational Databases (relational model, querying)
– XML Data
– JSON Data
– Relational Algebra
– SQL (main commands)
– Relational Design Theory
– Querying XML
– UML
– SQL: Indexes
– SQL: Transactions
– SQL: Constraints and Triggers
– SQL: Views
– SQL: Authorization
– SQL: Recursion
– SQL: On-Line Analytical Processing
– NoSQL

Of course, SQL is the 800 pound gorilla you just won’t get around. However, the supplemental topics were well-selected. This course focussed more on practice than theory, which working professionals might appreciate. Still, you’ll be taught substantial amounts of database theory, meaning that this course is not lacking in that regard either. If you want to thoroughly go through each module, you should plan to invest a good five to eight hours a week.

Overall, Introduction to Databases truck me as well-designed. The quality of the teaching was top-notch, too, with Jennifer Widom taking great care to illustrate every concept, and spending copious amounts of time demonstrating them in practice. The course was surprisingly hands-on, and a significant part of the videos were screencasts showing the instructor querying a database, explaining peculiarities of SQL syntax along the way, or pointing out common mistakes.

The course is currently offered in self-service mode. If you want to learn about databases, it is an excellent starting point. Please note that even though this course does refer to three database textbooks, you won’t have to buy any of them since the teaching materials are of a very good quality. I own a copy of Jeffrey Ullman and Jennifer Widom’s A First Course in Database Systems, which complements this online course. However, it was merely in the “nice to have” category, meaning that I think it’s quite possible to make it through the course without a textbook.

Most MOOCs follow a strict schedule with weekly deadlines. I was thus quite surprised when I learnt that Think Again: How to Reason and Argue, which started on 26 November 2012, had just one deadline for all assignments, which happened to be 11 March 2013. Since I had some spare time last week, I decided to go through the entire class in the evenings.

At first, I expected a class that was more similar to Michael Genesereth’s Introduction to Logic, but this wasn’t the case at all. Instead, Walter Sinnott-Armstrong (Duke University) and Ram Neta (University of North Carolina at Chapel Hill) gave a 12 week course on “informal logic”. This isn’t a term I just made up, though, but one I have taken from the subtitle of the recommended textbook.

Compared to a typical course in logic as you might find it on a computer science or philosophy curriculum, Think Again has a much different focus as it attempts to teach how one could evaluate arguments in everyday contexts.

The syllabus structures the course into four distinct parts:

• Analyzing argument
• Deductive arguments
• Inductive arguments
• Fallacies and refutations

Later on, this review will sound rather critical, therefore let me straight away state that Walter Sinnott-Armstrong and Ram Neta largely succeeded. If you make it through the course, you will be able to more critically analyze the kind of arguments you encounter on your favorite blogs or in mainstream media, and if you watch Fox News, you’ll probably stop doing so by week two or three.

The first part consisted little more than textual analysis. You’ll learn how arguments can be strengthened or weakened, or why it’s not that great of an idea to make unfalsifiable statements. Overall, that part moved very slowly, but the pace picked up in the second part. Ram Neta offered a gentle introduction to propositional and categorical logic. I would have expected some first-order logic, and at least an intro to proofs in propositional logic.

I found Ram Neta’s teaching to be of uneven quality, which was also reflected in many negative comments on the forums. I did enjoy his sometimes whimsical presentation, like his illustration of the logical connective “and” by putting a book between his foot and his hand:

That’s nothing any of my former philosophy professors would have done.

However, he sadly had a tendency to often ramble on and repeat himself over and over. This was even more bothersome because some concepts which you would have expected to be taught in the lectures were introduced in the quizzes. Out of the blue you were hit with the definition of, say, a conjunction introduction argument, and asked to identify it among the answer choices. I can’t imagine that this is how those two professor teach their course at Duke. I’d assume that to some degree the lectures were ad libbed, and when they later on realized that they forgot to cover some concepts, they simply added them to the quizzes.

The third part was refreshing as it incorporates probability theory and set theory, but nothing that should be unknown if you’ve paid attention in high school. Knowledge about various statistical biases would do a lot of people good. I do believe that those who misinform with statistics do so deliberately, but some very basic knowledge of statistics could at least make some people more skeptical of the kind of flawed reasoning that is all too common in mass media.

In the final part, various logical fallacies were discussed, and in the final week, arguments that were submitted by students were picked apart. Overall, part four was rather weak, and many of the minor issues you were willing to overlook in the previous became quite annoying. I will now proceed with describing what I perceive to be the main problems with “informal logic” ask taught in Think Again, but before I do so, let me reiterate that I thought the course to be pretty good. What follows is therefore a discussion of some of the negative aspects that are in no way representative of the entire course.

One big problem I saw with “informal logic” is that natural languages are inherently vague, and therefore any attempt to precisely reason about language is doomed to failure. I will give one particularly striking example. In one of the lectures, Ram Neta discusses what philosophers call the “sorites paradox”. The two premises are:

1) One grain of sand does not constitute a heap.
2) Adding one additional grain of sand doesn’t turn a non-heap into a heap.

This might seem plausible enough, but the conclusion is that no matter how many grains of sand you add, you’ll never end up with a heap, which is obviously nonsensical. The problem is simply that “heap” is a vague term. I consider it to be perfectly valid when someone states that he knows what a heap is when he sees one, just as you know when someone is, say, tall, rich, or bald.

We could of course spend many hours debating the problem of vagueness in natural language and feel oh-so-smart when we point out that if losing one hair doesn’t make the difference between being bald or not-bald, then you can never get bald. However, it’s probably more productive to simply formalize the language and move on. Vagueness may be an interesting philosophical problem, but it’s of limited relevance in everyday contexts. Just imagine you were to write a program that determined which number of grain constitute a heap. You’d simply pick a reasonable value and be done with it.

The same is true of another linguistic ambiguity. Consider the following sentence:

Carol went to the bank.

If you now sit back in awe, then you’d fit right into a freshman philosophy class. Of course, “bank” could refer to a financial institution, or it could refer to the bank of a river. However, it’s incredibly difficult to mess this up in normal conversations, and the examples people construct in philosophy classes are more than just a bit tiring. Yes, this is a case of ambiguity, and to get around this, you could either use an artificial language like Lojban, or you point out to your students that there is no equivalence between both words. In a dictionary, there are two entries for “bank” for this very reason. In a philosophy class, you could make it clear to the densest student by using subscripts or variables. Let the bank of a river be P, and the financial institution Q. So, Carol went either to P or Q, and with that there is suddenly no need to waste any time discussing particularities of the English language. All of a sudden, many “philosophical” problems are cleared up.

Logical fallacies are certainly an important topic, yet in this course, the professors went overboard. Instead of discussing, say, the structure of an ad hominem argument, Ram Neta insisted on classifying them as “deniers”, “dismissers”, and “silencers”, and each could be either justified or unjustified. Yet, knowing an arbitrary distinction is not really what matters, given that recognizing an ad hominem in the first place is much more valuable.

Of course, knowing those distinctions was necessary for passing the quizzes. Here is one argument, taken from week 10:

During Senate testimony concerning the BP oil spill in the Gulf of Mexico, BP executives were being questioned about the safety procedures that they implemented on all their oil rigs. An environmental activist who was sitting in the audience stood up during one executive’s testimony and interrupted the testimony, shouting that the executive was lying, and holding up physical evidence that belied the claims that the executive was making. Given the circumstances, we must not consider the activist’s testimony or evidence. Therefore, unless we have any evidence independent of what the activist provided, we must simply assume that the executive was telling the truth.

You then had to choose between the six possible permutations of ad hominem arguments. The supposedly correct answer was that it was a “justified silencer”. Look at this explanation:

[T]he argument is a justified silencer because, in order to have a right to testify in court, one’s testimony has to be presented in accordance with the rules of the courtroom. Since it goes against these rules to merely stand up and start shouting during the testimony of another witness, it follows that, if the activist in (4) presents her testimony by shouting it during a Senator’s testimony, she does not, in fact, have the right to be heard then.

I found this to be a particularly bad example since it relied not on the validity of the argument itself but on circumstances. If the activist did indeed present compelling evidence, then merely being in a court of law does not negate it. You could very well believe that the executive was lying, based on the evidence the activist provided, even though the court (!) would not be in a position to acknowledge the evidence. To spell it out: Yes, the evidence can be inadmissible in court but at the same time prove beyond doubt that the executive has been lying.

Here is another example argument from the course:

A large majority of Americans currently believe that the most effective way for the US Government to grow the economy in the next 3 years is by reducing its spending. In a democracy, the voters decide. Therefore, the most effective way for the US Government to grow the economy in the next 3 years is by reducing its spending.

Sadly, there are too many problems with this argument. First, there is no way to reliably estimate what a “large majority of Americans” currently believe. At best, it’s an extrapolation based on a sample. Second, in a democracy, the voters don’t decide at all. You’ve got the chance to cast a vote every few years, and your choice may, like in the US, be limited to two candidates from a very similar political spectrum. Modern democracy has little to do with the Greek root of the word, and even ancient Greeks democracy wasn’t a democracy, considering that their society was built on a minority exploiting slaves. You’d really wish you’d only see some Ps and Qs on the screen instead of this garbled mess, and if you wanted to discuss this argument, it would best be done in essay form.

This wasn’t all, though. Here is the very next argument from that exercise set:

According to one recent lexicographical study, a large majority of native English speakers in the United States think that ‘chillaxing’ is a synonym of ‘relaxing’. Therefore, it is true that ‘chillaxing’ is a synonym of ‘relaxing’.

(a) justified appeal to popular opinion
(b) unjustified appeal to popular opinion
(c) justified appeal to authority
(d) unjustified appeal to authority
(e) none of the above

Again, you can’t accurately determine what a larger majority of English speakers thinks, but even if we ignore this problem, there is an even larger issue that may be familiar to you if you’ve studied linguistics. The issue is whether linguistics should be descriptive or prescriptive. One one extreme you’ve got people who believe speakers of a language do not have the authority to determine what constitutes acceptable use. Instead, they think they have to set rules for them. A prominent example is the French Academy, which is the authority on the French language. Thus, if 70% of French speakers thought that “chillaxer” was a synonym for “relaxer”, but the French Academy didn’t, then it simply wouldn’t be a synonym. To use an argument by analogy, the French Academy acts as a law maker, and decides what is right or wrong, and they can of course be completely off the mark, just like lawmakers sometimes are as well. Still, the fact that they are an authority means that they can ignore what the majority might think.

The consequence is therefore that you could justify at least three of the answer choices. The third is “none of the above”, which might imply that more than one choice is correct.

An integral part of the MOOC experience are the discussion forums. However, I have rarely seen this part executed well. I saw a few examples in more advanced courses where posts were generally of a high quality, and where moderators had no qualms just deleting inappropriate comments, or quickly pointing out errors in potentially misleading posts. Think Again followed a laissez-faire policy, and the consequence was that you could find staggering amounts of bullshit, which would have made Dunning and Kruger proud.

To provide you with the necessary context, here is an exercise from week 8:

What is the expected financial value of a bet where you will win $2652 if you draw a seven of spades and then a seven of clubs on two consecutive draws (without returning the first card to the deck)?

The correct answer is $0, which is the result of this calculation:
(1/52 * 1/51) * (2652-1) – (1 – (1/52 * 1/51)) * 1).

Let me just walk you through it step by step:

• There are 52 cards in a deck, and you don’t put the card you first draw back. Thus, the probability of drawing a seven of spades, and then a seven of clubs is 1/52 * 1/51 = 1/2652.
• If you draw this combination of cards, you win $2652 – $1, since the dollar you initially put in doesn’t get refunded. Considering the probability of this event, the gross gain from winning is therefore 2651/2652.
• The probability of losing is (1 – 1/2652) = 2651/2652. Multiply this value by the one dollar you put in.
• Add it all up and you end up with (1/2652) * $2651 – (2651/2652) * $1 = $0

But let’s look at a discussion in the forum. One guy was stumped, and asked for help. Plenty were willing to chime in, including one particular person who wrote:

[Y]ou are not accounting the fact that if the person doesn’t get the right card in the first draw, he wouldn’t pick the second card at all. So to calculate the probability of losing, you actually need to consider two cases:
1. the first card picked is not correct. the probability of this happening is 51/52. Since he doesn’t pick the second card here, we do not multiply this by the probability of picking the wrong card in next draw.
2. the first card picked is correct but the second card picked is incorrect. here the probability of losing is 1/52 * 50/51=50/2652.
now, both these cases are mutually exclusive. so you can add the two probabilities to get the probability of losing overall as 2651/2652, which gives you the correct answer.

This wrong. The question clearly states that two picks are made. There is no room for fantasizing that the bettor would walk away if he picked the wrong card in the first draw. This was bad enough, but then the person who originally asked the question replied:

[EDIT: I made a mistake here. That guy presented an alternative solution that is equally correct. I made the wrong evaluation after reading the first point. Please see the comment by Ed McCardell below if my error isn’t obvious to you.]

Thank you.

I have given up and have left this exercise (only because of Q7 and Q8) in a backburner until a deadline, but your post helped me to think over it again. In my post above I complicated the solution and confused myself (hopefully, not other forum readers!). Thanks again.

It’s a case of the blind leading the blind, and you can witness the same over and over. There is a good reason why universities don’t just put students in a room and tell them to figure everything out by themselves.

Lastly, there were some platform-related issues. Coursera is growing rapidly, and some technical difficulties are therefore to be expected. I did notice prolonged loading times when navigating course content. This refers both to waiting for a video to load as well as simply navigating the site. In addition, there was a particular problem with the videos, and it’s a bug I find quite easy to reproduce: whenever you pause a video, and too much time passes before you continue, then the remainder of it won’t get streamed. After a few moments the video freezes, and you’ll have to reload the page to continue.

I don’t want to end this review with just a lengthy discussion of the flaws I found, though. Overall, I did enjoy Think Again: How to Reason and Argue. It provided me with many enjoyable hours often filled with interesting problems. If you are looking for a gentle introduction into logic and enjoy abstract reasoning, then this course might be for you, provided you can look past some of the issues and won’t require much feedback from the forum. Expecting a flawless experience from a MOOC is probably too much to ask. I have taken dozens of courses in brick-and-mortar universities, and quite a few of those had issues far greater than any of the MOOCs I have taken since.

There are many obvious problems with the “peer review” model of essay grading Coursera is piloting. On the other hand, one may think that automatic grading of code submissions in MOOCs is much more reasonable. It’s certainly more practical, and it might explain why the majority of current online courses are related to computer science. Obviously, some piece of code either works, or it doesn’t. Yet, a few important issues seem to be neglected. Working code should of course be a necessity, but it’s not all there is to writing “good” code. While one may want to argue that the points I am going to raise are more relevant for real-life software development and less of an issue in university courses, bad practices can still lead to bad habits that only take more time to correct later on.

A big issue is that auto-graders do not recognize poorly structured code. If the code is working but sloppily written, without any comments and poor visual guidance, the software won’t notice it. To some degree this could be rectified, but the current versions of the auto-graders on Coursera, EdX and Udacity are oblivious to it. Questions of coding style are, of course, to some degree subjective. Yet, following well-thought out style guides can lead to greatly increased readability. Even just blank lines to separate functions and variables can go a long way, and certainly few people doubt that it’s a good idea to not write more than a certain amount of characters per line. (It may not be 80, though.) A good tutor, either in industry or at university, will point this out to you. The autograder won’t because it can’t.

Hand in hand with poor structure goes over-engineering. I’m currently going through MITx’s 6.00x Introduction to Computer Science and Programming. Among the introductory CS courses I’ve either taken or had a look at, it seems to be the best one by a wide margin. One reason is that the exercises extend beyond the merely mechanical application of knowledge. For instance, in an early exercise (lecture 4: problem 5), you are asked to find the maximum of three numbers without using conditional statements. Instead, you have to use the in-built min() and max() functions of Python.

Here is the code skeleton:

def clip(lo, x, hi):

'''
 Takes in three numbers and returns a value based on the value of x.
 Returns:
 - lo, when x < lo
 - hi, when x > hi
 - x, otherwise
 '''
 # Your code here

This is not an overly difficult exercise, but you may have to think about it for a minute or two.

One possible solution is:

return max(min(hi,x),lo)

One of the students, though, presented a fabulous case for overthinking the problem. He writes:

The trick here is to know that the boolean True and False have values you can use:

True == 1
False == 0

So you can test for the three conditions (lower, in-range, higher) and store each in a separate boolean variable. Then you can use those booleans (think now of 1 or 0) in a return statement that multiplies the low-test boolean by lo, the in-range boolean by x, and the high-test boolean by hi.

The return value is simply the booleans multiplied by each argument, and added together. That will result in math something like these

1 * lo + 0 * x + 0 * hi # too low
0 * lo + 1 * x + 0 * hi # in range
0 * lo + 0 * x + 1 * hi # too high

This is only possible on a weakly moderated platform, where any post with the veneer of plausibility, and written in an authoritative tone, tends to impress the inexperienced. Indeed, other students were busy thanking that guy for his “help”, until finally someone chimed in and clarified that this solution was entirely missing the point.

Poor structure and complicated code are not the only issues you may face. If you’ve ever worked with legacy code, then you’ve probably come across poorly chosen or variable names. This may not be an big issue in a 50-line script someone writes for his CS 101 class. However, in any context where someone else has to work with your code, this can quickly lead to problems because the other person may need much more time to familiarize himself with it. Besides, that other person may well be the original author in a few months, once he is no longer familiar with the problem he wanted to solve.

What you also sometimes see is people writing code in languages other than English. Yet, English is the de facto Lingua Franca of programming. It’s not so uncommon that someone posts a code snippet somewhere, and asks for help. If a variable is called, say, “naam” instead of name, I don’t necessarily need to know that this is Dutch to correctly guess its meaning. It remains a minor annoyance, though. Besides, there are enough languages out there, and a plethora of possible variable names that bear little to no resemblance to their English counterparts, that guessing won’t help much in the long run.

This is no trifling matter. Just think of Open Office, which is based on an office suite called Star Office. Star Office was originally developed by a team of German developers who, you guessed it, did not comment their code in English. The company behind Star Office was acquired by Sun in 1999, and the software was finally open sourced in 2000. In 2012, though, there are still German comments left in the source code, waiting to be translated into English. The German programmers could have written their comments in English, which may have taken a bit longer. However, by neglecting this proverbial “stitch in time” the problem got compounded. I don’t even want to speculate how much time was wasted on fixing this issue subsequently. Eric S. Raymond writes about this as well in “How To Become A Hacker“, where he points out that English is the “working language of the hacker culture and the Internet.” This situation hasn’t changed.

On a side note, the EdX team recently had some technical issues with their auto-grader. It couldn’t evaluate code that contained Unicode characters. However, this wasn’t a bug but a feature! If anything, you want to discourage students from putting special characters from all the languages in the world in their code. It may even make sense to artificially restrict the auto-grader to the ASCII standard. This doesn’t apply to the real world, but for CS 101 it’s probably appropriate.

In general, I think that having a few units on how to write clean code would be very beneficial as it would help students to acquire good habits. This doesn’t just apply to beginners. There are also some seemingly experienced programmers on those platforms, presumably using online courses to learn a new programming language or brushing up on their computer science knowledge. At least that’s my guess when I see someone posting nicely written code in Python, in which the lines end with semicolons. This only illustrates my point: Old habits die hard, so you better acquire good ones as early as you can.

Coursera certainly deserves praise for opening access to higher education, especially since they also offer courses in the humanities, unlike competitors like Udacity or EdX. Solutions to exercises in mathematics or computer science can easily be graded because there is an expected correct answer at undergraduate level courses, but assessing the relative merits of an essay in, say, literary studies isn’t quite so straightforward. Coursera attempts to solve this problem by letting students grade the essays of their peers.

I do see some issues with automated grading even in code submissions, but that’s a topic for another article. Right now I am more concerned with the peer review system Coursera has implemented. I am sure they will attempt to modify their system eventually, but at the moment there are some serious issues. Please note that I am not speaking as an observer on the sidelines. I have sampled numerous courses, and finished finished three so far. Especially in more technical courses, the content seems to be very good, and for a motivated self-learner you could easily substitute a course at a brick-and-mortar university by one of Coursera’s, if you are more concerned about learning something new and care little about getting a paper.

On the other hand, the humanities courses don’t seem to fare that well. You’d really have to lower your expectations. I’ll talk about the shortcomings in a moment, but before that, let’s review the ambitions Coursera has for their peer review process:

Following the literature on student peer reviews, we have developed a process in which students are first trained using a grading rubric to grade other assessments. This has been shown to result in accurate feedback to other students, and also provide a valuable learning experience for the students doing the grading. Second, we draw on ideas from the literature on crowd-sourcing, which studies how one can take many ratings (of varying degrees of reliability) and combine them to obtain a highly accurate score. Using such algorithms, we expect that by having multiple students grade each homework, we will be able to obtain grading accuracy comparable or even superior to that provided by a single teaching assistant.

What actually happens is that every student has to evaluate five essays to receive feedback on his own work, which in turn also gets evaluated by five other students. Your score is merely an average of the scores you have received from other students in the course. Scoring is done according to various rubrics, reflecting the criteria of the assignment. Let’s say you were asked to discuss events that happened in a certain time period. If you did that, you were supposed to get one point, if not, you got zero. Most of the rubrics were quite surprising if your assumption was that people writing those essays actually paid attention to the directions. It turned out that many didn’t. I read a few essays that were completely unrelated. The bar seems to be rather low.

An article on Inside HigherEd recently pointed out some issues with Coursera’s peer review system. I do think that the author had a rather narrow perspective, despite raising awareness of some issues with the nature of the feedback you receive from other students. She pointed out that feedback was highly variable, ranging from a “good job!” to downright abusive language. Further, feedback is one-directional, and there is no way to comment on feedback or challenge a perceived unfair assessment. Given the current state of Coursera with low-quality feedback being the norm, I am tempted to view this as a feature and not a bug. Yet, in a traditional university, a great part of the learning experience consists in discussing your work with your supervisor.

The other issues Inside HigherEd mention go hand in hand: Anonymity of feedback is not necessarily a problem, but the associated lack of community is. In many courses, the forum have a low signal to noise ratio. Of course, this won’t matter if someone finds it deeply gratifying to express “me too!” statements in poor English. I’d imagine that most exchanges via text messages or on Twitter follow exactly this pattern. Personally, I saw little value in threads in which over 100 people reveal which country they are from, though.

But let’s look at methodological problems of the crowd review system Coursera uses. The assumption is that a random number of students is perfectly able to objectively assess the works of others, despite their own shortcomings. This itself is a rather daring hypothesis that hardly passes the laugh test since it presupposes that all students, once they submit their essays, turn into omniscient and objective evaluators who are able to detect the kind of mistakes they made before their role switched from student to peer reviewer. Further down I’ll present an argument that is supposed to show that crowd review can’t even work in a best case scenario, but for now I’ll focus on what I perceive to be the current reality on Coursera.

Providers of MOOCs focus on vanity metrics like the number of “enrollments.” It seems that no matter what kind of course is launched, eventually between 50,000 and 100,000 students sign up. But this number itself is highly inflated. It’s common that not even half the students finish the first week’s assignment. This is not because they are all unmotivated or realize that they had bitten off more than they can chew. Instead, “enrolling” is the only way to sample the course content. If you just wanted to have a look, you’d have to register first. I am fairly certain that if, for instance, the first week of lectures and assignments was available immediately, then the big drop off in numbers could be largely avoided. This would be a straightforward solution, but how impressive would it be if Daphne Koller and Andrew Ng had to tell their investors that some recent changes lowered the number of enrollments by 50%? This is not just an issue of Coursera or Udacity. Twitter and Facebook are also full of inactive accounts, in addition to fake ones. Yet, yet their reported total number of users is rarely questioned in the press.

[EDIT: The world of MOOCs is moving fast. Coursera has very recently begun making all video lectures of about a dozen courses available without registration.]

After week one about half the students are left. This will still be an enormous amount of people. Yet, it doesn’t mean that they are all automatically well-qualified. “Opening up” higher education brings this to light, and only reveals educational deficits. I am far from putting the blame on the students. Many first-world countries rather invest in their military or propping up a global fantasy economy instead of creating equal opportunities by investing in schools or libraries. Therefore, it is little surprise that quite a few of the people who are active on the forums, or participate in the peer review process, would seem quite out of place at a proper university. Given that all it takes to sign up is an email address, this is to be expected.

Unfortunately, an immediate effect is the relatively low quality of the discussion forums, which I had touched upon before. If you skim a few threads, you’ll be amazed at the widespread lack of etiquette, and often poor grammar and orthography. It’s not as bad as a random collection of comments on YouTube, but at worst it’s quite close. As the courses go on, it normally gets better, but the first two or three weeks normally make you want to stay away from online discussions.

The rather diverse student body has ramifications for crowd reviewing as a whole. Put five students with a poor command of English in a room, tell them to grade the each others’s work, and the result won’t be especially awe-inspiring. While this example may sound absurd, it is a rather common occurrence on Coursera. Of course, if you only want to check for basic writing ability, then you can just run a grammar and spell checker. I don’t want to sound condescending, but about half the students whose essays I read did’t even bother with this. But if people can’t even meet this goal, then it’s arguably too much to ask for coherent arguments.

As Coursera expands, I can only see this getting worse, because September will never end. Without strong moderation, the quality of sites degrades. However, since strong moderation isn’t conducive to growth, good sites sooner or later degenerate. It’s probably only a matter of time until people will post “First!” or “+1” on a thread where someone asks a question about algorithmic efficiency.

Finally, let’s consider a best case scenario. One may want to think that in an ideal world you could have stellar students grading the work of other equally stellar students. It’s quite obvious that they would be able to give each other much more meaningful feedback. Yet, they don’t know what they don’t know. A motivated student will therefore learn much more from interacting with a tutor who is able to make him consider a different point of view. In fact, one of the benefits of attending a university course that focuses on writing is that you can discuss your work with someone who is more experienced and more educated. Of course, not all professors are, but at the better universities you should find a lot of smart professors and highly competent tutors. It will take significant breakthroughs in AI to replicate something similar.

In summary, it seems that in a best-case scenario Coursera will stunt the growth of motivated students. This ties in with a recent talk by David Patterson, where he stated that MOOCs are a “godsend for the 99%” since they were better than other options. The “1%” were CS majors at good schools, and their education was undeniably superior to what a MOOC could provide. He was referring to his own Software as a Service course, which was originally offered through Coursera, but is now available on EdX.

David Patterson on the “99%” in Education

Patterson was referring to a computer science course. In the humanities, though, a lot has to change until a similar claim could be made. The worst case scenario I described, which seems rather close to the reality on Coursera, has quite some room for improvement. I am tempted to say that this is an enormous problem, and I don’t see how the goals of “opening up higher education” in the humanities and raising standards can be achieved. In the end there may be the realization that high school education has to be fixed before MOOCs can reach their full potential. We’ll have to see how well Khan Academy will be doing on that front, but it certainly looks promising.