If you are aware of the need for students to eventually write code without a direct feedback system, then writing open-ended response questions is the right format to use. These problems are very difficult to grade, as the instructor needs to consider many different components. A student can make mistakes with syntax, logic, problem solving approaches, and even conceptual misunderstandings. It also puts pressure on students to complete a large task. Failure to respond to an open-ended question may not signal any localized conceptual misunderstandings, rather a fault in their process for writing code from scratch. An advantage to using these questions is in identifying rough tiers of students within a classroom. The ones that can’t write code in this setting may be using other students to survive in their lab time, and might not even be completing their own work. How this fits in the scheme of assigning grades can be a very tricky consideration.
It is very difficult to craft a stem for open-ended questions that is free from ambiguity. Students need to practice the skill of reading a problem description and converting it to a computational algorithm. If you choose to design open-ended questions for an exam, consider giving the students multiple chances to practice this during class sessions. Oftentimes, this will allow you to identify which aspects of your descriptions are hard for a student to interpret.
Consider the question below, taken from an exam that covers the use of variables. Students have not as of yet learned anything regarding conditional execution, loops, functions, or object oriented programming.
A program is designed to read in the coordinate values of three points on a Cartesian plane. The program then calculates the side lengths of each pair of points and the corresponding angle measurements.
A) Write the segment of code that collects one set of coordinate input from the users.
B) Assume that similar code for collecting the other two sets of coordinate pairs (x2, y2) and (x3, y3) exists. Write the segment of code that calculates one of the three side lengths based on the distance formula.
C) Write the segment of code that calculates the angle opposite your side length from part B. You can assume that all three side lengths were calculated and stored inside the variables a, b and c.
This question limits the scope to strictly variables, arithmetic, and input/output. Unfortunately, it also contains some domain knowledge from basic algebra and geometry. In a case like this, it would be important to provide the equations for any calculations so that a student would not have to demonstrate outside knowledge for this question. Even with the formulas, a student may not feel completely comfortable with the question. Some students don’t demonstrate a high-acumen for mathematical relationships until they have completed two levels of algebra or even a pre-calculus and trigonometry course. It will be up to the instructor to decide how to grade a question that incorporates information from outside their discipline.
The other major consideration with this question is the variety of ways students may choose to solve it. Be prepared to trace through a lot of student code that is not completely valid, and focus on the key concepts as much as possible. This is where the development of a rubric can come in to play.
A suggestion for grading open-ended response items is to take the exams, and place them in piles based on which responses are the best based on your intuition. From this point, you can identify within each tier the elements that a response has, or lacks. Then a rubric can be developed that allows a fair evaluation of each student’s demonstrated skill. This takes some time, but can usually improve the consistency in scoring complicated items.
Another good way to get started in designing good open-ended exam questions is to read through the AP Computer Science A free response questions from the past 15 years. These questions are public, and have complete rubrics for scoring. They also in more recent times provide an “ideal solution”, which can help make the expectations of the question more clear. AP free response questions can be found at: http://apcentral.collegeboard.com/apc/public/exam/exam_information/2000.html
