(7) Using PrairieTest

Author

Steven Bogaerts

Note

Due: Monday, September 21

All exams in this course will be taken in the Engineering Testing Facility(opens in new tab) (ETF). See the Exam Details page for more information on exam content and the ETF.

Each exam in this course will use a tool called PrairieTest, which is tightly integrated with PrairieLearn. Your experience taking an exam will be very similar to your current PrairieLearn experience, but there are a few key differences you should be aware of before the first exam. This tutorial steps you through this so you know exactly what to expect.

1 The Main Exam Page

Log into PrairieLearn and click on the PrairieTest Tutorial assessment in the Setup section. You’ll be asked to certify an Honor Code pledge and then you can start the exam.

You’ll be shown the main page for the exam, sometimes called the assessment overview. Read this entire page. It includes:

  • Reference Sheet:
    • Some exams (not all) will include a reference sheet, as specified in the Exam Details page. If a reference sheet is included, it will be linked on the assessment overview page, with the text “Click here for the exam reference sheet”. When taking an exam, before doing anything else, I recommend you open the reference sheet (if applicable) in a new browser tab. The link is only available from the assessment overview page, not on individual questions.
  • Question Zones:
    • There are two zones of questions, labeled “Post-Exam Results Only”, and “In-Exam Feedback”. Details are further below, but to summarize:
      • Post-Exam Results Only: These questions will not give you feedback while you’re taking the exam. You will only see your awarded points for these questions after you finish the exam. These are primarily multiple choice and short answer questions, but could include questions of any type.
      • In-Exam Feedback: These questions will give you feedback while you’re taking the exam. With each response you give, you’ll receive feedback about your answer and a current score. You’ll have a limited number of attempts to earn as many points as possible. These are primarily programming questions (with a larger text entry box), but could include questions of any type.
  • Question Links:
    • There are direct links to each question.
    • Within a zone, the questions are in randomized order. The zones themselves are always in a fixed order.
    • You may answer questions in any order, skip around, come back and change your answer, etc.
    • Each question includes an indication of whether or not you’ve answered it yet.
  • Total Points:
    • How many points the exam is worth overall – in this case, 9 points. (Except this tutorial is just practice; it will not count towards your grade!)
  • Points per Question:
    • How many points each question is worth. In this case, 2, 3, and 4 points. (I’ll explain the list of 4’s later.)
Warning

There is also a Finish assessment button on this page. DO NOT click it until you are certain you are done with the exam!

Important

Since this is just for practice with PrairieTest, please know that right and wrong answers do not matter for this practice. In the walkthrough below, we’ll experiment with getting some answers right and others wrong so you can see what happens in both cases.

2 Randomness in PrairieTest

The questions a given student sees on an exam are randomized in a few ways:

  • The order of questions is randomized within a zone.
  • For multiple choice, the order of the answer choices is randomized.
  • Each question is randomly selected from a pool of questions of similar difficulty.
  • For each question selected, there is randomness in precisely what is presented.

This PrairieTest Tutorial exam includes three questions, each drawn from a pool of related questions. Specifically:

  • One question is drawn randomly from a pool of multiple-choice questions on either addition or subtraction. For the question drawn, additional randomness is applied (e.g., changing some numbers).
  • One question is drawn randomly from a pool of short-answer questions on either addition or subtraction. For the question drawn, additional randomness is applied (e.g., changing some numbers).
  • One question is drawn randomly from a pool of programming questions on either adding or multiplying three non-negative integers. For the question drawn, additional randomness is applied.

3 Questions with No Autograder Feedback

Two of the three questions in this welcome exam include no feedback while you’re taking the exam. They are in the “Post-Exam Results Only” zone. In this exam, they are the questions worth 2 and 3 points. Start by clicking on the question worth 2 points.

Before we consider the question itself, note in the “Tools” section on the right that there is a link to a basic calculator. In EECS 183, you will have access to this calculator for every exam. It is always available within each question, as you see here.

You’ll see that the question itself is a multiple choice question on arithmetic. Go ahead and answer it incorrectly and click Save. You’ll see the confirmation that your answer is saved. Note that, unlike in a regular PrairieLearn assignment, you are not told whether or not the answer is correct, nor is there an option to try a new variant. You can change your answer as often as you want while taking the exam; just be sure to click Save again for that question to register your new answer each time. (This is analogous to erasing and writing a different answer on a paper exam.) Try this now.

Warning

Make sure you hit the Save button for that question any time you select or change your answer! If you don’t hit the save button for that question, your updated answer will not be recorded. There is no penalty for hitting save multiple times, if you’re unsure.

When your answer is saved, you can always navigate away from that question and then navigate back, and your answer will still be there. Go ahead and try this, by either hitting the Assessment overview button to go back to the main exam page, or by clicking Previous question or Next question. Then click back to the question you just answered. Your answer should still be there. Practice this question navigation to be certain that you are comfortable with how it works.

Navigate now to the question worth 3 points. It’s a short-answer question. Answer it as well and experiment as needed. I might recommend, just for the experience, that you intentionally save a correct answer for one question and an incorrect answer for the other. Near the end of this tutorial, you’ll see how the exam points are / are not awarded accordingly.

After answering the 2- and 3-point questions, navigate back to the assessment overview (the main page for the exam) and observe how the summary table reflects your work so far: you’ve answered two questions (saved for grading after finish), but you don’t know how you did on them (pending/2 and pending/3). The 4-point question is still unanswered, with -/4 points. Your total score is still listed as 0/9.

4 Questions with Autograder Feedback and Multiple Attempts

Click on the last remaining question, the one worth 4 points. It is in the “In-Exam Feedback” zone. Thus, in contrast to the other questions, you will receive immediate feedback on this question while you take the exam. You can then revise your solution and resubmit, to try to get more points. This particular question is a programming question with an autograder. In this exam, you have 10 attempts to try to earn the full 4 points. In general, I plan for each “In-Exam Feedback” question to give you enough attempts to have good opportunity for debugging / problem-solving, but not enough for blind guessing (“guess-and-check”).

Warning

Pay attention to how many attempts you have on an In-Exam Feedback question! If you use up all your attempts, you will not be able to submit on that question anymore. Of course, you keep any points you’ve already earned on the question even if you run out of attempts.

Read the problem statement. Depending on which the system chose for you, you’re being asked to complete either add_two_and_constant or multiply_two_and_constant.

4.1 For Your Reference: The Correct Answers

Before we go any further, let’s consider the correct answer for each. But don’t put the correct answer into your test yet, and don’t hit Save & Grade yet! We need to experiment.

Note that a single non-negative integer constant is randomly chosen for you in the question text, so the correct answer will depend on the specific value chosen. For example, suppose the constant is 5. Then the correct answer for add_two_and_constant is:

def add_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the sum of a, b, and the number 5.
    '''
    return a + b + 5

For multiply_two_and_constant:

def multiply_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the product of a, b, and the number 5.
    '''
    return a * b * 5
Tip

If you haven’t seen code like the above before (e.g., the return, or the (a: int, b: int)), then you might want to wait until after we talk about functions in class before moving on. But if you want to continue, consider this very brief explanation for now:

  • Think of a and b as placeholders for values that are provided to the function when it is called.
  • The return statement is how the function “reports back” its result.
  • When we want the function to be executed, we call it with specific values for a and b. For example, if we call add_two_and_constant(2, 3), then a is 2 and b is 3. If 5 is the constant (as in our example), then the function will return 10 (because 2 + 3 + 5 = 10).

4.2 Our First Attempt: Wrong Syntax

Note that the Save & Grade button currently says “10 attempts left”. There’s also a long list of 4’s shown in a couple places. This just indicates that each of your 10 attempts are worth 4 points. More precisely, you can earn a total of 4 points for this question, and you have 10 attempts to do so.

Let’s go ahead and answer the question incorrectly. Type in some gibberish, indented inside the function, like the following (depending on your question):

def add_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the sum of a, b, and the number 5.
    '''
    hgf
def multiply_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the product of a, b, and the number 5.
    '''
    hgf

Click Save & Grade. You’ll see that one of your attempts is used up (9 attempts left). Your Score is currently 0/56 (0%). The 56 is the sum of all values for the 24 test cases shown under Test Results. Click the down-arrow by the first test. You should see something like this (or an analogous message for the other function):

The callable 'multiply_two_and_constant' supplied in your code failed with an exception while it was being called by the grading code:Traceback (most recent call last):
  File "/grade/run/code_feedback.py", line 614, in call_user
    return f(*args, **kwargs)
  File "/grade/run/user_code.py", line 7, in multiply_two_and_constant
    hgf
NameError: name 'hgf' is not defined

It’s a Python interpreter error, just like you see in VS Code! Don’t be intimidated by the error message. Just look for the parts that are helpful: line 7 and NameError: name 'hgf' is not defined. Of course, the interpreter thinks that hgf is a variable we haven’t defined. We’ll fix our code in a moment.

Before we do that, though, click on the down-arrow for the second test. You should see something like this:

- Grading was skipped because an earlier test failed -

That’s the standard language PrairieLearn uses, but a better message would be “Grading was skipped because an earlier test crashed”. Your code crashed, so later test cases couldn’t be run. If the test case merely gives a wrong answer, then the later test cases can still be run. Thus, if you ever see a message like this, be sure you look at the earlier test cases to find the actual error message.

4.3 Our Second Attempt: Wrong Implementation

For the purposes of this tutorial, let’s not get the problem perfect yet. I’ll give you some new code to enter (for whichever problem you have), but make sure you change the constant from 5 to whatever constant was randomly chosen for you in the problem statement.

Enter one of the following, as appropriate:

def add_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the sum of a, b, and the number 5.
    '''
    return a + 5
def multiply_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the product of a, b, and the number 5.
    '''
    return a * 5

The above is still wrong, because it’s ignoring b entirely! But this example will help us understand the tests more.

Click Save & Grade again. Your attempts left decreases to 8, and you see a Score less than 100%.

If you click the down-arrow for a test you passed, it doesn’t show much, since you got it right. But many tests are still failing. I’ll explain the tests labeled Hidden in a moment; for now, click the down-arrow for one of the failing visible (non-“hidden”) tests. You should see a message indicating what (incorrect) value the function actually returned, and what the correct, expected value is.

Important

Score versus Total Points

Note at this time the difference between score and total points in this testing system. The score is based on how many test cases your code passes. The test cases for this problem are out of 56. This is used to proportionally calculate your total points, which is how many points the exam question is worth in calculating your exam grade.

For example, suppose that the autograder scores out of 56, and that your submission currently has a score of 20/56 (35.71%). Then your total points for this question is 35.71% of the 4 points available for this question: .3571 * 4 = 1.43 points. The total points is what matters for your exam grade! The score is just dependent on how many tests there are, and could be out of 10 or 150 or whatever.

If you have 1.43 points already, for a 4-point question, then of course there are 2.57 points remaining to be earned. That’s why the Available points list decreases as you increase your score. It’s not that you’re being penalized for using attempts! Rather, it’s because you’ve already earned 1.43 points. Those are locked in and can’t be lost, even if you later submit code that scores worse. The Available points are those you can still earn on top of what you’ve already locked in.

4.4 Debugging the Code

Each autograded programming problem on an exam will test not merely what you get perfect on the first try, but also how you use extensive test results to debug your code. So, before fixing this code specifically, let’s pause to talk about debugging in general in an exam context.

Debugging code takes practice! The best way to practice with the autograder is in the assigned Post programming questions you’ve been doing all semester. You have many practice problems to work with! Most of these have the same visible/hidden test structure. When you’re studying for an exam, practice these as if you were taking the exam at that moment – no notes, no internet, etc.

You also gain more general debugging practice with the zyBooks exercises, labs, and projects, so I recommend you try those again too. We’re all happy to help in office hours on debugging techniques as well!

4.4.1 Use the test case feedback

Suppose you’re taking a real exam right now, and that you don’t already know what’s wrong with the code we’ve been working on. Take a moment to think about how the feedback reflects what the function currently does correctly and incorrectly. How might this give you hints about what to change in your code?

4.4.2 Try an example by hand

Given the test results, you can sometimes identify what is wrong with your code by just thinking about which arguments lead to success and which lead to failure. Other times, you may need to dig a little deeper and try an example by hand. To do this, pick a visible test case your code fails – perhaps one in particular where both a and b are non-zero, for example. Imagine that a and b take on those values, and trace through your code line by line, just like the interpreter does. In this process, the necessary fix may become clear.

4.4.3 Submit at least once without syntax errors

If the autograder can’t run your code due to a syntax error, it will give you a score of 0, no matter how “close” your code is to being correct. Please note that these scores will not be reviewed / overridden after the exam. This is because one thing the exam is designed to test is your ability to fix syntax errors, with any syntax error messages given to you exactly as in VS Code. These error messages should provide a big hint about what is needed to get your code to run.

If you’re really stuck on syntax, make sure you submit at least once some version of your code with no errors, even if this is a greatly simplified version. That is, you might choose to submit code that obviously doesn’t work in general, but works for certain situations, just to lock in some points before targeting the full problem again. Don’t spend too long on this, though.

4.4.4 Skip as needed and come back later

Don’t be shy about skipping a problem and coming back to it later! Grab the points that are easiest for you first.

4.5 Our Third Attempt: Correct Implementation

Let’s return to the code we’re working on. Of course, in this example, the fix is clear. We need to make use of b as well. Here are the solutions again (and again, you’ll need to be sure to change the constant from 5 to whatever constant was randomly chosen for you in the problem statement):

def add_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the sum of a, b, and the number 5.
    '''
    return a + b + 5
def multiply_two_and_constant(a: int, b: int) -> int:
    '''
    Requires: a and b are non-negative integers.
    Modifies: Nothing
    Effects: Returns the product of a, b, and the number 5.
    '''
    return a * b * 5

Paste in your solution (with the changed constant value) and click Save & Grade again. At this point, you should receive a 56/56 100% score on the autograder, and have all 4 points for the exam question.

Tip

Once you get full credit on the autograder for a question, you are prevented from submitting again for that question, even if you would otherwise have attempts remaining. That’s no problem, because once the autograder gives you full credit, you are guaranteed to have earned all the points for that question on the exam.

4.6 Visible and Hidden Tests

You’ve probably noticed that some tests are simply labeled Hidden Test. In a hidden test, you are not told what input values are, nor the expected or actual output. If a hidden test does not pass, it will simply show the message Hidden test did not pass.

Use the visible (non-hidden) tests for debugging your code. The hidden tests are just additional test cases used to manage the scoring.

5 Finishing the Exam

When you’ve answered all the questions, go back to the main exam page. Recall that you can get there from any question by clicking the Assessment overview button.

In a real exam, do not click the Finish assessment button until you are certain you are done with the exam! Double check: Have you answered every question?

  • Every question should indicate either saved for grading after finish (for questions in the “Post-Exam Results Only” zone) or complete (for questions in the “In-Exam Feedback” zone).
  • If any question lists unanswered, go back and give an answer if you can. The Finish assessment button will also warn you first if you have unanswered questions.

When you’re certain that you are done with the exam and would like to turn it in, click the Finish assessment button. You’ll be asked to confirm; when you’re ready, click again to Finish assessment.

At that time, you’ll see your awarded points for every problem, including the ones you didn’t get feedback on previously. You’re done with the exam!

6 Taking an Exam at the ETF

Your experience at the ETF taking a real exam will be very similar. Of course, differences include the following:

  • The exam will be longer and more difficult than this welcome exam.
  • You will be taking the exam on a computer provided by the ETF, rather than your own laptop.
  • You’ll be guided through strict protocols about checking in, devices, material usage, etc. See the Exam Details page for more information.