Runtime

On this page, you have complete control over how you'd like your test's to run. This page allows you to update your assignment files, select which files are required for submissions, change the runtime environment, and set a time limit.

Open assignment runtime

1) To view your assignment's runtime, open the Assignments page by clicking the Assignment button in the activity bar, on the left side of the screen.

2) On the Assignments page, click the blue Open button that corresponds to the assignment that you'd like to open.

3) Now that the Assignments page is open, in the bar at the top of the screen, click the tab labeled Runtime.

Student files

This is where you may select which files you wish to require your students to submit. When you internally uploaded your assignment files, you need to include your assignment answers too. The auto grading system needs to know which files it should replace with the students files when setting up the project prior to testing. This drop down menu will only contain the file names of the files that you submitted; the auto grading system will then proceed to swap out your files with your student's assignment files.

The auto grader preforms a full refactoring prior to testing

The auto grader preforms a full refactoring when swapping files. This means the system is able to handle additional files that are not listed, so your students are allowed to submit helper packages or even an entire project if desired.

Additionally, dependencies are automatically resolved for you too. For example, if your original file imports package_a.package_b.function_a and your student imports the same function but has a different project structure such as package_c.package_b.function_a the auto grader will refactor the student's code to match your uploaded project structure.

Misspellings of file names is a very common issue. For example, you may tell your students to submit a file named functions_homework.py and you may receive functions homework.py, functions-homework.py, Functions_Homework.py, FunctionsHomework.py, functions_homework.java and functions_homework.docx. Fortunately, Grader Than handles this issue before the student's files are tested. By selecting the files you expect in the submission the system, you will be able to inform them of the missing files immediately. It does not matter if they submit an archive file or an entire project. The auto grader will be able to search for the required file(s) in a project or archive file (zip, rar, etc).

Partial Submissions

This feature enables your students to submit one or more of the expected student files at a time. This is helpful because they will be able to make incremental improvements. Notably, they must submit all files at once to gain full credit.

This is an advanced feature, it might break your unit test code if you statically import your student's code and import errors are not handled. It is best to leave this unchecked if your unit test code is not designed to dynamically import your students files or the code does not handle the situation when an expected file is missing from the submitted files. This feature is generally meant for Python because it can handle import errors if a function or class does not exist. Its recommended that you do NOT check this box if its a java project.

When to use partial submissions

• If you are working with command line applications (CLI) and you are starting your student's applications via command line, you may use this feature for both Java and Python CLI applications.
• If you are handling import errors in your python unit test code: Read more about handling import errors.

Environment

This is where you may select the runtime environment you'd like your code to run in. Typically, you don't need to worry about setting this because it will automatically be selected for you based on the files you submit.

Time Limit

With this option you can control the maximum time your student's code has to complete the necessary testing. This time limit includes the project setup and validation process, which takes about 5-20 seconds depending upon the project's size. Typically, you don't need to modify this setting. This feature is primarily meant to stop the system from stalling if it encounters inevitable infinite while-loops or an input statement a student forgot to remove. Based on our experience, you may want to consider increasing this setting in the following situations:

• If you know your student's code will take a long time to complete the tests because you are teaching a topic that requires CPU intensive tasks. An example of this would be, you are teaching an algorithms course and discussing exponential time or polynomial time algorithms \(O(k^n)\) or \(O(n^k)\).
• You are learning about http connections and your student's code downloads many files.

In our experience it's often best to set individual timeouts for your tests. For Java you can specify a timeout time within the Junit annotation. For python there is no built-in timeout support in the Unittest library but we recommended you use timeout-decorator library. Below is an example of how you would define a timeout value of 5 seconds for either programming language:

PythonJava

import unittest
import timeout_decorator


class ExampleTest(unittest.TestCase):

    @timeout_decorator.timeout(5)
    def test_student_code(self):
        ...

import org.junit.Assert;
import org.junit.Test;

import java.util.Random;

public class ExampleTest {
    @Test(timeout = 5000)
    public void testStudentCode(){
        ...
    }
}