How to Create a Programmatic Problem

A programmatic problem is assessed using unit tests, meaning you’ll write code to test your students' submitted code. Unit tests allow for more versatile and powerful testing, but they require a deeper understanding of programming and are recommended for teachers who are comfortable with coding.

In this example, we’ll ask the student to create a calculator function that accepts three inputs: a string or character (+, -, *, or /), an integer, and another integer. The function is expected to return the result of the operation.

Create a Problem Set First

Before starting, make sure you’ve already created a problem set. If not, please create a problem set first by following the previous tutorial.

Add a new problem

1. From the problem set editor page, press the plus button in the top right corner.

2. A modal will appear. Select the New problem button in the middle.

3. Give the problem a name (for this example, we’ll call it "Calculator Function") and click Create and add.

Create the problem prompt

1. Back on the main screen, a new problem editor will appear.

2. In the problem prompt section, type exactly what you want your students to do. Be as clear as possible since this will be used by the AI to provide assistance to students.

Markdown

The prompt supports both HTML and Markdown formatting. You’ll see a live rendering of the prompt in the right-hand box. For more on Markdown, check out this guide.

Add a Unit test

1. Scroll down to the bottom of the problem editor and toggle the button labeled Programmatic.

2. Keep the box labeled Add an example to the new test checked, as we’ll start with this.

3. Click Create a Programmatic test.

Python Java Javascript C++

The starter example code tests an add() function in Python using the standard unittest framework. However, all Python unit tests in Grader Than Feedback are executed using pytest, which is compatible with Python's standard unittest library. This means you can access the full capabilities of both pytest and the unittest library. For more details, refer to the pytest documentation or the Python standard library documentation.

import unittest

class AddTestCase(unittest.TestCase): # (1)!
    def test_add(self):
        try: # (2)!
            from submitted import add
        except ImportError:
            self.fail("The function 'add()' is missing!")

        result = add(3, 5) # (3)!
        self.assertEqual(result, 8)  # (4)!

if __name__ == '__main__':
    unittest.main()

1. This creates a test class. In unit testing, tests are grouped in classes.
2. Here, we attempt to import the student’s function from the submitted module. The Grader Than system will automatically place the student’s code in a file called submitted.py. We will access the add function from this module. If it doesn’t exist, Python will raise an error, and the test will fail gracefully.
3. This calls the student’s function and saves the output.
4. This checks that the function’s output matches the expected value, in this case, 8.

This code defines a unit test for the add() function using Python's unittest framework. It checks if the function add(3, 5) correctly returns 8 and fails the test if the function is missing or improperly named.

---

Modify the unit test for the calculator problem:

1. Rename AddTestCase to CalculatorTestCase.
2. Replace add with calculator.
3. Call the calculator function like this: calculator('+', 3, 5).

The final code should look like this. The lines that have been changes have be highlighted:

import unittest

class CalculatorTestCase(unittest.TestCase):
    def test_calculator(self):
        try:
            from submitted import calculator
        except ImportError:
            self.fail("The function 'calculator()' is missing!")

        result = calculator('+', 3, 5)
        self.assertEqual(result, 8)

    def test_subtraction(self):
        try:
            from submitted import calculator
        except ImportError:
            self.fail("The function 'calculator()' is missing!")

        result = calculator('-', 10, 5)
        self.assertEqual(result, 5)

    def test_multiplication(self):
        try:
            from submitted import calculator
        except ImportError:
            self.fail("The function 'calculator()' is missing!")

        result = calculator('*', 4, 5)
        self.assertEqual(result, 20)

    def test_division(self):
        try:
            from submitted import calculator
        except ImportError:
            self.fail("The function 'calculator()' is missing!")
        result = calculator('/', 20, 5)
        self.assertEqual(result, 4)

if __name__ == '__main__':
    unittest.main()

Create separate tests for each operator:
Instead of testing all operations in one method, we will create individual test methods for each operator: addition, subtraction, multiplication, and division. Each test method will focus on one specific operation and use Assertions.assertEquals() to compare the output of the function with the expected result.

The starter example code tests an add() function in Java using the JUnit framework, which is the standard testing framework for all Java unit tests in Grader Than Feedback. For more details on how to write and run tests with JUnit, refer to the JUnit documentation.

// Import necessary junit dependencies
import org.junit.jupiter.api.Assertions; // (1)!
import org.junit.jupiter.api.Test;

// Import the student's Calculator class
import Calculator; // (2)!

public class CalculatorTest { // (3)!

    // A single unit test. You may add more!
    @Test
    public void testAdd() { // (4)!

        Calculator calculator = new Calculator(); // (5)!

        int result = calculator.add(5, 3); // (6)!

        Assertions.assertEquals(8, result); // (7)!
    }
}

1. These lines import the JUnit dependencies needed for testing. JUnit is the framework we use to write unit tests in Java.

2. This imports the student's Calculator class from their submitted code. In the background, the Grader Than system places the student's code in a file that can be accessed in this unit test.

3. This creates a test class called CalculatorTest. In JUnit, test cases are grouped into classes like this.

4. The @Test annotation marks the method as a unit test. In this case, we’re testing the add() method from the student's Calculator class.

5. This line creates an instance of the Calculator class, which will allow us to call the add() method.

6. Here, we call the add() method on the calculator object, passing in the arguments 5 and 3. The result is stored in the result variable.

7. Finally, we use Assertions.assertEquals() to check if the result is equal to 8. If the result is correct, the test passes. If not, the test fails.

You must tell the student the class name you expect

In your problem prompt, you need to tell the student the name of the class you expect to find in your unit test. In this case, we expect the class to be named Calculator. This is critical for the code to compile.

---

Modifying the Unit Test for the Calculator Problem

1. Replace the method call

   Change the method call from add() to calculate(), since the task requires testing a calculate() function that accepts an operator (+, -, *, /) and two integers.

2. Update the method signature

   Adjust the calculate() method to handle three inputs: the operator (as a char), and two integer values.

3. Create separate tests for each operator

   Instead of testing all operations in one method, we will create individual test methods for each operator: addition, subtraction, multiplication, and division. Each test method will focus on one specific operation and use Assertions.assertEquals() to compare the output of the function with the expected result.

Here is the final version of the unit test with separate tests for each operator. The lines that have been changes have be highlighted::

// Import necessary junit dependencies
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

// Import the student's Calculator class
import Calculator;

public class CalculatorTest {

    // Test addition
    @Test
    public void testAdd() {
        // Create an instance of the Calculator class
        Calculator calculator = new Calculator();

        // Test addition
        int result = calculator.calculate('+', 3, 5);
        Assertions.assertEquals(8, result);  // Check if the result is 8
    }

    // Test subtraction
    @Test
    public void testSubtract() {
        // Create an instance of the Calculator class
        Calculator calculator = new Calculator();

        // Test subtraction
        int result = calculator.calculate('-', 10, 5);
        Assertions.assertEquals(5, result);  // Check if the result is 5
    }

    // Test multiplication
    @Test
    public void testMultiply() {
        // Create an instance of the Calculator class
        Calculator calculator = new Calculator();

        // Test multiplication
        int result = calculator.calculate('*', 4, 5);
        Assertions.assertEquals(20, result);  // Check if the result is 20
    }

    // Test division
    @Test
    public void testDivide() {
        // Create an instance of the Calculator class
        Calculator calculator = new Calculator();

        // Test division
        int result = calculator.calculate('/', 20, 5);
        Assertions.assertEquals(4, result);  // Check if the result is 4
    }
}

Explanation of Changes

• Separate Tests for Each Operator Each operator (+, -, *, /) has its own dedicated test method: testAdd(), testSubtract(), testMultiply(), and testDivide(). This approach makes it easier to isolate and test each operation individually.

By separating the tests, students can earn partial credit for the tests the pass.

The starter example code tests an add() function in JavaScript using the Jest framework, which is the standard testing framework for all JavaScript unit tests in Grader Than. For more details on how to write and run tests with Jest, refer to the Jest documentation.

const { add } = require('./submitted');

describe('add()', () => {
it('should return the sum of two numbers', () => {
    // Test case: Adding 2 and 3 should return 5
    const result = add(2, 3);
    expect(result).toBe(5);
});

it('should return zero when both numbers are zero', () => {
    // Test case: Adding 0 and 0 should return 0
    const result = add(0, 0);
    expect(result).toBe(0);
});

it('should handle negative numbers correctly', () => {
    // Test case: Adding -5 and 10 should return 5
    const result = add(-5, 10);
    expect(result).toBe(5);
});
});

This code tests the add() function by passing in different numbers and verifying that the correct result is returned. However, for the calculator problem, we need to modify the test to account for different operations like addition, subtraction, multiplication, and division.

---

Changes to Make:

1. Replace add() with calculate():
   The task is to create a calculate() function that accepts three inputs: the operator (+, -, *, /), and two integers. We need to change all references to add() to calculate().

2. Separate Tests for Each Operation:
   Instead of testing all operations in one test, we will create separate tests for each operation (+, -, *, /). This not only makes it easier to organize the tests but also allows students to earn partial credit for correctly implementing individual operations.

---

Modified Unit Test Example

Here’s how the updated test should look for the calculator problem:

const { calculate } = require('./submitted');

describe('calculate()', () => {
    // Test addition
    it('should return the sum of two numbers when using "+"', () => {
        const result = calculate('+', 3, 5);
        expect(result).toBe(8);  // 3 + 5 = 8
    });

    // Test subtraction
    it('should return the difference of two numbers when using "-"', () => {
        const result = calculate('-', 10, 5);
        expect(result).toBe(5);  // 10 - 5 = 5
    });

    // Test multiplication
    it('should return the product of two numbers when using "*"', () => {
        const result = calculate('*', 4, 5);
        expect(result).toBe(20);  // 4 * 5 = 20
    });

    // Test division
    it('should return the quotient of two numbers when using "/"', () => {
        const result = calculate('/', 20, 5);
        expect(result).toBe(4);  // 20 / 5 = 4
    });
});

---

Explanation of the Changes:

1. Changing the Function Name:
   We replaced add() with calculate() since the students are required to implement a calculator function that can handle multiple operations, not just addition. The calculate() function accepts three parameters: an operator (+, -, *, /) and two numbers.

2. Creating Separate Tests for Each Operator:
   Each mathematical operation is tested in its own it() block. This structure makes the code more modular and easier to debug. If one test fails (e.g., subtraction), the other tests (e.g., addition or multiplication) can still pass, allowing students to earn partial credit for the sections they have implemented correctly.

The starter example code tests an add() function in C++ using the GoogleTest framework, which is the standard testing framework for all C++ unit tests in Grader Than Feedback. For more details on how to write and run tests with GoogleTest, refer to the GoogleTest documentation.

#include <gtest/gtest.h>

// NOTE: Don't worry about including a header file or a main() function
// that will be added automatically!

// Test case
TEST(AddTest, SumOfTwoIntegers) {
// Test inputs
int a = 2;
int b = 3;

// Expected result
int expectedSum = 5;

// Call the function being tested
int actualSum = add(a, b);

// Assertion
ASSERT_EQ(actualSum, expectedSum);
}

This test is designed to check if the add() function correctly returns the sum of two integers. However, for the calculator problem, we need to test a calculate() function that handles different operations: addition, subtraction, multiplication, and division.

---

Steps to Modify the Code

1. Replace the add() function with calculate():
   Since the task requires students to implement a calculator function, all calls to add() must be replaced with calculate(). The new function will accept three arguments: an operator (+, -, *, /), and two integers.

2. Create Separate Tests for Each Operation:
   We will separate the tests into individual test cases for addition, subtraction, multiplication, and division. This ensures each operation is evaluated independently, which allows for partial credit if a student correctly implements one operation but not the others.

3. Partial Credit Explanation:
   By creating individual tests for each mathematical operation, students can earn points for completing specific parts of the task. If, for example, a student only completes the addition and multiplication portions of the calculator, they will still receive partial credit for their progress.

---

Modified Unit Test Example

Here is the updated code with separate test cases for each operator:

#include <gtest/gtest.h>

// NOTE: Don't worry about including a header file or a main() function
// that will be added automatically!

// Test case for addition
TEST(CalculatorTest, Addition) {
// Test inputs
int a = 3;
int b = 5;

// Expected result
int expected = 8;

// Call the function being tested
int actual = calculate('+', a, b);

// Assertion
ASSERT_EQ(actual, expected);
}

// Test case for subtraction
TEST(CalculatorTest, Subtraction) {
// Test inputs
int a = 10;
int b = 5;

// Expected result
int expected = 5;

// Call the function being tested
int actual = calculate('-', a, b);

// Assertion
ASSERT_EQ(actual, expected);
}

// Test case for multiplication
TEST(CalculatorTest, Multiplication) {
// Test inputs
int a = 4;
int b = 5;

// Expected result
int expected = 20;

// Call the function being tested
int actual = calculate('*', a, b);

// Assertion
ASSERT_EQ(actual, expected);
}

// Test case for division
TEST(CalculatorTest, Division) {
// Test inputs
int a = 20;
int b = 5;

// Expected result
int expected = 4;

// Call the function being tested
int actual = calculate('/', a, b);

// Assertion
ASSERT_EQ(actual, expected);
}

---

Explanation of the Changes

1. Function Name Change:
   We replaced all instances of add() with calculate(), reflecting the function students are required to implement. This function accepts three arguments: an operator (+, -, *, /) and two integers.

2. Separate Tests for Each Operator:
   Instead of testing all operations in a single test case, we created a separate test for each operation: Addition, Subtraction, Multiplication, and Division. This way, each test focuses on one specific mathematical operation.

3. Assertion Using ASSERT_EQ():
   Each test uses ASSERT_EQ() to compare the actual result returned by the calculate() function with the expected result. For example, in the addition test, we check whether calculate('+', 3, 5) returns 8.

4. Partial Credit:
   By separating the operations into their own test cases, students can earn partial credit if they correctly implement some of the operations but not all. For example, if the addition and subtraction tests pass but the multiplication and division tests fail, students still receive credit for what they completed successfully.