From b05ef223f8127a5b2e4926cc84a492d3a9253337 Mon Sep 17 00:00:00 2001 From: Eliot Robson Date: Tue, 10 Nov 2026 16:00:01 -0500 Subject: [PATCH 1/5] Add a guide for converting questions from the legacy Python grader Also document leading and trailing code, query_setup and the limits on setup code in the quick start, and fix the module-level timeout example, which used a variable the plugin doesn't read. Co-Authored-By: Claude Opus 5.5 --- README.md | 3 + converting.md | 330 +++++++++++++++++++++++++++++++++++++++++++++++++ quick_start.md | 48 ++++++- 3 files changed, 375 insertions(+), 6 deletions(-) create mode 100644 converting.md diff --git a/README.md b/README.md index faf1b15..b89460a 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,9 @@ pip install pytest-prairielearn-grader[notebook] See [quick_start.md](quick_start.md) for a full tutorial with PrairieLearn integration examples. +To convert a question from the legacy Python grader (`prairielearn/grader-python`), follow +[converting.md](converting.md). It's written so that you can hand it to an AI coding agent. + ## Key Features ### Sandboxed Execution diff --git a/converting.md b/converting.md new file mode 100644 index 0000000..290a2a7 --- /dev/null +++ b/converting.md @@ -0,0 +1,330 @@ +# Converting Questions from the Legacy Python Grader + +This guide explains how to convert a PrairieLearn question from the legacy Python autograder +(`prairielearn/grader-python`, with `tests/test.py` and `PLTestCase`) to this pytest grader +(`prairielearn/grader-python-pytest`). It's written so that you can give it to a person or an AI +coding agent and have them do the conversion from start to finish. + +[quick_start.md](quick_start.md) documents every feature in detail. This guide covers what changes, +in what order to do it, and the differences that are easy to miss. + +Converted questions in the PrairieLearn example course are good references: + +| Question (under `exampleCourse/questions/demo/`) | Shows | +| ------------------------------------------------ | ------------------------------------------------------------- | +| `autograder/python/square` | `setup_code.py`, `query_setup`, a scalar with a tolerance | +| `autograder/python/pandas` | Checking a DataFrame | +| `autograder/python/randomFunctionName` | `server.py` parameters, `data_json`, partial credit, timeouts | +| `autograder/codeEditor` | A function question | +| `annotated/MarkovChainGroupActivity/*` | Notebook workspaces, `helper_modules`, `data_files` | + +## 1. Check whether the question can be converted + +Read every file in the question, including `info.json`, `question.html`, `server.py` and everything in +`tests/`. Stop and report back instead of converting if the question needs one of these: + +- **Plot grading.** `PLTestCaseWithPlot` with tests that read `self.st.plt` or call + `Feedback.check_plot`. The pytest grader can only grade plots that student code builds with + `MatplotlibSerializer` and returns from a function (see + [Testing Matplotlib Plots](quick_start.md#testing-matplotlib-plots)), so the student's task would + change. `PLTestCaseWithPlot` on its own, with no test that reads the plot, isn't a blocker: use the + normal conversion. +- **Course libraries.** `externalGradingOptions.serverFilesCourse` or imports of course-wide + modules. The legacy grader put `/grade/serverFilesCourse` on the import path, and this grader + doesn't. +- **A custom grader image** or an `entrypoint` in `externalGradingOptions`. +- **Values the sandbox can't send back.** Tests can only receive values that can be serialized: + numbers, strings, `None`, lists, dicts, numpy arrays and scalars, pandas DataFrames and plots + built with `MatplotlibSerializer`. A test that needs a student's class instance, generator or + open file has to be rewritten to call student functions that return plain values. Do that only + if the student's task stays the same. + +## 2. Change the files + +| Legacy file | What to do | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| `info.json` | Change `externalGradingOptions.image`. Keep `timeout` and everything else. See [Choosing an image](#choosing-an-image). | +| `question.html` | Change the file editor's `file-name` to `student_code.py`, or keep it and set `student_code_pattern`. | +| `tests/test.py` | Replace with `tests/test_student.py`. See [Rewrite the tests](#3-rewrite-the-tests). | +| `tests/ans.py` | Delete it. Compute expected values in `test_student.py`. Don't leave a reference solution in `tests/`. | +| `tests/setup_code.py` | Keep it if it's needed, and update it. See [Update the setup code](#4-update-the-setup-code). | +| `tests/leading_code.py` | Keep it. It's prepended to the student's code, as before. | +| `tests/trailing_code.py` | Keep it. It's appended to the student's code, as before. | +| `tests/initial_code.py` | Keep it. The grader doesn't read it. Fix starter code that doesn't parse (see below). | +| `server.py` | Usually unchanged. | + +Other changes: + +- **Starter code that doesn't parse.** A submission with a `SyntaxError` is ungradable, so it + doesn't cost the student an attempt, but the student sees no test results. If `initial_code.py` + has placeholders like `x = # fill this in`, replace them with something that parses, such as + `x = 0 # fill this in`, so an unchanged submission gets 0 points with feedback. +- **``.** Keep these elements. The `names_for_user` list still controls + which setup variables student code gets. The grader ignores `names_from_user`, which only + displays the table. +- **The file name.** The grader looks for `student_code*.py` by default, while the legacy grader looked + for `user_code.py`. Change `file-name="user_code.py"` to `file-name="student_code.py"` in + `question.html`, or add `student_code_pattern="user_code.py"` to the `ConfigObject`. For workspace + and notebook questions, set `student_code_pattern` to the graded file's name, such as + `"Markov-Chains-1.ipynb"`. + +### Choosing an image + +- `prairielearn/grader-python-pytest:lite` when the student code, setup code and tests use only the + standard library. +- `prairielearn/grader-python-pytest:latest` for anything else, including numpy, pandas, + matplotlib, sympy and notebooks. + +See [Choosing an Image](quick_start.md#choosing-an-image) for what each includes. + +## 3. Rewrite the tests + +Each `test_*` method on the legacy `Test` class becomes a `test_*` function in +`tests/test_student.py`. Keep each test's name, points and order, and keep checking the same things +with the same tolerances. Don't make the question easier or harder. + +Here's the `square` question before and after: + +```python +# Legacy tests/test.py +class Test(PLTestCaseWithPlot): + @points(10) + @name("x_sq") + def test_0(self): + if Feedback.check_scalar("x_sq", self.ref.x_sq, self.st.x_sq, accuracy_critical=False): + Feedback.set_score(1) + else: + Feedback.set_score(0) +``` + +```python +# tests/test_student.py +import pytest +from pytest_prairielearn_grader.assertions import assert_approx_equal +from pytest_prairielearn_grader.fixture import StudentFixture +from pytest_prairielearn_grader.utils import GradingOutputLevel + + +@pytest.mark.output(level=GradingOutputLevel.Friendly) +@pytest.mark.grading_data(name="x_sq", points=10) +def test_x_sq(sandbox: StudentFixture) -> None: + x = sandbox.query_setup("x") + assert_approx_equal(sandbox.query("x_sq"), x**2, description="x_sq") +``` + +### API mapping + +| Legacy | pytest grader | +| -------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| `class Test(PLTestCase)` with `test_*` methods | Module-level `test_*` functions that take the `sandbox` fixture | +| `@points(10)` and `@name("...")` | `@pytest.mark.grading_data(name="...", points=10)` | +| Tests run in sorted name order | Tests run in file order | +| `self.st.x` | `sandbox.query("x")` | +| `Feedback.call_user(self.st.f, a, b)` | `sandbox.query_function("f", a, b)`, or `query_function_raw` to inspect exceptions and timeouts | +| `self.ref.x` from `ans.py` | Compute the expected value in the test | +| A setup variable, such as `x` from `setup_code.py` | `sandbox.query_setup("x")` | +| `self.data["params"]["k"]` | The `data_json` fixture: `data_json["params"]["k"]` | +| `Feedback.set_score(s)` | `feedback.set_score(s)` with the `feedback` fixture. See [Scoring](#scoring). | +| `Feedback.add_feedback("...")` | `feedback.add_message("...")` | +| `accuracy_critical=True` | A failed assertion ends the test. Other tests still run. | +| `total_iters = 3` and `@not_repeated` | `@pytest.mark.parametrize` over the inputs, or a loop with partial credit | +| `Feedback.not_allowed` | Define your own function in `setup_code.py`. See [Update the setup code](#4-update-the-setup-code). | +| `repeated_setup()` | Not needed: setup code runs again for each sandbox. | + +### Checks + +Replace the `Feedback.check_*` functions with the helpers in `pytest_prairielearn_grader.assertions`, +or with numpy and pandas functions wrapped in `assert_true`: + +| Legacy | pytest grader | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `check_scalar(name, ref, st)` | `assert_approx_equal(value, expected, description=name)`, keeping `rtol=1e-5, atol=1e-8` unless the test set others | +| Exact comparisons | `assert_equal(value, expected, description=name)` | +| `call_user` plus a comparison | `assert_fn_equal(sandbox, "f", args=(...), expected=...)` or `assert_fn_approx_equal` | +| `check_numpy_array_allclose(name, ref, st)` | The four array checks below | +| `check_numpy_array_features(name, ref, st)` | The first three array checks below | +| `check_list(name, ref, st, entry_type=T)` | `assert_true(isinstance(value, list), ...)`, a length check and an `isinstance` check on each entry | +| `check_tuple`, `check_dict` | `assert_equal` | +| `check_dataframe(name, ref, st)` | `pandas.testing.assert_frame_equal(expected, df, check_like=True)`, as in the `pandas` example question | + +For a numpy array, check the type, dtype, shape and values in that order, so that each failure has a +clear message and a wrong shape doesn't reach `np.allclose`: + +```python +value = sandbox.query("x") +assert_true(isinstance(value, np.ndarray), msg="x should be a numpy array") +assert_true(np.issubdtype(value.dtype, np.number), msg="x should contain numbers") +assert_true(value.shape == expected.shape, msg=f"x should have shape {expected.shape}") +assert_true(np.allclose(value, expected, rtol=1e-5, atol=1e-8), msg="x is inaccurate") +``` + +`assert_approx_equal` is for scalars. Don't use it for arrays. + +### Feedback messages + +Use `@pytest.mark.output(level=GradingOutputLevel.Friendly)` on each test, as the example questions +do, or `output_level="friendly"` in the `ConfigObject`. At this level the student sees only the +assertion's message. + +Pick assertions with that in mind: + +- **Bare `assert`.** Don't use it in tests. pytest rewrites it, so the student sees + `assert 6 == 12345` and the fixture's `repr` along with your message, which gives away the + expected value. +- **numpy and pandas assertions.** `np.testing.assert_allclose` and similar include both arrays in + their messages. Use `assert_true(np.allclose(...), msg=...)` instead. +- **`assert_equal` and the other comparison helpers.** These show the expected value. That's fine + when the legacy feedback showed it too, or when the test's inputs are visible to the student. For + hidden test cases, use `assert_true` with a message that doesn't include the answer. +- **Wording.** Keep the wording of the legacy feedback where you can, such as `"x is inaccurate"`. + +### Scoring + +- A test that finishes without failing gets full points, even if it never calls `feedback.set_score`. +- A test that fails gets the last score it set with `feedback.set_score`, or 0. Call `set_score` + after each check that earns partial credit. +- `feedback.set_score_final(s)` sets the score even if the test then fails. +- The legacy pattern of `if check: set_score(1) else: set_score(0)` becomes a plain assertion. + +### Querying + +- `query` and `query_function` raise `NameError` if the student didn't define the name, which fails + the test with a message that says so. You don't need to check this yourself. +- Each call to `query_function` has a 1 second timeout by default. Pass `query_timeout=` for slower + functions. +- When a test calls a student function several times, use `query_function_raw` and stop after the + first `"timeout"` status: see the `randomFunctionName` example question. + +## 4. Update the setup code + +Setup code runs inside the sandbox, so some things that worked in the legacy grader don't: + +- **Parameters.** `data` isn't defined. Use `__data_params`, which is `data["params"]`. Values that + `server.py` stored with `pl.to_json` arrive still encoded: decode them with + `__from_server_json(__data_params["A"])`. The `prairielearn` package isn't available. +- **Restrictions.** Setup code runs with the same builtins and import rules as student code. For + example, it can't call `open` or import `os`. To read a data file, list it in `data_files` and use + a library function such as `np.loadtxt`. +- **Variables for the student.** Student code only gets the setup variables named in + `names_for_user`. This list usually comes from `` in `question.html`. If the question defines these names somewhere + else, or not at all, set `names_for_user=[...]` in the `ConfigObject`. +- **Fresh values per test.** The `sandbox` fixture runs setup code and student code again for each + test, so random values in setup code change between tests. Read them with `sandbox.query_setup` in + the same test that uses them, as `square` does. Don't compute expected values from random setup + values at module level. +- **Disallowed functions.** `Feedback` isn't available. Define a replacement in setup code: + + ```python + import numpy.linalg as la + + + def not_allowed(*args, **kwargs): + raise RuntimeError("Usage of this function is not allowed in this question.") + + + la.inv = not_allowed + ``` + +If the setup code only defined values that tests need, and student code doesn't use them, move that +code into `test_student.py` and delete `setup_code.py`. + +## 5. Configure the grader + +Put a `ConfigObject` named `autograder_config` at the top of `test_student.py` when the defaults +don't fit: + +```python +from pytest_prairielearn_grader import ConfigObject + +autograder_config = ConfigObject( + student_code_pattern="Markov-Chains-1.ipynb", + notebook_cell_tag="#grade", + # Importing numpy in student code can take longer than the default timeout. + sandbox_timeout=15, +) +``` + +When a `ConfigObject` is present, the module-level `sandbox_timeout` variable and the +`@pytest.mark.sandbox_timeout` marker are ignored. + +| Situation | Setting | +| --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| Student or setup code imports numpy, pandas or other large libraries | `sandbox_timeout=15`. The default of 1 second covers starting the student's code. | +| The graded file isn't `student_code*.py` | `student_code_pattern="..."` | +| A notebook where only some cells should run | `notebook_cell_tag="#grade"` | +| Student code or `leading_code.py` imports a module from `tests/` | `helper_modules=["module_name"]`. See [Helper Modules](quick_start.md#helper-modules). | +| Student code or `leading_code.py` reads a file from `tests/` | `data_files=["file.txt"]`, and read it in tests with the `tests_dir` fixture. See [Data Files](quick_start.md#data-files). | +| Student code imports `os`, `sys`, `subprocess`, `pathlib` or `shutil` | `import_blacklist=[...]` without that module. These are blocked by default. | +| The legacy setup code blocked imports or builtins | `import_whitelist`, `import_blacklist` or `builtin_whitelist` | +| A multi-file workspace project with `.py` files | `workspace_mode=True`. See [Workspace Grading](quick_start.md#workspace-grading). | + +`test_student.py` itself can't import other files from `tests/`. Put helper code for the tests in +`test_student.py`, and read data files through `tests_dir`. + +## 6. Notebook questions + +For questions where students submit a Jupyter notebook from a workspace: + +- Set `student_code_pattern` to the notebook's name from `workspaceOptions.gradedFiles`. +- Legacy notebook questions grade only the cells that start with `#grade`. Keep that with + `notebook_cell_tag="#grade"`. Without it, every code cell runs, including plots and scratch work. +- Set `sandbox_timeout=15` if the notebook imports numpy or similar libraries. +- Notebook questions use the normal `sandbox` fixture, not `workspace_mode`. +- The grader runs the graded cells as a script, so anything a graded cell needs must be defined in + a graded cell or in `leading_code.py`. +- If the notebook reads a data file, list it in `data_files`. If the notebook imports a `.py` file + that's in the workspace, copy that file into `tests/` and list it in `helper_modules`. + +## Behavior differences + +These differences from the legacy grader are easy to miss: + +- **`__name__`.** Student code runs with `__name__ == "__main__"`, so code under + `if __name__ == "__main__":` runs during grading. The legacy grader skipped it. If the starter code + has such a block, check that running it during grading is harmless. A `leading_code.py` that only + sets `__name__ = "__main__"` is no longer needed. +- **No reference run.** `ans.py` isn't run. Expected values come from the test code. +- **Setup code runs for each test**, and so does the student's code. Output that student code prints + at the top level appears in each test's feedback, unless the test sets + `include_stdout_feedback=False` in `grading_data`. +- **Default import blacklist.** `os`, `sys`, `subprocess`, `pathlib` and `shutil` are blocked unless + you set `import_blacklist`. +- **Syntax errors** make the submission ungradable instead of scoring 0. + +## 7. Test the conversion + +Test the converted question with the grader image before calling the conversion done. Run each case +in a fresh copy of the layout, because the image moves files around: + +```bash +mkdir -p grade/tests grade/student grade/data +cp -R path/to/question/tests/. grade/tests/ +cp solution.py grade/student/student_code.py +echo '{"params": {}}' > grade/data/data.json +docker run --rm -v "$PWD/grade:/grade" prairielearn/grader-python-pytest:latest +cat grade/results/results.json +``` + +- **`data.json`.** Put in `params` whatever the question's `server.py` and elements would set. In + particular, `` sets + `params["names_for_user"]` to a list of `{"name": ..., "description": ..., "type": ...}` objects. + Without it, student code doesn't get the setup variables. +- **Cases to run.** At least a correct solution, which should get full points, the unchanged starter + code, which should get 0 without crashing the grader, and one wrong answer for each test. Compare + the scores with what the legacy grader would give. +- **Messages.** Read `message` for each test in `results.json`, and check that hidden test cases don't + reveal their expected values. +- **Notebooks.** Use the solution notebook as the submission, with the file name from `gradedFiles`. + +Finally, test the question in PrairieLearn itself, with a local PrairieLearn server and the example +or course repository. A person usually does this step. If you couldn't, say so when you report back. + +## 8. Report back + +When you're done, report: + +- Which files you changed, added and deleted. +- Any check whose behavior changed, such as a different tolerance or message, and why. +- The cases you ran and their scores. +- Anything you couldn't convert or test. diff --git a/quick_start.md b/quick_start.md index 3996040..0bca1ad 100644 --- a/quick_start.md +++ b/quick_start.md @@ -5,8 +5,9 @@ covers the basic usage and functionality with examples. The grader uses the Dock `prairielearn/grader-python-pytest:latest` which is powered by the `pytest-prairielearn-grader` pytest plugin. -The following discussion is based on converted example questions in PrairieLearn. -For a real example, see: https://github.com/PrairieLearn/PrairieLearn/pull/12603 +The following discussion is based on converted example questions in PrairieLearn. To convert a +question from the legacy Python grader, follow [converting.md](converting.md), which also lists +example questions to compare against. ## Editor Setup @@ -28,9 +29,18 @@ The required file structure for a PrairieLearn question using this grader is: - tests/ ├── initial_code.py (optional: starter code for students) ├── setup_code.py (optional: test setup and parameters) + ├── leading_code.py (optional: code prepended to the student's code) + ├── trailing_code.py (optional: code appended to the student's code) └── test_student.py (required: test cases) ``` +`leading_code.py` and `trailing_code.py` run in the same namespace as the student's code, so +student code can use anything that `leading_code.py` defines. The grader doesn't read +`initial_code.py`. It's only used by the file editor in `question.html`. + +`test_student.py` can't import the other files in `tests/`. To share code with the sandbox, see +[Helper Modules](#helper-modules). To read a data file in a test, use the `tests_dir` fixture. + **Important**: The file editor element in `question.html` should have `file-name="student_code.py"`. The autograder looks for `student_code.py` by default. You can customize this by setting `student_code_pattern = "your_filename.py"` at the global scope of `test_student.py`. @@ -81,6 +91,16 @@ In this example, only `A`, `b`, and `n` (specified in `names_for_user`) are acce student code. The function blocking demonstrates how to prevent students from using certain library functions. +Setup code runs in the sandbox before the student's code, with the same builtins and import +restrictions. For example, it can't call `open` or import `os`. To read a data file, list it in +[`data_files`](#data-files). Parameters that `server.py` stored with `pl.to_json` are still encoded +in `__data_params`. Decode them with `__from_server_json`, for example +`A = __from_server_json(__data_params["A"])`. + +With the `sandbox` fixture, setup code and student code run again for each test, so random values +in setup code differ between tests. To use a setup value in a test, read it in that test with +[`sandbox.query_setup`](#5-query-setup-variables). + ## Test Cases (`test_student.py`) Test cases use pytest fixtures provided by the `pytest-prairielearn-grader` package. @@ -267,6 +287,16 @@ Control execution time for individual function calls: result = sandbox.query_function("slow_computation", data, query_timeout=2.0) ``` +### 5. Query Setup Variables + +```python +x = sandbox.query_setup("x") +``` + +Retrieves a variable defined by `setup_code.py`, whether or not it's in `names_for_user`. The value +comes from the same sandbox as the student's code, so a random value matches the one the student's +code received. + **Note**: Student code must define the queried symbols, and return values must be JSON-serializable. Supported types include: `int`, `float`, `str`, `list`, `dict`, `bool`, `None`, numpy arrays, pandas DataFrames, and plots built with `MatplotlibSerializer` (see [Testing Matplotlib Plots](#testing-matplotlib-plots)). @@ -766,15 +796,16 @@ Variable queries via `sandbox.query()` do not have timeouts since they're simple #### Module-level Default Timeout -Set a default timeout for sandbox initialization in all tests in a file: +Set a default timeout for sandbox initialization in all tests in a file. This variable and the +`sandbox_timeout` marker below are ignored when the file has an `autograder_config`, so use +`ConfigObject(sandbox_timeout=...)` there instead. ```python -# At the top of test_student.py (before imports) -initialization_timeout = 2.0 # 2 second timeout for initialization - import pytest from pytest_prairielearn_grader.fixture import StudentFixture +sandbox_timeout = 2.0 # 2 second timeout for initialization + @pytest.mark.grading_data(name="Test 1", points=1) def test_with_default_timeout(sandbox: StudentFixture) -> None: @@ -1162,6 +1193,11 @@ Pinned version tags (e.g., `:v1.2.3` and `:v1.2.3-lite`) are also published alon 8. **Configure security restrictions**: Use `import_whitelist` and `builtin_whitelist` in `server.py` to control what modules and functions students can access, preventing security issues and enforcing pedagogical constraints +9. **Watch for `__main__` blocks**: Student code runs with `__name__ == "__main__"`, so code under + `if __name__ == "__main__":` runs during grading +10. **Use friendly messages that don't give away answers**: A bare `assert` or a numpy testing + assertion shows the expected value to students. Use `assert_true(condition, msg=...)` for + hidden test cases --- From 2740800fa07abb124acd3ee1c3349936964c1220 Mon Sep 17 00:00:00 2001 From: Eliot Robson Date: Tue, 10 Nov 2026 17:22:03 -0500 Subject: [PATCH 2/5] Encourage moving leading, trailing and setup code into tests when converting Co-Authored-By: Claude Opus 5.5 --- converting.md | 59 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 51 insertions(+), 8 deletions(-) diff --git a/converting.md b/converting.md index 290a2a7..364ead3 100644 --- a/converting.md +++ b/converting.md @@ -18,6 +18,45 @@ Converted questions in the PrairieLearn example course are good references: | `autograder/codeEditor` | A function question | | `annotated/MarkovChainGroupActivity/*` | Notebook workspaces, `helper_modules`, `data_files` | +## Where code belongs + +Legacy questions often run grading code next to the student's code, in `leading_code.py`, +`trailing_code.py` and `setup_code.py`. In this grader, keep as much as possible in +`test_student.py`, where each test runs it on its own and controls the feedback. A converted +question should usually have only `test_student.py`, and `setup_code.py` only when student code +needs something from it. + +When you convert a question, move each piece of code to the first place in this table that works: + +| Code | Where to put it | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Expected values and reference solutions | The test | +| Calls to the student's functions, often in `trailing_code.py` | `sandbox.query_function` in the test | +| Inputs to the student's functions, including random ones | The test, passed as arguments. Use `@pytest.mark.parametrize` for several inputs. | +| Variables that student code reads and that `server.py` randomizes | `data["params"]` in `server.py`, listed in `names_for_user`. Tests read them from the `data_json` fixture. This works for plain JSON values, not ones stored with `pl.to_json`. | +| Fixed variables that student code reads | `starting_vars` in the `ConfigObject`, listed in `names_for_user` | +| Variables that student code reads and that need Python to build | `setup_code.py`, such as numpy arrays or values decoded with `__from_server_json` | +| Blocking a library function | `setup_code.py`. To block a whole module, use `import_blacklist` instead. | +| Imports or definitions that the student's code relies on | The starter code in `initial_code.py`, so that the student sees them. Use `helper_modules` for a module the student imports. Keep `leading_code.py` only when neither works. | + +Why this matters: + +- **Trailing code runs when the sandbox starts.** If it calls a student function that raises, every + test fails with the same error, including tests of other functions. When a test calls the function + with `query_function`, only that test fails, and the message names the function. +- **Setup code runs again for each test**, in the sandbox, with the same restrictions as student + code. Random values in it change between tests, and errors in it are reported as if the student's + code failed. +- **Leading code is hidden.** The student's file only works with code they can't see, so it behaves + differently when they run it in a workspace or on their own computer. +- **Values in `data["params"]`** are the same in every test and match what `question.html` shows, + and tests can read them without starting a sandbox. + +Don't change what the student has to write to make this work. If the student's task is to set +variables at the top level, keep it that way, and read the variables with `sandbox.query`. If +students have already submitted to the question, check that removing `leading_code.py` doesn't break +those submissions when they're regraded. + ## 1. Check whether the question can be converted Read every file in the question, including `info.json`, `question.html`, `server.py` and everything in @@ -47,9 +86,9 @@ Read every file in the question, including `info.json`, `question.html`, `server | `question.html` | Change the file editor's `file-name` to `student_code.py`, or keep it and set `student_code_pattern`. | | `tests/test.py` | Replace with `tests/test_student.py`. See [Rewrite the tests](#3-rewrite-the-tests). | | `tests/ans.py` | Delete it. Compute expected values in `test_student.py`. Don't leave a reference solution in `tests/`. | -| `tests/setup_code.py` | Keep it if it's needed, and update it. See [Update the setup code](#4-update-the-setup-code). | -| `tests/leading_code.py` | Keep it. It's prepended to the student's code, as before. | -| `tests/trailing_code.py` | Keep it. It's appended to the student's code, as before. | +| `tests/setup_code.py` | Move what you can out of it, then delete it or update it. See [Update the setup code](#4-update-the-setup-code). | +| `tests/leading_code.py` | Move its code to the starter code or `helper_modules`, and delete it. See [Where code belongs](#where-code-belongs). | +| `tests/trailing_code.py` | Move its code into the tests and delete it. See [Where code belongs](#where-code-belongs). | | `tests/initial_code.py` | Keep it. The grader doesn't read it. Fix starter code that doesn't parse (see below). | | `server.py` | Usually unchanged. | @@ -197,7 +236,13 @@ Pick assertions with that in mind: ## 4. Update the setup code -Setup code runs inside the sandbox, so some things that worked in the legacy grader don't: +First, move what you can out of the setup code, as described in [Where code belongs](#where-code-belongs): + +- Values that only tests use go in `test_student.py`. +- Random variables that student code reads go in `server.py`, if they're plain JSON values. +- Fixed variables that student code reads go in `starting_vars`. + +If nothing is left, delete `setup_code.py`. Otherwise, update what's left. Setup code runs inside the sandbox, so some things that worked in the legacy grader don't: - **Parameters.** `data` isn't defined. Use `__data_params`, which is `data["params"]`. Values that `server.py` stored with `pl.to_json` arrive still encoded: decode them with @@ -226,9 +271,6 @@ params-name="names_for_user">` in `question.html`. If the question defines these la.inv = not_allowed ``` -If the setup code only defined values that tests need, and student code doesn't use them, move that -code into `test_student.py` and delete `setup_code.py`. - ## 5. Configure the grader Put a `ConfigObject` named `autograder_config` at the top of `test_student.py` when the defaults @@ -272,7 +314,8 @@ For questions where students submit a Jupyter notebook from a workspace: - Set `sandbox_timeout=15` if the notebook imports numpy or similar libraries. - Notebook questions use the normal `sandbox` fixture, not `workspace_mode`. - The grader runs the graded cells as a script, so anything a graded cell needs must be defined in - a graded cell or in `leading_code.py`. + a graded cell. Tag the cells with the notebook's imports `#grade`. Use `leading_code.py` only + when students already have copies of the notebook that you can't change. - If the notebook reads a data file, list it in `data_files`. If the notebook imports a `.py` file that's in the workspace, copy that file into `tests/` and list it in `helper_modules`. From 2c9bf4141610cded07d7f4bde9466bfad1615a62 Mon Sep 17 00:00:00 2001 From: Eliot Robson Date: Tue, 10 Nov 2026 17:34:31 -0500 Subject: [PATCH 3/5] Reorganize quick_start.md for new users and keep converting.md conversion-specific quick_start.md now starts with a first question, local testing and where code belongs, then the API reference. It fixes the list of default builtins and the examples that relied on bare asserts. converting.md links to it instead of repeating general guidance, rewrites the square example to avoid setup code, and calls out the restricted builtins. Co-Authored-By: Claude Opus 5.5 --- converting.md | 163 ++--- quick_start.md | 1670 ++++++++++++++++++++++++++---------------------- 2 files changed, 981 insertions(+), 852 deletions(-) diff --git a/converting.md b/converting.md index 364ead3..b663ddb 100644 --- a/converting.md +++ b/converting.md @@ -5,52 +5,34 @@ This guide explains how to convert a PrairieLearn question from the legacy Pytho (`prairielearn/grader-python-pytest`). It's written so that you can give it to a person or an AI coding agent and have them do the conversion from start to finish. -[quick_start.md](quick_start.md) documents every feature in detail. This guide covers what changes, -in what order to do it, and the differences that are easy to miss. +[quick_start.md](quick_start.md) shows how to write a question for this grader and documents its +full API. This guide covers only what's specific to converting: what changes, in what order to do +it, and the differences from the legacy grader that are easy to miss. Converted questions in the PrairieLearn example course are good references: -| Question (under `exampleCourse/questions/demo/`) | Shows | -| ------------------------------------------------ | ------------------------------------------------------------- | -| `autograder/python/square` | `setup_code.py`, `query_setup`, a scalar with a tolerance | -| `autograder/python/pandas` | Checking a DataFrame | -| `autograder/python/randomFunctionName` | `server.py` parameters, `data_json`, partial credit, timeouts | -| `autograder/codeEditor` | A function question | -| `annotated/MarkovChainGroupActivity/*` | Notebook workspaces, `helper_modules`, `data_files` | +| Question (under `exampleCourse/questions/demo/`) | Shows | +| ------------------------------------------------ | ------------------------------------------------------------------------------------------ | +| `autograder/python/square` | A scalar with a tolerance. It still keeps `x` in `setup_code.py`, unlike the version below | +| `autograder/python/pandas` | Checking a DataFrame | +| `autograder/python/randomFunctionName` | `server.py` parameters, `data_json`, partial credit, timeouts | +| `autograder/codeEditor` | A function question | +| `annotated/MarkovChainGroupActivity/*` | Notebook workspaces, `helper_modules`, `data_files` | ## Where code belongs Legacy questions often run grading code next to the student's code, in `leading_code.py`, -`trailing_code.py` and `setup_code.py`. In this grader, keep as much as possible in -`test_student.py`, where each test runs it on its own and controls the feedback. A converted -question should usually have only `test_student.py`, and `setup_code.py` only when student code -needs something from it. - -When you convert a question, move each piece of code to the first place in this table that works: - -| Code | Where to put it | -| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Expected values and reference solutions | The test | -| Calls to the student's functions, often in `trailing_code.py` | `sandbox.query_function` in the test | -| Inputs to the student's functions, including random ones | The test, passed as arguments. Use `@pytest.mark.parametrize` for several inputs. | -| Variables that student code reads and that `server.py` randomizes | `data["params"]` in `server.py`, listed in `names_for_user`. Tests read them from the `data_json` fixture. This works for plain JSON values, not ones stored with `pl.to_json`. | -| Fixed variables that student code reads | `starting_vars` in the `ConfigObject`, listed in `names_for_user` | -| Variables that student code reads and that need Python to build | `setup_code.py`, such as numpy arrays or values decoded with `__from_server_json` | -| Blocking a library function | `setup_code.py`. To block a whole module, use `import_blacklist` instead. | -| Imports or definitions that the student's code relies on | The starter code in `initial_code.py`, so that the student sees them. Use `helper_modules` for a module the student imports. Keep `leading_code.py` only when neither works. | - -Why this matters: - -- **Trailing code runs when the sandbox starts.** If it calls a student function that raises, every - test fails with the same error, including tests of other functions. When a test calls the function - with `query_function`, only that test fails, and the message names the function. -- **Setup code runs again for each test**, in the sandbox, with the same restrictions as student - code. Random values in it change between tests, and errors in it are reported as if the student's - code failed. -- **Leading code is hidden.** The student's file only works with code they can't see, so it behaves - differently when they run it in a workspace or on their own computer. -- **Values in `data["params"]`** are the same in every test and match what `question.html` shows, - and tests can read them without starting a sandbox. +`trailing_code.py` and `setup_code.py`. Don't carry that over. Follow +[Where Code Belongs](quick_start.md#where-code-belongs), which says where each kind of code goes in +this grader and why. A converted question should usually have only `initial_code.py` and +`test_student.py`, plus `setup_code.py` only when student code needs something from it: + +- **`ans.py`**: compute the expected values in the tests. +- **`trailing_code.py`**: call the student's functions from the tests with `query_function`. +- **`leading_code.py`**: move imports and definitions to the starter code, or to `helper_modules`. +- **`setup_code.py`**: move random plain values to `server.py`, fixed values to `starting_vars` and + values that only tests use to the tests. Keep what's left, such as numpy arrays and blocked + functions. Don't change what the student has to write to make this work. If the student's task is to set variables at the top level, keep it that way, and read the variables with `sandbox.query`. If @@ -122,10 +104,30 @@ Each `test_*` method on the legacy `Test` class becomes a `test_*` function in `tests/test_student.py`. Keep each test's name, points and order, and keep checking the same things with the same tolerances. Don't make the question easier or harder. -Here's the `square` question before and after: +Here's the `square` question, which asks students to set `x_sq` to the square of a random `x`. +The legacy version has four files in `tests/`: + +```python +# tests/setup_code.py +import random + +x = random.random() +``` + +```python +# tests/ans.py +x_sq = x**2 +``` ```python -# Legacy tests/test.py +# tests/initial_code.py +import numpy as np + +x_sq = # fill in this code here... +``` + +```python +# tests/test.py class Test(PLTestCaseWithPlot): @points(10) @name("x_sq") @@ -136,21 +138,48 @@ class Test(PLTestCaseWithPlot): Feedback.set_score(0) ``` +Converted, following [Where code belongs](#where-code-belongs): + +- `x` is a plain random value that student code reads, so `server.py` generates it instead of + `setup_code.py`. `question.html` already lists `x` in + ``, so student code still gets it. +- The expected value comes from the test, so `ans.py` is deleted. +- The starter code parses, so an unchanged submission gets 0 points with feedback. +- `question.html` changes `file-name="user_code.py"` to `file-name="student_code.py"`, and + `info.json` changes the image to `prairielearn/grader-python-pytest:lite`. + +```python +# server.py +import random + + +def generate(data): + data["params"]["x"] = random.random() +``` + +```python +# tests/initial_code.py +x_sq = 0 # fill in this code here... +``` + ```python # tests/test_student.py import pytest from pytest_prairielearn_grader.assertions import assert_approx_equal from pytest_prairielearn_grader.fixture import StudentFixture -from pytest_prairielearn_grader.utils import GradingOutputLevel + +pytestmark = pytest.mark.output(level="friendly") -@pytest.mark.output(level=GradingOutputLevel.Friendly) @pytest.mark.grading_data(name="x_sq", points=10) -def test_x_sq(sandbox: StudentFixture) -> None: - x = sandbox.query_setup("x") +def test_x_sq(sandbox: StudentFixture, data_json: dict) -> None: + x = data_json["params"]["x"] assert_approx_equal(sandbox.query("x_sq"), x**2, description="x_sq") ``` +The converted question has no `setup_code.py`. `x` is now the same in every test and every +regrade of a variant, and `question.html` could show it with `{{params.x}}`. + ### API mapping | Legacy | pytest grader | @@ -180,24 +209,13 @@ or with numpy and pandas functions wrapped in `assert_true`: | `check_scalar(name, ref, st)` | `assert_approx_equal(value, expected, description=name)`, keeping `rtol=1e-5, atol=1e-8` unless the test set others | | Exact comparisons | `assert_equal(value, expected, description=name)` | | `call_user` plus a comparison | `assert_fn_equal(sandbox, "f", args=(...), expected=...)` or `assert_fn_approx_equal` | -| `check_numpy_array_allclose(name, ref, st)` | The four array checks below | -| `check_numpy_array_features(name, ref, st)` | The first three array checks below | +| `check_numpy_array_allclose(name, ref, st)` | The four [array checks](quick_start.md#hiding-expected-values) on type, dtype, shape and values | +| `check_numpy_array_features(name, ref, st)` | The first three of those array checks | | `check_list(name, ref, st, entry_type=T)` | `assert_true(isinstance(value, list), ...)`, a length check and an `isinstance` check on each entry | | `check_tuple`, `check_dict` | `assert_equal` | | `check_dataframe(name, ref, st)` | `pandas.testing.assert_frame_equal(expected, df, check_like=True)`, as in the `pandas` example question | -For a numpy array, check the type, dtype, shape and values in that order, so that each failure has a -clear message and a wrong shape doesn't reach `np.allclose`: - -```python -value = sandbox.query("x") -assert_true(isinstance(value, np.ndarray), msg="x should be a numpy array") -assert_true(np.issubdtype(value.dtype, np.number), msg="x should contain numbers") -assert_true(value.shape == expected.shape, msg=f"x should have shape {expected.shape}") -assert_true(np.allclose(value, expected, rtol=1e-5, atol=1e-8), msg="x is inaccurate") -``` - -`assert_approx_equal` is for scalars. Don't use it for arrays. +Keep `check_scalar`'s default tolerances of `rtol=1e-5, atol=1e-8` unless the legacy test set others. ### Feedback messages @@ -205,17 +223,10 @@ Use `@pytest.mark.output(level=GradingOutputLevel.Friendly)` on each test, as th do, or `output_level="friendly"` in the `ConfigObject`. At this level the student sees only the assertion's message. -Pick assertions with that in mind: - -- **Bare `assert`.** Don't use it in tests. pytest rewrites it, so the student sees - `assert 6 == 12345` and the fixture's `repr` along with your message, which gives away the - expected value. -- **numpy and pandas assertions.** `np.testing.assert_allclose` and similar include both arrays in - their messages. Use `assert_true(np.allclose(...), msg=...)` instead. -- **`assert_equal` and the other comparison helpers.** These show the expected value. That's fine - when the legacy feedback showed it too, or when the test's inputs are visible to the student. For - hidden test cases, use `assert_true` with a message that doesn't include the answer. -- **Wording.** Keep the wording of the legacy feedback where you can, such as `"x is inaccurate"`. +Follow [Hiding Expected Values](quick_start.md#hiding-expected-values): don't use a bare `assert` +or numpy testing functions, and use `assert_true` for hidden test cases. The comparison helpers show +the expected value, which is fine when the legacy feedback showed it too. Keep the wording of the +legacy feedback where you can, such as `"x is inaccurate"`. ### Scoring @@ -256,7 +267,7 @@ params-name="names_for_user">` in `question.html`. If the question defines these else, or not at all, set `names_for_user=[...]` in the `ConfigObject`. - **Fresh values per test.** The `sandbox` fixture runs setup code and student code again for each test, so random values in setup code change between tests. Read them with `sandbox.query_setup` in - the same test that uses them, as `square` does. Don't compute expected values from random setup + the same test that uses them. Don't compute expected values from random setup values at module level. - **Disallowed functions.** `Feedback` isn't available. Define a replacement in setup code: @@ -298,6 +309,7 @@ When a `ConfigObject` is present, the module-level `sandbox_timeout` variable an | Student code or `leading_code.py` imports a module from `tests/` | `helper_modules=["module_name"]`. See [Helper Modules](quick_start.md#helper-modules). | | Student code or `leading_code.py` reads a file from `tests/` | `data_files=["file.txt"]`, and read it in tests with the `tests_dir` fixture. See [Data Files](quick_start.md#data-files). | | Student code imports `os`, `sys`, `subprocess`, `pathlib` or `shutil` | `import_blacklist=[...]` without that module. These are blocked by default. | +| Solutions use builtins that aren't defaults, such as `sum` or `super` | `builtin_whitelist=[...]`. See [Builtin Function Control](quick_start.md#builtin-function-control). | | The legacy setup code blocked imports or builtins | `import_whitelist`, `import_blacklist` or `builtin_whitelist` | | A multi-file workspace project with `.py` files | `workspace_mode=True`. See [Workspace Grading](quick_start.md#workspace-grading). | @@ -331,14 +343,19 @@ These differences from the legacy grader are easy to miss: - **Setup code runs for each test**, and so does the student's code. Output that student code prints at the top level appears in each test's feedback, unless the test sets `include_stdout_feedback=False` in `grading_data`. +- **Restricted builtins.** Student code and setup code get only a default set of builtins, which + leaves out common ones such as `sum`, `any`, `all`, `type`, `getattr`, `next` and `super`. Check + the solution and the starter code, and add what they use to `builtin_whitelist`. A missing builtin + shows up as a `NameError` when you test the conversion. - **Default import blacklist.** `os`, `sys`, `subprocess`, `pathlib` and `shutil` are blocked unless you set `import_blacklist`. - **Syntax errors** make the submission ungradable instead of scoring 0. ## 7. Test the conversion -Test the converted question with the grader image before calling the conversion done. Run each case -in a fresh copy of the layout, because the image moves files around: +Test the converted question with the grader image before calling the conversion done, as described +in [Testing Locally](quick_start.md#testing-locally). Run each case in a fresh copy of the layout, +because the image moves files around: ```bash mkdir -p grade/tests grade/student grade/data diff --git a/quick_start.md b/quick_start.md index 0bca1ad..3c4f858 100644 --- a/quick_start.md +++ b/quick_start.md @@ -1,13 +1,16 @@ # Quick Start Guide -This is a quick start guide for using the new Python autograder for PrairieLearn. This guide -covers the basic usage and functionality with examples. The grader uses the Docker image -`prairielearn/grader-python-pytest:latest` which is powered by the `pytest-prairielearn-grader` +This guide shows how to write a PrairieLearn question that's graded with the pytest-based Python +autograder, and then documents the grader's full API. The grader runs in the +`prairielearn/grader-python-pytest` Docker images and is built on the `pytest-prairielearn-grader` pytest plugin. -The following discussion is based on converted example questions in PrairieLearn. To convert a -question from the legacy Python grader, follow [converting.md](converting.md), which also lists -example questions to compare against. +Tests are ordinary pytest test functions. Each test gets a fresh sandbox that runs the student's +code, and the test asks the sandbox for the values and functions it wants to check. Student code +never runs in the same process as the tests. + +If you're converting a question that uses the legacy `prairielearn/grader-python` image, follow +[converting.md](converting.md) instead. ## Editor Setup @@ -19,155 +22,255 @@ pip install pytest pytest-prairielearn-grader This enables IDE features like autocomplete, type checking, and inline documentation when writing test cases. -## File Structure +## Your First Question -The required file structure for a PrairieLearn question using this grader is: +This section builds a question that asks students to write a function `average(values)`. A +question that's graded by this grader has these files: ``` -- info.json -- question.html -- tests/ - ├── initial_code.py (optional: starter code for students) - ├── setup_code.py (optional: test setup and parameters) - ├── leading_code.py (optional: code prepended to the student's code) - ├── trailing_code.py (optional: code appended to the student's code) - └── test_student.py (required: test cases) +average/ +├── info.json +├── question.html +└── tests/ + ├── initial_code.py (starter code shown in the file editor) + └── test_student.py (the tests) +``` + +### `info.json` + +Set `gradingMethod` to `"External"` and pick a grader image: + +```json +{ + "uuid": "...", + "title": "Average of a list", + "topic": "Functions", + "type": "v3", + "gradingMethod": "External", + "externalGradingOptions": { + "enabled": true, + "image": "prairielearn/grader-python-pytest:lite", + "timeout": 30 + } +} ``` -`leading_code.py` and `trailing_code.py` run in the same namespace as the student's code, so -student code can use anything that `leading_code.py` defines. The grader doesn't read -`initial_code.py`. It's only used by the file editor in `question.html`. +### Choosing an Image + +Two image variants are published. Pick the smallest one that includes what your question needs — a smaller image means faster cold-start pull times and a reduced security surface. + +| Tag | Size (approx.) | Includes | Best for | +| --------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| `:latest` | ~400–500 MB | Full scientific stack: `numpy`, `pandas`, `scipy`, `matplotlib` + `matplotcheck2` + `plot-serializer`, `seaborn`, `scikit-learn`, `sympy`, `networkx`, `nltk`, `nbformat`, and more | Any course that needs scientific computing, data science, plotting, or Jupyter notebook grading | +| `:lite` | ~60–80 MB | Grader core only: `pytest-prairielearn-grader`, `Pygments`, `beautifulsoup4`, `defusedxml`, `requests` | Introductory CS courses where student code uses only the Python standard library | + +> **Note:** If a student's code imports a package that isn't present in the image (e.g., `numpy` in `:lite`), it will raise an `ImportError` and the test will fail with an exception. Make sure the image you choose contains everything both your test code and the expected student solution require. + +Pinned version tags (e.g., `:v1.2.3` and `:v1.2.3-lite`) are also published alongside each release if you need a reproducible build. -`test_student.py` can't import the other files in `tests/`. To share code with the sandbox, see -[Helper Modules](#helper-modules). To read a data file in a test, use the `tests_dir` fixture. +### `question.html` -**Important**: The file editor element in `question.html` should have `file-name="student_code.py"`. -The autograder looks for `student_code.py` by default. You can customize this by setting -`student_code_pattern = "your_filename.py"` at the global scope of `test_student.py`. +Students write their code in a file editor. The grader looks for a file named `student_code*.py`, +so name the file `student_code.py`, or set [`student_code_pattern`](#configobject-parameters): -## Setup Code (`setup_code.py`) +```html + +

+ Write a function average(values) that returns the mean of a + non-empty list of numbers. +

+ + -The `setup_code.py` file defines variables and functions that are available to the student -code. Only variables listed in the `names_for_user` entry in `data.json` (via the -`pl-external-grader-variables` element) are accessible to the student. + + + + +``` -### Accessing `data.json` Parameters +### `tests/initial_code.py` -Inside `setup_code.py`, you can access parameters from `data.json` via the special `__data_params` variable, which contains the `params` dict (i.e., `data.json["params"]`): +The starter code. The grader doesn't read this file, so it's only used by the file editor. Make +sure it parses: a submission with a syntax error is [ungradable](#ungradable-submission-detection), +so an unchanged starter file should run and get 0 points with feedback. ```python -# Access parameters passed from PrairieLearn -coefficient = __data_params["coefficient"] -matrix_size = __data_params["matrix_size"] +def average(values): + return 0 ``` -### Example `setup_code.py` +### `tests/test_student.py` + +Each test is a function that takes the `sandbox` fixture, calls the student's code through it, and +checks the result: ```python -import numpy as np -import numpy.linalg as la +import pytest +from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_fn_approx_equal +from pytest_prairielearn_grader.fixture import StudentFixture +autograder_config = ConfigObject(output_level="friendly", builtin_whitelist=["sum"]) -def not_allowed(*args, **kwargs): - raise RuntimeError("Usage of this function is not allowed in this question.") +@pytest.mark.grading_data(name="Average of one number", points=1) +def test_single(sandbox: StudentFixture) -> None: + assert_fn_approx_equal(sandbox, "average", args=([5],), expected=5) -# Set up parameters -n = np.random.randint(4, 16) -# Generate a random full-rank matrix -X = la.qr(np.random.random_sample((n, n)))[0] -D = np.diag(np.random.random_sample(n) * 10 + 1) -A = X.T @ D @ X +@pytest.mark.parametrize("values", [[1, 2, 3], [2.5, -1, 0, 7], list(range(100))]) +@pytest.mark.grading_data(name="Average of several numbers", points=2) +def test_several(sandbox: StudentFixture, values: list[float]) -> None: + assert_fn_approx_equal(sandbox, "average", args=(values,), expected=sum(values) / len(values)) +``` -b = np.random.random(n) +- `grading_data` sets the name that students see and the test's points. A parametrized test gets + those points for each parameter. +- `assert_fn_approx_equal` calls `average` in the sandbox and compares the result with a + tolerance. On a failure, the student sees which call was checked, the expected value and their + value. See [Checking Answers](#checking-answers) for the other helpers and for checks that hide the + expected value. +- The `friendly` output level shows students only the assertion's message, without a traceback. +- Student code gets a restricted set of builtins that doesn't include `sum`, so this question + allows it with `builtin_whitelist`. See [Builtin Function Control](#builtin-function-control). +- Expected values are computed in the test. Don't put a reference solution in a file that student + code can read. -# Block certain functions -la.inv = not_allowed -la.pinv = not_allowed +### Testing Locally + +The grader image's entrypoint expects PrairieLearn's `/grade` layout. Build it from your question +and a solution, run the image, and read the results: + +```bash +mkdir -p grade/tests grade/student grade/data +cp -R average/tests/. grade/tests/ +cp solution.py grade/student/student_code.py +echo '{"params": {}}' > grade/data/data.json +docker run --rm -v "$PWD/grade:/grade" prairielearn/grader-python-pytest:lite +cat grade/results/results.json +``` + +The image moves files around, so make a fresh copy of the layout for each run. Put in +`data.json` whatever `params` your `server.py` and elements would set. Try at least a correct +solution, the unchanged starter code and a wrong answer for each test, and read each test's +`message` in `results.json`. + +To run pytest directly while you write tests, install the packages from [Editor Setup](#editor-setup) +and put the files that the grader reads in a folder named after the test file, next to it: + +``` +tests/ +├── test_student.py +└── test_student/ + ├── student_code_correct.py (each student_code*.py file is graded separately) + ├── student_code_wrong.py + ├── setup_code.py (if the question has one) + └── data.json (if the tests need params) ``` -In this example, only `A`, `b`, and `n` (specified in `names_for_user`) are accessible in the -student code. The function blocking demonstrates how to prevent students from using certain -library functions. +Then run `pytest tests/test_student.py`. -Setup code runs in the sandbox before the student's code, with the same builtins and import -restrictions. For example, it can't call `open` or import `os`. To read a data file, list it in -[`data_files`](#data-files). Parameters that `server.py` stored with `pl.to_json` are still encoded -in `__data_params`. Decode them with `__from_server_json`, for example -`A = __from_server_json(__data_params["A"])`. +## Where Code Belongs -With the `sandbox` fixture, setup code and student code run again for each test, so random values -in setup code differ between tests. To use a setup value in a test, read it in that test with -[`sandbox.query_setup`](#5-query-setup-variables). +Besides `test_student.py`, the `tests` folder can hold these files: -## Test Cases (`test_student.py`) +| File | What the grader does with it | +| ------------------ | ------------------------------------------------------------------------------------------------------------------- | +| `setup_code.py` | Runs in the sandbox before the student's code. See [Giving Values to Student Code](#giving-values-to-student-code). | +| `leading_code.py` | Prepended to the student's code, in the same namespace. | +| `trailing_code.py` | Appended to the student's code, in the same namespace. | +| Helper modules | Modules that student code can import, when listed in [`helper_modules`](#helper-modules). | +| Data files | Files that student code can read, when listed in [`data_files`](#data-files). | -Test cases use pytest fixtures provided by the `pytest-prairielearn-grader` package. +Keep as much as possible in `test_student.py`, where each test runs on its own and controls its +feedback. Most questions need only `initial_code.py` and `test_student.py`. Put each piece of code +in the first place in this table that works: -### Basic Example +| Code | Where to put it | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Expected values and reference solutions | The test | +| Calls to the student's functions | `sandbox.query_function` or an assertion helper in the test | +| Inputs to the student's functions, including random ones | The test, passed as arguments. Use `@pytest.mark.parametrize` for several inputs. | +| Variables that student code reads and that `server.py` randomizes | `data["params"]` in `server.py`, listed in `names_for_user`. Tests read them from the `data_json` fixture. This works for plain JSON values, not ones stored with `pl.to_json`. | +| Fixed variables that student code reads | `starting_vars` in the `ConfigObject`, listed in `names_for_user` | +| Variables that student code reads and that need Python to build | `setup_code.py`, such as numpy arrays or values decoded with `__from_server_json` | +| Blocking a library function | `setup_code.py`. To block a whole module, use `import_blacklist` instead. | +| Imports or definitions that the student's code relies on | The starter code, so that the student sees them. Use `helper_modules` for a module the student imports. Use `leading_code.py` only when neither works. | -```python -import numpy as np -import numpy.linalg as la -import pytest -from pytest_prairielearn_grader.fixture import StudentFixture +Why: +- **Trailing code runs when the sandbox starts.** If it calls a student function that raises, every + test fails with the same error, including tests of other functions. When a test calls the function + with `query_function`, only that test fails, and the message names the function. +- **Setup code runs again for each test**, in the sandbox, with the same restrictions as student + code. Random values in it change between tests, and errors in it are reported as if the student's + code failed. +- **Leading code is hidden.** The student's file only works with code they can't see, so it behaves + differently when they run it in a workspace or on their own computer. +- **Values in `data["params"]`** are the same in every test and match what `question.html` shows, + and tests can read them without starting a sandbox. -@pytest.mark.grading_data(name="x", points=1) -def test_array_all_close(sandbox: StudentFixture) -> None: - """Test that student's solution x solves the linear system.""" - correct_x = la.solve(sandbox.query("A"), sandbox.query("b")) - np.testing.assert_allclose( - sandbox.query("x"), correct_x, err_msg="x is not correct" - ) +`test_student.py` can't import the other files in `tests/`. Put helper code for the tests in +`test_student.py`, and read data files in tests with the [`tests_dir`](#available-fixtures) +fixture. + +## Writing Tests + +Tests are pytest test functions in `tests/test_student.py`. They run in the order they appear in +the file. The examples below leave out their imports, which usually are: + +```python +import pytest +from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_fn_equal, assert_true +from pytest_prairielearn_grader.fixture import FeedbackFixture, StudentFixture ``` ### Test Markers -The `@pytest.mark.grading_data` decorator specifies test metadata: +The `@pytest.mark.grading_data` decorator sets what PrairieLearn shows for the test: - `name`: Test name displayed to students - `points`: Maximum points for the test - `include_stdout_feedback`: (Optional, default=`True`) Whether to include student code's stdout in feedback -Example with stdout control: - -```python -@pytest.mark.grading_data(name="Test Output", points=2, include_stdout_feedback=True) -def test_with_output(sandbox: StudentFixture) -> None: - result = sandbox.query_function("process_data", data) - assert result == expected_value - # Student's print statements will appear in feedback -``` +`@pytest.mark.output(level=...)` sets how much of a failure students see. See +[Output Levels](#output-levels). ### Available Fixtures These fixtures are provided by the `pytest-prairielearn-grader` package: -1. **`sandbox: StudentFixture`**: Provides sandboxed access to student code. Use this to query variables - and call functions from the student's submission. +1. **`sandbox: StudentFixture`**: Runs the setup code and student code in a new sandbox for each + test. Use it to query variables and call functions from the student's submission. + +2. **`module_sandbox: StudentFixture`**: Like `sandbox`, but shares one sandbox across all tests in + the file. See [Module-Scoped Sandbox](#module-scoped-sandbox). -2. **`module_sandbox: StudentFixture`**: Similar to `sandbox`, but maintains state across all tests in - a module. Useful when you want student code initialization to persist between tests (e.g., testing - stateful classes or persistent data structures). +3. **`feedback: FeedbackFixture`**: Sets partial credit and adds feedback messages. See + [Partial Credit and Feedback](#partial-credit-and-feedback). -3. **`feedback: FeedbackFixture`**: Manages partial credit and custom feedback messages for students. +4. **`data_json: dict | None`**: The contents of PrairieLearn's `data.json`, such as + `data_json["params"]`. -4. **`data_json: DataFixture`**: Provides access to parameters from PrairieLearn's `data.json` file - (generated via `pl-external-grader-variables` and other elements). +5. **`tests_dir: Path`**: The question's tests folder. Use it to read instructor files in a test, + for example `np.loadtxt(tests_dir / "pagerank_large.txt")`, instead of building the path from + `__file__`. -5. **`tests_dir: Path`**: The question's tests folder, which holds `leading_code.py` and any data files. - Use it to read instructor files in a test, for example `np.loadtxt(tests_dir / "pagerank_large.txt")`, - instead of building the path from `__file__`. +6. **`workspace_sandbox: WorkspaceFixture`**: For multi-file workspace projects. See + [Workspace Grading](#workspace-grading). For implementation details, see the [fixture source code](https://github.com/prairielearn/grader-python-v2/blob/main/src/pytest_prairielearn_grader/fixture.py). -## Querying Student Code +### Querying Student Code The `sandbox` fixture provides methods to interact with student code: -### 1. Query Variables +#### 1. Query Variables ```python value = sandbox.query("variable_name") @@ -182,10 +285,10 @@ Raises a `NameError` if the variable doesn't exist, and a `RuntimeError` with a @pytest.mark.grading_data(name="Check Variable", points=2) def test_variable(sandbox: StudentFixture) -> None: coefficient = sandbox.query("coefficient") - assert coefficient > 0, "Coefficient must be positive" + assert_true(coefficient > 0, msg="coefficient must be positive") ``` -### 2. Query Functions +#### 2. Query Functions ```python result = sandbox.query_function("function_name", arg1, arg2, kwarg1=value1) @@ -202,27 +305,18 @@ Calls a function defined in the student code with the given arguments and **retu ```python @pytest.mark.grading_data(name="Test Function", points=5) def test_function(sandbox: StudentFixture) -> None: - # Function returns value directly result = sandbox.query_function("calculate", 10, 20) - assert result == 30, f"Expected 30, got {result}" + assert_equal(result, 30, description="calculate(10, 20)") - # With keyword arguments result = sandbox.query_function("process", x=5, y=10) - assert result == expected_value + assert_equal(result, 15, description="process(x=5, y=10)") ``` -**Error handling:** +You don't need to catch these errors: they fail the test with a message saying what went wrong. -```python -@pytest.mark.grading_data(name="Test with Error Handling", points=3) -def test_with_error_handling(sandbox: StudentFixture) -> None: - try: - result = sandbox.query_function("risky_function", data) - assert result == expected_value - except RuntimeError as e: - # The error message includes the original exception details - pytest.fail(f"Function raised an error: {e}") -``` +Each call has a 1 second timeout by default. See [Function Timeout](#4-function-timeout). When a +test calls a student function many times, use `query_function_raw` and stop after the first +`"timeout"` status, so that a slow function doesn't use up the grading job's time. **Testing for specific exceptions with `query_function_raw`:** @@ -232,22 +326,18 @@ use `query_function_raw` to inspect the response directly: ```python @pytest.mark.grading_data(name="Test Exception Handling", points=3) def test_raises_value_error(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: - """Test that student code properly validates input and raises ValueError.""" - # Call function with invalid input - use query_function_raw to get the full response response = sandbox.query_function_raw("validate_input", -1) - # Check that an exception was raised - assert response["status"] == "exception", "Function should raise an exception for negative input" + assert_true(response["status"] == "exception", msg="validate_input(-1) should raise an exception") feedback.set_score(0.5) - # Verify it's the correct exception type - assert response["exception_name"] == "ValueError", \ - f"Expected ValueError, got {response['exception_name']}" + assert_equal(response["exception_name"], "ValueError", description="the exception raised by validate_input(-1)") feedback.set_score(0.75) - # Optionally check the error message - assert "negative" in response["exception_message"].lower(), \ - "Error message should mention negative values" + assert_true( + "negative" in response["exception_message"].lower(), + msg="The error message should mention negative values", + ) feedback.set_score(1.0) ``` @@ -260,7 +350,7 @@ The `query_function_raw` response contains: - `exception_message`: The exception message string - `traceback`: Full traceback string for debugging -### 3. Get Captured Output +#### 3. Get Captured Output ```python output = sandbox.get_accumulated_stdout() @@ -275,10 +365,10 @@ Retrieves stdout captured from student code execution across all function calls. def test_output(sandbox: StudentFixture) -> None: sandbox.query_function("print_greeting", "Alice") output = sandbox.get_accumulated_stdout() - assert "Hello, Alice!" in output, "Greeting not found in output" + assert_true("Hello, Alice!" in output, msg="Greeting not found in output") ``` -### 4. Function Timeout +#### 4. Function Timeout Control execution time for individual function calls: @@ -287,7 +377,7 @@ Control execution time for individual function calls: result = sandbox.query_function("slow_computation", data, query_timeout=2.0) ``` -### 5. Query Setup Variables +#### 5. Query Setup Variables ```python x = sandbox.query_setup("x") @@ -297,909 +387,907 @@ Retrieves a variable defined by `setup_code.py`, whether or not it's in `names_f comes from the same sandbox as the student's code, so a random value matches the one the student's code received. -**Note**: Student code must define the queried symbols, and return values must be JSON-serializable. -Supported types include: `int`, `float`, `str`, `list`, `dict`, `bool`, `None`, numpy arrays, pandas DataFrames, +**Note**: Values sent back from the sandbox must be serializable. Supported types include: `int`, `float`, `str`, `list`, `dict`, `bool`, `None`, numpy arrays, pandas DataFrames, and plots built with `MatplotlibSerializer` (see [Testing Matplotlib Plots](#testing-matplotlib-plots)). -## Partial Credit and Feedback +### Checking Answers -Tests flow linearly, allowing partial credit after certain assertions pass. When an assertion -fails, the student receives the last partial credit value set before the failure. +The `pytest_prairielearn_grader.assertions` module has helpers that raise an `AssertionError` with a +structured message: -```python -@pytest.mark.grading_data(name="Multi-step Test", points=10) -def test_with_partial_credit(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: - # Check basic requirements (worth 30%) - result = sandbox.query("data_loaded") - assert result is not None, "Data must be loaded" - feedback.set_score(0.3) +``` +Checking: add(2, 3) +Expected output: 5 +Your code output: -1 +The expected and actual output do not match. +``` - # Check intermediate computation (worth 60%) - intermediate = sandbox.query("processed_data") - assert len(intermediate) > 0, "Data processing failed" - feedback.set_score(0.6) +**`assert_equal(actual, expected, *, msg=None, description=None)`** - # Check final result (worth 100%) - final = sandbox.query("final_result") - assert final == expected_value, "Final result incorrect" - feedback.set_score(1.0) +Compare two values for equality: - # Add custom feedback - feedback.add_message("Excellent work! All steps completed correctly.") +```python +assert_equal(sandbox.query("x"), 42, description="variable 'x'") ``` -**Key Methods**: - -- `feedback.set_score(fraction)`: Set partial credit (0.0 to 1.0) -- `feedback.set_score_final(fraction)`: Set final score (prevents further updates) -- `feedback.add_message(msg)`: Add custom feedback message +**`assert_approx_equal(actual, expected, *, rtol=1e-5, atol=1e-8, msg=None, description=None)`** -## Advanced Features +Compare numbers with a tolerance. It's for scalars: see below for arrays. -### Configuration with ConfigObject +```python +result = sandbox.query_function("compute_pi") +assert_approx_equal(result, 3.14159, rtol=1e-3, description="compute_pi()") +``` -For complex test scenarios, you can use the `ConfigObject` class to configure all autograder settings -in a single, type-safe, immutable object. This is especially useful when you need to override multiple -settings or want better IDE support with autocomplete and type checking. +**`assert_fn_equal(sandbox, func_name, *, args=(), kwargs=None, expected, msg=None, description=None, query_timeout=1.0)`** -**Key Features:** +Call a student function and compare its return value. The description defaults to the call, such as +`add(2, 3)`: -- **Type-safe**: All parameters are type-checked with validation -- **Immutable**: Configuration cannot be modified after creation (frozen dataclass) -- **Keyword-only**: Must use explicit parameter names for clarity -- **Comprehensive**: Supports all security and execution settings +```python +assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) +``` -#### Basic Usage +**`assert_fn_approx_equal(sandbox, func_name, *, args=(), kwargs=None, expected, rtol=1e-5, atol=1e-8, msg=None, query_timeout=1.0)`** -Import and create a `ConfigObject` at the module level in your `test_student.py`: +Same as `assert_fn_equal`, with a tolerance: ```python -from pytest_prairielearn_grader import ConfigObject -from pytest_prairielearn_grader.fixture import StudentFixture -import pytest +assert_fn_approx_equal(sandbox, "compute_area", args=(3.0,), expected=28.274, rtol=1e-2) +``` -# Module-level configuration - detected automatically by the plugin -autograder_config = ConfigObject( - sandbox_timeout=2.0, - import_whitelist=["numpy", "pandas"], - builtin_whitelist=["len", "range", "sum", "print"], - starting_vars={"coefficient": 10, "threshold": 5.0}, - names_for_user=["coefficient", "threshold"], -) +**`assert_true(condition, *, msg=None)` / `assert_false(condition, *, msg=None)`** -@pytest.mark.grading_data(name="Test with Config", points=5) -def test_with_config(sandbox: StudentFixture) -> None: - # Configuration is automatically applied - result = sandbox.query_function("process", data) - assert result == expected_value -``` +Check a condition, and show only `msg` when it fails: -#### ConfigObject Parameters +```python +assert_true(sandbox.query_function("is_valid", data), msg="is_valid() should accept valid input.") +``` -All parameters are optional with sensible defaults: +#### Hiding Expected Values -- **`sandbox_timeout`** (float, default=1.0): Timeout in seconds for sandbox initialization and operations -- **`import_whitelist`** (list[str] | None): Allowed import modules (whitelist mode) -- **`import_blacklist`** (list[str] | None): Blocked import modules (blacklist mode) -- **`builtin_whitelist`** (list[str] | None): Allowed builtin functions -- **`helper_modules`** (list[str] | None): Modules from the tests folder that can be imported (see [Helper Modules](#helper-modules)) -- **`data_files`** (list[str] | None): Files from the tests folder that code in the sandbox can read by relative path (see [Data Files](#data-files)) -- **`names_for_user`** (list[str] | None): List of variable names to inject into student code -- **`student_code_pattern`** (str, default="student_code\*.py"): Glob pattern for finding student files -- **`starting_vars`** (dict[str, Any], default={}): Dictionary of variable values to provide for injection +The comparison helpers show the expected value, which is fine when the student can see the test's +inputs. For hidden test cases, use `assert_true` with a message that doesn't include the answer. -#### Helper Modules +Don't use a bare `assert` or numpy and pandas testing functions for checks that students see. +pytest rewrites a bare `assert`, so its message includes both values, such as `assert 6 == 12345`. +`np.testing.assert_allclose` and similar include both arrays. Use +`assert_true(np.allclose(...), msg=...)` instead. -The tests folder isn't on the import path in the sandbox, and `import_whitelist` only permits a -name. To let setup code, `leading_code.py` and student code import a file from that folder, name it -in `helper_modules`: +For a numpy array, check the type, dtype, shape and values in that order, so that each failure has +a clear message and a wrong shape doesn't reach `np.allclose`: ```python -# tests/test_student.py, with the helper in tests/helper_function.py -autograder_config = ConfigObject(helper_modules=["helper_function"]) +value = sandbox.query("x") +assert_true(isinstance(value, np.ndarray), msg="x should be a numpy array") +assert_true(np.issubdtype(value.dtype, np.number), msg="x should contain numbers") +assert_true(value.shape == expected.shape, msg=f"x should have shape {expected.shape}") +assert_true(np.allclose(value, expected), msg="x is inaccurate") ``` -```python -# tests/leading_code.py, or the student's code -import helper_function as hf -``` +For a pandas DataFrame, `pandas.testing.assert_frame_equal(expected, df)` works, but its message +shows both frames. -- **Names:** each entry is a plain module name, without dots, paths or `.py`. Packages aren't supported. -- **Everything else stays out:** other files in the folder, such as `test_student.py`, still can't be - imported, even if they're in `import_whitelist`. -- **Import restrictions:** a listed module doesn't need to be in `import_whitelist`. Listing one that - `import_blacklist` also blocks is a configuration error. -- **Inside a helper module:** it runs with the same builtins and import rules as student code. For - example, it can't use `sum` or import `numpy` unless those are allowed. -- **Configuration errors:** if a listed file is missing or has a syntax error, or the module is - blacklisted, the submission is ungradable. The students don't lose an attempt, and each test's - message says what's wrong. -- **Fixtures:** it works with `sandbox` and `module_sandbox`, for `.py` and `.ipynb` submissions, and - with `workspace_sandbox`. -- **Workspace mode:** students can have several files that import each other, so a listed name is - reserved. If a student submits a `helper_function.py` or a `helper_function/` folder, the - submission fails with a message asking them to rename it and counts as an attempt. Don't give - students a copy of a helper module in their workspace. A listed module keeps the restricted builtins - that the student's own modules don't have. Dotted queries such as - `query_function("helper_function.triple")` only search the student's project, so they won't call a - helper module. +#### Output Levels -> **Warning:** Students can read everything in a helper module, so it must not contain answers. This -> setting also doesn't make the rest of the tests folder private. `importlib` isn't blacklisted by -> default, so student code can load files in that folder by path. +The output level sets how much of a failure students see: -#### Data Files +| Level | Shows to student | +| ------------- | ------------------------------------------------------------- | +| `"none"` | Exception class name only (e.g., `AssertionError`) | +| `"message"` | Exception name + first line of message _(default when unset)_ | +| `"traceback"` | Exception name + message + full traceback | +| `"friendly"` | Only the exception message text — no class name, no traceback | -In a workspace, code such as `np.loadtxt("pagerank_large.txt")` reads a file next to the notebook. To -make the same code work during grading, put the file in the tests folder and name it in `data_files`: +Use `"friendly"` with the assertion helpers. Set it on one test: ```python -# tests/test_student.py, with the data in tests/pagerank_large.txt -autograder_config = ConfigObject( - data_files=["pagerank_large.txt"], - student_code_pattern="Markov-Chains-3.ipynb", - notebook_cell_tag="#grade", -) +@pytest.mark.output(level="friendly") +@pytest.mark.grading_data(name="Test addition", points=5) +def test_add(sandbox: StudentFixture) -> None: + assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) +``` +On every test in a file, with a module-level `pytestmark`: -def test_pagerank(sandbox: StudentFixture, tests_dir: Path) -> None: - edges = np.loadtxt(tests_dir / "pagerank_large.txt") # the instructor's original - ... +```python +pytestmark = pytest.mark.output(level="friendly") ``` +Or in the `ConfigObject`: + ```python -# tests/leading_code.py, or the student's code -edges = np.loadtxt("pagerank_large.txt") +autograder_config = ConfigObject(output_level="friendly") ``` -- **Paths:** each entry is relative to the tests folder, such as `"pagerank_large.txt"` or - `"data/points.csv"`, and keeps that relative path in the sandbox. Absolute paths and `..` aren't - accepted. -- **Starting folder:** the sandbox starts in a new folder that holds only copies of the listed files. - Without `data_files`, the starting folder isn't changed. -- **Read-only copies:** the copies and their folder are read-only for student code, and a fresh copy is - made for each sandbox. Tests should read the originals through the `tests_dir` fixture. -- **Reading, not importing:** the folder isn't on the import path, so a listed `.py` file can't be - imported. Use `helper_modules` for that. -- **Configuration errors:** if a listed file is missing, the submission is ungradable. The students - don't lose an attempt, and each test's message says what's wrong. -- **Fixtures:** it works with `sandbox` and `module_sandbox`, for `.py` and `.ipynb` submissions, and - with `workspace_sandbox`. In workspace mode, relative paths resolve against the data files' folder, - not the student's workspace folder. +With the `ConfigObject` approach, individual tests can still override it with +`@pytest.mark.output(level="traceback")`. The `"message"` and `"traceback"` levels can help +advanced students, who can learn from reading tracebacks. -> **Warning:** Students can read the listed files, so they must not contain answers. +### Partial Credit and Feedback -**How variable injection works**: +Tests flow linearly, allowing partial credit after certain assertions pass. When an assertion +fails, the student receives the last partial credit value set before the failure. -- Only variables listed in `names_for_user` are injected into the student code namespace -- Values are resolved in this priority order: - 1. **Highest**: Variables defined in `setup_code.py` execution - 2. **Medium**: Values from `ConfigObject.starting_vars` - 3. **Lowest**: Values from `data.json` params -- If `names_for_user` is not specified, no variables are injected (prevents variable leaking) -- `starting_vars` provides values but does NOT automatically inject them - variables must still be listed in `names_for_user` +```python +@pytest.mark.grading_data(name="Multi-step Test", points=10) +def test_with_partial_credit(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: + # Check basic requirements (worth 30%) + result = sandbox.query("data_loaded") + assert_true(result is not None, msg="Data must be loaded") + feedback.set_score(0.3) -#### Configuration Priority + # Check intermediate computation (worth 60%) + intermediate = sandbox.query("processed_data") + assert_true(len(intermediate) > 0, msg="Data processing failed") + feedback.set_score(0.6) -When `ConfigObject` is defined, it overrides all other configuration sources: + # Check final result (worth 100%) + final = sandbox.query("final_result") + assert_true(final == expected_value, msg="Final result incorrect") + feedback.set_score(1.0) -1. **Highest priority**: `ConfigObject` (when `autograder_config` variable exists) -2. **Medium priority**: Module-level variables (e.g., `sandbox_timeout = 2.0`) -3. **Lowest priority**: `data.json` params from PrairieLearn + # Add custom feedback + feedback.add_message("Excellent work! All steps completed correctly.") +``` -```python -# This ConfigObject overrides everything -autograder_config = ConfigObject( - sandbox_timeout=3.0, # Overrides module-level timeout - import_whitelist=["numpy"], # Overrides data.json import_whitelist - starting_vars={"x": 10}, # Overrides data.json params["x"] - names_for_user=["x"], -) +**Key Methods**: -# This module-level timeout is ignored when ConfigObject is present -sandbox_timeout = 1.0 -``` +- `feedback.set_score(fraction)`: Set partial credit (0.0 to 1.0) +- `feedback.set_score_final(fraction)`: Set final score (prevents further updates) +- `feedback.add_message(msg)`: Add custom feedback message -#### Complete Example +### Capturing and Testing Output + +Control whether student code output appears in feedback: ```python -from pytest_prairielearn_grader import ConfigObject -from pytest_prairielearn_grader.fixture import StudentFixture, FeedbackFixture -import pytest - -# Comprehensive configuration -autograder_config = ConfigObject( - # Execution settings - sandbox_timeout=2.5, - - # Security: only allow scientific computing libraries - import_whitelist=["numpy", "scipy", "matplotlib"], - - # Allow specific builtins for data processing - builtin_whitelist=["len", "range", "sum", "min", "max", "sorted"], - - # Provide test data - starting_vars={ - "input_data": [1, 2, 3, 4, 5], - "multiplier": 2.5, - "threshold": 10, - }, +@pytest.mark.grading_data(name="With Output", points=2, include_stdout_feedback=True) +def test_with_output(sandbox: StudentFixture) -> None: + """Student's print statements will appear in feedback.""" + result = sandbox.query_function("process_data") + # Any print() calls in student code are captured and shown + assert_equal(result, expected) - # Explicitly list what gets injected - names_for_user=["input_data", "multiplier", "threshold"], -) -@pytest.mark.grading_data(name="Test Processing", points=10) -def test_data_processing(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: - # All configuration from ConfigObject is automatically applied +@pytest.mark.grading_data(name="Without Output", points=2, include_stdout_feedback=False) +def test_without_output(sandbox: StudentFixture) -> None: + """Student's print statements will NOT appear in feedback.""" result = sandbox.query_function("process_data") + assert_equal(result, expected) - assert len(result) > 0, "Result should not be empty" - feedback.set_score(0.5) - assert result == expected_output, "Incorrect processing result" - feedback.set_score(1.0) +@pytest.mark.grading_data(name="Manual Output Check", points=2) +def test_output_manually(sandbox: StudentFixture) -> None: + """Manually inspect and test stdout.""" + sandbox.query_function("print_greeting", "Alice") + output = sandbox.get_accumulated_stdout() + assert_true("Hello, Alice!" in output, msg="Greeting message not found in output") ``` -#### Validation +### Module-Scoped Sandbox -`ConfigObject` validates all parameters at creation time: +Use `module_sandbox` instead of `sandbox` when you want student code state to persist +across multiple tests: ```python -# ✓ Valid configuration -config = ConfigObject( - sandbox_timeout=2.0, - import_whitelist=["numpy"], -) +@pytest.mark.grading_data(name="Initialize", points=1) +def test_initialization(module_sandbox: StudentFixture) -> None: + """First test initializes state.""" + result = module_sandbox.query_function("initialize_counter") + assert_equal(result, 0) -# ✓ Valid: can combine whitelist and blacklist -config = ConfigObject( - import_whitelist=["numpy", "matplotlib"], - import_blacklist=["matplotlib.pyplot"] # Block pyplot specifically -) -# ✗ Error: timeout must be positive -config = ConfigObject(sandbox_timeout=-1.0) # ValueError +@pytest.mark.grading_data(name="Increment 1", points=1) +def test_increment_1(module_sandbox: StudentFixture) -> None: + """State persists - counter should be 1.""" + result = module_sandbox.query_function("increment_counter") + assert_equal(result, 1) -# ✗ Error: must use keyword arguments -config = ConfigObject(2.0) # TypeError -# ✗ Error: names_for_user must contain strings -config = ConfigObject(names_for_user=[123, 456]) # ValueError +@pytest.mark.grading_data(name="Increment 2", points=1) +def test_increment_2(module_sandbox: StudentFixture) -> None: + """State still persists - counter should be 2.""" + result = module_sandbox.query_function("increment_counter") + assert_equal(result, 2) ``` -### Security: Import and Builtin Restrictions +**Use Cases for `module_sandbox`**: -Control which Python modules and builtin functions students can use in their code. This is crucial for: +- Testing stateful classes or modules +- Expensive initialization that should only run once +- Testing persistent data structures (databases, file systems, etc.) +- Simulating multi-step workflows -- **Security**: Preventing access to file system, network, or system operations -- **Pedagogical constraints**: Requiring students to implement functionality from scratch -- **Resource management**: Blocking operations that consume excessive resources +**Important**: With `module_sandbox`, student code is loaded once and shared across all tests +in the module. Use regular `sandbox` for independent test execution. -#### Import Control +### Testing Matplotlib Plots -Restrict which Python modules students can import using whitelists or blacklists. +To grade a plot, student code builds it with `MatplotlibSerializer` from the `plot-serializer` package and returns the serializer. The test receives a matplotlib `Figure`. -**Import Whitelist**: Only allow specific modules: +`plot-serializer` is included in the `:latest` image but not in `:lite`. Returning a plain matplotlib `Figure` from student code isn't supported. ```python -# In server.py (PrairieLearn question) -def generate(data): - data["params"]["import_whitelist"] = ["numpy", "math", "statistics"] - # Only numpy, math, and statistics can be imported - # Attempting to import any other module raises ImportError -``` +# Student code +from plot_serializer.matplotlib.serializer import MatplotlibSerializer -**Import Blacklist**: Block specific modules while allowing all others: -```python -# In server.py -def generate(data): - data["params"]["import_blacklist"] = ["os", "subprocess", "sys"] - # os, subprocess, and sys cannot be imported - # All other modules are allowed +def create_plot(data): + serializer = MatplotlibSerializer() + fig, ax = serializer.subplots() + ax.plot(data) + ax.set_xlabel("Time") + ax.set_ylabel("Value") + ax.set_title("Data Visualization") + return serializer ``` -**Combining Whitelist and Blacklist**: Use both for fine-grained control (blacklist checked first): - ```python -# In server.py -def generate(data): - # Allow scientific libraries but block specific dangerous submodules - data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib"] - data["params"]["import_blacklist"] = ["matplotlib.pyplot"] # Block pyplot even though matplotlib is allowed - # Result: numpy, scipy, and matplotlib are allowed, but matplotlib.pyplot is blocked -``` +# Test code +import matplotlib +matplotlib.use("Agg") # Non-GUI backend for server environments -**Example student code with import whitelist:** +import pytest +from matplotlib.figure import Figure +from matplotcheck.base import PlotTester +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture -```python -# data.json has "import_whitelist": ["numpy", "math"] -import numpy as np # ✓ Allowed -import math # ✓ Allowed -from numpy import array # ✓ Allowed +@pytest.mark.grading_data(name="Plot Test", points=5) +def test_student_plot(sandbox: StudentFixture) -> None: + """Test that student creates a correct plot.""" + # Student function returns a MatplotlibSerializer, which arrives here as a Figure + plot = sandbox.query_function("create_plot", data) -import os # ✗ ImportError: Module 'os' is not allowed to be imported -import pandas # ✗ ImportError: Module 'pandas' is not allowed to be imported -``` + assert_true(isinstance(plot, Figure)) + assert_equal(len(plot.axes), 1) -**Example with import inside function:** + # Use matplotcheck for detailed plot testing + ax = plot.axes[0] + pt = PlotTester(ax) -```python -def my_function(): - import os # ✗ This will raise ImportError when the function is called - return os.getcwd() + # Check plot properties + pt.assert_plot_type("line") + pt.assert_axis_label_contains(axis="x", strings_expected=["Time"]) + pt.assert_axis_label_contains(axis="y", strings_expected=["Value"]) + pt.assert_title_contains(["Data Visualization"]) ``` -**Note**: Blacklist is checked first, then whitelist. If a whitelist is specified, only those modules (minus any in blacklist) can be imported. If only a blacklist is specified, those modules are blocked but all others are allowed. +The grader automatically serializes/deserializes: -A blacklisted module also blocks its submodules, however they're imported. With `"os"` in the blacklist, `import os.path` is blocked as well as `import os`. With `"matplotlib.pyplot"`, both `import matplotlib.pyplot` and `from matplotlib import pyplot` are blocked, but `import matplotlib` still works. +- Plots built with `MatplotlibSerializer` (received as a `Figure`) +- NumPy arrays +- Pandas DataFrames +- Standard Python types (int, float, str, list, dict, bool, None) -The blacklist only checks import statements, so it isn't a security boundary. For example, modules that student code is allowed to import can expose `os` as an attribute. +## Giving Values to Student Code -#### Builtin Function Control +Some questions give students variables to use, such as a matrix to factor. Student code only gets +the variables named in `names_for_user`. Each name's value comes from the first of these that +defines it: -Python's builtin functions (like `open()`, `eval()`, `exec()`) are automatically restricted to a safe subset. The autograder provides **only safe builtins by default**, including: +1. A variable that `setup_code.py` defines. +2. `starting_vars` in the [`ConfigObject`](#configuration-with-configobject). +3. `data["params"]` from `data.json`. -- Standard types: `int`, `float`, `str`, `bool`, `list`, `dict`, `tuple`, `set`, `frozenset` -- Type checking: `isinstance`, `issubclass`, `type` -- Iteration: `iter`, `next`, `enumerate`, `zip`, `range`, `reversed`, `sorted`, `filter`, `map` -- Math: `abs`, `min`, `max`, `sum`, `round`, `pow`, `divmod` -- Conversion: `chr`, `ord`, `bin`, `oct`, `hex`, `hash` -- Common functions: `len`, `print`, `repr`, `getattr`, `hasattr`, `format` -- Python exceptions (for proper error handling) +`names_for_user` comes from the `ConfigObject` if it's set there, as a list of strings. Otherwise +it comes from `data["params"]["names_for_user"]`, a list of `{"name": ..., "description": ..., +"type": ...}` objects, which `` in +`question.html` usually sets. Without `names_for_user`, student code gets no variables, and using +one fails with a `NameError`. `names_from_user` only displays a table in the question, and the +grader ignores it. -**Dangerous builtins automatically blocked** include: `open`, `eval`, `exec`, `compile`, `__import__`, `input`, `exit`, `quit`, and others that could compromise security. +### Parameters from `server.py` -**Builtin Whitelist**: Grant access to additional builtin functions: +When `server.py` randomizes plain values, list them in `names_for_user` and read them in tests with +the `data_json` fixture: ```python -# In server.py +# server.py +import random + + def generate(data): - data["params"]["builtin_whitelist"] = ["dict", "sorted", "enumerate"] - # Students can now use dict(), sorted(), and enumerate() in addition to safe defaults - # This is useful when you want to allow specific advanced builtins + data["params"]["n"] = random.randint(5, 10) ``` -**Example with builtin whitelist:** +```html + + + The number of terms + +``` ```python -# data.json has "builtin_whitelist": ["dict"] +# tests/test_student.py +@pytest.mark.grading_data(name="total", points=1) +def test_total(sandbox: StudentFixture, data_json: dict) -> None: + n = data_json["params"]["n"] + assert_equal(sandbox.query("total"), n * (n + 1) // 2, description="total") +``` -# Safe builtins work normally: -my_list = [1, 2, 3] # ✓ list is safe by default -length = len(my_list) # ✓ len is safe by default -print(length) # ✓ print is safe by default +### Fixed Values -# Whitelisted builtins work: -my_dict = dict(a=1, b=2) # ✓ dict is in whitelist +For a value that's the same in every variant, use `starting_vars`: -# Dangerous builtins are blocked: -f = open("file.txt") # ✗ NameError: 'open' is not defined -result = eval("1+1") # ✗ NameError: 'eval' is not defined +```python +autograder_config = ConfigObject(starting_vars={"threshold": 5.0}, names_for_user=["threshold"]) ``` -#### Combining Security Features +### Setup Code (`setup_code.py`) -You can combine import and builtin restrictions for comprehensive control: +Use setup code when a value needs Python to build, such as a numpy array, or to block library +functions. It runs in the sandbox before the student's code: ```python -# In server.py -def generate(data): - # Allow only numpy and math imports - data["params"]["import_whitelist"] = ["numpy", "math"] +import numpy as np +import numpy.linalg as la - # Allow dict builtin for student convenience - data["params"]["builtin_whitelist"] = ["dict"] - # Now students can: - # - Import numpy and math - # - Use dict() plus safe builtins - # - Cannot import other modules - # - Cannot use dangerous builtins like open(), eval() -``` +def not_allowed(*args, **kwargs): + raise RuntimeError("Usage of this function is not allowed in this question.") -#### PrairieLearn Server.py Configuration -In your PrairieLearn question's `server.py`, add these parameters during the `generate()` step: +n = np.random.randint(4, 16) -```python -import prairielearn as pl +# A random full-rank matrix +X = la.qr(np.random.random_sample((n, n)))[0] +D = np.diag(np.random.random_sample(n) * 10 + 1) +A = X.T @ D @ X +b = np.random.random(n) -def generate(data): - # Set up question parameters - data["params"]["n"] = 5 - data["params"]["A"] = pl.to_json(np.random.rand(5, 5)) +la.inv = not_allowed +la.pinv = not_allowed +``` - # Configure security restrictions - data["params"]["import_whitelist"] = ["numpy", "numpy.linalg"] - data["params"]["builtin_whitelist"] = ["dict", "sorted"] +With `names_for_user` set to `A`, `b` and `n`, student code gets those three variables, and its +`la.inv` raises an error. - # Define variables accessible to student code - data["params"]["names_for_user"] = [ - {"name": "n", "description": "Matrix dimension", "type": "integer"}, - {"name": "A", "description": "Input matrix", "type": "numpy array"} - ] -``` +- **Parameters.** `__data_params` holds `data["params"]`. Values that `server.py` stored with + `pl.to_json` are still encoded: decode them with `__from_server_json`, for example + `A = __from_server_json(__data_params["A"])`. The `prairielearn` package isn't available. +- **Restrictions.** Setup code has the same builtins and import rules as student code. For example, + it can't call `open` or import `os`. To read a data file, list it in [`data_files`](#data-files). +- **Fresh values.** With the `sandbox` fixture, setup code runs again for each test, so random + values differ between tests. Read a setup value in the test that uses it, with + [`sandbox.query_setup`](#5-query-setup-variables). -These parameters are automatically passed to the autograder via `/grade/data/data.json` and enforced during student code execution. +## Configuration -In the grader images, `run.sh` passes this path to pytest with `--data-json-path` and makes the file readable only by root, so student code can't open it. The file can contain `correct_answers`. Without that option, for example when running locally, the fixture reads `data.json` next to your test file. The sandbox still receives `params` through `__data_params`, so don't rely on values in `params` staying hidden from student code. +### Configuration with ConfigObject -#### Testing Import/Builtin Restrictions Locally +For complex test scenarios, you can use the `ConfigObject` class to configure all autograder settings +in a single, type-safe, immutable object. This is especially useful when you need to override multiple +settings or want better IDE support with autocomplete and type checking. -To test your restrictions in the test scenarios: +**Key Features:** + +- **Type-safe**: All parameters are type-checked with validation +- **Immutable**: Configuration cannot be modified after creation (frozen dataclass) +- **Keyword-only**: Must use explicit parameter names for clarity +- **Comprehensive**: Supports all security and execution settings + +#### Basic Usage + +Import and create a `ConfigObject` at the module level in your `test_student.py`: ```python -# tests/test_student.py -import pytest +from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import StudentFixture +import pytest -@pytest.mark.grading_data(name="Test with restrictions", points=5) -def test_security(sandbox: StudentFixture) -> None: - """Test that imports are properly restricted.""" - # If student code tries to import blocked modules, - # the sandbox will raise an ImportError during initialization - result = sandbox.query_function("safe_function", data) - assert result == expected -``` +# Module-level configuration - detected automatically by the plugin +autograder_config = ConfigObject( + sandbox_timeout=2.0, + import_whitelist=["numpy", "pandas"], + builtin_whitelist=["len", "range", "sum", "print"], + starting_vars={"coefficient": 10, "threshold": 5.0}, + names_for_user=["coefficient", "threshold"], +) -```python -# data.json -{ - "params": { - "import_whitelist": ["numpy", "math"], - "builtin_whitelist": ["dict"], - "names_for_user": [...] - } -} +@pytest.mark.grading_data(name="Test with Config", points=5) +def test_with_config(sandbox: StudentFixture) -> None: + # Configuration is automatically applied + result = sandbox.query_function("process", data) + assert_equal(result, expected_value) ``` -#### Common Use Cases +#### ConfigObject Parameters -**1. Force students to implement algorithms from scratch:** +All parameters are optional: + +| Parameter | Default | Description | +| -------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `sandbox_timeout` | `1.0` | Seconds allowed for starting the sandbox, which runs the setup code and student code. See [Timeout Configuration](#timeout-configuration). | +| `output_level` | `None` | Default [output level](#output-levels) for every test. | +| `names_for_user` | `None` | Variables to give student code. See [Giving Values to Student Code](#giving-values-to-student-code). | +| `starting_vars` | `{}` | Values for names in `names_for_user`. | +| `student_code_pattern` | `"student_code*.py"` | Glob pattern for the student's file. | +| `import_whitelist` | `None` | Modules that code in the sandbox may import. `None` allows all. See [Import Control](#import-control). | +| `import_blacklist` | `["os", "sys", "subprocess", "pathlib", "shutil"]` | Modules that code in the sandbox may not import. | +| `builtin_whitelist` | `None` | Builtins to allow beyond the safe defaults. See [Builtin Function Control](#builtin-function-control). | +| `helper_modules` | `None` | Modules from the tests folder that can be imported. See [Helper Modules](#helper-modules). | +| `data_files` | `None` | Files from the tests folder that code in the sandbox can read by relative path. See [Data Files](#data-files). | +| `syntax_errors_ungradable` | `True` | Whether a syntax error makes the submission ungradable. See [Ungradable Submission Detection](#ungradable-submission-detection). | +| `notebook_cell_tag` | `None` | Grade only notebook cells that start with this tag. See [Jupyter Notebook Grading](#jupyter-notebook-grading). | +| `workspace_mode` | `False` | Grade a multi-file project. See [Workspace Grading](#workspace-grading). | +| `workspace_student_dir` | `None` | The student's project folder in workspace mode. | +| `workspace_exec_entry` | `None` | A file to run at startup in workspace mode. | -```python -# Require students to implement their own sorting -data["params"]["import_whitelist"] = [] # No imports allowed -data["params"]["builtin_whitelist"] = [] # No additional builtins beyond safe defaults -# Students must implement sorting without sorted() or external libraries -``` +#### Helper Modules -**2. Allow numerical computing but block system access:** +The tests folder isn't on the import path in the sandbox, and `import_whitelist` only permits a +name. To let setup code and student code import a file from that folder, name it +in `helper_modules`: ```python -# Scientific computing course -data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib", "pandas"] -# Safe builtins are automatically enforced (no open, eval, etc.) +# tests/test_student.py, with the helper in tests/helper_function.py +autograder_config = ConfigObject(helper_modules=["helper_function"]) ``` -**3. Block dangerous operations explicitly:** - ```python -# Block system and file operations -data["params"]["import_blacklist"] = ["os", "sys", "subprocess", "pathlib", "shutil"] -# All other imports allowed, but dangerous ones explicitly blocked +# the student's code, or setup code +import helper_function as hf ``` -### Timeout Configuration - -Control execution time limits for sandbox initialization and function calls to prevent infinite loops or slow student code. - -**Important**: Timeouts apply to: - -- **Sandbox initialization** (loading and executing student code at startup) -- **Function calls** via `sandbox.query_function()` with `query_timeout` parameter +- **Names:** each entry is a plain module name, without dots, paths or `.py`. Packages aren't supported. +- **Everything else stays out:** other files in the folder, such as `test_student.py`, still can't be + imported, even if they're in `import_whitelist`. +- **Import restrictions:** a listed module doesn't need to be in `import_whitelist`. Listing one that + `import_blacklist` also blocks is a configuration error. +- **Inside a helper module:** it runs with the same builtins and import rules as student code. For + example, it can't use `sum` or import `numpy` unless those are allowed. +- **Configuration errors:** if a listed file is missing or has a syntax error, or the module is + blacklisted, the submission is ungradable. The students don't lose an attempt, and each test's + message says what's wrong. +- **Fixtures:** it works with `sandbox` and `module_sandbox`, for `.py` and `.ipynb` submissions, and + with `workspace_sandbox`. +- **Workspace mode:** students can have several files that import each other, so a listed name is + reserved. If a student submits a `helper_function.py` or a `helper_function/` folder, the + submission fails with a message asking them to rename it and counts as an attempt. Don't give + students a copy of a helper module in their workspace. A listed module keeps the restricted builtins + that the student's own modules don't have. Dotted queries such as + `query_function("helper_function.triple")` only search the student's project, so they won't call a + helper module. -Variable queries via `sandbox.query()` do not have timeouts since they're simple lookups. +> **Warning:** Students can read everything in a helper module, so it must not contain answers. This +> setting also doesn't make the rest of the tests folder private. `importlib` isn't blacklisted by +> default, so student code can load files in that folder by path. -#### Module-level Default Timeout +#### Data Files -Set a default timeout for sandbox initialization in all tests in a file. This variable and the -`sandbox_timeout` marker below are ignored when the file has an `autograder_config`, so use -`ConfigObject(sandbox_timeout=...)` there instead. +In a workspace, code such as `np.loadtxt("pagerank_large.txt")` reads a file next to the notebook. To +make the same code work during grading, put the file in the tests folder and name it in `data_files`: ```python -import pytest -from pytest_prairielearn_grader.fixture import StudentFixture +# tests/test_student.py, with the data in tests/pagerank_large.txt +autograder_config = ConfigObject( + data_files=["pagerank_large.txt"], + student_code_pattern="Markov-Chains-3.ipynb", + notebook_cell_tag="#grade", +) -sandbox_timeout = 2.0 # 2 second timeout for initialization +def test_pagerank(sandbox: StudentFixture, tests_dir: Path) -> None: + edges = np.loadtxt(tests_dir / "pagerank_large.txt") # the instructor's original + ... +``` -@pytest.mark.grading_data(name="Test 1", points=1) -def test_with_default_timeout(sandbox: StudentFixture) -> None: - # The sandbox was initialized with 2 second timeout - # Now we can safely query variables and call functions - result = sandbox.query_function("compute_result") - assert result == 5 +```python +# the student's code, or setup code +edges = np.loadtxt("pagerank_large.txt") ``` -#### Per-test Timeout Override +- **Paths:** each entry is relative to the tests folder, such as `"pagerank_large.txt"` or + `"data/points.csv"`, and keeps that relative path in the sandbox. Absolute paths and `..` aren't + accepted. +- **Starting folder:** the sandbox starts in a new folder that holds only copies of the listed files. + Without `data_files`, the starting folder isn't changed. +- **Read-only copies:** the copies and their folder are read-only for student code, and a fresh copy is + made for each sandbox. Tests should read the originals through the `tests_dir` fixture. +- **Reading, not importing:** the folder isn't on the import path, so a listed `.py` file can't be + imported. Use `helper_modules` for that. +- **Configuration errors:** if a listed file is missing, the submission is ungradable. The students + don't lose an attempt, and each test's message says what's wrong. +- **Fixtures:** it works with `sandbox` and `module_sandbox`, for `.py` and `.ipynb` submissions, and + with `workspace_sandbox`. In workspace mode, relative paths resolve against the data files' folder, + not the student's workspace folder. -Override the default initialization timeout with the `@pytest.mark.sandbox_timeout` marker: +> **Warning:** Students can read the listed files, so they must not contain answers. -```python -@pytest.mark.grading_data(name="Fast Test", points=1) -@pytest.mark.sandbox_timeout(0.5) # 0.5 second initialization timeout -def test_with_custom_timeout(sandbox: StudentFixture) -> None: - # This sandbox was initialized with 0.5 second timeout - result = sandbox.query_function("quick_computation") - assert result == 5 -``` +#### Configuration Priority -#### Per-function Timeout +When `ConfigObject` is defined, it overrides all other configuration sources: -Set timeout for individual function calls using the `query_timeout` parameter: +1. **Highest priority**: `ConfigObject` (when `autograder_config` variable exists) +2. **Medium priority**: Module-level variables (e.g., `sandbox_timeout = 2.0`) +3. **Lowest priority**: `data.json` params from PrairieLearn ```python -@pytest.mark.grading_data(name="Function Test", points=2) -def test_function_timeout(sandbox: StudentFixture) -> None: - # This specific function call has a 1 second timeout - result = sandbox.query_function("compute", data, query_timeout=1.0) - assert result == expected_value +# This ConfigObject overrides everything +autograder_config = ConfigObject( + sandbox_timeout=3.0, # Overrides module-level timeout + import_whitelist=["numpy"], # Overrides data.json import_whitelist + starting_vars={"x": 10}, # Overrides data.json params["x"] + names_for_user=["x"], +) - # This function call uses the default timeout (no per-function limit) - result2 = sandbox.query_function("another_compute", data) - assert result2 == expected_value2 +# This module-level timeout is ignored when ConfigObject is present +sandbox_timeout = 1.0 ``` -**Using ConfigObject for timeouts:** +#### Complete Example ```python from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture, FeedbackFixture +import pytest +# Comprehensive configuration autograder_config = ConfigObject( - sandbox_timeout=3.0, # Sets initialization timeout -) - -@pytest.mark.grading_data(name="Test", points=1) -def test_with_config_timeout(sandbox: StudentFixture) -> None: - # Initialized with 3 second timeout from ConfigObject - result = sandbox.query_function("process", query_timeout=1.0) - assert result == expected -``` + # Execution settings + sandbox_timeout=2.5, -### Module-Scoped Sandbox + # Security: only allow scientific computing libraries + import_whitelist=["numpy", "scipy", "matplotlib"], -Use `module_sandbox` instead of `sandbox` when you want student code state to persist -across multiple tests: + # Allow specific builtins for data processing + builtin_whitelist=["len", "range", "sum", "min", "max", "sorted"], -```python -@pytest.mark.grading_data(name="Initialize", points=1) -def test_initialization(module_sandbox: StudentFixture) -> None: - """First test initializes state.""" - result = module_sandbox.query_function("initialize_counter") - assert result == 0 + # Provide test data + starting_vars={ + "input_data": [1, 2, 3, 4, 5], + "multiplier": 2.5, + "threshold": 10, + }, + # Explicitly list what gets injected + names_for_user=["input_data", "multiplier", "threshold"], +) -@pytest.mark.grading_data(name="Increment 1", points=1) -def test_increment_1(module_sandbox: StudentFixture) -> None: - """State persists - counter should be 1.""" - result = module_sandbox.query_function("increment_counter") - assert result == 1 +@pytest.mark.grading_data(name="Test Processing", points=10) +def test_data_processing(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: + # All configuration from ConfigObject is automatically applied + result = sandbox.query_function("process_data") + assert_true(len(result) > 0, msg="Result should not be empty") + feedback.set_score(0.5) -@pytest.mark.grading_data(name="Increment 2", points=1) -def test_increment_2(module_sandbox: StudentFixture) -> None: - """State still persists - counter should be 2.""" - result = module_sandbox.query_function("increment_counter") - assert result == 2 + assert_true(result == expected_output, msg="Incorrect processing result") + feedback.set_score(1.0) ``` -**Use Cases for `module_sandbox`**: +#### Validation -- Testing stateful classes or modules -- Expensive initialization that should only run once -- Testing persistent data structures (databases, file systems, etc.) -- Simulating multi-step workflows +`ConfigObject` validates all parameters at creation time: -**Important**: With `module_sandbox`, student code is loaded once and shared across all tests -in the module. Use regular `sandbox` for independent test execution. +```python +# ✓ Valid configuration +config = ConfigObject( + sandbox_timeout=2.0, + import_whitelist=["numpy"], +) -### Capturing and Testing Output +# ✓ Valid: can combine whitelist and blacklist +config = ConfigObject( + import_whitelist=["numpy", "matplotlib"], + import_blacklist=["matplotlib.pyplot"] # Block pyplot specifically +) -Control whether student code output appears in feedback: +# ✗ Error: timeout must be positive +config = ConfigObject(sandbox_timeout=-1.0) # ValueError -```python -@pytest.mark.grading_data(name="With Output", points=2, include_stdout_feedback=True) -def test_with_output(sandbox: StudentFixture) -> None: - """Student's print statements will appear in feedback.""" - result = sandbox.query_function("process_data") - # Any print() calls in student code are captured and shown - assert result == expected +# ✗ Error: must use keyword arguments +config = ConfigObject(2.0) # TypeError +# ✗ Error: names_for_user must contain strings +config = ConfigObject(names_for_user=[123, 456]) # ValueError +``` -@pytest.mark.grading_data(name="Without Output", points=2, include_stdout_feedback=False) -def test_without_output(sandbox: StudentFixture) -> None: - """Student's print statements will NOT appear in feedback.""" - result = sandbox.query_function("process_data") - assert result == expected +### Timeout Configuration +Control execution time limits for sandbox initialization and function calls to prevent infinite loops or slow student code. -@pytest.mark.grading_data(name="Manual Output Check", points=2) -def test_output_manually(sandbox: StudentFixture) -> None: - """Manually inspect and test stdout.""" - sandbox.query_function("print_greeting", "Alice") - output = sandbox.get_accumulated_stdout() - assert "Hello, Alice!" in output, "Greeting message not found in output" -``` +**Important**: Timeouts apply to: -### Student-Friendly Feedback Mode +- **Sandbox initialization** (loading and executing student code at startup) +- **Function calls** via `sandbox.query_function()` with `query_timeout` parameter -By default, when a test fails, students see the Python exception class name, a "failed with an -exception:" prefix, and (depending on the output level) a full traceback. This is useful for -debugging but can be intimidating for introductory-level students. +Variable queries via `sandbox.query()` do not have timeouts since they're simple lookups. -The **friendly** output level suppresses all traceback noise and shows only the human-readable -assertion message. Combine it with the built-in assertion helpers to produce clean, structured -feedback like: +#### Module-level Default Timeout -``` -Checking: add(2, 3) -Expected output: 5 -Your code output: -1 -The expected and actual output do not match. +Set a default timeout for sandbox initialization in all tests in a file. This variable and the +`sandbox_timeout` marker below are ignored when the file has an `autograder_config`, so use +`ConfigObject(sandbox_timeout=...)` there instead. + +```python +import pytest +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture + +sandbox_timeout = 2.0 # 2 second timeout for initialization + + +@pytest.mark.grading_data(name="Test 1", points=1) +def test_with_default_timeout(sandbox: StudentFixture) -> None: + # The sandbox was initialized with 2 second timeout + # Now we can safely query variables and call functions + result = sandbox.query_function("compute_result") + assert_equal(result, 5) ``` -#### Enabling Friendly Output +#### Per-test Timeout Override -Apply `@pytest.mark.output(level="friendly")` to individual tests: +Override the default initialization timeout with the `@pytest.mark.sandbox_timeout` marker: ```python -import pytest -from pytest_prairielearn_grader.fixture import StudentFixture -from pytest_prairielearn_grader.assertions import assert_fn_equal, assert_equal - -@pytest.mark.output(level="friendly") -@pytest.mark.grading_data(name="Test addition", points=5) -def test_add(sandbox: StudentFixture) -> None: - assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) +@pytest.mark.grading_data(name="Fast Test", points=1) +@pytest.mark.sandbox_timeout(0.5) # 0.5 second initialization timeout +def test_with_custom_timeout(sandbox: StudentFixture) -> None: + # This sandbox was initialized with 0.5 second timeout + result = sandbox.query_function("quick_computation") + assert_equal(result, 5) ``` -Or apply it to every test in a file with a module-level `pytestmark`: +#### Per-function Timeout + +Set timeout for individual function calls using the `query_timeout` parameter: ```python -import pytest +@pytest.mark.grading_data(name="Function Test", points=2) +def test_function_timeout(sandbox: StudentFixture) -> None: + # This specific function call has a 1 second timeout + result = sandbox.query_function("compute", data, query_timeout=1.0) + assert_equal(result, expected_value) -pytestmark = pytest.mark.output(level="friendly") + # This function call uses the default timeout (no per-function limit) + result2 = sandbox.query_function("another_compute", data) + assert_equal(result2, expected_value2) ``` -Or set it globally via `ConfigObject` (recommended for intro courses): +**Using ConfigObject for timeouts:** ```python from pytest_prairielearn_grader import ConfigObject autograder_config = ConfigObject( - output_level="friendly", # All tests use friendly output by default + sandbox_timeout=3.0, # Sets initialization timeout ) -``` -With the `ConfigObject` approach, individual tests can still override with -`@pytest.mark.output(level="traceback")` if needed. +@pytest.mark.grading_data(name="Test", points=1) +def test_with_config_timeout(sandbox: StudentFixture) -> None: + # Initialized with 3 second timeout from ConfigObject + result = sandbox.query_function("process", query_timeout=1.0) + assert_equal(result, expected) +``` -**All output levels:** +### Security: Import and Builtin Restrictions -| Level | Shows to student | -|-------|-----------------| -| `"none"` | Exception class name only (e.g., `AssertionError`) | -| `"message"` | Exception name + first line of message *(default when unset)* | -| `"traceback"` | Exception name + message + full traceback | -| `"friendly"` | Only the exception message text — no class name, no traceback | +Control which Python modules and builtin functions students can use in their code. This is crucial for: -#### Assertion Helpers +- **Security**: Preventing access to file system, network, or system operations +- **Pedagogical constraints**: Requiring students to implement functionality from scratch +- **Resource management**: Blocking operations that consume excessive resources -The `pytest_prairielearn_grader.assertions` module provides helpers that produce the -structured "Checking / Expected / Your output" messages. They raise `AssertionError` -with the formatted message, which the friendly output level then shows as-is. +#### Import Control -**`assert_equal(actual, expected, *, msg=None, description=None)`** +Restrict which Python modules students can import using whitelists or blacklists. -Compare two values for equality: +**Import Whitelist**: Only allow specific modules: ```python -value = sandbox.query("x") -assert_equal(value, 42, description="variable 'x'") -# On failure: -# Checking: variable 'x' -# Expected output: 42 -# Your code output: 99 -# The expected and actual output do not match. +# In server.py (PrairieLearn question) +def generate(data): + data["params"]["import_whitelist"] = ["numpy", "math", "statistics"] + # Only numpy, math, and statistics can be imported + # Attempting to import any other module raises ImportError ``` -**`assert_approx_equal(actual, expected, *, rtol=1e-5, atol=1e-8, msg=None, description=None)`** - -Compare numeric values with tolerance: +**Import Blacklist**: Block specific modules while allowing all others: ```python -result = sandbox.query_function("compute_pi") -assert_approx_equal(result, 3.14159, rtol=1e-3, description="compute_pi()") +# In server.py +def generate(data): + data["params"]["import_blacklist"] = ["os", "subprocess", "sys"] + # os, subprocess, and sys cannot be imported + # All other modules are allowed ``` -**`assert_true(condition, *, msg=None)` / `assert_false(condition, *, msg=None)`** - -Check boolean conditions with a clean message: +**Combining Whitelist and Blacklist**: Use both for fine-grained control (blacklist checked first): ```python -result = sandbox.query_function("is_valid", data) -assert_true(result, msg="Expected is_valid() to return True for valid input.") +# In server.py +def generate(data): + # Allow scientific libraries but block specific dangerous submodules + data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib"] + data["params"]["import_blacklist"] = ["matplotlib.pyplot"] # Block pyplot even though matplotlib is allowed + # Result: numpy, scipy, and matplotlib are allowed, but matplotlib.pyplot is blocked ``` -**`assert_fn_equal(sandbox, func_name, *, args=(), kwargs=None, expected, msg=None, description=None, query_timeout=1.0)`** - -Call a student function and compare its return value — the most common helper. It -automatically generates the description from the function call: +**Example student code with import whitelist:** ```python -assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) -# On failure: -# Checking: add(2, 3) -# Expected output: 5 -# Your code output: -1 -# The expected and actual output do not match. -``` +# data.json has "import_whitelist": ["numpy", "math"] -**`assert_fn_approx_equal(sandbox, func_name, *, args=(), kwargs=None, expected, rtol=1e-5, atol=1e-8, msg=None, query_timeout=1.0)`** +import numpy as np # ✓ Allowed +import math # ✓ Allowed +from numpy import array # ✓ Allowed + +import os # ✗ ImportError: Module 'os' is not allowed to be imported +import pandas # ✗ ImportError: Module 'pandas' is not allowed to be imported +``` -Same as `assert_fn_equal` but uses approximate comparison: +**Example with import inside function:** ```python -assert_fn_approx_equal(sandbox, "compute_area", args=(3.0,), expected=28.274, rtol=1e-2) +def my_function(): + import os # ✗ This will raise ImportError when the function is called + return os.getcwd() ``` -#### Combining with Partial Credit +**Note**: Blacklist is checked first, then whitelist. If a whitelist is specified, only those modules (minus any in blacklist) can be imported. If only a blacklist is specified, those modules are blocked but all others are allowed. -Friendly mode works seamlessly with `FeedbackFixture`: +A blacklisted module also blocks its submodules, however they're imported. With `"os"` in the blacklist, `import os.path` is blocked as well as `import os`. With `"matplotlib.pyplot"`, both `import matplotlib.pyplot` and `from matplotlib import pyplot` are blocked, but `import matplotlib` still works. -```python -@pytest.mark.output(level="friendly") -@pytest.mark.grading_data(name="Multi-step", points=10) -def test_multi_step(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: - # Step 1: basic check - assert_fn_equal(sandbox, "initialize", expected=0) - feedback.set_score(0.5) +The blacklist only checks import statements, so it isn't a security boundary. For example, modules that student code is allowed to import can expose `os` as an attribute. - # Step 2: harder check — if this fails, student keeps 50% - assert_fn_equal(sandbox, "compute", args=(10,), expected=100) - feedback.set_score(1.0) -``` +#### Builtin Function Control -#### When to Use Friendly Mode +Setup code, student code and helper modules get only these builtins by default: -- **Introductory courses**: Students who haven't seen Python tracebacks benefit from simpler messages. -- **Lite image questions**: Pairs well with the `:lite` Docker image for intro CS courses. -- **Standardized feedback**: When you want all failure messages to follow a consistent format. +- Types: `bool`, `bytes`, `complex`, `dict`, `float`, `int`, `list`, `set`, `slice`, `str`, `tuple` +- Type checking: `callable`, `isinstance`, `issubclass` +- Iteration: `enumerate`, `map`, `range`, `sorted`, `zip` +- Math: `abs`, `divmod`, `max`, `min`, `pow`, `round` +- Conversion: `chr`, `hash`, `hex`, `id`, `oct`, `ord`, `repr` +- Other: `len`, `print`, `None`, `True`, `False`, and class definitions +- The standard exception classes, such as `ValueError` -For advanced courses, the default output levels (`message` or `traceback`) may be more appropriate -since students can learn from reading full tracebacks. +Many common builtins aren't in this list, including `sum`, `any`, `all`, `type`, `getattr`, +`hasattr`, `iter`, `next`, `reversed`, `filter`, `frozenset`, `format`, `object` and `super`. Student +code that uses one fails with a `NameError`, so add the ones your question's solutions need to +`builtin_whitelist`. -### Testing Matplotlib Plots +**Dangerous builtins automatically blocked** include: `open`, `eval`, `exec`, `compile`, `__import__`, `input`, `exit`, `quit`, and others that could compromise security. -To grade a plot, student code builds it with `MatplotlibSerializer` from the `plot-serializer` package and returns the serializer. The test receives a matplotlib `Figure`. +**Builtin Whitelist**: Grant access to additional builtin functions: -`plot-serializer` is included in the `:latest` image but not in `:lite`. Returning a plain matplotlib `Figure` from student code isn't supported. +```python +# In server.py +def generate(data): + data["params"]["builtin_whitelist"] = ["sum", "any", "all"] + # Students can now use sum(), any() and all() in addition to the defaults +``` + +**Example with builtin whitelist:** ```python -# Student code -from plot_serializer.matplotlib.serializer import MatplotlibSerializer +# data.json has "builtin_whitelist": ["sum"] +# Default builtins work normally: +my_list = [1, 2, 3] # ✓ list is a default +length = len(my_list) # ✓ len is a default +print(length) # ✓ print is a default -def create_plot(data): - serializer = MatplotlibSerializer() - fig, ax = serializer.subplots() - ax.plot(data) - ax.set_xlabel("Time") - ax.set_ylabel("Value") - ax.set_title("Data Visualization") - return serializer +# Whitelisted builtins work: +total = sum(my_list) # ✓ sum is in the whitelist + +# Other builtins are blocked: +found = any(my_list) # ✗ NameError: name 'any' is not defined + +# Dangerous builtins are blocked: +f = open("file.txt") # ✗ NameError: 'open' is not defined +result = eval("1+1") # ✗ NameError: 'eval' is not defined ``` +#### Combining Security Features + +You can combine import and builtin restrictions for comprehensive control: + ```python -# Test code -import matplotlib -matplotlib.use("Agg") # Non-GUI backend for server environments +# In server.py +def generate(data): + # Allow only numpy and math imports + data["params"]["import_whitelist"] = ["numpy", "math"] -import pytest -from matplotlib.figure import Figure -from matplotcheck.base import PlotTester -from pytest_prairielearn_grader.fixture import StudentFixture + # Allow sum for student convenience + data["params"]["builtin_whitelist"] = ["sum"] + # Now students can: + # - Import numpy and math + # - Use sum() plus the default builtins + # - Cannot import other modules + # - Cannot use dangerous builtins like open(), eval() +``` -@pytest.mark.grading_data(name="Plot Test", points=5) -def test_student_plot(sandbox: StudentFixture) -> None: - """Test that student creates a correct plot.""" - # Student function returns a MatplotlibSerializer, which arrives here as a Figure - plot = sandbox.query_function("create_plot", data) +#### PrairieLearn Server.py Configuration - assert isinstance(plot, Figure) - assert len(plot.axes) == 1 +In your PrairieLearn question's `server.py`, add these parameters during the `generate()` step: - # Use matplotcheck for detailed plot testing - ax = plot.axes[0] - pt = PlotTester(ax) +```python +import prairielearn as pl - # Check plot properties - pt.assert_plot_type("line") - pt.assert_axis_label_contains(axis="x", strings_expected=["Time"]) - pt.assert_axis_label_contains(axis="y", strings_expected=["Value"]) - pt.assert_title_contains(["Data Visualization"]) +def generate(data): + # Set up question parameters + data["params"]["n"] = 5 + data["params"]["A"] = pl.to_json(np.random.rand(5, 5)) + + # Configure security restrictions + data["params"]["import_whitelist"] = ["numpy", "numpy.linalg"] + data["params"]["builtin_whitelist"] = ["sum"] + + # Define variables accessible to student code + data["params"]["names_for_user"] = [ + {"name": "n", "description": "Matrix dimension", "type": "integer"}, + {"name": "A", "description": "Input matrix", "type": "numpy array"} + ] ``` -The grader automatically serializes/deserializes: +These parameters are automatically passed to the autograder via `/grade/data/data.json` and enforced during student code execution. -- Plots built with `MatplotlibSerializer` (received as a `Figure`) -- NumPy arrays -- Pandas DataFrames -- Standard Python types (int, float, str, list, dict, bool, None) +In the grader images, `run.sh` passes this path to pytest with `--data-json-path` and makes the file readable only by root, so student code can't open it. The file can contain `correct_answers`. Without that option, for example when running locally, the fixture reads `data.json` next to your test file. The sandbox still receives `params` through `__data_params`, so don't rely on values in `params` staying hidden from student code. -## More Examples +#### Testing Import/Builtin Restrictions Locally -To see more examples of what is possible in these test files, look at the test cases in -[this folder](https://github.com/prairielearn/grader-python-v2/tree/main/tests/scenario_root). Each test file is called `scenario.py`. +To test your restrictions in the test scenarios: -## PrairieLearn Configuration +```python +# tests/test_student.py +import pytest +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture -In your PrairieLearn question's `info.json`, specify the grader image: +@pytest.mark.grading_data(name="Test with restrictions", points=5) +def test_security(sandbox: StudentFixture) -> None: + """Test that imports are properly restricted.""" + # If student code tries to import blocked modules, + # the sandbox will raise an ImportError during initialization + result = sandbox.query_function("safe_function", data) + assert_equal(result, expected) +``` -```json +```python +# data.json { - "title": "Your Question Title", - "topic": "Your Topic", - "tags": ["your-tags"], - "type": "v3", - "gradingMethod": "External", - "externalGradingOptions": { - "enabled": true, - "image": "prairielearn/grader-python-pytest:latest", - "timeout": 30 + "params": { + "import_whitelist": ["numpy", "math"], + "builtin_whitelist": ["sum"], + "names_for_user": [...] } } ``` -### Choosing an Image - -Two image variants are published. Pick the smallest one that includes what your question needs — a smaller image means faster cold-start pull times and a reduced security surface. - -| Tag | Size (approx.) | Includes | Best for | -| --------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -| `:latest` | ~400–500 MB | Full scientific stack: `numpy`, `pandas`, `scipy`, `matplotlib` + `matplotcheck2` + `plot-serializer`, `seaborn`, `scikit-learn`, `sympy`, `networkx`, `nltk`, `nbformat`, and more | Any course that needs scientific computing, data science, plotting, or Jupyter notebook grading | -| `:lite` | ~60–80 MB | Grader core only: `pytest-prairielearn-grader`, `Pygments`, `beautifulsoup4`, `defusedxml`, `requests` | Introductory CS courses where student code uses only the Python standard library | +#### Common Use Cases -To use the lite image, change the `image` field: +**1. Force students to implement algorithms from scratch:** -```json -"externalGradingOptions": { - "enabled": true, - "image": "prairielearn/grader-python-pytest:lite", - "timeout": 30 -} +```python +# Require students to write their own sum +data["params"]["import_whitelist"] = [] # No imports allowed +# Don't set builtin_whitelist: sum isn't a default builtin, so students can't call it. +# sorted is a default, so this doesn't stop students from sorting with it. ``` -> **Note:** If a student's code imports a package that isn't present in the image (e.g., `numpy` in `:lite`), it will raise an `ImportError` and the test will fail with an exception. Make sure the image you choose contains everything both your test code and the expected student solution require. - -Pinned version tags (e.g., `:v1.2.3` and `:v1.2.3-lite`) are also published alongside each release if you need a reproducible build. +**2. Allow numerical computing but block system access:** -## Tips and Best Practices +```python +# Scientific computing course +data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib", "pandas"] +# Safe builtins are automatically enforced (no open, eval, etc.) +``` -1. **Use descriptive test names**: The `name` in `@pytest.mark.grading_data` is shown to students -2. **Provide clear error messages**: Use informative assertion messages to guide students -3. **Test incrementally**: Break complex problems into smaller tests with partial credit -4. **Control output visibility**: Use `include_stdout_feedback=False` for tests where student output - would be confusing or reveal answers -5. **Set appropriate timeouts**: Prevent infinite loops while allowing reasonable execution time -6. **Use `module_sandbox` sparingly**: Only when you need persistent state across tests -7. **Block prohibited functions**: Use the setup code to prevent students from using disallowed functions - (like in the `la.inv = not_allowed` example above) -8. **Configure security restrictions**: Use `import_whitelist` and `builtin_whitelist` in `server.py` - to control what modules and functions students can access, preventing security issues and - enforcing pedagogical constraints -9. **Watch for `__main__` blocks**: Student code runs with `__name__ == "__main__"`, so code under - `if __name__ == "__main__":` runs during grading -10. **Use friendly messages that don't give away answers**: A bare `assert` or a numpy testing - assertion shows the expected value to students. Use `assert_true(condition, msg=...)` for - hidden test cases +**3. Block dangerous operations explicitly:** ---- +```python +# Block system and file operations +data["params"]["import_blacklist"] = ["os", "sys", "subprocess", "pathlib", "shutil"] +# All other imports allowed, but dangerous ones explicitly blocked +``` ## Ungradable Submission Detection @@ -1207,7 +1295,7 @@ When student code has a **SyntaxError** (e.g., missing colons, unmatched bracket the grader automatically marks the submission as **ungradable**. This means: - The student does **not** lose a grading attempt -- They see a clear message: *"Your code could not be parsed. Please fix the syntax errors and resubmit."* +- They see a clear message: _"Your code could not be parsed. Please fix the syntax errors and resubmit."_ - The `format_errors` field in results contains the specific error This is especially useful for introductory courses where students may accidentally submit code @@ -1251,8 +1339,6 @@ The grader also detects when **test files themselves** fail to collect (e.g., im your test code, missing dependencies). These are marked as ungradable with a message directing course staff to check the logs — students are never penalized for grader bugs. ---- - ## Jupyter Notebook Grading PrairieLearn supports JupyterLab as a workspace environment @@ -1367,6 +1453,7 @@ import pytest import numpy as np from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import FeedbackFixture, StudentFixture autograder_config = ConfigObject( @@ -1386,7 +1473,7 @@ def test_solve(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: feedback.set_score(0.5) expected = np.linalg.solve(A, b) - assert np.allclose(result, expected), f"Expected {expected}, got {result}" + assert_true(np.allclose(result, expected), msg="solve(A, b) is inaccurate") feedback.set_score(1.0) @@ -1394,7 +1481,7 @@ def test_solve(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: def test_result_variable(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: feedback.set_score(0.0) x = sandbox.query("x") - assert x is not None, "Variable 'x' was not defined" + assert_true(x is not None, msg="Variable 'x' was not defined") feedback.set_score(1.0) ``` @@ -1420,8 +1507,6 @@ questions/my_notebook_question/ └── notebook.ipynb # local stand-in for the student's submission ``` ---- - ## Workspace Grading PrairieLearn workspace questions give students a full IDE (VS Code, JupyterLab, etc.) running in a @@ -1516,6 +1601,7 @@ Use `workspace_sandbox` and query student code with **dotted module paths**: ```python from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import WorkspaceFixture import pytest @@ -1527,19 +1613,19 @@ autograder_config = ConfigObject( @pytest.mark.grading_data(name="Add two numbers", points=2) def test_add(workspace_sandbox: WorkspaceFixture) -> None: result = workspace_sandbox.query_function("calculator.add", 3, 4) - assert result == 7 + assert_equal(result, 7) @pytest.mark.grading_data(name="Clamp from sub-package", points=2) def test_clamp(workspace_sandbox: WorkspaceFixture) -> None: result = workspace_sandbox.query_function("utils.helpers.clamp", 15, 0, 10) - assert result == 10 + assert_equal(result, 10) @pytest.mark.grading_data(name="Module-level constant", points=1) def test_constant(workspace_sandbox: WorkspaceFixture) -> None: pi = workspace_sandbox.query("calculator.PI_APPROX") - assert abs(pi - 3.14159) < 1e-5 + assert_true(abs(pi - 3.14159) < 1e-5) ``` The dotted path is split on the **last dot**: `"calculator.add"` imports the module `calculator` @@ -1564,8 +1650,8 @@ result = workspace_sandbox.query_function("models.Classifier.predict", X_test) # query_function_raw — inspects the full response without raising response = workspace_sandbox.query_function_raw("calculator.divide", 1, 0) -assert response["status"] == "exception" -assert response["exception_name"] == "ValueError" +assert_equal(response["status"], "exception") +assert_equal(response["exception_name"], "ValueError") ``` ### Optional: Entry-Point Execution @@ -1583,7 +1669,7 @@ autograder_config = ConfigObject( # Now the sandbox namespace includes everything main.py defined at module level def test_global_state(workspace_sandbox: WorkspaceFixture) -> None: db = workspace_sandbox.query("database") # flat name - set by main.py via exec - assert db is not None + assert_true(db is not None) ``` Without `workspace_exec_entry` (the default), no file is executed at startup and all access @@ -1672,6 +1758,7 @@ autograder_config = ConfigObject( import pytest from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import FeedbackFixture, WorkspaceFixture autograder_config = ConfigObject( @@ -1686,30 +1773,31 @@ def test_add(workspace_sandbox: WorkspaceFixture, feedback: FeedbackFixture) -> feedback.set_score(0.0) response = workspace_sandbox.query_function_raw("calculator.add", 2, 3) - assert response["status"] != "not_found", "calculator.add is not defined" + assert_true(response["status"] != "not_found", msg="calculator.add is not defined") feedback.set_score(0.5) - assert response["status"] == "success", ( - f"calculator.add raised {response['exception_name']}: {response['exception_message']}" + assert_true( + response["status"] == "success", + msg=f"calculator.add raised {response['exception_name']}: {response['exception_message']}", ) feedback.set_score(0.8) - assert response["value"] == 5, f"Expected 5, got {response['value']}" + assert_equal(response["value"], 5, description="calculator.add(2, 3)") feedback.set_score(1.0) @pytest.mark.grading_data(name="divide() raises ValueError on zero", points=2) def test_divide_guard(workspace_sandbox: WorkspaceFixture) -> None: response = workspace_sandbox.query_function_raw("calculator.divide", 10, 0) - assert response["status"] == "exception", "divide() should raise on zero denominator" - assert response["exception_name"] == "ValueError" + assert_true(response["status"] == "exception", msg="divide() should raise on zero denominator") + assert_equal(response["exception_name"], "ValueError") @pytest.mark.grading_data(name="PI_APPROX constant exists", points=1) def test_constant(workspace_sandbox: WorkspaceFixture) -> None: pi = workspace_sandbox.query("calculator.PI_APPROX") - assert isinstance(pi, float), "PI_APPROX should be a float" - assert abs(pi - 3.14) < 0.01 + assert_true(isinstance(pi, float), msg="PI_APPROX should be a float") + assert_true(abs(pi - 3.14) < 0.01) ``` **Student workspace (`student/calculator.py`):** @@ -1725,3 +1813,27 @@ def divide(a, b): raise ValueError("Cannot divide by zero") return a / b ``` + +## More Examples + +To see more examples of what is possible in these test files, look at the test cases in +[this folder](https://github.com/prairielearn/grader-python-v2/tree/main/tests/scenario_root). Each test file is called `scenario.py`. + +## Tips and Best Practices + +1. **Keep code in the tests**: See [Where Code Belongs](#where-code-belongs). +2. **Use descriptive test names**: The `name` in `@pytest.mark.grading_data` is shown to students. +3. **Use friendly messages that don't give away answers**: Use the assertion helpers with the + `friendly` output level, and `assert_true` for hidden test cases. +4. **Test incrementally**: Break complex problems into smaller tests with partial credit. +5. **Prefer `parametrize`**: Use `@pytest.mark.parametrize` for several inputs instead of a loop, + unless the inputs share partial credit. +6. **Control output visibility**: Use `include_stdout_feedback=False` for tests where student output + would be confusing or reveal answers. +7. **Set appropriate timeouts**: Student code that imports numpy or pandas usually needs a + `sandbox_timeout` of several seconds. +8. **Use `module_sandbox` sparingly**: Only when you need persistent state across tests. +9. **Watch for `__main__` blocks**: Student code runs with `__name__ == "__main__"`, so code under + `if __name__ == "__main__":` runs during grading. +10. **Configure security restrictions**: Use `import_whitelist`, `import_blacklist` and + `builtin_whitelist` to control what modules and functions students can access. From 1f8aff329926d80def08fb08919ad7d3e831f235 Mon Sep 17 00:00:00 2001 From: Eliot Robson Date: Tue, 10 Nov 2026 17:56:51 -0500 Subject: [PATCH 4/5] Recommend import_blacklist for blocking library functions Co-Authored-By: Claude Opus 5.5 --- converting.md | 52 +++++++++++++++++++++++---------------------- quick_start.md | 57 +++++++++++++++++++++++++++++++++++++------------- 2 files changed, 69 insertions(+), 40 deletions(-) diff --git a/converting.md b/converting.md index b663ddb..7601437 100644 --- a/converting.md +++ b/converting.md @@ -182,22 +182,22 @@ regrade of a variant, and `question.html` could show it with `{{params.x}}`. ### API mapping -| Legacy | pytest grader | -| -------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -| `class Test(PLTestCase)` with `test_*` methods | Module-level `test_*` functions that take the `sandbox` fixture | -| `@points(10)` and `@name("...")` | `@pytest.mark.grading_data(name="...", points=10)` | -| Tests run in sorted name order | Tests run in file order | -| `self.st.x` | `sandbox.query("x")` | -| `Feedback.call_user(self.st.f, a, b)` | `sandbox.query_function("f", a, b)`, or `query_function_raw` to inspect exceptions and timeouts | -| `self.ref.x` from `ans.py` | Compute the expected value in the test | -| A setup variable, such as `x` from `setup_code.py` | `sandbox.query_setup("x")` | -| `self.data["params"]["k"]` | The `data_json` fixture: `data_json["params"]["k"]` | -| `Feedback.set_score(s)` | `feedback.set_score(s)` with the `feedback` fixture. See [Scoring](#scoring). | -| `Feedback.add_feedback("...")` | `feedback.add_message("...")` | -| `accuracy_critical=True` | A failed assertion ends the test. Other tests still run. | -| `total_iters = 3` and `@not_repeated` | `@pytest.mark.parametrize` over the inputs, or a loop with partial credit | -| `Feedback.not_allowed` | Define your own function in `setup_code.py`. See [Update the setup code](#4-update-the-setup-code). | -| `repeated_setup()` | Not needed: setup code runs again for each sandbox. | +| Legacy | pytest grader | +| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `class Test(PLTestCase)` with `test_*` methods | Module-level `test_*` functions that take the `sandbox` fixture | +| `@points(10)` and `@name("...")` | `@pytest.mark.grading_data(name="...", points=10)` | +| Tests run in sorted name order | Tests run in file order | +| `self.st.x` | `sandbox.query("x")` | +| `Feedback.call_user(self.st.f, a, b)` | `sandbox.query_function("f", a, b)`, or `query_function_raw` to inspect exceptions and timeouts | +| `self.ref.x` from `ans.py` | Compute the expected value in the test | +| A setup variable, such as `x` from `setup_code.py` | `sandbox.query_setup("x")` | +| `self.data["params"]["k"]` | The `data_json` fixture: `data_json["params"]["k"]` | +| `Feedback.set_score(s)` | `feedback.set_score(s)` with the `feedback` fixture. See [Scoring](#scoring). | +| `Feedback.add_feedback("...")` | `feedback.add_message("...")` | +| `accuracy_critical=True` | A failed assertion ends the test. Other tests still run. | +| `total_iters = 3` and `@not_repeated` | `@pytest.mark.parametrize` over the inputs, or a loop with partial credit | +| `Feedback.not_allowed` | `import_blacklist=["numpy.linalg.inv"]` in the `ConfigObject`. See [Update the setup code](#4-update-the-setup-code). | +| `repeated_setup()` | Not needed: setup code runs again for each sandbox. | ### Checks @@ -252,6 +252,7 @@ First, move what you can out of the setup code, as described in [Where code belo - Values that only tests use go in `test_student.py`. - Random variables that student code reads go in `server.py`, if they're plain JSON values. - Fixed variables that student code reads go in `starting_vars`. +- Functions that the setup code blocks, such as `la.inv = not_allowed`, go in `import_blacklist`. If nothing is left, delete `setup_code.py`. Otherwise, update what's left. Setup code runs inside the sandbox, so some things that worked in the legacy grader don't: @@ -269,19 +270,20 @@ params-name="names_for_user">` in `question.html`. If the question defines these test, so random values in setup code change between tests. Read them with `sandbox.query_setup` in the same test that uses them. Don't compute expected values from random setup values at module level. -- **Disallowed functions.** `Feedback` isn't available. Define a replacement in setup code: +- **Disallowed functions.** `Feedback.not_allowed` isn't available. Instead of overriding the + function in setup code, list it in `import_blacklist`, which blocks it however student code + reaches it. The default blacklist is replaced, so keep its entries: ```python - import numpy.linalg as la - - - def not_allowed(*args, **kwargs): - raise RuntimeError("Usage of this function is not allowed in this question.") - - - la.inv = not_allowed + autograder_config = ConfigObject( + import_blacklist=["os", "sys", "subprocess", "pathlib", "shutil", "numpy.linalg.inv"], + sandbox_timeout=15, + ) ``` + See [Blocking Functions](quick_start.md#blocking-functions). If the setup code computed a value + with the blocked function, compute it in the test instead. + ## 5. Configure the grader Put a `ConfigObject` named `autograder_config` at the top of `test_student.py` when the defaults diff --git a/quick_start.md b/quick_start.md index 3c4f858..593fc95 100644 --- a/quick_start.md +++ b/quick_start.md @@ -198,7 +198,7 @@ in the first place in this table that works: | Variables that student code reads and that `server.py` randomizes | `data["params"]` in `server.py`, listed in `names_for_user`. Tests read them from the `data_json` fixture. This works for plain JSON values, not ones stored with `pl.to_json`. | | Fixed variables that student code reads | `starting_vars` in the `ConfigObject`, listed in `names_for_user` | | Variables that student code reads and that need Python to build | `setup_code.py`, such as numpy arrays or values decoded with `__from_server_json` | -| Blocking a library function | `setup_code.py`. To block a whole module, use `import_blacklist` instead. | +| Blocking a library function | [`import_blacklist`](#blocking-functions), such as `["numpy.linalg.inv"]` | | Imports or definitions that the student's code relies on | The starter code, so that the student sees them. Use `helper_modules` for a module the student imports. Use `leading_code.py` only when neither works. | Why: @@ -718,18 +718,13 @@ autograder_config = ConfigObject(starting_vars={"threshold": 5.0}, names_for_use ### Setup Code (`setup_code.py`) -Use setup code when a value needs Python to build, such as a numpy array, or to block library -functions. It runs in the sandbox before the student's code: +Use setup code when a value needs Python to build, such as a numpy array. It runs in the sandbox +before the student's code: ```python import numpy as np import numpy.linalg as la - -def not_allowed(*args, **kwargs): - raise RuntimeError("Usage of this function is not allowed in this question.") - - n = np.random.randint(4, 16) # A random full-rank matrix @@ -737,13 +732,11 @@ X = la.qr(np.random.random_sample((n, n)))[0] D = np.diag(np.random.random_sample(n) * 10 + 1) A = X.T @ D @ X b = np.random.random(n) - -la.inv = not_allowed -la.pinv = not_allowed ``` -With `names_for_user` set to `A`, `b` and `n`, student code gets those three variables, and its -`la.inv` raises an error. +With `names_for_user` set to `A`, `b` and `n`, student code gets those three variables. To stop +students from calling a library function such as `la.inv`, use +[`import_blacklist`](#blocking-functions) rather than setup code. - **Parameters.** `__data_params` holds `data["params"]`. Values that `server.py` stored with `pl.to_json` are still encoded: decode them with `__from_server_json`, for example @@ -807,7 +800,7 @@ All parameters are optional: | `starting_vars` | `{}` | Values for names in `names_for_user`. | | `student_code_pattern` | `"student_code*.py"` | Glob pattern for the student's file. | | `import_whitelist` | `None` | Modules that code in the sandbox may import. `None` allows all. See [Import Control](#import-control). | -| `import_blacklist` | `["os", "sys", "subprocess", "pathlib", "shutil"]` | Modules that code in the sandbox may not import. | +| `import_blacklist` | `["os", "sys", "subprocess", "pathlib", "shutil"]` | Modules and functions that code in the sandbox may not use. See [Blocking Functions](#blocking-functions). | | `builtin_whitelist` | `None` | Builtins to allow beyond the safe defaults. See [Builtin Function Control](#builtin-function-control). | | `helper_modules` | `None` | Modules from the tests folder that can be imported. See [Helper Modules](#helper-modules). | | `data_files` | `None` | Files from the tests folder that code in the sandbox can read by relative path. See [Data Files](#data-files). | @@ -1136,6 +1129,40 @@ A blacklisted module also blocks its submodules, however they're imported. With The blacklist only checks import statements, so it isn't a security boundary. For example, modules that student code is allowed to import can expose `os` as an attribute. +#### Blocking Functions + +To stop students from using one library function, such as `numpy.linalg.inv` in a question about +solving linear systems, add the function to `import_blacklist`: + +```python +autograder_config = ConfigObject( + import_blacklist=["os", "sys", "subprocess", "pathlib", "shutil", "numpy.linalg.inv", "numpy.linalg.pinv"], + sandbox_timeout=15, +) +``` + +Setting `import_blacklist` replaces the default list, so keep its entries. + +Before setup code runs, the sandbox replaces each listed function in its module with one that +raises an error. So `np.linalg.inv(A)` and `la.inv(A)` raise +`RuntimeError: 'numpy.linalg.inv' is blacklisted and cannot be used.`, and +`from numpy.linalg import inv` raises an `ImportError`. This is the preferred way to block a +function: don't override it in setup code. + +- **Which entries:** an entry is treated as a function when the part before its last dot is a module + that can be imported and the last part is a callable that isn't a module. Other entries, such as + `"os"` or `"matplotlib.pyplot"`, block imports as described above. If the module isn't installed + in the image, the entry does nothing. +- **Setup code and helper modules** can't call the function either, because they share the + sandbox's modules. Compute values that need it in the test instead. +- **Startup time:** the function's module is imported when the sandbox starts, even if the student's + code doesn't use it. +- **Not a security boundary:** the function can still be reached under another name, such as + numpy's internal `numpy.linalg._linalg.inv`. + +> **Note:** blocking functions needs a grader release that includes +> [PrairieLearn/grader-python-v2#39](https://github.com/PrairieLearn/grader-python-v2/pull/39). + #### Builtin Function Control Setup code, student code and helper modules get only these builtins by default: @@ -1703,7 +1730,7 @@ imported normally. The list of loaded modules can change between Python versions | `workspace_exec_entry` | `str \| None` | `None` | Relative path (from workspace root) to an entry-point file to `exec` at startup. | | `sandbox_timeout` | `float` | `1.0` | Timeout for sandbox startup (increase for larger projects). | | `import_whitelist` | `list[str] \| None` | `None` | Allowed imports. `None` = allow all. | -| `import_blacklist` | `list[str] \| None` | default list | Blocked imports. | +| `import_blacklist` | `list[str] \| None` | default list | Blocked imports and functions. | | `helper_modules` | `list[str] \| None` | `None` | Modules from the tests folder to make importable. See [Helper Modules](#helper-modules). | ### Production vs. Local Development From f6b0b193c11aec7f5991d370aec501aea9a373ad Mon Sep 17 00:00:00 2001 From: Eliot Robson Date: Wed, 11 Nov 2026 00:56:15 -0500 Subject: [PATCH 5/5] Describe the default builtins from #40 Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- converting.md | 11 +++++----- quick_start.md | 58 +++++++++++++++++++++++++------------------------- 3 files changed, 36 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index b89460a..5fc6ee5 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ from pytest_prairielearn_grader import ConfigObject autograder_config = ConfigObject( sandbox_timeout=2.0, import_whitelist=["numpy", "math"], - builtin_whitelist=["len", "range", "sum"], + builtin_whitelist=["getattr"], output_level="friendly", syntax_errors_ungradable=True, starting_vars={"coefficient": 10}, diff --git a/converting.md b/converting.md index 7601437..d0b8bfd 100644 --- a/converting.md +++ b/converting.md @@ -311,7 +311,7 @@ When a `ConfigObject` is present, the module-level `sandbox_timeout` variable an | Student code or `leading_code.py` imports a module from `tests/` | `helper_modules=["module_name"]`. See [Helper Modules](quick_start.md#helper-modules). | | Student code or `leading_code.py` reads a file from `tests/` | `data_files=["file.txt"]`, and read it in tests with the `tests_dir` fixture. See [Data Files](quick_start.md#data-files). | | Student code imports `os`, `sys`, `subprocess`, `pathlib` or `shutil` | `import_blacklist=[...]` without that module. These are blocked by default. | -| Solutions use builtins that aren't defaults, such as `sum` or `super` | `builtin_whitelist=[...]`. See [Builtin Function Control](quick_start.md#builtin-function-control). | +| Solutions use builtins that aren't defaults, such as `getattr` | `builtin_whitelist=[...]`. See [Builtin Function Control](quick_start.md#builtin-function-control). | | The legacy setup code blocked imports or builtins | `import_whitelist`, `import_blacklist` or `builtin_whitelist` | | A multi-file workspace project with `.py` files | `workspace_mode=True`. See [Workspace Grading](quick_start.md#workspace-grading). | @@ -345,10 +345,11 @@ These differences from the legacy grader are easy to miss: - **Setup code runs for each test**, and so does the student's code. Output that student code prints at the top level appears in each test's feedback, unless the test sets `include_stdout_feedback=False` in `grading_data`. -- **Restricted builtins.** Student code and setup code get only a default set of builtins, which - leaves out common ones such as `sum`, `any`, `all`, `type`, `getattr`, `next` and `super`. Check - the solution and the starter code, and add what they use to `builtin_whitelist`. A missing builtin - shows up as a `NameError` when you test the conversion. +- **Restricted builtins.** Student code and setup code get only a default set of builtins. It + includes common ones such as `sum`, `any` and `super`, but not `getattr`, `hasattr`, `setattr`, + `vars`, `dir`, `globals`, `open`, `eval` or `exec`. Check the solution and the starter code, and + add what they use to `builtin_whitelist`. A missing builtin shows up as a `NameError` when you + test the conversion. - **Default import blacklist.** `os`, `sys`, `subprocess`, `pathlib` and `shutil` are blocked unless you set `import_blacklist`. - **Syntax errors** make the submission ungradable instead of scoring 0. diff --git a/quick_start.md b/quick_start.md index 593fc95..2be11a7 100644 --- a/quick_start.md +++ b/quick_start.md @@ -114,7 +114,7 @@ from pytest_prairielearn_grader import ConfigObject from pytest_prairielearn_grader.assertions import assert_fn_approx_equal from pytest_prairielearn_grader.fixture import StudentFixture -autograder_config = ConfigObject(output_level="friendly", builtin_whitelist=["sum"]) +autograder_config = ConfigObject(output_level="friendly") @pytest.mark.grading_data(name="Average of one number", points=1) @@ -135,8 +135,6 @@ def test_several(sandbox: StudentFixture, values: list[float]) -> None: value. See [Checking Answers](#checking-answers) for the other helpers and for checks that hide the expected value. - The `friendly` output level shows students only the assertion's message, without a traceback. -- Student code gets a restricted set of builtins that doesn't include `sum`, so this question - allows it with `builtin_whitelist`. See [Builtin Function Control](#builtin-function-control). - Expected values are computed in the test. Don't put a reference solution in a file that student code can read. @@ -776,7 +774,7 @@ import pytest autograder_config = ConfigObject( sandbox_timeout=2.0, import_whitelist=["numpy", "pandas"], - builtin_whitelist=["len", "range", "sum", "print"], + builtin_whitelist=["getattr", "hasattr"], starting_vars={"coefficient": 10, "threshold": 5.0}, names_for_user=["coefficient", "threshold"], ) @@ -832,7 +830,7 @@ import helper_function as hf - **Import restrictions:** a listed module doesn't need to be in `import_whitelist`. Listing one that `import_blacklist` also blocks is a configuration error. - **Inside a helper module:** it runs with the same builtins and import rules as student code. For - example, it can't use `sum` or import `numpy` unless those are allowed. + example, it can't use `getattr` or import `numpy` unless those are allowed. - **Configuration errors:** if a listed file is missing or has a syntax error, or the module is blacklisted, the submission is ungradable. The students don't lose an attempt, and each test's message says what's wrong. @@ -928,8 +926,8 @@ autograder_config = ConfigObject( # Security: only allow scientific computing libraries import_whitelist=["numpy", "scipy", "matplotlib"], - # Allow specific builtins for data processing - builtin_whitelist=["len", "range", "sum", "min", "max", "sorted"], + # Allow builtins that the defaults leave out + builtin_whitelist=["getattr", "hasattr"], # Provide test data starting_vars={ @@ -1167,18 +1165,21 @@ function: don't override it in setup code. Setup code, student code and helper modules get only these builtins by default: -- Types: `bool`, `bytes`, `complex`, `dict`, `float`, `int`, `list`, `set`, `slice`, `str`, `tuple` +- Types: `bool`, `bytearray`, `bytes`, `complex`, `dict`, `float`, `frozenset`, `int`, `list`, + `object`, `set`, `slice`, `str`, `tuple`, `type` - Type checking: `callable`, `isinstance`, `issubclass` -- Iteration: `enumerate`, `map`, `range`, `sorted`, `zip` -- Math: `abs`, `divmod`, `max`, `min`, `pow`, `round` -- Conversion: `chr`, `hash`, `hex`, `id`, `oct`, `ord`, `repr` -- Other: `len`, `print`, `None`, `True`, `False`, and class definitions +- Iteration: `all`, `any`, `enumerate`, `filter`, `iter`, `map`, `next`, `range`, `reversed`, + `sorted`, `zip` +- Math: `abs`, `divmod`, `max`, `min`, `pow`, `round`, `sum` +- Conversion: `ascii`, `bin`, `chr`, `format`, `hash`, `hex`, `id`, `oct`, `ord`, `repr` +- Classes: class definitions, `classmethod`, `property`, `staticmethod`, `super` +- Other: `len`, `print`, `None`, `True`, `False`, `Ellipsis`, `NotImplemented` - The standard exception classes, such as `ValueError` -Many common builtins aren't in this list, including `sum`, `any`, `all`, `type`, `getattr`, -`hasattr`, `iter`, `next`, `reversed`, `filter`, `frozenset`, `format`, `object` and `super`. Student -code that uses one fails with a `NameError`, so add the ones your question's solutions need to -`builtin_whitelist`. +Builtins that read or change attributes and namespaces by name, such as `getattr`, `hasattr`, +`setattr`, `vars`, `dir` and `globals`, aren't in this list. Student code that uses one fails with a +`NameError`, so add the ones your question's solutions need to `builtin_whitelist`. A builtin in +the default list can't be removed with `builtin_whitelist`. **Dangerous builtins automatically blocked** include: `open`, `eval`, `exec`, `compile`, `__import__`, `input`, `exit`, `quit`, and others that could compromise security. @@ -1187,14 +1188,14 @@ code that uses one fails with a `NameError`, so add the ones your question's sol ```python # In server.py def generate(data): - data["params"]["builtin_whitelist"] = ["sum", "any", "all"] - # Students can now use sum(), any() and all() in addition to the defaults + data["params"]["builtin_whitelist"] = ["getattr", "hasattr"] + # Students can now use getattr() and hasattr() in addition to the defaults ``` **Example with builtin whitelist:** ```python -# data.json has "builtin_whitelist": ["sum"] +# data.json has "builtin_whitelist": ["getattr"] # Default builtins work normally: my_list = [1, 2, 3] # ✓ list is a default @@ -1202,10 +1203,10 @@ length = len(my_list) # ✓ len is a default print(length) # ✓ print is a default # Whitelisted builtins work: -total = sum(my_list) # ✓ sum is in the whitelist +count = getattr(my_list, "count")(1) # ✓ getattr is in the whitelist # Other builtins are blocked: -found = any(my_list) # ✗ NameError: name 'any' is not defined +names = dir(my_list) # ✗ NameError: name 'dir' is not defined # Dangerous builtins are blocked: f = open("file.txt") # ✗ NameError: 'open' is not defined @@ -1222,12 +1223,12 @@ def generate(data): # Allow only numpy and math imports data["params"]["import_whitelist"] = ["numpy", "math"] - # Allow sum for student convenience - data["params"]["builtin_whitelist"] = ["sum"] + # Allow getattr for student convenience + data["params"]["builtin_whitelist"] = ["getattr"] # Now students can: # - Import numpy and math - # - Use sum() plus the default builtins + # - Use getattr() plus the default builtins # - Cannot import other modules # - Cannot use dangerous builtins like open(), eval() ``` @@ -1246,7 +1247,7 @@ def generate(data): # Configure security restrictions data["params"]["import_whitelist"] = ["numpy", "numpy.linalg"] - data["params"]["builtin_whitelist"] = ["sum"] + data["params"]["builtin_whitelist"] = ["getattr"] # Define variables accessible to student code data["params"]["names_for_user"] = [ @@ -1283,7 +1284,7 @@ def test_security(sandbox: StudentFixture) -> None: { "params": { "import_whitelist": ["numpy", "math"], - "builtin_whitelist": ["sum"], + "builtin_whitelist": ["getattr"], "names_for_user": [...] } } @@ -1294,10 +1295,9 @@ def test_security(sandbox: StudentFixture) -> None: **1. Force students to implement algorithms from scratch:** ```python -# Require students to write their own sum +# Require students to implement algorithms without libraries data["params"]["import_whitelist"] = [] # No imports allowed -# Don't set builtin_whitelist: sum isn't a default builtin, so students can't call it. -# sorted is a default, so this doesn't stop students from sorting with it. +# Default builtins such as sorted() and sum() are still available. ``` **2. Allow numerical computing but block system access:**