diff --git a/README.md b/README.md index faf1b15..5fc6ee5 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,9 @@ pip install pytest-prairielearn-grader[notebook] See [quick_start.md](quick_start.md) for a full tutorial with PrairieLearn integration examples. +To convert a question from the legacy Python grader (`prairielearn/grader-python`), follow +[converting.md](converting.md). It's written so that you can hand it to an AI coding agent. + ## Key Features ### Sandboxed Execution @@ -102,7 +105,7 @@ from pytest_prairielearn_grader import ConfigObject autograder_config = ConfigObject( sandbox_timeout=2.0, import_whitelist=["numpy", "math"], - builtin_whitelist=["len", "range", "sum"], + builtin_whitelist=["getattr"], output_level="friendly", syntax_errors_ungradable=True, starting_vars={"coefficient": 10}, diff --git a/converting.md b/converting.md new file mode 100644 index 0000000..d0b8bfd --- /dev/null +++ b/converting.md @@ -0,0 +1,393 @@ +# Converting Questions from the Legacy Python Grader + +This guide explains how to convert a PrairieLearn question from the legacy Python autograder +(`prairielearn/grader-python`, with `tests/test.py` and `PLTestCase`) to this pytest grader +(`prairielearn/grader-python-pytest`). It's written so that you can give it to a person or an AI +coding agent and have them do the conversion from start to finish. + +[quick_start.md](quick_start.md) shows how to write a question for this grader and documents its +full API. This guide covers only what's specific to converting: what changes, in what order to do +it, and the differences from the legacy grader that are easy to miss. + +Converted questions in the PrairieLearn example course are good references: + +| Question (under `exampleCourse/questions/demo/`) | Shows | +| ------------------------------------------------ | ------------------------------------------------------------------------------------------ | +| `autograder/python/square` | A scalar with a tolerance. It still keeps `x` in `setup_code.py`, unlike the version below | +| `autograder/python/pandas` | Checking a DataFrame | +| `autograder/python/randomFunctionName` | `server.py` parameters, `data_json`, partial credit, timeouts | +| `autograder/codeEditor` | A function question | +| `annotated/MarkovChainGroupActivity/*` | Notebook workspaces, `helper_modules`, `data_files` | + +## Where code belongs + +Legacy questions often run grading code next to the student's code, in `leading_code.py`, +`trailing_code.py` and `setup_code.py`. Don't carry that over. Follow +[Where Code Belongs](quick_start.md#where-code-belongs), which says where each kind of code goes in +this grader and why. A converted question should usually have only `initial_code.py` and +`test_student.py`, plus `setup_code.py` only when student code needs something from it: + +- **`ans.py`**: compute the expected values in the tests. +- **`trailing_code.py`**: call the student's functions from the tests with `query_function`. +- **`leading_code.py`**: move imports and definitions to the starter code, or to `helper_modules`. +- **`setup_code.py`**: move random plain values to `server.py`, fixed values to `starting_vars` and + values that only tests use to the tests. Keep what's left, such as numpy arrays and blocked + functions. + +Don't change what the student has to write to make this work. If the student's task is to set +variables at the top level, keep it that way, and read the variables with `sandbox.query`. If +students have already submitted to the question, check that removing `leading_code.py` doesn't break +those submissions when they're regraded. + +## 1. Check whether the question can be converted + +Read every file in the question, including `info.json`, `question.html`, `server.py` and everything in +`tests/`. Stop and report back instead of converting if the question needs one of these: + +- **Plot grading.** `PLTestCaseWithPlot` with tests that read `self.st.plt` or call + `Feedback.check_plot`. The pytest grader can only grade plots that student code builds with + `MatplotlibSerializer` and returns from a function (see + [Testing Matplotlib Plots](quick_start.md#testing-matplotlib-plots)), so the student's task would + change. `PLTestCaseWithPlot` on its own, with no test that reads the plot, isn't a blocker: use the + normal conversion. +- **Course libraries.** `externalGradingOptions.serverFilesCourse` or imports of course-wide + modules. The legacy grader put `/grade/serverFilesCourse` on the import path, and this grader + doesn't. +- **A custom grader image** or an `entrypoint` in `externalGradingOptions`. +- **Values the sandbox can't send back.** Tests can only receive values that can be serialized: + numbers, strings, `None`, lists, dicts, numpy arrays and scalars, pandas DataFrames and plots + built with `MatplotlibSerializer`. A test that needs a student's class instance, generator or + open file has to be rewritten to call student functions that return plain values. Do that only + if the student's task stays the same. + +## 2. Change the files + +| Legacy file | What to do | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| `info.json` | Change `externalGradingOptions.image`. Keep `timeout` and everything else. See [Choosing an image](#choosing-an-image). | +| `question.html` | Change the file editor's `file-name` to `student_code.py`, or keep it and set `student_code_pattern`. | +| `tests/test.py` | Replace with `tests/test_student.py`. See [Rewrite the tests](#3-rewrite-the-tests). | +| `tests/ans.py` | Delete it. Compute expected values in `test_student.py`. Don't leave a reference solution in `tests/`. | +| `tests/setup_code.py` | Move what you can out of it, then delete it or update it. See [Update the setup code](#4-update-the-setup-code). | +| `tests/leading_code.py` | Move its code to the starter code or `helper_modules`, and delete it. See [Where code belongs](#where-code-belongs). | +| `tests/trailing_code.py` | Move its code into the tests and delete it. See [Where code belongs](#where-code-belongs). | +| `tests/initial_code.py` | Keep it. The grader doesn't read it. Fix starter code that doesn't parse (see below). | +| `server.py` | Usually unchanged. | + +Other changes: + +- **Starter code that doesn't parse.** A submission with a `SyntaxError` is ungradable, so it + doesn't cost the student an attempt, but the student sees no test results. If `initial_code.py` + has placeholders like `x = # fill this in`, replace them with something that parses, such as + `x = 0 # fill this in`, so an unchanged submission gets 0 points with feedback. +- **``.** Keep these elements. The `names_for_user` list still controls + which setup variables student code gets. The grader ignores `names_from_user`, which only + displays the table. +- **The file name.** The grader looks for `student_code*.py` by default, while the legacy grader looked + for `user_code.py`. Change `file-name="user_code.py"` to `file-name="student_code.py"` in + `question.html`, or add `student_code_pattern="user_code.py"` to the `ConfigObject`. For workspace + and notebook questions, set `student_code_pattern` to the graded file's name, such as + `"Markov-Chains-1.ipynb"`. + +### Choosing an image + +- `prairielearn/grader-python-pytest:lite` when the student code, setup code and tests use only the + standard library. +- `prairielearn/grader-python-pytest:latest` for anything else, including numpy, pandas, + matplotlib, sympy and notebooks. + +See [Choosing an Image](quick_start.md#choosing-an-image) for what each includes. + +## 3. Rewrite the tests + +Each `test_*` method on the legacy `Test` class becomes a `test_*` function in +`tests/test_student.py`. Keep each test's name, points and order, and keep checking the same things +with the same tolerances. Don't make the question easier or harder. + +Here's the `square` question, which asks students to set `x_sq` to the square of a random `x`. +The legacy version has four files in `tests/`: + +```python +# tests/setup_code.py +import random + +x = random.random() +``` + +```python +# tests/ans.py +x_sq = x**2 +``` + +```python +# tests/initial_code.py +import numpy as np + +x_sq = # fill in this code here... +``` + +```python +# tests/test.py +class Test(PLTestCaseWithPlot): + @points(10) + @name("x_sq") + def test_0(self): + if Feedback.check_scalar("x_sq", self.ref.x_sq, self.st.x_sq, accuracy_critical=False): + Feedback.set_score(1) + else: + Feedback.set_score(0) +``` + +Converted, following [Where code belongs](#where-code-belongs): + +- `x` is a plain random value that student code reads, so `server.py` generates it instead of + `setup_code.py`. `question.html` already lists `x` in + ``, so student code still gets it. +- The expected value comes from the test, so `ans.py` is deleted. +- The starter code parses, so an unchanged submission gets 0 points with feedback. +- `question.html` changes `file-name="user_code.py"` to `file-name="student_code.py"`, and + `info.json` changes the image to `prairielearn/grader-python-pytest:lite`. + +```python +# server.py +import random + + +def generate(data): + data["params"]["x"] = random.random() +``` + +```python +# tests/initial_code.py +x_sq = 0 # fill in this code here... +``` + +```python +# tests/test_student.py +import pytest +from pytest_prairielearn_grader.assertions import assert_approx_equal +from pytest_prairielearn_grader.fixture import StudentFixture + +pytestmark = pytest.mark.output(level="friendly") + + +@pytest.mark.grading_data(name="x_sq", points=10) +def test_x_sq(sandbox: StudentFixture, data_json: dict) -> None: + x = data_json["params"]["x"] + assert_approx_equal(sandbox.query("x_sq"), x**2, description="x_sq") +``` + +The converted question has no `setup_code.py`. `x` is now the same in every test and every +regrade of a variant, and `question.html` could show it with `{{params.x}}`. + +### API mapping + +| Legacy | pytest grader | +| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `class Test(PLTestCase)` with `test_*` methods | Module-level `test_*` functions that take the `sandbox` fixture | +| `@points(10)` and `@name("...")` | `@pytest.mark.grading_data(name="...", points=10)` | +| Tests run in sorted name order | Tests run in file order | +| `self.st.x` | `sandbox.query("x")` | +| `Feedback.call_user(self.st.f, a, b)` | `sandbox.query_function("f", a, b)`, or `query_function_raw` to inspect exceptions and timeouts | +| `self.ref.x` from `ans.py` | Compute the expected value in the test | +| A setup variable, such as `x` from `setup_code.py` | `sandbox.query_setup("x")` | +| `self.data["params"]["k"]` | The `data_json` fixture: `data_json["params"]["k"]` | +| `Feedback.set_score(s)` | `feedback.set_score(s)` with the `feedback` fixture. See [Scoring](#scoring). | +| `Feedback.add_feedback("...")` | `feedback.add_message("...")` | +| `accuracy_critical=True` | A failed assertion ends the test. Other tests still run. | +| `total_iters = 3` and `@not_repeated` | `@pytest.mark.parametrize` over the inputs, or a loop with partial credit | +| `Feedback.not_allowed` | `import_blacklist=["numpy.linalg.inv"]` in the `ConfigObject`. See [Update the setup code](#4-update-the-setup-code). | +| `repeated_setup()` | Not needed: setup code runs again for each sandbox. | + +### Checks + +Replace the `Feedback.check_*` functions with the helpers in `pytest_prairielearn_grader.assertions`, +or with numpy and pandas functions wrapped in `assert_true`: + +| Legacy | pytest grader | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `check_scalar(name, ref, st)` | `assert_approx_equal(value, expected, description=name)`, keeping `rtol=1e-5, atol=1e-8` unless the test set others | +| Exact comparisons | `assert_equal(value, expected, description=name)` | +| `call_user` plus a comparison | `assert_fn_equal(sandbox, "f", args=(...), expected=...)` or `assert_fn_approx_equal` | +| `check_numpy_array_allclose(name, ref, st)` | The four [array checks](quick_start.md#hiding-expected-values) on type, dtype, shape and values | +| `check_numpy_array_features(name, ref, st)` | The first three of those array checks | +| `check_list(name, ref, st, entry_type=T)` | `assert_true(isinstance(value, list), ...)`, a length check and an `isinstance` check on each entry | +| `check_tuple`, `check_dict` | `assert_equal` | +| `check_dataframe(name, ref, st)` | `pandas.testing.assert_frame_equal(expected, df, check_like=True)`, as in the `pandas` example question | + +Keep `check_scalar`'s default tolerances of `rtol=1e-5, atol=1e-8` unless the legacy test set others. + +### Feedback messages + +Use `@pytest.mark.output(level=GradingOutputLevel.Friendly)` on each test, as the example questions +do, or `output_level="friendly"` in the `ConfigObject`. At this level the student sees only the +assertion's message. + +Follow [Hiding Expected Values](quick_start.md#hiding-expected-values): don't use a bare `assert` +or numpy testing functions, and use `assert_true` for hidden test cases. The comparison helpers show +the expected value, which is fine when the legacy feedback showed it too. Keep the wording of the +legacy feedback where you can, such as `"x is inaccurate"`. + +### Scoring + +- A test that finishes without failing gets full points, even if it never calls `feedback.set_score`. +- A test that fails gets the last score it set with `feedback.set_score`, or 0. Call `set_score` + after each check that earns partial credit. +- `feedback.set_score_final(s)` sets the score even if the test then fails. +- The legacy pattern of `if check: set_score(1) else: set_score(0)` becomes a plain assertion. + +### Querying + +- `query` and `query_function` raise `NameError` if the student didn't define the name, which fails + the test with a message that says so. You don't need to check this yourself. +- Each call to `query_function` has a 1 second timeout by default. Pass `query_timeout=` for slower + functions. +- When a test calls a student function several times, use `query_function_raw` and stop after the + first `"timeout"` status: see the `randomFunctionName` example question. + +## 4. Update the setup code + +First, move what you can out of the setup code, as described in [Where code belongs](#where-code-belongs): + +- Values that only tests use go in `test_student.py`. +- Random variables that student code reads go in `server.py`, if they're plain JSON values. +- Fixed variables that student code reads go in `starting_vars`. +- Functions that the setup code blocks, such as `la.inv = not_allowed`, go in `import_blacklist`. + +If nothing is left, delete `setup_code.py`. Otherwise, update what's left. Setup code runs inside the sandbox, so some things that worked in the legacy grader don't: + +- **Parameters.** `data` isn't defined. Use `__data_params`, which is `data["params"]`. Values that + `server.py` stored with `pl.to_json` arrive still encoded: decode them with + `__from_server_json(__data_params["A"])`. The `prairielearn` package isn't available. +- **Restrictions.** Setup code runs with the same builtins and import rules as student code. For + example, it can't call `open` or import `os`. To read a data file, list it in `data_files` and use + a library function such as `np.loadtxt`. +- **Variables for the student.** Student code only gets the setup variables named in + `names_for_user`. This list usually comes from `` in `question.html`. If the question defines these names somewhere + else, or not at all, set `names_for_user=[...]` in the `ConfigObject`. +- **Fresh values per test.** The `sandbox` fixture runs setup code and student code again for each + test, so random values in setup code change between tests. Read them with `sandbox.query_setup` in + the same test that uses them. Don't compute expected values from random setup + values at module level. +- **Disallowed functions.** `Feedback.not_allowed` isn't available. Instead of overriding the + function in setup code, list it in `import_blacklist`, which blocks it however student code + reaches it. The default blacklist is replaced, so keep its entries: + + ```python + autograder_config = ConfigObject( + import_blacklist=["os", "sys", "subprocess", "pathlib", "shutil", "numpy.linalg.inv"], + sandbox_timeout=15, + ) + ``` + + See [Blocking Functions](quick_start.md#blocking-functions). If the setup code computed a value + with the blocked function, compute it in the test instead. + +## 5. Configure the grader + +Put a `ConfigObject` named `autograder_config` at the top of `test_student.py` when the defaults +don't fit: + +```python +from pytest_prairielearn_grader import ConfigObject + +autograder_config = ConfigObject( + student_code_pattern="Markov-Chains-1.ipynb", + notebook_cell_tag="#grade", + # Importing numpy in student code can take longer than the default timeout. + sandbox_timeout=15, +) +``` + +When a `ConfigObject` is present, the module-level `sandbox_timeout` variable and the +`@pytest.mark.sandbox_timeout` marker are ignored. + +| Situation | Setting | +| --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| Student or setup code imports numpy, pandas or other large libraries | `sandbox_timeout=15`. The default of 1 second covers starting the student's code. | +| The graded file isn't `student_code*.py` | `student_code_pattern="..."` | +| A notebook where only some cells should run | `notebook_cell_tag="#grade"` | +| Student code or `leading_code.py` imports a module from `tests/` | `helper_modules=["module_name"]`. See [Helper Modules](quick_start.md#helper-modules). | +| Student code or `leading_code.py` reads a file from `tests/` | `data_files=["file.txt"]`, and read it in tests with the `tests_dir` fixture. See [Data Files](quick_start.md#data-files). | +| Student code imports `os`, `sys`, `subprocess`, `pathlib` or `shutil` | `import_blacklist=[...]` without that module. These are blocked by default. | +| Solutions use builtins that aren't defaults, such as `getattr` | `builtin_whitelist=[...]`. See [Builtin Function Control](quick_start.md#builtin-function-control). | +| The legacy setup code blocked imports or builtins | `import_whitelist`, `import_blacklist` or `builtin_whitelist` | +| A multi-file workspace project with `.py` files | `workspace_mode=True`. See [Workspace Grading](quick_start.md#workspace-grading). | + +`test_student.py` itself can't import other files from `tests/`. Put helper code for the tests in +`test_student.py`, and read data files through `tests_dir`. + +## 6. Notebook questions + +For questions where students submit a Jupyter notebook from a workspace: + +- Set `student_code_pattern` to the notebook's name from `workspaceOptions.gradedFiles`. +- Legacy notebook questions grade only the cells that start with `#grade`. Keep that with + `notebook_cell_tag="#grade"`. Without it, every code cell runs, including plots and scratch work. +- Set `sandbox_timeout=15` if the notebook imports numpy or similar libraries. +- Notebook questions use the normal `sandbox` fixture, not `workspace_mode`. +- The grader runs the graded cells as a script, so anything a graded cell needs must be defined in + a graded cell. Tag the cells with the notebook's imports `#grade`. Use `leading_code.py` only + when students already have copies of the notebook that you can't change. +- If the notebook reads a data file, list it in `data_files`. If the notebook imports a `.py` file + that's in the workspace, copy that file into `tests/` and list it in `helper_modules`. + +## Behavior differences + +These differences from the legacy grader are easy to miss: + +- **`__name__`.** Student code runs with `__name__ == "__main__"`, so code under + `if __name__ == "__main__":` runs during grading. The legacy grader skipped it. If the starter code + has such a block, check that running it during grading is harmless. A `leading_code.py` that only + sets `__name__ = "__main__"` is no longer needed. +- **No reference run.** `ans.py` isn't run. Expected values come from the test code. +- **Setup code runs for each test**, and so does the student's code. Output that student code prints + at the top level appears in each test's feedback, unless the test sets + `include_stdout_feedback=False` in `grading_data`. +- **Restricted builtins.** Student code and setup code get only a default set of builtins. It + includes common ones such as `sum`, `any` and `super`, but not `getattr`, `hasattr`, `setattr`, + `vars`, `dir`, `globals`, `open`, `eval` or `exec`. Check the solution and the starter code, and + add what they use to `builtin_whitelist`. A missing builtin shows up as a `NameError` when you + test the conversion. +- **Default import blacklist.** `os`, `sys`, `subprocess`, `pathlib` and `shutil` are blocked unless + you set `import_blacklist`. +- **Syntax errors** make the submission ungradable instead of scoring 0. + +## 7. Test the conversion + +Test the converted question with the grader image before calling the conversion done, as described +in [Testing Locally](quick_start.md#testing-locally). Run each case in a fresh copy of the layout, +because the image moves files around: + +```bash +mkdir -p grade/tests grade/student grade/data +cp -R path/to/question/tests/. grade/tests/ +cp solution.py grade/student/student_code.py +echo '{"params": {}}' > grade/data/data.json +docker run --rm -v "$PWD/grade:/grade" prairielearn/grader-python-pytest:latest +cat grade/results/results.json +``` + +- **`data.json`.** Put in `params` whatever the question's `server.py` and elements would set. In + particular, `` sets + `params["names_for_user"]` to a list of `{"name": ..., "description": ..., "type": ...}` objects. + Without it, student code doesn't get the setup variables. +- **Cases to run.** At least a correct solution, which should get full points, the unchanged starter + code, which should get 0 without crashing the grader, and one wrong answer for each test. Compare + the scores with what the legacy grader would give. +- **Messages.** Read `message` for each test in `results.json`, and check that hidden test cases don't + reveal their expected values. +- **Notebooks.** Use the solution notebook as the submission, with the file name from `gradedFiles`. + +Finally, test the question in PrairieLearn itself, with a local PrairieLearn server and the example +or course repository. A person usually does this step. If you couldn't, say so when you report back. + +## 8. Report back + +When you're done, report: + +- Which files you changed, added and deleted. +- Any check whose behavior changed, such as a different tolerance or message, and why. +- The cases you ran and their scores. +- Anything you couldn't convert or test. diff --git a/quick_start.md b/quick_start.md index 3996040..2be11a7 100644 --- a/quick_start.md +++ b/quick_start.md @@ -1,12 +1,16 @@ # Quick Start Guide -This is a quick start guide for using the new Python autograder for PrairieLearn. This guide -covers the basic usage and functionality with examples. The grader uses the Docker image -`prairielearn/grader-python-pytest:latest` which is powered by the `pytest-prairielearn-grader` +This guide shows how to write a PrairieLearn question that's graded with the pytest-based Python +autograder, and then documents the grader's full API. The grader runs in the +`prairielearn/grader-python-pytest` Docker images and is built on the `pytest-prairielearn-grader` pytest plugin. -The following discussion is based on converted example questions in PrairieLearn. -For a real example, see: https://github.com/PrairieLearn/PrairieLearn/pull/12603 +Tests are ordinary pytest test functions. Each test gets a fresh sandbox that runs the student's +code, and the test asks the sandbox for the values and functions it wants to check. Student code +never runs in the same process as the tests. + +If you're converting a question that uses the legacy `prairielearn/grader-python` image, follow +[converting.md](converting.md) instead. ## Editor Setup @@ -18,136 +22,253 @@ pip install pytest pytest-prairielearn-grader This enables IDE features like autocomplete, type checking, and inline documentation when writing test cases. -## File Structure +## Your First Question -The required file structure for a PrairieLearn question using this grader is: +This section builds a question that asks students to write a function `average(values)`. A +question that's graded by this grader has these files: ``` -- info.json -- question.html -- tests/ - ├── initial_code.py (optional: starter code for students) - ├── setup_code.py (optional: test setup and parameters) - └── test_student.py (required: test cases) +average/ +├── info.json +├── question.html +└── tests/ + ├── initial_code.py (starter code shown in the file editor) + └── test_student.py (the tests) +``` + +### `info.json` + +Set `gradingMethod` to `"External"` and pick a grader image: + +```json +{ + "uuid": "...", + "title": "Average of a list", + "topic": "Functions", + "type": "v3", + "gradingMethod": "External", + "externalGradingOptions": { + "enabled": true, + "image": "prairielearn/grader-python-pytest:lite", + "timeout": 30 + } +} ``` -**Important**: The file editor element in `question.html` should have `file-name="student_code.py"`. -The autograder looks for `student_code.py` by default. You can customize this by setting -`student_code_pattern = "your_filename.py"` at the global scope of `test_student.py`. +### Choosing an Image + +Two image variants are published. Pick the smallest one that includes what your question needs — a smaller image means faster cold-start pull times and a reduced security surface. + +| Tag | Size (approx.) | Includes | Best for | +| --------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| `:latest` | ~400–500 MB | Full scientific stack: `numpy`, `pandas`, `scipy`, `matplotlib` + `matplotcheck2` + `plot-serializer`, `seaborn`, `scikit-learn`, `sympy`, `networkx`, `nltk`, `nbformat`, and more | Any course that needs scientific computing, data science, plotting, or Jupyter notebook grading | +| `:lite` | ~60–80 MB | Grader core only: `pytest-prairielearn-grader`, `Pygments`, `beautifulsoup4`, `defusedxml`, `requests` | Introductory CS courses where student code uses only the Python standard library | + +> **Note:** If a student's code imports a package that isn't present in the image (e.g., `numpy` in `:lite`), it will raise an `ImportError` and the test will fail with an exception. Make sure the image you choose contains everything both your test code and the expected student solution require. + +Pinned version tags (e.g., `:v1.2.3` and `:v1.2.3-lite`) are also published alongside each release if you need a reproducible build. + +### `question.html` -## Setup Code (`setup_code.py`) +Students write their code in a file editor. The grader looks for a file named `student_code*.py`, +so name the file `student_code.py`, or set [`student_code_pattern`](#configobject-parameters): -The `setup_code.py` file defines variables and functions that are available to the student -code. Only variables listed in the `names_for_user` entry in `data.json` (via the -`pl-external-grader-variables` element) are accessible to the student. +```html + +

+ Write a function average(values) that returns the mean of a + non-empty list of numbers. +

+ + -### Accessing `data.json` Parameters + + + + +``` + +### `tests/initial_code.py` -Inside `setup_code.py`, you can access parameters from `data.json` via the special `__data_params` variable, which contains the `params` dict (i.e., `data.json["params"]`): +The starter code. The grader doesn't read this file, so it's only used by the file editor. Make +sure it parses: a submission with a syntax error is [ungradable](#ungradable-submission-detection), +so an unchanged starter file should run and get 0 points with feedback. ```python -# Access parameters passed from PrairieLearn -coefficient = __data_params["coefficient"] -matrix_size = __data_params["matrix_size"] +def average(values): + return 0 ``` -### Example `setup_code.py` +### `tests/test_student.py` + +Each test is a function that takes the `sandbox` fixture, calls the student's code through it, and +checks the result: ```python -import numpy as np -import numpy.linalg as la +import pytest +from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_fn_approx_equal +from pytest_prairielearn_grader.fixture import StudentFixture +autograder_config = ConfigObject(output_level="friendly") -def not_allowed(*args, **kwargs): - raise RuntimeError("Usage of this function is not allowed in this question.") +@pytest.mark.grading_data(name="Average of one number", points=1) +def test_single(sandbox: StudentFixture) -> None: + assert_fn_approx_equal(sandbox, "average", args=([5],), expected=5) -# Set up parameters -n = np.random.randint(4, 16) -# Generate a random full-rank matrix -X = la.qr(np.random.random_sample((n, n)))[0] -D = np.diag(np.random.random_sample(n) * 10 + 1) -A = X.T @ D @ X +@pytest.mark.parametrize("values", [[1, 2, 3], [2.5, -1, 0, 7], list(range(100))]) +@pytest.mark.grading_data(name="Average of several numbers", points=2) +def test_several(sandbox: StudentFixture, values: list[float]) -> None: + assert_fn_approx_equal(sandbox, "average", args=(values,), expected=sum(values) / len(values)) +``` -b = np.random.random(n) +- `grading_data` sets the name that students see and the test's points. A parametrized test gets + those points for each parameter. +- `assert_fn_approx_equal` calls `average` in the sandbox and compares the result with a + tolerance. On a failure, the student sees which call was checked, the expected value and their + value. See [Checking Answers](#checking-answers) for the other helpers and for checks that hide the + expected value. +- The `friendly` output level shows students only the assertion's message, without a traceback. +- Expected values are computed in the test. Don't put a reference solution in a file that student + code can read. + +### Testing Locally + +The grader image's entrypoint expects PrairieLearn's `/grade` layout. Build it from your question +and a solution, run the image, and read the results: -# Block certain functions -la.inv = not_allowed -la.pinv = not_allowed +```bash +mkdir -p grade/tests grade/student grade/data +cp -R average/tests/. grade/tests/ +cp solution.py grade/student/student_code.py +echo '{"params": {}}' > grade/data/data.json +docker run --rm -v "$PWD/grade:/grade" prairielearn/grader-python-pytest:lite +cat grade/results/results.json ``` -In this example, only `A`, `b`, and `n` (specified in `names_for_user`) are accessible in the -student code. The function blocking demonstrates how to prevent students from using certain -library functions. +The image moves files around, so make a fresh copy of the layout for each run. Put in +`data.json` whatever `params` your `server.py` and elements would set. Try at least a correct +solution, the unchanged starter code and a wrong answer for each test, and read each test's +`message` in `results.json`. -## Test Cases (`test_student.py`) +To run pytest directly while you write tests, install the packages from [Editor Setup](#editor-setup) +and put the files that the grader reads in a folder named after the test file, next to it: -Test cases use pytest fixtures provided by the `pytest-prairielearn-grader` package. +``` +tests/ +├── test_student.py +└── test_student/ + ├── student_code_correct.py (each student_code*.py file is graded separately) + ├── student_code_wrong.py + ├── setup_code.py (if the question has one) + └── data.json (if the tests need params) +``` -### Basic Example +Then run `pytest tests/test_student.py`. -```python -import numpy as np -import numpy.linalg as la -import pytest -from pytest_prairielearn_grader.fixture import StudentFixture +## Where Code Belongs +Besides `test_student.py`, the `tests` folder can hold these files: -@pytest.mark.grading_data(name="x", points=1) -def test_array_all_close(sandbox: StudentFixture) -> None: - """Test that student's solution x solves the linear system.""" - correct_x = la.solve(sandbox.query("A"), sandbox.query("b")) - np.testing.assert_allclose( - sandbox.query("x"), correct_x, err_msg="x is not correct" - ) +| File | What the grader does with it | +| ------------------ | ------------------------------------------------------------------------------------------------------------------- | +| `setup_code.py` | Runs in the sandbox before the student's code. See [Giving Values to Student Code](#giving-values-to-student-code). | +| `leading_code.py` | Prepended to the student's code, in the same namespace. | +| `trailing_code.py` | Appended to the student's code, in the same namespace. | +| Helper modules | Modules that student code can import, when listed in [`helper_modules`](#helper-modules). | +| Data files | Files that student code can read, when listed in [`data_files`](#data-files). | + +Keep as much as possible in `test_student.py`, where each test runs on its own and controls its +feedback. Most questions need only `initial_code.py` and `test_student.py`. Put each piece of code +in the first place in this table that works: + +| Code | Where to put it | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Expected values and reference solutions | The test | +| Calls to the student's functions | `sandbox.query_function` or an assertion helper in the test | +| Inputs to the student's functions, including random ones | The test, passed as arguments. Use `@pytest.mark.parametrize` for several inputs. | +| Variables that student code reads and that `server.py` randomizes | `data["params"]` in `server.py`, listed in `names_for_user`. Tests read them from the `data_json` fixture. This works for plain JSON values, not ones stored with `pl.to_json`. | +| Fixed variables that student code reads | `starting_vars` in the `ConfigObject`, listed in `names_for_user` | +| Variables that student code reads and that need Python to build | `setup_code.py`, such as numpy arrays or values decoded with `__from_server_json` | +| Blocking a library function | [`import_blacklist`](#blocking-functions), such as `["numpy.linalg.inv"]` | +| Imports or definitions that the student's code relies on | The starter code, so that the student sees them. Use `helper_modules` for a module the student imports. Use `leading_code.py` only when neither works. | + +Why: + +- **Trailing code runs when the sandbox starts.** If it calls a student function that raises, every + test fails with the same error, including tests of other functions. When a test calls the function + with `query_function`, only that test fails, and the message names the function. +- **Setup code runs again for each test**, in the sandbox, with the same restrictions as student + code. Random values in it change between tests, and errors in it are reported as if the student's + code failed. +- **Leading code is hidden.** The student's file only works with code they can't see, so it behaves + differently when they run it in a workspace or on their own computer. +- **Values in `data["params"]`** are the same in every test and match what `question.html` shows, + and tests can read them without starting a sandbox. + +`test_student.py` can't import the other files in `tests/`. Put helper code for the tests in +`test_student.py`, and read data files in tests with the [`tests_dir`](#available-fixtures) +fixture. + +## Writing Tests + +Tests are pytest test functions in `tests/test_student.py`. They run in the order they appear in +the file. The examples below leave out their imports, which usually are: + +```python +import pytest +from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_fn_equal, assert_true +from pytest_prairielearn_grader.fixture import FeedbackFixture, StudentFixture ``` ### Test Markers -The `@pytest.mark.grading_data` decorator specifies test metadata: +The `@pytest.mark.grading_data` decorator sets what PrairieLearn shows for the test: - `name`: Test name displayed to students - `points`: Maximum points for the test - `include_stdout_feedback`: (Optional, default=`True`) Whether to include student code's stdout in feedback -Example with stdout control: - -```python -@pytest.mark.grading_data(name="Test Output", points=2, include_stdout_feedback=True) -def test_with_output(sandbox: StudentFixture) -> None: - result = sandbox.query_function("process_data", data) - assert result == expected_value - # Student's print statements will appear in feedback -``` +`@pytest.mark.output(level=...)` sets how much of a failure students see. See +[Output Levels](#output-levels). ### Available Fixtures These fixtures are provided by the `pytest-prairielearn-grader` package: -1. **`sandbox: StudentFixture`**: Provides sandboxed access to student code. Use this to query variables - and call functions from the student's submission. +1. **`sandbox: StudentFixture`**: Runs the setup code and student code in a new sandbox for each + test. Use it to query variables and call functions from the student's submission. + +2. **`module_sandbox: StudentFixture`**: Like `sandbox`, but shares one sandbox across all tests in + the file. See [Module-Scoped Sandbox](#module-scoped-sandbox). -2. **`module_sandbox: StudentFixture`**: Similar to `sandbox`, but maintains state across all tests in - a module. Useful when you want student code initialization to persist between tests (e.g., testing - stateful classes or persistent data structures). +3. **`feedback: FeedbackFixture`**: Sets partial credit and adds feedback messages. See + [Partial Credit and Feedback](#partial-credit-and-feedback). -3. **`feedback: FeedbackFixture`**: Manages partial credit and custom feedback messages for students. +4. **`data_json: dict | None`**: The contents of PrairieLearn's `data.json`, such as + `data_json["params"]`. -4. **`data_json: DataFixture`**: Provides access to parameters from PrairieLearn's `data.json` file - (generated via `pl-external-grader-variables` and other elements). +5. **`tests_dir: Path`**: The question's tests folder. Use it to read instructor files in a test, + for example `np.loadtxt(tests_dir / "pagerank_large.txt")`, instead of building the path from + `__file__`. -5. **`tests_dir: Path`**: The question's tests folder, which holds `leading_code.py` and any data files. - Use it to read instructor files in a test, for example `np.loadtxt(tests_dir / "pagerank_large.txt")`, - instead of building the path from `__file__`. +6. **`workspace_sandbox: WorkspaceFixture`**: For multi-file workspace projects. See + [Workspace Grading](#workspace-grading). For implementation details, see the [fixture source code](https://github.com/prairielearn/grader-python-v2/blob/main/src/pytest_prairielearn_grader/fixture.py). -## Querying Student Code +### Querying Student Code The `sandbox` fixture provides methods to interact with student code: -### 1. Query Variables +#### 1. Query Variables ```python value = sandbox.query("variable_name") @@ -162,10 +283,10 @@ Raises a `NameError` if the variable doesn't exist, and a `RuntimeError` with a @pytest.mark.grading_data(name="Check Variable", points=2) def test_variable(sandbox: StudentFixture) -> None: coefficient = sandbox.query("coefficient") - assert coefficient > 0, "Coefficient must be positive" + assert_true(coefficient > 0, msg="coefficient must be positive") ``` -### 2. Query Functions +#### 2. Query Functions ```python result = sandbox.query_function("function_name", arg1, arg2, kwarg1=value1) @@ -182,27 +303,18 @@ Calls a function defined in the student code with the given arguments and **retu ```python @pytest.mark.grading_data(name="Test Function", points=5) def test_function(sandbox: StudentFixture) -> None: - # Function returns value directly result = sandbox.query_function("calculate", 10, 20) - assert result == 30, f"Expected 30, got {result}" + assert_equal(result, 30, description="calculate(10, 20)") - # With keyword arguments result = sandbox.query_function("process", x=5, y=10) - assert result == expected_value + assert_equal(result, 15, description="process(x=5, y=10)") ``` -**Error handling:** +You don't need to catch these errors: they fail the test with a message saying what went wrong. -```python -@pytest.mark.grading_data(name="Test with Error Handling", points=3) -def test_with_error_handling(sandbox: StudentFixture) -> None: - try: - result = sandbox.query_function("risky_function", data) - assert result == expected_value - except RuntimeError as e: - # The error message includes the original exception details - pytest.fail(f"Function raised an error: {e}") -``` +Each call has a 1 second timeout by default. See [Function Timeout](#4-function-timeout). When a +test calls a student function many times, use `query_function_raw` and stop after the first +`"timeout"` status, so that a slow function doesn't use up the grading job's time. **Testing for specific exceptions with `query_function_raw`:** @@ -212,22 +324,18 @@ use `query_function_raw` to inspect the response directly: ```python @pytest.mark.grading_data(name="Test Exception Handling", points=3) def test_raises_value_error(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: - """Test that student code properly validates input and raises ValueError.""" - # Call function with invalid input - use query_function_raw to get the full response response = sandbox.query_function_raw("validate_input", -1) - # Check that an exception was raised - assert response["status"] == "exception", "Function should raise an exception for negative input" + assert_true(response["status"] == "exception", msg="validate_input(-1) should raise an exception") feedback.set_score(0.5) - # Verify it's the correct exception type - assert response["exception_name"] == "ValueError", \ - f"Expected ValueError, got {response['exception_name']}" + assert_equal(response["exception_name"], "ValueError", description="the exception raised by validate_input(-1)") feedback.set_score(0.75) - # Optionally check the error message - assert "negative" in response["exception_message"].lower(), \ - "Error message should mention negative values" + assert_true( + "negative" in response["exception_message"].lower(), + msg="The error message should mention negative values", + ) feedback.set_score(1.0) ``` @@ -240,7 +348,7 @@ The `query_function_raw` response contains: - `exception_message`: The exception message string - `traceback`: Full traceback string for debugging -### 3. Get Captured Output +#### 3. Get Captured Output ```python output = sandbox.get_accumulated_stdout() @@ -255,10 +363,10 @@ Retrieves stdout captured from student code execution across all function calls. def test_output(sandbox: StudentFixture) -> None: sandbox.query_function("print_greeting", "Alice") output = sandbox.get_accumulated_stdout() - assert "Hello, Alice!" in output, "Greeting not found in output" + assert_true("Hello, Alice!" in output, msg="Greeting not found in output") ``` -### 4. Function Timeout +#### 4. Function Timeout Control execution time for individual function calls: @@ -267,11 +375,134 @@ Control execution time for individual function calls: result = sandbox.query_function("slow_computation", data, query_timeout=2.0) ``` -**Note**: Student code must define the queried symbols, and return values must be JSON-serializable. -Supported types include: `int`, `float`, `str`, `list`, `dict`, `bool`, `None`, numpy arrays, pandas DataFrames, +#### 5. Query Setup Variables + +```python +x = sandbox.query_setup("x") +``` + +Retrieves a variable defined by `setup_code.py`, whether or not it's in `names_for_user`. The value +comes from the same sandbox as the student's code, so a random value matches the one the student's +code received. + +**Note**: Values sent back from the sandbox must be serializable. Supported types include: `int`, `float`, `str`, `list`, `dict`, `bool`, `None`, numpy arrays, pandas DataFrames, and plots built with `MatplotlibSerializer` (see [Testing Matplotlib Plots](#testing-matplotlib-plots)). -## Partial Credit and Feedback +### Checking Answers + +The `pytest_prairielearn_grader.assertions` module has helpers that raise an `AssertionError` with a +structured message: + +``` +Checking: add(2, 3) +Expected output: 5 +Your code output: -1 +The expected and actual output do not match. +``` + +**`assert_equal(actual, expected, *, msg=None, description=None)`** + +Compare two values for equality: + +```python +assert_equal(sandbox.query("x"), 42, description="variable 'x'") +``` + +**`assert_approx_equal(actual, expected, *, rtol=1e-5, atol=1e-8, msg=None, description=None)`** + +Compare numbers with a tolerance. It's for scalars: see below for arrays. + +```python +result = sandbox.query_function("compute_pi") +assert_approx_equal(result, 3.14159, rtol=1e-3, description="compute_pi()") +``` + +**`assert_fn_equal(sandbox, func_name, *, args=(), kwargs=None, expected, msg=None, description=None, query_timeout=1.0)`** + +Call a student function and compare its return value. The description defaults to the call, such as +`add(2, 3)`: + +```python +assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) +``` + +**`assert_fn_approx_equal(sandbox, func_name, *, args=(), kwargs=None, expected, rtol=1e-5, atol=1e-8, msg=None, query_timeout=1.0)`** + +Same as `assert_fn_equal`, with a tolerance: + +```python +assert_fn_approx_equal(sandbox, "compute_area", args=(3.0,), expected=28.274, rtol=1e-2) +``` + +**`assert_true(condition, *, msg=None)` / `assert_false(condition, *, msg=None)`** + +Check a condition, and show only `msg` when it fails: + +```python +assert_true(sandbox.query_function("is_valid", data), msg="is_valid() should accept valid input.") +``` + +#### Hiding Expected Values + +The comparison helpers show the expected value, which is fine when the student can see the test's +inputs. For hidden test cases, use `assert_true` with a message that doesn't include the answer. + +Don't use a bare `assert` or numpy and pandas testing functions for checks that students see. +pytest rewrites a bare `assert`, so its message includes both values, such as `assert 6 == 12345`. +`np.testing.assert_allclose` and similar include both arrays. Use +`assert_true(np.allclose(...), msg=...)` instead. + +For a numpy array, check the type, dtype, shape and values in that order, so that each failure has +a clear message and a wrong shape doesn't reach `np.allclose`: + +```python +value = sandbox.query("x") +assert_true(isinstance(value, np.ndarray), msg="x should be a numpy array") +assert_true(np.issubdtype(value.dtype, np.number), msg="x should contain numbers") +assert_true(value.shape == expected.shape, msg=f"x should have shape {expected.shape}") +assert_true(np.allclose(value, expected), msg="x is inaccurate") +``` + +For a pandas DataFrame, `pandas.testing.assert_frame_equal(expected, df)` works, but its message +shows both frames. + +#### Output Levels + +The output level sets how much of a failure students see: + +| Level | Shows to student | +| ------------- | ------------------------------------------------------------- | +| `"none"` | Exception class name only (e.g., `AssertionError`) | +| `"message"` | Exception name + first line of message _(default when unset)_ | +| `"traceback"` | Exception name + message + full traceback | +| `"friendly"` | Only the exception message text — no class name, no traceback | + +Use `"friendly"` with the assertion helpers. Set it on one test: + +```python +@pytest.mark.output(level="friendly") +@pytest.mark.grading_data(name="Test addition", points=5) +def test_add(sandbox: StudentFixture) -> None: + assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) +``` + +On every test in a file, with a module-level `pytestmark`: + +```python +pytestmark = pytest.mark.output(level="friendly") +``` + +Or in the `ConfigObject`: + +```python +autograder_config = ConfigObject(output_level="friendly") +``` + +With the `ConfigObject` approach, individual tests can still override it with +`@pytest.mark.output(level="traceback")`. The `"message"` and `"traceback"` levels can help +advanced students, who can learn from reading tracebacks. + +### Partial Credit and Feedback Tests flow linearly, allowing partial credit after certain assertions pass. When an assertion fails, the student receives the last partial credit value set before the failure. @@ -281,17 +512,17 @@ fails, the student receives the last partial credit value set before the failure def test_with_partial_credit(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: # Check basic requirements (worth 30%) result = sandbox.query("data_loaded") - assert result is not None, "Data must be loaded" + assert_true(result is not None, msg="Data must be loaded") feedback.set_score(0.3) # Check intermediate computation (worth 60%) intermediate = sandbox.query("processed_data") - assert len(intermediate) > 0, "Data processing failed" + assert_true(len(intermediate) > 0, msg="Data processing failed") feedback.set_score(0.6) # Check final result (worth 100%) final = sandbox.query("final_result") - assert final == expected_value, "Final result incorrect" + assert_true(final == expected_value, msg="Final result incorrect") feedback.set_score(1.0) # Add custom feedback @@ -304,85 +535,304 @@ def test_with_partial_credit(sandbox: StudentFixture, feedback: FeedbackFixture) - `feedback.set_score_final(fraction)`: Set final score (prevents further updates) - `feedback.add_message(msg)`: Add custom feedback message -## Advanced Features +### Capturing and Testing Output -### Configuration with ConfigObject +Control whether student code output appears in feedback: -For complex test scenarios, you can use the `ConfigObject` class to configure all autograder settings -in a single, type-safe, immutable object. This is especially useful when you need to override multiple -settings or want better IDE support with autocomplete and type checking. +```python +@pytest.mark.grading_data(name="With Output", points=2, include_stdout_feedback=True) +def test_with_output(sandbox: StudentFixture) -> None: + """Student's print statements will appear in feedback.""" + result = sandbox.query_function("process_data") + # Any print() calls in student code are captured and shown + assert_equal(result, expected) -**Key Features:** -- **Type-safe**: All parameters are type-checked with validation -- **Immutable**: Configuration cannot be modified after creation (frozen dataclass) -- **Keyword-only**: Must use explicit parameter names for clarity -- **Comprehensive**: Supports all security and execution settings +@pytest.mark.grading_data(name="Without Output", points=2, include_stdout_feedback=False) +def test_without_output(sandbox: StudentFixture) -> None: + """Student's print statements will NOT appear in feedback.""" + result = sandbox.query_function("process_data") + assert_equal(result, expected) -#### Basic Usage -Import and create a `ConfigObject` at the module level in your `test_student.py`: +@pytest.mark.grading_data(name="Manual Output Check", points=2) +def test_output_manually(sandbox: StudentFixture) -> None: + """Manually inspect and test stdout.""" + sandbox.query_function("print_greeting", "Alice") + output = sandbox.get_accumulated_stdout() + assert_true("Hello, Alice!" in output, msg="Greeting message not found in output") +``` + +### Module-Scoped Sandbox + +Use `module_sandbox` instead of `sandbox` when you want student code state to persist +across multiple tests: ```python -from pytest_prairielearn_grader import ConfigObject -from pytest_prairielearn_grader.fixture import StudentFixture -import pytest +@pytest.mark.grading_data(name="Initialize", points=1) +def test_initialization(module_sandbox: StudentFixture) -> None: + """First test initializes state.""" + result = module_sandbox.query_function("initialize_counter") + assert_equal(result, 0) -# Module-level configuration - detected automatically by the plugin -autograder_config = ConfigObject( - sandbox_timeout=2.0, - import_whitelist=["numpy", "pandas"], - builtin_whitelist=["len", "range", "sum", "print"], - starting_vars={"coefficient": 10, "threshold": 5.0}, - names_for_user=["coefficient", "threshold"], -) -@pytest.mark.grading_data(name="Test with Config", points=5) -def test_with_config(sandbox: StudentFixture) -> None: - # Configuration is automatically applied - result = sandbox.query_function("process", data) - assert result == expected_value +@pytest.mark.grading_data(name="Increment 1", points=1) +def test_increment_1(module_sandbox: StudentFixture) -> None: + """State persists - counter should be 1.""" + result = module_sandbox.query_function("increment_counter") + assert_equal(result, 1) + + +@pytest.mark.grading_data(name="Increment 2", points=1) +def test_increment_2(module_sandbox: StudentFixture) -> None: + """State still persists - counter should be 2.""" + result = module_sandbox.query_function("increment_counter") + assert_equal(result, 2) ``` -#### ConfigObject Parameters +**Use Cases for `module_sandbox`**: -All parameters are optional with sensible defaults: +- Testing stateful classes or modules +- Expensive initialization that should only run once +- Testing persistent data structures (databases, file systems, etc.) +- Simulating multi-step workflows -- **`sandbox_timeout`** (float, default=1.0): Timeout in seconds for sandbox initialization and operations -- **`import_whitelist`** (list[str] | None): Allowed import modules (whitelist mode) -- **`import_blacklist`** (list[str] | None): Blocked import modules (blacklist mode) -- **`builtin_whitelist`** (list[str] | None): Allowed builtin functions -- **`helper_modules`** (list[str] | None): Modules from the tests folder that can be imported (see [Helper Modules](#helper-modules)) -- **`data_files`** (list[str] | None): Files from the tests folder that code in the sandbox can read by relative path (see [Data Files](#data-files)) -- **`names_for_user`** (list[str] | None): List of variable names to inject into student code -- **`student_code_pattern`** (str, default="student_code\*.py"): Glob pattern for finding student files -- **`starting_vars`** (dict[str, Any], default={}): Dictionary of variable values to provide for injection +**Important**: With `module_sandbox`, student code is loaded once and shared across all tests +in the module. Use regular `sandbox` for independent test execution. -#### Helper Modules +### Testing Matplotlib Plots -The tests folder isn't on the import path in the sandbox, and `import_whitelist` only permits a -name. To let setup code, `leading_code.py` and student code import a file from that folder, name it -in `helper_modules`: +To grade a plot, student code builds it with `MatplotlibSerializer` from the `plot-serializer` package and returns the serializer. The test receives a matplotlib `Figure`. + +`plot-serializer` is included in the `:latest` image but not in `:lite`. Returning a plain matplotlib `Figure` from student code isn't supported. ```python -# tests/test_student.py, with the helper in tests/helper_function.py -autograder_config = ConfigObject(helper_modules=["helper_function"]) +# Student code +from plot_serializer.matplotlib.serializer import MatplotlibSerializer + + +def create_plot(data): + serializer = MatplotlibSerializer() + fig, ax = serializer.subplots() + ax.plot(data) + ax.set_xlabel("Time") + ax.set_ylabel("Value") + ax.set_title("Data Visualization") + return serializer ``` ```python -# tests/leading_code.py, or the student's code -import helper_function as hf -``` +# Test code +import matplotlib +matplotlib.use("Agg") # Non-GUI backend for server environments -- **Names:** each entry is a plain module name, without dots, paths or `.py`. Packages aren't supported. -- **Everything else stays out:** other files in the folder, such as `test_student.py`, still can't be - imported, even if they're in `import_whitelist`. -- **Import restrictions:** a listed module doesn't need to be in `import_whitelist`. Listing one that - `import_blacklist` also blocks is a configuration error. -- **Inside a helper module:** it runs with the same builtins and import rules as student code. For - example, it can't use `sum` or import `numpy` unless those are allowed. -- **Configuration errors:** if a listed file is missing or has a syntax error, or the module is - blacklisted, the submission is ungradable. The students don't lose an attempt, and each test's +import pytest +from matplotlib.figure import Figure +from matplotcheck.base import PlotTester +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture + + +@pytest.mark.grading_data(name="Plot Test", points=5) +def test_student_plot(sandbox: StudentFixture) -> None: + """Test that student creates a correct plot.""" + # Student function returns a MatplotlibSerializer, which arrives here as a Figure + plot = sandbox.query_function("create_plot", data) + + assert_true(isinstance(plot, Figure)) + assert_equal(len(plot.axes), 1) + + # Use matplotcheck for detailed plot testing + ax = plot.axes[0] + pt = PlotTester(ax) + + # Check plot properties + pt.assert_plot_type("line") + pt.assert_axis_label_contains(axis="x", strings_expected=["Time"]) + pt.assert_axis_label_contains(axis="y", strings_expected=["Value"]) + pt.assert_title_contains(["Data Visualization"]) +``` + +The grader automatically serializes/deserializes: + +- Plots built with `MatplotlibSerializer` (received as a `Figure`) +- NumPy arrays +- Pandas DataFrames +- Standard Python types (int, float, str, list, dict, bool, None) + +## Giving Values to Student Code + +Some questions give students variables to use, such as a matrix to factor. Student code only gets +the variables named in `names_for_user`. Each name's value comes from the first of these that +defines it: + +1. A variable that `setup_code.py` defines. +2. `starting_vars` in the [`ConfigObject`](#configuration-with-configobject). +3. `data["params"]` from `data.json`. + +`names_for_user` comes from the `ConfigObject` if it's set there, as a list of strings. Otherwise +it comes from `data["params"]["names_for_user"]`, a list of `{"name": ..., "description": ..., +"type": ...}` objects, which `` in +`question.html` usually sets. Without `names_for_user`, student code gets no variables, and using +one fails with a `NameError`. `names_from_user` only displays a table in the question, and the +grader ignores it. + +### Parameters from `server.py` + +When `server.py` randomizes plain values, list them in `names_for_user` and read them in tests with +the `data_json` fixture: + +```python +# server.py +import random + + +def generate(data): + data["params"]["n"] = random.randint(5, 10) +``` + +```html + + + The number of terms + +``` + +```python +# tests/test_student.py +@pytest.mark.grading_data(name="total", points=1) +def test_total(sandbox: StudentFixture, data_json: dict) -> None: + n = data_json["params"]["n"] + assert_equal(sandbox.query("total"), n * (n + 1) // 2, description="total") +``` + +### Fixed Values + +For a value that's the same in every variant, use `starting_vars`: + +```python +autograder_config = ConfigObject(starting_vars={"threshold": 5.0}, names_for_user=["threshold"]) +``` + +### Setup Code (`setup_code.py`) + +Use setup code when a value needs Python to build, such as a numpy array. It runs in the sandbox +before the student's code: + +```python +import numpy as np +import numpy.linalg as la + +n = np.random.randint(4, 16) + +# A random full-rank matrix +X = la.qr(np.random.random_sample((n, n)))[0] +D = np.diag(np.random.random_sample(n) * 10 + 1) +A = X.T @ D @ X +b = np.random.random(n) +``` + +With `names_for_user` set to `A`, `b` and `n`, student code gets those three variables. To stop +students from calling a library function such as `la.inv`, use +[`import_blacklist`](#blocking-functions) rather than setup code. + +- **Parameters.** `__data_params` holds `data["params"]`. Values that `server.py` stored with + `pl.to_json` are still encoded: decode them with `__from_server_json`, for example + `A = __from_server_json(__data_params["A"])`. The `prairielearn` package isn't available. +- **Restrictions.** Setup code has the same builtins and import rules as student code. For example, + it can't call `open` or import `os`. To read a data file, list it in [`data_files`](#data-files). +- **Fresh values.** With the `sandbox` fixture, setup code runs again for each test, so random + values differ between tests. Read a setup value in the test that uses it, with + [`sandbox.query_setup`](#5-query-setup-variables). + +## Configuration + +### Configuration with ConfigObject + +For complex test scenarios, you can use the `ConfigObject` class to configure all autograder settings +in a single, type-safe, immutable object. This is especially useful when you need to override multiple +settings or want better IDE support with autocomplete and type checking. + +**Key Features:** + +- **Type-safe**: All parameters are type-checked with validation +- **Immutable**: Configuration cannot be modified after creation (frozen dataclass) +- **Keyword-only**: Must use explicit parameter names for clarity +- **Comprehensive**: Supports all security and execution settings + +#### Basic Usage + +Import and create a `ConfigObject` at the module level in your `test_student.py`: + +```python +from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture +import pytest + +# Module-level configuration - detected automatically by the plugin +autograder_config = ConfigObject( + sandbox_timeout=2.0, + import_whitelist=["numpy", "pandas"], + builtin_whitelist=["getattr", "hasattr"], + starting_vars={"coefficient": 10, "threshold": 5.0}, + names_for_user=["coefficient", "threshold"], +) + +@pytest.mark.grading_data(name="Test with Config", points=5) +def test_with_config(sandbox: StudentFixture) -> None: + # Configuration is automatically applied + result = sandbox.query_function("process", data) + assert_equal(result, expected_value) +``` + +#### ConfigObject Parameters + +All parameters are optional: + +| Parameter | Default | Description | +| -------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `sandbox_timeout` | `1.0` | Seconds allowed for starting the sandbox, which runs the setup code and student code. See [Timeout Configuration](#timeout-configuration). | +| `output_level` | `None` | Default [output level](#output-levels) for every test. | +| `names_for_user` | `None` | Variables to give student code. See [Giving Values to Student Code](#giving-values-to-student-code). | +| `starting_vars` | `{}` | Values for names in `names_for_user`. | +| `student_code_pattern` | `"student_code*.py"` | Glob pattern for the student's file. | +| `import_whitelist` | `None` | Modules that code in the sandbox may import. `None` allows all. See [Import Control](#import-control). | +| `import_blacklist` | `["os", "sys", "subprocess", "pathlib", "shutil"]` | Modules and functions that code in the sandbox may not use. See [Blocking Functions](#blocking-functions). | +| `builtin_whitelist` | `None` | Builtins to allow beyond the safe defaults. See [Builtin Function Control](#builtin-function-control). | +| `helper_modules` | `None` | Modules from the tests folder that can be imported. See [Helper Modules](#helper-modules). | +| `data_files` | `None` | Files from the tests folder that code in the sandbox can read by relative path. See [Data Files](#data-files). | +| `syntax_errors_ungradable` | `True` | Whether a syntax error makes the submission ungradable. See [Ungradable Submission Detection](#ungradable-submission-detection). | +| `notebook_cell_tag` | `None` | Grade only notebook cells that start with this tag. See [Jupyter Notebook Grading](#jupyter-notebook-grading). | +| `workspace_mode` | `False` | Grade a multi-file project. See [Workspace Grading](#workspace-grading). | +| `workspace_student_dir` | `None` | The student's project folder in workspace mode. | +| `workspace_exec_entry` | `None` | A file to run at startup in workspace mode. | + +#### Helper Modules + +The tests folder isn't on the import path in the sandbox, and `import_whitelist` only permits a +name. To let setup code and student code import a file from that folder, name it +in `helper_modules`: + +```python +# tests/test_student.py, with the helper in tests/helper_function.py +autograder_config = ConfigObject(helper_modules=["helper_function"]) +``` + +```python +# the student's code, or setup code +import helper_function as hf +``` + +- **Names:** each entry is a plain module name, without dots, paths or `.py`. Packages aren't supported. +- **Everything else stays out:** other files in the folder, such as `test_student.py`, still can't be + imported, even if they're in `import_whitelist`. +- **Import restrictions:** a listed module doesn't need to be in `import_whitelist`. Listing one that + `import_blacklist` also blocks is a configuration error. +- **Inside a helper module:** it runs with the same builtins and import rules as student code. For + example, it can't use `getattr` or import `numpy` unless those are allowed. +- **Configuration errors:** if a listed file is missing or has a syntax error, or the module is + blacklisted, the submission is ungradable. The students don't lose an attempt, and each test's message says what's wrong. - **Fixtures:** it works with `sandbox` and `module_sandbox`, for `.py` and `.ipynb` submissions, and with `workspace_sandbox`. @@ -418,7 +868,7 @@ def test_pagerank(sandbox: StudentFixture, tests_dir: Path) -> None: ``` ```python -# tests/leading_code.py, or the student's code +# the student's code, or setup code edges = np.loadtxt("pagerank_large.txt") ``` @@ -439,16 +889,6 @@ edges = np.loadtxt("pagerank_large.txt") > **Warning:** Students can read the listed files, so they must not contain answers. -**How variable injection works**: - -- Only variables listed in `names_for_user` are injected into the student code namespace -- Values are resolved in this priority order: - 1. **Highest**: Variables defined in `setup_code.py` execution - 2. **Medium**: Values from `ConfigObject.starting_vars` - 3. **Lowest**: Values from `data.json` params -- If `names_for_user` is not specified, no variables are injected (prevents variable leaking) -- `starting_vars` provides values but does NOT automatically inject them - variables must still be listed in `names_for_user` - #### Configuration Priority When `ConfigObject` is defined, it overrides all other configuration sources: @@ -474,6 +914,7 @@ sandbox_timeout = 1.0 ```python from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import StudentFixture, FeedbackFixture import pytest @@ -485,8 +926,8 @@ autograder_config = ConfigObject( # Security: only allow scientific computing libraries import_whitelist=["numpy", "scipy", "matplotlib"], - # Allow specific builtins for data processing - builtin_whitelist=["len", "range", "sum", "min", "max", "sorted"], + # Allow builtins that the defaults leave out + builtin_whitelist=["getattr", "hasattr"], # Provide test data starting_vars={ @@ -504,10 +945,10 @@ def test_data_processing(sandbox: StudentFixture, feedback: FeedbackFixture) -> # All configuration from ConfigObject is automatically applied result = sandbox.query_function("process_data") - assert len(result) > 0, "Result should not be empty" + assert_true(len(result) > 0, msg="Result should not be empty") feedback.set_score(0.5) - assert result == expected_output, "Incorrect processing result" + assert_true(result == expected_output, msg="Incorrect processing result") feedback.set_score(1.0) ``` @@ -538,221 +979,6 @@ config = ConfigObject(2.0) # TypeError config = ConfigObject(names_for_user=[123, 456]) # ValueError ``` -### Security: Import and Builtin Restrictions - -Control which Python modules and builtin functions students can use in their code. This is crucial for: - -- **Security**: Preventing access to file system, network, or system operations -- **Pedagogical constraints**: Requiring students to implement functionality from scratch -- **Resource management**: Blocking operations that consume excessive resources - -#### Import Control - -Restrict which Python modules students can import using whitelists or blacklists. - -**Import Whitelist**: Only allow specific modules: - -```python -# In server.py (PrairieLearn question) -def generate(data): - data["params"]["import_whitelist"] = ["numpy", "math", "statistics"] - # Only numpy, math, and statistics can be imported - # Attempting to import any other module raises ImportError -``` - -**Import Blacklist**: Block specific modules while allowing all others: - -```python -# In server.py -def generate(data): - data["params"]["import_blacklist"] = ["os", "subprocess", "sys"] - # os, subprocess, and sys cannot be imported - # All other modules are allowed -``` - -**Combining Whitelist and Blacklist**: Use both for fine-grained control (blacklist checked first): - -```python -# In server.py -def generate(data): - # Allow scientific libraries but block specific dangerous submodules - data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib"] - data["params"]["import_blacklist"] = ["matplotlib.pyplot"] # Block pyplot even though matplotlib is allowed - # Result: numpy, scipy, and matplotlib are allowed, but matplotlib.pyplot is blocked -``` - -**Example student code with import whitelist:** - -```python -# data.json has "import_whitelist": ["numpy", "math"] - -import numpy as np # ✓ Allowed -import math # ✓ Allowed -from numpy import array # ✓ Allowed - -import os # ✗ ImportError: Module 'os' is not allowed to be imported -import pandas # ✗ ImportError: Module 'pandas' is not allowed to be imported -``` - -**Example with import inside function:** - -```python -def my_function(): - import os # ✗ This will raise ImportError when the function is called - return os.getcwd() -``` - -**Note**: Blacklist is checked first, then whitelist. If a whitelist is specified, only those modules (minus any in blacklist) can be imported. If only a blacklist is specified, those modules are blocked but all others are allowed. - -A blacklisted module also blocks its submodules, however they're imported. With `"os"` in the blacklist, `import os.path` is blocked as well as `import os`. With `"matplotlib.pyplot"`, both `import matplotlib.pyplot` and `from matplotlib import pyplot` are blocked, but `import matplotlib` still works. - -The blacklist only checks import statements, so it isn't a security boundary. For example, modules that student code is allowed to import can expose `os` as an attribute. - -#### Builtin Function Control - -Python's builtin functions (like `open()`, `eval()`, `exec()`) are automatically restricted to a safe subset. The autograder provides **only safe builtins by default**, including: - -- Standard types: `int`, `float`, `str`, `bool`, `list`, `dict`, `tuple`, `set`, `frozenset` -- Type checking: `isinstance`, `issubclass`, `type` -- Iteration: `iter`, `next`, `enumerate`, `zip`, `range`, `reversed`, `sorted`, `filter`, `map` -- Math: `abs`, `min`, `max`, `sum`, `round`, `pow`, `divmod` -- Conversion: `chr`, `ord`, `bin`, `oct`, `hex`, `hash` -- Common functions: `len`, `print`, `repr`, `getattr`, `hasattr`, `format` -- Python exceptions (for proper error handling) - -**Dangerous builtins automatically blocked** include: `open`, `eval`, `exec`, `compile`, `__import__`, `input`, `exit`, `quit`, and others that could compromise security. - -**Builtin Whitelist**: Grant access to additional builtin functions: - -```python -# In server.py -def generate(data): - data["params"]["builtin_whitelist"] = ["dict", "sorted", "enumerate"] - # Students can now use dict(), sorted(), and enumerate() in addition to safe defaults - # This is useful when you want to allow specific advanced builtins -``` - -**Example with builtin whitelist:** - -```python -# data.json has "builtin_whitelist": ["dict"] - -# Safe builtins work normally: -my_list = [1, 2, 3] # ✓ list is safe by default -length = len(my_list) # ✓ len is safe by default -print(length) # ✓ print is safe by default - -# Whitelisted builtins work: -my_dict = dict(a=1, b=2) # ✓ dict is in whitelist - -# Dangerous builtins are blocked: -f = open("file.txt") # ✗ NameError: 'open' is not defined -result = eval("1+1") # ✗ NameError: 'eval' is not defined -``` - -#### Combining Security Features - -You can combine import and builtin restrictions for comprehensive control: - -```python -# In server.py -def generate(data): - # Allow only numpy and math imports - data["params"]["import_whitelist"] = ["numpy", "math"] - - # Allow dict builtin for student convenience - data["params"]["builtin_whitelist"] = ["dict"] - - # Now students can: - # - Import numpy and math - # - Use dict() plus safe builtins - # - Cannot import other modules - # - Cannot use dangerous builtins like open(), eval() -``` - -#### PrairieLearn Server.py Configuration - -In your PrairieLearn question's `server.py`, add these parameters during the `generate()` step: - -```python -import prairielearn as pl - -def generate(data): - # Set up question parameters - data["params"]["n"] = 5 - data["params"]["A"] = pl.to_json(np.random.rand(5, 5)) - - # Configure security restrictions - data["params"]["import_whitelist"] = ["numpy", "numpy.linalg"] - data["params"]["builtin_whitelist"] = ["dict", "sorted"] - - # Define variables accessible to student code - data["params"]["names_for_user"] = [ - {"name": "n", "description": "Matrix dimension", "type": "integer"}, - {"name": "A", "description": "Input matrix", "type": "numpy array"} - ] -``` - -These parameters are automatically passed to the autograder via `/grade/data/data.json` and enforced during student code execution. - -In the grader images, `run.sh` passes this path to pytest with `--data-json-path` and makes the file readable only by root, so student code can't open it. The file can contain `correct_answers`. Without that option, for example when running locally, the fixture reads `data.json` next to your test file. The sandbox still receives `params` through `__data_params`, so don't rely on values in `params` staying hidden from student code. - -#### Testing Import/Builtin Restrictions Locally - -To test your restrictions in the test scenarios: - -```python -# tests/test_student.py -import pytest -from pytest_prairielearn_grader.fixture import StudentFixture - -@pytest.mark.grading_data(name="Test with restrictions", points=5) -def test_security(sandbox: StudentFixture) -> None: - """Test that imports are properly restricted.""" - # If student code tries to import blocked modules, - # the sandbox will raise an ImportError during initialization - result = sandbox.query_function("safe_function", data) - assert result == expected -``` - -```python -# data.json -{ - "params": { - "import_whitelist": ["numpy", "math"], - "builtin_whitelist": ["dict"], - "names_for_user": [...] - } -} -``` - -#### Common Use Cases - -**1. Force students to implement algorithms from scratch:** - -```python -# Require students to implement their own sorting -data["params"]["import_whitelist"] = [] # No imports allowed -data["params"]["builtin_whitelist"] = [] # No additional builtins beyond safe defaults -# Students must implement sorting without sorted() or external libraries -``` - -**2. Allow numerical computing but block system access:** - -```python -# Scientific computing course -data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib", "pandas"] -# Safe builtins are automatically enforced (no open, eval, etc.) -``` - -**3. Block dangerous operations explicitly:** - -```python -# Block system and file operations -data["params"]["import_blacklist"] = ["os", "sys", "subprocess", "pathlib", "shutil"] -# All other imports allowed, but dangerous ones explicitly blocked -``` - ### Timeout Configuration Control execution time limits for sandbox initialization and function calls to prevent infinite loops or slow student code. @@ -766,404 +992,329 @@ Variable queries via `sandbox.query()` do not have timeouts since they're simple #### Module-level Default Timeout -Set a default timeout for sandbox initialization in all tests in a file: +Set a default timeout for sandbox initialization in all tests in a file. This variable and the +`sandbox_timeout` marker below are ignored when the file has an `autograder_config`, so use +`ConfigObject(sandbox_timeout=...)` there instead. ```python -# At the top of test_student.py (before imports) -initialization_timeout = 2.0 # 2 second timeout for initialization - import pytest +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import StudentFixture +sandbox_timeout = 2.0 # 2 second timeout for initialization -@pytest.mark.grading_data(name="Test 1", points=1) -def test_with_default_timeout(sandbox: StudentFixture) -> None: - # The sandbox was initialized with 2 second timeout - # Now we can safely query variables and call functions - result = sandbox.query_function("compute_result") - assert result == 5 -``` - -#### Per-test Timeout Override - -Override the default initialization timeout with the `@pytest.mark.sandbox_timeout` marker: - -```python -@pytest.mark.grading_data(name="Fast Test", points=1) -@pytest.mark.sandbox_timeout(0.5) # 0.5 second initialization timeout -def test_with_custom_timeout(sandbox: StudentFixture) -> None: - # This sandbox was initialized with 0.5 second timeout - result = sandbox.query_function("quick_computation") - assert result == 5 -``` - -#### Per-function Timeout - -Set timeout for individual function calls using the `query_timeout` parameter: - -```python -@pytest.mark.grading_data(name="Function Test", points=2) -def test_function_timeout(sandbox: StudentFixture) -> None: - # This specific function call has a 1 second timeout - result = sandbox.query_function("compute", data, query_timeout=1.0) - assert result == expected_value - # This function call uses the default timeout (no per-function limit) - result2 = sandbox.query_function("another_compute", data) - assert result2 == expected_value2 -``` - -**Using ConfigObject for timeouts:** - -```python -from pytest_prairielearn_grader import ConfigObject - -autograder_config = ConfigObject( - sandbox_timeout=3.0, # Sets initialization timeout -) - -@pytest.mark.grading_data(name="Test", points=1) -def test_with_config_timeout(sandbox: StudentFixture) -> None: - # Initialized with 3 second timeout from ConfigObject - result = sandbox.query_function("process", query_timeout=1.0) - assert result == expected -``` - -### Module-Scoped Sandbox - -Use `module_sandbox` instead of `sandbox` when you want student code state to persist -across multiple tests: - -```python -@pytest.mark.grading_data(name="Initialize", points=1) -def test_initialization(module_sandbox: StudentFixture) -> None: - """First test initializes state.""" - result = module_sandbox.query_function("initialize_counter") - assert result == 0 - - -@pytest.mark.grading_data(name="Increment 1", points=1) -def test_increment_1(module_sandbox: StudentFixture) -> None: - """State persists - counter should be 1.""" - result = module_sandbox.query_function("increment_counter") - assert result == 1 - - -@pytest.mark.grading_data(name="Increment 2", points=1) -def test_increment_2(module_sandbox: StudentFixture) -> None: - """State still persists - counter should be 2.""" - result = module_sandbox.query_function("increment_counter") - assert result == 2 -``` - -**Use Cases for `module_sandbox`**: - -- Testing stateful classes or modules -- Expensive initialization that should only run once -- Testing persistent data structures (databases, file systems, etc.) -- Simulating multi-step workflows - -**Important**: With `module_sandbox`, student code is loaded once and shared across all tests -in the module. Use regular `sandbox` for independent test execution. - -### Capturing and Testing Output - -Control whether student code output appears in feedback: - -```python -@pytest.mark.grading_data(name="With Output", points=2, include_stdout_feedback=True) -def test_with_output(sandbox: StudentFixture) -> None: - """Student's print statements will appear in feedback.""" - result = sandbox.query_function("process_data") - # Any print() calls in student code are captured and shown - assert result == expected - - -@pytest.mark.grading_data(name="Without Output", points=2, include_stdout_feedback=False) -def test_without_output(sandbox: StudentFixture) -> None: - """Student's print statements will NOT appear in feedback.""" - result = sandbox.query_function("process_data") - assert result == expected - - -@pytest.mark.grading_data(name="Manual Output Check", points=2) -def test_output_manually(sandbox: StudentFixture) -> None: - """Manually inspect and test stdout.""" - sandbox.query_function("print_greeting", "Alice") - output = sandbox.get_accumulated_stdout() - assert "Hello, Alice!" in output, "Greeting message not found in output" -``` - -### Student-Friendly Feedback Mode - -By default, when a test fails, students see the Python exception class name, a "failed with an -exception:" prefix, and (depending on the output level) a full traceback. This is useful for -debugging but can be intimidating for introductory-level students. - -The **friendly** output level suppresses all traceback noise and shows only the human-readable -assertion message. Combine it with the built-in assertion helpers to produce clean, structured -feedback like: - -``` -Checking: add(2, 3) -Expected output: 5 -Your code output: -1 -The expected and actual output do not match. +@pytest.mark.grading_data(name="Test 1", points=1) +def test_with_default_timeout(sandbox: StudentFixture) -> None: + # The sandbox was initialized with 2 second timeout + # Now we can safely query variables and call functions + result = sandbox.query_function("compute_result") + assert_equal(result, 5) ``` -#### Enabling Friendly Output +#### Per-test Timeout Override -Apply `@pytest.mark.output(level="friendly")` to individual tests: +Override the default initialization timeout with the `@pytest.mark.sandbox_timeout` marker: ```python -import pytest -from pytest_prairielearn_grader.fixture import StudentFixture -from pytest_prairielearn_grader.assertions import assert_fn_equal, assert_equal - -@pytest.mark.output(level="friendly") -@pytest.mark.grading_data(name="Test addition", points=5) -def test_add(sandbox: StudentFixture) -> None: - assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) +@pytest.mark.grading_data(name="Fast Test", points=1) +@pytest.mark.sandbox_timeout(0.5) # 0.5 second initialization timeout +def test_with_custom_timeout(sandbox: StudentFixture) -> None: + # This sandbox was initialized with 0.5 second timeout + result = sandbox.query_function("quick_computation") + assert_equal(result, 5) ``` -Or apply it to every test in a file with a module-level `pytestmark`: +#### Per-function Timeout + +Set timeout for individual function calls using the `query_timeout` parameter: ```python -import pytest +@pytest.mark.grading_data(name="Function Test", points=2) +def test_function_timeout(sandbox: StudentFixture) -> None: + # This specific function call has a 1 second timeout + result = sandbox.query_function("compute", data, query_timeout=1.0) + assert_equal(result, expected_value) -pytestmark = pytest.mark.output(level="friendly") + # This function call uses the default timeout (no per-function limit) + result2 = sandbox.query_function("another_compute", data) + assert_equal(result2, expected_value2) ``` -Or set it globally via `ConfigObject` (recommended for intro courses): +**Using ConfigObject for timeouts:** ```python from pytest_prairielearn_grader import ConfigObject autograder_config = ConfigObject( - output_level="friendly", # All tests use friendly output by default + sandbox_timeout=3.0, # Sets initialization timeout ) -``` -With the `ConfigObject` approach, individual tests can still override with -`@pytest.mark.output(level="traceback")` if needed. +@pytest.mark.grading_data(name="Test", points=1) +def test_with_config_timeout(sandbox: StudentFixture) -> None: + # Initialized with 3 second timeout from ConfigObject + result = sandbox.query_function("process", query_timeout=1.0) + assert_equal(result, expected) +``` -**All output levels:** +### Security: Import and Builtin Restrictions -| Level | Shows to student | -|-------|-----------------| -| `"none"` | Exception class name only (e.g., `AssertionError`) | -| `"message"` | Exception name + first line of message *(default when unset)* | -| `"traceback"` | Exception name + message + full traceback | -| `"friendly"` | Only the exception message text — no class name, no traceback | +Control which Python modules and builtin functions students can use in their code. This is crucial for: -#### Assertion Helpers +- **Security**: Preventing access to file system, network, or system operations +- **Pedagogical constraints**: Requiring students to implement functionality from scratch +- **Resource management**: Blocking operations that consume excessive resources -The `pytest_prairielearn_grader.assertions` module provides helpers that produce the -structured "Checking / Expected / Your output" messages. They raise `AssertionError` -with the formatted message, which the friendly output level then shows as-is. +#### Import Control -**`assert_equal(actual, expected, *, msg=None, description=None)`** +Restrict which Python modules students can import using whitelists or blacklists. -Compare two values for equality: +**Import Whitelist**: Only allow specific modules: ```python -value = sandbox.query("x") -assert_equal(value, 42, description="variable 'x'") -# On failure: -# Checking: variable 'x' -# Expected output: 42 -# Your code output: 99 -# The expected and actual output do not match. +# In server.py (PrairieLearn question) +def generate(data): + data["params"]["import_whitelist"] = ["numpy", "math", "statistics"] + # Only numpy, math, and statistics can be imported + # Attempting to import any other module raises ImportError ``` -**`assert_approx_equal(actual, expected, *, rtol=1e-5, atol=1e-8, msg=None, description=None)`** - -Compare numeric values with tolerance: +**Import Blacklist**: Block specific modules while allowing all others: ```python -result = sandbox.query_function("compute_pi") -assert_approx_equal(result, 3.14159, rtol=1e-3, description="compute_pi()") +# In server.py +def generate(data): + data["params"]["import_blacklist"] = ["os", "subprocess", "sys"] + # os, subprocess, and sys cannot be imported + # All other modules are allowed ``` -**`assert_true(condition, *, msg=None)` / `assert_false(condition, *, msg=None)`** - -Check boolean conditions with a clean message: +**Combining Whitelist and Blacklist**: Use both for fine-grained control (blacklist checked first): ```python -result = sandbox.query_function("is_valid", data) -assert_true(result, msg="Expected is_valid() to return True for valid input.") +# In server.py +def generate(data): + # Allow scientific libraries but block specific dangerous submodules + data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib"] + data["params"]["import_blacklist"] = ["matplotlib.pyplot"] # Block pyplot even though matplotlib is allowed + # Result: numpy, scipy, and matplotlib are allowed, but matplotlib.pyplot is blocked ``` -**`assert_fn_equal(sandbox, func_name, *, args=(), kwargs=None, expected, msg=None, description=None, query_timeout=1.0)`** +**Example student code with import whitelist:** + +```python +# data.json has "import_whitelist": ["numpy", "math"] + +import numpy as np # ✓ Allowed +import math # ✓ Allowed +from numpy import array # ✓ Allowed + +import os # ✗ ImportError: Module 'os' is not allowed to be imported +import pandas # ✗ ImportError: Module 'pandas' is not allowed to be imported +``` -Call a student function and compare its return value — the most common helper. It -automatically generates the description from the function call: +**Example with import inside function:** ```python -assert_fn_equal(sandbox, "add", args=(2, 3), expected=5) -# On failure: -# Checking: add(2, 3) -# Expected output: 5 -# Your code output: -1 -# The expected and actual output do not match. +def my_function(): + import os # ✗ This will raise ImportError when the function is called + return os.getcwd() ``` -**`assert_fn_approx_equal(sandbox, func_name, *, args=(), kwargs=None, expected, rtol=1e-5, atol=1e-8, msg=None, query_timeout=1.0)`** +**Note**: Blacklist is checked first, then whitelist. If a whitelist is specified, only those modules (minus any in blacklist) can be imported. If only a blacklist is specified, those modules are blocked but all others are allowed. + +A blacklisted module also blocks its submodules, however they're imported. With `"os"` in the blacklist, `import os.path` is blocked as well as `import os`. With `"matplotlib.pyplot"`, both `import matplotlib.pyplot` and `from matplotlib import pyplot` are blocked, but `import matplotlib` still works. + +The blacklist only checks import statements, so it isn't a security boundary. For example, modules that student code is allowed to import can expose `os` as an attribute. + +#### Blocking Functions -Same as `assert_fn_equal` but uses approximate comparison: +To stop students from using one library function, such as `numpy.linalg.inv` in a question about +solving linear systems, add the function to `import_blacklist`: ```python -assert_fn_approx_equal(sandbox, "compute_area", args=(3.0,), expected=28.274, rtol=1e-2) +autograder_config = ConfigObject( + import_blacklist=["os", "sys", "subprocess", "pathlib", "shutil", "numpy.linalg.inv", "numpy.linalg.pinv"], + sandbox_timeout=15, +) ``` -#### Combining with Partial Credit +Setting `import_blacklist` replaces the default list, so keep its entries. -Friendly mode works seamlessly with `FeedbackFixture`: +Before setup code runs, the sandbox replaces each listed function in its module with one that +raises an error. So `np.linalg.inv(A)` and `la.inv(A)` raise +`RuntimeError: 'numpy.linalg.inv' is blacklisted and cannot be used.`, and +`from numpy.linalg import inv` raises an `ImportError`. This is the preferred way to block a +function: don't override it in setup code. -```python -@pytest.mark.output(level="friendly") -@pytest.mark.grading_data(name="Multi-step", points=10) -def test_multi_step(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: - # Step 1: basic check - assert_fn_equal(sandbox, "initialize", expected=0) - feedback.set_score(0.5) +- **Which entries:** an entry is treated as a function when the part before its last dot is a module + that can be imported and the last part is a callable that isn't a module. Other entries, such as + `"os"` or `"matplotlib.pyplot"`, block imports as described above. If the module isn't installed + in the image, the entry does nothing. +- **Setup code and helper modules** can't call the function either, because they share the + sandbox's modules. Compute values that need it in the test instead. +- **Startup time:** the function's module is imported when the sandbox starts, even if the student's + code doesn't use it. +- **Not a security boundary:** the function can still be reached under another name, such as + numpy's internal `numpy.linalg._linalg.inv`. - # Step 2: harder check — if this fails, student keeps 50% - assert_fn_equal(sandbox, "compute", args=(10,), expected=100) - feedback.set_score(1.0) -``` +> **Note:** blocking functions needs a grader release that includes +> [PrairieLearn/grader-python-v2#39](https://github.com/PrairieLearn/grader-python-v2/pull/39). -#### When to Use Friendly Mode +#### Builtin Function Control -- **Introductory courses**: Students who haven't seen Python tracebacks benefit from simpler messages. -- **Lite image questions**: Pairs well with the `:lite` Docker image for intro CS courses. -- **Standardized feedback**: When you want all failure messages to follow a consistent format. +Setup code, student code and helper modules get only these builtins by default: -For advanced courses, the default output levels (`message` or `traceback`) may be more appropriate -since students can learn from reading full tracebacks. +- Types: `bool`, `bytearray`, `bytes`, `complex`, `dict`, `float`, `frozenset`, `int`, `list`, + `object`, `set`, `slice`, `str`, `tuple`, `type` +- Type checking: `callable`, `isinstance`, `issubclass` +- Iteration: `all`, `any`, `enumerate`, `filter`, `iter`, `map`, `next`, `range`, `reversed`, + `sorted`, `zip` +- Math: `abs`, `divmod`, `max`, `min`, `pow`, `round`, `sum` +- Conversion: `ascii`, `bin`, `chr`, `format`, `hash`, `hex`, `id`, `oct`, `ord`, `repr` +- Classes: class definitions, `classmethod`, `property`, `staticmethod`, `super` +- Other: `len`, `print`, `None`, `True`, `False`, `Ellipsis`, `NotImplemented` +- The standard exception classes, such as `ValueError` -### Testing Matplotlib Plots +Builtins that read or change attributes and namespaces by name, such as `getattr`, `hasattr`, +`setattr`, `vars`, `dir` and `globals`, aren't in this list. Student code that uses one fails with a +`NameError`, so add the ones your question's solutions need to `builtin_whitelist`. A builtin in +the default list can't be removed with `builtin_whitelist`. -To grade a plot, student code builds it with `MatplotlibSerializer` from the `plot-serializer` package and returns the serializer. The test receives a matplotlib `Figure`. +**Dangerous builtins automatically blocked** include: `open`, `eval`, `exec`, `compile`, `__import__`, `input`, `exit`, `quit`, and others that could compromise security. -`plot-serializer` is included in the `:latest` image but not in `:lite`. Returning a plain matplotlib `Figure` from student code isn't supported. +**Builtin Whitelist**: Grant access to additional builtin functions: ```python -# Student code -from plot_serializer.matplotlib.serializer import MatplotlibSerializer +# In server.py +def generate(data): + data["params"]["builtin_whitelist"] = ["getattr", "hasattr"] + # Students can now use getattr() and hasattr() in addition to the defaults +``` +**Example with builtin whitelist:** -def create_plot(data): - serializer = MatplotlibSerializer() - fig, ax = serializer.subplots() - ax.plot(data) - ax.set_xlabel("Time") - ax.set_ylabel("Value") - ax.set_title("Data Visualization") - return serializer +```python +# data.json has "builtin_whitelist": ["getattr"] + +# Default builtins work normally: +my_list = [1, 2, 3] # ✓ list is a default +length = len(my_list) # ✓ len is a default +print(length) # ✓ print is a default + +# Whitelisted builtins work: +count = getattr(my_list, "count")(1) # ✓ getattr is in the whitelist + +# Other builtins are blocked: +names = dir(my_list) # ✗ NameError: name 'dir' is not defined + +# Dangerous builtins are blocked: +f = open("file.txt") # ✗ NameError: 'open' is not defined +result = eval("1+1") # ✗ NameError: 'eval' is not defined ``` +#### Combining Security Features + +You can combine import and builtin restrictions for comprehensive control: + ```python -# Test code -import matplotlib -matplotlib.use("Agg") # Non-GUI backend for server environments +# In server.py +def generate(data): + # Allow only numpy and math imports + data["params"]["import_whitelist"] = ["numpy", "math"] -import pytest -from matplotlib.figure import Figure -from matplotcheck.base import PlotTester -from pytest_prairielearn_grader.fixture import StudentFixture + # Allow getattr for student convenience + data["params"]["builtin_whitelist"] = ["getattr"] + # Now students can: + # - Import numpy and math + # - Use getattr() plus the default builtins + # - Cannot import other modules + # - Cannot use dangerous builtins like open(), eval() +``` -@pytest.mark.grading_data(name="Plot Test", points=5) -def test_student_plot(sandbox: StudentFixture) -> None: - """Test that student creates a correct plot.""" - # Student function returns a MatplotlibSerializer, which arrives here as a Figure - plot = sandbox.query_function("create_plot", data) +#### PrairieLearn Server.py Configuration - assert isinstance(plot, Figure) - assert len(plot.axes) == 1 +In your PrairieLearn question's `server.py`, add these parameters during the `generate()` step: - # Use matplotcheck for detailed plot testing - ax = plot.axes[0] - pt = PlotTester(ax) +```python +import prairielearn as pl - # Check plot properties - pt.assert_plot_type("line") - pt.assert_axis_label_contains(axis="x", strings_expected=["Time"]) - pt.assert_axis_label_contains(axis="y", strings_expected=["Value"]) - pt.assert_title_contains(["Data Visualization"]) +def generate(data): + # Set up question parameters + data["params"]["n"] = 5 + data["params"]["A"] = pl.to_json(np.random.rand(5, 5)) + + # Configure security restrictions + data["params"]["import_whitelist"] = ["numpy", "numpy.linalg"] + data["params"]["builtin_whitelist"] = ["getattr"] + + # Define variables accessible to student code + data["params"]["names_for_user"] = [ + {"name": "n", "description": "Matrix dimension", "type": "integer"}, + {"name": "A", "description": "Input matrix", "type": "numpy array"} + ] ``` -The grader automatically serializes/deserializes: +These parameters are automatically passed to the autograder via `/grade/data/data.json` and enforced during student code execution. -- Plots built with `MatplotlibSerializer` (received as a `Figure`) -- NumPy arrays -- Pandas DataFrames -- Standard Python types (int, float, str, list, dict, bool, None) +In the grader images, `run.sh` passes this path to pytest with `--data-json-path` and makes the file readable only by root, so student code can't open it. The file can contain `correct_answers`. Without that option, for example when running locally, the fixture reads `data.json` next to your test file. The sandbox still receives `params` through `__data_params`, so don't rely on values in `params` staying hidden from student code. -## More Examples +#### Testing Import/Builtin Restrictions Locally -To see more examples of what is possible in these test files, look at the test cases in -[this folder](https://github.com/prairielearn/grader-python-v2/tree/main/tests/scenario_root). Each test file is called `scenario.py`. +To test your restrictions in the test scenarios: -## PrairieLearn Configuration +```python +# tests/test_student.py +import pytest +from pytest_prairielearn_grader.assertions import assert_equal, assert_true +from pytest_prairielearn_grader.fixture import StudentFixture -In your PrairieLearn question's `info.json`, specify the grader image: +@pytest.mark.grading_data(name="Test with restrictions", points=5) +def test_security(sandbox: StudentFixture) -> None: + """Test that imports are properly restricted.""" + # If student code tries to import blocked modules, + # the sandbox will raise an ImportError during initialization + result = sandbox.query_function("safe_function", data) + assert_equal(result, expected) +``` -```json +```python +# data.json { - "title": "Your Question Title", - "topic": "Your Topic", - "tags": ["your-tags"], - "type": "v3", - "gradingMethod": "External", - "externalGradingOptions": { - "enabled": true, - "image": "prairielearn/grader-python-pytest:latest", - "timeout": 30 + "params": { + "import_whitelist": ["numpy", "math"], + "builtin_whitelist": ["getattr"], + "names_for_user": [...] } } ``` -### Choosing an Image - -Two image variants are published. Pick the smallest one that includes what your question needs — a smaller image means faster cold-start pull times and a reduced security surface. - -| Tag | Size (approx.) | Includes | Best for | -| --------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -| `:latest` | ~400–500 MB | Full scientific stack: `numpy`, `pandas`, `scipy`, `matplotlib` + `matplotcheck2` + `plot-serializer`, `seaborn`, `scikit-learn`, `sympy`, `networkx`, `nltk`, `nbformat`, and more | Any course that needs scientific computing, data science, plotting, or Jupyter notebook grading | -| `:lite` | ~60–80 MB | Grader core only: `pytest-prairielearn-grader`, `Pygments`, `beautifulsoup4`, `defusedxml`, `requests` | Introductory CS courses where student code uses only the Python standard library | +#### Common Use Cases -To use the lite image, change the `image` field: +**1. Force students to implement algorithms from scratch:** -```json -"externalGradingOptions": { - "enabled": true, - "image": "prairielearn/grader-python-pytest:lite", - "timeout": 30 -} +```python +# Require students to implement algorithms without libraries +data["params"]["import_whitelist"] = [] # No imports allowed +# Default builtins such as sorted() and sum() are still available. ``` -> **Note:** If a student's code imports a package that isn't present in the image (e.g., `numpy` in `:lite`), it will raise an `ImportError` and the test will fail with an exception. Make sure the image you choose contains everything both your test code and the expected student solution require. +**2. Allow numerical computing but block system access:** -Pinned version tags (e.g., `:v1.2.3` and `:v1.2.3-lite`) are also published alongside each release if you need a reproducible build. +```python +# Scientific computing course +data["params"]["import_whitelist"] = ["numpy", "scipy", "matplotlib", "pandas"] +# Safe builtins are automatically enforced (no open, eval, etc.) +``` -## Tips and Best Practices +**3. Block dangerous operations explicitly:** -1. **Use descriptive test names**: The `name` in `@pytest.mark.grading_data` is shown to students -2. **Provide clear error messages**: Use informative assertion messages to guide students -3. **Test incrementally**: Break complex problems into smaller tests with partial credit -4. **Control output visibility**: Use `include_stdout_feedback=False` for tests where student output - would be confusing or reveal answers -5. **Set appropriate timeouts**: Prevent infinite loops while allowing reasonable execution time -6. **Use `module_sandbox` sparingly**: Only when you need persistent state across tests -7. **Block prohibited functions**: Use the setup code to prevent students from using disallowed functions - (like in the `la.inv = not_allowed` example above) -8. **Configure security restrictions**: Use `import_whitelist` and `builtin_whitelist` in `server.py` - to control what modules and functions students can access, preventing security issues and - enforcing pedagogical constraints - ---- +```python +# Block system and file operations +data["params"]["import_blacklist"] = ["os", "sys", "subprocess", "pathlib", "shutil"] +# All other imports allowed, but dangerous ones explicitly blocked +``` ## Ungradable Submission Detection @@ -1171,7 +1322,7 @@ When student code has a **SyntaxError** (e.g., missing colons, unmatched bracket the grader automatically marks the submission as **ungradable**. This means: - The student does **not** lose a grading attempt -- They see a clear message: *"Your code could not be parsed. Please fix the syntax errors and resubmit."* +- They see a clear message: _"Your code could not be parsed. Please fix the syntax errors and resubmit."_ - The `format_errors` field in results contains the specific error This is especially useful for introductory courses where students may accidentally submit code @@ -1215,8 +1366,6 @@ The grader also detects when **test files themselves** fail to collect (e.g., im your test code, missing dependencies). These are marked as ungradable with a message directing course staff to check the logs — students are never penalized for grader bugs. ---- - ## Jupyter Notebook Grading PrairieLearn supports JupyterLab as a workspace environment @@ -1331,6 +1480,7 @@ import pytest import numpy as np from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import FeedbackFixture, StudentFixture autograder_config = ConfigObject( @@ -1350,7 +1500,7 @@ def test_solve(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: feedback.set_score(0.5) expected = np.linalg.solve(A, b) - assert np.allclose(result, expected), f"Expected {expected}, got {result}" + assert_true(np.allclose(result, expected), msg="solve(A, b) is inaccurate") feedback.set_score(1.0) @@ -1358,7 +1508,7 @@ def test_solve(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: def test_result_variable(sandbox: StudentFixture, feedback: FeedbackFixture) -> None: feedback.set_score(0.0) x = sandbox.query("x") - assert x is not None, "Variable 'x' was not defined" + assert_true(x is not None, msg="Variable 'x' was not defined") feedback.set_score(1.0) ``` @@ -1384,8 +1534,6 @@ questions/my_notebook_question/ └── notebook.ipynb # local stand-in for the student's submission ``` ---- - ## Workspace Grading PrairieLearn workspace questions give students a full IDE (VS Code, JupyterLab, etc.) running in a @@ -1480,6 +1628,7 @@ Use `workspace_sandbox` and query student code with **dotted module paths**: ```python from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import WorkspaceFixture import pytest @@ -1491,19 +1640,19 @@ autograder_config = ConfigObject( @pytest.mark.grading_data(name="Add two numbers", points=2) def test_add(workspace_sandbox: WorkspaceFixture) -> None: result = workspace_sandbox.query_function("calculator.add", 3, 4) - assert result == 7 + assert_equal(result, 7) @pytest.mark.grading_data(name="Clamp from sub-package", points=2) def test_clamp(workspace_sandbox: WorkspaceFixture) -> None: result = workspace_sandbox.query_function("utils.helpers.clamp", 15, 0, 10) - assert result == 10 + assert_equal(result, 10) @pytest.mark.grading_data(name="Module-level constant", points=1) def test_constant(workspace_sandbox: WorkspaceFixture) -> None: pi = workspace_sandbox.query("calculator.PI_APPROX") - assert abs(pi - 3.14159) < 1e-5 + assert_true(abs(pi - 3.14159) < 1e-5) ``` The dotted path is split on the **last dot**: `"calculator.add"` imports the module `calculator` @@ -1528,8 +1677,8 @@ result = workspace_sandbox.query_function("models.Classifier.predict", X_test) # query_function_raw — inspects the full response without raising response = workspace_sandbox.query_function_raw("calculator.divide", 1, 0) -assert response["status"] == "exception" -assert response["exception_name"] == "ValueError" +assert_equal(response["status"], "exception") +assert_equal(response["exception_name"], "ValueError") ``` ### Optional: Entry-Point Execution @@ -1547,7 +1696,7 @@ autograder_config = ConfigObject( # Now the sandbox namespace includes everything main.py defined at module level def test_global_state(workspace_sandbox: WorkspaceFixture) -> None: db = workspace_sandbox.query("database") # flat name - set by main.py via exec - assert db is not None + assert_true(db is not None) ``` Without `workspace_exec_entry` (the default), no file is executed at startup and all access @@ -1581,7 +1730,7 @@ imported normally. The list of loaded modules can change between Python versions | `workspace_exec_entry` | `str \| None` | `None` | Relative path (from workspace root) to an entry-point file to `exec` at startup. | | `sandbox_timeout` | `float` | `1.0` | Timeout for sandbox startup (increase for larger projects). | | `import_whitelist` | `list[str] \| None` | `None` | Allowed imports. `None` = allow all. | -| `import_blacklist` | `list[str] \| None` | default list | Blocked imports. | +| `import_blacklist` | `list[str] \| None` | default list | Blocked imports and functions. | | `helper_modules` | `list[str] \| None` | `None` | Modules from the tests folder to make importable. See [Helper Modules](#helper-modules). | ### Production vs. Local Development @@ -1636,6 +1785,7 @@ autograder_config = ConfigObject( import pytest from pytest_prairielearn_grader import ConfigObject +from pytest_prairielearn_grader.assertions import assert_equal, assert_true from pytest_prairielearn_grader.fixture import FeedbackFixture, WorkspaceFixture autograder_config = ConfigObject( @@ -1650,30 +1800,31 @@ def test_add(workspace_sandbox: WorkspaceFixture, feedback: FeedbackFixture) -> feedback.set_score(0.0) response = workspace_sandbox.query_function_raw("calculator.add", 2, 3) - assert response["status"] != "not_found", "calculator.add is not defined" + assert_true(response["status"] != "not_found", msg="calculator.add is not defined") feedback.set_score(0.5) - assert response["status"] == "success", ( - f"calculator.add raised {response['exception_name']}: {response['exception_message']}" + assert_true( + response["status"] == "success", + msg=f"calculator.add raised {response['exception_name']}: {response['exception_message']}", ) feedback.set_score(0.8) - assert response["value"] == 5, f"Expected 5, got {response['value']}" + assert_equal(response["value"], 5, description="calculator.add(2, 3)") feedback.set_score(1.0) @pytest.mark.grading_data(name="divide() raises ValueError on zero", points=2) def test_divide_guard(workspace_sandbox: WorkspaceFixture) -> None: response = workspace_sandbox.query_function_raw("calculator.divide", 10, 0) - assert response["status"] == "exception", "divide() should raise on zero denominator" - assert response["exception_name"] == "ValueError" + assert_true(response["status"] == "exception", msg="divide() should raise on zero denominator") + assert_equal(response["exception_name"], "ValueError") @pytest.mark.grading_data(name="PI_APPROX constant exists", points=1) def test_constant(workspace_sandbox: WorkspaceFixture) -> None: pi = workspace_sandbox.query("calculator.PI_APPROX") - assert isinstance(pi, float), "PI_APPROX should be a float" - assert abs(pi - 3.14) < 0.01 + assert_true(isinstance(pi, float), msg="PI_APPROX should be a float") + assert_true(abs(pi - 3.14) < 0.01) ``` **Student workspace (`student/calculator.py`):** @@ -1689,3 +1840,27 @@ def divide(a, b): raise ValueError("Cannot divide by zero") return a / b ``` + +## More Examples + +To see more examples of what is possible in these test files, look at the test cases in +[this folder](https://github.com/prairielearn/grader-python-v2/tree/main/tests/scenario_root). Each test file is called `scenario.py`. + +## Tips and Best Practices + +1. **Keep code in the tests**: See [Where Code Belongs](#where-code-belongs). +2. **Use descriptive test names**: The `name` in `@pytest.mark.grading_data` is shown to students. +3. **Use friendly messages that don't give away answers**: Use the assertion helpers with the + `friendly` output level, and `assert_true` for hidden test cases. +4. **Test incrementally**: Break complex problems into smaller tests with partial credit. +5. **Prefer `parametrize`**: Use `@pytest.mark.parametrize` for several inputs instead of a loop, + unless the inputs share partial credit. +6. **Control output visibility**: Use `include_stdout_feedback=False` for tests where student output + would be confusing or reveal answers. +7. **Set appropriate timeouts**: Student code that imports numpy or pandas usually needs a + `sandbox_timeout` of several seconds. +8. **Use `module_sandbox` sparingly**: Only when you need persistent state across tests. +9. **Watch for `__main__` blocks**: Student code runs with `__name__ == "__main__"`, so code under + `if __name__ == "__main__":` runs during grading. +10. **Configure security restrictions**: Use `import_whitelist`, `import_blacklist` and + `builtin_whitelist` to control what modules and functions students can access.