Skip to content

README.md

Latest commit

 

History

History
125 lines (100 loc) · 6.89 KB

README.md

File metadata and controls

125 lines (100 loc) · 6.89 KB
Raw

OkPy Backups CLI

Setup

  1. Install uv
  2. cd into this directory (src/backups)
  3. Run uv sync to install dependencies and create the virtual environment (.venv/)
  4. Optional, only if you wish to run the request command to request OkPy backups: Create a .env file (follow the .env-template):
    • Update the .env file with your OkPy access token. You must have staff permissions for the course you want to query and you must periodically re-request your token.

Note

In VS Code, you may need to select your Python interpreter manually to get language features in your editor. Alternatively, open the src/backups directory in its own VS Code window so that VS Code can detect the correct environment (.venv).

Commands

cd src/backups

# Extract student emails from Gradescope .csv roster
uv run python3 main.py emails

# Make HTTP requests to OkPy server to create .json dump of
# student emails to the backups for all of their assignments
uv run python3 main.py request

# Given the .json dump, store the file contents of the backups
# locally and also write the backup metadata to a sqlite database
uv run python3 main.py store

# Compute lint_errors table given .json output of running
# ruff linter on backup files.
# STEP 0: Configure ruff as you wish: https://docs.astral.sh/ruff/configuration/

# STEP 1: Run ruff, replacing $PATH_TO_DIR and $PATH_TO_OUTPUT_JSON with your values
uvx ruff check ../../data/private/$PATH_TO_DIR --output-format json --output-file ../../data/private/$PATH_TO_OUTPUT_JSON

# STEP 2: Run the lint command
uv run python3 main.py lint

# Compute num_lines table given that backup files
# have been stored already. This table stores the number of lines
# for each file in each backup
uv run python3 main.py backup-file-metadata

Tip

If you get tired of prefixing all commands with uv run you can activate and deactivate the virtual environment manually with source .venv/bin/activate and deactivate, respectively.

Run --help with any of the commands for more information.

Create a configuration file to save yourself the effort of typing a bunch of CLI arguments. An example can be found in ./configs/dev/backup_config.json. All fields are required (e.g. they must either be provided in the config or via the CLI). If you provide both a config and CLI arguments, the CLI arguments will override anything in the config.

Uploading backup files to AWS S3

Once you have run the store command, there should be files within the data/private/ directory that you can upload to AWS S3 to be retrieved later by the web app. We recommend one of two methods:

  1. Manually upload files to the S3 console (fast if you have < 10 files to upload, very slow otherwise)
  2. Automatically upload files through the AWS CLI

For both methods, you can refer to the documentation linked above. To save yourself some reading, here is an example of the commands you would need to run for method 2, assuming you have already configured and authenticated through the AWS CLI:

  1. cd into the folder that you want to upload, replacing $FILE_PATH with your desired path, e.g. cal/cs88/fa25/ants:
cd data/private/$FILE_PATH
  1. Run the following command to synchronize the contents of the folder you are currently inside to the folder in our AWS S3 bucket, replacing $BUCKET_NAME with your desired bucket (ucb-assignment-snapshots-eae254943a2c4f51bef67654e99560dd) and $FILE_PATH with your desired path, e.g. cal/cs88/fa25/ants:
aws s3 sync . s3://$BUCKET_NAME/$FILE_PATH

Note

We recommend keeping the $FILE_PATH the same for steps 1-2 above for consistency, although technically they can differ.

Dumping database from OkPy Backups CLI into Rails database

  1. Optional if not done already: Run backups CLI command(s) in this directory to create or update $OUTPUT_DB_NAME.db. If you are an internal contributor working with the dev dataset from data.zip, skip this step.
  2. Create .sql dump of output .db file. Replace $PATH_TO_DB_FILE and $PATH_TO_SQL_FILE with values of your choice in the root directory of the repository:
# General command
sqlite3 $PATH_TO_DB_FILE .dump > $PATH_TO_SQL_FILE

# Example using the dev dataset
sqlite3 data/private/out/dev/c88c_fa25.db .dump > data/private/out/dev/c88c_fa25.sql
  1. Update the SQL file:
    1. Remove ../../data/private/ prefix from paths. IMPORTANT: Make sure you are removing the trailing /.
    2. Remove/comment out CREATE TABLE statements since that will interfere with the Rails database migrations (Rails will already handle table creation on its own end, so if you have a duplicate CREATE TABLE statement Rails will error).
    3. Also remove CREATE INDEX statements (which may have been created through code written in src/notebooks)
  2. Optional if not done already: Generate corresponding Rails model(s) in the src/snapshots-app directory by running the following command. If you are an internal contributor working with the toy data from data.zip, skip this step.
rails generate model <model_name> <column_name:data_type> ...

Caution

THE FOLLOWING STEP WILL RESET (e.g. delete everything) AND RE-MIGRATE THE RAILS DB. BE CAREFUL!

  1. Run the following command in the src/snapshots-app directory:
rails db:migrate:reset
  1. Run the following command in the root directory of the repository to execute commands from output .sql dump into the Rails app development.sqlite3 database. Replace $PATH_TO_SQL_FILE with the same value from steps 1 and 2:
# General command
sqlite3 src/snapshots-app/storage/development.sqlite3 < $PATH_TO_SQL_FILE

# Example with dev dataset
sqlite3 src/snapshots-app/storage/development.sqlite3 < data/private/out/dev/c88c_fa25.sql
  1. Run the following command in the src/snapshots-app directory to seed the Rails database with hardcoded initial data:
rails db:seed
  1. Verify that your data has been loaded properly in the Rails console by using the ActiveRecord query interface. For example, run the following commands in the src/snapshots-app directory:
$ rails c
snapshots-app(dev)> BackupMetadatum.all # view data and check it looks correct, press q to exit long list
snapshots-app(dev)> OkpyMessage.all # view data and check it looks correct
snapshots-app(dev)> exit