Server Configuration

Table of Contents

1) Install uWSGI

The default catsoop installation doesn't install uWSGI, which is our recommended server for public-facing catsoop instances. You can install it with:

$ pip3 install uwsgi

2) Check Web Settings

Before going any further, double-check the following values in ~/.config/catsoop/config.py, which was automatically generated by catsoop configure:

• cs_url_root is the URL of the root of the CAT-SOOP installation.
• cs_checker_websocket tells CAT-SOOP where clients can make websocket connections to the checker.

For example:

cs_url_root = 'https://example.com/catsoop'
cs_checker_websocket = 'wss://example.com/reporter'

Typically, on a public-facing server, cs_url_root will start with https, and cs_checker_websocket will start with wss.

By default, running catsoop start will start several processes. The most important are the uWSGI server (default port 7667) and the websocket server (default port 7668). You can change these ports by setting additional variables cs_wsgi_server_port and cs_checker_server_port, respectively, in your config.py.

3) Install NGINX and SSL Certificate

Unless you have a different preferred method, we recommend using the NGINX reverse proxy to route requests to the catsoop server. On Debian systems, you can install NGINX with:

$ sudo apt install nginx

You will also want an SSL certificate. SSL/TLS Certificates are available gratis from Let's Encrypt, which is our recommended method for getting a certificate, even for servers running at MIT. Debian provides convenient packages, which can be installed with:

$ sudo apt install certbot python3-certbot-nginx

You can then acquire a certificate by running the following, replacing example.com with your domain name:

$ sudo certbot --nginx -d example.com

This will also set things up so that the certificates auto-renew when necessary.

4) Configure NGINX

Now that certificates have been set up, we will configure NGINX to redirect relevant traffic to the web server and the websocket server.

I usually do this by editing /etc/nginx/sites-available/default to include something like the following content, which will configure NGINX to route certain requests to CAT-SOOP. You can, of course, customize the endpoints (/catsoop and /reporter in the example below) to change the base URL for both the WSGI server and the websocket server.

    # the following will route requests to https://example.com/catsoop
    # to the uWSGI server.  change "catsoop" in the following lines if you
    # want to use a different URL.
    location /catsoop {
            rewrite /catsoop/?(.*) /$1 break;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_cache_bypass $http_upgrade;
            proxy_pass http://localhost:7667/;
    }

    # the following will route websocket requests to
    # wss://example.com/reporter to CAT-SOOP's websocket server.
    location /reporter {
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_cache_bypass $http_upgrade;
            proxy_pass http://localhost:7668/;
    }

Once you have updated the configuration file, you can check that it is valid with:

$ sudo nginx -t

and, if no errors are reported, you can then restart NGINX with:

$ sudo service nginx restart

4.1) (Optional) Configure NGINX to Serve Static Files

Using only the configuration above, CAT-SOOP will handle serving static files in the various __STATIC__ directories throughout the catsoop codebase and the courses it serves. This can be slow, especially for big files. You can configure things so that NGINX will serve those files by adding the following snippet and then restarting NGINX.

        location ~ ^/_static/_base/(?<path>.*)/(?<filename>[^\/]*) {
           root /home/catsoop/catsoop/catsoop/__STATIC__;
           try_files /$path/$filename =404;
        }

        location ~ ^/_static/_handler/(?<handler>[^\/]*)/(?<path>.*) {
           root /home/catsoop/catsoop/catsoop/__HANDLERS__;
           try_files /$handler/__STATIC__/$path =404;
        }
        location ~ ^/_static/_auth/(?<auth>[^\/]*)/(?<path>.*) {
           root /home/catsoop/catsoop/catsoop/__AUTH__;
           try_files /$auth/__STATIC__/$path =404;
        }
        location ~ ^/_static/_qtype/(?<qtype>[^\/]*)/(?<path>.*) {
           root /home/catsoop/catsoop/catsoop/__QTYPES__;
           try_files /$qtype/__STATIC__/$path =404;
        }

        location ~ ^/_static/_plugin/(?<plugin>[^\/]*)/(?<path>.*) {
           root /home/catsoop/.local/share/catsoop/plugins;
           try_files /$plugin/__STATIC__/$path =404;
        }

        location ~ ^/_static/(?<course>[^\/]*)(?<path>.*)/(?<filename>[^\/]*) {
           root /home/catsoop/.local/share/catsoop/courses;
           try_files /$course$path/__STATIC__/$filename /$course/__STATIC__$path/$filename =404;
        }

5) (Optional) Additional Configuration

5.1) Web Server

By default, public-facing CAT-SOOP instances use uWSGI to serve content. The number of worker processes spawned by uWSGI are controlled by the cs_wsgi_server_min_processes and cs_wsgi_server_max_processes variables.

You may also consider adding a cs_wsgi_server_worker_max_requests variable to your config.py. This variable causes uWSGI to restart each worker process after it has served some number of requests. For example, setting cs_wsgi_server_worker_max_requests = 10000 will cause each worker to restart after serving 10000 requests. This option is particularly useful in courses that do a lot of importing of numpy, matplotlib, etc, as a heavy-handed way of managing memory leaks.

5.2) Checker

By default, CAT-SOOP's checker will run at most 1 check at a time. If you have the resources available, you can configure the checker to run multiple checks in parallel by setting cs_checker_parallel_checks to a larger (integer) number in your config.py. Higher values cause wait times in the checker queues to go down, but the checks may run more slowly, and their timing may be less reliable.

6) Start CAT-SOOP

At this point, you could run catsoop start to start a webserver. However, on a public-facing server, it is important to start catsoop in such a way that it won't die when you hang up. One option is to run catsoop start inside of a GNU screen or a tmux session.

6.1) Starting catsoop via systemd Service

Another option, though, would be to set up a new systemd service to run catsoop. If you want to go that route, you can make a new file called /etc/systemd/catsoop.service with something like the following (customized to your own use-case if need be):

[Unit]
Description=CAT-SOOP
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/local/bin/catsoop start
User=catsoop
KillMode=mixed
TimeoutSec=10

[Install]
WantedBy=multi-user.target

Then running the following will start catsoop:

$ sudo service catsoop start

If you make any configuration changes or upgrade your catsoop, you can restart it with:

$ sudo service catsoop restart

And, importantly, the following will cause catsoop to start automatically when the machine is booted up:

$ sudo systemctl enable catsoop.service

7) Try It Out!

With all of the above done, direct your web browser to your cs_url_root and you should now see the CAT-SOOP default page!

8) (Optional) Configure Backups

All of CAT-SOOP's data are stored in files on disk in a directory called _logs in the cs_data_root location specified above (by default, ~/.local/share/catsoop). CAT-SOOP itself will not back these files up, but there are many strategies for backups using common utilities.

I have used many approaches in the past, but my usual approach involves setting the _logs directory up as a Git repository, and then setting up a cron job to commit all files in that repository and push to several locations. This approach has several advantages over simply using rsync or scp to copy the folder to a remote machine. In particular, it allows you to roll back to any past backup while keeping size down by only storing diffs (instead of storing a complete copy of each file for each backup).

Here, we'll set up a backup using Git. To set this up, first move yourself to the _logs directory and run git init, followed by git add -A. This will set your _logs directory up as a Git repository. You can then set up a cron job to commit all changes and push these changes to an arbitrary number of backup locations (local or remote).

Something like the following example script (/home/catsoop/do_backup.sh) has been used by several classes since around 2018. It commits local changes to a Git repository, and it then pushes those changes to one local location (on a separate disk) and to one remote location (and, of course, it can be edited to push to arbitrarily-many locations).

#!/bin/bash
cd /home/catsoop/.local/share/catsoop/_logs;
git add -A;
git commit -m "$(date +'%Y-%m-%d:%H:%M')";
git push /storage/logs_backup main;
git push backups@catsoop.org:py_backup main;

It can then be configured to run, for example, every hour at xx:05 and xx:35 with the following crontab entry:

5,35 * * * * /usr/bin/flock -n /tmp/backup.lockfile /home/catsoop/do_backup.sh 2>&1 >/dev/null

9) (Optional) Set Up Local Python Sandbox

By default, Python code that needs to be sandboxed (for example, student code from the pythonic or pythoncode question types) will be sent to sandbox.catsoop.org to be run. It is fine to leave things this way if you'd like. I will keep that service up as long as is feasible, and the sandbox doesn't log anything about the code it runs. That said, you may also wish to set things up so that the code runs on your machine. The main benefit of this approach is that you don't have to rely on an external service (network issues or our server's downtime won't affect you, and you have a sandbox to yourself instead of having to share with others) and that your machine is likely more powerful than the public sandbox (and you have it all to yourself).

Our recommended sandboxing approach involves creating a Python virtual environment to run student code, and limiting that interpreter's permissions using AppArmor and bubblewrap. This approach will largely isolate the student code from the system on which it is running, and it will also limit other resources (memory usage, etc).

9.1) Installing Necessary Software

Most of the tools we need should already be installed by this point, but we'll need a couple more:

$ sudo apt install apparmor apparmor-utils python3-venv

You'll also need to install bubblewrap. The version of bubblewrap that is available in the Debian repositories does not support some of the features we want to use, so you should compile from source. You can do so with the following sequence of commands (on Debian):

$ sudo apt build-dep bubblewrap
$ git clone https://github.com/projectatomic/bubblewrap
$ cd bubblewrap
$ ./autogen.sh
$ make
$ sudo make install

This will make an executable called bwrap, which our sandbox will use.

On Debian, you will also need to set a kernel parameter to allow unprivileged users to create new user namespaces:

$ sudo sysctl kernel.unprivileged_userns_clone=1

You should also set kernel.unprivileged_userns_clone=1 in /etc/sysctl.conf so that it persists across reboots.

9.2) Virtual Environment

Now that we have all of the necessary software, we'll set up a virtual environment. The sandboxed code will be run in this environment. Pick a location (one that is readable by the user running the web server) and create a new virtual environment there with the Python interpreter you want the checkers to use. In example below, we'll use the /usr/bin/python3 interpreter, and we'll set up the virtual environment in /home/catsoop/python/sandbox.

$ /usr/bin/python3 -m venv --copies /home/catsoop/python/sandbox

If you want to use a different Python version as the basis for the virtual environment, just change the first argument.

9.2.1) Installing Packages to the Sandbox

If you would like your checkers to be able to use any packages outside the standard library, you can install them in the virtual environment using the pip executable within the virtual environment. For most packages, you can simply use the pip module from this new virtual environment to install them. For example, to make pillow available within the sandbox, we could use:

$ /home/catsoop/python/sandbox/bin/python3 -m pip install pillow

However, some packages require special care when installing. For example, numpy normally uses multiple processes when computing its results. However, a desirable feature of the sandbox is that it prevents student code from launching new processes of any kind. To get around this, it is possible to compile numpy for the sandbox with all optimizations disabled, for example:

$ sudo apt build-dep python3-numpy
$ wget https://files.pythonhosted.org/packages/54/a4/f8188c4f3e07f7737683588210c073478abcb542048cf4ab6fedad0b458a/numpy-2.1.0.tar.gz
$ tar xvfa numpy-2.1.0.tar.gz
$ cd numpy-2.1.0
$ BLAS=None LAPACK=None ATLAS=None /home/catsoop/python/sandbox/bin/python3 setup.py install

9.3) AppArmor

Debian comes with AppArmor enabled by default, but you'll need to set up an AppArmor profile to limit your virtual environment's Python interpreter. Create a file /etc/apparmor.d/py3sandbox containing the following, but replacing /home/catsoop/python/sandbox/bin/python3.10 with your sandbox interpreter's location (if it is different), and tuning some of the other parameters if necessary:

#include <tunables/global>

/home/catsoop/python/sandbox/bin/python3.10 {
    /** wrix,

    set rlimit nproc <= 0,
    set rlimit fsize <= 1M,
    set rlimit as <= 500M,
}

This file does a couple of things:

• It allows access to the entire filesystem. This might seem dangerous, but we'll use bwrap to handle the filesystem sandboxing (though you can modify the entries above to further restrict things).
• It also introduces two resource limits:
  • student code will not be allowed to spawn any new processes
  • student code cannot write more than 1MB of data to files
  • student code will not be allowed to use more than 500MB of memory

All of these parameters are tunable, and other resources can also be limited, as documented here.

Finally, enable the profile with the following command:

$ sudo aa-enforce /etc/apparmor.d/py3sandbox

You can then test your setup by running the Python interpreter (in our example, /home/catsoop/python3_sandbox/bin/python3) and trying to write more than 1M of data to a file:

with open('/tmp/test', 'w') as f:
    f.write('a'*(1204**2+1))

This should produce an error, since this interpreter is not allowed to write that much data to disk.

$ sudo aa-disable /etc/apparmor.d/py3sandbox

$ sudo aa-enforce /etc/apparmor.d/py3sandbox

9.4) CAT-SOOP Configuration

Now that we have those pieces set up, we'll need to configure CAT-SOOP to use this new sandbox.

Add the following to the top-level preload.py of any course that you want to use the sandbox (so that all pages in the course inherit it), substituting your own values where appropriate:

csq_python_sandbox_type = 'bwrap'

# the following must match the line in /etc/apparmor.d/py3sandbox exactly
csq_python_sandbox_interpreter = '/home/catsoop/python/sandbox/bin/python3.10'

csq_bwrap_extra_ro_binds = [('/home/catsoop/python', '/home/catsoop/python')]

The csq_bwrap_extra_ro_binds variable tells bubblewrap to mount certain directories from the base system on the virtual filesystem available to the student's code in rea-only mode. In our case, it is necessary to include the directory from which our Python executable is available.

And that's it! It is worth runing a few tests as submissions to a pythoncode question after implementing this, to make sure things are working properly. For example, I would usually try to:

• call os.fork() and/or use the subprocess module to start a child process
• write too much data to a file
• list the files in a directory not included in the sandbox (e.g., someone's home directory)
• use too much memory
• cause an infinite loop

If the system properly stops the code from running in all of the examples above but works for a correct solution, then you're probably in good shape!