System Judge Proposal

Purpose

In some programming courses, we are used to evaluate students' code using an online judge system like DOMJudge. An online judge is a system that, upon receiving a submission, compiles and runs the program, provides specific input, checks the output, and then gives a result such as "Accepted" or "Wrong Answer." This automated process is fairer and more cost-effective than manual grading.

To improve the efficiency of learning system administration and implementation skills, we should introduce a similar mechanism: allowing students to complete projects, which are then evaluated by an automated judging system.

Basic Concepts about System Judge

We have multiple hosts in our system:

• Judge Server: Hosts the API server and components such as the VPN controller, Judge controller, and more.
• Common Infrastructure: Provides essential infrastructure and services necessary for the judging process.
• Kubernetes Cluster: Executes the tests.
• Container Registry: Stores the images we would use for judging. Here, we use Harbor.

We provide users with VPN certificates to facilitate NAT traversal. Once a test submission is received, a pod is created within the Kubernetes cluster to test the user's environment.

For network isolation, we make each VPN certificate connect to instance running in different VRF.

Workflow 1 - Create a problem

When an administrator submits a new problem, they must provide some basic information (such as the problem's name, deadline, status, etc.), subtasks (including a shell script for each subtask, the point value, and any dependencies between subtasks), and any other necessary files. The judge API server's Docker service will then build a container image that includes all these files and upload it to a remote registry.

Workflow 2 - Submit a test request

The work allocator assigns pending submissions to idle workers. The assigned worker is responsible for interacting with Kubernetes and the database to process the submission.

Concept 3 - Judging pod connects to user machine

Whenever a VPN certificate is requested, the service on the judge API server generates two configuration files: one for the user, and one for the judging pod. Due to VPN limitations, the judging process for a single submission cannot be parallelized.

Implementation

Network: Isolation

Although only VRFs were mentioned above, network namespaces are also used for network isolation. We use two veth pairs to forward traffic: one between and the namespace, and another between the namespace and the global network stack. We avoid using only VRFs because this setup would require multiple NAT operations, while network namespaces alone are insufficient because the WireGuard instance must run on the global network stack. Thus, access to the common infrastructure and network isolation can be achieved at the same time.

Database: MongoDB

Because of the complexity of the data, we finally used the NoSQL database instead of SQL. It could simplify a lot of works.

API Server: Flask

We applied the factory pattern to instantiate our application components. The main architecture follows the model-view-controller (MVC) pattern. Additionally, we incorporated the repository pattern, and the service pattern.