HiPerGator-RV Technical Details

The high-level view of the architecture is illustrated in Figure 1. HiPerGator-RV is completely separate from other computing infrastructure in the data center so that compromise of nearby systems cannot lead to a compromise of the systems in the data enclave. HiPerGator-RV systems sit in separate racks, with separate network switches and connections.

The general client computer is not trusted to hold ePHI data. Users can access HiPerGator-RV only through a secure SSL network connection after GatorLink authentication and all traffic will be carried encrypted to the HiPerGator-RV gateway system over all traversed networks. The gateway system provides further authentication through a secret key created for each user upon first use of the HiPerGator-RV system. The HiPerGator-RV gateway then establishes a secure, high-performance virtual desktop session with a VM running inside the vault. The system ensures that the researcher can use applications to analyze authorized data.

Interactive work on the data is done from VMs running on infrastructure inside the vault. The size of the VM can be controlled by the developers and researchers so that applications that need multiple cores or large amounts of memory have the resources to run efficiently. Very large workloads and complex, multi-task work flows can be run interactively or asynchronously on the HPC resources inside the enclave. The HPC resources have been configured to support high-performance access to the data storage system.

The initial configuration of the HiPerGator-RV system has 200 cores from Intel E5-2670v2 processors with 6.4 GB of RAM per core. It has a fast primary storage system connected by InfiniBand of 1.2 PetaByte and a 400 TeraByte secondary storage system that keeps the second copy as a replica of all user data. User data, all encrypted, is backed up daily to the tape robot.

The system configuration is very specialized and highly locked down. The configuration of the nodes is generated automatically and changes go through a rigorous process.

VIRTUAL MACHINE STARTUP SEQUENCE

1. A small bootstrap service installed on the VM downloads a signed configuration file and controller executable from a known location on the local network.
   1. The purpose of the bootstrap process is to allow updates to the configuration or controller software to be deployed without requiring a rebuild of the virtual machine image.
   2. The configuration file and controller executable must have a valid signature with a certificate chain rooted at a Tera Insights certificate baked into the bootstrap service executable.
   3. If signature verification fails, the startup sequence aborts and the virtual machine is destroyed.
2. The bootstrap service generates an RSA key-pair and registers with the ResVault backend server listed in the signed configuration file.
   1. Upon successful registration, the bootstrap service receives in response information about the user that owns the virtual machine, including that user’s public key. Similarly, the user is given the VM’s public key, which was included in the registration request.
3. The bootstrap service starts the controller executable and hands over the private key and configuration.
4. The controller connects to the HiPerGator-RV backend and waits for WebSocket connections from the user.

SECURE WEBSOCKET COMMUNICATION CHANNEL

1. Client connects to the HiPerGator-RV backend, which joins the connection to the VM.
   1. The connecting user must be either the VM’s owner or authorized by the owner to connect to that VM, otherwise the connection will be rejected before it gets to the VM.
2. The client and VM establish an encryption layer on top of the WebSocket connection (in addition to TLS for each leg of the connection).
   1. Key negotiated using Ephemeral-Key Diffie-Hellman, with each side’s ephemeral public key signed by the side’s RSA key.
   2. AES-GCM used as underlying cipher.
3. Communication channel then used to control VM, transfer file and drive encryption keys, etc.

ENCRYPTED DRIVES

In order for the VM to perform efficient computation, it needs access to fast, secure storage. This is provided in the form of encrypted drives. The drive encryption keys are generated and manipulated in the same manner as the file keys. The user-VM encrypted channel is used to transmit such keys to the VM during the start-up sequence.

The disk encryption happens inside the VM, under control of the user that owns the VM, not the administrators of the system. The Controller utilizes widely available and tested disk encryption solutions, like BitLocker and LUKS, to handle the disk-level encryption.

PROCESS FOR SECURE FILE SHARING WITHIN RESVAULT

• Each file, upon creation, is encrypted with a unique, independently generated AES-256 key.
• These keys are then encrypted with the public keys of the users (and groups) that have access to the file and are then stored on the server fully encrypted, unable to be accessed unless decrypted and downloaded by a user with a permissible private key.
• Those that have been given the permissions to share the file (or add others to groups) are then able to add the public keys of users they wish to share the file with, giving those users the ability to decrypt and download the file from the server with their private keys.
• Even if attackers are able to gain access to the servers where the files are stored, the files will remain worthless given the attackers will not have the proper private keys to decrypt the files before downloading.