Skip to main content
Version: 1.10

Vault

This tutorial guides you through deploying a Vault as a confidential deployment by using Contrast's built-in Vault support.

Vaults are an identity-based secrets and encryption management systems, which provide encryption services that are gated by authentication and authorization methods to ensure secure, auditable and restricted access to secrets, such as API keys, passwords, encryption keys or certificates.

There are several implementations of Vault systems, the most prominent being HashiCorp Vault and its open-source derivative, OpenBao. This example utilizes the Vault image provided by OpenBao. The associated APIs exposed by the Contrast coordinator are fully compatible with both HashiCorp Vault and OpenBao, ensuring interoperability across implementations.

Contrast can lift a Vault deployment into a confidential computing environment, further shielding the secrets from the workload operator.

Sealing and Unsealing of Vaults​

Sealing ensures that all sensitive data within the Vault remains inaccessible and protected when the system isn't in active use. It provides a security boundary that prevents unauthorized access during restarts or shutdowns.

Unsealing is a manual process required to transition Vault into an operational state, allowing authorized access to stored secrets. Vault implementations by default use a set of unseal keys derived from a master key, building up on Shamir's Secret Sharing scheme. Further to auto-unseal Vaults, the process can be delegated to another already initialized Vault by using an exposed transit secrets engine API as the unsealing mechanism.

Transit secrets engine API of Contrast Coordinator​

To automate the unsealing process in confidential deployments of Vault instances, the coordinator exposes a compatible transit secrets engine API on port 8200. Vault deployments can be configured to integrate this transit engine to enable auto-unsealing, ensuring immediate operational readiness and seamless integration within the secure Contrast environment.

Secure endpoints with mutual TLS​

All communication between the transit secrets engine API and Vault is secured through mutual TLS (mTLS). Only entities presenting a mesh certificate signed by the current mesh CA key are trusted. The Coordinator issues itself a valid certificate at the time of the transit secrets engine API call, while the Vault deployment obtains its certificate in the initialization phase, after attesting to the Coordinator.

Role of workloadSecretID​

To support persistence in the auto-unsealing process, the workloadSecretID is used to derive the encryption key utilized by the transit secrets engine. Beyond key derivation, the workloadSecretID also plays a critical role in authorization.

Access to a specific encryption key via the transit secrets engine API is permitted only if the requested key name matches the workloadSecretID embedded in the corresponding certificate extension of the Contrast mesh certificate. This ensures that each entity can only access their own set of encryption keys within the transit secrets engine.

For more details on how the workload secret is used, see Workload Secrets.

Prerequisites​

  • Installed Contrast CLI
  • A running Kubernetes cluster with support for confidential containers, either on AKS or on bare metal

Steps to deploy Vault with Contrast​

Download the deployment files​

The Vault deployment files are part of the Contrast release. You can download them by running:

curl -fLO https://github.com/edgelesssys/contrast/releases/download/v1.10.0/vault-demo.yml --create-dirs --output-dir deployment

Deploy the Contrast runtime​

Contrast depends on a custom Kubernetes RuntimeClass, which needs to be installed to the cluster initially. This consists of a RuntimeClass resource and a DaemonSet that performs installation on worker nodes. This step is only required once for each version of the runtime. It can be shared between Contrast deployments.

kubectl apply -f https://github.com/edgelesssys/contrast/releases/download/v1.10.0/runtime-aks-clh-snp.yml

Download the Contrast Coordinator resource​

Download the Kubernetes resource of the Contrast Coordinator, comprising a single replica deployment and a LoadBalancer service. Put it next to your resources:

curl -fLO https://github.com/edgelesssys/contrast/releases/download/v1.10.0/coordinator.yml --output-dir deployment

Generate policy annotations and manifest​

Run the generate command to generate the execution policies and add them as annotations to your deployment files. A manifest.json file with the reference values of your deployment will be created:

contrast generate --reference-values aks-clh-snp deployment/
Runtime class and Initializer

The deployment YAML shipped for this demo is already configured to be used with Contrast. A runtime class contrast-cc was added to the pods to signal they should be run as Confidential Containers. During the generation process, the Contrast Initializer will be added as an init container to these workloads. It will attest the pod to the Coordinator and fetch the workload certificates and the workload secret.

Deploy the Coordinator​

Deploy the Coordinator resource first by applying its resource definition:

kubectl apply -f deployment/coordinator.yml

Set the manifest​

Configure the Coordinator with a manifest. It might take up to a few minutes for the load balancer to be created and the Coordinator being available.

coordinator=$(kubectl get svc coordinator -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "The user API of your Contrast Coordinator is available at $coordinator:1313"
contrast set -c "${coordinator}:1313" deployment/

The CLI will use the reference values from the manifest to attest the Coordinator deployment during the TLS handshake. If the connection succeeds, it's ensured that the Coordinator deployment hasn't been tampered with.

Deploy Vault​

Now that the Coordinator has a manifest set, which defines the Vault deployment as an allowed workload, we can deploy the application:

kubectl apply -f deployment/

The Vault deployment is defined as a StatefulSet using the OpenBao Vault image, with a mounted block device for persistent storage. A Contrast Initializer, running as an init container, uses the workload secret located at /contrast/secrets/workload-secret-seed to generate an encryption key and initialize the block device as a LUKS-encrypted partition. Before the Vault container starts, the Initializer opens the LUKS device using the generated key. This unlocked device is then mounted by the Vault container and used as the backend storage volume. For the Vault application, this process is entirely transparent, and the device behaves like a standard volume mount. It’s important to clarify that the LUKS encryption of the block device is primarily a convenience feature, enabling persistent storage at the filesystem level on confidential virtual machines. The primary and security-relevant encryption mechanism remains Vault’s own sealing process, which provides cryptographic protection of secrets even if the underlying storage is compromised.

Because the workload-secret-seed is derived from the associated workloadSecretID, any change to the workloadSecretID after the block device has been initialized will result in deriving an invalid encryption key, making the mounted block device undecryptable.

Inter-deployment communication

The Contrast Coordinator issues mesh certificates after successfully validating workloads. These certificates can be used for secure inter-deployment communication. The Initializer sends an attestation report to the Coordinator, retrieves a service mesh certificate bound to it's provided public key, containing the certificate chain, as well as the current mesh CA cert. The Initializer then writes them to a volumeMount, allowing to build up the secure mTLS connections based on the service mesh. The received service mesh certificate also holds the certificate extension of the workloadSecretID, which is used to allow the authorization to a certain encryption key on the transit engine API.

The Vault's TCP listener is configured to accept connections only from mesh certificates issued under the same CA state used to sign the Vault’s own certificate, effectively restricting communication to attested Contrast deployments. Because updating the workloadSecretID after initializing the LUKS device will make it inaccessible, it's critical to ensure that the workloadSecretID is correctly aligned with the intended endpoint specified in Vault’s sealing configuration before the first contrast set is executed. This can be achieved by using the corresponding workload-secret-id annotation to directly overwrite the manifest with the required workloadSecretID.

Connecting to the application​

Other confidential containers can securely connect to the Vault server via the Service Mesh. As previously noted, access to the Vault endpoint is restricted to peers that present a service mesh certificate valid under the currently set manifest. While such a certificate enables mTLS-based communication with the Vault server, it doesn't, on its own, grant authorization to perform Vault-related operations. Permissions for accessing secrets within Vault must be explicitly configured using the root token obtained during Vault initialization. The configured openbao-client deployment is responsible for executing Vault-related operations, including initialization, secret creation, and sealing instructions.

For more information on the Vault management and administration, please follow the official OpenBao documentation.

Restarting the Vault deployment with auto-unsealing​

If you want to make changes to your resources, ensure that the workloadSecretID of the Vault remains unchanged. You can restart the Vault using:

kubectl rollout restart statefulset/vault

When a new Vault pod starts, it runs the Contrast Initializer as part of its startup sequence. The Initializer receives the same workload secret as before, allowing it to derive the correct encryption key and unlock the existing LUKS-encrypted block device. This process ensures that the Vault backend can reattach the previously encrypted volume and access all stored data transparently. However, while this step enables access to the filesystem-level storage, it doesn't unlock access to the actual secrets. Once Vault has been initialized, subsequent restarts rely on the auto-unsealing process, which is triggered via the transit secrets engine API provided by the Coordinator.

You can verify that the auto-unsealing process completed successful by inspecting the logs of the openbao-server container of the Vault pod:

kubectl logs vault-0 -c openbao-server

The log entries will indicate that the Vault has transitioned into unsealed state.