Setting up Vault HA with CockroachDB

Frankly most of this was pretty easy and not much to write about… auto unseal is a bit tricky to setup freely

Vault is well designed tool for secret management, I use it for mostly anything. I’m only able to achieve cross-region HA with until my recent rediscovery of CockroachDB. After a bit of grinding, I got this:

-> % vault status
Key                      Value
---                      -----
Seal Type                ocikms
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.17.3
Build Date               2024-08-06T14:28:45Z
Storage Type             cockroachdb
Cluster Name             vault-cluster-*******
Cluster ID               37aeb87d-7803-b1db-18f1-94becf334bcc
HA Enabled               true
HA Cluster               https://****.****.ts.net:8201
HA Mode                  active
Active Since             2024-08-23T23:57:41.294146452Z

Setup

CockroachDB part is very simple, I just used their free Cloud offering. Vault does a lot of INSERTS even “idling”, most of these accounts for acquiring ha locks. The 50Mln request units per month may not be enough for even light usage. I would probably deploy my own CockroachDB.

I’m using ‘ocikms’ for unsealing, although Oracle turns me off, but the same offering from GCP and AWS are not very clear on the cost of it. I don’t want to pay to use simple symetric aes key. I had similar setup with awskms in the past, and if I recall correctly the cost was negligible. But for sure OCI Vault is completely free, and I would like to use it.

Setting up OCI

Using web console, see afterwords for “reasoning”…

1. Setting up Vault OCI account for vault.service

   1. Login to Oracle Cloud, Goto “Identity & Security” -> “Domains” -> “Default” (The one I’m using)

   2. Goto “Groups”, “Create group” with name “Vault Serviceaccout Group”

   3. Goto “Users”, “Create user” with First name “Vault”, last name “Serviceaccount” and vault@domain.tld as email address, for email, reachability isn’t a problem.

   4. Click on the user you have just created, click on “API Keys” hiding in the lower left columns, then “Add API key”. Use a generated key or generate your own key, no much difference. After key generation, it will give a template for ~/.oci/config

   5. Save the secret key to /home/vault/.oci/privkey.pem, also paste the generated config to /home/vault/.oci/config. Here’s an example:

      grep vault /etc/passed # check where the home of vault is, mine at /home/vault (debian)
      sudo mkdir -p /home/vault/.oci
      # copy paste key and config
      sudo chmod 0600 /home/vault/.oci/privkey.pem
      sudo chown -R vault:vault /home/vault
      

   6. Goto “Identity & Security” -> “Key Management & Secret Management”, “Create Vault” if you have to

   7. Click on the Vault, “Create key” with Protection Mode -> HSM, Name -> “vault-unseal-xyz” -> Key Shape: Algorithm -> “AES(blablah)”, and Key Shape: Length -> “256 bits”. (Longest it can provide)

   8. Note down the following:

      1. Click on the key you’ve just created and note down it’s OCID
      2. Go back to Vault Details page, note down “Cryptographic Endpoint”
      3. On the same page, note down “Management Endpoint”

   9. Configure vault to use ocikms

      seal "ocikms" {
        auth_type_api_key = "true"
        crypto_endpoint = # fill
        management_endpoint = # fill
        key_id = # fill in OCID of the key
      }
      

   10. Make sure vault is okay to restart, then systemctl restart vault

   11. Unseal with vault operator unseal -migrate and you’re done

Afterwords

1. The offical guide from Vault didn’t say they need a symetric key, I wasn’t thinking much when creating that key, tried to use ECDSA and RSA. This makes sense when you think a little more about it. But still I wish there’s a word about the key algo.
2. You can generate “auth token” for users in the OCI Console, but the manual doesn’t include that token. The other manual about using a token is about the shortlived (1h) token derived from OAuth2. The most probable use of the “auth token” is against OCI docker registry.
3. I’m using terraform cloud (shame on me), so I don’t really know how to supply the key file into the environment, so I’m using the web console. (One can use agents in conjunction with tf cloud, but that beats my purpose of relying less on local resources)
4. Cross region HA slows down vault a bit. Maybe not just a bit.