How to get started with HashiCorp Vault for Secrets Management in your DevOps Pipeline – Part 2

This is part 2 of 3 in our blog series on how to build out a secrets management environment using HashiCorp Vault® (“Vault”, for short). In our first blog of the series (part 1) we covered what Vault is. We also went through an overview of the solution. In this second part of the series, we build out Vault:

  • Review Prerequisites

  • Install Vault on the Linux Server

  • Configure the Vault Service

This will position us so that in our final blog (part 3) we can retrieve a secret from Vault. We will use a Python script to do that. And, finally, we will cover some bonus material.


Building the Environment

This is where we will spend the bulk of our time and effort! First, I list out the prerequisites that are needed to build out both the server and the desktop. Then, I cover two of the three major steps to build out the environment. We will cover major step #3 in our next blog entry.


Prerequisites

  • Ubuntu Server

An Ubuntu server 20.04 LTS with a sudo user and an accessible IP address is required for hosting Vault.

  • The unzip and jq packages also need to be installed.

Issue the commands in Figure 1 to install these two packages on the Ubuntu server.

sudo apt-get install unzip 
sudo apt-get install jq  

Figure 1 - Install unzip and jq


  • Windows 10 Desktop

A Windows 10 desktop system with an IP address that can access the Ubuntu server is required for accessing Vault.

  • Visual Studio Code or your preferred IDE that can run Python scripts

  • Python Plugin for VS Code

  • Python v3.x – I recommend installing this for all users and adding it to the PATH variable.

  • Python requests library – Run the pip command in Figure 2 inside the VS Code console to add the requests library for Python into VS Code.

pip install requests

Figure 2 - Install Python requests Library


  • Create or obtain an SSL Certificate for Vault

An SSL certificate for the Ubuntu server is required. You can either obtain one from an already established CA (certificate authority) or you can create one on your own using OpenSSL. For proof-of-concept purposes, a self-signed certificate (created on your own) is acceptable. Best practice in production is to obtain a signed certificate from a trusted CA. I’ve described the two options below. I used a self-signed SSL certificate (option #2 below).


Option 1: Obtain an SSL Certificate from a CA

If you are working in an environment with a PKI, request a certificate from your CA (ask your CA administrator if you don’t know how). Make sure to include subject alternative name(s) (SAN) that represent the Vault server you are building. Use the DNS name and/or IP address for the Vault server you will build below. In these instructions we will be using the IP address of the server, but you can substitute the DNS name if you have that configured. The public key must be in the CER, CRT, or PEM formats. The private key must be in the PEM format.


Note: If you are starting with a PFX file you will need to convert and split it into the public and private keys using a tool like OpenSSL. When converting using OpenSSL, it may add values to the files which will cause Vault to fail to load the certificate. Keep only the info between and including “-----BEGIN” and “END-----".


Option 2: Generate a self-signed SSL certificate using OpenSSL

To do this, you will need OpenSSL installed. If you do not have it installed already on your Ubuntu server, there are a couple options for loading OpenSSL on your server. The easiest way is to use Ubuntu’s app-get command (sudo apt-get install -y openssl). Alternatively, you can download the latest version of OpenSSL from the openssl.org site: https://www.openssl.org/source/.

  • Create the OpenSSL Configuration File

You need to create an OpenSSL configuration file to ensure you create the correct type of certification. Use the following steps to create the configuration file. On the Ubuntu command line, type the command in Figure 3:


cat <<EOF>>vaultcert.cnf

Figure 3 - Create vaultcert.cnf


This command will create a flat file with everything until the EOF characters at the end.

Paste in the text in Figure 4.

Important Note: Be sure to modify {{{YOUR VAULT IP}}} at the end. This should be the IP of your Ubuntu server.


[req] 
distinguished_name = req_distinguished_name 
x509_extensions = v3_req 
prompt = no 
[req_distinguished_name] 
C = US 
ST = PA 
L = Lancaster 
O = TestCompany 
OU = TestOrg 
CN = vault.test 
[v3_req] 
keyUsage = nonRepudiation, digitalSignature, keyEncipherment 
extendedKeyUsage = serverAuth 
subjectAltName = @alt_names 
[alt_names] 
IP.1 = {{{YOUR VAULT IP}}} 
EOF 

Figure 4 - vaultcert.cnf


Run the command in Figure 5 to generate the certificate and key. Note: It must be pasted in as a single line on the command line.

openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyoutvaultserver.key -outvaultserver.cer -config vaultcert.cnf

Figure 5 - Generate Certificate and Key


This will generate two files in your current directory: vaultserver.key and vaultserver.cer. You will use the vaultserver.key and vaultserver.cer files when setting up the Vault configuration files below. These will be the files used for the SSL connection to the Vault.


Major Step #1: Installing Vault on the Linux Server


Once the Ubuntu system is up, log in with a user account with sudo access.


Create the vault user as a system service user. The user in this exercise will be called “vault”. Issue the command in Figure 6 to create the user.

sudouseradd -r --system vault

Figure 6 - Add vault User


Download and copy the Vault binary to the correct directory. Issue the commands in Figure 7.

sudo mkdir /usr/local/bin/vault.d 
wget https://releases.hashicorp.com/vault/1.7.0-rc2/vault_1.7.0-rc2_linux_amd64.zip 
unzip vault_1.7.0-rc2_linux_amd64.zip 
cp ~/vault /usr/local/bin/vault.d/ 

Figure 7 - Download Vault Binary


Note: Check the URL above (https://releases.hashicorp.com/vault/...) and modify it, if needed. I used HashiCorp Vault® 1.7.0 RC2. The latest version can be found at https://releases.hashicorp.com/vault/. Make sure to download the zip file for the hardware architecture you are running your Ubuntu server on.


Set the ownership and permissions on the Vault binary.


Issue the two commands in Figure 8 to change the user and group ownership to both be “vault” and the rights to be 775.

sudo chown -R vault:vault /usr/local/bin/vault.d 
sudo chmod -R 775 /usr/local/bin/vault.d  

Figure 8 - Set Rights on vault.d


Create the Vault configuration folder. Then move the Vault server certificate and key files into the certs directory. Use the commands in Figure 9.

sudo mkdir /etc/vault.d/ 
sudo mkdir /etc/vault.d/certs 
 
sudo mv vaultserver.cer /etc/vault.d/certs 
sudo mv vaultserver.key /etc/vault.d/certs 

Figure 9 - Move certificate and key files


Run the command in Figure 10 and paste the text in Figure 11 into the command window. This snippet will create the vault configuration file, vault.hcl.

cat <<EOF >> vault.hcl

Figure 10 - Create vault.hcl File

ui = true 
diable_mlock=true 
 
HTTP listener 
listener "tcp" { 
address="127.0.0.1:8200" 
tls_disable = 1 
} 
 
listener "tcp" { 
address = "0.0.0.0:8200" 
tls_disable = "false" 
tls_cert_file = "/etc/vault.d/certs/vaultserver.cer" 
tls_key_file = "/etc/vault.d/certs/vaultserver.key" 
} 
 
storage "raft" { 
path = "/opt/raft/data" 
node_id="node1" 
} 
 
cluster_addr="http://127.0.0.1:8201" 
api_addr="http://127.0.0.1:8200" 
EOF 

Figure 11 - vault.hcl


Move the vault.hcl file into the config directory and set the permissions. Use the commands in Figure 12.

mv vault.hcl /etc/vault.d 
sudo chown -R vault:vault /etc/vault.d 
sudo chmod -R 774 /etc/vault.d/ 

Figure 12 - Move vault.hcl File


Create the Vault Raft storage directory and set permissions. Issue the commands in Figure 13.


This is where Vault will store the encrypted secrets. (Note: “Raft” is a proprietary storage format that HashiCorp developed to store the secrets in an encrypted format. For more information about Raft, you can read HashiCorp’s documentation here.)

sudo mkdir /opt/raft 
sudo mkdir /opt/raft/data 
sudo chown -R vault:vault /opt/raft 
sudo chmod -R 775 /opt/raft 

Figure 13 - Setup raft Directory


Major Step #2: Configuring the Vault Service


We will create a vault service definition for systemd.


Run the command in Figure 14 and paste the text in Figure 15 into the command window to create the service definition (configuration) file.

cat << EOF >> vault.service

Figure 14 - Create vault.service File

[Unit] 
Requires=network-online.target 
After=network-online.target 
ConditionFileNotEmpty=/etc/vault.d/vault.hcl 
StartLimitBurst=3 
 
[Service] 
User=vault 
Group=vault 
ProtectSystem=full 
ProtectHome=read-only 
PrivateTmp=yes 
PrivateDevices=yes 
SecureBits=keep-caps 
AmbientCapabilities=CAP_IPC_LOCK 
Capabilities=CAP_IPC_LOCK+ep 
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK 
NoNewPrivileges=yes 
ExecStart=/usr/local/bin/vault.d/vault server -config=/etc/vault.d/vault.hcl 
ExecReload=/bin/kill --signal HUP $MAINPID 
KillMode=process 
KillSignal=SIGINT 
Restart=on-failure 
RestartSec=5 
TimeoutStopSec=30 
StartLimitInterval=60 
StartLimitIntervalSec=60 
StartLimitBurst=3 
LimitNOFILE=65536 
LimitMEMLOCK=infinity 
 
[Install] 
WantedBy=multi-user.target 
EOF 

Figure 15 - vault.service File


Run the commands in Figure 16 to copy the vault service definition file into the system directory, configure “read” permissions on the file, and then startup the Vault service.

sudo mv vault.service /etc/systemd/system 
sudo chmod 644 /etc/systemd/system/vault.service 
sudo chown root:root /etc/systemd/system/vault.service 
sudo systemctl enable vault.service 
sudo systemctl start vault.service 

Figure 16 - Start Vault Service


Verify that the Vault service is running by typing the command in Figure 17 to see the status of running services.

systemctl --type=service

Figure 17 - Verify Vault Service is Running


You should see the output in Figure 18.

Figure 18 - systemctl Output


If the vault service fails to start, it is usually a permissions issue, a missing directory, or a missing file. To troubleshoot the issue, run the command in Figure 19 to see the console output of the Vault binary. This may help you determine why the service is not starting.


sudo journalctl -u vault

Figure 19 - Run journalctl Command


Summary and What’s Next

Now that we’ve built out our Vault environment, we are ready for Major Step #3: Setting Up the Vault itself. And, finally, we will retrieve a secret from Vault. We’ll cover that in our final blog of the series (part 3).


Reach Out to Us

If you need more information about how to implement Hashicorp Vault or another vaulting solution, or if you need help in automating your CI/CD pipeline, reach out to us.