Qubes OS Split SSH With Only AppVMs (Probably)

The currently accepted best practice for implementing Split SSH¹ involves making several updates to a few templates, and editing files that are already around (specifically /rw/config/rc.local and ~/.bashrc). Editing files that were handled by other programs is error prone and inelegant. Qubes OS provides better ways to handle config files and so do most Linux distros, so let's use those.

tl;dr / Key Improvements

• Probably no need to make any templates, only openssh, ssh-agent, and ssh-askpass are required
• Instead of /rw/config/rc.local, use /rw/config/rc.local.d/qubes-split-ssh.conf
• Instead of modifying environment in ~/.bashrc, use ~/.config/environment.d/qubes-split-ssh.conf
• Instead of /etc/qubes-rpc/qubes.SshAgent, use /usr/local/etc/qubes-rpc/user.SshAgent
• RPC policies can use @default so as not to leak the vault qube's name

AppVM-Only Split SSH

Following the currently accepted best practice, we see a handful of files are being created or edited in both templates and AppVMs:

• In dom0, create /etc/qubes/policy.d/50-ssh.policy
• In the vaults' template, create /etc/qubes-rpc/qubes.SshAgent
• In any vault AppVMs, create ~/.config/autostart/ssh-add.desktop
• In any client AppVMs, edit /rw/config/rc.local and ~/.bashrc

This works well enough, but I don't like a few things about it:

• I don't love mixing custom users' configs (my own configs) with configs that are managed by the system. This means /etc/qubes/policy.d/50-ssh.policy, and /etc/qubes-rpc/qubes.SshAgent
• I really don't like editing files that are expected to handle many things or be edited for many functions. This means /rw/config/rc.local, and ~/.bashrc

Thankfully, with smarter use of SystemD's environment.d, Qubes OS's config files, and Qubes OS Qrexec, we can solve all the above.

SystemD can load environment variables from a user's ${HOME}/.config/environment.d/*.conf files, and likewise Qubes OS checks /rw/config/rc.local.d/*.rc for files, as well as all /usr/local counterparts for the root filesystem (so /usr/local/qubes-rpc is a thing). All of them require the files there to be executable. With these in mind, we design a split SSH implementation that's more modular, and simple.

Setting up the Vault

First, we can remove the need for a vault template by moving the RPC service to /usr/local/qubes-rpc. These are per-AppVM services that Qubes can use. Add the following to /usr/local/qubes-rpc/user.SshAgent:

#!/bin/bash --
set -euo pipefail

notify-send \
  "$(qubesdb-read /name) Split SSH" \
  "SSH agent access from ${QREXEC_REMOTE_DOMAIN} granted"

socat - "UNIX-CONNECT:${SSH_AUTH_SOCK}"

Make it executable:

sudo chmod +x /usr/local/qubes-rpc/user.SshAgent

Also, create the auto-start shortcut in ~/.config/autostart/ssh-add.desktop:

[Desktop Entry]
Name=ssh-add
Exec=ssh-add -c
Type=Application

Setting Up a Key

If you want to generate a key, use ssh-keygen:

ssh-keygen -t ed25519

If you have a hardware security key, you can use ed25519-sk if you attach the USB device to your vault:

ssh-keygen -t ed25519-sk # Will prompt your security key

Setting up the Client(s)

In our client AppVMs, add the following to /rw/config/rc.local.d/30-split-ssh.rc:

#!/bin/bash --
set -aeo pipefail

SSH_VAULT_VM="${SSH_VAULT_VM:-@default}"

sock_dir="/var/run/qubes-split-ssh/${SSH_VAULT_VM}"
install -Dd -o$(id -nu 1000) -gqubes "${sock_dir}"
export SSH_SOCK="${sock_dir}/S.agent"
rm -f "${SSH_SOCK}"
sudo -u user /bin/sh -c "umask 177 && exec socat 'UNIX-LISTEN:${SSH_SOCK},fork' 'EXEC:qrexec-client-vm ${SSH_VAULT_VM} user.SshAgent'" &

Then add the following to ~/.config/environment.d/30-split-ssh.conf:

SSH_VAULT_VM="${SSH_VAULT_VM:-@default}"
SSH_AUTH_SOCK="/var/run/qubes-split-ssh/${SSH_VAULT_VM}/S.agent"

Make them both executable:

sudo chmod +x /rw/config/rc.local.d/*
chmod ~/.config/environment.d/*

Writing a Simple Policy in dom0

Finally, let's write a simple policy in dom0. Add the following to /etc/qubes/policy.d/99-user.conf:

user.SshAgent  *  @anyvm  @default  target=${SSH_VAULT_VM} notify=yes

...and that should be it? Now your clients can use SSH proxied securely with qrexec to an air-gapped backend.

Next Steps

There are still several ways this can be improved!

• In dom0, can the /usr/local/etc/qubes/policy.d directory be used?
• The policy uses @anyvm as the source qube, but you could make this more strict by using @tag:ssh-client or something. Then only VMs with the ssh-client tag would be able to proxy SSH agent requests
• If the rc.local.d/30-split-ssh.rc was rewritten as a SystemD unit, it could use ConditionPathExists=/run/qubes-services/ssh-client (or something) in conjunction with the Qubes Services feature to offer even more flexibility