During the past months i’ve launched several initiatives to improve the security posture of our corporate infrastructure. As most companies we have the notion of a “internal” and an “external” network, which becomes more obsolete every day. For more background on this, look for good resources on “Zero Trust” networking and try to avoid marketing material.
Some of our assets are stored within GitLab, for example source-code, documentation, configuration, automation and build pipelines. As most Git server and CI implementations client access is offered through HTTPS and Git+SSH, with the latter being much more efficient. We have already moved web-browser access over HTTPS to flow through an authentication proxy some time ago. This means users run through our OpenID based single-sign on process before being granted access to GitLabs web interface.
When cloning or pushing Git repositories however, we still depend on static SSH keys. While SSH authentication using public/private keys is already a lot better than passwords, it still comes at the risk of losing the private key and by that allowing a third party elevated access to our repositories. GitLab is exposed to the Internet as we share lots of code with the open-source community. The main issue with those private keys is eternal trust and the fact that they are “only” protected by client-side security measures, some of which are optional and cannot really be attested, like encryption of the key material.
Teleport to the rescue
We’re using Gravitational Teleport for privileged access management, for example maintenance access to machines through SSH. It’s built around the idea that access is granted in an ephemeral way and that authentication runs through SSO, which means out-of-band techniques like 2FA can be used before access to SSH key material is provided.
Teleport works well with kubectl, which employs OpenSSH to control Kubernetes deployments. This quickly led to the idea of just using Teleport to provide access management to GitLab, which offers Git+SSH access through OpenSSH as well. In theory that’s pretty straight forward but came with some quirks related to GitLab.
In an optimal scenario a GitLab user would not upload any key material but get authenticated through Teleport and authorized through GitLab. Without knowing the users key fingerprint it’s however hard to map incoming SSH connections to user accounts and subsequently make authorization decisions. As a bonus, login to SSH works through a generic “git” user, so the users name and access permissions have to come from the certificate metadata.
Integrating Teleport
Let’s assume that Teleport is already up and running and users can tsh login to get their key material. On the Teleport side there is only one more thing to change, which is encoding specific “Principal” information to key material for users that are eligible to use GitLab. This information can be obtained by Teleport through the SSO system by checking what “claims” the user has, a LDAP backend or through static configuration. For the sake of this example let’s assume static configuration.
1 | spec: |
The next time a user logs in to Teleport and gets access to key material, it will have those “logins” encoded as principals.
1 | $ ssh-add -L | grep cert | ssh-keygen -L -f - |
At this point also make sure the “Key ID” matches your GitLab user, this is essential to allow authorization.
Integrating GitLab
Public key information provided by the user through GitLabs user settings is stored within the home directory of the “git” user, at /var/opt/gitlab/.ssh/authorized_keys. Examining that file shows that a bit more is going on, for example that a command is called which maps the key to a user within the GitLab database. This will not work when authenticating with an ephemeral key that is not known or mapped at GitLab. At the same time we won’t need any integration by GitLab to make this work. It may make sense to restrict HTTPS access to force users on Git+SSH and somehow remove all existing user SSH keys, but thats rather optional.
Integrating OpenSSH
Server-side
To solve this, we can configure OpenSSHd to positively authenticate all connections that use a valid certificate which got signed by the Teleport CA. This follows the normal “OpenSSH integration” guideline from Gravitational. Export the public key of the Teleport CA and put it to the OpenSSH configuration at the GitLab server using the TrustedUserCAKeys parameter.
1 | root@teleport $ tctl auth export --type=host > cluster_node_keys |
The other very important part is to use AuthorizedPrincipalsCommand to allow sessions of the SSH “git” user to get mapped to GitLab users. This command can be run as user “git” and contains the “Principal” to make sure only certs with the encoded value gain access. Finally, the “Key ID” value is inserted as %i to tell GitLab which user shall be authorized. Note that this information can only be encoded to the certificate by Teleport as only certificates signed by Teleports CA are accepted.
1 | root@gitlab $ vim /etc/ssh/sshd_config |
Client-side
Now the users OpenSSH client configuration needs to be updated to make sure the key material provided through Teleport is being used, instead of the users default key.
1 | user@workstation $ vim ~/.ssh/config |
This makes sure the key stored at ~/.tsh/keys/teleport/martin is being used for SSH connections to gitlab.heiland.io when using the git user. Git will use this configuration when performing remote operations through Git+SSH.
Wrapping up
Now users should be able to git clone and work with repositories for which they are authorized in GitLab - once they ran through Teleports authentication process. There is no need anymore to upload any per-user key material to GitLab. However, GitLab always allow to fall-back to SSH keys, which can still be very useful for non-interactive access.
Limitations
This examples showcases how access to GitLab can be controlled through Gravitational Teleport. It builds upon OpenSSH integration and does not require a premium subscription of Teleport. However, this comes at the disadvantage that access to GitLab is not logged or monitored by Teleport, which can be worked around by monitoring OpenSSH logs which contain all that information.