gitolite/doc/6-ssh-troubleshooting.mkd
Sitaram Chamarty b0ce84d47f document @SHELL feature, allow "info" for all,
...but still distinguish shell folks with a small extra line telling
them they have shell access
2009-12-23 19:57:36 +05:30

10 KiB

ssh troubleshooting

Update 2009-12-23: most of this document is now of historical interest and will be totally revamped when I have time. For now, just note this amendment.

The document below says "we can't use the same key for both [gitolite access and shell access]...". We've managed (thanks to an idea from Jesse Keating) to get around this. Now it is possible for a single key to allow both gitolite access and shell access.

This is done by placing such a user in a special @SHELL group in the gitolite config file. As usual, please see conf/example.conf for more info on this, since I'm using that as a central place to document anything concerned with the conf file.


Ssh has always been the biggest troublespot in all this. While gitolite makes it as easy as possible, you might still run into trouble sometimes.

In this document:

  • ssh sanity checks
  • explanation
    • files on the server
    • files on client
    • why two keys on client
  • more complex ssh setups
    • two gitolite servers to manage?
  • further reading

But before we get to all that, let's clarify that all this is applicable only to the gitolite admin. He's the only one who needs both a shell and gitolite access, so he has two pubkeys in play.

Normal users have only one pubkey, since they are only allowed to access gitolite itself. They do not need to worry about any of this stuff, and their repo urls are very simple, like: git@my.git.server:reponame.git.


ssh sanity checks

There are two quick sanity checks you can run:

  • running ssh gitolite should get you a list of repos you have rights to access, as described here
  • conversely, ssh git@server should get you a command line

If one or both of these does not work as expected, do this:

  • first, check that your ~/.ssh has two public keys, like below:

    $ ls -al ~/.ssh/*.pub
    -rw-r--r-- 1 sitaram sitaram 409 2008-04-21 17:42 /home/sitaram/.ssh/id_rsa.pub
    -rw-r--r-- 1 sitaram sitaram 409 2009-10-15 16:25 /home/sitaram/.ssh/sitaram.pub
    

    If it doesn't you have either lost your keys or you're on the wrong machine. As long as you have password access to the server you can alweys recover; just pretend you're installing from scratch and start over.

  • next, try running ssh-add -l. On my desktop the output looks like this:

    2048 63:ea:ab:10:d2:4f:88:f4:85:cb:d3:7d:3a:83:37:9a /home/sitaram/.ssh/id_rsa (RSA)
    2048 d7:23:89:12:5f:22:4f:ad:54:7d:7e:f8:f5:2a:e9:13 /home/sitaram/.ssh/sitaram (RSA)
    

    If you get only one line (typically the top one), you should ssh-add the other one, using (in my case) ssh-add ~/.ssh/sitaram.

    If you get no output, add both of them and check ssh-add -l again.

    If this error keeps happening please consider installing keychain or something similar, or add these commands to your bash startup scripts.

  • Finally, make sure your ~/.ssh/config has the required host gitolite para (see below for more on this).

Once these sanity checks have passed, things should be fine. However, if you still have problems, make sure that the "origin" URL in any clones looks like gitolite:reponame.git, not git@server:reponame.git.

explanation

Here's how it all hangs together.

files on the server

  • the authkeys file; this contains one line containing the pubkey of each user who is permitted to login without a password.

    Pubkey lines that give shell access look like this:

    ssh-rsa AAAAB3NzaC[snip]uPjrUiAUew== /home/sitaram/.ssh/id_rsa
    

    On a typical server there will be only one or two of these lines.

    Note that the last bit (/home/sitaram/.ssh/id_rsa) is purely a comment field and can be anything. Also, the actual lines are much longer, about 400 characters; I snipped 'em in the middle, as you can see.

    In contrast, pubkey lines that give access to git repos hosted by gitolite look like this:

    command="[some path]src/gl-auth-command sitaram",[some restrictions] ssh-rsa AAAAB3NzaC[snip]s18OnB42oQ== sitaram@sita-lt
    

    You will have many more of these lines -- one for every pubkey file in keydir/ of your gitolite-admin repo, with the corresponding username in place of "sitaram" in the example above.

    The "command=" at the beginning ensures that when someone with the corresponding private key logs in, they don't get a shell. Instead, the gl-auth-command program is run, and (in this example) is given the argument sitaram. This is how gitolite is invoked, (and is told the user logging in is "sitaram").

files on client

  • default keypair; used to get shell access to servers. You would have copied this pubkey to the gitolite server in order to log in without a password. (On Linux systems you may have used ssh-copy-id to do that). You would have done this before you ran the easy install script, because otherwise easy install won't run!

    ~/.ssh/id_rsa
    ~/.ssh/id_rsa.pub
    
  • gitolite keypair; the "sitaram" in this is the 3rd argument to the src/gl-easy-install command you ran; the easy install script does the rest

    ~/.ssh/sitaram
    ~/.ssh/sitaram.pub
    
  • config file; this file has an entry for gitolite access:

    ~/.ssh/config
    

    To understand why we need that, let's step back a bit. Normally, you might expect to access gitolite repos like this:

    ssh://git@server/reponame.git
    

    But this won't work, because this ends up using the default keypair (normally), which gives you a command line. Which means it won't invoke the gl-auth-command program at all, and so none of gitolite's access control will work.

    You need to force ssh to use the other keypair when performing a git operation. With normal ssh, that would be

    ssh -i ~/.ssh/sitaram git@server
    

    but git does not support putting an alternate keypair in the URL.

    Luckily, ssh has a very convenient way of capturing all the connection information (username, hostname, port number (if it's not the default 22), and keypair to be used) in one "paragraph" of ~/.ssh/config. This is what the para looks like for us (the easy install script puts it there the first time):

    host gitolite
         user git
         hostname server
         identityfile ~/.ssh/sitaram
    

    (The "gitolite" can be anything you want of course; it's like a group name for all the stuff below it). This ensures that typing

    ssh gitolite
    

    is equivalent to

    ssh -i ~/.ssh/sitaram git@server
    

    and therefore this:

    git clone gitolite:reponame.git
    

    now works as expected, invoking the special keypair instead of the default one.

why two keys on client

Why do I (the admin) need two different keypairs?

There are two types of access the admin will make to the server: a normal login, to get a shell prompt, and gitolite access (clone/fetch/push etc). The first access needs an authkeys line without any "command=" restrictions, while the second requires a line with such a restriction.

And we can't use the same key for both because there is no way to disambiguate them; the ssh server will always (always) pick the first one in sequence when the key is offered by the ssh client.

So the next question is usually "I have other ways to get a shell on that account, so why do I need a key for shell access at all?"

The answer to this is that the "easy install" script, being written for the most general case, needs shell access via ssh to do its stuff.

If you really, really, want to get rid of the extra key, here's a transcript that should have enough info to get you going (but it helps to know ssh well):

  • on "sitaram" user, on my workstation

    cd ~/.ssh
    cp id_rsa sitaram
    cp id_rsa.pub sitaram.pub
    cd ~/gitolite-clone
    src/gl-easy-install -q git my.git.server sitaram
    

    that last command produces something like the following:

    you are upgrading from     (unknown)     to v0.80-6-gdde8c4e
    setting up keypair...
        ...reusing /home/sitaram/.ssh/sitaram.pub...
    creating gitolite para in ~/.ssh/config...
    finding/creating gitolite rc...
    installing/upgrading...
    Pseudo-terminal will not be allocated because stdin is not a terminal.
    [master (root-commit) e717a89] start
     2 files changed, 11 insertions(+), 0 deletions(-)
     create mode 100644 conf/gitolite.conf
     create mode 100644 keydir/sitaram.pub
    cloning gitolite-admin repo...
    Initialized empty Git repository in /home/sitaram/gitolite-admin/.git/
    fatal: 'gitolite-admin.git' does not appear to be a git repository
    fatal: The remote end hung up unexpectedly
    

    notice that the final step (the clone of the newly created gitolite-admin repo) failed, as expected

  • now log on to the git hosting account (git@my.git.server in this example), edit ~/.ssh/authorized_keys, and delete the line with the first occurrence of your key (this should be before the # gitolite start line)

  • now go back to your workstation and

    git clone git@my.git.server:gitolite-admin
    

That should do it.

more complex ssh setups

What do you need to know in order to create more complex ssh setups (for instance if you have two gitolite servers you are administering)?

two gitolite servers to manage?

  • they can have the same key; no harm there (example, sitaram.pub)

  • instead of just one ssh/config para, you now have two (assuming that the remote user on both machines is called "git"):

    host gitolite
         user git
         hostname server
         identityfile ~/.ssh/sitaram
    
    host gitolite2
         user git
         hostname server2
         identityfile ~/.ssh/sitaram
    
  • now access one server's repos as gitolite:reponame.git and the other server's repos as gitolite2:reponame.git.

further reading

While this focused mostly on the client side ssh, you may also want to read this for a much more detailed explanation of the ssh magic on the server side.