gitolite/doc/6-ssh-troubleshooting.mkd

# ssh troubleshooting

In this document:

  * <a href="#the_most_common_problems_that_an_admin_will_see">the most common problems that an admin will see</a>
  * <a href="#basic_ssh_troubleshooting">basic ssh troubleshooting</a>
      * <a href="#passphrases_versus_passwords">passphrases versus passwords</a>
      * <a href="#ssh_agent_problems">ssh-agent problems</a>
      * <a href="#basic_ssh_troubleshooting_for_the_main_admin">basic ssh troubleshooting for the main admin</a>
      * <a href="#basic_ssh_troubleshooting_for_a_normal_user">basic ssh troubleshooting for a normal user</a>
  * <a href="#windows_issues">windows issues</a>
  * <a href="#details">details</a>
      * <a href="#files_on_the_server">files on the server</a>
      * <a href="#files_on_client">files on client</a>
      * <a href="#why_two_keys_on_client">why two keys on client</a>
  * <a href="#some_other_tips_and_tricks">some other tips and tricks</a>
      * <a href="#giving_shell_access_to_gitolite_users">giving shell access to gitolite users</a>
      * <a href="#losing_your_admin_key">losing your admin key</a>
      * <a href="#simulating_ssh_copy_id">simulating ssh-copy-id</a>

----

This document should help you troubleshoot ssh-related problems in accessing
gitolite *after* the install has completed successfully.

In addition, I **strongly** recommend reading
[doc/9-gitolite-and-ssh.mkd][doc9gas], which is a very detailed look at how
gitolite uses ssh's features on the server side.  Most people don't know ssh
as well as they *think* they do; even if you don't have any problems right
now, it's worth skimming over.

In addition to both these documents, there's now a program called
`sshkeys-lint` that you can run on your client.  Run it without arguments to
get help on how to run it and what inputs it needs.

<a name="the_most_common_problems_that_an_admin_will_see"></a>

### the most common problems that an admin will see

Ironically, these problems **only** happen to the person who installed
gitolite using easy-install (the "from-client" method in
[doc/0-INSTALL.mkd][doc0]), and has **utterly failed** to read/heed the
message that shows up at the end of running that command.  This is because
only the admin has *two* ssh keys to the server (see "basic ssh
troubleshooting for the main admin" section below for more on this).

Both these problems are caused by using the wrong key, thus **bypassing gitolite
completely**:

  * you get `fatal: 'reponame' does not appear to be a git repository`, and
    yet you are sure 'reponame' exists, you haven't mis-spelled it, etc.

  * you are able to clone repositories but are unable to push changes back
    (the error complains about the `GL_RC` environment variable not being set,
    and the `hooks/update` failing in some way).

Let us recap the message that appears on a successful run of the "easy-install"
program; it looks something like this (with suitable values substituted for
`<user>`, `<server>`, and `<port>`):

    IMPORTANT NOTE -- PLEASE READ!!!
        *Your* URL for cloning any repo on this server will be
            gitolite:reponame.git

        *Other* users you set up will have to use
            <user>@<server>:reponame.git
        However, if your server uses a non-standard ssh port, they should use
            ssh://<user>@<server>:<port>/reponame.git

        If this is your first time installing gitolite, please also:
            tail -31 src/gl-easy-install
        for next steps.

The first error above happens if you use `git@server:reponame` instead of
`gitolite:reponame`.  All your repos are actually in a subdirectory pointed to
by `$REPO_BASE` in the rc file (default: `repositories`).  Gitolite internally
prefixes this before calling the actual git command you invoked, but since
you're bypassing gitolite completely, this prefixing does not happen, and so
the repo is not found.

The second error happens if you use `git@server:repositories/reponame.git`
(assuming default `$REPO_BASE` setting)  -- that is, you used the full unix
path.  Since the "prefixing" mentioned above is not required, the shell finds
the repo and clones ok.  But when you push, gitolite's **update hook** kicks
in, and fails to run because you some of the environment variables it is
expecting are not present.

<a name="basic_ssh_troubleshooting"></a>

### basic ssh troubleshooting

[glb]: http://sitaramc.github.com/0-installing/9-gitolite-basics.html#IMPORTANT_overview_of_ssh

I assume the gitolite server is called "server" and the user hosting all the
gitolite repos is "git".  I will also be using "sitaram" as the *gitolite
username* of the admin.

Unless specifically mentioned, all these commands are run on the user's or
admin's workstation, not on the server.

<a name="passphrases_versus_passwords"></a>

#### passphrases versus passwords

When you create an ssh keypair, you have the option of protecting it with a
passphrase.  When you subsequently use that keypair to access a remote host,
your *local* ssh client needs to unlock the corresponding private key, and ssh
will probably ask for the passphrase you set when you created the keypair.

Do not confuse or mistake this prompt (`Enter passphrase for key
'/home/sitaram/.ssh/id_rsa':`) for a password prompt from the remote server!

You have two choices to avoid this prompt every time you try to access the
remote.  The first is to create keypairs *without* a passphrase (just hit
enter when prompted for one).  **Be sure to add a passphrase later, once
everything is working, using `ssh-keygen -p`**.

The second is to use `ssh-agent` (or `keychain`, which in turn uses
`ssh-agent`) or something like that to manage your keys.  Other than the next
section, further discussion of this is out of scope of this document.

<a name="ssh_agent_problems"></a>

#### ssh-agent problems

1.  Run `ssh-add -l`.  If this responds with either "The agent has no
    identities." or "Could not open a connection to your authentication
    agent.", skip this section.

2.  However, if it lists some keys, like this:

        2048 fc:c1:48:1e:06:31:97:a4:8b:fc:37:b2:76:14:c7:53 /home/sitaram/.ssh/id_rsa (RSA)
        2048 d2:e0:7f:fa:1a:89:22:41:bb:06:d9:ff:a7:27:36:5c /home/sitaram/.ssh/sitaram (RSA)

    then run `ls ~/.ssh` and make sure that all the keypairs you have there
    are represented in the `ssh-add -l` output.

3.  If you find any keypairs in `~/.ssh` that are not represented in the
    `ssh-add -l` output, add them.  For instance, if `ssh-add -l` showed me
    only the `id_rsa` key, but I also had a `sitaram` (and `sitaram.pub`)
    keypair, I'd run `ssh-add ~/.ssh/sitaram` to add it.

This is because ssh-agent has a quirk: if `ssh-add -l` shows *any* keys at
all, ssh will only use those keys.  Even if you explicitly specify an unlisted
key using `ssh -i` or an `identityfile` directive in the config file, it won't
use it.

<a name="basic_ssh_troubleshooting_for_the_main_admin"></a>

#### basic ssh troubleshooting for the main admin

You're the "main admin" if you're trying to access gitolite from the same
workstation and user account where you ran the "easy install" command.  You
should have two keypairs in your `~/.ssh` directory.  The pair called `id_rsa`
(and `id_rsa.pub`) was probably the first one you created, and you used this
to get passwordless (pubkey based) access to the server (which was a
pre-requisite for running the easy install command).

The second keypair has the same name as the last argument in the easy install
command you ran (in my case, `sitaram` and `sitaram.pub`).  It was probably
created by the easy install script, and is the key used for gitolite access.

In addition, you should have a "gitolite" paragraph in your `~/.ssh/config`,
looking something like this:

    host gitolite
         user git
         hostname server
         identityfile ~/.ssh/sitaram

If any of these are not true, you did something funky in your install; email
me or hop onto #git and hope for the best ;-)

Otherwise, run these checks:

1.  `ssh git@server` should get you a command line.

    If it asks you for a password, then your `id_rsa` keypair changed after
    you ran the easy install, or someone fiddled with the
    `~/.ssh/authorized_keys` file on the server.

    If it prints [gitolite version and access info][myrights], you managed to
    overwrite the `id_rsa` keypair with the `sitaram` keypair, or something
    equally weird.

2.  `ssh gitolite info` should print some [gitolite version and access
    info][myrights].  If you get the output of the GNU info command instead,
    you probably reused your `id_rsa` keypair as your `sitaram` keypair, or
    overwrote the `sitaram` keypair with the `id_rsa` keypair.

There are many ways to fix this, depending on where and what the damage is.
The most generic way (and therefore time-taking) is to re-install gitolite
from scratch:

  * make a backup of your gitolite-admin repo clone somewhere (basically your
    "keydir/*.pub" and your "conf/gitolite.conf").  If necessary get these
    files from the server's `~/.gitolite` directory.
  * log on to the server somehow (using some other account, using a password,
    su-ing in, etc) and delete `~/.ssh/authorized_keys`.  Rename or move aside
    `~/.gitolite` so that also looks like it is missing.
  * back on your workstation, make sure you have 2 keypairs (`id_rsa` and
    `sitaram`, along with corresponding `.pub` files).  Create them if needed.
    Also make sure they are *different* and not a copy of each other :-)
  * install gitolite normally:
      * run `ssh-copy-id -i ~/.ssh/id_rsa git@server` to get passwordless
        access to the server.  (Mac users may have to do this step manually)
      * make sure `ssh git@server pwd` prints the `$HOME` of `git@server`
        **without** asking for a password.  Do not proceed till this works.
      * run easy install again, (in my case: `cd gitolite-source;
        src/gl-easy-install -q git server sitaram`)
  * go to your gitolite-admin repo clone, and copy `conf/gitolite.conf` and
    `keydir/*.pub` from your backup to this directory
  * copy (be sure to overwrite!) `~/.ssh/sitaram.pub` also to keydir
  * now `git add keydir; git commit; git push -f`

That's a long sequence but it should work.

<a name="basic_ssh_troubleshooting_for_a_normal_user"></a>

#### basic ssh troubleshooting for a normal user

For a normal user, life is much simpler.  They should have only one pubkey,
which was previously sent to the gitolite admin to add into the admin repo's
`keydir` as "user.pub", and then "user" given permissions to some repo.

`ssh git@server info` should get you [gitolite version and access
info][myrights].  If it asks you for a password, your pubkey was not sent to
the server properly.  Check with your admin.

[myrights]: http://github.com/sitaramc/gitolite/blob/pu/doc/3-faq-tips-etc.mkd#myrights

If it gets you the GNU info command output, you have shell access.  This means
you had command line access to the server *before* you were added as a
gitolite user.  If you send that same key to your gitolite admin to include in
the admin repo, it won't work.  For reasons why, see below.

<a name="windows_issues"></a>

### windows issues

On windows, I have only used msysgit, and the openssh that comes with it.
Over time, I have grown to distrust putty/plink due to the number of people
who seem to have trouble when those beasts are involved (I myself have never
used them for any kind of git access).  If you have unusual ssh problems that
just don't seem to have any explanation, try removing all traces of
putty/plink, including environment variables, etc., and then try again.

If you can offer an *authoritative* account of the complications involved, and
how to resolve them and get things working, I'd be happy to credit you and
include it, either directly here if it is short enough or just an external
link, or in contrib/ if it's a longer piece of text.

<a name="details"></a>

### details

Here's how it all hangs together.

<a name="files_on_the_server"></a>

#### files on the server

  * the authkeys file; this contains one line containing the pubkey of each
    user who is permitted to login without a password.

    Pubkey lines that give shell access look like this:

        ssh-rsa AAAAB3NzaC[snip]uPjrUiAUew== /home/sitaram/.ssh/id_rsa

    On a typical server there will be only one or two of these lines.

    Note that the last bit (`/home/sitaram/.ssh/id_rsa`) is purely a *comment*
    field and can be anything.  Also, the actual lines are much longer, about
    400 characters; I snipped 'em in the middle, as you can see.

    In contrast, pubkey lines that give access to git repos hosted by gitolite
    look like this:

        command="[some path]src/gl-auth-command sitaram",[some restrictions] ssh-rsa AAAAB3NzaC[snip]s18OnB42oQ== sitaram@sita-lt

    You will have many more of these lines -- one for every pubkey file in
    `keydir/` of your gitolite-admin repo, with the corresponding username in
    place of "sitaram" in the example above.

    The "command=" at the beginning ensures that when someone with the
    corresponding private key logs in, they don't get a shell.  Instead, the
    `gl-auth-command` program is run, and (in this example) is given the
    argument `sitaram`.  This is how gitolite is invoked, (and is told the
    user logging in is "sitaram").

<a name="files_on_client"></a>

#### files on client

  * default keypair; used to get shell access to servers.  You would have
    copied this pubkey to the gitolite server in order to log in without a
    password.  (On Linux systems you may have used `ssh-copy-id` to do that).
    You would have done this *before* you ran the easy install script, because
    otherwise easy install won't run!

        ~/.ssh/id_rsa
        ~/.ssh/id_rsa.pub

  * gitolite keypair; the "sitaram" in this is the 3rd argument to the
    `src/gl-easy-install` command you ran; the easy install script does the
    rest

        ~/.ssh/sitaram
        ~/.ssh/sitaram.pub

  * config file; this file has an entry for gitolite access:

        ~/.ssh/config

    To understand why we need that, let's step back a bit.  Normally, you
    might expect to access gitolite repos like this:

        ssh://git@server/reponame.git

    But this won't work, because this ends up using the *default* keypair
    (normally), which gives you a command line.  Which means it won't invoke
    the `gl-auth-command` program at all, and so none of gitolite's access
    control will work.

    <a name="altkey"></a>

    You need to force ssh to use the *other* keypair when performing a git
    operation.  With normal ssh, that would be

        ssh -i ~/.ssh/sitaram git@server

    but git does not support putting an alternate keypair in the URL.

    Luckily, ssh has a very convenient way of capturing all the connection
    information (username, hostname, port number (if it's not the default 22),
    and keypair to be used) in one "paragraph" of `~/.ssh/config`.  This is
    what the para looks like for us (the easy install script puts it there the
    first time):

        host gitolite
             user git
             hostname server
             identityfile ~/.ssh/sitaram

    (The "gitolite" can be anything you want of course; it's like a group name
    for all the stuff below it).  This ensures that typing

        ssh gitolite

    is equivalent to

        ssh -i ~/.ssh/sitaram git@server

    and therefore this:

        git clone gitolite:reponame.git

    now works as expected, invoking the special keypair instead of the default
    one.

<a name="why_two_keys_on_client"></a>

#### why two keys on client

[This section is only applicable to installs done using the "from-client"
method; see [doc/0-INSTALL.mkd][doc0] for details].

Why do I (the admin) need two **different** keypairs?

There are two types of access the admin will make to the server: a normal
login, to get a shell prompt, and gitolite access (clone/fetch/push etc).  The
first access needs an authkeys line *without* any "command=" restrictions,
while the second requires a line *with* such a restriction.

And we can't use the same key for both because there is no way to disambiguate
them; the ssh server will always (*always*) pick the first one in sequence
when the key is offered by the ssh client.

So the next question is usually "I have other ways to get a shell on that
account, so why do I need a key for shell access at all?"

The answer to this is that the "easy install" script, being written for the
most general case, needs shell access via ssh to do its stuff.

If you really, really, want to get rid of the extra key, here's a transcript
that should have enough info to get you going (but it helps to know ssh well):

  * on "sitaram" user, on my workstation

        cd ~/.ssh
        cp id_rsa sitaram
        cp id_rsa.pub sitaram.pub
        cd ~/gitolite-clone
        src/gl-easy-install -q git my.git.server sitaram

    that last command produces something like the following:

        you are upgrading from     (unknown)     to v0.80-6-gdde8c4e
        setting up keypair...
            ...reusing /home/sitaram/.ssh/sitaram.pub...
        creating gitolite para in ~/.ssh/config...
        finding/creating gitolite rc...
        installing/upgrading...
        Pseudo-terminal will not be allocated because stdin is not a terminal.
        [master (root-commit) e717a89] start
         2 files changed, 11 insertions(+), 0 deletions(-)
         create mode 100644 conf/gitolite.conf
         create mode 100644 keydir/sitaram.pub
        cloning gitolite-admin repo...
        Initialized empty Git repository in /home/sitaram/gitolite-admin/.git/
        fatal: 'gitolite-admin.git' does not appear to be a git repository
        fatal: The remote end hung up unexpectedly

    notice that the final step (the clone of the newly created gitolite-admin
    repo) failed, as expected

  * now log on to the git hosting account (`git@my.git.server` in this
    example), edit `~/.ssh/authorized_keys`, and delete the line with the
    first occurrence of your key (this should be *before* the `# gitolite
    start` line)

  * now go back to your workstation and

        git clone git@my.git.server:gitolite-admin

That should do it.

<a name="some_other_tips_and_tricks"></a>

### some other tips and tricks

<a name="giving_shell_access_to_gitolite_users"></a>

#### giving shell access to gitolite users

We've managed (thanks to an idea from Jesse Keating) to make it possible for a
single key to allow both gitolite access *and* shell access.

This is done by copying the pubkey (to which you want to give shell access) to
the server and running either

    cd $HOME/.gitolite  # assuming default $GL_ADMINDIR in ~/.gitolite.rc
    src/gl-tool shell-add ~/foo.pub

or

    gl-tool shell-add ~/foo.pub

The first method is to be used if you used the **user-install** mode, while
the second method is for the **system-install followed by user-setup** mode
(see doc/0-INSTALL.mkd, section on "install methods", for more on this)

**IMPORTANT UPGRADE NOTE**: previous implementations of this feature were
crap.  There was no easy/elegant way to ensure that someone who had repo admin
access would not manage to get himself shell access.

Giving someone shell access requires that you should have shell access in the
first place, so the simplest way is to enable it from the server side only.

<a name="losing_your_admin_key"></a>

#### losing your admin key

If you lost the admin key, and need to re-establish ownership of the
gitolite-admin repository with a fresh key, take a look at the
`src/gl-emergency-addkey` program.  You will need shell access to the server
of course.  The top of the script has useful information on how to use it and
what it needs.

<a name="simulating_ssh_copy_id"></a>

#### simulating ssh-copy-id

don't have `ssh-copy-id`?  This is broadly what that command does, if you want
to replicate it manually.  The input is your pubkey, typically
`~/.ssh/id_rsa.pub` from your client/workstation.

  * it copies it to the server as some file

  * it appends that file to `~/.ssh/authorized_keys` on the server
    (creating it if it doesn't already exist)

  * it then makes sure that all these files/directories have go-w perms
    set (assuming user is "git"):

        /home/git/.ssh/authorized_keys
        /home/git/.ssh
        /home/git

[Actually, `sshd` requires that even directories *above* `~` (`/`, `/home`,
typically) also must be `go-w`, but that needs root.  And typically
they're already set that way anyway.  (Or if they're not, you've got
bigger problems than gitolite install not working!)]

[doc0]: http://github.com/sitaramc/gitolite/blob/pu/doc/0-INSTALL.mkd
[doc9gas]: http://github.com/sitaramc/gitolite/blob/pu/doc/9-gitolite-and-ssh.mkd