gitolite/doc/ssh-troubleshooting.mkd

# ssh troubleshooting

----

**This document must be read in full the first time.  If you start from some
nice looking section in the middle it may not help you unless you're already
an expert at ssh**.

In this document:

  * <a href="#_IMPORTANT_READ_THIS_FIRST">IMPORTANT -- READ THIS FIRST</a>
      * <a href="#_caveats">caveats</a>
      * <a href="#_naming_conventions_used">naming conventions used</a>
      * <a href="#_taking_stock_relevant_files_and_directories">taking stock -- relevant files and directories</a>
      * <a href="#_normal_gitolite_key_handling">normal gitolite key handling</a>
  * <a href="#_Other_resources_">(Other resources)</a>
  * <a href="#_common_problems">common problems</a>
  * <a href="#_step_by_step">step by step</a>
  * <a href="#_random_tips_tricks_and_notes">random tips, tricks, and notes</a>
      * <a href="#_giving_shell_access_to_gitolite_users">giving shell access to gitolite users</a>
      * <a href="#_losing_your_admin_key">losing your admin key</a>
      * <a href="#_simulating_ssh_copy_id">simulating ssh-copy-id</a>
      * <a href="#_problems_with_using_non_openssh_public_keys">problems with using non-openssh public keys</a>
      * <a href="#_windows_issues">windows issues</a>
  * <a href="#_appendix_1_ssh_daemon_asks_for_a_password">appendix 1: ssh daemon asks for a password</a>
  * <a href="#_appendix_2_which_key_is_which_running_sshkeys_lint">appendix 2: which key is which -- running sshkeys-lint</a>
      * <a href="#_typical_cause_s_">typical cause(s)</a>
  * <a href="#_appendix_3_ssh_client_may_not_be_offering_the_right_key">appendix 3: ssh client may not be offering the right key</a>
  * <a href="#_appendix_4_making_git_use_the_right_key_using_ssh_config_">appendix 4: making git use the right key using `~/.ssh/config`</a>
  * <a href="#_appendix_5_why_bypassing_gitolite_causes_a_problem">appendix 5: why bypassing gitolite causes a problem</a>

----

This document should help you troubleshoot ssh-related problems in installing
and accessing gitolite.

<a name="_IMPORTANT_READ_THIS_FIRST"></a>

### IMPORTANT -- READ THIS FIRST

<a name="_caveats"></a>

#### caveats

  * Before reading this document, it is **mandatory** to read and **completely
    understand** [doc/gitolite-and-ssh.mkd][doc9gas], which is a very detailed
    look at how gitolite uses ssh's features on the server side.  Don't assume
    you know all that; if you knew it, you wouldn't be needing *this* document
    either!

  * This document, and others linked from this, together comprise all the help
    I can give you in terms of the ssh aspect of using gitolite.  If you're
    installing gitolite, you're a "system admin", like it or not.  Ssh is
    therefore a necessary skill.  Please take the time to learn at least
    enough to get passwordless access working.

  * Please note that authentication is not really gitolite's job at all.  I'd
    rather spend time on actual gitolite features, code, and documentation
    than authentication (i.e., ssh, in the common case).

    Surprised?  [This][wo] might help explain better.

----

<a name="_naming_conventions_used"></a>

#### naming conventions used

  * Your workstation is the **client**.  Your userid on the client does not
    matter, and it has no relation to your gitolite username.

  * the server is called **server** and the "hosting user" is **git**.  If
    this is an RPM/DEB install, the hosting user is probably called
    "gitolite", however we will use "git" in this document.

  * If you installed gitolite using the `from-client` method of installation,
    then, in the following description, wherever you see `git@server` just
    substitute `gitolite`.  Example: `git clone git@server:foo` becomes `git
    clone gitolite:foo`.

      * No changes are required if you installed using one of the other 3
        methods.

      * Note: to determine which method you used, in case you forgot, run

            grep GL_PACKAGE ~/.gitolite.rc

        on the server.  If there are only two, **commented**, lines, you're
        using the `from-client` method.

<a name="_taking_stock_relevant_files_and_directories"></a>

#### taking stock -- relevant files and directories

  * the client has a `~/.ssh` containing a few keypairs.  It may also have a
    `config` file (if you used the [from-client][fc] method of install it
    definitely will have one).

  * the client also has a clone of the "gitolite-admin" repo, which contains a
    bunch of `*.pub` files in `keydir`.  We assume this clone is in `$HOME`;
    if it is not, adjust instructions accordingly when needed.

  * The git user on the server has a `~/.ssh/authorized_keys` file that the
    ssh daemon uses to authenticate incoming users.  We often call this file
    **authkeys** to save typing, and it always means the one on the server
    (we're not interested in this file on the client side).

  * the server also has a `~/.gitolite/keydir` which contains a bunch of
    `*.pub` files.

<a name="_normal_gitolite_key_handling"></a>

#### normal gitolite key handling

Here's how normal gitolite key handling works:

  * (on client) pub key changes like adding new ones, deleting old ones, etc.,
    are done in the `keydir` directory in the gitolite-admin repo clone.  Then
    the admin `git add`s and `git commit`s those changes, then `git push`es
    them to the server.

  * (on server) a successful push from the client makes git invoke the
    post-update hook in the gitolite-admin repo.  This hook is installed by
    gitolite, and it does a bunch of things which are quite transparent to
    the admin, but we'll describe briefly here:

      * the pubkey files from this push are checked-out into
        `~/.gitolite/keydir` (and similarly the config files into
        `~/.gitolite/conf`)

      * the "compile" script then runs, which uses these files to populate
        `~/.ssh/authorized_keys` on the server

        The authkeys file may have other, (non-gitolite) keys also.  Those
        lines are preserved.  Gitolite only touches lines that are found
        between gitolite's "marker" lines (`# gitolite start` and `# gitolite
        end`).

<a name="_Other_resources_"></a>

### (Other resources)

  * people who think installing gitolite is too hard should take a look at
    this [transcript][] to **see how simple it *actually* is**.  This
    transcript uses the 'from-client' method of install.

  * someone also wrote a tutorial, see [here][tut].  This uses the 'non-root'
    method of install, to provide a different perspective.

<a name="_common_problems"></a>

### common problems

Since I'm pretty sure at least some of you didn't bother to read the
"IMPORTANT: PLEASE READ FIRST" section above, let me take a minute to point
you there again.  Especially the first bullet.

Done?  OK, read on...

The following problem(s) indicate that pubkey access is not working at all, so
you should start with appendix 1.  If that doesn't fix the problem, continue
with the other appendices in sequence.

  * running any git clone/fetch/ls-remote or just `ssh git@server info` asks
    you for a password.

The following problem(s) indicate that your pubkey is bypassing gitolite and
going straight to a shell.  You should start with appendix 2 and continue with
the rest in sequence.  Appendix 5 has some background info.

  * running `ssh git@server info` gets you the output of the GNU 'info'
    command instead of gitolite's version and access info.

  * running `git clone git@server:repositories/reponame` (note presence of
    `repositories/` in URL) works.

    [A proper gitolite key will only let you `git clone git@server:reponame`
    (note absence of `repositories/`)]

  * you are able to clone repositories but are unable to push changes back
    (the error complains about the `GL_RC` environment variable not being set,
    and the `hooks/update` failing in some way).

    [If you run `git remote -v` you will find that your clone URL included the
    `repositories/` described above!]

  * conversely, using the correct syntax, `git clone git@server:reponame`
    (note absence of `repositories/` in the URL), gets you `fatal: 'reponame'
    does not appear to be a git repository`, and yet you are sure 'reponame'
    exists, you haven't mis-spelled it, etc.

<a name="_step_by_step"></a>

### step by step

Since I'm pretty sure at least some of you didn't bother to read the
"IMPORTANT: PLEASE READ FIRST" section above, let me take a minute to point
you there again.  Especially the first bullet.

Done?  OK, now the general outline for ssh troubleshooting is this:

  * make sure the server's overall setup even *allows* pubkey based login.
    I.e., check that git fetch/clone/ls-remote commands or a plain `ssh
    git@server info` do NOT ask for a password.  If you do get asked for a
    password, see appendix 1.

  * match client-side pubkeys (`~/.ssh/*.pub`) with the server's authkeys
    file.  To do this, run `sshkeys-lint`, which tells you in detail what key
    has what access.  See appendix 2.

  * at this point, we know that we have the right key, and that if sshd
    receives that key, things will work.  But we're not done yet.  We still
    need to make sure that this specific key is being offered/sent by the
    client, instead of the default key.  See appendix 3 and 4.

<a name="_random_tips_tricks_and_notes"></a>

### random tips, tricks, and notes

<a name="_giving_shell_access_to_gitolite_users"></a>

#### giving shell access to gitolite users

We've managed (thanks to an idea from Jesse Keating) to make it possible for a
single key to allow both gitolite access *and* shell access.

This is done by copying the pubkey (to which you want to give shell access) to
the server and running either

    cd $HOME/.gitolite  # assuming default $GL_ADMINDIR in ~/.gitolite.rc
    src/gl-tool shell-add ~/foo.pub

or

    gl-tool shell-add ~/foo.pub

The first method is applicable if you installed using the **from-client**
method, while the second method is for any of the other three (see
doc/1-INSTALL.mkd, section on "install methods", for more on this)

**IMPORTANT UPGRADE NOTE**: previous implementations of this feature were
crap.  There was no easy/elegant way to ensure that someone who had repo admin
access would not manage to get himself shell access.

Giving someone shell access requires that you should have shell access in the
first place, so the simplest way is to enable it from the server side only.

<a name="_losing_your_admin_key"></a>

#### losing your admin key

If you lost the admin key, and need to re-establish ownership of the
gitolite-admin repository with a fresh key, get a shell on the server and use
the program called `gl-admin-push` that comes with gitolite.  See instructions
[here][gssp].

[gssp]: http://sitaramc.github.com/gitolite/doc/3-faq-tips-etc.html#_gl_admin_push_bypassing_gitolite_for_the_gitolite_admin_repo

<a name="_simulating_ssh_copy_id"></a>

#### simulating ssh-copy-id

don't have `ssh-copy-id`?  This is broadly what that command does, if you want
to replicate it manually.  The input is your pubkey, typically
`~/.ssh/id_rsa.pub` from your client/workstation.

  * it copies it to the server as some file

  * it appends that file to `~/.ssh/authorized_keys` on the server
    (creating it if it doesn't already exist)

  * it then makes sure that all these files/directories have go-w perms
    set (assuming user is "git"):

        /home/git/.ssh/authorized_keys
        /home/git/.ssh
        /home/git

[Actually, `sshd` requires that even directories *above* `~` (`/`, `/home`,
typically) also must be `go-w`, but that needs root.  And typically
they're already set that way anyway.  (Or if they're not, you've got
bigger problems than gitolite install not working!)]

<a name="_problems_with_using_non_openssh_public_keys"></a>

#### problems with using non-openssh public keys

Gitolite accepts public keys only in openssh format.  Trying to use an "ssh2"
key (used by proprietary SSH software) results in:

    WARNING: a pubkey file can only have one line (key); ignoring YourName.pub

To convert ssh2-compatible keys to openssh run:

    ssh-keygen -i -f /tmp/ssh2/YourName.pub > /tmp/openssh/YourName.pub

then use the resulting pubkey as you normally would in gitolite.

<a name="_windows_issues"></a>

#### windows issues

On windows, I have only used msysgit, and the openssh that comes with it.
Over time, I have grown to distrust putty/plink due to the number of people
who seem to have trouble when those beasts are involved (I myself have never
used them for any kind of git access).  If you have unusual ssh problems that
just don't seem to have any explanation, try removing all traces of
putty/plink, including environment variables, etc., and then try again.

Thankfully, someone contributed [contrib/putty.mkd][putty].

----

<a name="_appendix_1_ssh_daemon_asks_for_a_password"></a>

### appendix 1: ssh daemon asks for a password

>   **NOTE**: This section should be useful to anyone trying to get
>   password-less access working.  It is not necessarily specific to gitolite,
>   so keep that in mind if the wording feels a little more general than you
>   were expecting.

You have generated a keypair on your workstation (`ssh-keygen`) and copied the
public part of it (`~/.ssh/id_rsa.pub`, by default) to the server.

On the server you have appended this file to `~/.ssh/authorized_keys`.  Or you
ran something, like the `gl-setup` or `gl-easy-install` steps during a
gitolite install, which should have done that for you.

You now expect to log in without having to type in a password, but when you
try, you are being asked for a password.

This is a quick checklist:

  * make sure you're being asked for a password and not a pass*phrase*.  Do
    not confuse or mistake a prompt saying `Enter passphrase for key
    '/home/sitaram/.ssh/id_rsa':` for a password prompt from the remote
    server!

    When you create an ssh keypair using `ssh-keygen`, you have the option of
    protecting it with a passphrase.  When you subsequently use that keypair
    to access a remote host, your *local* ssh client needs to unlock the
    corresponding private key, and ssh will probably ask for the passphrase
    you set when you created the keypair.

    You have two choices to avoid this prompt every time you try to use the
    private key.  The first is to create keypairs *without* a passphrase (just
    hit enter when prompted for one).  **Be sure to add a passphrase later,
    once everything is working, using `ssh-keygen -p`**.

    The second is to use `ssh-agent` (or `keychain`, which in turn uses
    `ssh-agent`) or something like that to manage your keys.  Other than
    discussing one more potential trouble-spot with ssh-agent (see below),
    further discussion of ssh-agent/keychain is out of scope of this document.

  * ssh is very sensitive to permissions.  An extremely conservative setup is
    given below, but be sure to do this on **both the client and the server**:

        cd $HOME
        chmod go-rwx .
        chmod -R go-rwx .ssh

  * actually, every component of the path to `~/.ssh/authorized_keys` all the
    way upto the root directory must be at least `chmod go-w`.  So be sure to
    check `/` and `/home` also.

  * while you're doing this, make sure the owner and group info for each of
    these components are correct.  `ls -ald ~ ~/.ssh ~/.ssh/authorized_keys`
    will tell you what they are.

  * you may also want to check `/etc/ssh/sshd_config` to see if the "git" user
    is allowed to login at all.  For example, if that file contains an
    `AllowUsers` config entry, then only users mentioned in that line are
    allowed to log in!

  * some OSs/distributions require that the "git" user should have a password
    and/or not be a locked account.  You may want to check that as well.

  * if all that fails, log onto the server as root, `cd /var/log`, and look
    for a file called `auth.log` or `secure` or some such name.  Look inside
    this file for messages matching the approximate time of your last attempt
    to login, to see if they tell you what is the problem.

<a name="_appendix_2_which_key_is_which_running_sshkeys_lint"></a>

### appendix 2: which key is which -- running sshkeys-lint

Follow these steps on the client:

  * get a copy of `~/.ssh/authorized_keys` from the server and put it in
    `/tmp/foo` or something

  * cd to `~/.ssh`

  * run `sshkeys-lint /tmp/foo`

This checks every `*.pub` file in your `~/.ssh`, in combination with the
server's authkeys file.  It then tells you which key gets you what type of
access (if any) to the server.

Note that it is not trying to log in or anything -- it's just comparing bits
of text (the contents of `/tmp/foo` and of each of the `*.pub` files).

>   Note: It's also a stand-alone program, so even if your gitolite version is
>   old, you can safely bring over just this program from a more recent
>   gitolite and use it, without having to upgrade gitolite itself.

If the pubkey file you're interested in appears to have the correct access to
the server, you're done with this step.

Otherwise you have to rename some keypairs and try again to get the effect you
need.  Be careful:

  * do not just rename the ".pub" file; you will have to rename the
    corresponding private key also (the one with the same basename but without
    an extension)

  * if you're running ssh-agent, you may have to delete (using `ssh-add -D`)
    and re-add identities for it to pick up the renamed ones correctly

<a name="_typical_cause_s_"></a>

#### typical cause(s)

For server-based installs (i.e., not the 'from-client' install mode), the
admin often has passwordless shell access to `git@server` already, and then
used that same key to get access to gitolite (i.e., copied that same pubkey as
YourName.pub and gave it to gitolite in some way).

As a result, the same key appears twice in the authkeys file now, and since
the ssh server will always use the first match, the second occurrence (which
invokes gitolite) is ignored.

To fix this, you have to use a different keypair for gitolite access.  The
best way to do this is to create a new keypair, copy the pubkey to the server
as YourName.pub, then run `gl-setup YourName.pub` on the server.  Remember to
adjust your agent identities using ssh-add -D and ssh-add if you're using
ssh-agent, otherwise these new keys may not work.

NOTE: If you installed using the 'from-client' install method, `gl-setup`
won't work.  You will have to use the more generic method described
[here][gssp].

<a name="_appendix_3_ssh_client_may_not_be_offering_the_right_key"></a>

### appendix 3: ssh client may not be offering the right key

  * make sure the right private key is being offered.  Run ssh in very
    verbose mode and look for the word "Offering", like so:

        ssh -vvv user@host pwd 2> >(grep -i offer)

    If some keys *are* being offered, but not the key that was supposed to be
    used, you may be using ssh-agent (next bullet).  You may also need to
    setup/edit `~/.ssh/config` (appendix 4).

  * (ssh-agent issues) If `ssh-add -l` responds with either "The agent has no
    identities." or "Could not open a connection to your authentication
    agent.", then you can skip this bullet.

    However, if `ssh-add -l` lists *any* keys at all, then something weird
    happens.  Due to a quirk in ssh-agent, ssh will now *only* use one of
    those keys, *even if you explicitly ask* for some other key to be used.

    In that case, add the key you want using `ssh-add ~/.ssh/YourName` and try
    the access again.

<a name="_appendix_4_making_git_use_the_right_key_using_ssh_config_"></a>

### appendix 4: making git use the right key using `~/.ssh/config`

If you have several pubkeys with access to the same server, you **need** to
create/edit a file called `~/.ssh/config` to make sure git picks up the right
one when it accesses the server.  There is no other way to do this, as far as
I know.

  * (side note) this is *mandatory* for the 'from-client' method of install,
    and indeed that mode automatically sets up all this anyway.  This install
    method runs from the client, so it clearly needs a shell-capable key which
    is distinct from the gitolite key for the admin user.

What you need is to add something like this to `~/.ssh/config`:

    host gitolite
         user git
         hostname server
         identityfile ~/.ssh/YourName

Now you can use `gitolite:reponame` as the URL to make ssh use the named
key.  (Using `git@server:reponame` will end up using the default key
`id_rsa` and, presumably, bypass gitolite).

----

[doc9gas]: http://sitaramc.github.com/gitolite/doc/gitolite-and-ssh.html
[install]: http://sitaramc.github.com/gitolite/doc/1-INSTALL.html
[o3]: http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#methods
[fc]: http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#fc
[urls]: http://sitaramc.github.com/gitolite/doc/1-INSTALL.html#URLs_for_gitolite_managed_repos
[repout]: http://sitaramc.github.com/gitolite/doc/report-output.html
[transcript]: http://sitaramc.github.com/gitolite/doc/install-transcript.html
[openssh56]: http://www.openssh.org/txt/release-5.6
[tut]: http://sites.google.com/site/senawario/home/gitolite-tutorial
[wo]: http://sitaramc.github.com/gitolite/doc/authentication-vs-authorisation.html
[putty]: http://sitaramc.github.com/gitolite/contrib/putty.html

<a name="_appendix_5_why_bypassing_gitolite_causes_a_problem"></a>

### appendix 5: why bypassing gitolite causes a problem

When you bypass gitolite, you end up running your normal shell instead of the
special gitolite entry point script `gl-auth-command`.

This means commands (like 'info') are interpreted by the shell instead of
gitolite.

It also means git operations look for repos in `$HOME`.

However, gitolite places all your repos in a subdirectory pointed to by
`$REPO_BASE` in the rc file (default: `repositories`), and internally prefixes
this before calling the actual git command you invoked.  Thus, the pathname of
the repo that you use on the client is almost never the correct pathname on
the server.  (This is by design.  Don't argue...)

This means that, you get 2 kinds of errors if you bypass gitolite

  * when you use `git@server:reponame` with a key that bypasses gitolite
    (i.e., gets you a shell), this prefixing does not happen, and so the repo
    is not found.  Neither a clone/fetch nor a push will work.

  * conversely, consider `git@server:repositories/reponame.git`.  The clone
    operation will work -- you're using the full Unix path, (assuming default
    `$REPO_BASE` setting), and so the shell finds the repo where you said it
    would be.  However, when you push, gitolite's **update hook** kicks in,
    and fails to run because some of the environment variables it is expecting
    are not present.