and give a suitable message, fix the doc, etc. (error found by "ry" on #git backlog; couldn't contact him later)
465 lines
20 KiB
Markdown
465 lines
20 KiB
Markdown
# ssh troubleshooting
|
|
|
|
In this document:
|
|
|
|
* <a href="#the_most_common_problems_that_an_admin_will_see">the most common problems that an admin will see</a>
|
|
* <a href="#basic_ssh_troubleshooting">basic ssh troubleshooting</a>
|
|
* <a href="#passphrases_versus_passwords">passphrases versus passwords</a>
|
|
* <a href="#ssh_agent_problems">ssh-agent problems</a>
|
|
* <a href="#basic_ssh_troubleshooting_for_the_main_admin">basic ssh troubleshooting for the main admin</a>
|
|
* <a href="#basic_ssh_troubleshooting_for_a_normal_user">basic ssh troubleshooting for a normal user</a>
|
|
* <a href="#windows_issues">windows issues</a>
|
|
* <a href="#details">details</a>
|
|
* <a href="#files_on_the_server">files on the server</a>
|
|
* <a href="#files_on_client">files on client</a>
|
|
* <a href="#why_two_keys_on_client">why two keys on client</a>
|
|
* <a href="#some_other_tips_and_tricks">some other tips and tricks</a>
|
|
* <a href="#giving_shell_access_to_gitolite_users">giving shell access to gitolite users</a>
|
|
* <a href="#losing_your_admin_key">losing your admin key</a>
|
|
* <a href="#simulating_ssh_copy_id">simulating ssh-copy-id</a>
|
|
|
|
----
|
|
|
|
This document should help you troubleshoot ssh-related problems in accessing
|
|
gitolite *after* the install has completed successfully.
|
|
|
|
In addition, I **strongly** recommend reading
|
|
[doc/9-gitolite-and-ssh.mkd][doc9gas], which is a very detailed look at how
|
|
gitolite uses ssh's features on the server side. Most people don't know ssh
|
|
as well as they *think* they do; even if you don't have any problems right
|
|
now, it's worth skimming over.
|
|
|
|
In addition to both these documents, there's now a program called
|
|
`sshkeys-lint` that you can run on your client. Run it without arguments to
|
|
get help on how to run it and what inputs it needs.
|
|
|
|
**IMPORTANT NOTE: A lot of this document applies only if you installed using
|
|
the [from-client method][fc] . If you used one of the [other 3 methods][o3] ,
|
|
some of it may or may not apply. At the very least, any mention of `ssh
|
|
gitolite` below will change to `ssh git@server`; see [this][urls] for why.**
|
|
|
|
<a name="the_most_common_problems_that_an_admin_will_see"></a>
|
|
|
|
### the most common problems that an admin will see
|
|
|
|
Ironically, these problems **only** happen to the person who installed
|
|
gitolite using easy-install (the "from-client" method in
|
|
[doc/0-INSTALL.mkd][doc0]), and has **utterly failed** to read/heed the
|
|
message that shows up at the end of running that command. This is because
|
|
only the admin has *two* ssh keys to the server (see "basic ssh
|
|
troubleshooting for the main admin" section below for more on this).
|
|
|
|
Both these problems are caused by using the wrong key, thus **bypassing gitolite
|
|
completely**:
|
|
|
|
* you get `fatal: 'reponame' does not appear to be a git repository`, and
|
|
yet you are sure 'reponame' exists, you haven't mis-spelled it, etc.
|
|
|
|
* you are able to clone repositories but are unable to push changes back
|
|
(the error complains about the `GL_RC` environment variable not being set,
|
|
and the `hooks/update` failing in some way).
|
|
|
|
Let us recap the message that appears on a successful run of the "easy-install"
|
|
program; it looks something like this (with suitable values substituted for
|
|
`<user>`, `<server>`, and `<port>`):
|
|
|
|
IMPORTANT NOTE -- PLEASE READ!!!
|
|
*Your* URL for cloning any repo on this server will be
|
|
gitolite:reponame.git
|
|
|
|
*Other* users you set up will have to use
|
|
<user>@<server>:reponame.git
|
|
However, if your server uses a non-standard ssh port, they should use
|
|
ssh://<user>@<server>:<port>/reponame.git
|
|
|
|
If this is your first time installing gitolite, please also:
|
|
tail -31 src/gl-easy-install
|
|
for next steps.
|
|
|
|
The first error above happens if you use `git@server:reponame` instead of
|
|
`gitolite:reponame`. All your repos are actually in a subdirectory pointed to
|
|
by `$REPO_BASE` in the rc file (default: `repositories`). Gitolite internally
|
|
prefixes this before calling the actual git command you invoked, but since
|
|
you're bypassing gitolite completely, this prefixing does not happen, and so
|
|
the repo is not found.
|
|
|
|
The second error happens if you use `git@server:repositories/reponame.git`
|
|
(assuming default `$REPO_BASE` setting) -- that is, you used the full unix
|
|
path. Since the "prefixing" mentioned above is not required, the shell finds
|
|
the repo and clones ok. But when you push, gitolite's **update hook** kicks
|
|
in, and fails to run because you some of the environment variables it is
|
|
expecting are not present.
|
|
|
|
<a name="basic_ssh_troubleshooting"></a>
|
|
|
|
### basic ssh troubleshooting
|
|
|
|
[glb]: http://sitaramc.github.com/0-installing/9-gitolite-basics.html#IMPORTANT_overview_of_ssh
|
|
|
|
I assume the gitolite server is called "server" and the user hosting all the
|
|
gitolite repos is "git". I will also be using "sitaram" as the *gitolite
|
|
username* of the admin.
|
|
|
|
Unless specifically mentioned, all these commands are run on the user's or
|
|
admin's workstation, not on the server.
|
|
|
|
<a name="passphrases_versus_passwords"></a>
|
|
|
|
#### passphrases versus passwords
|
|
|
|
When you create an ssh keypair, you have the option of protecting it with a
|
|
passphrase. When you subsequently use that keypair to access a remote host,
|
|
your *local* ssh client needs to unlock the corresponding private key, and ssh
|
|
will probably ask for the passphrase you set when you created the keypair.
|
|
|
|
Do not confuse or mistake this prompt (`Enter passphrase for key
|
|
'/home/sitaram/.ssh/id_rsa':`) for a password prompt from the remote server!
|
|
|
|
You have two choices to avoid this prompt every time you try to access the
|
|
remote. The first is to create keypairs *without* a passphrase (just hit
|
|
enter when prompted for one). **Be sure to add a passphrase later, once
|
|
everything is working, using `ssh-keygen -p`**.
|
|
|
|
The second is to use `ssh-agent` (or `keychain`, which in turn uses
|
|
`ssh-agent`) or something like that to manage your keys. Other than the next
|
|
section, further discussion of this is out of scope of this document.
|
|
|
|
<a name="ssh_agent_problems"></a>
|
|
|
|
#### ssh-agent problems
|
|
|
|
1. Run `ssh-add -l`. If this responds with either "The agent has no
|
|
identities." or "Could not open a connection to your authentication
|
|
agent.", skip this section.
|
|
|
|
2. However, if it lists some keys, like this:
|
|
|
|
2048 fc:c1:48:1e:06:31:97:a4:8b:fc:37:b2:76:14:c7:53 /home/sitaram/.ssh/id_rsa (RSA)
|
|
2048 d2:e0:7f:fa:1a:89:22:41:bb:06:d9:ff:a7:27:36:5c /home/sitaram/.ssh/sitaram (RSA)
|
|
|
|
then run `ls ~/.ssh` and make sure that all the keypairs you have there
|
|
are represented in the `ssh-add -l` output.
|
|
|
|
3. If you find any keypairs in `~/.ssh` that are not represented in the
|
|
`ssh-add -l` output, add them. For instance, if `ssh-add -l` showed me
|
|
only the `id_rsa` key, but I also had a `sitaram` (and `sitaram.pub`)
|
|
keypair, I'd run `ssh-add ~/.ssh/sitaram` to add it.
|
|
|
|
This is because ssh-agent has a quirk: if `ssh-add -l` shows *any* keys at
|
|
all, ssh will only use those keys. Even if you explicitly specify an unlisted
|
|
key using `ssh -i` or an `identityfile` directive in the config file, it won't
|
|
use it.
|
|
|
|
<a name="basic_ssh_troubleshooting_for_the_main_admin"></a>
|
|
|
|
#### basic ssh troubleshooting for the main admin
|
|
|
|
You're the "main admin" if you're trying to access gitolite from the same
|
|
workstation and user account where you ran the "easy install" command. You
|
|
should have two keypairs in your `~/.ssh` directory. The pair called `id_rsa`
|
|
(and `id_rsa.pub`) was probably the first one you created, and you used this
|
|
to get passwordless (pubkey based) access to the server (which was a
|
|
pre-requisite for running the easy install command).
|
|
|
|
The second keypair has the same name as the last argument in the easy install
|
|
command you ran (in my case, `sitaram` and `sitaram.pub`). It was probably
|
|
created by the easy install script, and is the key used for gitolite access.
|
|
|
|
In addition, you should have a "gitolite" paragraph in your `~/.ssh/config`,
|
|
looking something like this:
|
|
|
|
host gitolite
|
|
user git
|
|
hostname server
|
|
identityfile ~/.ssh/sitaram
|
|
|
|
If any of these are not true, you did something funky in your install; email
|
|
me or hop onto #git and hope for the best ;-)
|
|
|
|
Otherwise, run these checks:
|
|
|
|
1. `ssh git@server` should get you a command line.
|
|
|
|
If it asks you for a password, then your `id_rsa` keypair changed after
|
|
you ran the easy install, or someone fiddled with the
|
|
`~/.ssh/authorized_keys` file on the server.
|
|
|
|
If it prints [gitolite version and access info][myrights], you managed to
|
|
overwrite the `id_rsa` keypair with the `sitaram` keypair, or something
|
|
equally weird.
|
|
|
|
2. `ssh gitolite info` should print some [gitolite version and access
|
|
info][myrights]. If you get the output of the GNU info command instead,
|
|
you probably reused your `id_rsa` keypair as your `sitaram` keypair, or
|
|
overwrote the `sitaram` keypair with the `id_rsa` keypair.
|
|
|
|
There are many ways to fix this, depending on where and what the damage is.
|
|
The most generic way (and therefore time-taking) is to re-install gitolite
|
|
from scratch:
|
|
|
|
* make a backup of your gitolite-admin repo clone somewhere (basically your
|
|
"keydir/*.pub" and your "conf/gitolite.conf"). If necessary get these
|
|
files from the server's `~/.gitolite` directory.
|
|
* log on to the server somehow (using some other account, using a password,
|
|
su-ing in, etc) and delete `~/.ssh/authorized_keys`. Rename or move aside
|
|
`~/.gitolite` so that also looks like it is missing.
|
|
* back on your workstation, make sure you have 2 keypairs (`id_rsa` and
|
|
`sitaram`, along with corresponding `.pub` files). Create them if needed.
|
|
Also make sure they are *different* and not a copy of each other :-)
|
|
* install gitolite normally:
|
|
* run `ssh-copy-id -i ~/.ssh/id_rsa git@server` to get passwordless
|
|
access to the server. (Mac users may have to do this step manually)
|
|
* make sure `ssh git@server pwd` prints the `$HOME` of `git@server`
|
|
**without** asking for a password. Do not proceed till this works.
|
|
* run easy install again, (in my case: `cd gitolite-source;
|
|
src/gl-easy-install -q git server sitaram`)
|
|
* go to your gitolite-admin repo clone, and copy `conf/gitolite.conf` and
|
|
`keydir/*.pub` from your backup to this directory
|
|
* copy (be sure to overwrite!) `~/.ssh/sitaram.pub` also to keydir
|
|
* now `git add keydir; git commit; git push -f`
|
|
|
|
That's a long sequence but it should work.
|
|
|
|
<a name="basic_ssh_troubleshooting_for_a_normal_user"></a>
|
|
|
|
#### basic ssh troubleshooting for a normal user
|
|
|
|
For a normal user, life is much simpler. They should have only one pubkey,
|
|
which was previously sent to the gitolite admin to add into the admin repo's
|
|
`keydir` as "user.pub", and then "user" given permissions to some repo.
|
|
|
|
`ssh git@server info` should get you [gitolite version and access
|
|
info][myrights]. If it asks you for a password, your pubkey was not sent to
|
|
the server properly. Check with your admin.
|
|
|
|
[myrights]: http://github.com/sitaramc/gitolite/blob/pu/doc/3-faq-tips-etc.mkd#myrights
|
|
|
|
If it gets you the GNU info command output, you have shell access. This means
|
|
you had command line access to the server *before* you were added as a
|
|
gitolite user. If you send that same key to your gitolite admin to include in
|
|
the admin repo, it won't work. For reasons why, see below.
|
|
|
|
<a name="windows_issues"></a>
|
|
|
|
### windows issues
|
|
|
|
On windows, I have only used msysgit, and the openssh that comes with it.
|
|
Over time, I have grown to distrust putty/plink due to the number of people
|
|
who seem to have trouble when those beasts are involved (I myself have never
|
|
used them for any kind of git access). If you have unusual ssh problems that
|
|
just don't seem to have any explanation, try removing all traces of
|
|
putty/plink, including environment variables, etc., and then try again.
|
|
|
|
If you can offer an *authoritative* account of the complications involved, and
|
|
how to resolve them and get things working, I'd be happy to credit you and
|
|
include it, either directly here if it is short enough or just an external
|
|
link, or in contrib/ if it's a longer piece of text.
|
|
|
|
<a name="details"></a>
|
|
|
|
### details
|
|
|
|
Here's how it all hangs together.
|
|
|
|
<a name="files_on_the_server"></a>
|
|
|
|
#### files on the server
|
|
|
|
* the authkeys file; this contains one line containing the pubkey of each
|
|
user who is permitted to login without a password.
|
|
|
|
Pubkey lines that give shell access look like this:
|
|
|
|
ssh-rsa AAAAB3NzaC[snip]uPjrUiAUew== /home/sitaram/.ssh/id_rsa
|
|
|
|
On a typical server there will be only one or two of these lines.
|
|
|
|
Note that the last bit (`/home/sitaram/.ssh/id_rsa`) is purely a *comment*
|
|
field and can be anything. Also, the actual lines are much longer, about
|
|
400 characters; I snipped 'em in the middle, as you can see.
|
|
|
|
In contrast, pubkey lines that give access to git repos hosted by gitolite
|
|
look like this:
|
|
|
|
command="[some path]src/gl-auth-command sitaram",[some restrictions] ssh-rsa AAAAB3NzaC[snip]s18OnB42oQ== sitaram@sita-lt
|
|
|
|
You will have many more of these lines -- one for every pubkey file in
|
|
`keydir/` of your gitolite-admin repo, with the corresponding username in
|
|
place of "sitaram" in the example above.
|
|
|
|
The "command=" at the beginning ensures that when someone with the
|
|
corresponding private key logs in, they don't get a shell. Instead, the
|
|
`gl-auth-command` program is run, and (in this example) is given the
|
|
argument `sitaram`. This is how gitolite is invoked, (and is told the
|
|
user logging in is "sitaram").
|
|
|
|
<a name="files_on_client"></a>
|
|
|
|
#### files on client
|
|
|
|
* default keypair; used to get shell access to servers. You would have
|
|
copied this pubkey to the gitolite server in order to log in without a
|
|
password. (On Linux systems you may have used `ssh-copy-id` to do that).
|
|
You would have done this *before* you ran the easy install script, because
|
|
otherwise easy install won't run!
|
|
|
|
~/.ssh/id_rsa
|
|
~/.ssh/id_rsa.pub
|
|
|
|
* gitolite keypair; the "sitaram" in this is the 3rd argument to the
|
|
`src/gl-easy-install` command you ran; the easy install script does the
|
|
rest
|
|
|
|
~/.ssh/sitaram
|
|
~/.ssh/sitaram.pub
|
|
|
|
* config file; this file has an entry for gitolite access:
|
|
|
|
~/.ssh/config
|
|
|
|
To understand why we need that, let's step back a bit. Normally, you
|
|
might expect to access gitolite repos like this:
|
|
|
|
ssh://git@server/reponame.git
|
|
|
|
But this won't work, because this ends up using the *default* keypair
|
|
(normally), which gives you a command line. Which means it won't invoke
|
|
the `gl-auth-command` program at all, and so none of gitolite's access
|
|
control will work.
|
|
|
|
<a name="altkey"></a>
|
|
|
|
You need to force ssh to use the *other* keypair when performing a git
|
|
operation. With normal ssh, that would be
|
|
|
|
ssh -i ~/.ssh/sitaram git@server
|
|
|
|
but git does not support putting an alternate keypair in the URL.
|
|
|
|
Luckily, ssh has a very convenient way of capturing all the connection
|
|
information (username, hostname, port number (if it's not the default 22),
|
|
and keypair to be used) in one "paragraph" of `~/.ssh/config`. This is
|
|
what the para looks like for us (the easy install script puts it there the
|
|
first time):
|
|
|
|
host gitolite
|
|
user git
|
|
hostname server
|
|
identityfile ~/.ssh/sitaram
|
|
|
|
(The "gitolite" can be anything you want of course; it's like a group name
|
|
for all the stuff below it). This ensures that typing
|
|
|
|
ssh gitolite
|
|
|
|
is equivalent to
|
|
|
|
ssh -i ~/.ssh/sitaram git@server
|
|
|
|
and therefore this:
|
|
|
|
git clone gitolite:reponame.git
|
|
|
|
now works as expected, invoking the special keypair instead of the default
|
|
one.
|
|
|
|
<a name="why_two_keys_on_client"></a>
|
|
|
|
#### why two keys on client
|
|
|
|
[This section is only applicable to installs done using the "from-client"
|
|
method; see [doc/0-INSTALL.mkd][doc0] for details].
|
|
|
|
Why do I (the admin) need two **different** keypairs?
|
|
|
|
There are two types of access the admin will make to the server: a normal
|
|
login, to get a shell prompt, and gitolite access (clone/fetch/push etc). The
|
|
first access needs an authkeys line *without* any "command=" restrictions,
|
|
while the second requires a line *with* such a restriction.
|
|
|
|
And we can't use the same key for both because there is no way to disambiguate
|
|
them; the ssh server will always (*always*) pick the first one in sequence
|
|
when the key is offered by the ssh client.
|
|
|
|
So the next question is usually "I have other ways to get a shell on that
|
|
account (like `su - git` from some other account), so why do I need a key for
|
|
shell access at all?"
|
|
|
|
The answer to this is that the "easy install" script, being written for the
|
|
most general case, needs shell access via ssh to do its stuff. If you have
|
|
access otherwise, you really should use one of the other 3 install methods to
|
|
install gitolite. Please see the [install doc][install] for details.
|
|
|
|
<a name="some_other_tips_and_tricks"></a>
|
|
|
|
### some other tips and tricks
|
|
|
|
<a name="giving_shell_access_to_gitolite_users"></a>
|
|
|
|
#### giving shell access to gitolite users
|
|
|
|
We've managed (thanks to an idea from Jesse Keating) to make it possible for a
|
|
single key to allow both gitolite access *and* shell access.
|
|
|
|
This is done by copying the pubkey (to which you want to give shell access) to
|
|
the server and running either
|
|
|
|
cd $HOME/.gitolite # assuming default $GL_ADMINDIR in ~/.gitolite.rc
|
|
src/gl-tool shell-add ~/foo.pub
|
|
|
|
or
|
|
|
|
gl-tool shell-add ~/foo.pub
|
|
|
|
The first method is to be used if you used the **user-install** mode, while
|
|
the second method is for the **system-install followed by user-setup** mode
|
|
(see doc/0-INSTALL.mkd, section on "install methods", for more on this)
|
|
|
|
**IMPORTANT UPGRADE NOTE**: previous implementations of this feature were
|
|
crap. There was no easy/elegant way to ensure that someone who had repo admin
|
|
access would not manage to get himself shell access.
|
|
|
|
Giving someone shell access requires that you should have shell access in the
|
|
first place, so the simplest way is to enable it from the server side only.
|
|
|
|
<a name="losing_your_admin_key"></a>
|
|
|
|
#### losing your admin key
|
|
|
|
If you lost the admin key, and need to re-establish ownership of the
|
|
gitolite-admin repository with a fresh key, take a look at the
|
|
`src/gl-emergency-addkey` program. You will need shell access to the server
|
|
of course. Run it without arguments to get instructions.
|
|
|
|
<a name="simulating_ssh_copy_id"></a>
|
|
|
|
#### simulating ssh-copy-id
|
|
|
|
don't have `ssh-copy-id`? This is broadly what that command does, if you want
|
|
to replicate it manually. The input is your pubkey, typically
|
|
`~/.ssh/id_rsa.pub` from your client/workstation.
|
|
|
|
* it copies it to the server as some file
|
|
|
|
* it appends that file to `~/.ssh/authorized_keys` on the server
|
|
(creating it if it doesn't already exist)
|
|
|
|
* it then makes sure that all these files/directories have go-w perms
|
|
set (assuming user is "git"):
|
|
|
|
/home/git/.ssh/authorized_keys
|
|
/home/git/.ssh
|
|
/home/git
|
|
|
|
[Actually, `sshd` requires that even directories *above* `~` (`/`, `/home`,
|
|
typically) also must be `go-w`, but that needs root. And typically
|
|
they're already set that way anyway. (Or if they're not, you've got
|
|
bigger problems than gitolite install not working!)]
|
|
|
|
[doc0]: http://github.com/sitaramc/gitolite/blob/pu/doc/0-INSTALL.mkd
|
|
[doc9gas]: http://github.com/sitaramc/gitolite/blob/pu/doc/9-gitolite-and-ssh.mkd
|
|
[install]: http://github.com/sitaramc/gitolite/blob/pu/doc/0-INSTALL.mkd
|
|
[o3]: http://github.com/sitaramc/gitolite/blob/pu/doc/0-INSTALL.mkd#installation_and_setup
|
|
[fc]: http://github.com/sitaramc/gitolite/blob/pu/doc/0-INSTALL.mkd#from_client_method_install_from_the_client_to_the_server
|
|
[urls]: http://github.com/sitaramc/gitolite/blob/pu/doc/0-INSTALL.mkd#URLs_for_gitolite_managed_repos
|