445 lines
18 KiB
Markdown
445 lines
18 KiB
Markdown
# ssh troubleshooting
|
|
|
|
In this document:
|
|
|
|
* the most common problems that an admin will see
|
|
* basic ssh troubleshooting
|
|
* passphrases versus passwords
|
|
* ssh-agent problems
|
|
* basic ssh troubleshooting for the main admin
|
|
* basic ssh troubleshooting for a normal user
|
|
* details
|
|
* files on the server
|
|
* files on client
|
|
* why two keys on client
|
|
* more complex ssh setups
|
|
* two gitolite servers to manage?
|
|
* giving shell access to gitolite users
|
|
|
|
----
|
|
|
|
This document should help you troubleshoot ssh-related problems in accessing
|
|
gitolite *after* the install has completed successfully.
|
|
|
|
In addition, I **strongly** recommend reading [this document][glb] -- it's a
|
|
very detailed look at how gitolite uses ssh's features on the server side.
|
|
Most people don't know ssh as well as they *think* they do; even if you don't
|
|
have any problems right now, it's worth skimming over.
|
|
|
|
In addition to both these documents, there's now a program called
|
|
`sshkeys-lint` that you can run on your client. Run it without arguments to
|
|
get help on how to run it and what inputs it needs.
|
|
|
|
### the most common problems that an admin will see
|
|
|
|
Ironically, these problems **only** happen to the person who installed
|
|
gitolite using easy-install, and has **utterly failed** to read/heed the
|
|
message that shows up at the end of running that command. This is because
|
|
only the admin has *two* ssh keys to the server (see "basic ssh
|
|
troubleshooting for the main admin" section below for more on this).
|
|
|
|
Both these problems are caused by using the wrong key, thus **bypassing gitolite
|
|
completely**:
|
|
|
|
* you get `fatal: 'reponame' does not appear to be a git repository`, and
|
|
yet you are sure 'reponame' exists, you haven't mis-spelled it, etc.
|
|
|
|
* you are able to clone repositories but are unable to push changes back.
|
|
|
|
Let us recap the message that appears on a successful run of the "easy-install"
|
|
program; it looks something like this (with suitable values substituted for
|
|
`<user>`, `<server>`, and `<port>`):
|
|
|
|
IMPORTANT NOTE -- PLEASE READ!!!
|
|
*Your* URL for cloning any repo on this server will be
|
|
gitolite:reponame.git
|
|
|
|
*Other* users you set up will have to use
|
|
<user>@<server>:reponame.git
|
|
However, if your server uses a non-standard ssh port, they should use
|
|
ssh://<user>@<server>:<port>/reponame.git
|
|
|
|
If this is your first time installing gitolite, please also:
|
|
tail -31 src/gl-easy-install
|
|
for next steps.
|
|
|
|
The first error above happens if you use `git@server:reponame` instead of
|
|
`gitolite:reponame`. All your repos are actually in a subdirectory pointed to
|
|
by `$REPO_BASE` in the rc file (default: `repositories`). Gitolite internally
|
|
prefixes this before calling the actual git command you invoked, but since
|
|
you're bypassing gitolite completely, this prefixing does not happen, and so
|
|
the repo is not found.
|
|
|
|
The second error happens if you use `git@server:repositories/reponame.git`
|
|
(assuming default `$REPO_BASE` setting) -- that is, you used the full unix
|
|
path. Since the "prefixing" mentioned above is not required, the shell finds
|
|
the repo and clones ok. But when you push, gitolite's **update hook** kicks
|
|
in, and fails to run because you some of the environment variables it is
|
|
expecting are not present.
|
|
|
|
----
|
|
|
|
<a name="basic"></a>
|
|
|
|
### basic ssh troubleshooting
|
|
|
|
[glb]: http://sitaramc.github.com/0-installing/9-gitolite-basics.html#IMPORTANT_overview_of_ssh
|
|
|
|
I assume the gitolite server is called "server" and the user hosting all the
|
|
gitolite repos is "git". I will also be using "sitaram" as the *gitolite
|
|
username* of the admin.
|
|
|
|
Unless specifically mentioned, all these commands are run on the user's or
|
|
admin's workstation, not on the server.
|
|
|
|
#### passphrases versus passwords
|
|
|
|
When you create an ssh keypair, you have the option of protecting it with a
|
|
passphrase. When you subsequently use that keypair to access a remote host,
|
|
your *local* ssh client needs to unlock the corresponding private key, and ssh
|
|
will probably ask for the passphrase you set when you created the keypair.
|
|
|
|
Do not confuse or mistake this prompt (`Enter passphrase for key
|
|
'/home/sitaram/.ssh/id_rsa':`) for a password prompt from the remote server!
|
|
|
|
You have two choices to avoid this prompt every time you try to access the
|
|
remote. The first is to create keypairs *without* a passphrase (just hit
|
|
enter when prompted for one). **Be sure to add a passphrase later, once
|
|
everything is working, using `ssh-keygen -p`**.
|
|
|
|
The second is to use `ssh-agent` (or `keychain`, which in turn uses
|
|
`ssh-agent`) or something like that to manage your keys. Other than the next
|
|
section, further discussion of this is out of scope of this document.
|
|
|
|
#### ssh-agent problems
|
|
|
|
1. Run `ssh-add -l`. If this responds with either "The agent has no
|
|
identities." or "Could not open a connection to your authentication
|
|
agent.", skip this section.
|
|
|
|
2. However, if it lists some keys, like this:
|
|
|
|
2048 fc:c1:48:1e:06:31:97:a4:8b:fc:37:b2:76:14:c7:53 /home/sitaram/.ssh/id_rsa (RSA)
|
|
2048 d2:e0:7f:fa:1a:89:22:41:bb:06:d9:ff:a7:27:36:5c /home/sitaram/.ssh/sitaram (RSA)
|
|
|
|
then run `ls ~/.ssh` and make sure that all the keypairs you have there
|
|
are represented in the `ssh-add -l` output.
|
|
|
|
3. If you find any keypairs in `~/.ssh` that are not represented in the
|
|
`ssh-add -l` output, add them. For instance, if `ssh-add -l` showed me
|
|
only the `id_rsa` key, but I also had a `sitaram` (and `sitaram.pub`)
|
|
keypair, I'd run `ssh-add ~/.ssh/sitaram` to add it.
|
|
|
|
This is because ssh-agent has a quirk: if `ssh-add -l` shows *any* keys at
|
|
all, ssh will only use those keys. Even if you explicitly specify an unlisted
|
|
key using `ssh -i` or an `identityfile` directive in the config file, it won't
|
|
use it.
|
|
|
|
#### basic ssh troubleshooting for the main admin
|
|
|
|
You're the "main admin" if you're trying to access gitolite from the same
|
|
workstation and user account where you ran the "easy install" command. You
|
|
should have two keypairs in your `~/.ssh` directory. The pair called `id_rsa`
|
|
(and `id_rsa.pub`) was probably the first one you created, and you used this
|
|
to get passwordless (pubkey based) access to the server (which was a
|
|
pre-requisite for running the easy install command).
|
|
|
|
The second keypair has the same name as the last argument in the easy install
|
|
command you ran (in my case, `sitaram` and `sitaram.pub`). It was probably
|
|
created by the easy install script, and is the key used for gitolite access.
|
|
|
|
In addition, you should have a "gitolite" paragraph in your `~/.ssh/config`,
|
|
looking something like this:
|
|
|
|
host gitolite
|
|
user git
|
|
hostname server
|
|
identityfile ~/.ssh/sitaram
|
|
|
|
If any of these are not true, you did something funky in your install; email
|
|
me or hop onto #git and hope for the best ;-)
|
|
|
|
Otherwise, run these checks:
|
|
|
|
1. `ssh git@server` should get you a command line.
|
|
|
|
If it asks you for a password, then your `id_rsa` keypair changed after
|
|
you ran the easy install, or someone fiddled with the
|
|
`~/.ssh/authorized_keys` file on the server.
|
|
|
|
If it prints [gitolite version and access info][myrights], you managed to
|
|
overwrite the `id_rsa` keypair with the `sitaram` keypair, or something
|
|
equally weird.
|
|
|
|
2. `ssh gitolite info` should print some [gitolite version and access
|
|
info][myrights]. If you get the output of the GNU info command instead,
|
|
you probably reused your `id_rsa` keypair as your `sitaram` keypair, or
|
|
overwrote the `sitaram` keypair with the `id_rsa` keypair.
|
|
|
|
There are many ways to fix this, depending on where and what the damage is.
|
|
The most generic way (and therefore time-taking) is to re-install gitolite
|
|
from scratch:
|
|
|
|
* make a backup of your gitolite-admin repo clone somewhere (basically your
|
|
"keydir/*.pub" and your "conf/gitolite.conf"). If necessary get these
|
|
files from the server's `~/.gitolite` directory.
|
|
* log on to the server somehow (using some other account, using a password,
|
|
su-ing in, etc) and delete `~/.ssh/authorized_keys`. Rename or move aside
|
|
`~/.gitolite` so that also looks like it is missing.
|
|
* back on your workstation, make sure you have 2 keypairs (`id_rsa` and
|
|
`sitaram`, along with corresponding `.pub` files). Create them if needed.
|
|
Also make sure they are *different* and not a copy of each other :-)
|
|
* install gitolite normally:
|
|
* run `ssh-copy-id -i ~/.ssh/id_rsa git@server` to get passwordless
|
|
access to the server. (Mac users may have to do this step manually)
|
|
* make sure `ssh git@server pwd` prints the `$HOME` of `git@server`
|
|
**without** asking for a password. Do not proceed till this works.
|
|
* run easy install again, (in my case: `cd gitolite-source;
|
|
src/gl-easy-install -q git server sitaram`)
|
|
* go to your gitolite-admin repo clone, and copy `conf/gitolite.conf` and
|
|
`keydir/*.pub` from your backup to this directory
|
|
* copy (be sure to overwrite!) `~/.ssh/sitaram.pub` also to keydir
|
|
* now `git add keydir; git commit; git push -f`
|
|
|
|
That's a long sequence but it should work.
|
|
|
|
#### basic ssh troubleshooting for a normal user
|
|
|
|
For a normal user, life is much simpler. They should have only one pubkey,
|
|
which was previously sent to the gitolite admin to add into the admin repo's
|
|
`keydir` as "user.pub", and then "user" given permissions to some repo.
|
|
|
|
`ssh git@server info` should get you [gitolite version and access
|
|
info][myrights]. If it asks you for a password, your pubkey was not sent to
|
|
the server properly. Check with your admin.
|
|
|
|
[myrights]: http://github.com/sitaramc/gitolite/blob/pu/doc/3-faq-tips-etc.mkd#myrights
|
|
|
|
If it gets you the GNU info command output, you have shell access. This means
|
|
you had command line access to the server *before* you were added as a
|
|
gitolite user. If you send that same key to your gitolite admin to include in
|
|
the admin repo, it won't work. For reasons why, see below.
|
|
|
|
<a name="details"></a>
|
|
|
|
### details
|
|
|
|
Here's how it all hangs together.
|
|
|
|
#### files on the server
|
|
|
|
* the authkeys file; this contains one line containing the pubkey of each
|
|
user who is permitted to login without a password.
|
|
|
|
Pubkey lines that give shell access look like this:
|
|
|
|
ssh-rsa AAAAB3NzaC[snip]uPjrUiAUew== /home/sitaram/.ssh/id_rsa
|
|
|
|
On a typical server there will be only one or two of these lines.
|
|
|
|
Note that the last bit (`/home/sitaram/.ssh/id_rsa`) is purely a *comment*
|
|
field and can be anything. Also, the actual lines are much longer, about
|
|
400 characters; I snipped 'em in the middle, as you can see.
|
|
|
|
In contrast, pubkey lines that give access to git repos hosted by gitolite
|
|
look like this:
|
|
|
|
command="[some path]src/gl-auth-command sitaram",[some restrictions] ssh-rsa AAAAB3NzaC[snip]s18OnB42oQ== sitaram@sita-lt
|
|
|
|
You will have many more of these lines -- one for every pubkey file in
|
|
`keydir/` of your gitolite-admin repo, with the corresponding username in
|
|
place of "sitaram" in the example above.
|
|
|
|
The "command=" at the beginning ensures that when someone with the
|
|
corresponding private key logs in, they don't get a shell. Instead, the
|
|
`gl-auth-command` program is run, and (in this example) is given the
|
|
argument `sitaram`. This is how gitolite is invoked, (and is told the
|
|
user logging in is "sitaram").
|
|
|
|
#### files on client
|
|
|
|
* default keypair; used to get shell access to servers. You would have
|
|
copied this pubkey to the gitolite server in order to log in without a
|
|
password. (On Linux systems you may have used `ssh-copy-id` to do that).
|
|
You would have done this *before* you ran the easy install script, because
|
|
otherwise easy install won't run!
|
|
|
|
~/.ssh/id_rsa
|
|
~/.ssh/id_rsa.pub
|
|
|
|
* gitolite keypair; the "sitaram" in this is the 3rd argument to the
|
|
`src/gl-easy-install` command you ran; the easy install script does the
|
|
rest
|
|
|
|
~/.ssh/sitaram
|
|
~/.ssh/sitaram.pub
|
|
|
|
* config file; this file has an entry for gitolite access:
|
|
|
|
~/.ssh/config
|
|
|
|
To understand why we need that, let's step back a bit. Normally, you
|
|
might expect to access gitolite repos like this:
|
|
|
|
ssh://git@server/reponame.git
|
|
|
|
But this won't work, because this ends up using the *default* keypair
|
|
(normally), which gives you a command line. Which means it won't invoke
|
|
the `gl-auth-command` program at all, and so none of gitolite's access
|
|
control will work.
|
|
|
|
<a name="altkey"></a>
|
|
|
|
You need to force ssh to use the *other* keypair when performing a git
|
|
operation. With normal ssh, that would be
|
|
|
|
ssh -i ~/.ssh/sitaram git@server
|
|
|
|
but git does not support putting an alternate keypair in the URL.
|
|
|
|
Luckily, ssh has a very convenient way of capturing all the connection
|
|
information (username, hostname, port number (if it's not the default 22),
|
|
and keypair to be used) in one "paragraph" of `~/.ssh/config`. This is
|
|
what the para looks like for us (the easy install script puts it there the
|
|
first time):
|
|
|
|
host gitolite
|
|
user git
|
|
hostname server
|
|
identityfile ~/.ssh/sitaram
|
|
|
|
(The "gitolite" can be anything you want of course; it's like a group name
|
|
for all the stuff below it). This ensures that typing
|
|
|
|
ssh gitolite
|
|
|
|
is equivalent to
|
|
|
|
ssh -i ~/.ssh/sitaram git@server
|
|
|
|
and therefore this:
|
|
|
|
git clone gitolite:reponame.git
|
|
|
|
now works as expected, invoking the special keypair instead of the default
|
|
one.
|
|
|
|
<a name="twokeys"></a>
|
|
|
|
#### why two keys on client
|
|
|
|
Why do I (the admin) need two **different** keypairs?
|
|
|
|
There are two types of access the admin will make to the server: a normal
|
|
login, to get a shell prompt, and gitolite access (clone/fetch/push etc). The
|
|
first access needs an authkeys line *without* any "command=" restrictions,
|
|
while the second requires a line *with* such a restriction.
|
|
|
|
And we can't use the same key for both because there is no way to disambiguate
|
|
them; the ssh server will always (*always*) pick the first one in sequence
|
|
when the key is offered by the ssh client.
|
|
|
|
So the next question is usually "I have other ways to get a shell on that
|
|
account, so why do I need a key for shell access at all?"
|
|
|
|
The answer to this is that the "easy install" script, being written for the
|
|
most general case, needs shell access via ssh to do its stuff.
|
|
|
|
If you really, really, want to get rid of the extra key, here's a transcript
|
|
that should have enough info to get you going (but it helps to know ssh well):
|
|
|
|
* on "sitaram" user, on my workstation
|
|
|
|
cd ~/.ssh
|
|
cp id_rsa sitaram
|
|
cp id_rsa.pub sitaram.pub
|
|
cd ~/gitolite-clone
|
|
src/gl-easy-install -q git my.git.server sitaram
|
|
|
|
that last command produces something like the following:
|
|
|
|
you are upgrading from (unknown) to v0.80-6-gdde8c4e
|
|
setting up keypair...
|
|
...reusing /home/sitaram/.ssh/sitaram.pub...
|
|
creating gitolite para in ~/.ssh/config...
|
|
finding/creating gitolite rc...
|
|
installing/upgrading...
|
|
Pseudo-terminal will not be allocated because stdin is not a terminal.
|
|
[master (root-commit) e717a89] start
|
|
2 files changed, 11 insertions(+), 0 deletions(-)
|
|
create mode 100644 conf/gitolite.conf
|
|
create mode 100644 keydir/sitaram.pub
|
|
cloning gitolite-admin repo...
|
|
Initialized empty Git repository in /home/sitaram/gitolite-admin/.git/
|
|
fatal: 'gitolite-admin.git' does not appear to be a git repository
|
|
fatal: The remote end hung up unexpectedly
|
|
|
|
notice that the final step (the clone of the newly created gitolite-admin
|
|
repo) failed, as expected
|
|
|
|
* now log on to the git hosting account (`git@my.git.server` in this
|
|
example), edit `~/.ssh/authorized_keys`, and delete the line with the
|
|
first occurrence of your key (this should be *before* the `# gitolite
|
|
start` line)
|
|
|
|
* now go back to your workstation and
|
|
|
|
git clone git@my.git.server:gitolite-admin
|
|
|
|
That should do it.
|
|
|
|
<a name="complex"></a>
|
|
|
|
### more complex ssh setups
|
|
|
|
What do you need to know in order to create more complex ssh setups (for
|
|
instance if you have *two* gitolite servers you are administering)?
|
|
|
|
#### two gitolite servers to manage?
|
|
|
|
* they can have the same key; no harm there (example, sitaram.pub)
|
|
|
|
* instead of just one ssh/config para, you now have two (assuming that the
|
|
remote user on both machines is called "git"):
|
|
|
|
host gitolite
|
|
user git
|
|
hostname server
|
|
identityfile ~/.ssh/sitaram
|
|
|
|
host gitolite2
|
|
user git
|
|
hostname server2
|
|
identityfile ~/.ssh/sitaram
|
|
|
|
* now access one server's repos as `gitolite:reponame.git` and the other
|
|
server's repos as `gitolite2:reponame.git`.
|
|
|
|
<a name="shell"></a>
|
|
|
|
### giving shell access to gitolite users
|
|
|
|
We've managed (thanks to an idea from Jesse Keating) to make it possible for a
|
|
single key to allow both gitolite access *and* shell access.
|
|
|
|
This is done by copying the pubkey (to which you want to give shell access) to
|
|
the server and running either
|
|
|
|
cd $HOME/.gitolite # assuming default $GL_ADMINDIR in ~/.gitolite.rc
|
|
src/gl-tool shell-add ~/foo.pub
|
|
|
|
or
|
|
|
|
gl-tool shell-add ~/foo.pub
|
|
|
|
The first method is to be used if you used the **user-install** mode, while
|
|
the second method is for the **system-install followed by user-setup** mode
|
|
(see doc/0-INSTALL.mkd, section on "install methods", for more on this)
|
|
|
|
**IMPORTANT UPGRADE NOTE**: previous implementations of this feature were
|
|
crap. There was no easy/elegant way to ensure that someone who had repo admin
|
|
access would not manage to get himself shell access.
|
|
|
|
Giving someone shell access requires that you should have shell access in the
|
|
first place, so the simplest way is to enable it from the server side only.
|