gitolite/doc/extras/sts.mkd

473 lines
20 KiB
Markdown
Raw Normal View History

2012-05-18 09:18:43 +02:00
# #sts ssh troubleshooting and tips
2012-03-16 02:54:47 +01:00
**This document must be read in full the first time. If you start from some
nice looking section in the middle it may not help you unless you're already
an expert at ssh**.
This document should help you troubleshoot ssh-related problems in installing
2012-05-18 09:18:43 +02:00
and accessing gitolite. It also has a section of random ssh-related tips and
tricks that gitolite can do.
2012-03-16 02:54:47 +01:00
2012-05-18 09:18:43 +02:00
----
2012-03-16 02:54:47 +01:00
2012-05-18 09:18:43 +02:00
[[TOC]]
----
## IMPORTANT -- READ THIS FIRST
### caveats
2012-03-16 02:54:47 +01:00
* Before reading this document, it is **mandatory** to read and **completely
understand** [this][ssh], which is a very detailed look at how gitolite
uses ssh's features on the server side. Don't assume you know all that;
if you did, you wouldn't be needing *this* document either!
2012-03-16 02:54:47 +01:00
* This document, and others linked from this, together comprise all the help
I can give you in terms of the ssh aspect of using gitolite. If you're
installing gitolite, you're a "system admin", like it or not. Ssh is
therefore a necessary skill. Please take the time to learn at least
enough to get passwordless access working.
* Please note that authentication is not really gitolite's job at all. I'd
rather spend time on actual gitolite features, code, and documentation
than authentication (i.e., ssh, in the common case).
Surprised? [This][auth] might help explain better.
2012-05-18 09:18:43 +02:00
### naming conventions used
2012-03-16 02:54:47 +01:00
* Your workstation is the **client**. Your userid on the client does not
matter, and it has no relation to your gitolite username.
2012-04-17 03:13:13 +02:00
* The server is called **server** and the "hosting user" is **git**. If
2012-03-16 02:54:47 +01:00
this is an RPM/DEB install, the hosting user is probably called
"gitolite", however we will use "git" in this document.
2012-05-18 09:18:43 +02:00
### taking stock -- relevant files and directories
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* The client has a `~/.ssh` containing a few keypairs. It may also have a
2012-03-16 02:54:47 +01:00
`config` file.
2012-04-17 03:13:13 +02:00
* The client also has a clone of the "gitolite-admin" repo, which contains a
2012-03-16 02:54:47 +01:00
bunch of `*.pub` files in `keydir`. We assume this clone is in `$HOME`;
if it is not, adjust instructions accordingly when needed.
* The git user on the server has a `~/.ssh/authorized_keys` file that the
ssh daemon uses to authenticate incoming users. We often call this file
**authkeys** to save typing, and it always means the one on the server
(we're not interested in this file on the client side).
2012-04-17 03:13:13 +02:00
* The server also has a `~/.gitolite/keydir` which contains a bunch of
2012-03-16 02:54:47 +01:00
`*.pub` files.
2012-05-18 09:18:43 +02:00
### normal gitolite key handling
2012-03-16 02:54:47 +01:00
Here's how normal gitolite key handling works:
2012-04-17 03:13:13 +02:00
* (On client) pub key changes like adding new ones, deleting old ones, etc.,
2012-03-16 02:54:47 +01:00
are done in the `keydir` directory in the gitolite-admin repo clone. Then
the admin `git add`s and `git commit`s those changes, then `git push`es
them to the server.
2012-04-17 03:13:13 +02:00
* (On server) a successful push from the client makes git invoke the
2012-03-16 02:54:47 +01:00
post-update hook in the gitolite-admin repo. This hook is installed by
gitolite, and it does a bunch of things which are quite transparent to
the admin, but we'll describe briefly here:
2012-04-17 03:13:13 +02:00
* The pubkey files from this push are checked-out into
2012-03-16 02:54:47 +01:00
`~/.gitolite/keydir` (and similarly the config files into
2012-04-17 03:13:13 +02:00
`~/.gitolite/conf`).
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* The "compile" script then runs, which uses these files to populate
`~/.ssh/authorized_keys` on the server.
2012-03-16 02:54:47 +01:00
The authkeys file may have other, (non-gitolite) keys also. Those
lines are preserved. Gitolite only touches lines that are found
between gitolite's "marker" lines (`# gitolite start` and `# gitolite
end`).
2012-05-18 09:18:43 +02:00
## common ssh problems
2012-03-16 02:54:47 +01:00
Since I'm pretty sure at least some of you didn't bother to read the
"IMPORTANT: PLEASE READ FIRST" section above, let me take a minute to point
you there again. Especially the first bullet.
Done? OK, read on...
The following problem(s) indicate that pubkey access is not working at all, so
you should start with [appendix 1][stsapp1]. If that doesn't fix the problem, continue
2012-03-16 02:54:47 +01:00
with the other appendices in sequence.
2012-04-17 03:13:13 +02:00
* Running any git clone/fetch/ls-remote or just `ssh git@server info` asks
2012-03-16 02:54:47 +01:00
you for a password.
The following problem(s) indicate that your pubkey is bypassing gitolite and
going straight to a shell. You should start with [appendix 2][sshkeys-lint]
and continue with the rest in sequence. [Appendix 5][ybpfail] has some
background info.
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* Running `ssh git@server info` gets you the output of the GNU 'info'
2012-03-16 02:54:47 +01:00
command instead of gitolite's version and access info.
2012-04-17 03:13:13 +02:00
* Running `git clone git@server:repositories/reponame` (note presence of
2012-03-16 02:54:47 +01:00
`repositories/` in URL) works.
[A proper gitolite key will only let you `git clone git@server:reponame`
(note absence of `repositories/`)]
2012-04-17 03:13:13 +02:00
* You are able to clone repositories but are unable to push changes back
(the error complains about the `GL_BINDIR` environment variable not being
set, and the `hooks/update` failing in some way).
2012-03-16 02:54:47 +01:00
[If you run `git remote -v` you will find that your clone URL included the
`repositories/` described above!]
2012-04-17 03:13:13 +02:00
* Conversely, using the correct syntax, `git clone git@server:reponame`
2012-03-16 02:54:47 +01:00
(note absence of `repositories/` in the URL), gets you `fatal: 'reponame'
does not appear to be a git repository`, and yet you are sure 'reponame'
exists, you haven't mis-spelled it, etc.
2012-05-18 09:18:43 +02:00
## step by step
2012-03-16 02:54:47 +01:00
Since I'm pretty sure at least some of you didn't bother to read the
"IMPORTANT: PLEASE READ FIRST" section above, let me take a minute to point
you there again. Especially the first bullet.
Done? OK, now the general outline for ssh troubleshooting is this:
2012-04-17 03:13:13 +02:00
* Make sure the server's overall setup even *allows* pubkey based login.
2012-03-16 02:54:47 +01:00
I.e., check that git fetch/clone/ls-remote commands or a plain `ssh
git@server info` do NOT ask for a password. If you do get asked for a
password, see [appendix 1][stsapp1].
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* Match client-side pubkeys (`~/.ssh/*.pub`) with the server's authkeys
2012-03-16 02:54:47 +01:00
file. To do this, run `sshkeys-lint`, which tells you in detail what key
has what access. See [appendix 2][sshkeys-lint].
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* At this point, we know that we have the right key, and that if sshd
2012-03-16 02:54:47 +01:00
receives that key, things will work. But we're not done yet. We still
need to make sure that this specific key is being offered/sent by the
client, instead of the default key. See [appendix 3][stsapp3] and
[appendix 4][ssh-ha].
2012-03-16 02:54:47 +01:00
2012-05-18 09:18:43 +02:00
## random tips, tricks, and notes
2012-03-16 02:54:47 +01:00
### #giving-shell giving shell access to gitolite users
2012-03-16 02:54:47 +01:00
Thanks to an idea from Jesse Keating, a single key can allow both gitolite
access *and* shell access.
2012-03-16 02:54:47 +01:00
To do this:
2012-03-16 02:54:47 +01:00
* add the list of users who will have shell access -- one username per line,
no extra whitespace -- to a plain text file of your choice.
2012-03-16 02:54:47 +01:00
* put the name of this file in a new rc variable `SHELL_USERS_LIST`. For
example it could be
2012-03-16 02:54:47 +01:00
SHELL_USERS_LIST => "$ENV{HOME}/.gitolite.shell-users",
2012-03-16 02:54:47 +01:00
* add the line `'Shell::input',` to the `INPUT` list in the rc file. This
must be the first item on the list (possibly preceded by CpuTime, if
you're using that).
2012-03-16 02:54:47 +01:00
* add the line `'post-compile/ssh-authkeys-shell-users',` to the
`POST_COMPILE` list, *after* the `'post-compile/ssh-authkeys',` line.
Then run `gitolite compile; gitolite trigger POST_COMPILE` or push a dummy
change to the admin repo.
2012-03-16 02:54:47 +01:00
### #kfn distinguishing one key from another
Since a user can have [more than one key][multi-key], it is sometimes useful
to distinguish one key from another. Sshd does not tell you even the
fingerprint of the key that finally matched, so normally all you have is the
`GL_USER` env var.
However, if you replace
'post-compile/ssh-authkeys',
in the `POST_COMPILE` trigger list in the rc file with
'post-compile/ssh-authkeys --key-file-name',
then an extra argument is added after the username in the "command" variable
of the authkeys file. That is, instead of this:
command="/home/g3/gitolite/src/gitolite-shell u3",no-port-forwarding,...
you get this:
command="/home/g3/gitolite/src/gitolite-shell u3 keydir/u3.pub",no-port-forwarding,...
You can then write an INPUT trigger to do whatever you need with the file
name, which is in `$ARGV[1]` (the second argument). The actual file is
available at `$ENV{GL_ADMIN_BASE}/$ARGV[1]` if you need its contents.
2012-05-18 09:18:43 +02:00
### simulating ssh-copy-id
2012-03-16 02:54:47 +01:00
don't have `ssh-copy-id`? This is broadly what that command does, if you want
to replicate it manually. The input is your pubkey, typically
`~/.ssh/id_rsa.pub` from your client/workstation.
2012-04-17 03:13:13 +02:00
* It copies it to the server as some file.
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* It appends that file to `~/.ssh/authorized_keys` on the server
(creating it if it doesn't already exist).
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* It then makes sure that all these files/directories have go-w perms
2012-03-16 02:54:47 +01:00
set (assuming user is "git"):
/home/git/.ssh/authorized_keys
/home/git/.ssh
/home/git
[Actually, `sshd` requires that even directories *above* `~` (`/`, `/home`,
typically) also must be `go-w`, but that needs root. And typically
they're already set that way anyway. (Or if they're not, you've got
bigger problems than gitolite install not working!)]
2012-05-18 09:18:43 +02:00
### problems with using non-openssh public keys
2012-03-16 02:54:47 +01:00
Gitolite accepts public keys only in openssh format. Trying to use an "ssh2"
key (used by proprietary SSH software) will not be a happy experience.
src/triggers/post-compile/ssh-authkeys can be made to detect non-openssh
formats and automatically convert them; patches welcome!
2012-03-16 02:54:47 +01:00
The actual conversion command, if you want to just do it manually for now and
be done with it, is:
2012-03-16 02:54:47 +01:00
ssh-keygen -i -f /tmp/ssh2/YourName.pub > /tmp/openssh/YourName.pub
then use the resulting pubkey as you normally would in gitolite.
2012-05-18 09:18:43 +02:00
### windows issues
2012-03-16 02:54:47 +01:00
On windows, I have only used msysgit, and the openssh that comes with it.
Over time, I have grown to distrust putty/plink due to the number of people
who seem to have trouble when those beasts are involved (I myself have never
used them for any kind of git access). If you have unusual ssh problems that
just don't seem to have any explanation, try removing all traces of
putty/plink, including environment variables, etc., and then try again.
Thankfully, someone contributed [this][putty].
2012-03-16 02:54:47 +01:00
2012-05-18 09:18:43 +02:00
## #stsapp1 appendix 1: ssh daemon asks for a password
2012-03-16 02:54:47 +01:00
> **NOTE**: This section should be useful to anyone trying to get
> password-less access working. It is not necessarily specific to gitolite,
> so keep that in mind if the wording feels a little more general than you
> were expecting.
You have generated a keypair on your workstation (`ssh-keygen`) and copied the
public part of it (`~/.ssh/id_rsa.pub`, by default) to the server.
On the server you have appended this file to `~/.ssh/authorized_keys`. Or you
ran something, like the `gitolite setup` step during a gitolite install, which
2012-03-16 02:54:47 +01:00
should have done that for you.
You now expect to log in without having to type in a password, but when you
try, you are being asked for a password.
This is a quick checklist:
2012-04-17 03:13:13 +02:00
* Make sure you're being asked for a password and not a pass*phrase*. Do
2012-03-16 02:54:47 +01:00
not confuse or mistake a prompt saying `Enter passphrase for key
'/home/sitaram/.ssh/id_rsa':` for a password prompt from the remote
server!
When you create an ssh keypair using `ssh-keygen`, you have the option of
protecting it with a passphrase. When you subsequently use that keypair
to access a remote host, your *local* ssh client needs to unlock the
corresponding private key, and ssh will probably ask for the passphrase
you set when you created the keypair.
You have two choices to avoid this prompt every time you try to use the
private key. The first is to create keypairs *without* a passphrase (just
hit enter when prompted for one). **Be sure to add a passphrase later,
once everything is working, using `ssh-keygen -p`**.
The second is to use `ssh-agent` (or `keychain`, which in turn uses
`ssh-agent`) or something like that to manage your keys. Other than
discussing one more potential trouble-spot with ssh-agent (see below),
further discussion of ssh-agent/keychain is out of scope of this document.
2012-04-17 03:13:13 +02:00
* Ssh is very sensitive to permissions. An extremely conservative setup is
2012-03-16 02:54:47 +01:00
given below, but be sure to do this on **both the client and the server**:
cd $HOME
chmod go-rwx .
chmod -R go-rwx .ssh
2012-04-17 03:13:13 +02:00
* Actually, every component of the path to `~/.ssh/authorized_keys` all the
2012-03-16 02:54:47 +01:00
way upto the root directory must be at least `chmod go-w`. So be sure to
check `/` and `/home` also.
2012-04-17 03:13:13 +02:00
* While you're doing this, make sure the owner and group info for each of
2012-03-16 02:54:47 +01:00
these components are correct. `ls -ald ~ ~/.ssh ~/.ssh/authorized_keys`
will tell you what they are.
2012-04-17 03:13:13 +02:00
* You may also want to check `/etc/ssh/sshd_config` to see if the "git" user
2012-03-16 02:54:47 +01:00
is allowed to login at all. For example, if that file contains an
`AllowUsers` config entry, then only users mentioned in that line are
allowed to log in!
2012-04-17 03:13:13 +02:00
* While you're in there, check that file does NOT have a setting for
`AuthorizedKeysFile`. See `man sshd_config` for details. This setting is
a show stopper for gitolite to use ssh.
2012-04-17 03:13:13 +02:00
* Some OSs/distributions require that the "git" user should have a password
2012-03-16 02:54:47 +01:00
and/or not be a locked account. You may want to check that as well.
* If your server is running SELinux, and you install gitolite to
`/var/gitolite` or another location unsupported by default SELinux
policies, then SELinux will prevent sshd from reading
`.ssh/authorized_keys`. Consider installing gitolite to
`/var/lib/gitolite`, which is a supported location by default SELinux
policies.
2012-04-17 03:13:13 +02:00
* If all that fails, log onto the server as root, `cd /var/log`, and look
2012-03-16 02:54:47 +01:00
for a file called `auth.log` or `secure` or some such name. Look inside
this file for messages matching the approximate time of your last attempt
to login, to see if they tell you what is the problem.
2012-05-18 09:18:43 +02:00
## #sshkeys-lint appendix 2: which key is which -- running sshkeys-lint
2012-03-16 02:54:47 +01:00
The sshkeys-lint program can be run on the server or the client. Run it with
'-h' to get a help message.
On the server you can run `gitolite sshkeys-lint` and it will tell you, for
each key in the admin directory's keydir, what access is available. This is
especially good at finding duplicate keys and such.
To run it on the client you have to copy the file src/commands/sshkeys-lint
from some gitolite clone, then follow these steps:
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* Get a copy of `~/.ssh/authorized_keys` from the server and put it in
`/tmp/foo` or something.
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* cd to `~/.ssh`.
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* Run `/path/to/sshkeys-lint *.pub < /tmp/foo`.
2012-03-16 02:54:47 +01:00
Note that it is not trying to log in or anything -- it's just comparing
fingerprints as computed by `ssh-keygen -l`.
2012-03-16 02:54:47 +01:00
If the pubkey file you're interested in appears to have the correct access to
the server, you're done with this step.
Otherwise you have to rename some keypairs and try again to get the effect you
need. Be careful:
2012-04-17 03:13:13 +02:00
* Do not just rename the ".pub" file; you will have to rename the
2012-03-16 02:54:47 +01:00
corresponding private key also (the one with the same basename but without
2012-04-17 03:13:13 +02:00
an extension).
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* If you're running ssh-agent, you may have to delete (using `ssh-add -D`)
and re-add identities for it to pick up the renamed ones correctly.
2012-03-16 02:54:47 +01:00
2012-05-18 09:18:43 +02:00
### typical cause(s)
2012-03-16 02:54:47 +01:00
The admin often has passwordless shell access to `git@server` already, and
then used that same key to get access to gitolite (i.e., copied that same
pubkey as YourName.pub and ran `gitolite setup` on it).
2012-03-16 02:54:47 +01:00
As a result, the same key appears twice in the authkeys file now, and since
the ssh server will always use the first match, the second occurrence (which
invokes gitolite) is ignored.
To fix this, you have to use a different keypair for gitolite access. The
best way to do this is to create a new keypair, copy the pubkey to the server
as YourName.pub, then run `gitolite setup -pk YourName.pub` on the server.
Remember to adjust your agent identities using ssh-add -D and ssh-add if
you're using ssh-agent, otherwise these new keys may not work.
2012-03-16 02:54:47 +01:00
2012-05-18 09:18:43 +02:00
## #stsapp3 appendix 3: ssh client may not be offering the right key
2012-03-16 02:54:47 +01:00
2012-04-17 03:13:13 +02:00
* Make sure the right private key is being offered. Run ssh in very
2012-03-16 02:54:47 +01:00
verbose mode and look for the word "Offering", like so:
ssh -vvv user@host pwd 2> >(grep -i offer)
If some keys *are* being offered, but not the key that was supposed to be
used, you may be using ssh-agent (next bullet). You may also need to
create some host aliases in `~/.ssh/config` ([appendix 4][ssh-ha]).
2012-03-16 02:54:47 +01:00
* (ssh-agent issues) If `ssh-add -l` responds with either "The agent has no
identities." or "Could not open a connection to your authentication
agent.", then you can skip this bullet.
However, if `ssh-add -l` lists *any* keys at all, then something weird
happens. Due to a quirk in ssh-agent, ssh will now *only* use one of
those keys, *even if you explicitly ask* for some other key to be used.
In that case, add the key you want using `ssh-add ~/.ssh/YourName` and try
the access again.
2012-05-18 09:18:43 +02:00
## #ssh-ha appendix 4: ssh host aliases
2012-03-16 02:54:47 +01:00
(or "making git use the right options for ssh")
The ssh command has several options for non-default items to be specified.
Two common examples are `-p` for the port number if it is not 22, and `-i` for
the public key file if you do not want to use just `~/.ssh/id_rsa` or such.
Git has two ssh-based URL syntaxes, but neither allows specifying a
non-default public key file. And a port number is only allowed in one of
them. (See `man git-clone` for details). Finally, hosts often have to be
referred with IP addresses (such is life), or the name is very long, or hard
to remember.
Using a "host" para in `~/.ssh/config` lets you nicely encapsulate all this
within ssh and give it a short, easy-to-remember, name. Example:
host gitolite
user git
hostname a.long.server.name.or.annoying.IP.address
port 22
identityfile ~/.ssh/id_rsa
Now you can simply use the one word `gitolite` (which is the host alias we
defined here) and ssh will infer all those details defined under it -- just
say `ssh gitolite` and `git clone gitolite:reponame` and things will work.
(By the way, the 'port' and 'identityfile' lines are needed only if you have
non-default values, although I put them in anyway just to be complete).
If you have *more than one* pubkey with access to the *same* server, you
**must** use this method to make git pick up the right key. There is no other
way to do this, as far as I know.
[tut]: http://sites.google.com/site/senawario/home/gitolite-tutorial
2012-05-18 09:18:43 +02:00
## #ybpfail appendix 5: why bypassing gitolite causes a problem
2012-03-16 02:54:47 +01:00
When you bypass gitolite, you end up running your normal shell instead of the
special gitolite entry point script `gitolite-shell`.
2012-03-16 02:54:47 +01:00
This means commands (like 'info') are interpreted by the shell instead of
gitolite.
It also means git operations look for repos in `$HOME`.
However, gitolite places all your repos in a subdirectory pointed to by
`$REPO_BASE` in the rc file (default: `repositories`), and internally prefixes
this before calling the actual git command you invoked. Thus, the pathname of
the repo that you use on the client is almost never the correct pathname on
the server. (This is by design. Don't argue...)
This means that, you get 2 kinds of errors if you bypass gitolite
2012-04-17 03:13:13 +02:00
* When you use `git@server:reponame` with a key that bypasses gitolite
2012-03-16 02:54:47 +01:00
(i.e., gets you a shell), this prefixing does not happen, and so the repo
is not found. Neither a clone/fetch nor a push will work.
2012-04-17 03:13:13 +02:00
* Conversely, consider `git@server:repositories/reponame.git`. The clone
2012-03-16 02:54:47 +01:00
operation will work -- you're using the full Unix path, (assuming default
`$REPO_BASE` setting), and so the shell finds the repo where you said it
would be. However, when you push, gitolite's **update hook** kicks in,
and fails to run because some of the environment variables it is expecting
are not present.