deac/gitolite

Fork 0

Sitaram Chamarty c41fcc2653 added doc and helper for complex mirroring setups, to contrib

2011-09-02 22:41:27 +05:30

12 KiB

Raw Blame History

semi-autonomous mirroring setup example

This document describes one way to do this. Gitolite is powerful so you can probably find other ways to suit you.

In this document:

overview of problem
overview of setup
- gitolite feature recap
pre-requisites
quick setup
step by step
next steps
appendix A: delegation helper files

overview of problem

The example is from real life, with the following characteristics:

multiple servers (hosts)
multiple "products", each product has one or more git repos
different products are "native to" (mastered on) different hosts
a product may be mirrored to zero or more other hosts (mirrored to zero hosts means it is local to the host)

The admin requirements are:

the overall system will have one or more master admins; they manage the main config file, for instance.
each host will have one or more host admins
these host admins should be allowed to create any repos they need (within one of a set of directory names allocated to them), and assign access to whomever they please
they should not be able to "step on each other" -- setup access rules for repos not in their control

The following can only be done by the master admins:

authentication (ssh keys) are centrally managed and distributed. Host admins should not be allowed to do this.
mirroring setup -- who's the master and who're the slaves for any repo
allowing redirected pushes from slaves

overview of setup

We will use p1 as the product, with sam as the master and frodo as a slave. Assume equivalent text/code for other product/master/slave combos.

This setup imposes the condition that all repos should be under some directory name; either a product name or, for local repos, a hostname. In our example, these directory names would be p1 or sam on the host sam, and frodo on the host frodo.

gitolite feature recap

We use delegation, to ensure that admins for sam can only write files whose names start with master/sam/. The actual files they will write are master/sam/p1.conf etc., one for each product that is mastered on their server.

We use subconf. When you say subconf "path/to/foo.conf, then within that file (and anything included from it), access can only be defined for repos that regex-match one of the elements of @foo.

pre-requisites

First, install mirroring on all servers according to the main mirroring document. Set it up so that the gitolite-admin repo is mastered at one server and everyone else slaves it.

Also, after (or during) the normal mirroring install, edit ~/.gitolite.rc on all servers and set $GL_WILDREPOS to 1 (from its default of 0).

quick setup

edit your gitolite.conf file as given in step 1 below
- ignore all the comments, even the ones that tell you to do something :-)
- change only the names of the admin users, and the names of the servers in the config lines. Everything else stays the same
now copy mirror-conf-helper from the contrib directory to your home or some easy to type place, and run it to setup your hosts, products, and slaves

A typical sequence with that script is:

# cd to your gitolite-admin clone

# create a new host
~/mirror-conf-helper newhost sam sam-admin

# create the actual repository and access rules.  This is done by the host
# admin for the host on which it is mastered (or you can do it yourself).
# See step 3 below for details.
mkdir -p conf/master/sam
vim      conf/master/sam/p1.conf
    # now add in some "repo p1/..." and "RW ..." lines for the repos in
    # product p1 and save

# add product p1 to the list of repos sam is allowed to control
~/mirror-conf-helper newprod sam p1

# add a slave
~/mirror-conf-helper newslave sam p1 frodo

You can then treat the detailed steps described below as extra information or "background reading" ;-)

step by step

If the script is not cutting it for you and want to vary the technique for some reason, or you simply want to gain a better understanding of what is happening, it may be better to do each step manually instead of just using the script.

Note: all files mentioned below are assumed to be under conf/. The only place where you have to explicitly state this is in the delegation code in the appendix. The rest of the time, "conf/" is assumed.

(1) `gitolite.conf`

The main config file has these items in it. Please add them in this order.

If you follow this document completely, your gitolite.conf file can be pretty static, changing only if the master admin changes or you need to add a new host as slave to the gitolite-admin repo. Therefore you can set it up first.

Here's what it looks like:

# (1.1)---------------------------------------------------------------------
# First the main setup:

@master-admins    =   sitaram dilbert
repo gitolite-admin
    RW+ =   @master-admins
    config gitolite.mirror.master   =   "master"
    config gitolite.mirror.slaves   =   "list of slave servers"
    # you cannot use continuation lines for this; sorry!  You have to list
    # them all in ONE long line within one set of double quotes...

# (1.2)---------------------------------------------------------------------
# If you have any files with "convenience" group definitions, pull them in:

include "groups/users.conf"
include "groups/repos.conf"

# (1.3)---------------------------------------------------------------------
# Next is delegation.  If you don't want delegation, omit this section,
# and replace all "subconf" commands with "include" in the rest of this
# document.

include "host-product-map.conf"
    # create this file; see step A1 in appendix A

repo gitolite-admin
    # now that you're adding a NAME/ section, you need this for master
    # admins to retain their access
    RW+ NAME/                   =   @master-admins

include "NAME-restrictions.conf"
    # create this file; see step A2 in appendix A

# (1.4)---------------------------------------------------------------------
# Now you include the access rules for native repos
# (example: master/sam.conf)

subconf "master/HOSTNAME.conf"

# (1.5)---------------------------------------------------------------------
# After this you have the mirror config for native repos.  We place this
# *after* the access rules above, to make sure we override any mirror
# config lines accidentally added by a host admin!
# (example: mirrors/sam/p1.conf)

include "mirrors/HOSTNAME/*.conf"

# (1.6)---------------------------------------------------------------------
# Now we pull in all setup (mirror config *and* access rules) for
# non-native repos.  For the product "p1", this file will get pulled in
# when the config is processed on frodo.
# (example: slave/frodo/sam.conf)

subconf "slave/HOSTNAME/*.conf"

You'll get some warnings about missing include files; ignore them.

(2) `master/sam.conf`

For each host sam, one file called master/sam.conf is needed. This file contains just one line:

include "master/sam/*.conf"

It is pulled in by the main config file using subconf "master/HOSTNAME.conf", which -- on host sam -- translates to subconf "master/sam.conf". The only purpose of this is to setup the subconf restriction on the combined contents of master/sam/*.conf.

(3) host admins only -- `master/sam/p1.conf`

(recap: the host admins for sam can only write files in master/sam).

For each product p1 with master on host sam, the admins for host sam will create a file master/sam/p1.conf. This file will contain reponames (must start with p1/) and access rules for these repos.

If they have some common groupnames etc., they can probably put them in master/sam/users.conf or some such file and pull those in into each of their product.conf files.

By default, everything is local to their server. (Mirroring can only be setup by the master admins).

(4) `mirrors/sam/p1.conf`

For each product p1 mastered on host sam, a file called mirrors/sam/p1.conf will be created, containing mirror config lines for all repos of product p1. In this case, it could be

repo    p1/..*
        config gitolite.mirror.master   =   "sam"
        config gitolite.mirror.slaves   =   "frodo"

If this file does not exist, p1 is local to sam and not mirrored.

(5) `slave/frodo/sam.conf`

For each product that slave frodo gets from master sam, this file has the following lines

# pull in the access lines
include "master/sam/p1.conf"

# pull in the mirror config lines
include "mirrors/sam/p1.conf"

This file is pulled in on a slave server via a subconf slave/HOSTNAME/*.conf line in the main config file. On frodo, this would pull in slave/frodo/sam.conf (among others), establishing, again, a subconf restriction on @sam.

Security note: what this achieves is that the access lines, which were written by sam's admins, are parsed on frodo under the subconf restriction of "sam". This is important to prevent sam's admins from writing rules for repos they don't own and having them processed on other servers!

(6) manual sync

The new repo(s) you just created would not have been synced up to frodo. You can either make an empty commit and push, or log on to sam and run

 gl-mirror-shell request-push p1/reponame

next steps

Once you've done the initial setup, here's what ongoing additions will require.

any new repos that are created for the same product require only step 3
a new product will require steps A1, 3, 4 and 5
a new host will require additions in all the steps, including adding the hostname in the slaves list for the admin repo (this is in the main gitolite.conf file)

appendix A: delegation helper files

These two files were briefly mentioned in the delegation setup.

(A1) conf/host-product-map.conf has the following contents:

# For each host foo, there will be one line:
#   @foo    =   foo/..*
# line (for local repos for the host).

# For each product bar whose master is a host foo, there will be one line:
#   @foo    =   bar/..*
# to add bar/..* to the allowed patterns when subconf "foo" is in effect

# ------------------------------------------------------------------------------
# so for our example:

@sam        =   sam/..*
@sam        =   p1/..*

@frodo      =   frodo/..*

(A2) conf/NAME-restrictions.conf has the following contents:

# For each host foo, there will be two lines:
#   RW                          =   username-of-foo-host-admin
#   RW  NAME/conf/master/foo/   =   username-of-foo-host-admin
#           IMPORTANT: DO NOT MISS THE TRAILING SLASH IN THE LINE ABOVE!

# ------------------------------------------------------------------------------

repo gitolite-admin     # this line is required

    RW                          =   sam-admin
    RW  NAME/conf/master/sam/   =   sam-admin

    RW                          =   frodo-admin
    RW  NAME/conf/master/frodo/ =   frodo-admin

12 KiB Raw Blame History