(mirroring) make cron jobs easier to write

gl-mirror-shell will now take a list of slaves and/or keys, expanding
the keys in place.  See doc for even more improvements and conveniences.
This commit is contained in:
Sitaram Chamarty 2011-08-14 10:06:16 +05:30
parent aa7ff8ac27
commit b11d44e036
4 changed files with 132 additions and 55 deletions

View file

@ -303,44 +303,99 @@ So here's how our example would go:
#### commands to (re-)sync mirrors
Sometimes there's a network problem and a mirror will not receive an update
immediately on a push. When the network is back up, you can do one of these
things to get it back in sync.
You don't have to put all the slaves in `gitolite.mirror.slaves`. For
example, let's say you have some repos that are very active, and two of your
mirrors that are halfway across the world are getting pushed very frequently.
But you don't need those mirrors to be that closely updated, perhaps *because*
they are halfway across the world and those guys are asleep ;-)
Or maybe there was a network glitch and even the default slaves are now
lagging, so they need to be manually synced.
Or a slave realised that one of its repos is lagging for some reason, and
wants to request an immediate update.
Whatever the reason, you need ways to sync a repo from a command line. Here
are ways to do that:
1. On the master server, you can start a **background** job to mirror a repo.
For example, this:
The command/syntax is
gl-mirror-shell request-push reponame [list of keys/slaves]
The list at the end is optional, and can be a mix of slave names or your
own gitolite mirror config keys. (Yes, you can have any key, named
anything you like, as long as it starts with `gitolite.mirror.`).
If the list is not supplied, the `gitolite.mirror.slaves` key is used.
Keys can have values that in turn contain a list of keys/slaves. The list
is recursively *expanded* but recursion is not *detected*. Order is
preserved while duplicates are removed. If you didn't get that, see the
example :-)
**Warning**: the `gitolite.mirror.slaves` key should have only hosts, no
keys, in it.
The program exits with a return value of "1" if it found no slaves in the
list passed, otherwise it fires off the background job, prints an
informative message, and exits with a return value of "0".
We'll take an example. Let's say your gitolite config file has this:
repo ip1
config gitolite.mirror.master = "frodo"
config gitolite.mirror.slaves = "sam merry pippin"
config gitolite.mirror.hourly = "sam legolas"
config gitolite.mirror.nightly = "gitolite.mirror.hourly gimli"
config gitolite.mirror.all = "gitolite.mirror.nightly gitolite.mirror.hourly gitolite.mirror.slaves"
Then the following commands have the results described in comments:
gl-mirror-shell request-push ip1
triggers a mirror-push of repo "ip1" to all slaves listed in that repo's
"gitolite.mirror.slaves" config.
On the hand, this:
# which is the same as:
gl-mirror-shell request-push ip1 gitolite.mirror.slaves
# pushes to sam, merry, pippin
gl-mirror-shell request-push ip1 gollum
# pushes only to gollum. Note that gollum is not a member of any of
# the slave lists we defined.
triggers a mirror-push of "ip1" *only* to the gollum server, regardless of
what servers are listed as slaves in the config.
gl-mirror-shell request-push ip1 gitolite.mirror.slaves gollum
# pushes to sam, merry, pippin, gollum
Note that this invocation does not even check if gollum is listed as a
slave for "ip1"; since you're doing it at the command line on the master
server, you're allowed to push it to *any* slave that will accept it.
gl-mirror-shell request-push ip1 gitolite.mirror.slaves gitolite.mirror.hourly
# pushes to sam, merry, pippin, legolas
<font color="gray">
gl-mirror-shell request-push ip1 gitolite.mirror.all
# pushes to sam, legolas, gimli, merry, pippin
> Side note: if you want to start a **foreground** job, the syntax is
> `gl-mirror-shell request-push ip1 -fg gollum`. Foreground mode
> requires one (and only one) slave name -- you cannot send to an
> implicit list, nor to more than one slave.
The last two examples show recursive expansion with order-preserving
duplicate removal (hey there's now a published conference paper on
gitolite, so we have to use jargon *somewhere* or they won't accept
follow-on papers!).
</font>
If you do something like this:
2. Cronjobs and custom mirroring schemes are now very easy to do. Just use
the second form of the command above to push any repo to any slave, and it
can form the basis of any scheme you like. Appendix A contains an example
setup.
config gitolite.mirror.nightly = "gimli gitolite.mirror.nightly"
3. Once in a while a slave will realise it needs an update, and wants to ask
or this:
config gitolite.mirror.nightly = "gimli gitolite.mirror.hourly"
config gitolite.mirror.hourly = "legolas gitolite.mirror.nightly"
you deserve what you get.
2. If you want to start a **foreground** job, the syntax is `gl-mirror-shell
request-push ip1 -fg gollum`. Foreground mode requires one (and only one)
slave name -- you cannot send to an implicit list, nor to more than one
slave.
3. Cronjobs and custom mirroring schemes are now very easy to do. Use either
of the command forms above and write a script around it. Appendix A
contains an example setup.
4. Once in a while a slave will realise it needs an update, and wants to ask
for one. It can run this command to do so:
ssh sam request-push ip2
@ -389,6 +444,10 @@ config file right?). These are:
See the section on "redirecting pushes"
* In addition, you can create your own slave lists, named whatever you want,
except they have to start with `gitolite.mirror.`. The section on
"commands to (re-)sync mirrors" has some examples.
<a name="_redirecting_pushes"></a>
### redirecting pushes
@ -517,34 +576,31 @@ pros/cons:
### appendix A: example cronjob based mirroring
Let's say you have some repos that are so active that you're pushing halfway
across the world every few seconds. The slaves do not need to be that closely
updated, and it is sufficient to update them once an hour instead. Here's how
you might do that:
Let's say you have some repos that are very active. You're pushing halfway
across the world every few seconds, but those slaves do not need to be that closely
updated, perhaps *because* they are halfway across the world and those guys
are asleep ;-)
You'd like to update them once an hour instead. Here's how you might do that.
First add this line to the configuration for those repos:
repo foo bar frob/nitz
config gitolite.mirror.hourly = "slave1 slave2 slave3"
Then you'd write a cron job that looks like this (untested):
Then write a cron job that looks like this (untested).
#!/bin/bash
REPO_BASE=`${0%/*}/gl-query-rc REPO_BASE`
GL_BINDIR=`${0%/*}/gl-query-rc GL_BINDIR`
cd $REPO_BASE
find . -type d -name "*.git" -prune | while read r
do
cd $REPO_BASE; cd $r
# get reponame as gitolite knows it
r=${r:2}
r=${r%.git}
# get slaves list
slaves=`git config --get gitolite.mirror.hourly`
gl-mirror-shell request-push $r $slaves
gl-mirror-shell request-push $r gitolite.mirror.hourly
# that command backgrounds the push, so you'd best wait a few seconds
# before hitting the next one, otherwise you'll have all your repos

View file

@ -16,7 +16,9 @@
# but we won't!), and (b) we can't distinguish easily between that and this
# case (the slave receiving a mirror push case)
[ -z "$GL_REPO" ] && { echo $0: GL_REPO not set -- this is BAD >&2; exit 1; }
[ -z "$GL_BINDIR" ] && { echo $0: GL_BINDIR not set -- this is BAD >&2; exit 1; }
[ -z "$GL_REPO" ] && die GL_REPO not set
[ -z "$GL_BINDIR" ] && die GL_BINDIR not set
$GL_BINDIR/gl-mirror-push $GL_REPO
slaves=`git config --get gitolite.mirror.slaves`
[ -z "$slaves" ] && exit 0
$GL_BINDIR/gl-mirror-push $GL_REPO $slaves

View file

@ -49,16 +49,11 @@ gmm=${gmm:-local}
# ----------
# normal (self-backgrounding) mode. Any number of slaves. If none are given,
# use the slave list from the repo config
# normal (self-backgrounding) mode, one or more slaves
[ -z "$1" ] && die fatal: missing list of slaves
export slaves
if [ -n "$1" ]
then
slaves="$*"
else
slaves=`git config --get gitolite.mirror.slaves`
fi
# ----------

View file

@ -46,14 +46,38 @@ my $soc = $ENV{SSH_ORIGINAL_COMMAND} || '';
# on the "master", run from a shell, for one specific repo, with an optional
# list of slaves, like so:
# gl-mirror-shell request-push some-repo [optional-list-of-slaves]
# gl-mirror-shell request-push some-repo [optional list of slaves/keys]
if ( ($ARGV[0] || '') eq 'request-push' and not $soc) {
shift;
# rest of the arguments are fit to go directly to gl-mirror-push
# (reponame, optional list of slaves)
system("gl-mirror-push", @ARGV);
my $repo = shift or die "fatal: missing reponame\n";
-d "$REPO_BASE/$repo.git" or die "fatal: no such repo?\n";
exit;
# this is the default argument if no slave list or key is supplied
@ARGV = ('gitolite.mirror.slaves') unless @ARGV;
my @slaves = ();
my %seen = ();
# each argument in @ARGV is either a slave name, or a gitolite mirroring
# key to be replaced with its value, split into a list of slaves
while (@ARGV) {
$a = shift @ARGV;
if ($a =~ /^gitolite\.mirror\.[\w.-]+$/) {
my @values = split(' ', `git config --file $REPO_BASE/$repo.git/config --get $a` || '');
unshift @ARGV, @values;
} else {
push @slaves, $a unless $seen{$a}++;
}
}
exit 1 unless @slaves;
# we don't want to complain louder than that because the most common
# use of this script on the master server is via cron, run against
# *all* known repos without checking their individual key values
print STDERR "info: mirror-push $repo ", join(" ", @slaves), "\n";
system("gl-mirror-push", $repo, @slaves);
exit 0;
}
unless (@ARGV) { print STDERR "fatal: missing command\n"; exit 1; }