deac/gitolite
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

(major change in big-config mode) split the compiled config file

Browse source 
Fedora's config has over 11,000 repositories and the compiled config
file is over 20 MB in size.  Although negligible on a server class
machine, on my laptop just parsing this file takes a good 2.5 seconds.

Even if you use GL_ALL_READ_ALL (see a couple of commits before this
one) to remove the overhead for 'read's, that's still a pretty big
overhead for writes.  And GL_ALL_READ_ALL is not really a solution for
most people anyway.

With this commit, using GL_BIG_CONFIG adds another optimisation; see
doc/big-config.mkd for details (look for the word "split config" to find
the section that talks about it).

----

Implementation notes:

  - the check for GL_NO_CREATE_REPOS has moved *into* the loop (which it
    completely bypassed earlier) so that write_1_compiled_conf can be
    called on each item
This commit is contained in:
Sitaram Chamarty 2011-01-01 15:14:54 +05:30
parent 7fc1e9459f
commit 10a30c961d
9 changed files with 326 additions and 161 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

229
doc/big-config.mkd
View file

@ -4,6 +4,8 @@ In this document:
* <a href="#_when_why_do_we_need_it_">when/why do we need it?</a> * <a href="#_when_why_do_we_need_it_">when/why do we need it?</a>
* <a href="#_how_do_we_use_it_">how do we use it?</a> * <a href="#_how_do_we_use_it_">how do we use it?</a>
* <a href="#_access_rules_for_groups">access rules for groups</a>
* <a href="#_access_rules_for_individual_repos_split_config_">access rules for individual repos (split config)</a>
* <a href="#_other_optimisations">other optimisations</a> * <a href="#_other_optimisations">other optimisations</a>
* <a href="#_disabling_various_defaults">disabling various defaults</a> * <a href="#_disabling_various_defaults">disabling various defaults</a>
* <a href="#_optimising_the_authkeys_file">optimising the authkeys file</a> * <a href="#_optimising_the_authkeys_file">optimising the authkeys file</a>
@ -18,10 +20,10 @@ In this document:
### when/why do we need it? ### when/why do we need it?
A "big config" is anything that has a few thousand users and a few thousand A "big config" is anything that has a few thousand users and a few thousand
repos, organised into groups that are much smaller in number (like maybe a few repos, resulting in a very large 'compiled' config file.
hundreds of repogroups and a few dozens of usergroups).
So let's say you have To understand the problem, consider what happens if you have something like
this in your gitolite conf file:
@wbr = lynx firefox @wbr = lynx firefox
@devs = alice bob @devs = alice bob
@ -30,15 +32,15 @@ So let's say you have
RW+ next = @devs RW+ next = @devs
RW master = @devs RW master = @devs
Gitolite internally translates this to Without the 'big config' setting, gitolite internally translates this to:
repo lynx firefox repo lynx firefox
RW+ next = alice bob RW+ next = alice bob
RW master = alice bob RW master = alice bob
Not just that -- it now generates the actual config rules once for each and then generates the actual config rules once for each user-repo-ref
user-repo-ref combination (there are 8 combinations above; the compiled config combination (there are 8 combinations above); the compiled config file looks
file looks partly like this: somewhat like this:
%repos = ( %repos = (
'firefox' => { 'firefox' => {
@ -51,20 +53,28 @@ file looks partly like this:
'bob' => 1 'bob' => 1
}, },
'alice' => [ 'alice' => [
{ [
'refs/heads/next' => 'RW+' 0,
}, 'refs/heads/next',
{ 'RW+'
'refs/heads/master' => 'RW' ],
} [
4,
'refs/heads/master',
'RW'
]
], ],
'bob' => [ 'bob' => [
{ [
'refs/heads/next' => 'RW+' 1,
}, 'refs/heads/next',
{ 'RW+'
'refs/heads/master' => 'RW' ],
} [
5,
'refs/heads/master',
'RW'
]
] ]
}, },
'lynx' => { 'lynx' => {
@ -77,54 +87,73 @@ file looks partly like this:
'bob' => 1 'bob' => 1
}, },
'alice' => [ 'alice' => [
{ [
'refs/heads/next' => 'RW+' 2,
}, 'refs/heads/next',
{ 'RW+'
'refs/heads/master' => 'RW' ],
} [
6,
'refs/heads/master',
'RW'
]
], ],
'bob' => [ 'bob' => [
{ [
'refs/heads/next' => 'RW+' 3,
}, 'refs/heads/next',
{ 'RW+'
'refs/heads/master' => 'RW' ],
} [
7,
'refs/heads/master',
'RW'
]
] ]
} }
); );
Phew! Phew!
You can imagine what that does when you have 10,000 users and 10,000 repos. Of course, the output is the same whether you used groups (like `@wbr` and
Let's just say it's not pretty :) `@devs` in the example above) or listed the repos directly on the 'repo'
lines.
Anyway, you can imagine what that does when you have 10,000 users and 10,000
repos. Let's just say it's not pretty :)
<a name="_how_do_we_use_it_"></a> <a name="_how_do_we_use_it_"></a>
### how do we use it? ### how do we use it?
Now, if you had all those 10,000 users and repos explicitly listed (no
groups), then there is no help. But if, like the above example, you had
groups like we used above, there is hope.
Just set Just set
$GL_BIG_CONFIG = 1; $GL_BIG_CONFIG = 1;
in the `~/.gitolite.rc` file on the server (see next section for more in the `~/.gitolite.rc` file on the server (see next section for more
variables). When you do that, and push this configuration, the compiled file variables). When you do that, and push this configuration, one of two things
looks like this: happens.
<a name="_access_rules_for_groups"></a>
#### access rules for groups
If you used group names in the 'repo' lines (as in `repo @wbr`), then the
compiled config looks like this:
%repos = ( %repos = (
'@wbr' => { '@wbr' => {
'@devs' => [ '@devs' => [
{ [
'refs/heads/next' => 'RW+' 0,
}, 'refs/heads/next',
{ 'RW+'
'refs/heads/master' => 'RW' ],
} [
1,
'refs/heads/master',
'RW'
]
], ],
'R' => { 'R' => {
'@devs' => 1 '@devs' => 1
@ -132,7 +161,7 @@ looks like this:
'W' => { 'W' => {
'@devs' => 1 '@devs' => 1
} }
}, }
); );
%groups = ( %groups = (
'@devs' => { '@devs' => {
@ -148,6 +177,62 @@ looks like this:
That's a lot smaller, and allows orders of magintude more repos and groups to That's a lot smaller, and allows orders of magintude more repos and groups to
be supported. be supported.
<a name="_access_rules_for_individual_repos_split_config_"></a>
#### access rules for individual repos (split config)
If, on the other hand, you had the repos listed individually, (as in `repo
lynx firefox`), then the main config file would now look like this:
%repos = ();
%split_conf = (
'firefox' => 1,
'lynx' => 1
);
And each individual repo's configuration would go its own directory. For
instance, `~/repositories/lynx.git/gl-conf` would look like this:
%one_repo = (
'lynx' => {
'R' => {
'alice' => 1,
'bob' => 1
},
'W' => {
'alice' => 1,
'bob' => 1
},
'alice' => [
[
0,
'refs/heads/next',
'RW+'
],
[
4,
'refs/heads/master',
'RW'
]
],
'bob' => [
[
1,
'refs/heads/next',
'RW+'
],
[
5,
'refs/heads/master',
'RW'
]
]
}
);
That does not reduce the overall size of the repo config (because you did not
group the repos), but the main repo config is now even smaller!
<a name="_other_optimisations"></a> <a name="_other_optimisations"></a>
### other optimisations ### other optimisations
@ -169,22 +254,18 @@ if you *do* have a large number of repositories, and do *not* use gitolite's
support for gitweb or git-daemon access (see "[easier to specify gitweb support for gitweb or git-daemon access (see "[easier to specify gitweb
description and gitweb/daemon access][gwd]" for details). This will save a description and gitweb/daemon access][gwd]" for details). This will save a
lot of time when you push the gitolite-admin repo with changes. This variable lot of time when you push the gitolite-admin repo with changes. This variable
also control whether "git config" lines (such as `config hooks.emailprefix = also controls whether "git config" lines (such as `config hooks.emailprefix =
"[gitolite]"`) will be processed or not. "[gitolite]"`) will be processed or not.
Setting this is relatively harmless to a normal installation, unlike the next You should be a lot more careful with `GL_NO_CREATE_REPOS` and
two variables :-) `GL_NO_CREATE_REPOS` and `GL_NO_SETUP_AUTHKEYS` are meant `GL_NO_SETUP_AUTHKEYS`. These are meant for installations where some backend
for installations where some backend system already exists that does all the system already exists that does all the actual repo creation, (including
actual repo creation, and all the authentication setup (ssh auth keys), setting up the proper hooks -- very important for access control), and all the
respectively. authentication setup (ssh auth keys), respectively.
Summary: Please **leave those two variables alone** unless you're initials are Summary: Please **leave those two variables alone** unless you're initials are
"JK" ;-) "JK" ;-)
Also note that using all 3 of the `GL_NO_*` variables will result in
*everything* after the config compile being skipped. In other words, gitolite
is being used **only** for its access control language.
<a name="_optimising_the_authkeys_file"></a> <a name="_optimising_the_authkeys_file"></a>
#### optimising the authkeys file #### optimising the authkeys file
@ -228,15 +309,29 @@ this (note the clever date command that always gets you last months log file!)
### what are the downsides? ### what are the downsides?
There is one minor issue. There are some downsides. The first one applies in all cases:
If you use the delegation feature, you can no longer define or extend * If you use the delegation feature, you can no longer define or extend
@groups in a fragment, for security reasons. It will also not let you use any @groups in a fragment, for security reasons. It will also not let you use
group other than the @fragname itself (specifically, groups which contained a any group other than the @fragname itself (specifically, groups which
subset of the allowed @fragname, which would work normally, do not work now). contained a subset of the allowed @fragname, which would work normally, do
not work now).
(If you didn't understand all that, you're probably not using delegation, so (If you didn't understand all that, you're probably not using delegation,
feel free to ignore it!) so feel free to ignore it!)
The following apply if individual ("split") conf files are written, which in
turn only happens if you used repo names instead of group names on the `repo`
lines:
* the compile (gitolite-admin push) is now slower, because it potentially
has to write a few thousand small files instead of one large one. Since
the compile should be relatively infrequent compared to developer access,
this is ok -- the main config file is parsed much faster now, so every hit
to the server will benefit.
* we can no longer distinguish 'repo not found on disk' from 'you dont have
access'. They both now look like 'you dont have access'.
<a name="_storing_usergroup_information_outside_gitolite_like_in_LDAP_"></a> <a name="_storing_usergroup_information_outside_gitolite_like_in_LDAP_"></a>
@ -298,10 +393,10 @@ path to this program, set `$GL_BIG_CONFIG` to 1, and that will be that.
### implementation notes ### implementation notes
To understand how big-config works, we'll first look at how it works without To understand how big-config works (at least when you're using grouped repos),
this setting. Think back to the example at the top, and assume 'alice' is we'll first look at how it works without this setting. Think back to the
accessing the 'lynx' repo. The various rights are governed by the following example at the top, and assume 'alice' is accessing the 'lynx' repo. The
hash elements: various rights are governed by the following hash elements:
# for the first level checks # for the first level checks
$repos{'lynx'}{'R'}{'alice'} = 1 $repos{'lynx'}{'R'}{'alice'} = 1

48
src/gitolite.pm
View file

@ -44,9 +44,14 @@ our ($REPO_UMASK, $GL_WILDREPOS, $GL_PACKAGE_CONF, $GL_PACKAGE_HOOKS, $REPO_BASE
our %repos; our %repos;
our %groups; our %groups;
our %git_configs; our %git_configs;
our %split_conf;;
our $data_version; our $data_version;
our $current_data_version = '1.7'; our $current_data_version = '1.7';
# the following are read in from individual repo's gl-conf files, if present
our %one_repo;
our %one_git_config;
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# convenience subs # convenience subs
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
@ -180,33 +185,19 @@ sub ln_sf
} }
} }
# collect repo patterns for all %repos # list physical repos
sub list_phy_repos
# for each repo passed (actual repos only please!), use either its own name if
# it exists as is in the repos hash, or find and use the pattern that matches
sub collect_repo_patts
{ {
my $repos_p = shift; my @phy_repos;
my %repo_patts = ();
wrap_chdir("$ENV{GL_REPO_BASE_ABS}"); wrap_chdir("$ENV{GL_REPO_BASE_ABS}");
for my $repo (`find . -type d -name "*.git"`) { for my $repo (`find . -type d -name "*.git"`) {
chomp ($repo); chomp ($repo);
$repo =~ s(\./(.*)\.git$)($1); $repo =~ s(\./(.*)\.git$)($1);
# the key has to be in the list, since the repo physically exists push @phy_repos, $repo;
# -- my($perm, $creator, $wild) = &repo_rights($repo);
# -- $repo_patts{$repo} = $wild || $repo;
# turns out we're not using the value anywhere, so no point wasting
# all those cycles getting all repos' rights, at least until a real
# use for it comes along. But when it does come along, remember that
# $wild is now a space separated list of matching patterns (or empty
# if no wild patterns matched $repo). It is NOT a single value
# anymore!
$repo_patts{$repo} = 1;
} }
return %repo_patts; return @phy_repos;
} }
@ -337,6 +328,7 @@ sub new_repo
# really care; we just pull it in once and save it for the rest of # really care; we just pull it in once and save it for the rest of
# the run # the run
do $GL_CONF_COMPILED; do $GL_CONF_COMPILED;
add_repo_conf($repo) if $repo;
%cached_groups = %groups; %cached_groups = %groups;
$cache_filled++; $cache_filled++;
} }
@ -559,8 +551,6 @@ sub parse_acl
%repos = %saved_repos; %groups = %saved_groups; %repos = %saved_repos; %groups = %saved_groups;
} else { } else {
die "parse $GL_CONF_COMPILED failed: " . ($! or $@) unless do $GL_CONF_COMPILED; die "parse $GL_CONF_COMPILED failed: " . ($! or $@) unless do $GL_CONF_COMPILED;
$saved_crwu = "$creator,$perm_cats_sig,$gl_user";
%saved_repos = %repos; %saved_groups = %groups;
} }
unless (defined($data_version) and $data_version eq $current_data_version) { unless (defined($data_version) and $data_version eq $current_data_version) {
# this cannot happen for 'easy-install' cases, by the way... # this cannot happen for 'easy-install' cases, by the way...
@ -569,6 +559,9 @@ sub parse_acl
die "parse $GL_CONF_COMPILED failed: " . ($! or $@) unless do $GL_CONF_COMPILED; die "parse $GL_CONF_COMPILED failed: " . ($! or $@) unless do $GL_CONF_COMPILED;
} }
$saved_crwu = "$creator,$perm_cats_sig,$gl_user";
%saved_repos = %repos; %saved_groups = %groups;
add_repo_conf($repo) if $repo;
# basic access reporting doesn't send $repo, and doesn't need to; you just # basic access reporting doesn't send $repo, and doesn't need to; you just
# want the config dumped as is, really # want the config dumped as is, really
@ -607,6 +600,17 @@ sub parse_acl
return ($wild); return ($wild);
} }
# add repo conf from repo.git/gl-conf
sub add_repo_conf
{
my ($repo) = shift;
return unless $split_conf{$repo};
do "$ENV{GL_REPO_BASE_ABS}/$repo.git/gl-conf" or return;
$repos{$repo} = $one_repo{$repo};
$git_configs{$repo} = $one_git_config{$repo};
}
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# print a report of $user's basic permissions # print a report of $user's basic permissions
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
@ -643,6 +647,8 @@ sub report_basic
local $ENV{GL_USER} = $user; local $ENV{GL_USER} = $user;
&parse_acl($GL_CONF_COMPILED, "", "CREATOR"); &parse_acl($GL_CONF_COMPILED, "", "CREATOR");
# all we need is for 'keys %repos' to come up with all the names, so:
@repos{ keys %split_conf } = values %split_conf if %split_conf;
# send back some useful info if no command was given # send back some useful info if no command was given
&report_version($GL_ADMINDIR, $user); &report_version($GL_ADMINDIR, $user);

1
src/gl-auth-command
View file

@ -38,6 +38,7 @@ our ($R_COMMANDS, $W_COMMANDS, $REPONAME_PATT, $REPOPATT_PATT, $ADC_CMD_ARGS_PAT
our %repos; our %repos;
our %groups; our %groups;
our %git_configs; our %git_configs;
our %split_conf;;
# the common setup module is in the same directory as this running program is # the common setup module is in the same directory as this running program is
my $bindir = $0; my $bindir = $0;

150
src/gl-compile-conf
View file

@ -92,6 +92,9 @@ our %groups = ();
# in between :) # in between :)
my %repos = (); my %repos = ();
# repos whose ACLs don't make it into the main compiled config file
my %split_conf = ();
# rule sequence number # rule sequence number
my $rule_seq = 0; my $rule_seq = 0;
@ -398,26 +401,31 @@ for my $fragment_file (glob("conf/fragments/*.conf"))
parse_conf_file($fragment_file, $fragment); parse_conf_file($fragment_file, $fragment);
} }
my $compiled_fh = wrap_open( ">", "$GL_CONF_COMPILED.new" ); sub write_compiled_conf
my $data_version = $current_data_version; {
print $compiled_fh Data::Dumper->Dump([$data_version], [qw(*data_version)]); my $compiled_fh = wrap_open( ">", "$GL_CONF_COMPILED.new" );
my $dumped_data = Data::Dumper->Dump([\%repos], [qw(*repos)]); my $data_version = $current_data_version;
$dumped_data .= Data::Dumper->Dump([\%git_configs], [qw(*git_configs)]) if %git_configs; print $compiled_fh Data::Dumper->Dump([$data_version], [qw(*data_version)]);
# the dump uses single quotes, but we convert any strings containing $creator my $dumped_data = Data::Dumper->Dump([\%repos], [qw(*repos)]);
# and $gl_user to double quoted strings. A bit sneaky, but not too much... $dumped_data .= Data::Dumper->Dump([\%git_configs], [qw(*git_configs)]) if %git_configs;
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g; # the dump uses single quotes, but we convert any strings containing $creator
print $compiled_fh $dumped_data; # and $gl_user to double quoted strings. A bit sneaky, but not too much...
if (%groups) {
$dumped_data = Data::Dumper->Dump([\%groups], [qw(*groups)]);
$dumped_data =~ s/\bCREAT[EO]R\b/\$creator/g;
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g; $dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
print $compiled_fh $dumped_data; print $compiled_fh $dumped_data;
if (%groups) {
$dumped_data = Data::Dumper->Dump([\%groups], [qw(*groups)]);
$dumped_data =~ s/\bCREAT[EO]R\b/\$creator/g;
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
print $compiled_fh $dumped_data;
}
print $compiled_fh Data::Dumper->Dump([\%split_conf], [qw(*split_conf)]) if %split_conf;
close $compiled_fh or die "$ABRT close compiled-conf failed: $!\n";
rename "$GL_CONF_COMPILED.new", "$GL_CONF_COMPILED";
} }
close $compiled_fh or die "$ABRT close compiled-conf failed: $!\n";
rename "$GL_CONF_COMPILED.new", "$GL_CONF_COMPILED";
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# (that ends the config file compiler and write) # (that ends the config file compiler, though we postpone the writing
# for now to deal with the latest GL_BIG_CONFIG innovation!)
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
@ -443,25 +451,31 @@ die "\n\t\t***** AAARGH! *****\n" .
"\tthe newer features, please upgrade.\n" "\tthe newer features, please upgrade.\n"
if $git_version < 10602; # that's 1.6.2 to you if $git_version < 10602; # that's 1.6.2 to you
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# the rest of this program can be "switched off"; see doc/big-config.mkd for # most of the rest of this program can be "switched off"; see
# details. # doc/big-config.mkd for details.
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# ---------------------------------------------------------------------------- # repo-base needs to be an absolute path due to all the jumping around we do,
# any new repos to be created?
# ----------------------------------------------------------------------------
# repo-base needs to be an absolute path for this loop to work right
# so if it was not already absolute, prefix $HOME. # so if it was not already absolute, prefix $HOME.
$ENV{GL_REPO_BASE_ABS} = ( $REPO_BASE =~ m(^/) ? $REPO_BASE : "$ENV{HOME}/$REPO_BASE" ); $ENV{GL_REPO_BASE_ABS} = ( $REPO_BASE =~ m(^/) ? $REPO_BASE : "$ENV{HOME}/$REPO_BASE" );
unless ($GL_NO_CREATE_REPOS) { # process the normal repos in %repos. This includes creating them if needed
# (and GL_NO_CREATE_REPOS is not set), checking hooks, and finally, if
# GL_BIG_CONFIG is set, writing out the one-repo config file for directly
# specified repos (i.e., "repo foo", not "@grp = foo" + "repo @grp")
do_normal_repos();
write_compiled_conf(); # write out the final compiled config
# ----------------------------------------------------------------------------
# process the normal repos in %repos (create, hook, one_repo config...)
# ----------------------------------------------------------------------------
sub do_normal_repos
{
wrap_chdir("$ENV{GL_REPO_BASE_ABS}"); wrap_chdir("$ENV{GL_REPO_BASE_ABS}");
# autocreate repos. Start with the ones that are normal repos in %repos # start with the ones that are normal repos in %repos
my @repos = grep { $_ =~ $REPONAME_PATT and not /^@/ } sort keys %repos; my @repos = grep { $_ =~ $REPONAME_PATT and not /^@/ } sort keys %repos;
# then, for each repogroup, find the members of the group and add them in # then, for each repogroup, find the members of the group and add them in
map { push @repos, keys %{ $groups{$_} } } grep { /^@/ } keys %repos; map { push @repos, keys %{ $groups{$_} } } grep { /^@/ } keys %repos;
@ -470,39 +484,67 @@ unless ($GL_NO_CREATE_REPOS) {
@repos = sort keys %seen; @repos = sort keys %seen;
for my $repo (sort @repos) { for my $repo (sort @repos) {
next unless $repo =~ $REPONAME_PATT; next unless $repo =~ $REPONAME_PATT; # skip repo patterns
next if $repo =~ m(^\@|EXTCMD/); # these are not real repos next if $repo =~ m(^\@|EXTCMD/); # skip groups and fake repos
unless (-d "$repo.git") {
print STDERR "creating $repo...\n"; unless ($GL_NO_CREATE_REPOS) {
new_repo($repo, "$GL_ADMINDIR/hooks/common"); unless (-d "$repo.git") {
# new_repo would have chdir'd us away; come back print STDERR "creating $repo...\n";
wrap_chdir("$ENV{GL_REPO_BASE_ABS}"); new_repo($repo, "$GL_ADMINDIR/hooks/common");
# new_repo would have chdir'd us away; come back
wrap_chdir("$ENV{GL_REPO_BASE_ABS}");
}
# when repos are copied over from elsewhere, one had to run easy install
# once again to make the new (OS-copied) repo contain the proper update
# hook. Perhaps we can make this easier now, and eliminate the easy
# install, with a quick check (and a new, empty, "hook" as a sentinel)
unless (-l "$repo.git/hooks/gitolite-hooked") {
ln_sf("$GL_ADMINDIR/hooks/common", "*", "$repo.git/hooks");
# in case of package install, GL_ADMINDIR is no longer the top cop;
# override with the package hooks
ln_sf("$GL_PACKAGE_HOOKS/common", "*", "$repo.git/hooks") if $GL_PACKAGE_HOOKS;
}
} }
# when repos are copied over from elsewhere, one had to run easy install # write a one_repo config for normal repos declared directly (not just via a group)
# once again to make the new (OS-copied) repo contain the proper update write_1_compiled_conf($repo) if $GL_BIG_CONFIG and $repos{$repo} and -d "$repo.git";
# hook. Perhaps we can make this easier now, and eliminate the easy
# install, with a quick check (and a new, empty, "hook" as a sentinel)
unless (-l "$repo.git/hooks/gitolite-hooked") {
ln_sf("$GL_ADMINDIR/hooks/common", "*", "$repo.git/hooks");
# in case of package install, GL_ADMINDIR is no longer the top cop;
# override with the package hooks
ln_sf("$GL_PACKAGE_HOOKS/common", "*", "$repo.git/hooks") if $GL_PACKAGE_HOOKS;
}
} }
} }
sub write_1_compiled_conf
{
# warning: writes and *deletes* it from %repos and %git_configs
my ($repo) = shift;
my (%one_repo, %one_git_config);
open(my $compiled_fh, ">", "$repo.git/gl-conf") or return;
$one_repo{$repo} = $repos{$repo};
delete $repos{$repo};
my $dumped_data = Data::Dumper->Dump([\%one_repo], [qw(*one_repo)]);
if ($git_configs{$repo}) {
$one_git_config{$repo} = $git_configs{$repo};
delete $git_configs{$repo};
$dumped_data .= Data::Dumper->Dump([\%one_git_config], [qw(*one_git_config)]);
}
# the dump uses single quotes, but we convert any strings containing $creator
# and $gl_user to double quoted strings. A bit sneaky, but not too much...
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
print $compiled_fh $dumped_data;
close $compiled_fh;
$split_conf{$repo} = 1;
}
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# collect repo_patt for each actual repo # get a list of physical repos for later
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# go through each actual repo on disk, and match it to either its own name in my @phy_repos = ();
# the config (non-wild) or a wild pattern that matches it. Lots of things @phy_repos = &list_phy_repos() unless $GL_NO_DAEMON_NO_GITWEB;
# later will need this correspondence so we may as well snarf it in one shot
my %repo_patts = ();
%repo_patts = &collect_repo_patts(\%repos) unless $GL_NO_DAEMON_NO_GITWEB;
# NOTE: we're overloading GL_NO_DAEMON_NO_GITWEB to mean "no git config" also. # NOTE: we're overloading GL_NO_DAEMON_NO_GITWEB to mean "no git config" also.
# In fact anything that requires trawling through the existing repos doing # In fact anything that requires trawling through the existing repos doing
@ -520,8 +562,6 @@ my %repo_patts = ();
# update repo configurations, gitweb description, daemon export-ok, etc # update repo configurations, gitweb description, daemon export-ok, etc
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# all these require a "chdir" to the repo, so we club them for efficiency
my %projlist = (); my %projlist = ();
# for each real repo (and remember this will be empty, thus skipping all this, # for each real repo (and remember this will be empty, thus skipping all this,
@ -530,13 +570,13 @@ my %projlist = ();
# note: we do them in 2 separate loops to avoid breaking the optimisation in # note: we do them in 2 separate loops to avoid breaking the optimisation in
# sub parse_acl (look for variable $saved_crwu) # sub parse_acl (look for variable $saved_crwu)
for my $repo (keys %repo_patts) { for my $repo (@phy_repos) {
wrap_chdir("$ENV{GL_REPO_BASE_ABS}/$repo.git"); wrap_chdir("$ENV{GL_REPO_BASE_ABS}/$repo.git");
# daemon is easy # daemon is easy
&setup_daemon_access($repo); &setup_daemon_access($repo);
} }
for my $repo (keys %repo_patts) { for my $repo (@phy_repos) {
wrap_chdir("$ENV{GL_REPO_BASE_ABS}/$repo.git"); wrap_chdir("$ENV{GL_REPO_BASE_ABS}/$repo.git");
# gitweb is a little more complicated. Here're some notes: # gitweb is a little more complicated. Here're some notes:
# - "setup_gitweb_access" also sets "owner", despite the name # - "setup_gitweb_access" also sets "owner", despite the name

26
t/out/t01-repo-groups.2
View file

@ -23,7 +23,21 @@ $data_version = '1.7';
'@g1' => 1, '@g1' => 1,
'@g2' => 1 '@g2' => 1
} }
}, }
);
%groups = (
'@g1' => {
'aa' => 'master',
'bb' => 'master'
}
);
%split_conf = (
'gitolite-admin' => 1,
'testing' => 1
);
repositories/gitolite-admin.git/gl-conf
repositories/testing.git/gl-conf
%one_repo = (
'gitolite-admin' => { 'gitolite-admin' => {
'R' => { 'R' => {
'tester' => 1 'tester' => 1
@ -38,7 +52,9 @@ $data_version = '1.7';
'RW+' 'RW+'
] ]
] ]
}, }
);
%one_repo = (
'testing' => { 'testing' => {
'@all' => [ '@all' => [
[ [
@ -55,9 +71,3 @@ $data_version = '1.7';
} }
} }
); );
%groups = (
'@g1' => {
'aa' => 'master',
'bb' => 'master'
}
);

6
t/t01-repo-groups
View file

@ -39,8 +39,8 @@ echo "
RW = u2 u3 RW = u2 u3
" | ugc " | ugc
catconf catconfs
expect_filesame $TESTDIR/out/t01-repo-groups.1 expect_filesame $TESTDIR/out/t01-repo-groups.1bs
# ---------- # ----------
$TESTDIR/rollback || die "rollback failed" $TESTDIR/rollback || die "rollback failed"
@ -54,7 +54,7 @@ echo "
RW = @g2 RW = @g2
" | ugc " | ugc
catconf catconfs
expect_filesame $TESTDIR/out/t01-repo-groups.2 expect_filesame $TESTDIR/out/t01-repo-groups.2
name INTERNAL name INTERNAL

8
t/t02-user-groups
View file

@ -42,8 +42,8 @@ echo "
RW = u2 u3 RW = u2 u3
" | ugc " | ugc
catconf catconfs
expect_filesame $TESTDIR/out/t02-user-groups.1 expect_filesame $TESTDIR/out/t02-user-groups.1bs
# ---------- # ----------
$TESTDIR/rollback || die "rollback failed" $TESTDIR/rollback || die "rollback failed"
@ -60,7 +60,7 @@ echo "
RW = @g2 RW = @g2
" | ugc " | ugc
catconf catconfs
expect_filesame $TESTDIR/out/t02-user-groups.2 expect_filesame $TESTDIR/out/t02-user-groups.2bs
name INTERNAL name INTERNAL

12
t/t59-repo-not-on-disk
View file

@ -76,15 +76,21 @@ do
cd ~/td cd ~/td
name "repo on disk missing: u1" name "repo on disk missing: u1"
runlocal git clone u1:aa runlocal git clone u1:aa
expect "fatal: 'repositories/aa.git' does not appear to be a git repository" [ "$bc" = "0" ] && expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
[ "$bc" = "1" ] && expect "R access for aa DENIED to u1"
[ "$bc" = "1" ] && expect "Or there may be no repository at the given path. Did you spell it correctly?"
name "repo on disk missing: tester" name "repo on disk missing: tester"
runlocal git clone gitolite:aa runlocal git clone gitolite:aa
expect "fatal: 'repositories/aa.git' does not appear to be a git repository" [ "$bc" = "0" ] && expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
[ "$bc" = "1" ] && expect "R access for aa DENIED to tester"
[ "$bc" = "1" ] && expect "Or there may be no repository at the given path. Did you spell it correctly?"
name "repo on disk missing: u4" name "repo on disk missing: u4"
runlocal git clone u4:aa runlocal git clone u4:aa
expect "fatal: 'repositories/aa.git' does not appear to be a git repository" [ "$bc" = "0" ] && expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
[ "$bc" = "1" ] && expect "R access for aa DENIED to u4"
[ "$bc" = "1" ] && expect "Or there may be no repository at the given path. Did you spell it correctly?"
name "repo on disk missing: u6" name "repo on disk missing: u6"
runlocal git clone u6:aa runlocal git clone u6:aa

7
t/test-driver.sh
View file

@ -14,6 +14,13 @@ runremote() { ssh gitolite-test@localhost "$@" > ~/1 2> ~/2; }
listrepos() { ssh gitolite-test@localhost find repositories -type d -name "*.git" | sort > ~/1 2> ~/2; } listrepos() { ssh gitolite-test@localhost find repositories -type d -name "*.git" | sort > ~/1 2> ~/2; }
# remote cat compiled pm # remote cat compiled pm
catconf() { ssh gitolite-test@localhost cat .gitolite/conf/gitolite.conf-compiled.pm > ~/1 2> ~/2; } catconf() { ssh gitolite-test@localhost cat .gitolite/conf/gitolite.conf-compiled.pm > ~/1 2> ~/2; }
catconfs() {
(
ssh gitolite-test@localhost cat .gitolite/conf/gitolite.conf-compiled.pm
ssh gitolite-test@localhost find repositories -name gl-conf \| sort
ssh gitolite-test@localhost find repositories -name gl-conf \| sort \| xargs cat
) > ~/1 2> ~/2
}
# remote cat ~/.gitolite.rc # remote cat ~/.gitolite.rc
catrc() { ssh gitolite-test@localhost cat .gitolite.rc > ~/1 2> ~/2; } catrc() { ssh gitolite-test@localhost cat .gitolite.rc > ~/1 2> ~/2; }
# tail gitolite logfile # tail gitolite logfile