(major change in big-config mode) split the compiled config file
Fedora's config has over 11,000 repositories and the compiled config
file is over 20 MB in size. Although negligible on a server class
machine, on my laptop just parsing this file takes a good 2.5 seconds.
Even if you use GL_ALL_READ_ALL (see a couple of commits before this
one) to remove the overhead for 'read's, that's still a pretty big
overhead for writes. And GL_ALL_READ_ALL is not really a solution for
most people anyway.
With this commit, using GL_BIG_CONFIG adds another optimisation; see
doc/big-config.mkd for details (look for the word "split config" to find
the section that talks about it).
----
Implementation notes:
- the check for GL_NO_CREATE_REPOS has moved *into* the loop (which it
completely bypassed earlier) so that write_1_compiled_conf can be
called on each item
This commit is contained in:
Sitaram Chamarty
parent
7fc1e9459f
commit
10a30c961d
|
|
@ -4,6 +4,8 @@ In this document:
|
|||
|
||||
* <a href="#_when_why_do_we_need_it_">when/why do we need it?</a>
|
||||
* <a href="#_how_do_we_use_it_">how do we use it?</a>
|
||||
* <a href="#_access_rules_for_groups">access rules for groups</a>
|
||||
* <a href="#_access_rules_for_individual_repos_split_config_">access rules for individual repos (split config)</a>
|
||||
* <a href="#_other_optimisations">other optimisations</a>
|
||||
* <a href="#_disabling_various_defaults">disabling various defaults</a>
|
||||
* <a href="#_optimising_the_authkeys_file">optimising the authkeys file</a>
|
||||
|
|
@ -18,10 +20,10 @@ In this document:
|
|||
### when/why do we need it?
|
||||
|
||||
A "big config" is anything that has a few thousand users and a few thousand
|
||||
repos, organised into groups that are much smaller in number (like maybe a few
|
||||
hundreds of repogroups and a few dozens of usergroups).
|
||||
repos, resulting in a very large 'compiled' config file.
|
||||
|
||||
So let's say you have
|
||||
To understand the problem, consider what happens if you have something like
|
||||
this in your gitolite conf file:
|
||||
|
||||
@wbr = lynx firefox
|
||||
@devs = alice bob
|
||||
|
|
@ -30,15 +32,15 @@ So let's say you have
|
|||
RW+ next = @devs
|
||||
RW master = @devs
|
||||
|
||||
Gitolite internally translates this to
|
||||
Without the 'big config' setting, gitolite internally translates this to:
|
||||
|
||||
repo lynx firefox
|
||||
RW+ next = alice bob
|
||||
RW master = alice bob
|
||||
|
||||
Not just that -- it now generates the actual config rules once for each
|
||||
user-repo-ref combination (there are 8 combinations above; the compiled config
|
||||
file looks partly like this:
|
||||
and then generates the actual config rules once for each user-repo-ref
|
||||
combination (there are 8 combinations above); the compiled config file looks
|
||||
somewhat like this:
|
||||
|
||||
%repos = (
|
||||
'firefox' => {
|
||||
|
|
@ -51,20 +53,28 @@ file looks partly like this:
|
|||
'bob' => 1
|
||||
},
|
||||
'alice' => [
|
||||
{
|
||||
'refs/heads/next' => 'RW+'
|
||||
},
|
||||
{
|
||||
'refs/heads/master' => 'RW'
|
||||
}
|
||||
[
|
||||
0,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
4,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
],
|
||||
'bob' => [
|
||||
{
|
||||
'refs/heads/next' => 'RW+'
|
||||
},
|
||||
{
|
||||
'refs/heads/master' => 'RW'
|
||||
}
|
||||
[
|
||||
1,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
5,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
]
|
||||
},
|
||||
'lynx' => {
|
||||
|
|
@ -77,54 +87,73 @@ file looks partly like this:
|
|||
'bob' => 1
|
||||
},
|
||||
'alice' => [
|
||||
{
|
||||
'refs/heads/next' => 'RW+'
|
||||
},
|
||||
{
|
||||
'refs/heads/master' => 'RW'
|
||||
}
|
||||
[
|
||||
2,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
6,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
],
|
||||
'bob' => [
|
||||
{
|
||||
'refs/heads/next' => 'RW+'
|
||||
},
|
||||
{
|
||||
'refs/heads/master' => 'RW'
|
||||
}
|
||||
[
|
||||
3,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
7,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
]
|
||||
}
|
||||
);
|
||||
|
||||
Phew!
|
||||
|
||||
You can imagine what that does when you have 10,000 users and 10,000 repos.
|
||||
Let's just say it's not pretty :)
|
||||
Of course, the output is the same whether you used groups (like `@wbr` and
|
||||
`@devs` in the example above) or listed the repos directly on the 'repo'
|
||||
lines.
|
||||
|
||||
Anyway, you can imagine what that does when you have 10,000 users and 10,000
|
||||
repos. Let's just say it's not pretty :)
|
||||
|
||||
<a name="_how_do_we_use_it_"></a>
|
||||
|
||||
### how do we use it?
|
||||
|
||||
Now, if you had all those 10,000 users and repos explicitly listed (no
|
||||
groups), then there is no help. But if, like the above example, you had
|
||||
groups like we used above, there is hope.
|
||||
|
||||
Just set
|
||||
|
||||
$GL_BIG_CONFIG = 1;
|
||||
|
||||
in the `~/.gitolite.rc` file on the server (see next section for more
|
||||
variables). When you do that, and push this configuration, the compiled file
|
||||
looks like this:
|
||||
variables). When you do that, and push this configuration, one of two things
|
||||
happens.
|
||||
|
||||
<a name="_access_rules_for_groups"></a>
|
||||
|
||||
#### access rules for groups
|
||||
|
||||
If you used group names in the 'repo' lines (as in `repo @wbr`), then the
|
||||
compiled config looks like this:
|
||||
|
||||
%repos = (
|
||||
'@wbr' => {
|
||||
'@devs' => [
|
||||
{
|
||||
'refs/heads/next' => 'RW+'
|
||||
},
|
||||
{
|
||||
'refs/heads/master' => 'RW'
|
||||
}
|
||||
[
|
||||
0,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
1,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
],
|
||||
'R' => {
|
||||
'@devs' => 1
|
||||
|
|
@ -132,7 +161,7 @@ looks like this:
|
|||
'W' => {
|
||||
'@devs' => 1
|
||||
}
|
||||
},
|
||||
}
|
||||
);
|
||||
%groups = (
|
||||
'@devs' => {
|
||||
|
|
@ -148,6 +177,62 @@ looks like this:
|
|||
That's a lot smaller, and allows orders of magintude more repos and groups to
|
||||
be supported.
|
||||
|
||||
<a name="_access_rules_for_individual_repos_split_config_"></a>
|
||||
|
||||
#### access rules for individual repos (split config)
|
||||
|
||||
If, on the other hand, you had the repos listed individually, (as in `repo
|
||||
lynx firefox`), then the main config file would now look like this:
|
||||
|
||||
%repos = ();
|
||||
%split_conf = (
|
||||
'firefox' => 1,
|
||||
'lynx' => 1
|
||||
);
|
||||
|
||||
And each individual repo's configuration would go its own directory. For
|
||||
instance, `~/repositories/lynx.git/gl-conf` would look like this:
|
||||
|
||||
%one_repo = (
|
||||
'lynx' => {
|
||||
'R' => {
|
||||
'alice' => 1,
|
||||
'bob' => 1
|
||||
},
|
||||
'W' => {
|
||||
'alice' => 1,
|
||||
'bob' => 1
|
||||
},
|
||||
'alice' => [
|
||||
[
|
||||
0,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
4,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
],
|
||||
'bob' => [
|
||||
[
|
||||
1,
|
||||
'refs/heads/next',
|
||||
'RW+'
|
||||
],
|
||||
[
|
||||
5,
|
||||
'refs/heads/master',
|
||||
'RW'
|
||||
]
|
||||
]
|
||||
}
|
||||
);
|
||||
|
||||
That does not reduce the overall size of the repo config (because you did not
|
||||
group the repos), but the main repo config is now even smaller!
|
||||
|
||||
<a name="_other_optimisations"></a>
|
||||
|
||||
### other optimisations
|
||||
|
|
@ -169,22 +254,18 @@ if you *do* have a large number of repositories, and do *not* use gitolite's
|
|||
support for gitweb or git-daemon access (see "[easier to specify gitweb
|
||||
description and gitweb/daemon access][gwd]" for details). This will save a
|
||||
lot of time when you push the gitolite-admin repo with changes. This variable
|
||||
also control whether "git config" lines (such as `config hooks.emailprefix =
|
||||
also controls whether "git config" lines (such as `config hooks.emailprefix =
|
||||
"[gitolite]"`) will be processed or not.
|
||||
|
||||
Setting this is relatively harmless to a normal installation, unlike the next
|
||||
two variables :-) `GL_NO_CREATE_REPOS` and `GL_NO_SETUP_AUTHKEYS` are meant
|
||||
for installations where some backend system already exists that does all the
|
||||
actual repo creation, and all the authentication setup (ssh auth keys),
|
||||
respectively.
|
||||
You should be a lot more careful with `GL_NO_CREATE_REPOS` and
|
||||
`GL_NO_SETUP_AUTHKEYS`. These are meant for installations where some backend
|
||||
system already exists that does all the actual repo creation, (including
|
||||
setting up the proper hooks -- very important for access control), and all the
|
||||
authentication setup (ssh auth keys), respectively.
|
||||
|
||||
Summary: Please **leave those two variables alone** unless you're initials are
|
||||
"JK" ;-)
|
||||
|
||||
Also note that using all 3 of the `GL_NO_*` variables will result in
|
||||
*everything* after the config compile being skipped. In other words, gitolite
|
||||
is being used **only** for its access control language.
|
||||
|
||||
<a name="_optimising_the_authkeys_file"></a>
|
||||
|
||||
#### optimising the authkeys file
|
||||
|
|
@ -228,15 +309,29 @@ this (note the clever date command that always gets you last months log file!)
|
|||
|
||||
### what are the downsides?
|
||||
|
||||
There is one minor issue.
|
||||
There are some downsides. The first one applies in all cases:
|
||||
|
||||
If you use the delegation feature, you can no longer define or extend
|
||||
@groups in a fragment, for security reasons. It will also not let you use any
|
||||
group other than the @fragname itself (specifically, groups which contained a
|
||||
subset of the allowed @fragname, which would work normally, do not work now).
|
||||
* If you use the delegation feature, you can no longer define or extend
|
||||
@groups in a fragment, for security reasons. It will also not let you use
|
||||
any group other than the @fragname itself (specifically, groups which
|
||||
contained a subset of the allowed @fragname, which would work normally, do
|
||||
not work now).
|
||||
|
||||
(If you didn't understand all that, you're probably not using delegation, so
|
||||
feel free to ignore it!)
|
||||
(If you didn't understand all that, you're probably not using delegation,
|
||||
so feel free to ignore it!)
|
||||
|
||||
The following apply if individual ("split") conf files are written, which in
|
||||
turn only happens if you used repo names instead of group names on the `repo`
|
||||
lines:
|
||||
|
||||
* the compile (gitolite-admin push) is now slower, because it potentially
|
||||
has to write a few thousand small files instead of one large one. Since
|
||||
the compile should be relatively infrequent compared to developer access,
|
||||
this is ok -- the main config file is parsed much faster now, so every hit
|
||||
to the server will benefit.
|
||||
|
||||
* we can no longer distinguish 'repo not found on disk' from 'you dont have
|
||||
access'. They both now look like 'you dont have access'.
|
||||
|
||||
<a name="_storing_usergroup_information_outside_gitolite_like_in_LDAP_"></a>
|
||||
|
||||
|
|
@ -298,10 +393,10 @@ path to this program, set `$GL_BIG_CONFIG` to 1, and that will be that.
|
|||
|
||||
### implementation notes
|
||||
|
||||
To understand how big-config works, we'll first look at how it works without
|
||||
this setting. Think back to the example at the top, and assume 'alice' is
|
||||
accessing the 'lynx' repo. The various rights are governed by the following
|
||||
hash elements:
|
||||
To understand how big-config works (at least when you're using grouped repos),
|
||||
we'll first look at how it works without this setting. Think back to the
|
||||
example at the top, and assume 'alice' is accessing the 'lynx' repo. The
|
||||
various rights are governed by the following hash elements:
|
||||
|
||||
# for the first level checks
|
||||
$repos{'lynx'}{'R'}{'alice'} = 1
|
||||
|
|
|
|||
|
|
@ -44,9 +44,14 @@ our ($REPO_UMASK, $GL_WILDREPOS, $GL_PACKAGE_CONF, $GL_PACKAGE_HOOKS, $REPO_BASE
|
|||
our %repos;
|
||||
our %groups;
|
||||
our %git_configs;
|
||||
our %split_conf;;
|
||||
our $data_version;
|
||||
our $current_data_version = '1.7';
|
||||
|
||||
# the following are read in from individual repo's gl-conf files, if present
|
||||
our %one_repo;
|
||||
our %one_git_config;
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# convenience subs
|
||||
# ----------------------------------------------------------------------------
|
||||
|
|
@ -180,33 +185,19 @@ sub ln_sf
|
|||
}
|
||||
}
|
||||
|
||||
# collect repo patterns for all %repos
|
||||
|
||||
# for each repo passed (actual repos only please!), use either its own name if
|
||||
# it exists as is in the repos hash, or find and use the pattern that matches
|
||||
|
||||
sub collect_repo_patts
|
||||
# list physical repos
|
||||
sub list_phy_repos
|
||||
{
|
||||
my $repos_p = shift;
|
||||
my %repo_patts = ();
|
||||
my @phy_repos;
|
||||
|
||||
wrap_chdir("$ENV{GL_REPO_BASE_ABS}");
|
||||
for my $repo (`find . -type d -name "*.git"`) {
|
||||
chomp ($repo);
|
||||
$repo =~ s(\./(.*)\.git$)($1);
|
||||
# the key has to be in the list, since the repo physically exists
|
||||
# -- my($perm, $creator, $wild) = &repo_rights($repo);
|
||||
# -- $repo_patts{$repo} = $wild || $repo;
|
||||
# turns out we're not using the value anywhere, so no point wasting
|
||||
# all those cycles getting all repos' rights, at least until a real
|
||||
# use for it comes along. But when it does come along, remember that
|
||||
# $wild is now a space separated list of matching patterns (or empty
|
||||
# if no wild patterns matched $repo). It is NOT a single value
|
||||
# anymore!
|
||||
$repo_patts{$repo} = 1;
|
||||
push @phy_repos, $repo;
|
||||
}
|
||||
|
||||
return %repo_patts;
|
||||
return @phy_repos;
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -337,6 +328,7 @@ sub new_repo
|
|||
# really care; we just pull it in once and save it for the rest of
|
||||
# the run
|
||||
do $GL_CONF_COMPILED;
|
||||
add_repo_conf($repo) if $repo;
|
||||
%cached_groups = %groups;
|
||||
$cache_filled++;
|
||||
}
|
||||
|
|
@ -559,8 +551,6 @@ sub parse_acl
|
|||
%repos = %saved_repos; %groups = %saved_groups;
|
||||
} else {
|
||||
die "parse $GL_CONF_COMPILED failed: " . ($! or $@) unless do $GL_CONF_COMPILED;
|
||||
$saved_crwu = "$creator,$perm_cats_sig,$gl_user";
|
||||
%saved_repos = %repos; %saved_groups = %groups;
|
||||
}
|
||||
unless (defined($data_version) and $data_version eq $current_data_version) {
|
||||
# this cannot happen for 'easy-install' cases, by the way...
|
||||
|
|
@ -569,6 +559,9 @@ sub parse_acl
|
|||
|
||||
die "parse $GL_CONF_COMPILED failed: " . ($! or $@) unless do $GL_CONF_COMPILED;
|
||||
}
|
||||
$saved_crwu = "$creator,$perm_cats_sig,$gl_user";
|
||||
%saved_repos = %repos; %saved_groups = %groups;
|
||||
add_repo_conf($repo) if $repo;
|
||||
|
||||
# basic access reporting doesn't send $repo, and doesn't need to; you just
|
||||
# want the config dumped as is, really
|
||||
|
|
@ -607,6 +600,17 @@ sub parse_acl
|
|||
return ($wild);
|
||||
}
|
||||
|
||||
# add repo conf from repo.git/gl-conf
|
||||
sub add_repo_conf
|
||||
{
|
||||
my ($repo) = shift;
|
||||
return unless $split_conf{$repo};
|
||||
do "$ENV{GL_REPO_BASE_ABS}/$repo.git/gl-conf" or return;
|
||||
$repos{$repo} = $one_repo{$repo};
|
||||
$git_configs{$repo} = $one_git_config{$repo};
|
||||
}
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# print a report of $user's basic permissions
|
||||
# ----------------------------------------------------------------------------
|
||||
|
|
@ -643,6 +647,8 @@ sub report_basic
|
|||
local $ENV{GL_USER} = $user;
|
||||
|
||||
&parse_acl($GL_CONF_COMPILED, "", "CREATOR");
|
||||
# all we need is for 'keys %repos' to come up with all the names, so:
|
||||
@repos{ keys %split_conf } = values %split_conf if %split_conf;
|
||||
|
||||
# send back some useful info if no command was given
|
||||
&report_version($GL_ADMINDIR, $user);
|
||||
|
|
|
|||
|
|
@ -38,6 +38,7 @@ our ($R_COMMANDS, $W_COMMANDS, $REPONAME_PATT, $REPOPATT_PATT, $ADC_CMD_ARGS_PAT
|
|||
our %repos;
|
||||
our %groups;
|
||||
our %git_configs;
|
||||
our %split_conf;;
|
||||
|
||||
# the common setup module is in the same directory as this running program is
|
||||
my $bindir = $0;
|
||||
|
|
|
|||
|
|
@ -92,6 +92,9 @@ our %groups = ();
|
|||
# in between :)
|
||||
my %repos = ();
|
||||
|
||||
# repos whose ACLs don't make it into the main compiled config file
|
||||
my %split_conf = ();
|
||||
|
||||
# rule sequence number
|
||||
my $rule_seq = 0;
|
||||
|
||||
|
|
@ -398,26 +401,31 @@ for my $fragment_file (glob("conf/fragments/*.conf"))
|
|||
parse_conf_file($fragment_file, $fragment);
|
||||
}
|
||||
|
||||
my $compiled_fh = wrap_open( ">", "$GL_CONF_COMPILED.new" );
|
||||
my $data_version = $current_data_version;
|
||||
print $compiled_fh Data::Dumper->Dump([$data_version], [qw(*data_version)]);
|
||||
my $dumped_data = Data::Dumper->Dump([\%repos], [qw(*repos)]);
|
||||
$dumped_data .= Data::Dumper->Dump([\%git_configs], [qw(*git_configs)]) if %git_configs;
|
||||
# the dump uses single quotes, but we convert any strings containing $creator
|
||||
# and $gl_user to double quoted strings. A bit sneaky, but not too much...
|
||||
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
|
||||
print $compiled_fh $dumped_data;
|
||||
if (%groups) {
|
||||
sub write_compiled_conf
|
||||
{
|
||||
my $compiled_fh = wrap_open( ">", "$GL_CONF_COMPILED.new" );
|
||||
my $data_version = $current_data_version;
|
||||
print $compiled_fh Data::Dumper->Dump([$data_version], [qw(*data_version)]);
|
||||
my $dumped_data = Data::Dumper->Dump([\%repos], [qw(*repos)]);
|
||||
$dumped_data .= Data::Dumper->Dump([\%git_configs], [qw(*git_configs)]) if %git_configs;
|
||||
# the dump uses single quotes, but we convert any strings containing $creator
|
||||
# and $gl_user to double quoted strings. A bit sneaky, but not too much...
|
||||
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
|
||||
print $compiled_fh $dumped_data;
|
||||
if (%groups) {
|
||||
$dumped_data = Data::Dumper->Dump([\%groups], [qw(*groups)]);
|
||||
$dumped_data =~ s/\bCREAT[EO]R\b/\$creator/g;
|
||||
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
|
||||
print $compiled_fh $dumped_data;
|
||||
}
|
||||
print $compiled_fh Data::Dumper->Dump([\%split_conf], [qw(*split_conf)]) if %split_conf;
|
||||
close $compiled_fh or die "$ABRT close compiled-conf failed: $!\n";
|
||||
rename "$GL_CONF_COMPILED.new", "$GL_CONF_COMPILED";
|
||||
}
|
||||
close $compiled_fh or die "$ABRT close compiled-conf failed: $!\n";
|
||||
rename "$GL_CONF_COMPILED.new", "$GL_CONF_COMPILED";
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# (that ends the config file compiler and write)
|
||||
# (that ends the config file compiler, though we postpone the writing
|
||||
# for now to deal with the latest GL_BIG_CONFIG innovation!)
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
|
|
@ -443,25 +451,31 @@ die "\n\t\t***** AAARGH! *****\n" .
|
|||
"\tthe newer features, please upgrade.\n"
|
||||
if $git_version < 10602; # that's 1.6.2 to you
|
||||
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# the rest of this program can be "switched off"; see doc/big-config.mkd for
|
||||
# details.
|
||||
# most of the rest of this program can be "switched off"; see
|
||||
# doc/big-config.mkd for details.
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# any new repos to be created?
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
# repo-base needs to be an absolute path for this loop to work right
|
||||
# repo-base needs to be an absolute path due to all the jumping around we do,
|
||||
# so if it was not already absolute, prefix $HOME.
|
||||
$ENV{GL_REPO_BASE_ABS} = ( $REPO_BASE =~ m(^/) ? $REPO_BASE : "$ENV{HOME}/$REPO_BASE" );
|
||||
|
||||
unless ($GL_NO_CREATE_REPOS) {
|
||||
# process the normal repos in %repos. This includes creating them if needed
|
||||
# (and GL_NO_CREATE_REPOS is not set), checking hooks, and finally, if
|
||||
# GL_BIG_CONFIG is set, writing out the one-repo config file for directly
|
||||
# specified repos (i.e., "repo foo", not "@grp = foo" + "repo @grp")
|
||||
do_normal_repos();
|
||||
write_compiled_conf(); # write out the final compiled config
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# process the normal repos in %repos (create, hook, one_repo config...)
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
sub do_normal_repos
|
||||
{
|
||||
wrap_chdir("$ENV{GL_REPO_BASE_ABS}");
|
||||
|
||||
# autocreate repos. Start with the ones that are normal repos in %repos
|
||||
# start with the ones that are normal repos in %repos
|
||||
my @repos = grep { $_ =~ $REPONAME_PATT and not /^@/ } sort keys %repos;
|
||||
# then, for each repogroup, find the members of the group and add them in
|
||||
map { push @repos, keys %{ $groups{$_} } } grep { /^@/ } keys %repos;
|
||||
|
|
@ -470,8 +484,10 @@ unless ($GL_NO_CREATE_REPOS) {
|
|||
@repos = sort keys %seen;
|
||||
|
||||
for my $repo (sort @repos) {
|
||||
next unless $repo =~ $REPONAME_PATT;
|
||||
next if $repo =~ m(^\@|EXTCMD/); # these are not real repos
|
||||
next unless $repo =~ $REPONAME_PATT; # skip repo patterns
|
||||
next if $repo =~ m(^\@|EXTCMD/); # skip groups and fake repos
|
||||
|
||||
unless ($GL_NO_CREATE_REPOS) {
|
||||
unless (-d "$repo.git") {
|
||||
print STDERR "creating $repo...\n";
|
||||
new_repo($repo, "$GL_ADMINDIR/hooks/common");
|
||||
|
|
@ -490,19 +506,45 @@ unless ($GL_NO_CREATE_REPOS) {
|
|||
ln_sf("$GL_PACKAGE_HOOKS/common", "*", "$repo.git/hooks") if $GL_PACKAGE_HOOKS;
|
||||
}
|
||||
}
|
||||
|
||||
# write a one_repo config for normal repos declared directly (not just via a group)
|
||||
write_1_compiled_conf($repo) if $GL_BIG_CONFIG and $repos{$repo} and -d "$repo.git";
|
||||
}
|
||||
}
|
||||
|
||||
sub write_1_compiled_conf
|
||||
{
|
||||
# warning: writes and *deletes* it from %repos and %git_configs
|
||||
my ($repo) = shift;
|
||||
my (%one_repo, %one_git_config);
|
||||
|
||||
open(my $compiled_fh, ">", "$repo.git/gl-conf") or return;
|
||||
|
||||
$one_repo{$repo} = $repos{$repo};
|
||||
delete $repos{$repo};
|
||||
my $dumped_data = Data::Dumper->Dump([\%one_repo], [qw(*one_repo)]);
|
||||
|
||||
if ($git_configs{$repo}) {
|
||||
$one_git_config{$repo} = $git_configs{$repo};
|
||||
delete $git_configs{$repo};
|
||||
$dumped_data .= Data::Dumper->Dump([\%one_git_config], [qw(*one_git_config)]);
|
||||
}
|
||||
|
||||
# the dump uses single quotes, but we convert any strings containing $creator
|
||||
# and $gl_user to double quoted strings. A bit sneaky, but not too much...
|
||||
$dumped_data =~ s/'(?=[^']*\$(?:creator|gl_user))~?(.*?)'/"$1"/g;
|
||||
print $compiled_fh $dumped_data;
|
||||
close $compiled_fh;
|
||||
|
||||
$split_conf{$repo} = 1;
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# collect repo_patt for each actual repo
|
||||
# get a list of physical repos for later
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
# go through each actual repo on disk, and match it to either its own name in
|
||||
# the config (non-wild) or a wild pattern that matches it. Lots of things
|
||||
# later will need this correspondence so we may as well snarf it in one shot
|
||||
|
||||
|
||||
my %repo_patts = ();
|
||||
%repo_patts = &collect_repo_patts(\%repos) unless $GL_NO_DAEMON_NO_GITWEB;
|
||||
my @phy_repos = ();
|
||||
@phy_repos = &list_phy_repos() unless $GL_NO_DAEMON_NO_GITWEB;
|
||||
|
||||
# NOTE: we're overloading GL_NO_DAEMON_NO_GITWEB to mean "no git config" also.
|
||||
# In fact anything that requires trawling through the existing repos doing
|
||||
|
|
@ -520,8 +562,6 @@ my %repo_patts = ();
|
|||
# update repo configurations, gitweb description, daemon export-ok, etc
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
# all these require a "chdir" to the repo, so we club them for efficiency
|
||||
|
||||
my %projlist = ();
|
||||
|
||||
# for each real repo (and remember this will be empty, thus skipping all this,
|
||||
|
|
@ -530,13 +570,13 @@ my %projlist = ();
|
|||
# note: we do them in 2 separate loops to avoid breaking the optimisation in
|
||||
# sub parse_acl (look for variable $saved_crwu)
|
||||
|
||||
for my $repo (keys %repo_patts) {
|
||||
for my $repo (@phy_repos) {
|
||||
wrap_chdir("$ENV{GL_REPO_BASE_ABS}/$repo.git");
|
||||
# daemon is easy
|
||||
&setup_daemon_access($repo);
|
||||
}
|
||||
|
||||
for my $repo (keys %repo_patts) {
|
||||
for my $repo (@phy_repos) {
|
||||
wrap_chdir("$ENV{GL_REPO_BASE_ABS}/$repo.git");
|
||||
# gitweb is a little more complicated. Here're some notes:
|
||||
# - "setup_gitweb_access" also sets "owner", despite the name
|
||||
|
|
|
|||
|
|
@ -23,7 +23,21 @@ $data_version = '1.7';
|
|||
'@g1' => 1,
|
||||
'@g2' => 1
|
||||
}
|
||||
},
|
||||
}
|
||||
);
|
||||
%groups = (
|
||||
'@g1' => {
|
||||
'aa' => 'master',
|
||||
'bb' => 'master'
|
||||
}
|
||||
);
|
||||
%split_conf = (
|
||||
'gitolite-admin' => 1,
|
||||
'testing' => 1
|
||||
);
|
||||
repositories/gitolite-admin.git/gl-conf
|
||||
repositories/testing.git/gl-conf
|
||||
%one_repo = (
|
||||
'gitolite-admin' => {
|
||||
'R' => {
|
||||
'tester' => 1
|
||||
|
|
@ -38,7 +52,9 @@ $data_version = '1.7';
|
|||
'RW+'
|
||||
]
|
||||
]
|
||||
},
|
||||
}
|
||||
);
|
||||
%one_repo = (
|
||||
'testing' => {
|
||||
'@all' => [
|
||||
[
|
||||
|
|
@ -55,9 +71,3 @@ $data_version = '1.7';
|
|||
}
|
||||
}
|
||||
);
|
||||
%groups = (
|
||||
'@g1' => {
|
||||
'aa' => 'master',
|
||||
'bb' => 'master'
|
||||
}
|
||||
);
|
||||
|
|
|
|||
|
|
@ -39,8 +39,8 @@ echo "
|
|||
RW = u2 u3
|
||||
" | ugc
|
||||
|
||||
catconf
|
||||
expect_filesame $TESTDIR/out/t01-repo-groups.1
|
||||
catconfs
|
||||
expect_filesame $TESTDIR/out/t01-repo-groups.1bs
|
||||
|
||||
# ----------
|
||||
$TESTDIR/rollback || die "rollback failed"
|
||||
|
|
@ -54,7 +54,7 @@ echo "
|
|||
RW = @g2
|
||||
" | ugc
|
||||
|
||||
catconf
|
||||
catconfs
|
||||
expect_filesame $TESTDIR/out/t01-repo-groups.2
|
||||
|
||||
name INTERNAL
|
||||
|
|
|
|||
|
|
@ -42,8 +42,8 @@ echo "
|
|||
RW = u2 u3
|
||||
" | ugc
|
||||
|
||||
catconf
|
||||
expect_filesame $TESTDIR/out/t02-user-groups.1
|
||||
catconfs
|
||||
expect_filesame $TESTDIR/out/t02-user-groups.1bs
|
||||
|
||||
# ----------
|
||||
$TESTDIR/rollback || die "rollback failed"
|
||||
|
|
@ -60,7 +60,7 @@ echo "
|
|||
RW = @g2
|
||||
" | ugc
|
||||
|
||||
catconf
|
||||
expect_filesame $TESTDIR/out/t02-user-groups.2
|
||||
catconfs
|
||||
expect_filesame $TESTDIR/out/t02-user-groups.2bs
|
||||
|
||||
name INTERNAL
|
||||
|
|
|
|||
|
|
@ -76,15 +76,21 @@ do
|
|||
cd ~/td
|
||||
name "repo on disk missing: u1"
|
||||
runlocal git clone u1:aa
|
||||
expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
|
||||
[ "$bc" = "0" ] && expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
|
||||
[ "$bc" = "1" ] && expect "R access for aa DENIED to u1"
|
||||
[ "$bc" = "1" ] && expect "Or there may be no repository at the given path. Did you spell it correctly?"
|
||||
|
||||
name "repo on disk missing: tester"
|
||||
runlocal git clone gitolite:aa
|
||||
expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
|
||||
[ "$bc" = "0" ] && expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
|
||||
[ "$bc" = "1" ] && expect "R access for aa DENIED to tester"
|
||||
[ "$bc" = "1" ] && expect "Or there may be no repository at the given path. Did you spell it correctly?"
|
||||
|
||||
name "repo on disk missing: u4"
|
||||
runlocal git clone u4:aa
|
||||
expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
|
||||
[ "$bc" = "0" ] && expect "fatal: 'repositories/aa.git' does not appear to be a git repository"
|
||||
[ "$bc" = "1" ] && expect "R access for aa DENIED to u4"
|
||||
[ "$bc" = "1" ] && expect "Or there may be no repository at the given path. Did you spell it correctly?"
|
||||
|
||||
name "repo on disk missing: u6"
|
||||
runlocal git clone u6:aa
|
||||
|
|
|
|||
|
|
@ -14,6 +14,13 @@ runremote() { ssh gitolite-test@localhost "$@" > ~/1 2> ~/2; }
|
|||
listrepos() { ssh gitolite-test@localhost find repositories -type d -name "*.git" | sort > ~/1 2> ~/2; }
|
||||
# remote cat compiled pm
|
||||
catconf() { ssh gitolite-test@localhost cat .gitolite/conf/gitolite.conf-compiled.pm > ~/1 2> ~/2; }
|
||||
catconfs() {
|
||||
(
|
||||
ssh gitolite-test@localhost cat .gitolite/conf/gitolite.conf-compiled.pm
|
||||
ssh gitolite-test@localhost find repositories -name gl-conf \| sort
|
||||
ssh gitolite-test@localhost find repositories -name gl-conf \| sort \| xargs cat
|
||||
) > ~/1 2> ~/2
|
||||
}
|
||||
# remote cat ~/.gitolite.rc
|
||||
catrc() { ssh gitolite-test@localhost cat .gitolite.rc > ~/1 2> ~/2; }
|
||||
# tail gitolite logfile
|
||||
|
|
|
|||
Loading…
Reference in a new issue