How to clone all projects of a group at once in GitLab?

In my GitLab repository, I have a group with 20 projects. I want to clone all projects at once. Is that possible?

6 Answers
6

Not really, unless:

you have a 21st project which references the other 20 as submodules.
(in which case a clone followed by a git submodule update --init would be enough to get all 20 projects cloned and checked out)

git submodule update --init

or you somehow list the projects you have access (GitLab API for projects), and loop on that result to clone each one (meaning that can be scripted, and then executed as "one" command)

Here's an example in Python 3:

from urllib.request import urlopen
import json
import subprocess, shlex

allProjects = urlopen("http://[yourServer:port]/api/v3/projects/all?private_token=[yourPrivateTokenFromUserProfile]")
allProjectsDict = json.loads(allProjects.read().decode())
for thisProject in allProjectsDict:
try:
thisProjectURL = thisProject['ssh_url_to_repo']
command = shlex.split('git clone %s' % thisProjectURL)
resultCode = subprocess.Popen(command)

except Exception as e:
print("Error on %s: %s" % (thisProjectURL, e.strerror))


urls = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*(),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+.git', allProjects.read().decode("latin-1"))

I built a script (curl, git, jq required) just for that. We use it and it works just fine: https://gist.github.com/JonasGroeger/1b5155e461036b557d0fb4b3307e1e75

To find out your namespace, its best to check the API quick:

curl "https://domain.com/api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN"


There, use "namespace.name" as NAMESPACE for your group.

NAMESPACE

The script essentially does:

PROJECT_SEARCH_PARAM

Get their path and ssh_url_to_repo

path

ssh_url_to_repo

2.1. If the directory path exists, cd into it and call git pull

path

git pull

2.2. If the directory path does not exist, call git clone

path

git clone

There is a tool called myrepos, which manages multiple version controls repositories. Updating all repositories simply requires one command:

mr update


In order to register all gitlab projects to mr, here is a small python script. It requires the package python-gitlab installed:

import os
from subprocess import call
from gitlab import Gitlab

# Register a connection to a gitlab instance, using its URL and a user private token
gl = Gitlab('http://192.168.123.107', 'JVNSESs8EwWRx5yDxM5q')
groupsToSkip = ['aGroupYouDontWantToBeAdded']

gl.auth() # Connect to get the current user

gitBasePathRelative = "git/"
gitBasePathRelativeAbsolut = os.path.expanduser("~/" + gitBasePathRelative)
os.makedirs(gitBasePathRelativeAbsolut,exist_ok=True)

for p in gl.Project():
if not any(p.namespace.path in s for s in groupsToSkip):
pathToFolder = gitBasePathRelative + p.namespace.name + "/" + p.name
commandArray = ["mr", "config", pathToFolder, "checkout=git clone '" + p.ssh_url_to_repo + "' '" + p.name + "'"]
call(commandArray)

os.chdir(gitBasePathRelativeAbsolut)

call(["mr", "update"])


Here is another example of a bash script to clone all the repos in a group. The only dependency you need to install is jq (https://stedolan.github.io/jq/). Simply place the script into the directory you want to clone your projects into. Then run it as follows:

./myscript <group name> <private token> <gitlab url>


i.e.

./myscript group1 abc123tyn234 http://yourserver.git.com

./myscript group1 abc123tyn234 http://yourserver.git.com

Script:

#!/bin/bash
if command -v jq >/dev/null 2>&1; then
echo "jq parser found";
else
echo "this script requires the 'jq' json parser (https://stedolan.github.io/jq/).";
exit 1;
fi

if [ -z "$1" ]
then
echo "a group name arg is required"
exit 1;
fi

if [ -z "$2" ]
then
echo "an auth token arg is required. See $3/profile/account"
exit 1;
fi

if [ -z "$3" ]
then
echo "a gitlab URL is required."
exit 1;
fi

TOKEN="$2";
URL="$3/api/v3"
PREFIX="ssh_url_to_repo";

echo "Cloning all git projects in group $1";

GROUP_ID=$(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups?search=$1 | jq '..id')
echo "group id was $GROUP_ID";
curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$GROUP_ID/projects?per_page=100 | jq --arg p "$PREFIX" '. | .[$p]' | xargs -L1 git clone


You can refer to this ruby script here:
https://gist.github.com/thegauraw/da2a3429f19f603cf1c9b3b09553728b

But you need to make sure that you have the link to the organization gitlab url (which looks like: https://gitlab.example.com/api/v3/ for example organization) and private token (which looks like: QALWKQFAGZDWQYDGHADS and you can get in: https://gitlab.example.com/profile/account once you are logged in). Also do make sure that you have httparty gem installed or gem install httparty

gem install httparty

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.