Bitbucket clone all team repositories

Question

I'd like to clone all my bitbucket team repositories using a bash script and http. I have found a few examples that use the Bitbucket api however, they all seem to return no repositories. Any ideas? Using mac.

Answer 1

Here is my simple solution. Make file downloader.sh

#!/bin/bash

USER=${1}
TEAM=${2}

rm -rf "$TEAM" && mkdir "$TEAM" && cd $TEAM

NEXT_URL="https://api.bitbucket.org/2.0/repositories/${TEAM}?pagelen=100"

while [ ! -z $NEXT_URL ] && [ $NEXT_URL != "null" ]
do
    curl -u $USER $NEXT_URL > repoinfo.json
    jq -r '.values[] | .links.clone[1].href' repoinfo.json > ../repos.txt
    NEXT_URL=`jq -r '.next' repoinfo.json`

    for repo in `cat ../repos.txt`
    do
        echo "Cloning" $repo
        if echo "$repo" | grep -q ".git"; then
            command="git"
        else
            command="hg"
        fi
        $command clone $repo
    done
done

cd ..

you can run it by:

sh downloader.sh username teamname # or username instead team name

Answer 2

Ensure you have your ssh keys setup in bitbucket and git is installed by cloning one repo manually:

This will clone all repos that are owned by you:

USER=bitbucket_username; curl --user ${USER} https://api.bitbucket.org/2.0/repositories/${USER} | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone

To backup your team repositories use the same script but hard code your bitbucket team name like this:

USER=bitbucket_username; curl --user ${USER} https://api.bitbucket.org/2.0/repositories/TEAMNAME | grep -o '"ssh:[^ ,]\+' | xargs -L1 git clone

Here's a better way:

curl -u ${1} https://api.bitbucket.org/1.0/users/TEAMNAME > repoinfo

for repo_name in `cat repoinfo | sed -r 's/("name": )/\n\1/g' | sed -r 's/"name": "(.*)"/\1/' | sed -e 's/{//' | cut -f1 -d\" | tr '\n' ' '`
do
    echo "Cloning " $repo_name
    git clone git@bitbucket.org:TEAMNAME/$repo_name.git
    echo "---"
done

Answer 3

Another alternative using jq.

#!/bin/bash

user=username:password

curl -u $user 'https://api.bitbucket.org/2.0/user/permissions/teams?pagelen=100' > teams.json

jq -r '.values[] | .team.username' teams.json > teams.txt

for team in `cat teams.txt`
do
  echo $team

  rm -rf "${team}"

  mkdir "${team}"

  cd "${team}"

  url="https://api.bitbucket.org/2.0/repositories/${team}?pagelen=100"

  echo $url

  curl -u $user $url > repoinfo.json

  jq -r '.values[] | .links.clone[0].href' repoinfo.json > repos.txt

  for repo in `cat repos.txt`
  do
    echo "Cloning" $repo
    git clone $repo
  done

  cd ..

done

Answer 4

You can only use a simple command if you have less than 100 repositories and add 'pagelen=100' to the query, since that is the most that the bitbucket API will report at one time. If you have more than 100 repositories, you need to process the "next" link in the JSON returned, to get the URL to query the next set of repositories, which would be easier with a script.

If you use http to clone instead of ssh, then you either need to enter the password for protected repositories, or else get a bitbucket app password and modify the URLs to insert that into them, so they look like:

https://bitbucketuserhere:apppasswordhere@bitbucket.org/teamorusername/repositoryname.git

Additionally, cloning will not get all versions of git LFS files, so be aware of that. According to bitbucket, use 'git lfs fetch --all' to copy all LFS file versions locally.

To get the list of your personal repositories, use a URL like:

https://api.bitbucket.org/2.0/repositories/BITBUCKETUSERNAME?pagelen=100

To get the list of your team repositories, use a URL like this to get the list of all repositories you are a member of:

https://api.bitbucket.org/2.0/repositories/TEAMNAME?pagelen=100&role=member

The following is an example perl script you could use to clone and then maintain copies of your repositories, using http instead of ssh to fetch. It makes --mirror clones instead of a fully populated working copy (perfect for moving or disaster recovery). It does not back up all LFS files.

#!/usr/bin/env perl

use warnings;
use strict;
use JSON::Parse 'parse_json';

# CONFIGURATION:
# Bitbucket team or user name to get list of repositories from
my $teamORuserName = "myteam";

# Bitbucket app password with access to query the API for the
# list of repositories. Format: "user-name:app-token"
my $appPassword= "frank-james:LAYDxtc8H6FGKUZeHEef";

#------------------------------------------------------------------------------

my $nextPageLink = "https://api.bitbucket.org/2.0/repositories/$teamORuserName?pagelen=100&role=member";
while (defined $nextPageLink)
{
    $nextPageLink =~ m/page=(\d+)/;
    print "Fetching page " . ($1 || 1). "\n";
    my $response = `curl -q --silent --request GET --user '$appPassword' '$nextPageLink'`;
    my $json = parse_json($response);
    my $values = $json->{values};

    foreach my $repo (@$values)
    {
        die "'$repo->{name}' is not a 'git' repo: $repo->{scm}" unless $repo->{scm} eq "git";
        my $links = $repo->{links} || die "no links data for '$repo->{name}'";
        my $clones = $links->{clone} || die "no clone data for '$repo->{name}'";
        my $url = $clones->[0]->{href} || die "no clone url found for $repo->{name}";

        # use uuid as directory name, to survive project name changes
        my $uuid = $repo->{uuid}; $uuid =~ s/[\{\}]//g;
        if (not -e $uuid)
        {
            print "cloning '$repo->{name}' into $uuid\n";
            # replace user name with token to avoid password prompts
            $url =~ s|(https?://).+(\@bitbucket.org)|$1$appPassword$2|;
            system("git clone --progress --mirror '$url' $uuid") == 0 or die "clone failed";
            # make a human friendly link to current repository name
            symlink $uuid, $repo->{slug} or warn "symlink failed: $!";
        }
        else
        {
            print "updating '$repo->{name}' in $uuid\n";
            system("cd $uuid && git fetch --all --tags --prune") == 0 or die "fetch failed";
        }
        print "\n";
    }

    $nextPageLink = $json->{next};
}
exit 0;

Answer 5

Building upon @eric-nord's answer, I've replaced sed with jq to get to the repo slug (which I found simpler, considering bitbucket outputs json).

So, if you have fewer than 100 repositories and you're looking to clone the repositories for a specific project, the following shell script should do the trick.

#!/usr/bin/env sh

if [ $# -eq 0 ] || [ "$1" == "-?" ] || [ "$1" == "--help" ] || [ "$1" == "-h" ]
  then
    echo "Usage: `basename $0` <URL> <USERNAME> <PROJECTNAME>"
    echo "e.g. `basename $0` bitbucket_url some_user some_project"
    exit 1
fi

curl -u ${2} https://${1}/rest/api/1.0/projects/${3}/repos?limit=100 > repos.json

for repo_name in `cat repos.json | jq -r '.values[] .slug'`
do
  echo "cloning" $repo_name
  git clone https://${2}@${1}/scm/${3}/$repo_name.git ./${3}/$repo_name
done

Answer 6

Here is a python script for cloning all the team or user's repositories in bitbucket. As the team repositories are usually private, I took care of that while using the bitbucket API. So just enter your bitbucket username, password, and the team's username and it will take care of cloning all the team repository for you.

import subprocess
import json

cmd = "curl -u <bitbucket_username>:<bitbucket_password>  https://api.bitbucket.org/2.0/repositories/<team_name_or_project_name>"
cmd = cmd.split()

while 1:
    from_api = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    from_api = from_api.communicate()
    json_from_api = json.loads(from_api[0])
    for unit_dict in json_from_api["values"]:
        clone_cmd = "git clone " + unit_dict["links"]["clone"][1]["href"]
        clone_cmd = clone_cmd.split()
        clone_out = subprocess.call(clone_cmd, shell=False)
    if "next" not in json_from_api:
        break
    else:
        cmd[-1] = json_from_api["next"]

Answer 7

Here is a node script to download all repositories in a bitbucket account. Please don't forget to add necessary npm packages.

const argv = require('yargs').argv
const request = require('request');
const nodegit = require('nodegit');

let repos = [];
let opts = {
    fetchOpts: {
        callbacks: {
            credentials: function() {
                return nodegit.Cred.userpassPlaintextNew(argv.username, argv.password);
            },
            certificateCheck: function() {
                return 1;
            }
        }
    }
};

function cloneRepository(index) {
    let repo = repos[index];
    console.log('Cloning ' + repo.full_name);
    nodegit.Clone(repo.links.clone[0].href, 'repositories/' + repo.full_name, opts)
    .then(function(repo) {
        if (repos.length - 1 == index) {
            console.log("All repositories cloned");
        } else {
            cloneRepository(index + 1);
        }
    })
    .catch(function(err) {
        if (err) {
            console.log(err);
        }
    });
}

function loadRepositories(url) {
    request.get(url, {
        'auth': {
            'user': argv.username,
            'pass': argv.password
        }
    }, function (err, response, body) {
        if (err) return console.log(err);
        let data = JSON.parse(body);
        for (var i = 0; i < data.values.length; i++) {
            repos.push(data.values[i]);
        }
        if (data.next){
            loadRepositories(data.next);    
        } else if (repos.length > 0) {
            console.log('Started cloning..');
            cloneRepository(0);
        } else {
            console.log("No repositories found");
        }
    });
}

if (argv.username && argv.password) {
    console.log('Loading all repositories..');
    loadRepositories('https://api.bitbucket.org/2.0/repositories/?role=member');
} else {
    console.log('Please specify both the --username and --password options');
}

You can also check out this GitHub repository. Bitbucket Repository Downloader

Answer 8

I built a CLI in rust for concurrent cloning and pulling git repositories (user and project). https://github.com/jensim/bitbucket_server_cli It works with interactive or batch-mode, has shell completion and ability to remember choices in the interactive mode, for a faster user experience.

For a mac user like yourself :

$> brew install jensim/bitbucket_server_cli/bitbucket_server_cli

$> bitbucket_server_cli clone

BitBucket server address: http://localhost
BitBucket username: jensim
✔ BitBucket password · ********
Clone/update all projects yes
Fetching users [00:00:15] [########################################] 2011/2011 (eta:0s)
Fetching projects [00:00:00] [########################################] 35/35 (eta:0s)
Working repos [00:01:07] [########################################] 1337/1337 (eta:0s)

Disclaimer : This is quite experimental, so its mostly used by mac people at the moment, as far as I'm aware.. But there are windows and debian binaries available on Github Releases. Also, this works concurrently, with a configurable number of simultanious http requests/git clone, in order not to kill your CI pipelines

Answer 9

This is what worked for me in Python:

import subprocess
import json
import cookielib
import urllib2
import base64

def _create_opener(proxy=None):
    cj = cookielib.LWPCookieJar()
    cookie_handler = urllib2.HTTPCookieProcessor(cj)
    if proxy:
        proxy_handler = urllib2.ProxyHandler(proxy)
        opener = urllib2.build_opener(cookie_handler, proxy_handler)
    else:
        opener = urllib2.build_opener(cookie_handler)
    return opener

def get_repos(_opener, _auth, user, password, team_username, paging):
    query_url = 'https://bitbucket.org/!api/2.0/repositories/'+team_username+paging
    try:
        req = urllib2.Request(query_url, None, {"Authorization": _auth })
        handler = _opener.open(req)
    except urllib2.HTTPError, e:
        print e.headers
        raise e
    for unit_dict in json.load(handler)["values"]:
        clone_cmd = "git clone " + unit_dict["links"]["clone"][0]["href"].replace('https://'+user,'https://'+password+':'+password)
        clone_cmd = clone_cmd.split()
        clone_out = subprocess.call(clone_cmd, shell=False)
encodedstring = base64.encodestring("%s:%s" % (user, password))[:-1]
_auth = "Basic %s" % encodedstring
_opener = _create_opener()
get_repos(_opener,_auth,'bitbucket-user','bitbucket-password','team-username','pagelen=100&page=1')

code credit to: this and this

Answer 10

Here's a Bash script that downloads via https.

Save to a file download_bitbucket_repos.sh, chmod +x download_bitbucket_repos.sh then ./download_bitbucket_repos.sh.

#!/bin/bash

USER='yourBitBucketUsername' #not email
PASS='yourPassword'
TEAM='teamNameInBitbucket'

curl -u $USER:$PASS https://api.bitbucket.org/1.0/users/$TEAM > repoinfo

for repo_name in `cat repoinfo | sed -r 's/("name": )/\n\1/g' | sed -r 's/"name": "(.*)"/\1/' | sed -e 's/{//' | cut -f1 -d\" | tr '\n' ' '`
do
    echo "Cloning " $repo_name
    git clone https://$USER@bitbucket.org/$TEAM/$repo_name.git
    echo "---"
done

Answer 11

I made a simple script available here: https://github.com/mimo84/cloner It works similarly to the answer from Nikolai, however it uses the api on bitbucket to fetch them. It has jq as dependency as well.

Usage: Download it (maybe doing git clone https://github.com/mimo84/cloner.git ) then

chmod +x ./bitbucket.sh

./bitbucket.sh username:password

It is going to create a new folder called "bitbucket_repos" above the current directory and in there clone all of your repos.

Answer 12

#!/bin/sh


# this script is inspired by the following url
#https://bytefreaks.net/gnulinux/bash/script-to-clone-all-git-repositories-from-all-projects-of-a-privately-hosted-bitbucket-server

project=myproject
stash_url=https://stash.of.my.company:8081
curl -s --location --request GET '$stash_url/8081/rest/api/1.0/projects/$project/repos' \
--header 'Authorization: Basic XXXXXXXXXXXXx' > repoinfo

for repo_name in `cat repoinfo | grep -o '\"ssh:[^ ,]\+' | xargs -L1`
do
    echo "Cloning " $repo_name
    git clone $repo_name
done