From time to time I install a Linux variant on one of my boxes and start fresh, good performance, lean install. I use git for all my projects now so I am building up repositories on remote hosts fast. Wouldn't it be nice to be able to clone them all to the newly installed host? Quite useful for quick migrations, so I wrote a quick bash script that achieves this ...

Assumptions

import all remote repos

  • 1. You have a REMOTE host that holds all repositories in a central place, ~/repositories in this example.
  • 2. I use Git as version control software (I didn't look further yet as it is amazing software!).
  • 3. You can ssh to the remote host. You can set up a key based login in 3 simple steps (on local host: ssh-keygen && ssh-copy-id -i ~/.ssh/id_rsa.pub your-remote-host).
  • 4. You remote repositories are "bare", see how to push code to a remote web server with Git as a reference. I use this technique for each new coding project now, it is highly efficient.
  • 5. My naming convention for remote repositories is project_name.git, located at ~/repositories/project_name.git.
  • 6. I have a CODE directory in my home directory ($HOME) on my LOCAL host. You should change lines 5-9 of the script below with your own settings ...
  • 7. You might want to refine the script if you are going to use it in a more professional (enterprise) environment. For me it fits the purpose for now though.

Code (also available here)

#!/bin/bash
# import multiple remote git repositories to local CODE dir

# settings / change this to your config
remoteHost=example.com
remoteUser=username
remoteDir="~/repositories/"
remoteRepos=$(ssh -l $remoteUser $remoteHost "ls $remoteDir")
localCodeDir="${HOME}/CODE/"

# if no output from the remote ssh cmd, bail out
if [ -z "$remoteRepos" ]; then
    echo "No results from remote repo listing (via SSH)"
    exit
fi

# for each repo found remotely, check if it exists locally
# assumption: name repo = repo.git, to be saved to repo (w/o .git)
# if dir exists, skip, if not, clone the remote git repo into it
for gitRepo in $remoteRepos
do
  localRepoDir=$(echo ${localCodeDir}${gitRepo}|cut -d'.' -f1)
  if [ -d $localRepoDir ]; then 	
		echo -e "Directory $localRepoDir already exits, skipping ...\n"
	else
		cloneCmd="git clone ssh://$remoteUser@$remoteHost/$remoteDir"
		cloneCmd=$cloneCmd"$gitRepo $localRepoDir"
		
		cloneCmdRun=$($cloneCmd 2>&1)

		echo -e "Running: \n$ $cloneCmd"
		echo -e "${cloneCmdRun}\n\n"
	fi
done

Explanation what some concepts mean ...

  • ssh -l user host "remote command" => allows you to run remote commands from your local machine, quite nice for automating tasks over the network.
  • remoteRepos=$(.. command ..) => $() executes the command and returns the output to a new variable called remoteRepos (remoteRepos for assigning values, $remoteRepos to access them).
  • -z "$remoteRepos" => checks for an empty string.
  • cut -d'.' -f1 => takes the .git off (-d'.' = set delimiter to dot (.) , -f1 = field #1).
  • echo -e => the -e flag allows "echo" to handle the new line character (\n).
  • cloneCmd=$cloneCmd"..string.." => concatenation technique to keep lines shorter (I ideally strive to not put more than 80 chars per line).
  • 2>&1 at the end of an Unix command sends the stderr output to stdout, basically I want to send all info to the same output stream. In bigger scripts I usually send it to a stderr directory with "2>stderr/file_unix_timestamp.err" (I got this tip from a more senior programmer; it has saved me time because I get crucial data for debugging!).

Example running this from cli

$ git_clone_all.sh 
Directory /home/bob/CODE/bamboo already exits, skipping ...

Directory /home/bob/CODE/codesnippets already exits, skipping ...

Directory /home/bob/CODE/myreadinglist already exits, skipping ...

Running: 
$ git clone ssh://[email protected]/~/repositories/myreadinglist_v2.git /home/bob/CODE/myreadinglist_v2
Cloning into '/home/bob/CODE/myreadinglist_v2'...


Running: 
$ git clone ssh://[email protected]/~/repositories/portfolio.git /home/bob/CODE/portfolio
Cloning into '/home/bob/CODE/portfolio'...


Running: 
$ git clone ssh://[email protected]/~/repositories/stock.git /home/bob/CODE/stock
Cloning into '/home/bob/CODE/stock'...


Directory /home/bob/CODE/tweetdigest already exits, skipping ...

Bob Belderbos

Software Developer, Pythonista, Data Geek, Student of Life. About me