How to create a local Android open source project mirror

How to create a local Android open source project mirror

We have two Android phone users in Deveo, me and our DevOps engineer Ahmed. Regardless of our small Android usage percentage, we love to support open source projects, and addition to our love to open source projects, our customers like to develop them even more.

Deveo 3.5.0 introduced the ability to host local Android open source project mirrors. At the time of writing, at least GitHub, Bitbucket nor Gitlab supports the naming convention used in Android Open Source Project repositories. Lack of the support for the naming convention makes the use of the repo tool impossible without modifying the manifest file used by the repo tool, and thus needing some extra hacks here and there.

The purpose of this post is to instruct how to set up a local Android Open source project mirror behind your firewall using Deveo. The same instructions can be applied to any source code hosting platform that supports the naming convention used in Android Open Source Projects repositories. Gitolite supports the naming convention at least, as well as Gerrit naturally.

Before you read any further, this post covers all of the steps in detail how to mirror android open source projects locally. If you are simply looking for a copy-paste solution, you can skip the jargon and jump right into the working example that is hosted in GitHub.

Getting down to business

So you are actually interested in the dirty details, let's carry on then. The process of mirroring android open source projects within your own infrastructure consists of six steps.

  1. Obtaining the necessary information
  2. Creating a Deveo project for our local Android open source project mirror
  3. Creating the repositories to Deveo
  4. Cloning the original repositories from android open source project official repository
  5. Pushing the repositories to the Deveo project
  6. Verifying that everything works

1. Obtain necessary information

In order to set up a local Android open source project mirror using Deveo, we need some information that we will be using later on. The needed information is listed below:

  • Android open source project Git server address
  • Deveo server address
  • Name for the Deveo project we wish to set up the local android open source project mirror
  • your Deveo plugin key
  • Deveo company key of your Deveo instance
  • Your Deveo account key
  • Your Deveo username
  • The company name in Deveo

We want to know the Android open source project git server address, which at the time of writing is android.googlesource.com. We'll naturally also need the address of the Deveo server we are using. If you are using Deveo cloud, that would be app.deveo.com. The Deveo project name will be the name of the project we create to Deveo for the local Android open source project mirror. The three keys, namely plugin key, company key and account key can be all found from user preferences located in the top right of the Deveo web interface. Your Deveo username can be also found from the same view, and the company name is the one you type whenever you log in to Deveo.

Deveo user profile

2. Create the project to Deveo for our mirror

Creating a Deveo project takes just a couple of clicks through the user interface, as explained in an earlier blog post. This time, we will create the project programmatically through Deveo REST APIs. We use the following curl command to create a new Deveo project to Deveo:

curl -X POST \  
  -H "Accept: application/vnd.deveo.v1" \
  -H "Authorization: deveo plugin_key='$PLUGIN_KEY',company_key='$COMPANY_KEY',account_key='$ACCOUNT_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "id": "$DEVEO_PROJECT", "name": "$DEVEO_PROJECT" }' \
  https://$DEVEO_URL/api/projects

Before executing the command, remember to export the Android open source project Git server URL, Deveo plugin key, company key and your account key, as well as the project name and the Deveo URL as shell variables with the following commands (note that the keys in the example below are imaginary, and you need to use the ones you find from your user profile):

export ANDROID_URL=android.googlesource.com  
export PLUGIN_KEY=3a12d59d6220aa024cb45b8c7b82db12  
export COMPANY_KEY=c18c6317641b13223228b1cd7254e9b8  
export ACCOUNT_KEY=462c0f154875824a626c81a26ab3212f  
export DEVEO_URL=app.deveo.com  
export DEVEO_PROJECT=my-android-open-source-project-fork  
export DEVEO_USERNAME=ilmarideveocom  

After executing the command, you may check the results from the web interface. There should be a new project in your Deveo dashboard with the appropriate name.

3. Creating the repositories to Deveo

With Deveo, you can create Git, Subversion and Mercurial repositories with two clicks. In this case, we need a lot of Git repositories. Instead of clicking around for hours, we can create a Git repository to Deveo using the curl snippet below:

curl -X POST \  
  -H "Accept: application/vnd.deveo.v1" \
  -H "Authorization: deveo plugin_key='$PLUGIN_KEY',company_key='$COMPANY_KEY',account_key='$ACCOUNT_KEY'" \
  -H "Content-Type: application/json" \
  -d '{ "id": "platform/manifest", "type": "git" }' \
  https://$DEVEO_URL/api/projects/$DEVEO_PROJECT/repositories

Since it would be tedious to copy-paste the code snippet above over and over for each repository in the manifest.xml file, we created an XSL template that can be used to read and create a file containing one curl snippet for each repository. The XSL template can be found from the working example.

4. Cloning the original repositories from Android open source project official repository

Cloning the repositories from the Android open source project Git server is done using simple git operations. We need to clone the repositories using --bare switch in order to store the repositories within the hierarchical structure in the local filesystem.

In the case of the manifest repository, we would do as follows:

git clone --bare https://$ANDROID_URL/platform/manifest platform/manifest  

We used similar XSL template based approach in the working example as with creating the repositories for going through all repositories under the manifest.xml file.

5. Pushing the repositories to the Deveo project

Pushing the locally cloned repositories to the repositories in Deveo with the same names can be achieved easily. Simply go to the directory of the repository, add a new remote to the repository pointing to the Deveo repository URL, and finally, push the changes using '--mirror' switch. Below is an example of doing this to the manifest repository.

cd platform/manifest  
git remote add deveo https://$DEVEO_USERNAME@$DEVEO_URL/$DEVEO_COMPANY/projects/$DEVEO_PROJECT/repositories/git/platform/manifest  
git push --mirror deveo  
cd -  

For this snippet to work we need to export Deveo username and company name as command line variables. Example below:

export DEVEO_USERNAME=ilmarideveocom  
export DEVEO_COMPANY=deveo  

Like in the previous two steps, our XSL template based approach takes care of the heavy lifting and copy-pasting. You'll find the details in the working example.

6. Testing whether everything works

Given you have all of the repositories pushed to Deveo, you may now test things out following the official Android open source project instructions, but using Deveo as the clone URL. Below is an example repo command to run to initialize a new client from your freshly created local mirror:

repo init -u https://ilmarideveocom@app.deveo.com/deveo/projects/aosp-testi/repositories/git/platform/manifest  

Conclusion

Creating a local Android open source project mirror takes some work. Luckily we created the XSL template to ease the process. Share this post around, if you found it useful.

Seamless software development.

Code management and collaboration platform with Git, Subversion, and Mercurial.

Sign up for free
comments powered by Disqus