How to manage large binary files in Git with Git LFS and Artifactory

How to manage large binary files in Git with Git LFS and Artifactory

We wrote earlier about storing large binary files in git repositories. Among one of the solutions was Git Large File Storage, more commonly known as Git LFS. The core idea behind Git LFS is that instead of writing large blobs to a Git repository, only a pointer file is written. The blobs are written to a separate server using the Git LFS HTTP API.

At the time of writing, JFrog Artifactory has the only production-ready implementation of Git LFS server that can be hosted on your own infrastructure. The solution Artifactory offers fits and combines perfectly with Deveo, that is on-premises code hosting and collaboration platform. Github also has support for Git LFS, however, it's no panacea for Open Source — quite the opposite

In this blog post, we are going to go through how to setup a Git repository in Deveo, a Git LFS repository in Artifactory, and how to configure the created Git repository to use the Git LFS Repository. The motivation behind this is to show that it is possible to manage Git repositories with large binary files behind your corporate firewall with Deveo and Artifactory. Git LFS can be used with any Git repository hosting solution.

Creating a Git repository in Deveo

Deveo dashboard

In Deveo all repositories belong to a certain project. For this purpose, we can create a new project from the dashboard. After creating the project, we can create a repository under it through repositories view. The empty repository shows the clone URL as well as instructions to set things up with Git client.

Deveo repository details

Creating a Git LFS repository in Artifactory

After cloning the repository, we can go to Artifactory and create a Git LFS repository. Creating a new repository in Artifactory is easy through its admin UI. The repository is created by going to Admin after which the Local repositories view is shown.

Artifactory Local repositories view

For Git LFS repositories, the only configurable parameter that is needed is the name of the repository. After the repository is created, it should be visible in Artifact Repository Browser.

Artifactory Artifact Repository browser

The "Set Me Up" link brings up the necessary settings we need to add to our Git configuration. However, first we need to install Git LFS client.

Setting up Git LFS client

Git LFS client can be downloaded from the official Git LFS homepage. The installation for OSX is simple. Download the package, run the install.sh shell script and you are all set. No administrator rights are required, which is a plus.

I assume readers of this article already know how to clone a Git repository, so we can move straight into setting up the Git LFS Specific settings to .gitconfig. Open your .gitconfig file with your favorite text editor, as well as open the "Set Me Up" link from the Git LFS repository properties in Artifactory.

Artifactory Git LFS repository set me up view

Copy the configuration from "Set Me Up" to your .gitconfig after which you are all set to test things out. I did a small test by first adding a simple README.md file to the Git repository and then enabling Git LFS for the repository. Given that the Git LFS client is installed, it can be enabled for a repository using the git-lfs init command. The command will make changes to a pre-push hook that you can see by looking at .git/hooks/pre-push.

Adding a file to be tracked by Git LFS happens using git-lfs track filename. You can also use patterns, such as *.psd, but in this case, it's sufficient to explicitly state the file we want to be pushed to the Git LFS repository.

When you do a push that contains new files or changes in files that are tracked by Git LFS, it requires you to authenticate against Artifactory. With current Git LFS version (1.0.2), in case you have enabled anonymous access, you need to skip the authentication questions with empty values. Git LFS can also utilize Git credential helpers, however, do note that Git stores credentials in plain text by default.

If we look at the contents of the file in Deveo browser after a push, we can see that it doesn't have the actual file content, but simply Git LFS specific metadata.

Git LFS metadate in Deveo file browser

The actual file content is stored to Artifactory and we can even download the file through the link provided by Artifactory. The file in Artifactory can be found with the SHA value provided in the Git repository.

Conclusion

As the aforementioned instructions show, Artifactory and any Git repository can be used together to set up production grade Git LFS setup behind your company firewall. With Deveo project hook (a hook that is triggered when a repository is created), it would be possible to automatically create Git LFS repositories for each created repository to Deveo. However, this is something to be studied further.

If you found these instructions useful, do share them around and let us know.

Seamless software development.

Code management and collaboration platform with Git, Subversion, and Mercurial.

Sign up for free
comments powered by Disqus