Sunday, January 7, 2007

Shelves in Subversion

There is a popular feature called "shelves" that was included in Microsoft's Visual Studio Team System. I am fairly certain that VSTS is not the first or only tool to have this feature and in this article I will show you how to get the same feature from Subversion. I think of shelves as essentially creating and using a branch to save some changes you have been working on, but now need to set aside, or shelve, for a while. Here is how Microsoft explains the feature in their documentation, which I found via a search:
Shelve your pending changes when you are not ready to or cannot check in a set of pending changes. There are primarily five shelve scenarios:
  • Interrupt When you have pending changes that are not ready for check in but you need to work on a different task, you can shelve your pending changes to set them aside.

  • Integration When you have pending changes that are not ready for check in but you need to share them with another team member, you can shelve your pending changes and ask your team member to unshelve them.

  • Review When you have pending changes that are ready for check-in and have to be code-reviewed, you can shelve your changes and inform the code reviewer of the shelveset.

  • Backup When you have work in progress that you want to back up, but are not ready to check in, you can shelve your changes to have them preserved on the Team Foundation server.

  • Handoff When you have work in progress that is to be completed by another team member, you can shelve your changes to make a handoff easier.

The branching model in Subversion, with its use of "cheap copies" is well suited to provide similar capabilities and handle all five of these scenarios. In the rest of this article, I will detail two different ways to shelve changes using branches in Subversion.

Quick Method: Create Branch from Working Copy

The first and easiest method is to simply create a branch from your working copy. A lot of Subversion users do not realize that you can do this. I actually use this technique a lot when creating complex tags. Just get your working copy exactly as you want the tag, and then create the tag based on your working copy.

svn copy . url://server/repos/project/tags/tagname -m "Create tagname"
This creates the tag in the repository based on what is present in the working copy. Subversion does not even need to send any file contents to the repository to create the tag, so it still runs very fast.

You can use this same concept to create a branch or shelf with the changes you are working on. Here is a screen shot from Subclipse showing how to do this:

Notice that the option to create the copy from the Working Copy has been selected. That is the key to this method. When the command runs, Subversion creates a copy in the repository based on the base revision of your working copy, however any local changes are also committed as updates. Likewise, if you have svn added or deleted any files/folders, then those are also included and it does this all in a single transaction.

After the command completes, your working copy will still appear to be modified (because the changes were committed to a different branch then the one associated with the working copy). To get your working copy back to a pristine state, before beginning your next task, you will need to run the svn revert command to remove all local modifications. The svn revert command does not remove files and folders that were added, so you need to manually delete those after the revert.

Hint: The Subclipse revert process will remove selected unversioned files and folders. So if you run the revert option twice, the first time you run it, added files are made unversioned and the second time you run it, they are deleted.

Creating a shelf using this technique is very easy. The problem with this technique comes when you want to work on your shelf again. To do so, you want to use the merge command to merge your changes from the shelf back into the current working copy which is associated with trunk or some other branch. The problem is that since the shelf was created in a single transaction, merging it is a little bit tricky. Suppose the shelve was created with revision 50. The normal way you might expect to merge it into your working copy would be to run a command like this:
svn merge -r49:50 url://server/repos/project1/branches/shelf
This tells Subversion to merge the changes that happened with revision 50 into the current working copy. However, in this scenario, when you run this command you will get an error something like this:
svn: Unable to find repository location for 'url://server/repos/project1/branches/shelf' in revision 49
The problem is that the shelf did not exist in revision 49, so the command errors out. The answer to this is that you have to run the merge using two URL's. The source URL would be the base location that your working copy was pointing to (which was the base of the copy) and the target URL would be the shelf. You also need to know what revision your working copy was at when you made the copy. You can determine this information by running the svn log command on the URL of the shelf. The output of the command will show both the URL and revision it was copied from. Unfortunately, this is also where it can get really complicated. There is a better than likely chance that your working copy contains mixed revisions (see my previous post on Mixed Revision Working Copies). Assuming that is the case, when you made the copy from your working copy, you will have gotten a fairly complex transaction in the repository. As an example, this is the output I get from svn log using a fairly simple mixed revision example:
r9 | markphip | 2007-01-08 09:34:06 -0500 (Mon, 08 Jan 2007) | 1 line
Changed paths:
A /branches/fromWC (from /trunk:3)
A /branches/fromWC/.classpath (from /trunk/.classpath:4)
A /branches/fromWC/.project (from /trunk/.project:4)
A /branches/fromWC/src (from /trunk/src:4)
A /branches/fromWC/src/org (from /trunk/src/org:5)
M /branches/fromWC/src/org/tigris/subversion/javahl/
R /branches/fromWC/src/org/tigris/subversion/javahl/
m /trunk/src/org/tigris/subversion/javahl/
R /branches/fromWC/src/org/tigris/subversion/javahl/ (from /trunk/src/org/tigris/subversion/javahl/

As you can see, I have files and folders copied from revisions 3, 4, 5, 7 and 8 and my shelf was created in revision 9. In this example, I know that revision 8 is the right value to use for the source of my copy, but in a more complex example, there may not be a right answer. Let's go back to our example, and just say that the shelf was created in revision 50 and it was created by copying trunk which was at revision 49. Assuming that is the case, then to merge the changes made in the shelf into your working copy, you would need to run a command like this:
svn merge url://server/repos/project1/trunk@49 url://server/repos/project1/branches/shelf@50

This tells Subversion to construct a diff of the changes between trunk @ revision 49 and the shelf @ revision 50 and then apply that diff to the current working copy. When this command runs, your working copy will now contain all of the changes you shelved. You can then go back to working on your change and eventually commit it to trunk or whatever the appropriate branch is to receive the change. Here is a screen shot of the same merge command from Subclipse:

Note that you just have to un-check the box that says to "Use From: URL". This then allows you to enter different "From" and "To" URL's and revisions.

To summarize this method, creating the shelf is easy, but using it again can be difficult. I will now show you what I think is a better, and much cleaner, method for creating shelves.

Better Method: Multiple Steps

There are really two main goals with this method:
  1. Remove the issue of mixed-revision working copies from the equation.
  2. Isolate the changes you want to shelve from the process of creating the shelf so that they are easy to merge later when you want to use them again.
This is always the technique that I use and it is generally much cleaner than the first approach.

The first thing to do is to create the shelf/branch. If you know your working copy is a little old, compared to the HEAD revision of your trunk or branch, then you can use the last revision you committed or updated to when creating the shelf. Otherwise, just use the HEAD revision:
svn copy -r HEAD url://server/repos/project1/trunk url://server/repos/project1/branches/shelf
This command is creating the shelf directly in the repository, based on whatever URL and revision your working copy is associated with. In this example, I used trunk. The next step is to use the svn switch command to switch your working copy so that it is pointing at the shelf URL:
svn switch url://server/repos/project1/branches/shelf

The switch command is also like an update. So if your working copy was not at the same revision you copied when you created the shelf, then it is possible that you will receive some updates, and perhaps even have some conflicts created in your working copy.

Here is a screen shot from Subclipse of the Create Branch dialog. Note the check box on the bottom that lets you create the branch and switch your working copy to it one step:

Once you have dealt with any conflicts that might been created in your working copy from the switch, you just need to commit your changes to the shelf using the normal commit command. When you are done, just use the switch command to switch your working copy back to trunk or wherever you need to be for your next task. Unlike the previous method, you will not need to cleanup your working copy when you do this.

At this point the shelf has been created and your changes have been committed in a way that will make it easy to merge them back later. The only part of this process that is potentially a bit more difficult than the first method is that you might have to resolve some conflicts before you can commit your changes to the shelf, and depending on why you are creating the shelf, you might not have the time to do this. If you create the shelf branch from the revision you have loaded in your working copy, you should minimize any potential for conflicts.

When you need to work on your shelved code again, it is easy. Just use the merge command to merge the changes from your shelf to your current working copy. Suppose you created the shelf with revision 44, spent some time resolving conflicts that were created when you switched, and then commited the changes to the shelf with revision 50. To merge the changes in the shelf to your current working copy, just run this command:
svn merge -r49:50 url://server/repos/project1/branches/shelf

This tells Subversion to merge the changes made in revision 50 to your current working copy (I could have used any value between 44 and 49 for the from revision). Since we separated the process of creating the shelf, from the changes you wanted to store in the shelf, this is an easy command to run. When the command completes your changes will have been merged into your working copy for you to pick up the work where you left off. When you are done, you just commit everything to trunk, or whatever branch is associated with your working copy, and you are done. The branch you created for the shelf can now be deleted from the repository if desired.

I went into a lot of detail in this article but hopefully you now know a lot more about how branching and merging work in Subversion. You can apply this information in your daily work process and use Subversion as a tool that helps you do your work better and more efficiently. I wrote this article under the pretense of the Shelving concept, but it is really just basic branching and merging in the end. I think one of the strengths of the Subversion design is that they kept it all very simple and you just use the basic functionality to build the process you want.


Javier said...

Excellent article, Mark. Thanks!

Unknown said...

I couldn't have found this article at a better time. I was just wondering if there was any way of sharing upcoming changes to one of my APIs without commiting it.

thanks a bunch, also plus points for including the commandline equivalent, as not everybody uses eclipse.

Anonymous said...

Good stuff.

The Darcs vcs has the best implementation of this sort of thing, due to several design choices:

First off, its Distributed - your working copy is a repository (supports pushing to a central repo). This means that you can record patches from the actual files, with out actually giving them to a central repository. This distribution also means that if the central one goes down, its not that big of a deal.

Next, even if you do multiple tasks before doing a record, the commit is interactive, allowing you to choose exactly what changes go in and what does not.

Anonymous said...

good post, though it shows what a lot of work subversion needs. What should be a trivial operation is very difficult. Shelving is essentially the task of storing a patch of uncommitted changes. Very simple concept that should be easy to implement. Once subversion gets merge tracking, maybe this will be easier to do with the first option you indicated.

Subversion also supports what you might call "local shelving" in the form of creating a patch. Unfortunately the unified diff format is inadequate to the task, so once subversion offers their new diff format, whatever it's going to be, then hopefully we'll see this as well.

Subversion is great. It's just amazing what room for growth it still has.

redsolo said...

Nice article! But it was not as easy as I hoped, perhaps the tools fronting subversion (as Subclipse) can help out. It should be much easier, and should be preferably just one user action.

Unknown said...

Great article, thanks for saving me hours! Well written and completely, thanks for taking the time to write a wonderful complete post.

Zachary Young said...

Wonderful post, Mark. Thanks for very much--it's great timing for me as well. I'm trying to move into a role as a Configuration Eng. and CI and Subversion would be some of my key responsibilities.

Dick T. said...

I've found info that makes "svn switch" look rather dodgy for anything but a clean working copy. See, for example. Have you run into any problems with "svn switch" for shelving?

Anonymous said...

I just posted in my small utility to do SVN shelves with one click. Check it out: