Open Letter to WANDisco

Shortly before we all went away on our Christmas holiday, one of the companies that sponsors developers in the Subversion community, WANDisco, delivered a big fu#k you to the rest of the community in the form of a press release and blog post from their CEO Dave Richards.

I commend my fellow members of the Subversion PMC for being able to take a deep breath and wait a couple weeks to respond with a cool head. Had it been up to me alone, I am sure we would have said something that felt good at the time but that we regret later. You can read the official response here on the Apache Software Foundation blog:

Apache Subversion to WANdisco: +1 on the code contributions, -1 on the attitude

I was, and am, deeply offended by Dave Richards and WANDisco in general. Their business model seems to be to issue press releases rather than actually doing stuff. In my experience, companies that choose to issue press releases BEFORE they start working on something are usually to be ignored and I think that is the case here too. The difference is that WANDisco is attempting to portray themselves as leading the Subversion community and as such that they are speaking for the community. As the blog post from the Subversion PMC illustrates, they neither lead the community nor are the welcome to speak for it.

That said, I get it. No one knows or cares who WANDisco is and by issuing press releases and generating controversy a few more people will now have heard of you. Congratulations. Bully for you. However, in the process your actions are only damaging the product and community you claim to care so deeply about. This reveals your true motives.

If you are so desperate for attention that you feel the need to issue press releases we probably cannot stop you. If you absolutely have to to issue a press release to try to garner some attention, then why could you not simply issue something that says "Hey we think features X, Y and Z are important and we wanted to let you know that we intend to direct our resources to work on those features." I could live with that even though I would rather see you work on the features before you crow about it. As it stands, just as you did a year ago with the Obliterate feature, you are just setting your people up for failure. You have declared that you are going to implement new features that the Subversion committers that work for you already know cannot be solved in the near term. You have brought no new ideas to the table nor any idea how any of the known obstacles that have blocked these features will be overcome. To top it off you have attempted to slap a release date on it. Good luck with that.

I really hope that we implement some of the features in your list and it would be even better if we can get some of them done in 2011. Most of your list was already on the roadmap we published last year. Unfortunately, the way you are going about this is not going to help any of that happen and if we do have some success it will likely be in spite of your efforts not because of them.

Introducing the CollabNet Merge Client

I have not posted to this blog in quite a while. I have been thinking about using it to blog some personal thoughts (like on the Red Sox) but I know it is syndicated in a few places for its Subversion/Subclipse content so I haven't. Instead, I have been doing all of my Subversion blogging over on the Submerged blog at openCollabNet.

It has been a busy summer. I have been more involved in the Subversion community than at any time in the past. Earlier in the summer I was made a full committer which was a proud moment and I have been doing a lot of work to help drive us to the final 1.5 release. Beyond all of that, what I have really been working on the most, however, is the development of a new GUI merge client for Subversion to coincide with the Subversion 1.5 release. Yesterday, we officially announced the client and released an early version. You can read the blog post here, or skip right to the details here.

The CollabNet Merge Client is built on top of Eclipse and Subclipse and has the simple goal of making merge easier. I think it does that and a lot more. It has really turned out great. There are details on the web site, so I will not reiterate them here.

As part of developing this client, we have also made a lot of really nice enhancements to Subclipse. We have focused on taking advantage of all of the great new features that are coming in Subversion 1.5 and also on general usability. I think the final result is going to be great.

The Subversion community is working hard to get the Subversion 1.5 release process started by mid-October so that it can be GA in December. The GA of Subclipse 1.4 and the CollabNet Merge Client will be at the same time.

If you use Subversion and Eclipse, even if you do not use Eclipse, I would encourage you to check this out and give it a try. It requires using the latest build of Subversion 1.5 on the client, but that is not as risky as it sounds, and we will be refreshing the client with new builds as needed. We are also providing Linux JavaHL binaries to make it easier for Linux users to try. Currently only OSX users are shut out, which is ironic because I do all my development on OSX. We could probably make a 1.5-trunk installer for OSX but it would take some work to make it co-exist with the existing 1.4.4 client for OSX. I know Jeremy has been busy so I have not asked him to do the work required. At worst this problem will go away when 1.5 is GA, but hopefully we will have something available before then.

CollabNet Desktop - Eclipse Edition

There was one other aspect to this, yet another thing I have not blogged about. That is the CollabNet Desktop - Eclipse Edition. This is a unified set of tools to access all CollabNet products from Eclipse. We include Subclipse for accessing CollabNet Subversion repositories. There is also a small UI layer on top of this to integrate into the CollabNet perspective and sites view.

It also includes connectors for Mylyn to integrate the issue trackers available in CollabNet Enterprise Edition (CEE) into Eclipse. There are a lot of popular open source sites that use CEE such as tigris.org, java.net, openoffice.org, netbeans.org, dev2dev.bea.com etc. Now the issue trackers for all of these sites are available in Eclipse and Mylyn.

We are also currently working on a Mylyn connector for the issue tracker in SourceForge Enterprise Edition, which is now a CollabNet product. Note: This is not the same product that runs on sourceforge.net.

Last but not least, we also provide integration and access to CollabNet CUBiT. This is a really cool product and the integration came out great. CUBiT allows you to virtualize your build and test environments. From Eclipse, I can provision a server in our CUBiT environment, this includes picking the operating system to run and possibly a set of other components I want to install such as MySQL, Apache etc. These servers are accessible via SSH, and the CollabNet Desktop client makes it simple to access the servers and setup port forwarding to access services running on the servers etc. We integrated the SSH Terminal from the Eclipse DSDP/TM project to provide terminal access. We also support launching external clients but none are required.

I use CUBiT to dynamically provision servers to test Subversion releases (which I have to test on Windows, Linux and Solaris), but most people are using it to build, run and debug server-based web applications. Anyway, the CollabNet Desktop was just another of the many things I have been working on this summer and it went GA a few weeks ago.

It has been a busy summer.

Unstoppable Subversion

Lots of Subversion news this week as the popularity and adoption continues to soar.

First, Forrester Research released their latest Forrester Wave where they analyzed the Software Configuration Management marketplace. For the first time, they included Subversion in their analysis, and what do you know, but Subversion actually came out on top for standalone SCM! They also analyzed integrated change management systems and included Subversion. It did surprisingly well considering that it really does not specifically address that space. You can download a copy of the report from CollabNet.

Next, the latest SecuritySpace.com numbers came out for the month of May and Subversion adoption continues to soar. See this chart for more information.

Finally, I participated in a webinar this week with Carey Schwaber of Forrester and my CollabNet colleague Auke Jilderda. You can view a replay here. This was my first time doing a webinar. I will probably try to make a post about the experience. It was a lot of work for 10 minutes and of course I screwed up the slides when control was transitioned to me. That really flustered me but I recovered. I just got up the courage to watch the replay this AM and it was a lot better than I thought, I actually came in at exactly 10 minutes. I am not sure how eager I will be to do another one anytime soon though because it just took up too much time to prepare. It is hard to do these well if you are not a pro speaker because you are so constrained for time. Anyway, the first 20 minutes of the webinar is Carey going over her report, the criteria used and an overview of SCM for distributed development teams. I then gave a quick overview of Subversion and why it is popular before covering the upcoming 1.5 release and some of its features. Auke then finished things off with an overview of the merge tracking feature which is the defining feature for 1.5. We then concluded with a fairly lengthy Q&A session. There were over 500 people connected which I understand to be a very large audience for one of these events. Clearly interest in Subversion is skyrocketing.

Subversion binaries for OS X

I am pleased to report that we are now providing OS X binaries (including JavaHL) on openCollabNet. This is an issue I raised before and soon after beginning employment at CollabNet, so I am glad to see it become a reality. Lack of a complete binary for OS X (including the JavaHL library) was a bit of a problem since Subversion 1.4 was released. There were some command line packages available, but nothing with JavaHL. I know this caused pain for a number of Subclipse users.

Thanks goes to Jeremy Whitlock for investing his own time and effort to get a Universal Build created, as well as learning the ins and outs of Mac packaging. I have been running this package on my Mac (which I now work on most of the time) for several weeks.

The other part of this news is that we are going to launch a project on openCollabNet to be the home for "community-driven" binaries. Basically, we can provide the bandwidth to host the downloads as well as community forums to solicit feedback etc. CollabNet does not have the resources to officially support every operating system users might be using, but this gives us a way to assist the community in providing quality binaries and also can allow us to gauge interest in the operating systems we are not officially supporting. I will follow-up to this post when the project launches.

In the mean time, I hope all you Mac users out there will download these binaries and try them out. You can post any questions or issues in the Subversion users forum on openCollabNet.

Subversion blog posts to read

I have been real busy with the new job and have not had time to blog. When I have had some time, it went into blogging over on the Submerged blog. Anyway, here are some links to some good blog posts I have seen recently about Subversion.

On the Submerged blog:

Computing the Differences Between Tags by yours truly :)
In this post I show the new svn diff --summarize command and how you can use it to get a list of files that changed between two repository locations.

Authz and Anon Authn Agony by Mike Pilato
In this post Mike explains a difficult problem you can run into when you want to provide anonymous access but also have private folders within the repository structure.

Multiple Subversion Repositories? by Guido Haarmans
This post touches on the issue of having multiple replicated repositories to support globally distributed development. More specifically, why you may not really need this with Subversion as you do with other tools I will not mention.

Subversion LDAP Authentication with Apache by Jeremy Whitlock
Jeremy gives a primer on how to configure Apache to authenticate users via an LDAP directory, such as Microsoft Active Directory.

How Subversion Conserves Disk Space by Guido Haarmans
This is a high-level overview of how the Subversion repository stores your data.

Finally, I have been meaning to link to another set of posts I have looked at. On his blog TrajicklyHip, Aaron West has written a whole series of detailed posts about setting up a Subversion environment and then working with various clients. He has good links within the posts themselves, so I will just link to my favorite, the one about Subclipse. :) I am fairly experienced with Subversion and these posts are aimed at getting people started, so I cannot say that I have poured over every word in detail. That being said, it all looks quite well done and what I read was accurate. I would recommend you check it out.

First blog post and article on openCollabNet

I have agreed to participate in the Submerged blog on openCollabNet and made my first post recently. I guess they did not read this blog too closely and notice my penchant for really long posts. Anyway, we decided to make the post into an article instead. The article is about installing the svnserve server option to run as a Windows service and you can find it here.

My first blog post then just became an introduction to the article. You can find that post here. If you have any comments on the article, you can leave them on the blog post. (Yes, that is four links to the same post if you are counting).

In the future, I will probably do more of my Subversion-specific posts on the Submerged blog and do the Subclipse posts here. I will figure it out as I have things I want to write about.

Happy Birthday Subversion

A belated Happy 3rd Birthday to Subversion and those that made it possible. One of the developer's, Mike Pilato, summarizes the history best in this blog entry.

I knew about the birthday from some entries to the Subversion mailing list, but I did not know about Mike's blog entry until I saw a link to it on the OpenCollabNet site.

openCollabNet

Just a quick post to point you to a site that you might not know about: openCollabNet. This is a site being hosted and supported by CollabNet to provide an online community for Subversion and CollabNet users.

It is a pretty cool site that seems to have a lot of information contained within it such as articles, forums, podcasts etc. I will probably start compiling a list of interesting links and add them to a new post in the future, but for now I would suggest you poke around the various parts of the site and you will likely find something useful.

One of the more immediate things you might find interesting on this site is that they are providing binaries for various Subversion components and tools (such as TortoiseSVN and Subclipse) and I believe they will provide support for those binaries. Currently, I think they are only offering Windows binaries. I would really like to see them provide some for OS X as there currently are not any good sources for up to date binaries for OS X. Metissian used to provide a very nice package, but they seem to have stopped providing them. Not many user's are going to want to download and install XCode so that they can build Subversion from source, and many that do seem to run into problems with various parts of the process -- at least those that are building JavaHL. Anyway, hopefully they have OS X support on their to-do list. Linux binaries would also be nice, but at the same time I think the Linux distributions are doing a decent job providing Subversion packages that work properly, so it is slightly less urgent when compared to OS X.

How to Undo a Commit in Subversion

A frequent question that is asked on the Subversion and Subclipse mailing lists is how to undo a change once it has been committed. A variant of the same request is how to get back a file that has been deleted.

If you need to undo a change completely, as in remove all traces it ever happened, there is really only one way to do it. You need to dump the repository up to the revision you want to remove, and then recreate and load the repository from the dump file. I will not be covering that option in any detail in this article. This article will just be about reversing the effects of a commit, as in restoring a deleted file/folder or some change to a file or set of files. Depending on the circumstances, there might be several ways you can accomplish this. For example, the svn copy command can be used to restore something that is deleted. The technique I am going to focus on in this post is called a "reverse merge". Essentially, you use the svn merge command with the revisions flipped around so that the merge command undoes the changes in the selected revisions. For command-line users, the process to do this is explained well in the Subversion book. See this section on undoing changes.

When using a GUI tool like Subclipse, you could just use the Merge option and fill out the dialog using similar values to what you would do from the command line. However, Subclipse offers a built-in technique that makes it very easy to undo a change.

To give proper credit where it is due, the technique that is described in this article first appeared in the TortoiseSVN product. The Subclipse feature described below is based on the TortoiseSVN feature.

Open the History View
The first step is to show the history of the item containing the change you want to restore. If you just want to restore the changes to one file, then select just that file. Otherwise, you can select your project or a folder so that you can undo changes to multiple files.

If you want to restore something that is deleted, then you need to show the history of the item's parent, so that the history will show you the deletion.

Select the Revision(s)
Now, just select the revision, or contiguous range of revisions, that contain the changes you want to undo. Right-click and choose the option to revert the changes from those revisions:

Review and Commit
When the option runs, it will invoke the merge command, supplying the proper parameters to do a reverse merge. Review the results of the merge and commit the changes to finish the process.

Partial Undo?
It would not be uncommon to only want to reverse part of the changes from a commit or series of commits. There are a couple of ways to deal with this:

1. Select the correct context. If you only want to undo the changes to files in a specific folder, then select that folder when you show the history. Then the merge will only undo the changes in the context of that folder and other changes in the commit will be skipped by the merge process.
2. Revert what you do not want. Suppose you delete five files from the same folder and want to restore one of them. You have to select the parent folder when you take the option, and this will restore all five files. When the command completes, just use the Revert option to undo any changes you do not want, and then commit the rest. The merge process only restores the change in your working copy. Nothing is forcing you to commit the results of the merge without editing it first.
   

Conclusion
Undoing the effects of a change in Subversion is a fairly easy process once you understand what it is that you have to do. The Subclipse and TortoiseSVN UI further simplifies the process and saves you from having to lookup and enter the revisions correctly in the merge dialog.

Integrating Subversion with your Issue Tracking System

This article will explore a technique for integrating Subversion development with your issue tracking system. This technique was cooperatively developed by the TortoiseSVN and Subclipse development teams, and both products support the features that will be described.

Background
Many popular issue tracking systems have developed procedures for integrating with version control tools like CVS and Subversion. A common approach that is used is for the issue tracker to provide one or more Subversion hook scripts that interrogate the message provided in the commit process, looking for references to issues in the message text. Depending on the tool being used, there will often be some meta data stored within the issue tracker to record the files that were modified for that issue. Some tools support features that change the state of the issue, others might perform some kind of validation and possibly reject the commit transaction. This part is really up to the issue tracker and the hook script(s) they provide.

The problem with this approach is that it only works as well as the ability of the issue tracker's hook script to correctly identify the references to issues in the commit message text. With no help being provided by the version control tool to help the user consistently format their message, this can be a difficult task to perform accurately. Also, even when everything works correctly, the integration is only in one place, the issue tracker. In other words, you can view an issue in the issue tracker and see what files were modified (or some other reference to the version control tool), but when looking at history in the version control tool, you do not have easy access to the related issues that were addressed by the commit.

With all this in mind, these were the goals of the issue tracking specification that we developed:

1. Provide a place in the UI of the commit dialog for the user to supply issue ID's. These issues are then inserted into the commit message in a consistent manner.
   
   



2. Provide for the possibility of some simple validations in the UI. Such as verifying that the issue is a number, or warning if no issue is entered.
   



3. When showing Subversion history in the UI, convert any references to issues in the commit message into clickable hyperlinks to the issue tracker.
   
   

You can view the full specification in the TortoiseSVN Subversion repository here. (username = guest with no password)

These features combine to give the user a better experience in terms of referencing their issue tracker from their Subversion client and also makes it much easier to write server side hook scripts that can accurately parse the commit messages. A nice aspect of this feature is that since the integration between Subversion and your issue tracker is still ultimately based on the text of the commit message, even users that are using a client that does not specifically support this feature can still participate in the integration by entering their commit messages in a format consistent with what this feature would have done for them.

Details
With all this in mind, how do you turn on this feature and/or configure it? The way that we settled on was to use Subversion properties to hold the configuration information. This allows one person to set it all up, and the configuration can then be pushed out to other users via the update and/or checkout process. In addition, the configuration just needs to be set on the top-level folder in the working copy. At runtime, the client will walk the working copy back to the root looking for the configuration. This means if you just set it on the root folder, that is enough. Keep in mind, that the client is walking the "working copy" not the "repository". If you have an environment where someone might conceivably check out at any folder level in the repository, then you might need to store this configuration as properties on every folder. Subclipse users are working in Eclipse, and Eclipse has a fairly rigid project structure. So for Eclipse users, it is pretty easy to configure as you just need to put the configuration on the project folder and you can be fairly confident that is what your users are checking out.

Once that is decided, there are then several properties all prefixed with "bugtraq" which determines the configuration of the feature. We chose bugtraq with the slightly odd spelling just to avoid any potential current or future namespace conflicts. Here are the properties and an explanation of each, this is actually copied from the spec verbatim:

NOTE: The specification has evolved a bit since it was first written and now breaks the feature down by a "basic" and "advanced" configuration where the latter is based on the use of regular expressions. Subclipse currently only supports the basic configuration options, and that is also all that this article covers.

Bugtraq Properties:

name : bugtraq:url
value : (string)
This value is the URL pointing to the bug tracking tool. The URL string should contain the substring "%BUGID%" which the client replaces with the issue number. That way the resulting URL will point directly to the correct issue.

NOTE: The URL must be properly URI encoded by the user. This URL can be used by clients to create a link to the bug tracking tool when showing the log message of a revision.

name : bugtraq:warnifnoissue
value : "true"/"yes" or "false"/"no"
If set to "true", then the clients will warn the user if the issue text box is left empty. But it must not block the commit, only give the user a chance to enter an issue number in case (s)he forgot it.

name : bugtraq:label
value : (string)
This can be used by a client as a GUI label describing the text box where the user has to enter the issue number. If this is not set, then a default value should be used, e.g. "Bug-ID / Issue-Number :". Keep in mind though that most GUI clients don't resize their windows according to the size of GUI elements. So keep the size of the label string below 20-25 chars.

name : bugtraq:message
value : (string)
If this property is set, then the client should show a text box in the commit window where the user can enter an issue number. The value string of this property is used by the client to create an additional line added to the log message. The string must contain the substring "%BUGID%"
which is replaced with the issue number the user entered before applied to the log message. The client will add the resulting string as a new line to the end of the log message the user entered:
logmessage + "\n" + resultstring

In case bugtraq:append is set to "false", then the log message is defined as resultstring + "\n" + logmessage

The client should make sure that the string doesn't have multiple lines. If more than one issue number is entered by the user (separated by commas), the client must make sure that there's no whitespace chars before and after the comma. And also the whole issue number string must be trimmed.

name : bugtraq:number
value : "true" or "false"
If this property is set to "false", then the client allows any character entered in the issue text box. Any other value or if the property is missing then only numbers are allowed as the issue number. One exception is the ',' char, which is used to separate multiple issues. The client must never complain if the text box is left empty, i.e. if no issue number is given. Not all commits are assigned to an issue!

name : bugtraq:append
value : "true" or "false"
If set to "false", then the bugtraq:message part is inserted at the top of the log message, if "yes" or not set, then it's appended to the log message at the bottom.

Since these are all Subversion properties, you need to use the svn propset command to set them on a folder within your working copy (there is no way to do it directly against the repository).

svn propset bugtraq:append false .
property 'bugtraq:append' set on '.'

You then need to commit those changes so that other users can get them. Keep in mind that in order to commit property changes to a folder, the revision of the folder in your working copy has to be the same as the HEAD revision of the folder in the repository.

Here is an example of the properties set on one of the Subclipse projects as an example:

If you refer to the previous screen shots, you can get an idea how these properties manifested in the UI of the commit dialog and the text that was inserted into the commit message.

Conclusion
This is a feature in Subclipse and TortoiseSVN that our users lobbied hard for and seem to love. I personally enjoy the easy access to issues that is created by the hyperlinks in commit messages when viewing history. Often reading the history contained within an issue is the best way to understand the context of why a particular change was made.

I do not know the complete list of Subversion clients that have added support for these properties, but I know that it goes beyond Subclipse and TortoiseSVN and includes several of the web-based repository browsers.

The Subclipse integration with Mylar includes partial support for this feature as well. When browsing history within Subclipse, the Mylar "Open Corresponding Task" option will be present and will determine the issue based on these properties and the content of the commit message. I assume this would mean if you were using Bugzilla or another issue tracker with a rich editor in Mylar that the rich editor would be opened in place of the web browser. Unfortunately the Mylar Change Set integration in Subclipse does not currently have a way to plug your issue number into the right part of the commit dialog. It does stick the URL of the issue in the commit message text, so all is not lost. Perhaps in the future it will be possible to enhance this so that the issue ID itself is populated.

Subversion 1.5: Tolerate obstructions during checkout/update/switch

This is my second article about new features in Subversion's trunk, features that will be included in Subversion 1.5. My previous article discussed enhancements in Subversion's move/copy commands. This article will discuss improvements that have been made to the checkout, update and switch commands -- specifically, the ability to tolerate obstructions.

Prior to these changes, if Subversion needed to add a file or folder to your working copy as part of the checkout/update/switch commands, and a file or folder with the same name already existed, then the command would abort with an error. The message you receive will vary based on the command and the situation, but it will look something like this:

svn: Failed to add file 'foo.c': object of the same name already exists

As of Subversion 1.5, you can add the --force switch to any of these commands to override this behavior and tell Subversion to tolerate the obstruction. The output you normally see from these commands will tell you when this happens. For example, it will look something like this:

E foo.c
A bar.c

A new code of 'E' for 'E'xists has been added to inform you of this situation. What Subversion does in this situation is create the ".svn" meta data, as it normally would. The "pristine" copy of the file is stored based on the repository, as it always would be. The file itself, however, remains unchanged. If the file does not match the file in the repository, then svn status will show the file as containing local modifications. Those modifications can be viewed with svn diff, or committed with svn commit. Likewise, you can use svn revert to restore the local file to the pristine copy.

These changes help with a number of scenarios that can happen in normal development. The original use case that we wanted to solve was a developer that downloaded a tarball of some product and winds up making some fixes. Since they started with a tarball, they do not have a Subversion working copy on their disk. If they want to commit their fixes, or even submit them as a patch, then they need to have a working copy. With this change, they could use the svn checkout command, and the --force switch, to checkout the code from the repository on top of their local copy. When the command completes, they will have a valid working copy. They can now create a patch with their changes, or commit them if they have access.

With the code that is currently in trunk, in this scenario, the svn checkout command will still need to pull down the same number of bytes from the server as a "normal" checkout. It is possible that it could be enhanced in the future to more selectively pull down just the differences, something like an rsync. A change of that nature has been deferred for some future release or whenever someone comes along with enough motivation to tackle it.

Here is another scenario where this change solves a problem. Suppose a developer has a project checked out. They create a patch to implement some feature or fix and that patch includes some new files. They submit the patch to the community and a developer reviews and commits it. When the original developer runs the update command, the command will fail unless they remembered to remove the new files they had submitted in their patch. Now, they can just use the --force option with the svn update command, and it will complete normally and just reconcile any issues. If the patch was committed exactly as it was submitted, the developers working copy should not show any modifications after the command completes. If the patch was tweaked, then a diff of the working copy would show whatever tweaks were made, and the svn revert command could be run to get the working copy back to a pristine state, ready for the next patch to be worked on.

We will be making use of this new feature in several places in Subclipse, including several where we have had to code our own workarounds to get around this problem. There are also a couple of new checkout and project share scenarios where we should be able to utilize this new option to implement features that our users have asked for.

Shelves in Subversion

There is a popular feature called "shelves" that was included in Microsoft's Visual Studio Team System. I am fairly certain that VSTS is not the first or only tool to have this feature and in this article I will show you how to get the same feature from Subversion. I think of shelves as essentially creating and using a branch to save some changes you have been working on, but now need to set aside, or shelve, for a while. Here is how Microsoft explains the feature in their documentation, which I found via a search:

Shelve your pending changes when you are not ready to or cannot check in a set of pending changes. There are primarily five shelve scenarios:

• Interrupt When you have pending changes that are not ready for check in but you need to work on a different task, you can shelve your pending changes to set them aside.

• Integration When you have pending changes that are not ready for check in but you need to share them with another team member, you can shelve your pending changes and ask your team member to unshelve them.

• Review When you have pending changes that are ready for check-in and have to be code-reviewed, you can shelve your changes and inform the code reviewer of the shelveset.

• Backup When you have work in progress that you want to back up, but are not ready to check in, you can shelve your changes to have them preserved on the Team Foundation server.

• Handoff When you have work in progress that is to be completed by another team member, you can shelve your changes to make a handoff easier.

The branching model in Subversion, with its use of "cheap copies" is well suited to provide similar capabilities and handle all five of these scenarios. In the rest of this article, I will detail two different ways to shelve changes using branches in Subversion.

Quick Method: Create Branch from Working Copy

The first and easiest method is to simply create a branch from your working copy. A lot of Subversion users do not realize that you can do this. I actually use this technique a lot when creating complex tags. Just get your working copy exactly as you want the tag, and then create the tag based on your working copy.

svn copy . url://server/repos/project/tags/tagname -m "Create tagname"

This creates the tag in the repository based on what is present in the working copy. Subversion does not even need to send any file contents to the repository to create the tag, so it still runs very fast.

You can use this same concept to create a branch or shelf with the changes you are working on. Here is a screen shot from Subclipse showing how to do this:

Notice that the option to create the copy from the Working Copy has been selected. That is the key to this method. When the command runs, Subversion creates a copy in the repository based on the base revision of your working copy, however any local changes are also committed as updates. Likewise, if you have svn added or deleted any files/folders, then those are also included and it does this all in a single transaction.

After the command completes, your working copy will still appear to be modified (because the changes were committed to a different branch then the one associated with the working copy). To get your working copy back to a pristine state, before beginning your next task, you will need to run the svn revert command to remove all local modifications. The svn revert command does not remove files and folders that were added, so you need to manually delete those after the revert.

Hint: The Subclipse revert process will remove selected unversioned files and folders. So if you run the revert option twice, the first time you run it, added files are made unversioned and the second time you run it, they are deleted.

Creating a shelf using this technique is very easy. The problem with this technique comes when you want to work on your shelf again. To do so, you want to use the merge command to merge your changes from the shelf back into the current working copy which is associated with trunk or some other branch. The problem is that since the shelf was created in a single transaction, merging it is a little bit tricky. Suppose the shelve was created with revision 50. The normal way you might expect to merge it into your working copy would be to run a command like this:

svn merge -r49:50 url://server/repos/project1/branches/shelf

This tells Subversion to merge the changes that happened with revision 50 into the current working copy. However, in this scenario, when you run this command you will get an error something like this:

svn: Unable to find repository location for 'url://server/repos/project1/branches/shelf' in revision 49

The problem is that the shelf did not exist in revision 49, so the command errors out. The answer to this is that you have to run the merge using two URL's. The source URL would be the base location that your working copy was pointing to (which was the base of the copy) and the target URL would be the shelf. You also need to know what revision your working copy was at when you made the copy. You can determine this information by running the svn log command on the URL of the shelf. The output of the command will show both the URL and revision it was copied from. Unfortunately, this is also where it can get really complicated. There is a better than likely chance that your working copy contains mixed revisions (see my previous post on Mixed Revision Working Copies). Assuming that is the case, when you made the copy from your working copy, you will have gotten a fairly complex transaction in the repository. As an example, this is the output I get from svn log using a fairly simple mixed revision example:

------------------------------------------------------------------------
r9 | markphip | 2007-01-08 09:34:06 -0500 (Mon, 08 Jan 2007) | 1 line
Changed paths:
A /branches/fromWC (from /trunk:3)
A /branches/fromWC/.classpath (from /trunk/.classpath:4)
A /branches/fromWC/.project (from /trunk/.project:4)
A /branches/fromWC/src (from /trunk/src:4)
A /branches/fromWC/src/org (from /trunk/src/org:5)
M /branches/fromWC/src/org/tigris/subversion/javahl/ChangePath.java
R /branches/fromWC/src/org/tigris/subversion/javahl/ClientException.java
(from /trunk/src/org/tigris/subversion/javahl/ClientException.java:7)
R /branches/fromWC/src/org/tigris/subversion/javahl/JNIError.java (from /trunk/src/org/tigris/subversion/javahl/JNIError.java:8)

As you can see, I have files and folders copied from revisions 3, 4, 5, 7 and 8 and my shelf was created in revision 9. In this example, I know that revision 8 is the right value to use for the source of my copy, but in a more complex example, there may not be a right answer. Let's go back to our example, and just say that the shelf was created in revision 50 and it was created by copying trunk which was at revision 49. Assuming that is the case, then to merge the changes made in the shelf into your working copy, you would need to run a command like this:

svn merge url://server/repos/project1/trunk@49 url://server/repos/project1/branches/shelf@50

This tells Subversion to construct a diff of the changes between trunk @ revision 49 and the shelf @ revision 50 and then apply that diff to the current working copy. When this command runs, your working copy will now contain all of the changes you shelved. You can then go back to working on your change and eventually commit it to trunk or whatever the appropriate branch is to receive the change. Here is a screen shot of the same merge command from Subclipse:

Note that you just have to un-check the box that says to "Use From: URL". This then allows you to enter different "From" and "To" URL's and revisions.

To summarize this method, creating the shelf is easy, but using it again can be difficult. I will now show you what I think is a better, and much cleaner, method for creating shelves.

Better Method: Multiple Steps

There are really two main goals with this method:

1. Remove the issue of mixed-revision working copies from the equation.
2. Isolate the changes you want to shelve from the process of creating the shelf so that they are easy to merge later when you want to use them again.
   

This is always the technique that I use and it is generally much cleaner than the first approach.

The first thing to do is to create the shelf/branch. If you know your working copy is a little old, compared to the HEAD revision of your trunk or branch, then you can use the last revision you committed or updated to when creating the shelf. Otherwise, just use the HEAD revision:

svn copy -r HEAD url://server/repos/project1/trunk url://server/repos/project1/branches/shelf

This command is creating the shelf directly in the repository, based on whatever URL and revision your working copy is associated with. In this example, I used trunk. The next step is to use the svn switch command to switch your working copy so that it is pointing at the shelf URL:

svn switch url://server/repos/project1/branches/shelf

The switch command is also like an update. So if your working copy was not at the same revision you copied when you created the shelf, then it is possible that you will receive some updates, and perhaps even have some conflicts created in your working copy.

Here is a screen shot from Subclipse of the Create Branch dialog. Note the check box on the bottom that lets you create the branch and switch your working copy to it one step:

Once you have dealt with any conflicts that might been created in your working copy from the switch, you just need to commit your changes to the shelf using the normal commit command. When you are done, just use the switch command to switch your working copy back to trunk or wherever you need to be for your next task. Unlike the previous method, you will not need to cleanup your working copy when you do this.

At this point the shelf has been created and your changes have been committed in a way that will make it easy to merge them back later. The only part of this process that is potentially a bit more difficult than the first method is that you might have to resolve some conflicts before you can commit your changes to the shelf, and depending on why you are creating the shelf, you might not have the time to do this. If you create the shelf branch from the revision you have loaded in your working copy, you should minimize any potential for conflicts.

When you need to work on your shelved code again, it is easy. Just use the merge command to merge the changes from your shelf to your current working copy. Suppose you created the shelf with revision 44, spent some time resolving conflicts that were created when you switched, and then commited the changes to the shelf with revision 50. To merge the changes in the shelf to your current working copy, just run this command:

svn merge -r49:50 url://server/repos/project1/branches/shelf

This tells Subversion to merge the changes made in revision 50 to your current working copy (I could have used any value between 44 and 49 for the from revision). Since we separated the process of creating the shelf, from the changes you wanted to store in the shelf, this is an easy command to run. When the command completes your changes will have been merged into your working copy for you to pick up the work where you left off. When you are done, you just commit everything to trunk, or whatever branch is associated with your working copy, and you are done. The branch you created for the shelf can now be deleted from the repository if desired.

Summary
I went into a lot of detail in this article but hopefully you now know a lot more about how branching and merging work in Subversion. You can apply this information in your daily work process and use Subversion as a tool that helps you do your work better and more efficiently. I wrote this article under the pretense of the Shelving concept, but it is really just basic branching and merging in the end. I think one of the strengths of the Subversion design is that they kept it all very simple and you just use the basic functionality to build the process you want.

Combating Sudden Slowdowns in Subversion

Martin Tomes writes about a 20x performance slow-down in checkouts after installing a new anti-virus program. There have been a lot of reports over the years of various anti-virus programs causing slow-down's or other problems with Subversion on Windows. It is a good thing to check if suddenly you are having performance problems or other errors. Hopefully you are at least using a program that lets you fine-tune what and how it is doing the checking.

Another item that people have reported to cause slow-down's is the XP drive indexing. This can be disabled by right-clicking on the drive and choosing Properties. There is an option at the bottom of the General tab that controls whether Indexing is allowed. I do not know if other tools that index your drive, such as the Google Desktop can cause similar problems, but I suspect they can.

Subversion 1.5: Improvements to move/copy commands

This is not an announcement of Subversion 1.5, and no, I do not have any idea when it will be released.

There are a number of nice enhancements to Subversion in trunk and these should therefore all be included in the next release, which will be 1.5. One such feature that I personally asked for was improvements to the copy and move commands. If you read my recent post on the move/rename process, you might assume that these enhancements address that problem -- they do not. Resolving that issue will be a major change that will probably not come until a 2.0 release, if ever. There were some other limitations in the copy/move process, and these really effect Subclipse users. Fortunately, in this case, the limitations were just an implementation issue, there was nothing in the Subversion design that caused them.

OK, enough already, what is the new feature? Prior to these changes, Subversion did not allow an item that was copied/moved to be copied/moved again until it had been committed. If a folder was copied/moved, then its children could not be copied/moved until the folder and children are committed. This is a huge limitation for Java users, especially when they want to use the refactoring tools provided by Eclipse. Let's give an example:

Suppose you have a Java package named: org.eclipse.project.core with a class in a file named Plugin.java. Now suppose you want to refactor the package name to be org.eclipse.project.internal.core and also refactor the class and file name to ProjectPlugin.java. When you try to do this, moving the folder to the new name would work OK, but when the refactoring got around to renaming the class file, it would fail with an error like this:

svn: Cannot copy or move 'Plugin.java': it's not in the repository yet; try committing first

Subclipse performs some gymnastics to detect this situation and workaround it, but part of that workaround means not using the svn move command, which means the file history is lost.

Fortunately, with Subversion 1.5, all of these limitations are now removed. The copy/move API's can now handle all of these scenarios and do the right thing. This allows us to remove all of our workarounds from the Subclipse code and just let Subversion handle everything as it should. The refactoring experience within Eclipse should become much nicer once Subclipse is up to this version.

Addendum: Some users of various Java Subversion integrations may be aware of SVNKit, formerly known as JavaSVN. SVNKit, has contained code that could handle this situation for a long time. They called the feature SmartMove and some of the tools that are based on SVNKit use it. Subclipse uses SVNKit via an Interface where it is mimicking the capabilities of the Subversion Java API (JavaHL), including its limitations. So this note is just to explain why Subclipse does not support this feature when SVNKit is being used.

Subversion Move/Rename Feature

I have read a lot of mailing list threads the last couple years where projects debate whether to move from CVS to Subversion. Usually this happens when their hosting provider adds Subversion support, as the Eclipse Foundation recently did: #131096. I see two primary features in Subversion that are mentioned when people want to switch:

1. Support for atomic transactions.
2. Support for tracking move/renamed files and folders.

Anyone that has ever been burned by the lack of atomic transactions in CVS knows the value of having that feature, and I'd argue the presence of that feature alone justifies the transition to Subversion. However, I'd also say that even more people mention wanting improved support for move/rename as the big reason for switching.

Here is the problem. The support for move/rename in Subversion is often not what user's expect it to be.

The support for move/rename in Subversion is really about maintaining history. If I rename a file, commit it and then examine its history, I can see the history of the file all the way back to when it was created, across any move boundaries. Likewise, if I want to get a copy of what a file looked like last year, I can do so without needing to know what the file was named last year. If these are the sort of features you are after, then Subversion provides them very well.

So what are the features that Subversion does not provide? For starters, you might want to take a look at issue #898 in the Subversion issue tracker.

Subversion implements a move/rename as a copy followed by a delete. The fact that the new file is a copy is the reason you get support for the previous history. The negative side effect of this can be seen in this simple scenario:

1. Suppose you have a versioned folder named foo with a file named bar.
2. Users A and B checkout foo to make some changes.
3. User A renames bar to baz and commits the change.
4. User B makes some changes to the code in bar and attempts to commit. The commit fails because bar is out of date. So far this is all normal and expected. User B then runs svn update. What happens?
5. svn update will add new file baz to the working copy of user B, and bar will remain in the working copy as an unversioned file (if it did not contain modifications it would have been deleted). I think that most people would expect that the local copy of bar would be renamed to baz and now contain user B's changes merged with whatever changes were in the repository as a normal update would do.
6. User B has to recognize this is what happened and transfer the changes to baz before committing. Of course, they might also run svn add on file bar to put it back and then commit it. Perhaps because they did not really recognize or understand what happened.
   

This is just a trivial example that shows a potentially big problem. Imagine the above scenario played out as a large refactoring -- which is entirely possible since that was one of the motivations in moving to Subversion to begin with.

This problem also manifests in other commands like merge. Imagine you have a branch with some customizations you are working on. A refactoring happens on trunk which involves the renaming of several files and folders. You now have lost the ability to use merge to bring the changes in the branch back to trunk, or vice versa. Or, more accurately, you have lost a lot of the automation that Subversion can normally perform for you during this operation.

If you read the referenced issue from the Subversion issue tracker you will see that solving this problem is a big challenge. I would not take the lack of this feature as a reason to not use Subversion, obviously I'd recommend that you DO use Subversion. However, it is important to understand how the tool works and what it does and does not do before making the switch. My big fear would just be for a project to switch so that they could do a bunch of refactoring work and then run into this limitation in Subversion and become angry. I think the advantage Subversion gives you is that you often can do those refactorings and take advantage of the history support that Subversion provides. However, if you are going to do this, you need to use communication with the rest of the team and try to coordinate it in a way that it does not impact the entire team.

Mixed Revision Working Copies

Subversion includes a powerful and useful feature known as Mixed Revision Working Copies. However, aspects of this feature regularly cause problems and confusion to new users of Subversion and its various clients like Subclipse. Problems related to this feature come up so often on the Subclipse mailing list that we had to create an FAQ for it.

The definitive explanation of mixed revision working copies can be found in the book that is maintained by the Subversion developers. See: Mixed Revision Working Copies. I'd encourage you to read this explanation, and really the entire chapter contained on the linked page, to gain a better understanding of Subversion and how it works. The following passage summarizes the feature:

"For example, suppose you have a working copy entirely at revision 10. You edit the file foo.html and then perform an svn commit, which creates revision 15 in the repository. After the commit succeeds, many new users would expect the working copy to be entirely at revision 15, but that's not the case! Any number of changes might have happened in the repository between revisions 10 and 15. The client knows nothing of those changes in the repository, since you haven't yet run svn update, and svn commit doesn't pull down new changes."

The end result of this example is that, after the commit, the file foo.html is at revision 15, but the rest of the working copy, including the parent folder of foo.html, is still at revision 10 and will remain at revision 10 until you run svn update.

This scenario trips up inexperienced users later on when they attempt to do something like commit a property change on the parent folder of foo.html or perhaps rename the parent folder. Subversion will not allow a property change, rename or delete of a folder to be commited unless the folder is at its HEAD revision in the repository. When you attempt to commit an item in this scenario, the commit will fail with an error message like this:

Transaction is out of date
svn: Commit failed (details follow):
svn: Out of date: '/Project1/src/folder1' in transaction '1221-1'

The solution at this point is to run the svn update command so that the folder in your working copy is updated to its HEAD revision. Of course all of this is also true for files. The difference is that with files it is much easier for the user to understand when and why they get out of date as it only happens when an explicit change is made to the file or its versioned properties.

In my opinion, users that are using a graphical Subversion client, or IDE integration, are much more likely to run into this problem than someone that uses the command line. The reason is simple, command line users are likely to simply run the svn update command at the root of their project in order to save typing. This has the beneficial side effect of bringing all of the folders in the working copy up to date with the repository. Conversely, a user in a graphical client will often take operations in a context that results in actions only being performed on the specific selections. In Subclipse, as an example, many users use the Eclipse Synchronize view to see all incoming and outgoing changes. This view encourages the user to only commit and update the specific files that have been modified, making it more likely that they will run into this issue.

First Post

This is the obligatory first post. What do I intend to do with this blog? Basically, I want to write about Subversion. I have not quite decided how I am going to do it, I imagine it will be a bit of trial and error. I am thinking of doing some form of schedule. Maybe one day will be a Subclipse tip or feature, one day a general Subversion tip or feature, one day post news from Subversion or Subclipse, such as a new feature in trunk. Maybe a news day where I link to some other Subversion and Subclipse information, and finally an essay day where I make a longer post about something with Subversion or Subclipse.

Of course, I also intend to post about other topics I am interested in or comment on something I read. We'll see and let this evolve over time.