I’ve been doing a lot of repository conversions recently, lifting ancient project histories from Subversion or even CVS into modern distributed version control systems. I’ve written about the technical problems with these conversions elsewhere but they also raise issues that are almost philosophical – and not unlike, actually, the challenges natural-language translators face moving a literary work between human languages.
In translations between human languages, there’s always an issue with constructions and idioms that are present in one language but not the other. This leads to a fundamental question about whether to prefer a literal or figurative translation. For example consider the French phrase “mon petit chou”; translating it literally gives you “my little cabbage”, which sounds very silly in English even if you know enough French to be well aware that it’s a term of endearment analogous to, say, “sweetie”. So most translators will opt for a free rendering that conveys authorial intent rather than wording – and then perhaps find they’re in technical trouble a few paragraphs later if another character makes a pun about cabbages.
Repository conversions raise startlingly similar issues. For a really low-level example of idioms mismatching, consider changeset references. In Subversion, a changeset reference is a number that increments from one; the conventional way of writing a reference just looks like “r235” for the 235th commit. All the commits take place on one server and are time-ordered, so monotonically increasing commit numbers make sense.
This isn’t so in DVCSes, which are built around history merging. They’re like moving from Newtonian to Einsteinian physics; there’s no reliable linear time ordering from a single clock anymore, just lots of change histories in different peer repos diverging and reconverging. A commit ID is a hash of the contents of an entire code tree (and the hashes of its parents); it describes content, not sequence.
So, when you’re converting a Subversion repo to a DVCS, and the change comments contain references to other changesets – like, say, “r235” – what do you do? You can’t leave it untranslated, it would just be a meaningless tag that would impair the readability of the history. That would be bad, because project histories (in a way more literary works never do) have a function. People read them to understand code and fix bugs. We never want to throw away the information that the author of commit A intended to refer to a specific other commit B in his comment; that could be important.
The tactic I use is to replace commit references in the language of any given VCS with a combination of committer name and commit timestamp (or unambiguous part thereof). In principle I could use the target DVCS’s hash for the commit, but why translate to a magic cookie that’s going to break – and need to be retranslated – if the codebase is ever moved to another DVCS?
So…”Reverted r235, it introduced a crash bug”, becomes “Reverted Fred Foonly’s commit at 2011-05-19T03:30:26Z, it introduced a crash bug.” I’m doing what a literary translator has to do: render intent instead of wording. And I’m justifying it in detail because hackers get weirdly twitchy about “rewriting history”.
There’s actually good reason for this twitchiness. People irresponsibly or maliciously messing with change histories threaten all the reputation incentives that make the whole hacker social machine hum. The weirdness and emotional intensity about it comes partly from how vitally important those incentives are, and partly from the fact that many hackers understand them only half-consciously and not very well. Consequently, they overprotect; I’ve actually had people accuse me of having made an evil tool because reposurgeon can easily edit comment histories.
Pressed, a hacker will admit that you pretty much have to rewrite commit references during a repo translation in order for them to make sense – but odds are he’ll feel uncomfortable, as though by modifying the original comment you’ve done something vaguely unclean or disreputable.
Now, one could translate like this: “Reverted Fred Foonly’s commit at 2011-05-19T03:30:26Z, it introduced a crash bug. (This reference was to ‘r235’ in the Subversion history.)” But, really, there wouldn’t be much functional point to this. The r235 is still meaningless in the new context; adding that parenthetical would be at best a ritual gesture of appeasement towards Thou Shall Not Rewrite History.
There is actually good reason not to perform that gesture. It’s a functional reason; the comment is there to convey a meaning, which the gesture distracts from. We’re actually near the territory of literary translation here, where one of the things a translator is not supposed to do is obtrude between the reader and the author.
Here’s another idiom mismatch where we don’t even have an equivalent of the ritual gesture available: tag names. A tag is just an arbitrary name for a version in the repository history. The most common use for tags is to identify shipped releases. So, for example, in the GPSD repository there’s a tag “3.3” for the release 3.3 I just shipped. (One wants this, of course, so that when someone reports a bug in public release 3.3 one can see exactly what that code was and what has changed since.)
The reason there’s a translation issue here is because older VCSes often had limitations on the form of a tag name – had to begin with a letter, couldn’t contain dots or other punctuation, that sort of thing. So tags from older systems tend to look like this: “release-3-3”.
When I translate a tag like that into modern, less restrictive VCS, I change it into “3.3”. It’s not what the original committer wrote, but it is probably what he/she was actually thinking. Intent, not wording. More generally, a really high-quality conversion of a Subversion or CVS repo into (for example) git should, as much as possible, look like what the developers would have created if they had been using git from day one – as opposed to carefully preserving Subversion/CVS idioms in amber because that’s more authentic.
Why? Again, repo translations are not primarily history or literature; they are intended to support the difficult work of understanding the code history so problems can be solved. The best translation of ancient history in a repo interferes as little as possible with the process of understanding the actual code.
Fortunately, this rule doesn’t collide with the social requirement that past hackers be properly credited (or blamed!) for their work. Nothing in a repository translation will ever require that anybody’s name be struck from the annals. Actually, translating to a modern DVCS requires that committers be identified by an unambiguous internet-wide identifier (an email address) rather than the old-style local usernames of CVS and Subversion. Thus, the attribution information in a DVCS is actually better than in older systems. It is impossible for hackers to object to this :-).
The rule that a good translation minimizes the reader’s cognitive friction – actually hides the degree to which it is a translation, except in a discreet note up front – is parallel to good practice in literary translation as well. But when you start applying it seriously, you will do things that make hackers twitchy.
Here’s one: When I’m reviewing old changeset comments during a repository lift, I fix obvious typos. Just one less thing for future hackers to trip over in the history when they’re trying to get serious analysis done. Intent, not wording!
Here’s another: On some translations where I was able to deduce the right referent for missing release tags, I silently added them…and attributed them to the project lead or release manager, back-dating them to the proper changeset. I chose to treat the repo as a slightly incomplete or damaged representation of the good practice its developers were trying to achieve, rather than authentically leaving the omissions in place.
One reason to take this attitude is that older automated repo-conversion tools such as cvs2svn really did have a tendency to damage that representation – often you just can’t tell whether the absence of a tag or slight garbling of other metadata was the result of a human slipup or a bug in a conversion tool.
The place where free translation raises the most issues is massaging comments into the form DVCSes prefer. Modern systems have log-summary tools that strongly encourage formatting change comments not as running paragraphs but as one standalone summary line optionally followed by a blank line and running paragraphs.
Most comments are one-liners anyway. Many of the multiline comments can be
turned into summary-line-plus form by inserting whitespace after the first sentence. But in extreme cases (about 0.5% of the time) I find I have to write entire summary sentences myself. (Yeah, that’ll send some of my readers to their fainting couches!)
The gods of Thou Shalt Not Rewrite History should not go entirely unappeased. I think it is a good idea to embed some indication of what you’ve changed in the repo history. Which is why the very first commit comment in the git translation of the Hercules repo now reads like this:
Initial repository setup
{You are actually looking at the result of two history lifts. This
project started out in CVS. In January of 2009 it was moved to
Subversion. On October 24 2011 it was moved to git, with the comment
history edited to turn Subversion commit references into references by
content and/or commit date, and massage comments into git form with a
summary line. Here and elsewhere, comment portions in curly braces
without preceding $ were added at conversion time.}
I’ve tried to hold the number of curly-brace comments to a minimum. I still haven’t decided whether I should sign my name in this one.
UPDATE: A commenter persuades me of two things:
1. The square brackets often used to mark ellipses and editorial comments may be a better convention – doubled, to avoid ambiguity with square brackets used in code snippets that might appear in comments. So: [[Note from the translator: …]]
2. For the sake of future browsing tools, inserted commit IDs should be in a uniform format, like this: “[[2011-10-25T15:11:09Z/fred@foonly.com]]”. This is an ISO8601 timestamp followed by an RFC822 mail address – unambiguous and easy to machine-parse. The brackets indicate a translation note and are not intended to be part of the ID format.
“It is impossible for hackers to object to this :-).”
Eric.
Come on now.
You know better than that…
Hackers will disagree on the color of the ceiling in the Big Room.
Of course they can disagree on technical stuff like this…
Oh, and the bit about “my little cabbage”: It makes a lot more sense when you know that French babies are found under a cabbage leaf, instead of being delivered by the stork…
You seem to be assuming that you and your conversion tools are infallible. If one of your “fixes” turns out to have been a mistake, you’ve already deleted a lot of information that would be useful in cleaning up the resulting mess.
>If one of your “fixes” turns out to have been a mistake, you’ve already deleted a lot of information that would be useful in cleaning up the resulting mess.
Huh? How does that follow from anything I said?
In particular, what information do you think I am deleting?
I was going to make eugine_nier’s point more gently. Considering how cheap storage is, is there a reason not to keep the whole old record in case something needs to be checked?
>Considering how cheap storage is, is there a reason not to keep the whole old record in case something needs to be checked?
None at all. But that is completely orthogonal to the issues I was discussing. Nothing prevents anyone from keeping copies of unconverted repositories as a historical record and check on the conversion. The issues I was discussing are about how you create a high-quality repository for future work.
If you’re that concerned, there’s no reason you can’t just tar up the old CVS/Subversion/what-have-you files and keep them around on a backup somewhere.
The implication is that cluttering the converted repository (is redundant and) will diminish the efficiency of future work in the new DVCS, and for those cases where the quality of the conversion needs to be checked, it is better to go refer to a copy of the unconverted repository.
Is there any tool that can refer to both the converted and unconverted repositories, and aid in viewing changes in the conversion process?
To those hackers flipping out about history rewriting, I would suggest two things: First, a cost/benefits analysis. I’ll pay the price of some tweaking of the history so that I don’t have to be stuck with some primitive source control tool for the next X years. If you actually work the multiplication out for how much additional effort you incur per commit-like event it adds up fast.
Second, esr is not creating the need to tweak history between two different source control systems, he is merely exposing it. Shooting reposurgeon is shooting the messenger. In the general case, conversion between two repository layouts is fundamentally ill-defined and human choices rigidly must be added to the mix for the process to work at all.
@Shelby
Have a look at this technique to enforce a sequenced prefix on the git commit hash:
https://github.com/vog/beautify_git_hash/commits/master
> In particular, what information do you think I am deleting?
My instinct was to wonder “what if r235 is ever used to mean something other than a commit?”
I might have solved this by turning it into “r235 {Fred Foonly’s commit at …}”. In a cost-benefit analysis I’m sure your way is positive compared to not changing anything; it might well be positive compared to mine.
I can see the logic behind reducing the cognitive load and all, but throwing away information always bothers me. I would prefer that any changes be recorded in some kind of data structure that is linked to the changed information, so that one can always follow the link to find the original information and the rationale for the change, just in case.
“Reverted r235, it introduced a crash bug”
“Reverted r235 [fred.foonly@example.com, 2007-10-11 12:01Z], it introduced a crash bug”
“Reverted r235 [2007-10-11/12:01/fred.foonly@example.com], it introduced a crash bug”
1. Square brackets are traditionally used for editorial insertions.
2. Readily parsable so future tools can turn into hyperlinks.
3. keeps original text!
>1. Square brackets are traditionally used for editorial insertions.
Hm. This is a good idea. I think you’ve persuaded me to (a) change my convention for metacomments, (b) write commit IDs in a uniform parseable manner, and (c) describe these recommendations in the reposurgeon documentation.
I think the uniform parseable reference should look like this: [2007-10-11T12:01/fred.foonly@example.com], That replaces your first slash with an IS8601 date stamp. The result is easier to parse. It’s even easy to unambiguously recognize with the [] omitted.
UPDATE: For a better version, see my next post.
@ben
Many of the discussions here are confusing several objectives:
1 Archiving should be done unchanged
2 Annotating should actually not be done in the original text ( as you do) but in an overlay that points to the original text
3 Translating should preserve the intention of the text (which might not be that of the author).
Note that 3 requires a non unique interpretation of the text. Different interpretations are always possible.
Note that old revision identifiers like r234 could in theory be preserved and used in modern DVCS conversion if they were made into tags… but from prectical point of view having one tag for one revision would absolutely destroy performance (at least in current versions of DVCS; for Git having many, many refs neds to be handled anyway because tools like Gerrit code review uses many, many refs).
@esr:
Yes, you should sign your name:
(1) You have performed a non-trivial and vital piece of work on the project’s (meta-)data, and that should be identified (for praise or blame, as appropriate :-D )
(2) Giving this history (meta-history??) could lead a future maintainer to dig for the original repo, if necessary, to glean further information and insight into the evolution of the project.
(3) In an exquisite bit of meta, I suggest that you should also put reposurgeon’s name and version number in. If necessary, future maintainers could go there to figure out how the one repo was mangled into the other.
Just my $0.02. @Jay Maynard, your thoughts?
[[p.s. quibble, quibble – it should be “Thou Shalt Not Rewrite History” ;-) ]]
I think inserting the version number of Reposurgeon is a great idea, in case it subtly garbles something and people need to troubleshoot and recover.
So, when are you going to convert Apache?
/me ducks.
@winter, the objective is that human readers, reading the up-converted repository, can work out what has gone on, with, if possible, the assistance of their tools.
There is always the original pre-convert archive for people who are interested in history. The question is what do you lift up in front of the developer’s eyes?
In light of the complex issues raised against commit author ID (email address, open ID not spamable in clear text, etc) in your new blog, it seems that conflating commits with commit authors is changing the semantics of version control. It would be better to not make such a semantic change during the conversion process.
I am wondering why you can’t replace the commit ID, e.g. “r235?, with the equivalent set of sequential DVCS commit hashes that identify the commit in question by way of differencing? Thus leaving the commit author data orthogonal to commit ID as was originally intended.
1) You wrote “Though Shalt Not…” ITYM “Thou Shalt Not…”
2) There is an Italian expression: Traduttore, traditore (“Translator, traitor”).
The words are almost the same in Italian, and the implied ironic jest is that a translator cannot help but betray the author.
@Shelby, are you suggesting that authors shouldn’t be identified? Or rather that the format of the identifier is inadequate?
The commit hash is sufficient for identifying a commit. If a commit’s comments refer to another commit (via a commit id), there is no need to mention the author.
Replacing the commit hash with the author’s id, is afaics a corruption of the comment. It unnecessarily conflates the author with a commit. And for what benefit?
The original comment did not mention the author, only refered to the commit. Referring to a commit, is not the same semantic as referring to an author and a timestamp.
I am sorry but I must emphatically disagree with Esr on this one. And this will end up corrupting a lot of open source.
Maybe I misunderstood the problem. Remove the “emphatic” from my prior comment. After further thought, I realized a commit hash will for a given set of changes will be different depending on which source code base it is merged into. Thus I assume a reference to a commit hash will only be valid for one branch of a repository. Thus, commit hashes are not a generally appropriate semantic in DVCS for referencing a commit.
When converting a centralized history, i.e. a single branch that is not distributed, the commit hashes will be constant, unless someone wants to fork that history.
So I surmise that Esr’s point is that since we need a general semantic for commit referencing in DVCS, then we should make them consistent when converting from centralized version control repositories.
Is there any other possible general semantic for a commit, that would be dependent only on the content of the changes, and not conflate the author and timestamp?
@Shishire, I didn’t fully answer your question. I am not saying the authors should or should not be identified in version control. I am saying that if a commit comment refers to another commit (i.e. a block of changes), then it should not be changed to refer to an author+timestamp. A block of changes is not the same semantic as an author+timestamp.
And now I realize that Esr’s proposed author+timestamp key is also ambiguous in the DVCS semantics. Afaik, DVCS has the capability to merge changes piecemeal, and thus the author+timestamp will not necessarily stay correlated to block of changes, just as the commit hashes will not stay constant for a block of changes as they are merged into different local copies of code bases.
The problem is that DVCS does not generally support referencing a block of changes, and using author+timestamp does not generally rectify that problem.
Thus Esr’s proposal to use author+timestamp is not a generally correct semantic, and thus I return to my “emphatic” stance and say that to be faithful to the centralized history, he should only be replacing commit ids with commit hashes.
This raises a question of how can blocks of changes be generally referenced in DVCS? As far as I can see, they can’t.
If there is a centralized master repository for a project, then those commit hashes should remain historically constant and can be referred to in any new commits in a DVCS copy of that master repository history (but the new distributed commits can not be referenced by other commits until they are merged into the master repository where they will obtain a permanent commit hash).
My work to create a new mainstream referentially transparent computer language might provide another solution. If projects become orthogonal modules, then centralized version control for each module, is a form of distributed version control, and without this general problem of how to reference commits in those modules, and without forcing a centralized master repository.
An improvement might be that DVCS might someday know to propagate commit hashes in comments, i.e. on merges modify each hash in comments so it refers to the same block of code changes. This wouldn’t be perfect, because the subset of code within the changeset being referred to might not have been merged. This could be mitigated by committing on each narrow issue changed where possible, instead of multiple issues per commit.
Thus, DVCS may need fields in the commit record for referencing prior commits, otherwise it is strong-AI problem to identify commit hashes in comments.
Link to a weakness in my logic and a clarification.