For those of you who have been following the development of reposurgeon, a pre-announcement: the next version, probably to be numbered 2.0, will directly read Subversion dumpfiles and repositories.
I’ve got this feature working now – it’s why my blogging has been scant recently – but I intend to have a really good regression-test suite in place and at least one large repo conversion done before I ship it for general use.
Note an important limitation: it will not write Subversion repos. So it will be useful as a conversion tool but not directly as an editor.
Remember, it MVST be “Reposurgeon 2 – Electric Boogaloo!”
>Remember, it MVST be “Reposurgeon 2 – Electric Boogaloo!”
Hmmph. I had, of course, already realized this.
Off-topic:
I’m surprised there hasn’t been any discussion of HP open sourcing WebOS. While it may be late given Android’s dominance and growth, it offers a non-Microsoft option to manufacturers if they have concerns about Google’s ownership of Motorola.
Of course, this would be have been more effective 12 months ago, but I suspect the new CEO had to come in and find her feet before making this decision.
Oops, looks like that discussion (WebOS) started about 3 threads back. Feel free to delete my comment above.
Subversion to Git via Reposurgeon? I might actually have something to celebrate this Christmas!
Seriously though, I don’t know who all’s involved (other than you), but I’m handing out mighty manly hugs ;)
>Seriously though, I don’t know who all’s involved (other than you)
This is a solo project.
Current state of play is this: I’ve crafted ten small repos which are self-describing, so you can see that the import-stream translation is correct by eyeball. They exercise almost all the cases I can think of – various repo layouts, properties, tagging, branching, svn:ignore, and so forth. I need to write two more, one to test translation of svn:executable and the other to test file copy translation.
Translation seems to be working perfectly on all of these. I check this not just by looking at the output import stream, but by then dumping it to a git repo and eyeballing the result with gitk. I can promise this much; you’re going to get better branch and tag conversion than any of the existing tools do, and (a theme in my projects) there are no required switches. reposurgeon discovers the branch layout itself and does the right thing.
Next, I need to do a big messy repo conversion, which is how to be sure the support is really production quality. I have one lined up.
…I suspect the new CEO had to come in and find her feet before making this decision….
Women…being late…
*ducks*
;)
I can promise this much; you’re going to get better branch and tag conversion than any of the existing tools do, and (a theme in my projects) there are no required switches. reposurgeon discovers the branch layout itself and does the right thing.
What? No configuration mini-language with nifty English-like syntax? :)
Seriously, though, It Just Works ™ is exactly what a tool like this needs to be.
>What? No configuration mini-language with nifty English-like syntax? :)
Nope. When I did fetchmail’s config language I had not yet attained enlightenment about this.
I’ll still write such things when necessary, but these days I only very rarely agree they’re necessary. Autoconfiguration is harder work than putting in a gallimaufry of options, but the payoff is huge. Among other things, fewer switches means fewer test cases.
…and (a theme in my projects) there are no required switches…
Heh…yeah…I’m not as adamant as you about such things, but I certainly appreciate and value your motives. GPSD kicks ass because of it…
…I need to do a big messy repo conversion…
Oh baby, I have the shag-nastyest pile of hippo shit to test this on….
>Oh baby, I have the shag-nastyest pile of hippo shit to test this on….
If $EMPLOYER won’t murder you for it, ship me a copy and I’ll make it part of my final testing.
I just wrote the svn:executable and filecopy checks. Next I plan to to write a test harness that will take a stream file, convert it to a git repo, and then walk through parallel checkout sequences looking for differences in the file manifests. Keywords and the way I handle svn:ignore might make this a little tricky…
this blog post is incomplete. the word “reposturgeon” has not yet been invoked.
>this blog post is incomplete. the word “reposturgeon” has not yet been invoked.
How insupportable of me. I shall rectify this error instanter.
>>this blog post is incomplete. the word “reposturgeon” has not yet been invoked.
>How insupportable of me. I shall rectify this error instanter.
now someone in the comments needs to exhibit confusion about whether the program is named “reposurgeon” or “reposturgeon.” then we can all go home happy.
>this blog post is incomplete. the word “reposturgeon” has not yet been invoked.
That’s because the reposturgeon is in the hospital. She was addressing a meeting of the Young Republicans when the entire crowd whipped out tiny spoons and mobbed the podium, chanting,
“…Mmmmmmm…….caviar….”
…If $EMPLOYER won’t murder you for it, ship me a copy and I’ll make it part of my final testing….
There’s no real danger of ‘murder’…just…it’s so fucking huge and awful and ghastly. Like staring Cthulhu in the face.
I need to think about how to chop something off to provide a respectable test case…
Listen, I have it on good authority that not only does Eric stare Cthulhu in the face, they trade bon mots over the sushi during their weekly luncheon date.
I would suggest that you tar the entire monstrosity up, wrap a pretty bow around it, and ship it off to him. The more real-world, mandrill-butt-ugly test cases reposurgeon goes through, the less `beta testing´ us end users are gonna have to suffer through….
Just my $0.02.
>Listen, I have it on good authority that not only does Eric stare Cthulhu in the face, they trade bon mots over the sushi during their weekly luncheon date.
You are misinformed, sir. I do not enjoy sushi. We exchange witticisms over the bloody, steaming entrails of my former enemies!
>The more real-world, mandrill-butt-ugly test cases reposurgeon goes through, the less `beta testing´ us end users are gonna have to suffer through….
Indeed it is so.
Morgan Greywolf Says:
> It Just Works ™ is exactly what a tool like this needs to be.
The thing about “It Just Works” is that “it” has to be pretty straightforward. (Note straightforward != simple.) If “it” is move this repo wholesale from here to there, that is straightforward, even if doing it is mind numbingly complex.
There is a saying in the Windows world that every option on the Options dialog represents a disagreement between the programmers or the product designers. However, in reality that isn’t true. For example, where I work there are hundreds of databases. Without command line parameters it is pretty hard to instruct a tool which db to use, how to output my results, what credentials to use and so forth. Without a language it is hard to describe how I want the data extracted or manipulated.
However, I guess there used to be a time when more options were always considered better. I am reminded of a statement allegedly written in a letter from Pascal to a colleague where he said “sorry this letter is so long, I didn’t have time to make it shorter.”
I sit corrected, sir. Ia! Ia! Cthulhu fhtagn!
@Jessica Boxer: “However, I guess there used to be a time when more options were always considered better. ”
I used to think that way. Then I read the man page for rsync. Cured me.
@Michael Hipp: But, rsync does, mostly, just do the right thing by default. In fact, if -a and -z were on by default, it would be perfect, IMHO. Simply typing ‘rsync -avz src-dir dest-host:/dest-dir’ does what you want most of the time without installing anything special or typing any other parameters.
@Jessica Boxer, in addition to Michael Hipp’s suggestion, check out the man page for what I consider to be one of the worst CLI designs ever: certutil from the NSS SSL tools.
> I need to think about how to chop something off to provide a respectable test case…
One of the things that I’ve learned working in the Enterprise Software ™ arena is that scale is a test case all of its own. Actually, an order of magnitude more test cases, really.
With all due respect to Eric, its possible that he’s accidentally leaked a reference count somewhere, or one of his corner-case handlers causes a state-machine to reset and obtain O(n) performance instead of O(log(n)) performance, that sort of thing. In a small test repository which just performs functional validation, the existing implementation could leak a page of memory for each file or directory processed (hell, every line of code) and *nobody would notice*. When you have a userspace application which is non-interactive, who’s memory image gets cleaned up on process exit, runs for seconds and executes on a system with GB of memory and a quad-core CPU, you won’t notice these things. A gigantic monstrosity such as you describe, on the other hand, is a great way to reduce the likelihood of an odd typo somewhere causing bizarre issues.
Eric – any chance you might write a Perforce module for this? I’ve got a great use case. I just need to be able to convert complex branched P4 repository with cross-integrations to something else while maintaining complete tracking of changes present, etc? :-)
>Eric – any chance you might write a Perforce module for this?
I’d need shell access to a working Perforce installation on a Unix box, and documentation.
But there may be an easier way. The Git wiki claims Git has a p4 conversion plugin.
@Garrett:
> With all due respect to Eric, its possible that he’s accidentally leaked a reference count somewhere
I thought it was pure Python. While it’s certainly possible to have performance-robbing memory issues in pure Python, they are usually fairly easy to locate and dispatch. Usually all it takes is to add some code like foo.bar = None to break a circular reference or two. In C, a memory problem is when you inadvertently stop referring to a chunk of memory. In Python, a memory problem is when you don’t stop referring to a chunk of memory.
More options ARE a good thing. Not needing to use them because the program autoconfigures itself to sensible defaults is even better. Having the options AND sensible defaults is the best.
Well, actually the best is to see your enemies slaughtered at your feet, and to hear the lamentations of their women. But lots of options you rarely have to use because the defaults are so well done, but CAN use when you need ’em is second-best.
@Patrick Maupin:
‘foo.bar = None’ will not always result in a release of memory. It depends _greatly_ on what foo.bar was pointing to. ;)
@Morgan:
Sure. Obviously, it could even add memory if foo didn’t previously have the attribute ‘bar.’
But I was talking about circular references, and was trying to make the point that, unlike in some other languages where you have to explicitly free memory, the very act of assigning None to foo.bar will decrement the ref count of whatever foo.bar used to be bound to, and automagically free it if the ref count goes to 0. You can also use “del foo.bar” but that doesn’t make the point quite so clearly (because it looks like you are doing an explicit free rather than simply decrementing a refcount). Of course, I can always depend on someone who understands the point perfectly to deliberately misunderstand… :-)
> I thought it was pure Python.
I was speaking more generally. The potential algorithm issues apply regardless. The memory allocation issues are less likely to apply in this specific instance. However, they are possible for the general case for similar utilities, but implemented in a different language.
@Garrett:
> The potential algorithm issues apply regardless.
Sure. Running on large and extra-large datasets is a good way to find out how well your algorithm scales.
> The memory allocation issues are less likely to apply in this specific instance.
You said it’s possible that Eric leaked a reference, so I was merely pointing out that seems unlikely…
> However, they are possible for the general case for similar utilities, but implemented in a different language.
I could be wrong, but ISTM that most unpaid programmers competent to write such an app would do so in a memory-managed language. (And migrate performance critical sections to C or C++ as necessary.)
Paid programmers are a different story. They have to provide value-add (e.g. massive speed) and protection (e.g. lack of source and easy decompilation) that makes it more likely that something like C or C++ would be used. But when you go down that path, refactoring to make your algorithms perform better is often quite difficult.
That’s easy to obtain (assuming, as is likely the case, that you already own a Unix box).
Perforce offers a free, time-unlimited trial version to single desktop users here: http://perforce.com/downloads/try_perforce_free
Well, if Eric has structured reposturgeon correctly, then you don’t need options. You’re either his bitch, or else you write your own top-level code which supplies the parameters you want to the mass of code beneath it. Basically, every Python program should be of the form:
#!/usr/bin/python
import mycode, sys
mycode.main(sys.argv)
And if that doesn’t make you happy, then you go look at …main() to see what it’s doing.
@esr: On the lack of subversion write: do you have an allergy to writing subversion repos or just no interest? I would like to enable $EMPLOYER to experiment with git, but being unable to demonstrate that I can get data back *into* subversion if it doesn’t work out makes the political barrier to entry that much higher. Would you accept a patch if I were to implement it?
Also, I’ve already found reposurgeon terribly useful on some of my own git repos and would love to run amok in some of ${EMPLOYER}s repos.
>On the lack of subversion write: do you have an allergy to writing subversion repos or just no interest?
I’m reluctant to try to support it because git-to-Subversion would be so lossy – you’d lose author information, full IDs, and branch merges. On the other hand…
>being unable to demonstrate that I can get data back *into* subversion if it doesn’t work out makes the political barrier to entry that much higher.
…that is a rather telling point.
>Would you accept a patch if I were to implement it?
Possibly. Depends on how well the patch fits the architecture.
I thought a major point of git was that some central authority didn’t need to accept patches… (/me ducks)
Ah, Patrick….such bait….but also a good point.
I have put zero time into this and won’t even know if I’m going to for several weeks more, but it would probably make merges down the road easier if I could pull from your repository. Do you publish a SCM repository for reposurgeon? I do not see such a link on any project pages or a quick Google search.
Hi Eric,
for converting http://www.PyInstaller.org to git, I fixed some bugs and added some features to reposurgeon.
Code can be found at https://github.com/htgoebel/reposurgeon, commit messages should state the change.
Please drop me a node so I can send you more issues I was not able to solve (and just worked around).
Regards
Hartmut
@esr: Do you have a semi-working version of this? I’ve got some personal subversion repos I’d like to migrate. I can do the git-svn thing and then apply reposurgeon but if you have something ready for “beta” level use I’d love to use it.
>Do you have a semi-working version of this?
Yes, but it has a really obscure bug that pops up if you follow a branch-to-trunk merge with a trunk-to-branch merge. That’s why I haven’t shipped 2.0 yet. If your repo history is linear or treelike you should be good. Alternatively, you can help me debug it…