The 2.13 release of reposurgeon has in it maybe the coolest new feature I’ve ever implemented in five minutes of programming – graphical visualization of a repository’s commit DAG.
In yet another episode of “The Awesome Power of Combining Unix Tools”, the way this actually works is that the new “graph” command emits to standard output a description of the commit graph in the DOT language used by the open-source graphviz suite. To turn this into an image view, just create the following little reposurgeon script and call it something like “show”:
shell dot </tmp/foo$$ -Tpng | display -; rm /tmp/foo$$
Now you can say “script show”; reposurgeon will call the graph command, redirecting the output to a tempfile (the reposurgeon interpreter knows that $$ in a script should be expanded to the process ID). Then it will spawn a shell and run a pipeline using the main graphviz rendering program, dot, to make an image from the DOT markup. That is then fed to the viewer of your choice, in this case display(1) from the ImageMagick suite.
Oh, and that $% in there? That says “substitute the selection set, if any, given the script command”. So you can use this to view selected subsets of the commit graph – useful for large repositories.
The stupid way to implement this feature would have been to hardwire assumptions about the image renderer and viewer into reposurgeon itself, or (slightly less stupidly) pass them in via environment variables or a dotfile. By using reposurgeon’s script feature I avoid all that sort of nonsense – the only graphics-specific thing reposurgeon itself knows how to do is emit DOT markup, which took me all of about five minutes to write after I read the graphviz documentation.
I added in-script expansion of the $$ and $% cookies once I knew ‘graph’ was a good idea (I already had ‘shell’ and > output redirection for reposurgeon commands). The $$ cookie is of course modeled on how Unix shells expand it – a perfectly reasonable notation that there is no reason not to re-use here, and which will be helpful for any other reposurgeon script in the future that needs to use tempfiles.
This is how it looks when you design your application as a domain-specific language and think in terms of adding simple, combinable, orthogonal primitives to it rather than big lumps implementing idiosyncratic features with code that never gets re-used for anything else. Total lines of code to implement DOT output plus $$ plus $% was certainly less than it would have been had I tried to write the equivalent of that script in Python inside reposurgeon, and this way each of these three little featurelets can pay off its complexity cost in the future by being useful in ways I’m not trying to anticipate now.
This design has one drawback, however. The reposurgeon scripting facility was really designed for writing regression tests and per-project repository-lifting scripts; there is at present no place reposurgeon looks for scripts like this that should be shared among all your projects – for the excellent reason that this is (probably) the first such shareable script ever written. I need to think about that; this will probably turn into another addition to the language.
The graph of reposurgeon itself is likely almost a straight line, but perhaps you might post some interesting graphs you’ve come across in doing the conversions.
The DOT language really is a design tour de force in its own way. I once had a meeting with about 20 developers in it as we were sketching out a dependency graph amongst all of our services, to analyze where the reliability bottlenecks were, and I was able to build a graph up in real time by simply typing it out into DOT and using a website that rendered it into a graphic every time it changed. Try that with Visio or anything else and you drag the meeting to a halt to rearrange things as it grows; this just handled everything as the changes came in. And I had only spent, as you say, about five minutes in advance learning how to use it… again, try that with Visio. Text FTW.
I’ll 2nd the request to see some examples. I don’t have anything at my fingertips that would allow me to make my own.
>I don’t have anything at my fingertips that would allow me to make my own.
1. Grab the reposurgeon distribution.
2. Change to the ‘test’ subdirectory.
3. Run reposurgeon. Read in ‘references.svn’.
4. graph >foo.dot. Exit reposurgeon.
5. dot
Screenshot or it didn’t happen. ;P
>Screenshot or it didn’t happen. ;P
This may take a while. The image uploader in my blog is busted due to a permissions problem at ibiblio. I’ve turned in a bug report.
Totally OT, but I did want to point out a disturbing trend. Have you noticed how many of your article titles end with an exclamation point?
>Totally OT, but I did want to point out a disturbing trend. Have you noticed how many of your article titles end with an exclamation point?
Well, yes – when I’m spoofing the names of crappy monster movies.
We had a library (very nearly a domain specific language ) where the path of execution traversed up and down a large complex tree.
It was a nightmare to debug, until we added a handler to dump the entire graph to Dot on any failure.
As Jeremy Bowers said, graphviz is a perfect example of Doing it Right. I’ve used it before. It’s really easy to autogenerate and to write by hand, and it uses the Unix text-based tradition to good effect.
> graph $% >/tmp/foo$$
>shell dot </tmp/foo$$ -Tpng | display -; rm /tmp/foo$$
Surely it would be better to allow graph to pipe into shell directly than to create and delete a tempfile.
Also, allow me to introduce you to xdot.py, a nifty little DOT viewer written in Python and PyGTK: http://xdot.jrfonseca.googlecode.com/git/xdot.py
You can pan and zoom the graph and it looks much nicer than a PNG rendered in ImageMagick. Simply replace "dot </tmp/foo$$ -Tpng | display -" with "python xdot.py /tmp/foo$$" in your example.
>Surely it would be better to allow graph to pipe into shell directly than to create and delete a tempfile.
Yes, but the reposurgeon command interpreter only partly emulates a shell – it does redirects, but not piprlines. You see a pipe in the second line because that line is handed off to a real shell for execution. I don’t think the effort required to implement full pipelines is justified by this one use.
>Also, allow me to introduce you to xdot.py,
Nice idea but it crashed on the first DOT I fed it. I think I know why and will ship a bug report.
>The stupid way to implement this feature would have been to hardwire assumptions about the image renderer and viewer into reposurgeon itself
In the interest of completeness let’s not forget the utterly braindead way; incorporate a renderer/viewer component into reposurgeon itself.
And now I am going to have nightmares tonight.
— Foo Quuxman
@Foo “In the interest of completeness let’s not forget the utterly braindead way; incorporate a renderer/viewer component into reposurgeon itself.”
My idea would be to rewrite reposurgeon as a PowerPoint plugin.
If you rename it Enterprise Reposurgeon Pro XL you could probably sell single copies for $749 each.
Pity I majored in engineering instead of marketing.
> We had a library (very nearly a domain specific language ) where the path of execution traversed up and down a large complex tree.
>
> It was a nightmare to debug, until we added a handler to dump the entire graph to Dot on any failure.
That reminds me that gprof2dot is not something built in…
http://code.google.com/p/jrfonseca/wiki/Gprof2Dot
@Jakub Narebski: That reminds me that gprof2dot is not something built in
That’s pretty cool, but it would not have helped in our case. The tree in question was not the theoretical tree implied by function calls, but rather a large data structure with functions stored in each node, and on some of the edges. The node functions would be called on the way down and the edges on the way back up. Any stack trace pointed back to walktree function, not to the last node executed.
If this sound needlessly complex, it is because I have left out significant details.
Not to rain on this one, but I have to admit that the gource repo commit visualization tool creates fascinating videos.
http://code.google.com/p/gource/wiki/Videos
You can see for example how some people focus on specific bits of a project, and others flit around…
It probably took longer to write, though.
Hello,
I want to remove the missing blobs and trees from my repo but reposurgeon chokes on read with:
fatal: unable to read destination tree (XXXXXXX)
where XXXXXXX is one of the missing trees. How do I proceed?
> How do I proceed?
Not enough information. Email me with more details, please.
Hello, Eric,
I was unable to find your e-mail, so I’ll write here. I hope that’s OK.
I have a Git repo with some missing trees and blobs. I have no backup and so I’d like to remove them completely because git gc does not work because of them. This is what I get from fsck –full:
broken link from tree 409d935d30de419083d9eceea508adec80b81bcc
to tree 87367ec02a45594b11a69bfde3f4c5b579b926cb
broken link from tree 94bd68bc14c07fc0ef3b71c5fa0428785ed7c499
to blob 871e24f05393d9b8879c5167835a4d49af55b732
broken link from tree c8de5623a3e937ae098407021ad31ad05eae6c63
to tree 869e9266cd3752441a23722fb509219847601f2f
broken link from tree 0b541e1198c2eef2234498c57bb8224427205f19
to tree cc4c8a5ef49cce800671b14a11d7f901023f2036
broken link from tree d51e86c9651abe6c14c2d534d65f6109cc07a9ac
to blob 87f15428004c061b71ba11078447364cefdd7e13
broken link from tree 8e420226ef2ed76f708087e835c06c11989bcb43
to tree 03f02016b2eb8dda9fe54e4d211f483e12e9d85a
// dangling stuf…
I tried reposurgeon on this repo and I got
fatal: unable to read destination tree (87367ec02a45594b11a69bfde3f4c5b579b926cb)
So I want to know if something can be done. Thanks.