If you were reading A&D a year ago, you may recall that I invented a new version-control system to occupy an odd little niche that none of the exiting ones serve very well.
Well, actually, it’s a shell around a very old version-control system that makes a reasonable fast version-storage manager but has a crappy UI. Thus, SRC – RCS reloaded, with a mission to serve cases where you don’t want per-directory changesets but prefer each file to have its own separate change history. Like a directory full of separate FAQs, or your ~/bin full of little scripts.
SRC gives you a modern UI in the svn/hg/git style (but much, much simpler than git’s) and lockless operation. It has full embedded documentation and an Emacs VC backend. If your little project goes multi-file, you can instantly fast-export to git.
Today I shipped Version 1.0. This could have happened sooner, but I’ve been focusing on NTPsec pretty hard in the last year. There was one odd bug in the behavior of multi-file commands that I just hadn’t got around to fixing. (Yes, you can do multi-file commands, but the files still have separate histories.)
The whole thing is just 2KLOC of Python, and that’s with the rather extensive embedded documentation. The sort of person who frequents this blog might find the FAQ entertaining.
I don’t see a Code of Conduct in the repository. Why are you implacably hostile to $VICTIM_GROUP ?
“Well, actually, it’s a shell around a very old version-control system that makes a reasonable fast version-storage manager but has a crappy UI”
Ironically in that respect it replicates your intimate fiend CVS ;-)
>Ironically in that respect it replicates your intimate fiend CVS ;-)
Sadly, no, CVS versions before 1.12 are crappy storage managers, too, because they lack the magic cookies required to unambiguously group file modifications into changesets. If not for this problem cvs-fast-export would have a well-defined behavioral target everywhere and be hugely simpler.
@Parallel: Go away, that’s fascist.
Is there any particular reason you’re sticking to python2.x for a somewhat “new” project? Why not use 3? And in 2, why are you doing things like your command line parsing by hand manually rather than using something that comes with the standard library?
>Is there any particular reason you’re sticking to python2.x for a somewhat “new” project?
It’s sufficient and it’s what I’m used to.
>And in 2, why are you doing things like your command line parsing by hand manually rather than using something that comes with the standard library?
Because neither getopt nor optparse are a good fit for SRC syntax. Anyway, I consider optparse rather overengineered and ugly and tend to avoid it; most of my Python programs use getopt.
This seems to have branching but no merging. Is this true (or am I missing something)? Is it an intentional design choice if true?
>This seems to have branching but no merging. Is this true (or am I missing something)? Is it an intentional design choice if true?
It’s not intentional exactly. It reflects a limitation in RCS, which doesn’t have a branch merge capability in anything like the modern sense – rcs merge is tricky and fragile, more like an RCS-sensitive patch(1). I’d have to write all the merge logic in the upper level, and it would probably about triple the complexity.
Still, I might do it if there’s demand. Is yours a theoretical request or a practical one?
I started implementing something similar some time ago. The idea was to have a “universal VCS”; providing a consistent set of commands for all VCSs as far as they could be made to work similarly. E.g. type “urc log” (urc = Universal Revision Control) and it would autodetect the VCS in use in the current directory and execute the appropriate log command, formatting the output to be also consistent with git. It hasn’t got around to supporting revision specifications, branches or anything that sophisticated, though, so it was never full-featured enough to release.
My reason for doing that was to support SCCS, which I still use for small single-file projects like your typical use case for src. Yes, stop laughing over there – it’s a hangover from my time at Sun when SCCS was the *only* supported VCS, on some historical repos that I’ve never got around to converting. So when that SCCS backend is written, I’ll start using it immediately…
>My reason for doing that was to support SCCS, which I still use for small single-file projects like your typical use case for src. Yes, stop laughing over there
I’m not laughing. I deliberately wrote SRC to isolate the RCS-specific stuff as much as possible in one back-end class, and the FAQ says “What *will* be funny is when we implement the back end to talk to SCCS.”
Do you know Python? If so, I bet you could write the SCCS back end in a day, tops. Please do it – I have the regression tests to verify correct operation, which is probably a bigger deal than writing the code.
The laughter I’d anticipated was that of amusement/derision at someone still seriously using SCCS in 2016…
I’m not really fluent in Python, but I’ll have a look at the backend anyway. It’ll be worth it just for the kudos of contributing to an ESR project.
>I’m not really fluent in Python, but I’ll have a look at the backend anyway.
I’ve just made it easier, If your look at the repo tip version you’ll see there’s now a class named SCCS full of stub methods. If you fill those in, modeling them on the methods in the preceding RCS class, we’ll have a working back end.
In most cases you’ll merely need to write an SCCS command invocation. The parse() method needs to scrape metadata from an SCCS log output – that’ll be most of your work.
There may be a gotcha near the is_branch() method in RevisionMixin – I don’t know if SCCS uses the same format for branch sticky tags as RCS, and it’s possible there are no branch stickies at all.
SRC over zit (http://git.oblomov.eu/zit / https://github.com/Oblomov/zit)?
I don’t see a Code of Conduct in the repository. Why are you implacably hostile to $VICTIM_GROUP ?
Because they refuse to specify themselves!
The question is pretty theoretical, since my use-case for src is just a linear history for little scripts and dotfiles. I know that, if you’re not a DAG-based VCS, you have trouble finding the ancestor for diff3 – but even naive ancestor-finding diff3s would be mildly useful.
@ esr
Well done! You’re a tireless coder and maintainer.
> [Python 2.x]’s sufficient and it’s what I’m used to.
And yet, in the current version of “How to Become a Hacker”, you point newbies at tutorials for Python 3.x. I guess that means that adopting the latter is the sensible choice for those who have yet to learn the language.
Similarly, you’ve stopped recommending XHTML, probably because of HTML5 (which HTML Dog now teaches, along with CSS3 and even JavaScript; so you might want to readmit it into your how-to).
@ James M
> stop laughing over there
Like ESR, I didn’t laugh; everyone has the right to use whatever tools they prefer.
Besides, I couldn’t laugh even if I wanted to: I’m still using a CRT monitor. After all, why fix that which is not broken? :-)
“I don’t see a Code of Conduct in the repository. Why are you implacably hostile to $VICTIM_GROUP ?”
Indeed, if esr’s Code of Merit is taking longer than expected (cause you know, he is actually engaged in contributing useful things to society and human kind as a whole, rather than having to waste his extremely valuable time in stating in writing things that all sane adults already abide by), this is something worth looking at: maybe to stick to any new (and existing) project.
https://github.com/domgetter/NCoC
http://voxday.blogspot.com.ar/2016/01/no-sjws-allowed.html
http://voxday.blogspot.com.ar/2016/01/more-sjw-attacks-in-tech.html
https://archive.is/tCRse
A must watch: https://youtu.be/dYu6qhd88_M
[Warning] If you fancy a sudden urge to vomit uncontrollably:
https://bugs.ruby-lang.org/issues/12004
Powered by Debian:
https://www.debian.org/intro/diversity
https://wiki.debian.org/AntiHarassment
http://blog.shrub.com/check-my-what/
http://lafalafu.com/krc/privilege.html
————————————————————————————-
For the record: I’m in no way interested in turning this blog entry into yet another waste of time and energy, so please esr erase this comment at your discretion. (Just wanted to let you have those links because I think they might be of great importance to you and the ongoing attack on the FOSS movement and culture. I’m well aware you don’t have the time deal with the amount of BS all by yourself. Just trying to help doing my part.
> Indeed, if esr’s Code of Merit is taking longer than expected […] this is something worth looking at: […]
For the middle of the road there is thought provoking and with some smart observations
http://sarah.thesharps.us/2015/10/06/what-makes-a-good-community/
Unfortunately it is halfway thorugh the post poisoned by SJW-isms and weasel-words (like “checking ones own privilege”, “microagression”, etc.). All in all, I think it is worth [critical] reading.
>> Is there any particular reason you’re sticking to python 2.x for a somewhat “new” project?
> It’s sufficient and it’s what I’m used to.
That was unexpected, but it does look like Python 2 still has a few years left in it:
When can we expect Python 2 to be a purely historical relic?
http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html#when-can-we-expect-python-2-to-be-a-purely-historical-relic
>That was unexpected, but it does look like Python 2 still has a few years left in it:
On the other hand, I just moved sshexport – the repo tip version is now in Python 3.
This was a finger exercise for when I have to port the really big stuff, like reposurgeon. I also just moved coverity-submit.
> On the other hand, I just moved sshexport – the repo tip version is now in Python 3
Wouldn’t it be better to have `python2` and `python3` branches instead?
>Wouldn’t it be better to have `python2` and `python3` branches instead?
In theory. Is there any major distro now on which “#!/usr/bin/env python3” fails to do the right thing?
Personally, I really dislike the env hack… I wish that the #! ‘protocol’ were A) smart enough not to need it and B) flexible enough that it doesn’t take away the one argument you’re allowed to pass in (a la “perl -w”)
It should actually be fairly easy to keep SRC working on both Python 2 and 3 at the same time (`#!/usr/bin/env python`). I had a branch once that did just that, I think it was pre-0.10. Perhaps I’ll resurrect it :)
>It should actually be fairly easy to keep SRC working on both Python 2 and 3 at the same time (`#!/usr/bin/env python`). I had a branch once that did just that, I think it was pre-0.10. Perhaps I’ll resurrect it :)
After limbering up with some successful py3k conversions of smaller Python projects (sshexport, coverity-submit, ski, irker), I just had a go at forward-porting reposurgeon and failed horribly. If you can manage it I will be seriously impressed.
What’s the main sticking point in a port of reposurgeon?
>What’s the main sticking point in a port of reposurgeon?
It’s 11.6KLOC of algorithmically dense code.
I can’t get a freshly-cloned reposurgeon to ‘make check’ – I get a huge pile of pylint noise and then “[pylint] Error 24” – is there some list of messages that need to be ignored from pylint or does 24 mean something else is going wrong?
Something else is going on. pylint passes here, I just checked.
Having gotten around that (and installed the “realpath” command, which was not documented as being required), I get the following error:
— bs.chk 2016-01-27 16:02:00.140712871 -0500
+++ /tmp/stripped30609 2016-01-27 16:28:59.845836383 -0500
@@ -1,3 +1,4 @@
+test-repo: No such file or directory
SVN-fs-dump-format-version: 2
>Having gotten around that (and installed the “realpath” command, which was not documented as being required), I get the following error:
Weird. What is your environment?
OT (about gpsd): Hmmm… if I understand it correctly NMEA 2000 is intimately tied to the CAN (Controller Area Network), all the while “there has also been a shift away from outdated network buses like Controller Area Network (CAN) to more robust alternatives like Ethernet Audio Video Bridging (AVB)” (in cars at least).
Ubuntu 14.04.3 LTS – with pylint 1.5.4 installed from pip because the version of pylint in the Ubuntu repository is broken.
The pylint messages I had to disable to get it to pass were:
1 (reposurgeon): R0101,C0410,C0113,C0121,R0204
2 (repodiffer): C0410,C0121,C0113
3 (repomapper): C0410
4 (repocutter): C0410,C0413,C0121,C0123,R0101
Though IMO some of these could be fixed instead of ignored… I’m not sure what’s the point of running pylint at all if you’re going to ignore everything your code doesn’t pass.
(Also, is there a reason for all the –quiet and @ in the makefile? Makes it harder to track where things go wrong.)
Realpath itself is available in the repository, but not installed by default. The package description says “This utility provides mostly the same functionality as `/bin/readlink -f’ in the coreutils package.” – could whichever test this affected be rewritten to use readlink?
> I’m not sure what’s the point of running pylint at all if you’re going to ignore everything your code doesn’t pass.
A lot of its warnings are silly; you need to suppress those. But it does catch real problems, which is why I keep using it.
>could whichever test this affected be rewritten to use readlink?
Possibly. I’m not sure that’s any less of a problem as a dependency, though.
(Yesterday, I was just having it skip pylint – today, I took the time to ignore each message number that came up, and doing so made it run successfully, so it’s probably simply a matter of a newer pylint than yours checking for more things.)
> it’s probably simply a matter of a newer pylint than yours checking for more things
That’s what it is. The repo tip version now suppresses most of those warnings and does cleanups for others.
> It’s 11.6KLOC of algorithmically dense code.
I meant something like, is there any particular class of python2/python3 incompatibility that accounts for most of its portability problems.
>I meant something like, is there any particular class of python2/python3 incompatibility that accounts for most of its portability problems.
String vs. unicode issues, as usual. Everything else in a 2 to 3 port is trivial stuff like changing module names.
I don’t program in Python3, but does it mean that unicode stuff in Python3 is misdesigned, because UTF-8 is not loseless / full encoding?
Maybe they should have went Perl 6 route with 8-bit safe UTF-8 variant (via surrogate codes for things outside UTF-8).
>I don’t program in Python3, but does it mean that unicode stuff in Python3 is misdesigned, because UTF-8 is not loseless / full encoding?
No, that’s not the problem. The problem is that in Python 2, strings were uninterpreted bytes. but in Python 3 they are Unicode code points. There’s a new type ‘bytes’ that has the old uninterpreted-byte-string behavior. There’s no implicit conversion between them; you have to do explicit encode and decode operations.
This change wreaks merry hell on any Python 2 code doing more than trivial string-bashing – Python 3 throws obscure exceptions when it sees bytes where it’s expecting strings and vice-versa. Where it hurts most is programs like reposurgeon that want to treat their data as uninterpreted bytes. You get partway by opening files in binary mode, which causes data from them to be type byte rather than type string, but resolving all the internal places where there need to be explicit type conversion between byte and string is a huge pain in the ass.
Difficulty tends to rise with square of program size – easy for small programs like sshexport or coverity-submit, horrifying for anything the size of reposurgeon or doclifter.
> The problem is that in Python 2, strings were uninterpreted bytes. but in Python 3 they are Unicode code points. There’s a new type ‘bytes’ that has the old uninterpreted-byte-string behavior. There’s no implicit conversion between them; you have to do explicit encode and decode operations.
But isn’t the problem that you need explicit conversion, because (I guess here) the conversion is lossy, because not every byte sequence has Unicode code point encoding (UTF-8, or whatever internal encoding Python3 uses; it might be UTF-16 / USC-2)?
Perl6 solved this in a bit strange way. Because Str there implements by default NFG (Normal Form Grapheme), i.e. one character is one graph, it needs synthetic Unicode code points (negative code points IIUC) for encoding those graphemes that do not have precomposed form (CR LF is one such grapheme). As a side effect then it can encode bytes that do not correspond to valid UTF-8 characters as synthetic Unicode code points, so that conversion Blob -> Str -> Blob is idempotent.
P.S. Nice that preview doesn’t now obscure “Post Comment” button…
>But isn’t the problem that you need explicit conversion, because (I guess here) the conversion is lossy, because not every byte sequence has Unicode code point encoding (UTF-8, or whatever internal encoding Python3 uses; it might be UTF-16 / USC-2)?
You do need explicit conversion, yes.
The porting problem arises because all your old code using string functions is now applied to bytes objects and has to be adjusted in unobvious ways, not just at the conversion points. It’s a huge PITA.
> The problem is that in Python 2, strings were uninterpreted bytes. but in Python 3 they are Unicode code points. There’s a new type ‘bytes’ that has the old uninterpreted-byte-string behavior. There’s no implicit conversion between them; you have to do explicit encode and decode operations.
Why didn’t they keep “strings” exactly the same (since the behavior would be identical between both things with the same name), and add a new type “u_string” etc. to be this new Unicode string type with new behavior? This ranks right up there with Microsoft’s “Program Files(x86)” and the corresponding WoW6432 Registry crap as exactly the wrong way to do things. Had the old 32-bit programs stayed where they were, and new explicit 64-bit locations for the new ones been created, then no programs would have been broken by being installed on a 64-bit OS. Similarly, if the two string-y types’ names were swapped, old Python code would tend to Just Work™ without needing to be rewritten.
> Indeed, if esr’s Code of Merit is taking longer than expected (cause you know, he is actually engaged in contributing useful things to society and human kind as a whole, rather than having to waste his extremely valuable time in stating in writing things that all sane adults already abide by), this is something worth looking at: maybe to stick to any new (and existing) project
Yet another nice blog post (from LedgerSMB dev):
http://ledgersmbdev.blogspot.com/2016/01/on-contributor-codes-of-conduct-and.html
There are other changes in Python 3 than strings and bytes that would not necessarily mean Python 2 code “Just Works” in the new one. it’s really a fundamental redesign of the language, and the removal of legacy baggage compared to the old. (I recall somewhere, however, the Python devs are less than impressed with the migration rate and Python 4 is highly unlikely to be incompatible with 3)
When redesigning the language, making it internally-consistent is perfectly good and fine.
@esr:
I have some experience with this on a couple of string-intensive projects, e.g. astor and pdfrw. The most string-intensive is pdfrw, which can read and write and manipulate PDFs.
It was practically impossible to maintain a common codebase across 2 and 3 until the devs relented and added enough backward compatibility to 3.3. I don’t even try to support 3.0 – 3.2.
From my perspective, the best thing to do is to do everything in native strings (which are essentially bytes on Python 2 and unicode on Python 3), and have conversions (or not) right at the file I/O layer.
This approach gives a 20% speed hit on Python 3, but it saves you from doing ugly and error-prone byte literals everywhere.
There is a project called six that theoretically allows you to write code that works in both versions. It uses a bit of voo-doo to wire itself into the interpreter, so now you really have 3 languages: Python 2, Python 3, and six. And potentially incompatible versions of six, used by different libraries.
Rather than all that, I use a fairly simple duck-typing style to handle 2 vs. 3 differences. E.g. for pdfrw, all the differences are incorporated here:
https://github.com/pmaupin/pdfrw/blob/master/pdfrw/py23_diffs.py
@Mike Swanson:
True, but most of those are much easier to work around than the string issues.
I wouldn’t call it fundamental, except in the sense that there’s a lot of purity testing going on. The designers obviously feel that there is no room in the universe for quasi-string/quasi-byte data (e.g. evaluate (‘a’ == b’a’) in Python 2 and Python 3), and that doing what third graders expect is more important than reducing mental friction for professional programmers who work in multiple languages (e.g. evaluate (3 / 4)).
The mandatory changes in 3.0 were a big “Fuck you!” to third party library supporters. What the hell did they think was going to happen? Notice that they’re still (slowly) fixing these fuckups — 3.5 now brings back the % string formatting operator, for example. The designers forgot or never learned the second part of Einstein’s formulation that starts with “Things should be made as simple as possible”.
There’s consistency, and then there’s “a foolish consistency” aka most lisp variants.