Last week I decided the time had come to bite the bullet and systematically port the fairly large volume of Python code I maintain from Python 2 to Python 3.
I straightaway ran into a problem, which is that for my purposes the Web resources on on how to do this are pretty awful. And not just in the general, unsurprising sense of being way too full of theory and generality and way too light on practical advice, either.
No, there’s a more specific problem as well. I write systems programs, things like SRC and reposurgeon that have to be able to do string-bashing-like things on binary data without upchucking or (worse) silently mangling that data.
Due to the Python 3 decision that strings are sequences of Unicode code points rather than bytes, this is significantly more difficult in Python 3 than it was in Python 2.
However, it is possible. When A&D regular Peter Donis volunteered to help me with my porting troubles, I proposed to him that we should write a HOWTO on the process.
That HOWTO now exists: Practical Python Porting for systems programmers.
The HOWTO points to SRC and reposurgeon as worked examples, and there are new releases of both to accompany it.
Thank you for putting that together. I get the sense that by the time I do anything serious with python, much of what I learned from the online 2.7 tutorials that exist will need some tweaking.
On another note Eric, what would you have to say about the effectiveness of tools like NLTK? An online friend who is entering data science keeps insisting it’s possible to extract insights from the federal register that would be useful to policy analysts and anyone affected by onerous regulations.
The case study he cites is a common training set (the Enron emails) that is used for understanding how machine learning can be used for fraud detection. He suspects that topic modeling, tokenization, and other techniques could better illustrate to laypeople how burdensome the Federal Register is and possibly even provide an automated “legal zoom” style means of crossing the hurdles to starting/running a business.
I don’t know how realistic that is, but the inspiration came from discussion of the “Ten-Thousand Commandments” study that Competitive Enterprise Institute puts out every year.
I certainly do think it would be a boon if there was a way of making it easier to wade through federal laws *and* make it clear why they are unnecessary. Could NLTK be useful in this area?
>On another note Eric, what would you have to say about the effectiveness of tools like NLTK?
Don’t know enough about it to have an opinion.
I’m not particularly looking forward to the day I need this, but I’m glad it’s there. Thanks. Also I suppose I should really start using Python 3 for new things.
Regarding the imports, why would one not use sys.version_info to make two different import blocks?
Instead of the try/catch?
I did the same for Numpy 1.8.x. The string problems were not as severe, so I have no recommendations for that. However, I would suggest deciding up front which Python versions to support, which at this point would probably be Python 2.7, 3.4, and 3.5. Here are a few other things that can help.
– The modernize program will attempt to make your program compatible with 2.6+. I didn’t use it myself, but have seen it recommended.
– The six Python 2 and 3 compatibility library. Six is also used by modenize by default.
– Include `from future import .from __future__ import division, absolute_import, print_function` at the top of the python files unless they are script files, in which case omit `absolute_import` as the necessary context will be missing. Including `unicode_literals` was troublesome at the time of the 1.8 work, but would probably be OK if the supported Python versions are as suggested. .
As an aside, I’ve often wondered why most programming languages only have one string type.
It seems like we have dozens of numerical types: int, long, float, signed, unsigned, BigDecimal, etc. And we know all about converting from one type to another, and the dangers thereof. So why not have both ASCIIString and UTF8String?
It seems to me that differentiating between the two string types, and allowing for formal conversion between them, would have saved you guys a fair bit of headaches and hassle. Is there a formal reason that language designers prefer a single string type, but are okay with multiple numeric types?
>It seems to me that differentiating between the two string types, and allowing for formal conversion between them, would have saved you guys a fair bit of headaches and hassle.
To be fair, Python 3 does this. That problems arise because Python 2 didn’t.
It is separately questionable whether Python 2 byte strings should have syntactically mapped to Python 3 Unicode strings. The alternative, which I would have liked better, would have been for Unicode to have remained the non-default string type.
>Is there a formal reason that language designers prefer a single string type, but are okay with multiple numeric types?
I can’t think of one. I think this is just hysterical raisins.
> Is there a formal reason that language designers prefer a single string type, but are okay with multiple numeric types?
First, numeric types are mainly correspond to hardware; there is no specialized hardware for string handling. Second, there are much more encodings than numeric types, and they change more often.
And yes, third is historical reasons.
> The alternative, which I would have liked better, would have been for Unicode to have remained the non-default string type.
Except then languages other than English are second-class citizens. I don’t think there’s a good reason to prefer byte strings for text manipulation.
“I don’t think there’s a good reason to prefer byte strings for text manipulation.”
Sure there is. Principle of least surprise. Doing things to Unicode strings will fail in sometimes surprising ways, especially as you switch the language the system is working in. They’ll try hard to get it right, but it’s a ridiculously tough problem to solve.
And anyone calling for a plethora of numeric types would do well to remember PL/I’s pounds sterling datatype, designed to handle British money pre-decimalization.
@Sean C.: why would one not use sys.version_info to make two different import blocks?
In principle, one could; but I think it’s a good general principle that one should test directly for capabilities, not infer them based on other data.
@Random832: I don’t think there’s a good reason to prefer byte strings for text manipulation.
In the case of src and reposurgeon, the key point is that they aren’t doing text manipulation; they’re doing binary data manipulation that just happens to include some operations that are usually thought of as “text” operations–things like tokenizing and parsing–but are still operating on binary data. “Binary” meaning that the requirement is not to display the data in a way that makes sense to humans, but to preserve the data unchanged.
Except then languages other than English are second-class citizens.
And?
Yeah, I’m almost over that. But this still rankles:
To be clear, the problem is not that there are a plethora of string types. The problem is that they are trying to cover the use-cases of bytes while pretending they aren’t strings.
They finally brought back the % operator for these, and they are converging on strings again, but the situation with strings is still as if someone decided that the comparison of (0 == 0.0) should return False.
Python 2 was a lovely language if you didn’t need Unicode every day. Python 3 has some even more lovely bits, but shenanigans like this and the // divide operator are utter bullshit.
>Python 3 has some even more lovely bits, but shenanigans like this and the // divide operator are utter bullshit.
I cannot disagree even a little, alas.
Heh.
It’s not uncommon to see a good language go off the rails. Luckily, Python 2.7 is so good that the switch to 3 isn’t required.
C# continues to do things that boggles my mind, until I get in there and use the new features, then I go ‘oh, ok’, but Python 3 is a long string of me yelling ‘WHY!!?!’ into the void.
Y’know, if Python only had strong static typing, not only would string problems in 2-to-3 conversions be caught without having to run the affected code, but linting tools could be written that identify likely such pain points and assist in the conversion process.
Never underestimate how much of a win static typing is.
“You know, if Python 3 had added strong static typing, the the other brain-dead decisions made in the 2-3 transition could be easily found and fixed by adding butt-ugly code constructs; nobody ever used Python because they thought it made their code look nice anyway.”
It’s a ridiculously EASY problem to solve: all strings carry encoding information, and strings with different encodings are mutually incompatible without a conversion step.
Unicode strings should ABSOLUTELY be a different type than byte strings; it’s possible to break a UTF-8 string (or virtually any multibyte-encoded string) with byte-string manipulations in undesirable ways; a separate Unicode string type allows you to impose invariants such that valid operations on strings will not break the encoding.
There are drawbacks: “get the nth character of the string” is no longer O(1). Deal with it.
Surely an old mainframe hand like yourself recognizes how mission-critical proper handling of currency amounts is to business applications?
Ideally there should be a separate numeric type for any sort of quantity used by the application: so mass has its own type, length has its own type, time has its own type, etc. Appropriate dimensional analysis should be performed when quantities of different units are multiplied or divided. Quantities should, where appropriate, be coerced to the appropriate SI unit (i.e., so a quantity of 1 ft appearing as a constant in code would be coerced to 0.3048 m).
Of course, these don’t have to be language built-ins — but any halfway decent numeric type system should contain, at a minimum: signed integers that fit in a machine word (fixnums), double-precision floating-point numbers (flonums), arbitrary precision integer and floating point values (bignums), rational numbers where the N and D can be of any integral type (D nonzero), and complex numbers where the R and I can be any real type.
The advantage of having Rat (rational numbers) as a type is that you won’t be surprised that 3*(1/3) != 1. The disadvantage of using Rat by default instead of Float is that it is slower…
And yes, there are a few less known languages that use Rat.
> Under Python 2, passing a byte-buffer object to str() just gives you the object back. Under Python 3 you get a string representation of the byte-buffer: thus str(b’23′) == b’23’.
Did you intend for there to be extra quotes here,
str(b'32') == "b'32'"? I had trouble following as-is.(Also, your first quote is a unicode close-quote, not an ascii apostrophe. Looks like wordpres is doing that to me too.)
> they’re doing binary data manipulation that just happens to include some operations that are usually thought of as “text” operations–things like tokenizing and parsing–but are still operating on binary data.
That is because they have terrible data formats. If a format is going to masquerade as a text format, then it should actually be text, with binary data encoded as base64 or mime or some other such thing. Git was designed late enough that the people who built these formats should have known better.
>That is because they have terrible data formats. If a format is going to masquerade as a text format, then it should actually be text, with binary data encoded as base64 or mime or some other such thing.
The import-stream format is simpler and, I think, petty well designed. The structural parts are all flat ASCII. The contents blobs are unencoded binary which will in most cases itself be plain text; each begins with a bte count ansd a newline. It’s unambiguous, minimal, and dead simple to parse.
@Random832:
Nice fantasy. Here’s mine: If a language is going to masquerade as a general purpose computer language, then it should actually deal with the real world as it comes.
Amen, Patrick. Computers serve people, not the other way around.
Well, at least there’s a way to test whether it’s Python 2 or Python 3:
(this line is a spacer)
if(‘b’ == b’b’):
python2
else:
   python3
Apologies for horrible formatting.
>if(‘b’ == b’b’):
> python2
In a subtle way, the best joke in the thread. *applauds*
OT: Is anyone else getting 502 errors from the dowload links at http://www.ntp.org/downloads.html and http://support.ntp.org/bin/view/Main/SoftwareDownloads ?
Is anyone aware of any alternate mirrors?
Actually, if I’m interpreting the GPSD Time Service HOWTO correctly, my above post is probably moot.
I did discover something that should probably go into the special notes for the Raspberry Pi at http://www.catb.org/gpsd/installation.html , though: Raspbian’s init uses port 2947 for something or other, so you need to start GPSD on another port.
Complete tangent: What’s the accepted way of punctuating a clause or sentence ending in a URL? The natural way to do it is to have the punctuation mark be the next character after the end of the URL (e.g. “An example of a URL is http://example.com.”), but I’m always afraid that some poorly written software or other might try interpreting the punctuation as part of the URL and choke if a reader tries following the link. Leaving whitespace between the URL and the punctuation (e.g. “An example of a URL is http://example.com .”) is unambiguous, but stylistically hideous.
I use the space. Stylistic considerations come in below accuracy.
Eric:
I’m a bit confused as to whether I’m getting PPS into GPSD or not. I know the GPS chip is exporting PPS, I can run ppstest with no problems. However, gpsmon shows nothing related to PPS, and GPSD at debug level five shows none of the carrier detect messages that the Time Service HOWTO suggests should be there. If I launch cgps, I get lines like the following while cgps is running, but only while cgps is running:
gpsd:PROG: Changed mask: {ONLINE|TIME|LATLON|SPEED|TRACK|PACKET|REPORT|PPSTIME} with reliable cycle detection
Does this indicate that GPSD is actually picking up PPS, or does it mean something else? Any idea why GPSD (run as root) would be unable to pick up PPS if the kernel is receiving it?
>I’m a bit confused as to whether I’m getting PPS into GPSD or not.
Ask on the gpsd-users list, please. The person with lots of experience troubleshooting PPS issues is Gary Miller, not me.
> It’s a ridiculously EASY problem to solve: all strings carry encoding information, and strings with different encodings are mutually incompatible without a conversion step.
But these new ridiculously-easy strings are not the existing strings. Being new things, they need new names, so that the old names may continue to refer to the old things, and statements using those old names will retain their existing truth value. Doing otherwise is to render every line of old code useless in the new version unless and until it’s been re-checked for correctness with the new nomenclature.
@The Monster
> But these new ridiculously-easy strings are not the existing strings.
Jeff’s answer is certainly simple, easy and wrong, but it would have been possible to make the new strings interoperate with the old strings without the separation you describe. The separation has deliberately been done the other way in Python as a deliberate movement of the sorts of strings that esr and I have found to be extremely useful into a second-class category.
Carrying type information is fine. Disallowing comparison (or worse, as Python 3 does now, simply returning False to denote inequality) between a raw ASCII character and the same character in Unicode (especially for a character between 0x20 and 0x7F) is, as I mentioned above, akin to deciding that 0.0 is not equal to 0.
I could see a language deciding that 0.0 should not be able to be compared to 0, or a language deciding that b’a’ should not be comparable to ‘a’ (e.g. by throwing an exception). But that language would not be Python — not even Python 3 (at least when it comes to equal/not-equal). Since it is allowed, even in Python 3, to compare b’a’ to ‘a’ for equality, it would be nice if the results of that comparison were useful in a realm other than merely guaranteeing co-hashability in the same dict. It would also be nice, once there was a useful comparison in one version, not to deliberately remove that utility in the next version.
> it would have been possible to make the new strings interoperate with the old strings without the separation you describe
What I describe is not “separation”. It’s that the existing strings should be invoked with the existing syntax, and the new strings (which may include the existing strings as a subset) get a new syntax to indicate their newness. Instead, what’s been described here is that the new strings got the old syntax, greatly impairing code reuse pending the line-by-line review I’ve described.
As I said on the other thread, this reminds me of when MS decided to put 64-bit code in “C:\Program Files” and move 32-bit code to “C:\Program Files (x86)” (and introduce the corresponding WoW6432 branch of the Registry) rather than leaving the 32-bit stuff where it was and creating a new “C:\ProgramFiles64” home for the new programs. These decisions are a collective “fsck you” to all of your developers, whose old programs are instantly rendered useless.
I haven’t been reading lately about how Perl 6 is coming along, but I was impressed by the way it planned to handle this issue by explicitly scoping when the programmer wants to use new, non-backward-compatible syntax. That is exactly the right way to handle these situations.
>deciding that 0.0 is not equal to 0.
There are contexts in which I want 0.0 != 0 (or even to throw an error rather than returning false), and others in which I want 0.0 == 0. (There are even contexts in which I want 0.000000000001==0, but I digress.) Give me an API that lets me decide which I want, but make existing code not break.
@The Monster – er, the point is that when you’re actually running an old program, it sees “Program Files” and normal registry entries that point to the x86/wow64 locations. The only programs that have to care about the new locations are 64-bit programs or programs that need to interact with the real filesystem (in order to see 64-bit programs’ files).
You’ve not described a scenario in which an old program is in fact rendered useless.
@Random832
In a previous job, I frequently had to deal with the fact that 32-bit installers written before 64-bit Windows came along would fail to work on 64-bit Windows. On some level, the “point to” mechanism you’re talking about worked, but not fully and completely, resulting in programs that did not function properly. Only after those installers were rewritten as 64-bit-aware did the problems go away.
Had things been done the other way, there would have been no need to “point to” anything, and therefore everything would have just worked the way it always did.
@Monster:
Just to clarify, your talking about situations where the installer runs and puts files on disk, but does not create a viable install, right? ‘Cause the biggest cause of trouble I’ve encountered regarding to running Win32 programs in a WoW64 environment is that a ton of developers packaged Win32 programs with Win16 installers.
Hi here. I’ve just published a response to the Porting HOWTO, based on my own experience.
Since it’s rather long, you can find it here:
https://github.com/dmerejkowsky/response-to-practical-python-porting-by-esr
Feedback welcome :)
>Hi here. I’ve just published a response to the Porting HOWTO, based on my own experience
Some good thoughts there. You are right, we should have mentioned the functional-print issue, I’m adding it to the checklist now.
You’re also right, I avoided six because I wanted to minimize external dependencies.
Would you like to become a contributor to Practical Python Porting? I think we can merge in some of your other stuff, and I’m sure Peter wouldn’t object to having another expert on board.
> Would you like to become a contributor to Practical Python Porting?
I would be honored, thanks. I do think we can merge in some stuff. I just created a gitlab account, I can send a pull request later this week if you like.
>I can send a pull request later this week if you like.
Please do.
@Jon Brase It’s been a few years now, but my recollection is that things like %PATH% settings, .ini file contents, Registry entries, and shortcuts that ought to reference a subdirectory of “c:\Program Files (x86)” instead had the corresponding “c:\Program Files” subdirectory, and therefore didn’t work.
This was precisely because some Clever Person at Microsoft thought that telling the installer a bunch of lies, and translating those lies to “point to” the real locations was a good idea, rather than the far simpler solution of leaving the existing well-known directories the hell alone and creating new ones for the expanded instruction set (such as system32, which ironically now holds 64-bit code).
That 32-bit code uses the old op codes, and is thus 100% backward compatible, while the 64-bit code uses new op codes. The AMD designers who created those new op codes knew better than to change the meaning of existing op codes. Guido et. al. apparently think they are smarter than AMD.
The MS team charged with implementing 64-bit Windows decided to put code with the new op codes in the old places, and the code with the old op codes in new places, then construct some epicycles to cover up the added complexity. On what planet does that make sense?
@esr: I’m sure Peter wouldn’t object to having another expert on board.
No objection at all. dmerej has some good items that should be included in the HOWTO.
>No objection at all. dmerej has some good items that should be included in the HOWTO.
Been done – take a look at version 1.2. I think that spans what we can merge in without changing the overall strategy described in the document.
@dmerej: Feedback welcome :)
Certainly. :-)
One comment on “UTF-8 everywhere”: your use case, as you note, is somewhat different from the use case assumed in the HOWTO. You actually care about the semantic meaning of bytes other than the ASCII range 0x00..0x7f. For this use case, yes, you need an encoding that can represent any Unicode code point, which Latin-1 obviously can’t. But you also have to accept the fact that your program will not work correctly if it is fed input bytes that are not UTF-8 encoded; in fact there will be byte sequences that cause it to throw a UnicodeDecodeError, since there are sequences of bytes that are not legal UTF-8. The advantage of Latin-1 encoding, for the use case assumed in the HOWTO, where we don’t care about the semantic meaning of bytes 0x80..0xff, is that every possible sequence of bytes is valid Latin-1.
So I’m not sure the “UTF-8 everywhere” advice is really a “don’t agree”; I think it’s more of a “different use cases require different solutions”.
@dmerej:
One other comment, on this:
“Instead, make sure that your string is UTF-8 encoded before sending it to sys.stdout or sys.stderr.”
In Python 3, the sys.stdin/out/err streams are Unicode, not bytes, so you don’t encode anything, you just write Unicode strings to them. But you also don’t control the encoding of those streams; it will depend on the system’s locale. If all of your users happen to have system locales that set UTF-8 as the encoding, you’re fine. If not, you might have problems if your code assumes that everything is UTF-8.
Unfortunately, there is currently no way in Python (2.7 or 3) to set the encoding of the standard streams, because the only way to do it for any Unicode stream is to do it in the constructor, and you don’t control that process for the standard streams. There is an open issue in the Python bug tracker to add the ability to set the encoding of streams after they are constructed, but it looks like the earliest it might be available is Python 3.6:
https://bugs.python.org/issue15216
The other thing to be aware of with the standard streams is that they have two very different uses (at least on Unix–I don’t know how much the second use comes into play on Windows): console I/O for interactive use, and pipes. The former is really the only case where the system locale is even relevant; but Python sets the encoding of the streams the same whether they are TTYs or not. So the best solution, once again, depends on your use case.
@esr: take a look at version 1.2
I pushed a few small edits. Overall looks good.
The core problem here was that the architectural decisions for 64-bit Windows XP were made for (Alpha and) Itanium. So instead of taking the x86-32 Windows design and naturally extending it to handle x86-64 code, you had a port of the x86-32 code to alien 64-bit processors, and then an x86-32 backwards compatibility module added to the system as a second-class citizen. On Itanium, allowing native-compiled software to target the “Program Files” directory without any special name-mangling was perfectly natural, as was segregating the non-native x86 software that would only run in compatibility mode into a “Program Files (x86)” directory.
Then, with Microsoft taken by surprise by the flop of Itanium and AMD’s introduction of x86-64, this Itanium version of Windows XP was quickly ported to x86-64. And now because Microsoft wasn’t willing to break compatibility with existing “64-bit Windows”, the design decisions that were made for non-x86 platforms were now built-in assumptions of the software environment of the x86-64 platform. Running foreign code in emulation on any system is always a bitch, and the Win64 design assumed that was the only way you’d ever be running 32-bit code on 64-bit Windows.
(Which is why it wasn’t until Windows 7 that they were willing to recommend 64-bit for consumer use; it took years to file down the bumps and for third-party software to adapt to the new environment and 64-bit versions to be ready. They probably would have advised waiting even longer than Windows 7 if systems hadn’t been shoved up hard against the 3.2 GB RAM barrier of 32-bit Windows.)
Which wouldn’t have been a problem if Windows had been extended from x86-32 to x86-64; for example, WINE on 64-bit Linux manages 16-bit Windows code just fine. But it would have been a low-return bitch to make 16-bit x86 code run on Itanium servers and workstations, so the architects of 64-bit Windows XP didn’t worry about it.