How To Learn Hacking: Version 1.2, with a new section on being original. Incorporates more feedback from here and G++
For those of you who wondered why this didn’t just become a major section in How To Become A Hacker, it’s because I think it might become long enough to make that document too bulky to read at one sitting.
Nice. Allow me to make, as usual, a couple of minor corrections.
The very first sentence of the new section includes the word “lean”, but the context suggests it should have been “learn” (“lean how to recombine musical patterns”). And in the second paragraph, you wrote: “at increaing complexity scales [emphasis added]”.
I couldn’t correct the content of what you write even if I wanted to, due to a lack of knowledge; and your writing style is, IMHO, always–or at least nearly always–impeccable. Keep up the good work! ^_^
http://www.catb.org/esr/whatsnew.html also has some mistakes. (secirity, ahipper, addes)
The page you linked to led me to “The Parable of the Sheep”, wich I hadn’t read. It’s beautiful, so thanks. I *had* read a different version, though; in case anyone’s interested, here it is.
Such writings are eerily relevant to my country (Argentina) nowadays: our authorities enable criminals–even rewarding people in jail by granting them a “salary”!–but make it quite difficult for us law-abiding citizens to protect ourselves.
Under “The Incremental-Hacking Cycle:”
1) In step (5) consider mentioning this: when making a change, use the existing coding style. There are practical reasons for this, which you may want to mention as well.
2) Regarding steps (4) and (5):
Why does (5) emphasis documenting ones change when (4) already included this: “It’s a good exercise to add explanatory comments and notes to the code as you figure out things about it. This will help your memory, and will help you organize your thoughts as well.”?
And while (4) mentions _where_ the comments go (in the code) and _how_ (by adding explanatory notes), (5) does not describe where and how documenting ones change is to be done (e.g., in a separate document to be submitted with the change–perhaps a README file?). And exactly what is one documenting? For whom?
3) Step (4) deals with seaching the code while (5) deals with making ones change. Should not this sentence be moved from (4) to (5)? “It’s a good exercise to add explanatory comments and notes to the code as you figure out things about it. This will help your memory, and will help you organize your thoughts as well.” Further (and this may be putting too fine a point on it): should “practice” be used instead of “exercise?” “
Not related to this document, but related tangentially as documents-ESR-maintains: does anyone other than me have character encoding issues with the Jargon File? I get garbage characters unless I explicitly set the encoding in my browser to Western.
The wc3 i8n checker reports conflicting character encodings between the page source and the http headers, e.g. here: http://validator.w3.org/i18n-checker/check?uri=catb.org%2Fjargon%2Fhtml%2FB%2Fbit-rot.html#validate-by-uri+
>does anyone other than me have character encoding issues with the Jargon File?
Everyone. It’s a toolchain problem I haven’t been able to fix yet.
> It’s a toolchain problem I haven’t been able to fix yet.
Okay; I suppose you don’t need another bug report on it then. Mind if I ask what the toolchain for producing the JF looks like, and where the problem appears to lie? Curiosity.
>Mind if I ask what the toolchain for producing the JF looks like, and where the problem appears to lie?
XMLDocBook. It generates a wrong character-set header.
I once had a sign…
“I don’t write complex code. I simplify complex designs and write simple code”.
This is the point of refactoring. Sometimes you have a blob that “works”. Maybe not efficiently, well, etc. but there is no specific design. Refactoring clarifies.
You don’t have a reference to it, but what you are referring to is “The Quality Plateau”.
I think of it as something like 3rd normal form for procedural programming. Where you cannot refactor further (can only trade-off, but not optimize). The Ah! moment.
Two cars I owned might show the difference. One was an AMC that had a carburetor, and had all kinds of tubes (vacuum control, but not vacuum tubes :) ). It was incomprehensible. The other was a Subaru that everything was clean, small, and the things I would need to touch were yellow.
Hacker refactoring is part of the understanding process. Instead of a widespread set of lines of code which are redundant, complex, etc. you reduce it to the minimal equivalent form. The sort of thing like a huge equation taking up an entire wall is reduced to X**2=2 after going through various transforms. You can’t understand 4000 lines, so take 4 redundant lines and make 1 line out of it, so you have 1000 lines. You can understand the strand of spaghetti, and straighten it out. Once enough things are straight, things clear up.
In a recent case, a buried set of “timer = 1000; … if( –timer )” entries were replaced with “eventtime = currenttime+1000 … if (eventtime >= currenttime)” that makes lots of nasty stuff just disappear.
Something ambiguous, wasteful, or complex but you can figure out what it is getting at is replaced with something simple, efficient and clean. Rinse, lather, and repeat.
I can’t fix a pile of spaghetti. I can reduce a pile of spaghetti into an equivalent linear run and fix the linear run of code.
ESR, tz, et al – What guides the size of a given refactoring step? Is it an intersection of skill, opportunity, and understanding of the nearby codebase?
I’ve been trying to apply what our host has explained, and I’m finding myself making a whole heap of baby-step refactors, and I’m wondering if possibly a more extensive, higher-level refactoring would obviate a whole bunch of the baby-step changes.
>I’ve been trying to apply what our host has explained, and I’m finding myself making a whole heap of baby-step refactors
That is perfectly normal in the earlier parts of the process. Relax: the larger steps will come in their time.
So I AM trying to rush things. Thanks.
> XMLDocBook. It generates a wrong character-set header.
It looks fine today… did you fix this recently?
Off-topic: I’d rather here about your rants on newer languages like Rust and Go. Also, I think the original “how to become a hacker” is somewhat outdated on its suggestion about programming languages. It still suggests Perl as a language of choice. I rarely see anyone use it anymore. Could we possibly at least get a blog post on that one some time?
Eric, I think it may be helpful for your text to address the question: What is the proper use of theory and books in practicing the incremental hacking style effectively? In How to Become a Hacker, you caution against relying on books too much. But one can also rely on them too little. How would an apprentice hacker find the Goldilocks zone here?
For a concrete example to clarify the question, suppose that a reader of your howto practices incremental hacking by reading and rewriting some standard Unix tools she uses. Consistent with your advice, she starts with small ones like true(1), cat(1), and ls(1), gradually working her way up to medium ones like bc(1) and chunks of behemoths like cc(1). Pretty soon in this process, she meets cksum(1).
As you know, cksum is on the tiny side of the Unix-tool spectrum. It merely reads all the bytes in a file, applies a trivial arithmetic operation on them, and prints out two integers as the result. Even inexperienced apprentices can code it easily.
But why does cksum perform this trivial arithmetic operation and not that one? While the tool is easy to code, your budding hacker won’t, on first reading the code, understand why it’s doing what it does. To gain the understanding she needs, she will need to hit a book, look up what a cyclic redundancy check is, learn how cyclic redundancy checks are useful in error correction, understand how the particular check in cksum makes sense, and explore what other checks might replace it, if any. After she gets all these points, she can go back to writing code and will be better at it for her investment in theory. But it shouldn’t take her longer than it has to.
So what strategy do you recommend for your readers here? What should they do to end up hitting the books hard enough to understand the theory of their code’s operation, but not so hard they never return to their main task of exploratory programming? I sense that the readers of your howto would benefit from a few paragraphs of guidance about the use and abuse of books in the incremental-hacking process.
Refactoring is a process, not an event. There is no “size”. Either the code is in a mathematically reduce-able form or it is not. See “The Quality Plateau” from “Thinking about Programming” from “The Programmer’s Stone”. And reducing it in one pass may expose more redundancies and inefficiencies. Sometimes things at a global level are redundant, but you will only see that when the local level is cleaned up. Even code of which I have no knowledge of what it does, I can ask, can I rewrite these N lines in M lines, M is much smaller than N. Can I eliminate any variables? Can I turn a global into a static.
For a lot of the deeper opensource (BSD, Linux) kernels, the answer will be “no”. For lots of other things, the answer will be “yes”. But the key thing is when you refactor, you build a mental map and UNDERSTAND the code, since the new lines have to have the identical behavior to the old lines.
Right now there is a new push for “standards” where I work. It has so much technical debt that it should be declared bankrupt, but like many systems it is TBTF (too big to fail). Yet the current proposal is to leave all the existing resource hogs alone, but to apply the exact Pantone color and amount of lipstick on new pigs added to the sty.
But adding a few dozen robo-documented headers and comments to 1000 lines (and making sure the curly braces are in the right places) is actually counterproductive when those 1000 lines could be reduce to 200 or even 100 with a proper refactoring. I can understand 100 lines usually at the first read. 1000 lines, even of the clearest, cleanest code requires a large effort. Complexity is exponential, not linear.
Even things like subroutines/functions. If it is static, and called from exactly one place, there should be a reason for it not to be inlined, and instead sending context across a boundary (you know it is wrong when there are more arguments to the called function than variables in the caller). There may be a reason but it is never in the function header. Usually it is a bad echo of the What. Occasionally it has a bit of How (a particular algorithm). Rarely does it say Why.
And the “Why” is important. When something is fully reduced, exceptions stand out and cause me to pause and ask why, and usually there is a comment saying it needs to be a certain way to address some external bug (software, compiler, hardware) or some side-effect behavior that couldn’t be discerned any other way.
When the code is hairy and bloated – and this does not mean it isn’t neat, or uncommented, or un-“documented”, these redundancies, inefficiencies, and the rest occur every few lines. Someone used an extra variable because they felt like it or didn’t bother cleaning it up. They though 50 lines were too much for a(), so at the end of a(), it just does return a_1(*entire context);, and a_1 in turn does the same with a_1_1. But the coding standards say the books can only be less than 20 pages long even if the storyline is perfectly linear.
The single upside of being in the embedded world is that usually such practices simply blow things up. Features need to be added and they eventually exhaust resources. An ATtiny85 (Arduino Trinket) has only 8,192 bytes of flash and a lot of that is for the bootloader. And 512 bytes of RAM. And I can get them to do incredible things (like reading multiple sensors and displaying the results on an LCD over I2C). No, I can’t add a ram chip or wait for Moore’s law to let everyone replace their black-boxes.
And the important thing is what I said about complexity being exponential. When I throw out a ball of tangled spaghetti – comments, data structures, and all, and replace it with a clean, clear implementation an order of magnitude smaller, no one complains they can’t understand it. Every line does something specific, every variable has specific meaning, and often there is no need to comment those.
There needs to be a new UNIX Koan for comments. Similar to the KLoC problem, the push for comments (headers or inline) at best expresses intent in (buggy?) opaque code, or is redundant in clear code. If in its native C form, the code is readable and understandable, then why port it to English? Assuming you could maintain the two codebases and keep them in sync.
And meticulously documenting a bad design will not change it into a good design.
Tz, ESR, thanks for the reassurance I’m not on a hiding to sweet bugger all.
Tz, I understand that refactoring is a process – I was asking about the size of individual steps in said process.
As I said earlier, I’m still doing a heap of baby steps – ESR said that’s normal, just keep buggering onwards, you’re still knowing the codebase better.
I’m still only clearing away the most obvious underbrush with each pass (each involving a whole bunch of small steps), although I have had a single “a HA! … I can see what needs fixing, but damned if I know how except by heap of baby steps or nukenpave” moment.
>I’m still only clearing away the most obvious underbrush with each pass (each involving a whole bunch of small steps), although I have had a single “a HA! … I can see what needs fixing, but damned if I know how except by heap of baby steps or nukenpave” moment.
Again, entirely normal. The kind of aha you are having is clue that you are about 20% in.
“When you master that level, go to the order of magnitude, 1000-5000 lines.”
From context, is this supposed to be “… go to the next order of magnitude…” ?
> 2) Regarding steps (4) and (5):
The difference in “documenting” or “adding notes” between (4) and (5) is that in (4) you’re reading code, and making notes about what you read. You’re annotating or documenting other people’s code, you’re adding the comments you wish the previous coder had put there.
In (5), you document your own changes;
I just sent you an up to date translation of this document Eric, did you get it?
>I just sent you an up to date translation of this document Eric, did you get it?
I’m not currently at my comp, I will send you soon the full French translation of this document.
Here is for now the translation of “How to Become a Hacker” in the case you didn’t receive it : http://www.leblogduhacker.fr/hacker.html
I will update this translation everytime you update the original document.
Please not that the Arabic, Belorussian, Dutch and Norwegian translations seem also down (404).
Thanks for your great job!
Here is your document translated in French : http://www.leblogduhacker.fr/hacking-howtofr.html
Please let me know if you need any modification.
Did you see my last comments?