Maybe you’re one of the tiny minority of programmers that, like me, already enjoys writing documentation and works hard at doing it right. If so,the rest of this essay is not for you and you can skip it.
Otherwise, you might want to re-read (or at least re-skim) Ground-Truth Documents before continuing. Because ground-truth documents are a special case of a more general reason why you might want to try to change your mindset about documentation.
In that earlier essay I used the term “knowledge capture” in passing. This is a term of art from AI; it refers to the process of extracting domain knowledge from the heads of human experts into a form that can be expressed as an algorithm executable by the literalistic logic of a computer.
What I invite you to think about now is how writing documentation for software you are working on can save you pain and effort by (a) capturing knowledge you have but don’t know you have, and (b) eliciting knowledge that you have not yet developed.
Humans, including me and you, are sloppy and analogical thinkers who tend to solve problems by pattern-matching against noisy data first and checking our intuitions with logic after the fact (if we actually get that far). There’s no point in protesting that it shouldn’t be that way, that we should use rigorous logic all the way down, because our brains simply aren’t wired for that. Evolved cognition is a kludge – more properly, multiple stacks of kludges – developed under selection to be just barely adequate at coping.
This kludginess is revealed by, for example, optical illusions. And by the famous 7±2 result about the very limited sized of the human working set. And the various well-documented ways that human beings are extremely bad at statistical reasoning. And in many other ways…
When you do work that is as demanding of rigor as software engineering, one of your central challenges is hacking around the limitations of your own brain. Sometimes this develops in very obvious ways; the increasing systematization of testing during development during the last couple of decades, for example.
Other brain hacks are more subtle. Which is why I am here to suggest that you try to stop thinking of documentation as a chore you do for others, and instead think of it as a way to explore your problem space. and the space in your head around your intuitions about the problem, so you can shine light into the murkier corners of both. Writing documentation can function as valuable knowledge capture about your problem domain even when you are the only expert about what you are trying to do.
This is why my projects often have a file called “designer’s notes” or “hacking guide”. Early in a project these may just be random jottings that are an aid to my own memory about why things are the way they are. They tend to develop into a technical briefing about the code internals for future contributors. This is a good habit to form if you want to have future contributors!
But even though the developed version of a “designer’s notes” looks other-directed, it’s really a thing I do to reduce my own friction costs. And not just in the communication-to-my-future-self way either. Yes, it’s tremendously valuable to have a document that, months or years after I wrote it, reminds me of my assumptions when I have half-forgotten them. And yes, a “designer’s notes” file is good practice for that reason alone. But its utility does not even start there, let alone end there.
Earlier, I wrote of (a) capturing knowledge you have but don’t know you have, and (b) eliciting knowledge that you have not yet developed. The process of writing your designer’s notes can be powerful and catalytic that way even if they’re never communicated. The thing you have to do in your brain to narratize your thoughts so they can be written down is itself an exploratory tool.
As with “designer’s notes” so with every other form of documentation from the one-line code comment to a user-oriented HOWTO. When you achieve right mindset about these they are no longer burdens; instead they become an integral part of your creative process, enabling you to design better and write better code with less total effort.
I understand that to a lot of programmers who now experience writing prose as difficult work this might seem like impossible advice. But I think there is a way from where you are to right mindset. That way is to let go of the desire for perfection in your prose, at least early on. Sentence fragments are OK. Misspellings are OK. Anything you write that explores the space is OK, no matter how barbarous it would look to your third-grade grammar teacher or the language pedants out there (including me).
It is more important to do the discovery process implied by writing down your ideas than it is for the result to look polished. If you hold on to that thought, get in the habit of this kind of knowledge capture, and start benefiting from it, then you might find that over time your standards rise and it gets easier to put more effort into polishing.
If that happens, sure; let it happen – but it’s not strictly necessary. The only thing that is necessary is that you occasionally police what you’ve recorded so it doesn’t drift into reporting something the software no longer does. That sort of thing is a land-mine for anyone else who might read your notes and very bad form.
Other than that, though, the way to get to where you do the documentation-as-knowledge-capture thing well is by starting small; allow a low bar for polish and completeness and grow the capability organically. You will know you have won when it starts being fun.
Getting to the next stage is actually trickier for me. I usually have a notes.txt file in my projects which just contains freeform prose that I think will be helpful to me in future, and serves something of the purpose of a hackers’ guide (except that, despite being very brilliant, the others in the lab do not like touching non-LabView code, so it’s not written with collaborators in mind). What trips me up is creating well-organised documentation that faces the user, even working from notes.txt.
For instance, on a man page for a CLI tool, you might have an introduction followed by a table of option descriptions. But one option’s description may only make sense alongside some context about the internal structure of the program – this context isn’t a prerequisite for understanding the other options, so it’s unwieldy to put it in the introduction; but neither does it look clean to shoehorn a couple paragraphs of exposition into the middle of the option blurbs.
Alternatively, two options may interact in complicated ways that are best explained with a differential set of examples (i.e. give input-output combinations for
foo -a,foo -b, andfoo -ab). On one hand, it would be least confusing that the example for each option alone should be in that option’s description; on the other, the illustrative value of the exercise is in having them all together where the contrast in the output is evident.I seem to always end up running into weird cases like these where there’s text that needs to be in the documentation somewhere, but any specific place I put it seems awkward and unsatisfying. Perhaps this is an inevitable consequence of trying to make it possible to learn only what one needs for a small task, while simultaneously including all the information necessary to use the full capabilities of the tool – in any case, I’ve come to regard writing the user-facing docs with some trepidation.
>I seem to always end up running into weird cases like these where there’s text that needs to be in the documentation somewhere, but any specific place I put it seems awkward and unsatisfying.
This means you’re paying attention.
In cases like that you just have to bite the bullet, accept that none of the N alternatives is ideal, and pick one.
Also, trust your future self. Time may reveal that the reason why none of the N places it could have gone was correct was that there was an N+1th place you hadn’t created yet, and this is just the first instance of the several bits of info that need to go there. You can’t know that until the rest of the info appears, but it’s a good enough chance in my experience that you should just roll with it and be as ready to refactor your docs as you are to refactor your code.
As often as not I create a simple text (today more likely asciidoc) file named “NOTES” in the top-level directory of a new project, as or even before I create the file that holds the program’s entry point. (I nearly typed “main()” there, betraying my heritage as an ancient C programmer. ;-} )
NOTES contains whatever random stuff I have to remember now; very often it gets refactored into better places as I go.
Version Control History FTW!
This is a large part of why I have found behavior-driven development to be a useful tool: It deliberately front-loads a substantial part of the clarification of exactly what the software is supposed to do that otherwise tends to take lots of back-and-forth during the development process. Business “feature owners” tend to be frustrated at first and within a couple of cycles discover the terra incognita of their own business process, then turn enthusiastic about the effort to align what’s produced with what the business really needs. It’s discussed in terms of “acceptance criteria” and “test cases”, but it’s fundamentally about knowledge capture and spreading (or agreeing on!) knowledge about the task to be done.
Ideally, one’s code is so perfect that it doesn’t need documentation.
</sarc>
Appreciate the humor ;)
I have known plenty of “the code is the documentation” people over the last many decades….I had varying degrees of success in getting them to understand that while parsing the code (if written sufficiently expressively) can indeed literally document its *function*, it never explicitly defines the creator’s logical *intent* – the “why” behind it all.
Hey, if it was hard to write it should be hard to read and even harder to modify.
Code doesn’t lie.
But it often mumbles.
“Self-documenting code isn’t.”
If that’s NOT a ‘Law of Computer Programming’ it dang well ought to be!
Yeah. I’ve been hired to come in after those guys a few times. Most of them were too elite to do input validation, bounds or error checking, or use more than single-letter names for variables…
No, I’m no sooper-haxor-dood. I’m just the guy who comes along after, cleans up the mess, and makes the wondercode actually work, while documenting the hell out of everything, because odds are better than even I’ll have to swan dive into that mess again in the future.
“When will it be working? We NEED this! We paid a lot of money for this.”
“If I was the one you’d paid, you wouldn’t have had to call someone in to fix it…”
>Most of them were too elite to do input validation, bounds or error checking, or use more than single-letter names for variables…
That sound you hear is me swearing sulphorously.
Got any examples? I’ve been running low on good variable names lately…
The guys who get paid to store the data that kind of code spews out aren’t too happy either.
As in, “you can’t have a PK on that table or the code will break”. I wish I were joking.
I’m not convinced this is true. Compare the the relative success of the logic based approach to AI and the pattern-matching neural-net based approach.
What part do you think is not true?
> Humans, including me and you, are sloppy and analogical thinkers who tend to solve problems by pattern-matching against noisy data first and checking our intuitions with logic after the fact (if we actually get that far)
I remember stumbling upon a description of psychological experiment, where when tested on case with limited or incomplete information our emotion-based heuristic approach performs better than logic-based one – which was checked by testing such cases on people with brain damage or genetic defect that are unable to do emotion-based approach.
This is a does-replicate phenomenon, at least as far as gambling goes. Lack of appropriate fear/anxiety response leads to absurd and catastrophic risk-taking.
In addition to knowledge capture, there’s something Mark Twain probably didn’t say: “It ain’t so much the things that people don’t know that makes trouble in this world, as it is the things that people know that ain’t so.” In my first quality assurance job, I had to test a backup program, so first I had to figure out what it was supposed to do, initially by figuring out what it actually did. Neither matched what the lead engineer had told me.
Thus began my career-long implementation of Cunningham’s Law: QA shouldn’t wait endlessly for engineering to provide the necessary documentation. Instead, write and post it yourself, then ask for corrections, with the subtext that that’s what you’ll be filing bugs against.
I personally hate writing documentation, but have discovered it valuable as you’ve said in clarifying my thinking. And I write the first version not for others, but as a reference for future me, who clearly is going to want to know why the me of 2 years ago concocted this design.
And it is important to do it at the beginning, and update it periodically, rather than do it when it’s done. It really clarifies the thinking, and for me, it points out the gaps in my understanding of the problem domain, and also the instances where I’m going from pure instinct and experience rather than reasoning from first principles. Sometimes I don’t change my mind after writing it down, but it gives me better focus and better ability to explain my design decisions to others.
I have looked at old code, wondered “who wrote this crap?” and realized it was me. So make good notes, because Future You can be a tough critic.
The problem with documentation is that very often then “what” is documented (which one can see anyway) and not the “why”. This tends to come from programmers focusing purely on the implementation phase of a project, which is fine if someone else is doing the rest, but when not then not. This is why I tend to not call myself a programmer, just someone who uses coding as one of the many tools to deal with problems. And of course if someone else has written a design document, it can get lost. It is amazing how much important information people put in nowhere else but an email that nobody can find a decade later – oh maybe in the Unix world yes, but in the mainstream world after a few Exchange migrations… emails are definitely not “fixed assets”.
(Besides, the disk-space-is-cheap philosophy is changing quickly now that businesses tend to put everything into the cloud, want to save cloud hosting costs, so just like I keep being asked to allow some testing databases to be deleted because 80GB cloud space just for testing is too expensive [I don’t have to tell you that every teenager has a terabyte at home just for games, really it seems to me this cloud thing is just asking for getting shafted and getting it good and hard], similarly, in this mindset they will quickly decide that 10 years old emails should be deleted because muh cloud is expensive.)
For this reason I am a fan of writing these into comments, there are tools that can lift them out of it when necessary. At least if you lose it, you lose both the code and the documentation, hence no problem :) Of course a design decision as a code comment might sound weird, but only because people tend to be very terse, formal and to and to the point in code comments. Why? I find it perfectly normal to use a code comment to make a joke, curse the vendor of some library, or talk about design philosophy. Why not? After all, every code is written primarily for humans to read. Let’s not keep that reading too boring.
But from other people I often only get very terse documentation on the “what” and not on the “why”. So I get into situations when I want to upgrade something and run into customizations done 10 years ago and nobody has any idea who wanted it, why, is it still being used and so on. And again the issue is people do not do this because they think they must write a super formal design document. While a one-liner comment like “C.A. from Purchasing wanted a report both in USD and EUR, hence we are storing both values here” can help identifying if is it still needed or not.
Generally speaking, don’t be a programmer in the bad sense of the word, someone who has horse-blinders on implementation, implementation, implementation, code in, features out, bugs down, that sort of narrow focus, but try to be someone who just uses it as a tool for reaching more abstractly or more humanly-defined goals. I understand that there people whose inexperience or low IQ makes implementation i.e. coding so hard for them that they cannot possibly have energy left for anything else, but the high-IQ and experienced people still often focus on increasing their already very high coding productivity, which is IMHO not the right direction.
I can’t claim to be overly fond of writing huge chunks of documentation when I could be writing actual code. Just as git helped me become a better programmer though, it has done a lot to make me a better documentation writer. (If you’re not a git fan substitute your preferred source control system here.)
The major benefits are almost identical. The very first version (commit) doesn’t need to be perfectly complete (or even perfectly coherent); and incremental changes don’t need to be perfect improvements. I can fix the problems I see now, confident that when I see another problem down the road, I can fix that one too. I can be aggressive about changes because I know I will never have a panic attack of the form “Oh crap, that section I deleted two weeks ago, I actually need that now.”
And the benefits come without loss of flexibility, the same as with writing code. If you’re a nitpicky stickler, constitutionally unable to change the code unless you immediately change the docs to match, then by all means commit the changes simultaneously. If you do better “emotionally” when you can just jot down whatever at the moment, knowing you will come back at a less frequent interval to do more aggressive editing, then you can keep the changes separate (keep all the docs in a separate branch, or even separate repo if that makes sense to you).
Sorry if the above was obvious to the majority here. My own experience of learning to use git was how astonished I was at its ability to drain ego issues out of my coding (“gah this is total _crap_, screw that guy (me of two weeks ago), I will just start a new branch and redo this garbage), and later doc-writing. So if anyone else is hampered when they start the writing / coding process by (possibly subconscious) fears that they won’t be able to get it “right” the first time, I hope this approach would help.
I’ve never understood that mentality – of “hating” writing documentation.
What I have grown to loathe is the corporate box-checking process-driven mundane paperwork that provides little of value and has no safeguards against stale information.
> I’ve never understood that mentality – of “hating” writing documentation.
Lots of reasons:
Most people are crappy prose writers. Programmers, speaking very generally, are people. I remember Back In The Day getting a resume from a buddy who was a Perl wizard. The resume was *shockingly* bad in terms of grammar an punctuation. I was almost ashamed to pass it on.
Many programmers are fuzzy on the why themselves, and the how is generally obvious…at the time.
It’s possible that for some people, they hated writing it because they were told to write it in a certain way.
A long time ago, a cow-orker produced a list of documentation guidelines for our Java project and insisted we follow them. Among the guidelines was a requirement that every accessor have its return value documented with a @return annotation. This is sensible only if you make an effort to say something that wasn’t galactically obvious from the surrounding code (apologies if this misformats):
/** @name getFrotz
* @return the frotz
*/
public Frotz getFrotz();
This might not be a case of self-documenting code, but if not, it’s damn close. In nearly all cases, it was so clear what an object’s Frotz was that there was truly nothing one could say in English that wasn’t already in the method name. Nevertheless, this guy would Khrushchev on guidelines like this, to the point of passive-aggressive behavior when we didn’t (slow-walking modules we needed, pretending not to hear our ideas at meetings, etc.). He didn’t get it. To him, we were putting ourselves above the law; we were special, and didn’t need to follow the rules, and he never considered that he was making himself special by demanding we all lived by his.
I’m not to the point of enjoying writing documentation, but I’m to the point of frustration when software lacks it, and relief when software has documentation that’s comprehensive and lets me understand the system I’m dealing with.
That includes software I write.
So that in itself is impetus for me to get off my ass and document.
I recommend that everybody here who programs, try writing a piece of software in literate programming style. In 2020 this is really easy to do with Emacs; you don’t need any additional tooling besides Emacs itself and its legendary org-mode. Just follow the instructions for literate programming in the org-babel documentation. The added clarity gained by explaining to yourself why you’re writing a particular bit of code, before or while you write it, is instructive.
In fact, I recommend you get familiar with org-mode itself, as it is a handy document-structuring, note-taking, and thought-organizing tool. There’s a reason (or two or most-positive-fixnum) why I stuck with Emacs for 25 years after learning to grok it in the 90s. (Though that’s not something I can recommend for everybody; zoomers in particular are likely to have started with Visual Studio Code or similar as their primary editor and have no reason to switch.)
I haven’t touched emacs in eons….but you’ve piqued my curiosity :)
> So that in itself is impetus for me to get off my ass and document.
If you *don’t* document, and you’re still working at the same place, you’ll wind up married to that code, which will follow you across departments and job descriptions. Some people have the idea it’s job security (it isn’t), but it’s also an anchor; “no we gave $EXCITING_NEW_PROJECT to the new guy because we can’t spare you from all the old code you’re married to.”
>(Though that’s not something I can recommend for everybody; zoomers in particular are likely to have started with Visual Studio Code or similar as their primary editor and have no reason to switch.)
I’m a millennial, not a zoomer, but, in my growing up years used Word, mainly, as my text editor. When I took programming classes in high school, it was whatever IDE the school was using for Java. But when I switched to Linux and really started to learn how computers worked and what tools were available to me for any given task on any given system, I stopped using word processors to edit text that didn’t need formatting pretty much by instinct, and the programming courses I took in college didn’t push any specific IDE for code, and so I stopped using word processors (except when something actually does need to be pretty-printed) and IDEs without really putting much thought into it. I took to using gedit when I browsed to a text file in a file-browser, and vi (mostly) when I was already in a terminal window when I decided to edit a particular file. I’ve since switched to kate as my graphical editor because “GTK3 Considered Harmful”, but I can really use any editor, and I think I’d recommend that anyone my age try vi, emacs, and some graphical editor other than what they currently use, just because, if you absolutely *can’t* figure out how to use each of those three, computers are probably not for you.
> I’m a millennial, not a zoomer, but, in my growing up years used Word, mainly, as my text editor.
Ouch!
I’m absolutely insistent about using Word for composing prose when it comes to any work longer than a blog comment. So much so that not only will I sell my soul to the Dark One for the templates, macros, and blasphemous arts needed to convert prose-in-Word to epub format or similar, but I will add a tip to the payment out of appreciation for the service provided. But using Word as a text editor? No. Just No.
I have a couple of text editors that I regularly use for editing text, because “Right tool for the job,” and composing prose and editing text are two different jobs despite their superficial similarities.
>But using Word as a text editor? No. Just No.
Iknow, but hey, I was a kid! And, in my defense, the system text editor was Win9x notepad. (Though I could have used DOS edit if I’d known about it). My dad had also purchased a third party editor, but I hadn’t clued in enough to use it when formatting wasn’t required.
Could be worse. I was in college / work tours during the 1990s, and cut my teeth on Framemaker. Toward the end, I could write and typeset mathematical formulas using keyboard shortcuts alone. My math homework was a joy to behold!
Framemaker still exists today… but how many people have seen an install of it sitting around?
I’m currently on the customer-facing side of documentation problems while trying to learn YAML, which seems to be carefully documented with the “I know what I’m doing, why don’t you know what I’m doing?” method of documentation, without basic examples, OR an understanding that someone might want to learn their language from scratch, OR an idea that someone might want to know how to pick apart the not-text, not-an-array, not-a-hash, not-a-variable chunk of something which the YAML library drops into your program.
Today I’m going to try again with a different programming language, (Python instead of Ruby) and see if someone from the Python community wrote something useful about YAML. Anyway, the documentation sucks!
Just to close out the story, I did look up YAML for python, and the first article I found made everything clear to me; it explained things really well and had an example which included a properly-written YAML sample. But every time Eric writes about better documentation, I want to cheer!
> OR an understanding that someone might want to learn their language from scratch
I recently worked on setting up a 2-machine UUCP network at home as a retrocomputing hobby project, and it was unbelievable to what degree the documentation for UUCP and netnews suites assumed that you were already familiar with the general operation of UUCP and netnews suites. The documentation was decent for transitioning from an older system to a newer one, but next to useless for anyone trying to learn everything from scratch. I ended up having to go all the way back to the original Bell Labs v7 Unix UUCP documentation (which couldn’t assume prior familiarity with UUCP because the v7 UUCP suite was the first of its kind) to learn certain things. Now that I’m done figuring that out, documentation exists that explains those details for “modern” news systems, but it was a long slog getting to that point.
I’ve also found it very difficult finding reading material on the hard sciences for STEM-minded readers that took a wrong turn in college. Most material either assumes that the reader is unwilling to engage with math at all, or else has all the mathematical background necessary for an undergraduate physics degree.
Interestingly, though, for all the gripes about poor documentation, I’ve generally found the various computing disciplines to be those that are best among the STEM disciplines at explaining their domain to motivated outsiders.
Writing tests have much the same effect as writing documentation. I find it often takes me longer to write a test than to write the original code.
That’s because you have to write a test that the original code will pass.
So, if you’re doing test driven development, you write the unit tests and at least some integration tests, before you start writing the code.
Recalibrate your sarcasm detector.
s/sarcasm/humor/, but yeah.
There must be thousands of documentation and source control horror stories, but these three are mine (two each).
1. Long ago (late 1970s), in a grad school far, far away, I was the assistant dean’s bit basher.
One of my duties was to run a FORTRAN 4 program that read faculty evaluation data from tape and summarized the results by department and faculty member on line-printer paper, semester A on the left, semester B on the right.
One day, my boss said the dean wanted to change the report.
My predecessor, a much, much better programmer than I’ve ever been, had whiled away the hours waiting for listings by minimizing the number of lines in the program. The code was a tight snarl of GOTOs on fewer than 100 cards including a few lines of CDC 6400 JCL.
There were no comments or other docs. The changes never happened.
2: A couple of years later, I was the bit basher for a faculty member whose career depended on research based on reels of data from a government agency in a mid-western city.
His few JCL and many FORTRAN 4 cards filled two boxes; 2000 cards per box if memory serves. There was only one copy of the cards, and a predecessor had removed the comments for convenience in moving them five miles between the office and computer center.
My boss let me charge a reel of tape to the department and copy the code to it.
3: In the early 1990s, I was a microeconomist on the faculty of a small liberal arts college. A physicist colleague and I were experimenting with a program that let students enter formulas and coefficients or data sets, and get numeric and graphic results. The program fit a 3.5 inch floppy disk, there was a nice user interface, it worked well, and students could use it.
The developer, a high energy entrepreneur, wrote the first version of the code. At the time he had one full-time programmer working on it. A paper-back textbook / user guide had been published. The project was on the edge of being rolled out commercially when the programmer lost the only copy of the source code. We were told, and I believe, that it was an accident.
I’m sure nothing like these stories happens in these enlightened times.
Yikes, #1 is like The Story of Mel with all the fun bits taken out. =:O
HaHaHa!
I hadn’t seen Mel before; good story!
Thanks!
So another, slightly more involved way of saying you don’t really understand something unless you can teach it.
I actually get really scared when I run into ‘software architects’ that can’t explain what their designs do and why. Not anyone here but I’ve run into a few.
Another twist on that concept is that you’ll never be sure you truly understand something until you can teach it.
….and yet another twist is – if you don’t understand something, try to teach it. I have actually benefited from “AHA!” moments on many occasions doing just this.
>if you don’t understand something, try to teach it
That’s actually a mental trick I do on occasion. When making sure I understand some new topic I’ve encountered recently I go through how I would teach what I’ve learned.
It clarifies your thoughts enormously. It’s also very clear from attempting to communicate with, well, almost anyone, that *very* few people do this.
A nice side effect of this is that it engages my brain during long commutes and road trips.
I am currently leaving Patreon because they decided that they own all the art of artists who use their services.
Is there any way to support you that’s not Patreon?
I’ve wanted to bring this up for a while. If Owen Benjamin’s arbitrations and lawsuit succeed, and it’s looking increasingly like they will, then I suspect there won’t be a Patreon at all by the end of this year. It might be time to start looking for a new source of income.
I hear there’s good money to be made in Nigerian email services
Source for that information about Paetreon please, A link to the TOS or whatever there. I was under the impression that the artists owned the copyright of whatever works that they created, and that they would probably have a non-exclusive license to display it on their site.
I would also be interested in the Patreon issue. If you’ve got a link please post it.
Patreon’s Terms of Service at https://www.patreon.com/policy/legal – scroll down to the section headed “Your creations”
So they claim they’re going to “play nice”, but they actually could take your stuff and e.g., sublicense it for a fee to someplace else, and keep that entire fee.
“Own all the art”? Not exactly – but they could try to undercut any financial benefit you would receive….
Of course, IANAL, don’t even play that role on the Internet, please consult with an appropriately trained and licensed attorney for legal advice, etc., etc.
>Is there any way to support you that’s not Patreon?
Recurring PayPal does nicely.
Done.
Peace and long life!
How do you deal with communities filled with unconscious incompetents (https://en.wikipedia.org/wiki/Four_stages_of_competence) that *will*not* read the documentation?
I’m an ex-math/CS-professor. I do write documentation — with (and frequently driven by) examples. The few who read it tell me they get a lot out of it. But the community is filled with meteorologists and atmospheric chemists (and DB coders who don’t understand HPC computing issues at all); these latter think they know everything even though when it comes to software engineering they know just enough to be dangerous.
As a software engineer in HPC environmental modeling, I frequently have to deal with this. Currently, the problem is with users and developers who won’t read documentation; in the earlier stages the problem was that new developers added to projects would not read the documentation and kept trying to insist on ideas that were proven bad two generations of modeling previous.
It’s frustrating…
> How do you deal with communities filled with unconscious incompetents
Change communities and leave them to themselves.
> But the community is filled with meteorologists and atmospheric chemists (and DB coders who don’t understand HPC computing issues at all);
Fuck’n shocker mate.
This is actually a great opportunity for you – to enlighten others. As a developer in a writing capacity, you have great influence. Once, I was on a contract where the staff never read the documentation. It was a field I am familiar with – GNU Linux based Internet of Things and Artificial Intelligence, right. So, I started “actually” reading the content, and found multiple things “wrong”.
I was reassigned to the AI group because, I was “too slow” – you know – actually reading the content… refusing to make ludicrous edits, pointing out what new libraries were needed because of a vulnerability – you know – they had no idea…
Finding another opportunity was ideal until I was reassigned back to the “non reading” group…. Still, in this case, the employees were good at social engineering, not software engineering…. until the one person – who actually knows the stuff – like you… comes along and ruins their day!
I just read the following on another blog, by a poster I will not name.
The year 2038 problem is going to be far, far worse, because much of the code will not be overtly handling dates, but will be deep inside code that nominally does something else entirely… …It isn’t just NTP, most security mechanisms, backups, application building mechanisms etc. that will fail, but any program that contains similar logic for other purposes.
…Because modern software is typically built by plugging existing components together and using ghastly scripting languages to bypass and kludge up failure modes, there is decreasing expertise in several critical infrastructure areas. I can witness that I failed to find ANY top-level expert in either NTP or TCP/IP in the UK when I needed one, and my contacts with those in the USA indicated that there at most a single digit number left, worldwide. Indeed, I was horrified to discover how few ‘experts’ knew even as much as I did :-(
On this matter, fixes to the 2038 problem will often be intimately bound up with C’s conversion and promotion rules. Hands up anyone who knows what they are, how they differ from C++, and (most importantly) what they have been over the period 1980-present. At one stage, I was one of about half a dozen people in the UK who could have put up my hand half-way, but am now too rusty.
This sounds very much like something Eric should be involved in, probably starting right now – this is pretty much his resume.
>This sounds very much like something Eric should be involved in, probably starting right now – this is pretty much his resume.
You’re not wrong. Now for the next question: who’s going to fund the work so I can pay rent and buy groceries while I do it?
No idea – I’m pretty broke these days – but I did suggest that the concerned poster on the other blog get in touch with you.
BTW, I notice that you haven’t posted in awhile. Is everything OK?
>BTW, I notice that you haven’t posted in awhile. Is everything OK?
Yes, it’s just that I’ve been busy and no essay topic has said “write me!” recently.
I’m glad to hear you’re OK.
Maybe you could post about the 2038 issue? Get some feedback on how bad the problem and whether anyone’s working to fix it?
>Maybe you could post about the 2038 issue?
In this case I doubt I could do better than the Wikipedia article on the topic, which told me some things I didn’t know about which Unixes have and have not gone to 64-bit time_t.
I can tell you that NTPsec is as on top of the problem as is possible without a wire-protocol rewrite.
The Wikipedia article was very helpful and it was relieving to notice that most of the major open-source Unixes are using 64-bit time.
It’s also good to hear that NTPsec is up-to-date. Does it allow the use of 128-bit time?
>It’s also good to hear that NTPsec is up-to-date. Does it allow the use of 128-bit time?
No. That would take a redesign of the wire protocol. Which we are thinking about, but that’s a very difficult transition involving IETF politics.
I think I’m up-to-date now. Thanks for discussing this with me.
https://lcamtuf.blogspot.com/2015/10/subjective-explainer-gun-debate-in-us.html?m=1
Always curious to see what other people think of knowledge management documentation – fantastic.