The C/UNIX library support for time and calendar programming is a nasty mess of historical contingency. I have grown tired of having to re-learn its quirks every time I’ve had to deal with it, so I’m doing something about that.
Announcing Time, Clock, and Calendar Programming In C, a document which attempts to chart the historical clutter (so you can ignore it once you know why it’s there) and explain the mysteries.
What I’ve released is an 0.9 beta version. My hope is that it will rapidly attract some thoroughgoing reviews so I can release a 1.0 in a week or so. More than that, I would welcome a subject matter expert as a collaborator.
Bravo!
Fun.
I think gettimeofday was first in BSD 4.2, not 4.3. I remember adjusting to all this stuff back in the day. And long for tv_usec
http://modman.unixdev.net/?sektion=2&page=gettimeofday&manpath=4.2BSD
Useful paper, thanks.
Historical anecdote regarding India’s half hour offset: hold an analogue watch set to Indian time upside down voila! GMT, thus avoiding a need for British army officers to own/carry two watches.
In any situation were absolute time actually matters, one is going to be using GPS time
GPS time drifts with respect to unix time.
I am not aware of any standard system for converting, though I suppose one must exist, probably inside every gps.
>I am not aware of any standard system for converting, though I suppose one must exist, probably inside every gps.
Date and time in GPS is represented as number of weeks from the start of zero second of 6 January 1980, plus number of seconds into the week. GPS time is not leap-second corrected, though satellites also broadcast a current leap-second correction which is updated on six-month boundaries according to rotational bulletins issued by the
International Earth Rotation and Reference Systems Service (IERS).
The leap-second correction is only included in the satellite subframre broadcast, roughly once ever 20 minutes. While the satellites do notify GPSes of upcoming leap-seconds, this notification is not necessarily processed correctly on consumer-grade devices, and will not be available at all when a GPS receiver has just cold-booted. Thus, UTC time reported from devices may be slightly inaccurate between a cold boot or leap second and the following
subframe broadcast.
GPS date and time are subject to a rollover problem in the 10-bit week number counter, which will re-zero every 1024 weeks (roughly every 20 years). The last rollover (and the first since GPS went live in 1980) was 0000 22 August 1999; the next would fall in 2019, but plans are afoot to upgrade the satellite counters to 13 bits; this will delay the next rollover until 2173.
For accurate time reporting, therefore, a GPS requires a supplemental time references sufficient to identify the current rollover period, e.g. accurate to within 512 weeks. Many NMEA GPSes have a wired-in assumption about the UTC time of the last rollover and will thus report incorrect times outside the rollover period they were designed in.
Link is not working for me :(
We are currently working on the tzdist standard for distributing timezone/DST information, but there is still some confusion on the base TZ database which has been wiping historic data prior to 1970 as ‘unreliable’ even where we have documented provenance.
What pigs me off is the fact that tzdist is being discussed on a base of iCalendar which is not designed to handle archival data at all. I’m trying to ensure that when we use a set of data to normalize historic material you can at least get back the information in the future.
It’s interesting that leap-second is still a moving target. There is still controversy as whether it should be added at all? My own time systems work on ‘day’ as the base unit, with fraction of a day providing the time. This allows 32 bit accurate for the date, and 32 bit accuracy for the time, with leap second just being a minor variation on the odd day. It does seem strange that even today systems are being built with known limited functional period?
>It’s interesting that leap-second is still a moving target.
Unavoidably. We can’t predict the glitches in the Earth’s rotation.
>There is still controversy as whether it should be added at all?
There is some. I’m in the camp that thinks leap-second hassle is worth it to keep our clocks describing solar time.
Under UNIX and UTC:
‘The absence of leap YEAR correction” I think should be “second”
“These are both intended solely for NTP’s use, and that is all we EILL have to say about them in this document.” “will”?
At the very beginning TIME+T was defined as a signed 32-bit quantity…
All the previous caveats about leap-second discontinuities ABD tiny variabilities due to NTP correction apply.
First, rough SYCHRONY with mean solar time would…
(assuNing you don’t go the everything-is-UTC route) is to
The *+r functions are, on the other hand, re-entrant, – should be *_r
though tzset(3) has been called, but that +localtime_r(3) is not …. *localtime_r(3) and font is code for has been…
explicit, +gmtime_r() as well. … *gmtime_r()
The mktime(3) function inverts +localtime(3), all in code font *localtime
description of +timegm(3) below. … *timegm
e) to a posi? tive value or
be used; use strftime(3) instead. If you are curio ..instead in code font
There seems to be a problem with posting here … have to retry several time to get in …
It is only assuming that time is a number of seconds rather than a fraction of a day which means that the Unix clock needs continual corrections. Our clocks don’t describe solar time anyway, just a rough approximation since the whole thing varies anyway?
Inspirational music?
http://www.satriani.com/discography/Time_Machine/
I’ve posted this to the tz@iana.org mailing list (where there is no shortage of subject matter experts).
After reading through it, I noticed that you don’t acknowledge the leap second support specific to that implementation and those derived from it (which include most modern unix systems, though they don’t usually ship with leap second support enabled) – which results in time_t not precisely matching the traditional notion of unix time (conversion functions are provided: time2posix and posix2time). The fact that it renders a system non-POSIX-compliant has made it understandably unpopular, but probably still worth mentioning given your position on leap seconds.
“[difftime] seems to have been added in C89 as a way to encapsulate time arithmetic on operating systems where time_t is not an integral or float type; it is possible (even likely) that no such environments actually exist.” – AFAIK time_t is required by the standard to be a numeric type. It is not required to represent a count of seconds, even disregarding differences due to different ways of implementing (or not) leap second support, and this is the reason for this function.
>After reading through it, I noticed that you don’t acknowledge the leap second support specific to that implementation and those derived from it
That’s because I don’t know about it. Documentation pointer?
Despite their nonexistence on Linux/GNU in particular, I think something claiming to be about these issues in C generally would be incomplete without a mention of Windows and FreeBSD’s (different) locale-specific versions of strftime, and the much-maligned (and not yet implemented on most Unix systems) C11 “secure” Annex K functions asctime_s, ctime_s, gmtime_s, and localtime_s.
re asctime/ctime: “In older Unixes the following functions were available to report time and date as a string. They wired in bad design choices (the date string is of unpredictable length and includes a trailing “\n”) and are not locale-aware.” – it might be worth mentioning that by way of compensation for the unpredictable length, some implementations will add an extra five spaces before the year if it is greater than four digits, so that a program attempting to extract the year from those exact four positions will get a blank string instead of a misleading four-digit year.
Some more stray + signs, these ones on function names that are not in monospace font and were presumably intended to be:
The most basic function is +time(3), which returns the current Unix second. All the previous caveats about leap-second discontinuities and tiny variabilities due to NTP correction apply.
The sleep and alarm function groups developed by accretion in early Unixes and the POSIX standard up to 2001. The sleep(3) group was often implemented with alarm(3) and a specialized SIGALRM handler; likewise for usleep(3) and +ualarm(3).
I enjoyed reading. However, the section on TZs made me laugh a little. To me it is a classic case of confusing the map from the territory (that is to say the subject matter, not your excellent treatment of it..)
Of course there is a lot of history there, but computers should absolutely be communicating in a standard time format, UTC or its predecessor GMT. Timezones then become entirely a display issue, and timezone databases become mostly irrelevant.
I know what the TZ is in my computer, and it is trivial to write a function to display the UTC time in local time format. It is no different as to whether you represent pi the circumference of a circle with a diameter of 100 meters as 3,141.593 m or 3.141,593 m or 3 141,593m or even 3,435.688 yards.
In one respect the cause of much of the gnashing of teeth with respect to TZs, in fact the very existence of the TZ database has to do with the need to represent communication protocols in text — user readable text. To cite one of the examples you mentioned, SMTP gives a date time with offset in its stream. SMTP should be for communicating between mail programs and servers, and should entirely communicate in a UTC like system. How your email program displays the date is something that can come from the locally configured TZ.
There are a few other reasons why a TZ database might be needed, and there are a whole range of questions as to how to display historical dates and times with respect to changing boundaries of TZs. But they are mostly esoteric edge cases that don’t really matter all that much except in specialized applications.
And in a sense it is a classic case of centralization verses local decision making. Should the IANA decide how dates and times are displayed on my computer, located a few miles west of Chicago, or should I decide irrespective of what some petty bureaucrat in some pseudo government agency decides.
>>It’s interesting that leap-second is still a moving target.
>Unavoidably. We can’t predict the glitches in the Earth’s rotation.
That’s true. I read that the Indonesian tsunami back in 2004 slowed the earth’s daily rotation by a measurable sub-second amount.
Another error: [TIME-ZONE] cited inline, footnote is [TIMEZONE].
Speaking of timezones, some of the history might be worth mentioning – like the Posix/System V style “TZ” environment variable (EST5EDT, then later some other text was added to specify the daylight savings offset [or reverse for the southern hemisphere, since on some implementations the first offset is assumed to be the “January” timezone rather than winter specifically] and transition times. Which can of course only specify a single pair of offsets and transition times. Tzdata uses these strings internally to represent far future transition rules, if possible (if not possible the file will include 400 years of data and hope for the best).
Strictly speaking, according to the standard, the TZ variable should be prefixed with a colon if it specifies something other than a Posix-style string.
>Speaking of timezones, some of the history might be worth mentioning
Very much so. Is it documented anywhere?
@Jessica Boxer that’s fine for you, a few miles west of Chicago, but someone living one state to the east of you might not want to have to manually configure everything by hand to represent that their county switched to Eastern time in 1991 and then back to Central in 2006, when they may have files from all three eras that they want to display the correct timestamp on. Your “territory” may have been stable since 1967 [barring national changes, which likewise have to be configured somewhere], others’ have not. “Pseudo-government agency” crack aside, this is no different from any large software project in that someone has to maintain and distribute it, and deal with all the edge cases that you, a few miles west of Chicago, have no knowledge about. And even you have to keep track of the fact that the national daylight savings rules changed in 1987 and 2007.
IMO the proper solution should have been for the timezone offset – or, indeed, the whole local time – to be stored with the file alongside the UTC time. But that ship has long since sailed (and storing only the local time, as Windows does on FAT filesystems, is worse).
Somehow, the famous Douglas Adams quote about time should be included. Seems appropriate.
@Random832
> but someone living one state to the east of you might not want to have to manually configure everything
Kudos to you for knowing how screwed up Indiana is with regards to timezones. But to be honest I think you are not really putting yourself in the shoes on a Hoosier. When the Indiana government switched the TZs around, then all the residents of the affected counties had to change a whole bunch of clocks. It doesn’t seem anything less than a trivial extension of this to change their computer TZ setting. In fact many people were probably confused that their computer “magically” changed.
> and deal with all the edge cases that you … have no knowledge about.
For sure, but I don’t need to know about the local time in Kazakhstan, do I? It leads to a situation where the IANA needs to make decisions about politics that they should not be making. For example, what is the timezone in eastern Ukraine right now, or Crimea? They are pretty much due south of Moscow, shouldn’t they be in the same timezone as Moscow? Or should they be within the common timezone of the rest of Ukraine? Who gets to decide? The IANA?
> or, indeed, the whole local time – to be stored with the file alongside the UTC time.
Yikes, I hate that idea. Like I said originally, time zones are really a display representation. Should we store lengths in meters, yards and fulongs too? After all the relationship between meters and yards has changed as recently as 1959.
OK catb.org is now accessible from here ( although who.is still reports it as inactive )
As I’ve indicated, I’ve adopted the Firebird format of storing timestamps, which handles the time element as a fraction of the day. This prevents any problem due to the count on number of seconds, which going back in time would probably be well out with respect to solar time anyway.
http://timedb.co.uk/ is my attempt to document the problems of using and identifying DST data to use.
My main objection is to the use of a useless ‘offset’ and calling it the TZoffset on a browser header. It was this which created my first example of why the whole system was wrong, when all of the meeting data switched an hour after a DST change. Several assumptions all made switches to the time, but the result was an hour out.
Now everything is ONLY stored as UTC, and then the information is displayed as UTC or as local time ONLY after a user actually logs in and registers their Timezone.
But even that does not work when TZ removes the data that was being used prior to 1970. We need a master set of data that has ALL valid provenance, and identifies areas where the historic data may be inaccurate.. Simply returning UTC for historic data is not acceptable.
> The oldest, alarm(2|3), sends SIGALRM for a specified number of seconds
Yikes! Is that from the days when computers hit the snooze button? Pretty sure it should be “after a specified number.” :-)
Re: “Some of the comments in [tzdata] make entertaining reading.” Concur, I’ve made a habit to seek out the release notes when there’s an update.
Sometimes more poignant than entertaining. From March of this year: “Crimea switches to Moscow time on 2014-03-30 at 02:00 local time.”
Another source of information on time from a hacker’s perspective is Naggum’s “A Long, Painful History of Time”:
http://naggum.no/lugm-time.html
@Jessica Boxer
>But to be honest I think you are not really putting yourself in the shoes on a Hoosier.
lol. I’ll admit not being from Starke County (the example just fit so well for having two post-1970 zone changes), but I started changing my clocks in 2006 just like most everyone else in the state.
> When the Indiana government switched the TZs around, then all the residents of the affected counties had to change a whole bunch of clocks. It doesn’t seem anything less than a trivial extension of this to change their computer TZ setting.
If they changed their TZ setting, then the timestamps on last year’s files would be wrong. Maybe accurate time on old files isn’t important to you, but are you prepared to impose on everyone a system where that’s not kept track of for anyone unless they do it themselves? Implementing that from scratch is a lot of things, but “trivial” isn’t one of them.
> In fact many people were probably confused that their computer “magically” changed.
In fact it probably didn’t, because new timezone IDs had to be created for the regions that changed, with the original timezone IDs (mostly, IIRC – it’s of course based on the named city, but Chicago, New York, and Indianapolis didn’t, except for the daylight savings change in 2006) staying with the region that didn’t change. And even if it did, it’s no more mystifying than the rest of the country having their computers and phones automatically ‘spring’ forward and ‘fall’ back.
@esr
> That’s because I don’t know about it. Documentation pointer?
It’s not particularly well-documented – probably the best actual formal documentation for it is the time2posix manpage – the tzfile manpage includes some information on how the information is stored, as well. Other than that you might have to look through the mailing list archive.
To test it out, look for the files in the “right” directory under your zoneinfo; that’s where a default tzdata build puts them. You have to replace GMT (ideally, just clear everything out and dump the contents of the “right” directory in the main zoneinfo directory) for it to behave correctly, and like I said, it does render a system non-POSIX-compliant (and AFAIK most programs don’t call the time2posix/posix2time functions for interchange like they “should”, since none of this got any traction with any formal standard). The fact that they’re not widely used also means that there’s no guarantee it will interoperate right with ntpdate or whatever else.
> Very much so. Is it documented anywhere?
Well, the POSIX TZ format itself is documented in numerous places – in addition to the standard itself, the Theory file explains it.
I just noticed after going looking for this today that there are brand-new “_z/_rz” variants that take a non-static timezone structure. Might also be worth mentioning. manpage
My previous post is in the moderation queue due to links.
P.S. apparently the _z functions I mentioned are originally from NetBSD.
General comment: why some declarations have the name of arguments (usually hinting what this argument is about), but not all? Compare with
Also, regarding “It is conceivable that in the future we may see further generations of these structures in which the subsecond offset is in picoseconds or even smaller units.” – it might be true for performance counters (though we don’t see increase in CPU clock frequency lately), but useless for wallclock time, due to clock drift… and special and general relativity. Fun fact: top level atomic clocks are so accurate that they drift apart moved meter up ;-). There was IIRC an article in Scientific American on this…
@Random832
> If they changed their TZ setting, then the timestamps on last year’s files would be wrong.
No they’d still be correct, since they’d be stored in UTC, and the Indiana government doesn’t have the capability to move the sun. How they are displayed is a matter for the user to decide. It is a lot like old style dates. For example, when was Adam Smith born? 5th June or 16th June? It is really up to you how you want to view it, but the actual day his mother brought him into the world doesn’t change just because you label the date differently.
> Maybe accurate time on old files isn’t important to you, but are you prepared to impose on everyone a system where that’s not kept track of for anyone unless they do it themselves?
On the contrary, I’d advocate giving anyone the power to display dates however they want.
> Yikes, I hate that idea. Like I said originally, time zones are really a display representation. Should we store lengths in meters, yards and fulongs too? After all the relationship between meters and yards has changed as recently as 1959.
Time zones represent the fact of what time it was on the wall clock when e.g. a file was created, or what time it will be on the wall clock when you’re expected to show up to a meeting. Those are arguably more meaningful data about a file, and definitely more meaningful data about a meeting (if you schedule a meeting for next week, and your timezone changes, the meeting’s UTC time needs to change right along with it), then the UTC time of that moment.
> On the contrary, I’d advocate giving anyone the power to display dates however they want.
In other words, you do want them to have to do it themselves, since your objection is to the existence of a large software project for the purpose of being a ready-made solution for the most common cases (and while regions changing their standard timezone offset arguably isn’t that common, four or five places a year seem to change their daylight saving rules.)
Looking at historic material falls foul of the problem that many computer systems only ever had ‘local’ time on their clocks, and when a 9am meeting was arranged it was held at 9am either side of the DST change. There would be no way to normalize that data to UTC offsets if one does not have the information on the rules they were following. I’ve been there running railway schedules where trains never ran over night, so the time change was never a problem. Until someone built a tunnel to another time zone :)
If you want the full detail on all of the problems then check the archive for both the TZ and now the TZDIST lists. The short version is the problems planning video link meetings between Europe and the Middle east where admittedly in a small number of countries, the DST change depends on astronomical events and change at short notice. The CURRENT system gives no indication to users that the time of a meeting MAY change and the current plans for tzdist are simply pushing the problem under the table again, yet we have all the data and all it needs is a warning message …
There are a few more scenarios that require a clean identification of the data used to normalize data, or to convert historic local times to UTC, but again the current plan is only to maintain a current set of TZ data. Returning historic views is to much trouble …
So we end up with the existing problems for yet another 15+ years?
This is great stuff; I’m not even using these functions and I devoured it as bedtime reading. :) I wish I could put my finger on what makes it this readable…
As a past volunteer proofreader I have a lot of typo corrections for you (tens of them); it might be easiest to send a patch rather than describing them all here. Is the asciidoc(?) source available somewhere?
>I wish I could put my finger on what makes it this readable…
To connect to another currently active thread: Narrativium. I’m not as good a writer as Terry Pratchett is, but I get that part, yes I do.
>Is the asciidoc(?) source available somewhere?
http://www.catb.org/esr/time-programming/index.asc
“[tzset] part of the Unix time interface is very old. Later portions of the Unix API design would almost completely avoid predefined global variables in favor of returns from function calls.
Later, in 4.2BSD, an attempt was made to replace this portion of the time interface with ftime(3).”
This is not correct. I believe tzset, along with these variables and the notion of a TZ environment variable is a System V function, and the TUHS archive shows it does not exist in 4.2BSD. ftime, on the other hand, dates back at least as far as 7th Edition unix.
“timezone” existed as a global variable hardcoded to the number of seconds for EST in libc (ctime.c) in 6th edition unix and in System III (which may explain its presence in System V). It was removed (at least from ctime.c) in 7th edition, and replaced with a call to ftime – which returned, then, a value hardcoded in the kernel (param.h). I suppose this was an improvement because rebuilding the kernel was easier than rebuilding all programs that linked in ctime.o.
In fact, ftime was already on its way out (it is labeled as “old” in the kernel sources, and lives in kern_xxx.c as “oftime”) in 4.2BSD; the approved mechanism appears to have been gettimeofday. A discussion of the (mostly disused these days) timezone aspect of gettimeofday/settimeofday, currently missing from your document, might not be out of place in this section.
>This is not correct. I believe tzset, along with these variables and the notion of a TZ environment variable is a System V function, and the TUHS archive shows it does not exist in 4.2BSD. ftime, on the other hand, dates back at least as far as 7th Edition unix.
Hmmmm. I believe you about ftime(3) and have amended to note that it goes back to V7. But I’m pretty sure I remember seeing tzset() and friends in System III circa 1982-83, which is supported by your recollection. We need to do more research, but I suspect we’ll find that tzset() originated in PWB.
Certainly the style of this part of the interface is very old – I mean, those globals hanging out there naked in front of God and everybody? It’s like it froze before dmr prosyletized everyone about function calls being cheap.
Of course, the approved mechanism these days isn’t to use the globals either, it’s to call tzset (or don’t, it’ll get called automatically) and rely on localtime’s use of inscrutable internal structures. The _z mechanism (which lets you pass around handles to inscrutable internal structures) appears to be the future.
You know, I think I’ll mention this on the TUHS mailing list, too, since your article is to some degree about a historical perspective.
>You know, I think I’ll mention this on the TUHS mailing list, too, since your article is to some degree about a historical perspective.
Do that. The responses may prove interesting.
There are two reasons, by the way, for the historical perspective. One is that it really is easier in this case to understand the shape of the API by how it evolved. The other one (and this answers a previous question about why it’s so readable) is that by using history as a narrative armature I was able to morph something that could have been merely a dry laundry list of technical details into an entertainment with some elements resembling an adventure story and others resembling a comedy of errors.
This was an Evil Writer Trick(tm). I know many others.
The timezone() function in V7 is another missing link – it’s how the date command got the timezone abbreviation before tzname and strftime were around
> which is supported by your recollection
Research. I’m looking at the source code of various versions in the TUHS archive. (not linking to avoid spam filter) minnie.tuhs.org/cgi-bin/utree.pl
Looking again, tzname is also present (and also a hardcoded value) in V6 and SysIII, as is daylight. So they are older than ftime. V5 has them as well, along with another constant called “nixonflg” to enable year-round daylight saving time (I’m guessing President Nixon once proposed for that to be made policy, and they were preparing for the possibility) – can you imagine if that had been immortalized as part of the API?
I’m guessing that System III was to some extent derived from 6th edition, which led to what began life as hardcoded libc values (these were used directly by the date command in addition to libc functions, so I can’t really call them internal) eventually becoming dynamic values in System V. Whereas BSD was derived from 7th edition, which used ftime?
>I’m guessing that System III was to some extent derived from 6th edition, which led to what began life as hardcoded libc values (these were used directly by the date command in addition to libc functions, so I can’t really call them internal) eventually becoming dynamic values in System V. Whereas BSD was derived from 7th edition, which used ftime?
That parallels my own guesses pretty closely.
For now I have simply become artfully vague about the provenance of these parts. Perhaps I can be more specific after more digging.
@Random832
> Time zones represent the fact of what time it was on the wall clock when e.g. a file was created,
Really? so if I have a server in Chicago and I am in California. A backup runs at 4pm Chicago time, and I change a file at 4:05pm California time (2:05pm Chicago) you don’t think it is a problem that my file won’t get backed up next time round? I think you are exactly wrong here. The time on the file should be in a fixed unit — namely UTC — not some variable unit like wall clock time.
> definitely more meaningful data about a meeting (if you schedule a meeting for next week, and your timezone changes, the meeting’s UTC time needs to change right along with it), then the UTC time of that moment.
Again, I think you are exactly wrong here. When you move timezones the meeting UTC time doesn’t change, the local clock time changes. Meetings across timezones are in fact particularly problematic if you don’t use a common time base like UTC. It is the job of your calendar to represent the local time of the meeting based on the UTC time of the meeting.
Error: There is no such thing as “daylight savings time”.
There is “daylight saving time” or even “daylight-saving time” (the older spelling). The s frequently appended suggests that the point of DST is that there is some sort of “savings account” into which one deposits daylight, then withdraws at a later date. Nothing could be further from the truth. The daylight is saved by waking earlier in the morning in the spring and summer than in the fall and winter, which people naturally did before clocks were invented.
@Jessica Boxer
Just because a backup tool shouldn’t use timezone/local times doesn’t mean it shouldn’t be stored at all. A backup tool shouldn’t use mtime anyway, it should only use ctime.
As for meetings – the thing is, if the place your office is in changes timezones, then every meeting that has been scheduled for after that switch does change its UTC time (if you are a normal company). As with daylight savings – if you schedule a meeting for 3:00 on tuesdays, you (most people) don’t want it to be on 2:00 when daylight savings is in effect, nor do you want some meetings at the boundaries of the change to end up being at 2:00 or 4:00 if the daylight savings rules change.
In a related vein:
There is no such thing as “Eastern Daylight Savings Time” or even “Eastern Daylight Saving Time” without the gratuitous terminal s. There is “Eastern Daylight Time”, which is why it’s abbreviated EDT in the first place.
By the way, props for recognizing that it is incorrect to quote a time as “EST” during DST. It’s not as big of a deal as it was when half of Indiana was on EST even during the time most of the country does DST. Keeping track of the difference between KC and South Bend was fun back in the day.
When talking about broken down time, you wrote: “The tm_year base value of 1900 rather than zero – easily forgotten and confusing, especially when interpreting negative values. Also, int is too short for this type, creating potential overflow issues. ”
How is an int too short? Even at its shortest (16 bits, although I think POSIX mandates at least 32 bits), you can still easily represent years between 30868 BCE and 30868 CE (the reliability of such dates is another issue).
>How is an int too short?
From the current version:
There is a potential overflow problem with +tm_year+ on systems where
sizeof(int) < sizeof(time_t); historically this was often true on 32-bit systems and may remain a hidden problem in embedded deployments. The year derived from a +time_t+ can be so large that it won't fit in an int, which means +localtime()+ and +gmtime()+ could silently fail on valid time stamps in the far past or far future. A subtler problem is that when tm_year > INT_MAX – 1900 the Gregorian year doesn’t fit in
an ‘int’; a lot of code gets this wrong. Because INT_MAX on a 32-bit
system is 2^31 = 2,147,483,647 this is unlikely to be a problem
for normal historical dates.
It’s not a huge problem, but it’s a flaw.
> We need to do more research, but I suspect we’ll find that tzset() originated in PWB.
TUHS has a tree called “PWB/UNIX 1.0” ctime.c still has the hardcoded values.
I was reading System III wrong – the variables are initialized to EST values, but tzset does exist. I was confused because on modern systems they’re initialized to blank.
I also found a usenet thread complaining of this exact difference between System III and V7: “TZ Rationalization Requested” from http://tech-insider.org/unix/advocacy.html (along with a number of other threads from that era about timezones).
Ultimately, I think the reason tzset (but not the variables it sets) won is because it became clear in the 1980s that, no, time zone information could not be described by a small handful of scalar variables, whether encoded in an environment value, set as global variables within libc, or returned by a system call.
>Ultimately, I think the reason tzset (but not the variables it sets) won is because it became clear in the 1980s that, no, time zone information could not be described by a small handful of scalar variables, whether encoded in an environment value, set as global variables within libc, or returned by a system call.
Not convincing. In other, similar cases where a system call had to return an aggregate of configuration it was bundled into a struct. The uname(2) call illustrates how it might have been done.
Something else must have been a factor here. Perhaps simple neglect.
No, my point was, the amount of data that had to be handled grew too large – too large for a single struct rather than several large arrays, and too large for it to be worthwhile to handle in kernelspace at all. Look at the size of the zoneinfo files. Two kilobytes is a typical size, and that’s got to all be parsed out from the big-endian packed format it’s stored in. And that’s hundreds to thousands of time_t’s, not a handful of long strings like uname. The problem isn’t the size of the data so much as the fact that it can’t be distilled down to a few simple things (even if those things are long strings as in uname).
Having it in the environment also gives you the ability to set it per-process (which means you’ve got to have files to load it from anyway, instead of putting it in even a large data structure in the kernel)
The variables themselves are just obsolete cruft, my point was more about the reason having an initialization library call rather than a system call purporting to return “everything you need” in a handful of scalars is the paradigm that won.
Plus the fact that System V had a head-start commercially and in the standards organizations.
Spotted an error — UTC is not “abbreviated in the French form”. The French acronym would be TUC. The English would of course be CUT. The Solomonic “compromise” was an acronym that uses the order of the words in *no* language (as far as I know) in which the letters of the TLA are “U”, “T” and “C”. Wikipedia points out that it does follow the pattern of the abbreviations for the various Universal Time schemes, which all start with “UT”, from the English.
Good article.
May want to reference the official opengroup.org site and its definition of the calculation of POSIX time.
IIRC SVs are updated with and transmit the TAI (leap second) offset between the GPS representation of the TAI time scale and UTC time, allowing GPS receivers to report UTC.
Some receivers report only GPS time, so may not decode this data.
The TAI offset is also reported in BIPM IERS Bulletin C which announces every 6 months whether and when new leap seconds are being added, and in the NIST site leap-seconds.list file, both of which list all the leap second corrections from the inception of the TAI time scale.
Recently added to IANA TZ NEWS file:
zdump’s gmtoff=N output is now documented, and its isdst=D output
is now documented to possibly output D values other than 0 or 1.
Some of us were aware that struct tm member tm_isdst could have the value 2 for times in zones during which double summer/savings time was in effect i.e. DST offset was +2:00.
You might want to consider expanding on the varieties of POSIX clock types and the ways in which they differ.
I always liked the appropriate C description of “Broken-down time”.
Where else in this decimal world do we get to deal with such a wide variety of “clock” arithmetic in various time dependent radixes e.g. 60 or 61 seconds per minute, also affecting day, month, year; and days per month, year affecting each century and quad-century.
Thank the Babylonians – perhaps Babel was a dscussion about time?
>May want to reference the official opengroup.org site and its definition of the calculation of POSIX time.
URL?
>Some of us were aware that struct tm member tm_isdst could have the value 2 for times in zones during which double summer/savings time was in effect i.e. DST offset was +2:00.
Hm. Paul Eggert just told me that Unix interprets values > 1 as 1. Is he wrong or are we talking different implementations here?
I know the gory details of GPS time and GPS leap-second correction, but consider them out of scope here.
I don’t know the gory details of GPS time and GPS leap second correction, whereas I already knew most of the gory details of unix time. So, could you recommend a document where GPS time is in scope?
>So, could you recommend a document where GPS time is in scope?
The header comment in this is pretty detailed:
http://git.savannah.gnu.org/cgit/gpsd.git/tree/timebase.c
For the tm_year to exceed an int, it would have to be the year 30867 or greater (ANSI C mandates that the minimum value for INT_MAX is 32767). While *technically* you could run into problems where sizeof(int) == 16 bits and sizeof(time_t) == 64 bits, but I’m still failing to see it as any type of real flaw.
Actually, I see the issue now … sizeof(int) == 32 bits and sizeof(time_t) == 64 bits could also be an issue. I think you need to make it a bit clearer in the text.
s/&eagrave;/è/
Damn…
make that ‘s/eagrave/egrave/’
Sean – The main problem is the size of the base unit, and what one uses as the base. So back in 16 bit days one could not even count the ‘seconds’ in a single integer, so many time processors had strange resolutions depending on the underlying hardware.
32 bit seemed to give a ‘solution’ to the problem, and a single variable counting seconds gave a long time frame when counting seconds. But it was always a bit of a bodge since not having fractional seconds make logging rather restricted? Even a simple switch to 64 bit does not solve THAT problem.
The database approach has been to use a pair of 32 bit numbers for some time ( back to VAX days? ) counting days and milliseconds. Durations are handled by using the millisecond count as fraction of a day, so masking any effect from the leap-second problem by simply adding 1000 to the divisor on the leap second days.
So all timestamps end up with milliscecond accuracy, but the value is essentially a single 64bit number simplifying indexing.
>Note that ANSI C allows for time_t to be a float value, but this choice seems never to have been made in a real operating system and is excluded in later POSIX revisions.
I believe early versions of BeOS had a floating-point time_t, but this proved to be enough of a problem for porting existing code — and the usual equality and comparison headaches with floating-point values — that they changed it pretty quickly.
The story I heard is that the problem with abbreviating Coordinated Universal Time as “CUT” is that the Dutch word “kut” is vulgar, cognate to the English word with an “N” in it.
Above someone linked to another sticks that UTC should properly be written add UT(C) (universal time (coordinated)) a la UT(1) and UT(2) (parenthetical indicating subscript).
See, that’s what got us into the whole Y2K mess. Programmers and managers who wouldn’t think ahead.
I bet Microsoft has a plan for it. G,D&R…
They don’t, but do *they* realize that?
http://en.wikipedia.org/wiki/Indiana_Pi_Bill
I would love to see an example of portable, thread-safe, and correct method to serialize and deserialize a time_t. I think all the pieces of the puzzle are presented, but a complete example would be illuminating I think.
Posting links separately to avoid getting my main post caught in the spam filter.
SVr3.1 email
V4 manpage for ctime(3), localtime, timezone, daylight
V4 manpage for time(2)
V3 manpage for time(2)
…the “main post” to which I refer to above is a lengthy reply to several points, and more information about timezone/daylight/altzone/etc, and disappeared without even “awaiting moderation”.
POSIX time definition – making it pretty clear that leap seconds are to be ignored by POSIX compliant systems.
> General comment: why some declarations have the name of arguments (usually hinting what this argument is about), but not all? Compare with
I meat to write:
Compare
unsigned int alarm(unsigned int seconds);
int adjtime(const struct timeval *delta, struct timeval *olddelta);
useconds_t ualarm(useconds_t usecs, useconds_t interval);
with
int nanosleep(const struct timespec *, struct timespec *);
int clock_gettime(clockid_t, struct timespec *);
…
In the float precision table (Table 1) shouldn’t 1 of those exponent bits actually be numerical sign? As here: http://en.wikipedia.org/wiki/File:Float_example.svg
>In the float precision table (Table 1) shouldn’t 1 of those exponent bits actually be numerical sign?
Right you are.
That’s why I put stuff like this up for review before I ship 1.0.
Fixed.
Typo: for “neans” read “means”.
The functions and groups of functions should be reordered by decreasing usefulness in modern programs.
You should clarify the distinction between time zone abbreviations like EST, which are potentially ambiguous (in North America it means one thing, in Australia something quite different) and potentially subject to localization into different languages on the one hand, and universal IANA/Olsen time zone names on the other. These latter names unify across areas that have had the same time since the epoch, but do not unify across national boundaries ever. In particular, strptime or similar functions cannot parse time zone abbreviations because of the ambiguity.
In the general case, past times are best represented as UTC with optional timezone information (for which local time plus then-offset is a reasonable proxy), but future times are best represented as local time with mandatory locality (for which IANA TZ name is a reasonable proxy) information. If you look, for example, at stock options, they say they expire on such and such a date at 5:00 PM, New York time. That is the only safe kind of future time.
As a result, programs that want to handle the past and the future uniformly have problems. I once tried to fix a friend’s Windows system that had been left on U.S. Pacific time (the then factory default), although he was in U.S. Eastern time, by changing the time zone setting. Unfortunately, that changed every one of his past and future appointments in Outlook by three hours. Fixing them all would have been intolerably tedious, so he simply had to live with the fact that his computer’s clock was three hours slow. Fortunately, he was not using any sort of distributed source control or similar application in which it mattered that his local time differed from other people’s time. (Note that this problem has nothing to do with the fact that Windows kernel time is local time.)
>The functions and groups of functions should be reordered by decreasing usefulness in modern programs.
Actually, my top ordering criterion is what produces good expository flow. But I’ll attempt to reorder this way as a tiebreaker.
You are one of several people that wants me to write a major section about timezones and timezone naming. I have two difficulties with this.
One is that it’s arguably out of scope and I’m not sure I want it in scope. Dealing with the IANA zone database may require – even deserve – an entire separate FAQ.
The other is that I have not yet found the primary documentation on which to base such a thing. Here is what I would need:
* URL to a description and rationale of the IANA/Olson naming scheme
* URL to a description and rationale of the POSIX naming scheme
* URL to a reasonably complete survey of civil zone name schemes (e.g. what people use when they’re not on computers).
* Pointers to reference pages for pre-POSIX zone naming schemes on Unix. Right now I don’t even know how many distinct kinds of these there were…
“You are one of several people that wants me to write a major section about timezones and timezone naming. I have two difficulties with this.
One is that it’s arguably out of scope and I’m not sure I want it in scope. Dealing with the IANA zone database may require – even deserve – an entire separate FAQ.”
This is currently a sticking point on the tzdist workgroup!
Not the least is the use of backslash in TZID references which cause problems making it a single element in URL lookups. And the second is identifying that different publishes may well not provide the same data for the same name … since that is out of scope of the charter of the workgroup :)
* URL to a description and rationale of the IANA/Olson naming scheme
The Theory file I linked earlier is the best you’re likely to get there.
* URL to a description and rationale of the POSIX naming scheme
TZ on this page.
The rationale is, realistically, that this came from SVR3.1 (with “std offset dst” coming from System III). The actual Open Group rationale mainly explains the one extension they made to the existing syntax.
* Pointers to reference pages for pre-POSIX zone naming schemes on Unix. Right now I don’t even know how many distinct kinds of these there were…
The closest thing I can find to a non-SysIII-style “zone naming scheme” is the timezone function in v7 and BSD, which looked up names from a table based on the offset, or (in 4.2BSD) from the TZNAME environment variable. BSD sets the timezone offset and daylight saving rules with the settimeofday function, independent of the name. This, in particular (in 4.2BSD) seems to be where names like “MET DST” [for Central European Summer Time] come from. The most complete set of names that I can find is in the modern OpenSolaris version of the function, which has more names than the modern FreeBSD version (but neither has, for example, MSK/MSD for Moscow time, which can be found in the Soviet version of Unix “DEMOS”.)
I don’t know what would be a good reference page – you could link to the source code or manpage of some version of the function on TUHS.
* URL to a reasonably complete survey of civil zone name schemes (e.g. what people use when they’re not on computers).
I don’t know if such a thing even exists.
>I don’t know if such a thing even exists.
I found a way to write the new section on timezone selection and representation in such a way as not to require it. Now uploaded.
> Unfortunately, that changed every one of his past and future appointments in Outlook by three hours.
Wouldn’t his appointments have showed up wrong on other people’s calendars (or appointments he was invited to by someone else show up at the wrong time) in that case?
@Lester Caine
“the use of backslash in TZID references ”
Where do TZID references use backslash?
Could you possibly mean the slashes in things like “America/Chicago”?
“The contents of the database entry for the designator describe the history of the zone name, zone offset and the DST start/end times (if any) for the location, as those have changed over time.”
It should be pointed out that an entry in the database does not actually correspond to a location, but rather simple to a set of information. That information may not be the proper history for a location, but a truncated set from a similar set of data, or it may be that even today the ‘official’ set of data for a location may be replaced with a different set much as different calendars are used in addition to the Gregorian calendar.
( and historical note 3 still needs a concerted exercise to be replaced with something that is actually correct all year – but that at least needs tzdist to be implemented )
Lester Caine, I am not at all clear on what changes your last comment was arguing for. Try to be more specific?
By the way, there is nice explanation about why floating point is a bad idea for representing dates in the blog post by Josh Haberman:
“What every computer programmer should know about floating point, part 1”
http://blog.reverberate.org/2014/09/what-every-computer-programmer-should.html
>“What every computer programmer should know about floating point, part 1?
That is good stuff. I’ve added a link.
Nb. it is funny that in the post (partially) about proper naming of dates we see WordPress software using “2014-10-02 at 04:13:17” date format *without* information about timezone (hint: it is not my local time…) ;-)
“By the way, there is nice explanation about why floating point is a bad idea for representing dates in the blog post by Josh Haberman:”
The ‘trick’ is to use floating point with days rather than seconds and part of the problem goes away. The main point here is that a lot of historic material we do not actually need the the time. i.e. the fractional part is ignored. On calendaring services date and time tend to be separate elements anyway which is why only using seconds counts for a date is what is the real problem.
I should perhaps add that the time element is not ACTUALLY a fraction, but rather a count which gets converted to a fraction only when calculating durations.
( And it’s now 10:00 here )
> By the way, there is nice explanation about why floating point is a bad idea for representing dates in the blog post by Josh Haberman:
The precision does get to be less as timestamps get larger (currently it is 1/2^22 second), but that’s not, by itself, a convincing reason to me. The “usual equality and comparison headaches” that Jon Lennox mentioned aren’t, either – it’s unusual to be attempting to compare a date for equality (rather than less-than-or-equal) and not to expect them to be exactly equal if the question you want answered (same version of a file e.g.) is true. It’s also not entirely clear why “micbiseconds” (1/2^20 seconds) are a worse unit to work in than microseconds, so “flaky in actual use due to floating point rounding” doesn’t fit either. The only way to truly solve that problem is to use an arbitrary-precision rational type – which might be appropriate for Python (which currently uses float for Unix times), but seems a bit heavyweight for C.
It would be nice if C had fixed-point (or fixed-denominator) types. Addition and subtraction is basically free in such types, as is multiplication or division by an integer (well, no worse than division generally), and multiplication or division of two instances of the type would be rare.
@Lester Caine: in short, the problem with using floating point (i.e. scientific exponential notation) is growing lack of precision the further you are from the starting point (time = 0.0).
“Lester Caine, I am not at all clear on what changes your last comment was arguing for. Try to be more specific?”
With regards TZID references?
The point is that locations ‘map’ to a particular reference. The reference is not a physical location but simply an id for a set of data. Just which set of data is used may depend on more than simply the location. This may seem a little academic, but current TZID data only applies from a particular date, and prior to that date a different set of rules need to be used to provide the offset, depending on how the supplied data has been truncated.
>The point is that locations ‘map’ to a particular reference.
So what?
That is, given the kind and level of exposition in the document, is there any actual point in making this distinction?
“@Lester Caine: in short, the problem with using floating point (i.e. scientific exponential notation) is growing lack of precision the further you are from the starting point (time = 0.0).”
It does depend on the manor of your floating point, but if the fractional part of the calculation is ‘part of day’ rather than part of second things can be looked at differently. I am STILL looking at this as a pair of 32 bit numbers, day and part of day. Makes storage and calculation easy on both 32 and 64 bit systems, removes the need to use 64 bit value if all you need is the date, and many other advantages. Using a 64bit floating point value fixes some problems, but still leaves the extra load of 32 unnecessary bits when handing date only material. If you need more than mS resolution on the time, then the time element needs to be 64 bit rather than 32, but for the majority of logging timestamps, millisecond accuracy is more than enough.
Along with tzset, the tzcode implementation (and so most modern unixes) also has “tzsetwall”, which sets the timezone to a central system-configured one [e.g. /etc/localtime] rather than one from the environment.
The non-static equivalent (on NetBSD and very recent versions of tzcode) to tzset, for use with localtime_rz and mktime_z, is tzalloc – which takes an argument that can either be NULL [like tzsetwall], empty string [UTC], or a zone name like in the environment variable.
>Along with tzset, the tzcode implementation (and so most modern unixes) also has “tzsetwall”
As I saus to Lester Caine: I think we’re getting unnecessarily far off into the weeds here, but if you can suggest a specific patch to the text I’ll consider it.
“The point is that locations ‘map’ to a particular reference.
So what?
That is, given the kind and level of exposition in the document, is there any actual point in making this distinction?”
I think there is since the information supplied only applies for a short period of time and in some cases the data may actually be different outside of the time period actually covered, but the information to that fact may not even be supplied by the source.
>I think there is since the information supplied only applies for a short period of time and in some cases the data may actually be different outside of the time period actually covered, but the information to that fact may not even be supplied by the source.
I think we’re getting unnecessarily far off into the weeds here, but if you can suggest a specific patch to the text I’ll consider it.
“I think we’re getting unnecessarily far off into the weeds ore, but if you can suggest a specific patch to the text I’ll consider it.”
While your text on timezones ( or is that time zones to keep some people happy ) covers most of the current state of play. It WOULD be useful to add the following.
…
Care should be taken when using TZ data for historic purposes as the TZ database is only scoped to provide data from 1970 and some historic data may be truncated depending on just what data is included when compiling a copy for local use.
…
If we get things right on the tzdist protocol, the actual cutoff dates will be identified properly but for now that information is not published.
> As I saus to Lester Caine: I think we’re getting unnecessarily far off into the weeds here, but if you can suggest a specific patch to the text I’ll consider it.
I don’t understand why you don’t consider these to be directly in scope, unless you are under the misapprehension that it is one of the “corpses of failed experiments”, as you seem to have been confused into believing tzset itself was an example of.
The problem is, the variables like timezone/tzname are, but the purpose of tzset on a modern system isn’t to set those variables, it’s to set the enormous amount of hidden state used by localtime and mktime. And tzalloc/tzfree/localtime_rz/mktime_z is looking like it’s going to be the new “right way to do things” once it’s sufficiently widely supported.
While on POSIX systems where time_t is required to be an integral type hilding -> holding
One caveat to watch out for – if called with a manually populated struct tm with all Standard C members populated, but without the tm_gmtoff and tm_zone members populated, strftime will, on some systems, not give sensible results for %z or %Z. Worse, it directly reads from the uninitialized tm_zone pointer.
>One caveat to watch out for
That’s a good one. Added.
More generally, %z and %Z for anything other than an immediate mktime() / strftime() pair can get wacky.
In particular:
* If you’re on a system without tm_gmtoff and tm_zone, they’ll print the local time zone even for a gmtime()-initialized struct tm.
* If you change the value of TZ between mktime() and strftime(), if you have tm_zone, %Z will use a random zone name abbreviations for the new zone, not the abbreviation for the old one. If the new zone has fewer abbreviations than the old one, tm_zone (and thus %Z) can even end up pointing into freed memory!
>More generally, %z and %Z for anything other than an immediate mktime() / strftime() pair can get wacky.
Yuck. This is all deserving of inclusion and I have done so.
>More generally, %z and %Z for anything other than an immediate mktime() / strftime() pair can get wacky.
You mean localtime.
Also, that section could use a rework. I doubt my observation deserves to be the lead, especially considering the gmtime issue is technically mandated by the standard (%z and %Z to only depend on tm_isdst), which tzcode and other systems with tm_gmtoff/tm_zone willfully violate because it’s dumb. Though, on the other hand, at least the dumb implementations never dereferenced a dangling pointer.
>Also, that section could use a rework.
I’d take a patch.
Also, glibc appears to use tzname[tm_isdst] in some cases, which works about as well as you would expect for tm_isdst > 1. Fortunately, this is rare and can never AFAIK be generated by glibc’s localtime or mktime.
Is the source form available so that patches can be made against that instead of the HTML?
>Is the source form available so that patches can be made against that instead of the HTML?
http://www.catb.org/esr/time-programming/index.asc