In the process of working on my Time, Clock, and Calendar Programming In C document, I have learned something sad but important: the standard Unix calendar API is irremediably broken.
The document list a lot of consequences of the breakage, but here I want to zero in on what I think is the primary causes. That is: the standard struct tm (a) fails to be an unambiguous representation of time, and (b) violates the SPOT (Single Point of Truth) design rule. It has some other more historically contingent problems as well, but these problems (and especially (a)) are the core of its numerous failure modes.
These problems cannot be solved in a backwards-compatible way. I think it’s time for a clean-sheet redesign. In the remainder of this post I’ll develop what I think the premises of the design ought to be, and some consequences.
The functions we are talking about here are tzset(), localtime(3), gmtime(3), mktime(3), strftime(3), and strptime() – everything (ignoring some obsolete entry points) that takes a struct tm argument and/or has timezone issues.
The central problem with this group of functions is the fact that the standard struct tm (what manual pages hilariously call “broken-down-time”) was designed to hold a local time/date without an offset from UTC time. The consequences of this omission cascade through the entire API in unfortunate ways.
Here are the standard members:
struct tm
{
int tm_sec; /* seconds [0,60] (60 for + leap second) */
int tm_min; /* minutes [0,59] */
int tm_hour; /* hour [0,23] */
int tm_mday; /* day of month [1,31] */
int tm_mon ; /* month of year [0,11] */
int tm_year; /* years since 1900 */
int tm_wday; /* day of week [0,6] (Sunday = 0) */
int tm_yday; /* day of year [0,365] */
int tm_isdst; /* daylight saving flag */
};
The presence of the day of year and day of week members violates SPOT. This leads to some strange behaviors – mktime(3) “normalizes” its input structure by fixing up these members. This can produce subtle gotchas.
Also, note that there is no way to represent dates with subsecond precision in this structure. Therefore strftime(3) cannot format them and strptime(3) cannot parse them.
The GNU C library takes a swing at the most serious problem by adding a GMT offset member, but only half-heartedly. Because it is concerned with maintaining backward compatibility, that member is underused.
Here’s what I think it ought to look like instead
struct gregorian
{
float sec; /* seconds [0,60] (60 for + leap second) */
int min; /* minutes [0,59] */
int hour; /* hour [0,23] */
int mday; /* day of month [1,31] */
int mon; /* month of year [1,12] */
int year; /* years Gregorian */
int zoffset; /* zone offset, seconds east of Greenwich */
char *zone; /* zone name or NULL */
int dst; /* daylight saving offset, seconds */
};
Some of you, I know, are looking at the float seconds member and bridling. What about roundoff errors? What about comparisons? Here’s where I introduce another basic premise of the redesign: integral floats are safe to play with..
That wasn’t true when the Unix calendar API was designed, but IEEE754 solved the problem. Most modern FPUs are well-behaved on integral quantities. There is not in fact a fuzziness risk if you stick to integral seconds values.
The other way to handle this – the classic Unix way – would have been to add a decimal subseconds member in some unit, probably nanoseconds in 2014. The problem with this is that it’s not future-proof. Who’s to say we won’t want finer resolution in a century?
Yes, this does means decimal subsecond times will have round-off issues when you do certain kinds of arithmetic on them. I think this is tolerable in calendar dates, where subsecond arithmetic is unusual thing to do to them.
The above structure fixes some quirks and inconsistencies, The silly 1900 offset for years is gone. Time divisions of a day or larger are consistently 1-origin as humans expect; this will reduce problems when writing and reading debug messages. SPOT is restored for the calendar portion of dates.
The zoffset/zone/dst group do not have the SPOT property – zone can be inconsistent with the other two members. This is, alas, unavoidable if we’re going to have a zone member at all, which is pretty much a requirement in order for the analogs of strftime(3) and strptime() to have good behavior.
Now I need to revisit another basic assumption of the Unix time API: that the basic time type is integral seconds since the epoch. In the HOWTO I pointed out that this assumption made sense in a world of 32-bit registers and expensive floating point, but no longer in a world of 64-bit machines and cheap floating point.
So here’s the other basic decision: the time scalar for this library is quad-precision seconds since the epoch in IEEE74 (that is, 112 bits of mantissa).
Now we can begin to sketch some function calls. Here are the basic two:
struct gregorian *unix_to_gregorian(double time, struct gregorian *date, char *zone)
Float seconds since epoch to broken-down time. A NULL zone argument means UTC, not local time. This is important because we want to be able to build a version of this code that doesn’t do lookups through the IANA zone database for embedded applications.
double gregorian_to_unix(struct gregorian *date)
Broken-down time to float seconds. No zone argument because it’s contained in the structure. Actually this function wouldn’t use the zone member but just the zoffset member; this is significant because we want to limit lookups to the timezone database for performance reasons.
struct gregorian *gregorian_to_local(struct gregorian *date, char *zone)
Broken-down time to broken-down time normalized for the specified zone. In this case a null zone just means normalize so there are no out-of-range structure elements (e.g. day 32 wraps to the 1st, 2nd, or 3rd of the next month) without applying any zone change. (Again, this is so the IANA timezone database is not a hard dependency).
Notice that both functions are re-entrant and can take constant arguments.
An auxiliary function we’ll need is:
char *local_timezone(void)
so we can say this:
unix_to_gregorian(time, datebuffer, local_timezone())
We only need two other functions: gregorian_strf() and gregorian_strp(), patterned after strftime() and strptime(). These present no great difficulties. Various strange bugs and glitches in the existing functions would disappear because zone offset and name are part of the structures they operate on.
Am I missing anything here? This seems like it would be a large improvement and not very difficult to write.
> struct gregorian *unix_to_gregorian(double time, struct gregorian *date, char *zone)
What is the second argument for?
gregorian_to_local seems superfluous (but useful for conciseness): it seems to be equivalent to unix_to_gregorian(gregorian_to_unix(.), .)
As far as floating point seconds are concerned: this doesn’t future-proof the structure, because the field has a small-sized mantissa.
The precision of your doubleword seconds-since-epoch is higher than the precision of “struct gregorian”: thus gregorian_to_unix(unix_to_gregorian(t, foo)) is not necessarily equal to t. This seems to me to be a desirable property.
Hm. The biggest issue I see is embedded applications without floating point hardware. Perhaps you can ameliorate it by making seconds an int and adding a float fractional_second value; that allows ignoring float math altogether if all you care about it integral seconds. Not sure what to think about the double unixtime, though.
I think calendar handling should ideally be almost entirely separate from timezone handling – a function that gives an hours/minutes/seconds breakdown should simply return an epoch count of days, which a calendar API (for gregorian or other calendars) can be asked questions about.
I believe that your “sec” member should not have an upper limit of 60 if you want to allow for subsecond precision as well as leap seconds, simultaneously. sec=59.999 is one millisecond before the end of a normal minute. A minute that includes a leap second would end one millisecond after sec=60.999. So the definition should be either [0,61] or maybe [0,61).
> The other way to handle this – the classic Unix way – would have been to add a decimal subseconds member in some unit, probably nanoseconds in 2014. The problem with this is that it’s not future-proof. Who’s to say we won’t want finer resolution in a century?
For *wallclock* time (as opposed to time period for e.g. profiling) we have limited precision because of physics: special and general relativity makes it impossible to synchronize (and stay synchronized) clocks beyond some finite precision.
Also GHz CPU frequency translates to nanosecond precision… and we don’t see THz computers (IIRC 8 GHz is maximum we have ever seen and will ever see).
>Also GHz CPU frequency translates to nanosecond precision… and we don’t see THz computers (IIRC 8 GHz is maximum we have ever seen and will ever see).
What’s your reason for believing this?
It would certainly be simpler if I could assume that nanoseconds is the most resolution we’ll ever need. In that case the new equivalent of time_t could be nanoseconds since the epoch and there would be no float operations amywhere. But confident predictions of “computers will never be faster than X” have had a way of coming unstuck in the past.
> Broken-down time to broken-down time normalized for the specified zone. In this case a null
> zone just means normalize so there are no out-of-range structure elements (e.g. day 32
> wraps to the 1st, 2nd, or 3rd of the next month) without applying any zone change. (Again,
> this is so the IANA timezone database is not a hard dependency).
I would’ve assumed NULL to mean conversion to UTC (and normalization) here.
>I would’ve assumed NULL to mean conversion to UTC (and normalization) here.
That was the other possibility. If you think through the use cases on a machine without access to the timezone database I think you’ll see why I landed here.
I think it’s actually fine to represent a timestamp without an offset – the problem with struct tm is that it purports to represent the timezone (via isdst in the standard), and that the functions that require this information therefore have no other way to acquire it.
Just putting some ideas out there:
struct timestruct { long day; int hour, minute, second; };
struct zonestruct { int gmtoff, isdst; char zonename[system dependent fixed width]; };
const struct zonestruct zone_UTC = {0, 0, “UTC”};
// zone_UTC for passing into formatting functions that require it
void current_utc(struct timestruct *out_time);
// need this because time_t is just as obsolete as the rest, it can’t represent leap seconds…
void utc_to_local(struct timestruct *in_time, struct timestruct *out_time, struct zonestruct *out_zone);
void unix_to_utc(time_t t, struct timestruct *out_time)
enum { CAL_YEAR, CAL_MONTH, CAL_MDAY, CAL_YDAY, CAL_YWEEK, CAL_WDAY, CAL_ERA }
struct calendar_api {
// all members are function pointers, declared as functions to save typing, consider this pseudocode
int get_component(days_t d);
days_t add_component(days_t orig, int component);
ssize_t name_component(char *buf, size_t len, int component, int value);
};
ssize_t time_format(char *buf, size_t len, struct timestruct *in_time, struct zonestruct *in_zone, struct calendar_api *calendar)
in_time is mandatory (or default to current time in local time?)
If zone is missing, zone-related components are not allowed.
If calendar is missing, calendar-related components (Y m d etc) are not allowed. (or default to proleptic gregorian with astronomical negative years? Do we want to continue with strftime’s ability to use %E for a locale (or in this case passed-in) calendar and plain %Y-etc for gregorian?)
> quad-precision seconds since the epoch in IEEE74 (that is, 52 bits of mantissa).
IEEE 754 quadruple-precision floats have 112 bits of mantissa. From the code below, it looks like you meant double-precision? If you’re measuring in seconds since the epoch, that’ll let you represent times around now with a granularity of about 240 ns or so.
>IEEE 754 quadruple-precision floats have 112 bits of mantissa.
My error. That’s the number I meant to use.
What springs to my mind is this:
http://xkcd.com/927/
I don’t think there is any need for a structure physically representing the calendar components, and especially month and mday components – they can be calculated on demand… the struct tm is just the result of calculating them all on-demand at once, with predictably poor consequences when the structure is used as an input.
A general calendar API needs to support at least the following cases:
One-era proleptic gregorian calendar with astronomical year numbering
Two-era Julian calendar with BC/AD eras (and a way to at least [i]format[/i] BC years without a minus sign)
Three-era? Julian/Gregorian BC/AD-OS/AD-NS calendar, including a switch of era in the middle of a month, and a month that does not have certain days in it.
Multi-era Japanese calendar.
Hebrew calendar with extra month on leap years.
>I don’t think there is any need for a structure physically representing the calendar components,
It’s useful when you want to do calendar-sensitive arithmetic – “date 6 months from now”, that sort of thing.
int zoffset; /* zone offset, seconds east of Greenwich */
char *zone; /* zone name or NULL */
int dst; /* daylight saving offset, seconds */
You are making the classic mistake of assuming that a timezone has exactly two offsets. Your structure also fails to answer which of the two offsets applies to the time it describes.
A moment in time has exactly one offset. A timezone in the form of the full rules applying to a location over many years can have a large number.
I would much rather see the data storage be needed on Julian days, probably stored in a long int, rather than carrying all the calendar date data. Then write simple functions to convert Julian dates to and from calendar datea and times in a specified timezone.
Locale should not be stored with the master copy of the date and time.
One long int for Julian date, an int or short for seconda that day, and an optional double for fractions of a second to very high precision. Voila, a unique representation of exactly one time and date, subject only to leap-second and relativistic effects.
> It’s useful when you want to do calendar-sensitive arithmetic – “date 6 months from now”, that sort of thing.
That was what my add_component was for. Doing arithmetic in a struct and renormalizing was always a hack. Speaking of, might need to add a flag for whether the month-end is “sticky” (i.e. Feb 28 to Mar 31) or not (Feb 28 to Mar 28).
I’d take a good hard look at the Java time and calendaring classes, if you haven’t. They’ve wound up having to attack the problem three times before they got to a really truly good solution, with all the edge, usability, and loophole cases adequately handled.
In fact, the third version of the APIs they finally adopted came from the Apache-licensed Joda Time project. See http://joda-time.sourceforge.net/old-index.html for details.
> One long int for Julian date, an int or short for seconda that day, and an optional double for fractions of a second to very high precision. Voila, a unique representation of exactly one time and date, subject only to leap-second and relativistic effects.
If we’re going to represent leap seconds, then minutes at least need to be separate from seconds, and I have reservations about allowing timezone offsets to be in seconds.
Obviously a C API will look different, but the approach of counting integral milliseconds since UTC epoch has always been at the heart of the Java approach, and there’s a vast amount of programmers and code using that standard, and all the calendar calculations for many different types of calendars have already been worked out.
I’ll offer a few observations and questions:
1. Perhaps I’m missing something, but I don’t understand the point of having a dst offset member. Isn’t zulu offset sufficient? Perhaps it’s just semantics, but in the UK, the way we think about “daylight savings” is that we switch timezones twice a year. Half the time we are in GMT (with an offset of 0), and half the time we are in BST (British Summer Time) (with an offset of +1). There’s no additional special dst offset; we just change our zoffset. That seems to me the correct model.
2. If you want to have the seconds member be a float, why not have a float for zoffset as well?
3. You say you want to have a ‘zone’ member ‘in order for the analogs of strftime(3) and strptime() to have good behaviour’. Could you explain this? It seems like a really desirable goal to me to have one member only that defines time zone (which would be zoffset). I don’t think zone is required to unambiguously represent a time.
4. Is the year int signed? I am thinking about representing times prior to 0AD.
5. Are you familiar with the types here: http://cr.yp.to/libtai/tai64.html (specifically the N and NA variations, which give nano- and atto-second precision timestamps respectively).
@esr:
A quick bit of digging shows that a 32-bit IEEE-754 float only provides 23 bits worth of mantissa. Assuming my calculations are correct, you need 20 bits to represent a value with the precision of a microsecond, and 30 bits to represent a value with the precision of a nanosecond. In this case, using a float for seconds value isn’t future-proofing against a nanosecond count – it’s providing less precision. A uint64_t of attoseconds would provide 12 times more precision than the smallest measured period of time.
For entertainment, and as a way to use two of your major interests at once, I’d suggest looping some of your sample raw GPS data through any proposed library/encoding scheme to see how much of a difference it makes on the resulting calculated location.
Finally, you’d probably need to add helper methods to allow somebody to extract the existing over-determined fields before (day of week, etc). I have to assume that somebody had a use case for them at some point in time.
I think Jay’s concern about floatless embedded systems is worth considering, though overwrought. This would be a new library function used in new systems. Also, something as primitive as an 8051 microcontroller has software floating-point libraries available for it. ARM now comes with hardware floating point.
Joda Time has been ported to .NET, Objective-C / Cocoa, and Android. I’ve found Joda Time support on Ruby and Python when run on the JVM, and talk of porting it to Rust, at least.
Ironically, when Joda Time was made part of the Java spec with JSR-310, the original author of Joda Time wound up adding additional improvements and refinements.
Time/Date/Calendaring is deceptively hard to get right.
Variable month lengths means that doing arithmetic in a component structure is pathological around month ends.
Proof:
Jan 31 + 1 month = Feb 31 = Feb 29 + 2 days
Feb 29 + 1 year = Feb 29 (of 28) = Feb 28 + 1 day
And don’t tell me to just add 86400 (or 1 day) to the un-calendared timestamp if I wanted to add days – the math for working in months and years alone isn’t hard, and doesn’t merit a normalization facility.
In effect, adding one month is equivalent to adding the length of the current month. Which isn’t bad per se, but people should have to think about what they mean by adding a number of months (whether or not that number is a multiple of twelve) to a date near the end of the month.
“Variable month lengths means that doing arithmetic in a component structure is pathological around month ends.
Proof:
Jan 31 + 1 month = Feb 31 = Feb 29 + 2 days
Feb 29 + 1 year = Feb 29 (of 28) = Feb 28 + 1 day
And don’t tell me to just add 86400 (or 1 day) to the un-calendared timestamp if I wanted to add days – the math for working in months and years alone isn’t hard, and doesn’t merit a normalization facility.”
This is exactly the kind of thing that the Java community took so much time to get a truly good solution to. ;-/
You could define fractions of a second using integral types by:
A single integral value, with a hardcoded #define for the number of units per second. Changing the #define is no more difficult than changing the number of bits.
An implementation-defined non-array type, with macros to convert it to/from milliseconds/microseconds/nanoseconds.
These macros would also let you make sure the conversions round down. With time fractions, just as 12:59:59 is still 12:59, 59.999999 is still 59.999 is still 59 – you are “in” this millisecond until the start of the next millisecond.
Why not just stop using time zones IRL for the most part. ;)
(when you can’t change the solution to fit the problem…)
Although, it would just be a continuation of the adoption of official rather than local time, caused by the railway.
Here’s a brief explanation for why Java wound up needing a third attempt to get calendaring right, from the JSR-310 document:
“Currently Java SE has two separate date and time APIs – java.util.Date and java.util.Calendar. Both APIs are consistently described as difficult to use by Java developers on weblogs and forums. Notably, both use a zero-index for months, which is a cause of many bugs. Calendar has also suffered from many bugs and performance issues over the years, primarily due to storing its state in two different ways internally.
One classic bug (4639407) prevented certain dates from being created in a Calendar object. A sequence of code could be written that could create a date in some years but not in others, having the effect of preventing some users from entering their correct birth dates. This was caused by the Calendar class only allowing a daylight savings time gain of one hour in summer, when historically it was plus 2 hours around the time of the second world war. While this bug is now fixed, if at some point in the future a country chose to introduce a daylight savings time gain of plus three hours in summer, then the Calendar class would again be broken.
..
As well as the problems with the classes that Java SE has for datetime, it has no classes for modelling other concepts. Non-time-zone dates or times, durations, periods and intervals have no class representation in Java SE. As a result, developers frequently use an int to represent a duration of time, with javadoc specifying the unit.
The lack of a comprehensive date and time model also results in many common operations being trickier than they should be. For example, calculating the number of days between two dates is a particularly hard problem at present.
This JSR will tackle the problem of a complete date and time model, including dates and times (with and without time zones), durations and time periods, intervals, formatting and parsing.”
Working out how to handle time and calendaring right in all cases is almost as bad as the struggles the Unicode folks have had to deal with.
“I don’t think there is any need for a structure physically representing the calendar components, and especially month and mday components – they can be calculated on demand… the struct tm is just the result of calculating them all on-demand at once, with predictably poor consequences when the structure is used as an input.”
This++.
The unix_to_gregorian and gregorian_to_unix functions still need access to a database of historical leap seconds.
Also, using strings to refer to refer to time zones bugs me. I’m not sure I have a better idea, but:
* It means we’ll be doing string-based database lookups a lot (every modifier function) in what could be performance-relevant code.
* It’s not clear what the behaviour should be if the user gives a string that’s not in the database. Ideally, there should be a way to check for this error condition that doesn’t drive programmers crazy.
* It’s asking for trouble next time there’s a civil war in Myanmar and the question of whether to refer to UTC+6:30 as “MST” or “BST” becomes one people get shot over. But maybe I’m being paranoid.
Eric, if you’re going to be using float number for seconds, it might worth having a field in the gregorian struct to explicitly indicate the precision to which the seconds are known.
Also, do you want to measure offset against GMT, or UTC? Do you intend a separate calendar struct for UTC time? Traditionally, Unix has used UTC for epoch, not GMT..
> The unix_to_gregorian and gregorian_to_unix functions still need access to a database of historical leap seconds.
No they don’t, because a unix timestamp can’t represent leap seconds. What you mean is that a conversion function for the time_t of the tzcode project’s attempt to support leap seconds (which is not a unix timestamp) would need one – but that format is not to be used for interchange, and on systems where it is used locally the database already exists.
According to
http://www.slideshare.net/JustinSDK/2013-java-developer-day-joda-timejsr310
the official timezones are relative to UTC, not GMT.
@ Daniel Speyer
BST already stands for British Summer Time. You can’t have +1 and -10 or so getting confused.
My problem with floating point is that at the epoch, you can go down to time values below planck’s constant, but geo-logical, you might not get second granularity.
Maybe if all floats are IEEE7xx, they behave well, but “ONLY USE THIS ON WELL BEHAVED HARDWARE” is not something I’d ever put in a spec. If it is dependent on a particular behavior, then use a specific soft-float double library known to have the correct behavior.
FPUs may play well with integers that stay integers, but what happens during a divide or square (or higher) root? “What day is 1/3d since X” is a question. You know the answer with metaphysical certainty?
Subtle and hidden bugs come from treating discrete values as continuous. They work well until some combination of these errors accumulates.
Older accounting packages had a BCD library for such purposes, post FPU.
Speaking of accounting, what API do common spreadsheets use? They have a superset of functions and didn’t need the historical baggage.
I’ll echo Jonathan Abbey’s call to take a serious look at Joda Time and JSR-310. It is an excellent library.
One of Joda’s main insights–and something that I’ve found completely true–is that it is a mistake to mix dates and timestamps. We humans use the two concepts in different fashions, even though they are conceptually linked. When you mix the two, you often find comparisons going subtly wrong, and are constantly adjusting timestamps to beginning-of-day or end-of-day. Joda explicitly differentiates between the two, forcing explicit conversions. This makes date handling a lot more sensible.
It occurs to me – there’s a lot to potentially disagree with… a lot I disagree with… in it – but a discussion of a new time API in C would be incomplete without a mention (and ideally a thorough analysis) of last century’s attempt to redesign the C time APIs.
And of course, deficiency number one in that proposal was: Not designed to be packaged as a third-party library. From use of reserved names to demands that its functions be declared in an existing standard header, this is absolutely 100% a standard-extension proposal and therefore fundamentally incapable of getting any grassroots uptake.
A (32 bit) float only has 24 bits of precision, or for 1 second, 1 part in 16 million. If seconds can be larger (e.g. 59.xxz), six of those bits will be needed for the integer, leaving 18 bits, or 1/262144, which can’t even do microsecond precision.
Hm, apparently Joda has standardized on nanoseconds since epoch, not milliseconds like Java’s original Date API.
Apparently the GPS system only goes down to 14 nanosecond resolution. Grace Hopper’s nanoseconds just ain’t that long:
http://blog.americanhistory.si.edu/.a/6a00e553a80e108834017d425591d0970c-500wi
And you can get time dilation variations of much more than a nanosecond a day without leaving the ground:
http://en.wikipedia.org/wiki/Hafele%E2%80%93Keating_experiment
It seems that nanoseconds should get the job done. Anything attempting to get below that will need to worry about far more than what the standard unix time code is.
Let me summarize:
Do. Not. Use. Floating. Point.
itype_t reciprocal; // typically 10 to a power specifying precision, e.g.1G for nS, 1M for mS, i.e.one second is this value, subseconds go from 0 to this-1 (maybe allow for overflow with threads and interrupts to avoid races if this directly reads hardware)
itype2_t subseconds; // number of subseconds
Later I’ll take a look at how the Joda API handles this problem, but:
“int tm_sec; /* seconds [0,60] (60 for + leap second) */”
“the time scalar for this library is quad-precision seconds since the epoch”
So it sounds like you’re abandoning the traditional jiggery-pokery of pretending not to have leap seconds, but as Daniel Speyer observes above, this means you need to load a leap-second database for all conversions to and from gregorian. I imagine this could be a problem for embedded systems.
One possibility for correct conversions without a database at hand would be to use TAI, which by definition actually has 86400 SI seconds/day, for those cases instead of UTC. However, under the current design it seems like one would want to treat TAI as a time zone, and would still need that leap-second database to set zoffset correctly…
I may be overthinking this :)
(oops: by “the current design” I meant “the design currently being proposed”, not the one currently in use!)
> No they don’t, because a unix timestamp can’t represent leap seconds.
The “double time” parameter is documented as “float seconds since epoch” and not “hideous kludge that needs to die in a fire”, but I suppose the function name does imply the latter.
This means that unix_to_gregorian(gregorian_to_unix(x),x->zone) will not return x for sec>60. That’s the sort of thing that’s likely to cause surprising and hard-to-debug problems.
Also, if we’re fixing unix time, handling leap seconds cleanly should be high priority. These other problems are annoying — leap seconds have been known to bring down large datacenters.
“It occurs to me – there’s a lot to potentially disagree with… a lot I disagree with… in it – but a discussion of a new time API in C would be incomplete without a mention (and ideally a thorough analysis) of last century’s attempt to redesign the C time APIs.”
Ooh, neat. Thanks Random.
The other problem is there are such things as legacy and embedded linux. Not even all ARM processors implement floating point in hardware. 64 bit seconds since epoch suffices (and if the sheet is clean, why not y2k instead of 1900 or 1970?).
I can implement the “broken” API in an arduino. I can’t do it with doubles. And time is one of the more universally needed items – even for a binary LED clock. (OpenWRT uses linux on really limited chips)
In an era of cheap ssds, why not one huge lookup table on disk?
You are only looking at the computer in front of you which has gained capability for a fixed price. But now you can run the same program on something for $20.
Linux and 99% of libc works fine without floating point.
There is nothing inherent in this time API which requires it or is made significantly cleaner by using floating point, and it introduces its own set of problems, and cuts off (or cripples) a significant amount of embedded linux that would have to use the old API (so if we have to keep it around, the new one is redundant) or write a third API devoid of float.
Floating point needs more cycles or transistors, so uses more power. Some things are designed with small batteries. Every bit flipped needs a small amount of power.
I believe that for real SPOT and future proof we should count just “day of the year” and not “month+day of month” as separation of year in month is quite stupid and non-metric.
> Speaking of accounting, what API do common spreadsheets use? They have a superset of functions and didn’t need the historical baggage.
Microsoft Excel uses a format which consists of a double precision floating point number representing a number of days since “1900-01-00”, with 1900 incorrectly defined as a leap year (1900-02-29 has a value of 60). Negative values are not allowed. Fractional seconds can only be displayed to a precision of milliseconds (the actual float value presently has a precision just over 5 microseconds), and are rounded rather than being truncated when displaying.
Most of this was inherited from Lotus 1-2-3.
Without a flag to the actual timezone data set being used how do you carry out comparisons between times which may be in different daylight saving offset periods?
The vast majority of changes to offset values relate to changes from a base of LMT local clock to a politically defined offset standard even prior to the start of what started as summer time, but then became referred to as daylight saving.
A timestamp that only contains an unqualified offset makes working with it just as bad a the current problem with the browser supplied ‘tz’ information.
Just to add to the fun, there is currently a thread on the TZ list relating to loosing politically sensitive names and replacing them with something generic instead.
Measuring time is, fundamentally, counting the ticks of something that ticks. It’s discrete, not continuous. Nothing is ever going to tick faster than the Planck time. So you can represent any sub-second time quantity in 150 bits (check the math, but I think it’s right). Memory keeps getting cheaper, and you can develop some clever representation that uses fewer bits for lower-precision values.
I’m afraid I, too, would prefer to avoid floating point whenever reasonably possible.
Random832 wrote: “Most of this was inherited from Lotus 1-2-3.”
Not only inherited but also deliberately kept for the sake of backwards compatibility, wasn’t it?
> Not only inherited but also deliberately kept for the sake of backwards compatibility, wasn’t it?
In the world of bad software designs, that’s what “inherited” means, I thought.
In terms of non-floating point, Java’s original 64 bit millisecond value went 320 million years before and after epoch. A 64 bit nanosecond value would do for a few hundred years around epoch, 96 bits will get you 2.5 trillion years around epoch.
One might compromise and use microseconds for the time code and a get a few hundred thousand around epoch.. maybe set epoch a couple of hundred thousand years hence to put off timecode extinction.. ;-)
I sometimes have to process large quantities of timestamps with an *unknown* timezone. It would be helpful to have a clear, unambiguous way of representing that, and an API call to then set (convert to) the correct timezones once I figure out what they actually are.
> Just to add to the fun, there is currently a thread on the TZ list relating to loosing politically sensitive names and replacing them with something generic instead.
That’s been going on for a while – ever since someone kicked over the anthill that is the Jerusalem zone, probably.
On my current installation, America/Chicago seems to be a symlink to US/Central, rather than vice versa.
Oh, but if you’re good with 128 bits, there’s no problem with nanoseconds. Plenty of space for the duration of millions of universes, possibly.
If you want to get fancy, you could add another 128 bit value for the point in the (observable?) universe that the time is being measured at, to ensure non-ambiguity after we discover Interstellar travel. ;-)
I think I said this before, but if you are leading something to correct time, I’ll say it again. What time it is, is independent of the way in which time is displayed to the user. Timezone has no place in a time structure.
There is certainly an argument for a wrapping it in a separate structure:
struct time { … }
and
struct localtime { struct time tm; char* tzone; } /* or whatever */
Adding local time into standard time makes for lots more complications in terms of arithmetic all of which are quite unnecessary.
> So it sounds like you’re abandoning the traditional jiggery-pokery of pretending not to have leap seconds, but as Daniel Speyer observes above, this means you need to load a leap-second database for all conversions to and from gregorian
Sounds like, but various standards declare the phrase “seconds since the epoch” to actually mean non-leap-seconds.
And I would object to that – you only need the database if you have linear timestamps to convert. I’m not convinced that we should be creating a new linear timestamp format at all.
I’m also not convinced of the use case for getting the number of seconds between two timestamps that isn’t better served by performance counters.
@Jessica Boxer It’s not just a matter of display, but also calculation. If I schedule lunch for noon monday, I may not want it to be 24 hours after noon sunday. Or 86400 seconds, if I’m the punctual sort and there’s a leap second.
If you’re going to advocate users should have to adjust their TZ manually for every change (and not have any way to encapsulate the past history of changes for their location), why not go all the way and abolish automatic daylight saving adjustment?
This article describes the complexities that the .NET port of Joda is meant to deal with that developers may not have realized existed:
http://blog.nodatime.org/2011/08/what-wrong-with-datetime-anyway.html
@Random832
> for noon monday, I may not want it to be 24 hours after noon sunday. Or 86400 seconds, if I’m the punctual sort and there’s a leap second.
Leap seconds are applied to UTC at the same UTC time throughout the world, irrespective of local TZ, and then this adjusted UTC is interpreted as a local time. In fact this is exactly the sort of reason you want to use UTC rather than local time.
“I think I said this before, but if you are leading something to correct time, I’ll say it again. What time it is, is independent of the way in which time is displayed to the user. Timezone has no place in a time structure.”
I’d totally agree with that. Store everything as UTC is rule one. To which end every server should be running with a synchronized UTC clock. But the problem is establishing just how to normalized data … and then ensure that the same rules are used when later handling that data. Especially when the data we are currently using keeps being ‘tidied’ :(
esr – a zone name is not a reliable means of identifying the correct data set. Certainly there is little point having two offset values without the information on where each applies, and in many cases a data set may have several offset values over time. We need a reliable source of data as stage one, something which TZ only provides post 1970 currently. Going back before 1970, the actual location of the event is required rather than the zone, but again there is no reliable data source that will correctly convert that reliably for the 19th and early 20th century. The location is also important in areas where boundaries may change in future.The Crimea being a case in point today. Which is why the statement that a TZID is NOT a location becomes important.
> Leap seconds are applied to UTC at the same UTC time throughout the world, irrespective of local TZ, and then this adjusted UTC is interpreted as a local time. In fact this is exactly the sort of reason you want to use UTC rather than local time.
The reason I mentioned leap seconds has nothing to do with timezones, it’s just another example where a time the next day will not occur exactly 86400 seconds after the previous one.
If I schedule a recurring appointment in UTC, they will be an hour off from when I want them to be for half the year. If I schedule a future appointment, recurring or otherwise, with UTC or a baked-in offset, then they will be an hour off from when I want to be if the timezone or daylight rules change between now and then. This is not an uncommon use case at all. Current and past times can be in UTC or have a mere offset, but future times need a reference to a set of rules for most things they are used for.
Floats (or doubles) are non-ideal for seconds, since resolution close to 0.0 is very fine, while much coarser when having to represent 59.* seconds.
When the value is greater than 32, the exponent is unchanging. When the value is less than 1, you have a lot of variability in the exponent (which allows a float to resolve nano-seconds for a very small window above 0.0, but not even close to that around 59.0).
@Random832
> If I schedule a recurring appointment in UTC, they will be an hour off from when I want them to be for half the year.
If you write your scheduler in a really stupid way perhaps, but if you write your scheduler so that it interprets the recurrence based on what one would expect — local time, then this is not correct. If you think a daily appointment is set up as an infinite number of appointments, each 86400 seconds ahead of the previous one, I’d suggest that is a crazy design.
I recommend you read RFC 2445 (iCalendar) which discusses recurring appointments at great length and some of the subtitles of them. The simple way you are proposing that they work bears absolutely no similarity to what actually happens. Five or six years ago I tried to implement it, it was a nightmare (especially with respect to recurring appts) so much so that I have only managed a partial implementation.
And Eric, FWIW, it might be a good thing for you to read and reference if you are putting out a new time type RFC.
>If you write your scheduler in a really stupid way perhaps, but if you write your scheduler so that it interprets the recurrence based on what one would expect — local time, then this is not correct.
That is EXACTLY the feature you are proposing to abolish. To interpret it as local time you have to store it in local time (not in UTC).
What iCalendar files I have had occasion to see the guts of, provided a finite list of specific timestamps that each instance of a repeating appointment occurs at. Your proposal was for storing things like this list in UTC, and displaying it in the user’s TZ.
“If you write your scheduler in a really stupid way perhaps, but if you write your scheduler so that it interprets the recurrence based on what one would expect — local time, then this is not correct. ”
If you are only scheduling meeting for a single location, one can make a few shortcuts. With the increasing use of video conferencing for meetings such as medical reviews, where sites can be several hours apart, the shortcuts don’t work. It is exactly these sorts of events we have occasional problems with where some sites have no DST while other have ‘conflicting’ ones. One has to work UTC based to ensure everyone is at the same time, but with Middle east sites the offset can change at short notice, and it is information that an offset may be subject to a religious/astronomically controlled change to the transition would be useful :)
Why not 64.64 fixed point representation for seconds? Arithmetic is pretty easy, you don’t need floating point hardware, you can do fewer fractional bits on really constrained systems, and it’s no bigger than the quad precision float that was suggested.
You do run into approximation issues for decimal times, but you can get pretty darn close even for femtoseconds. It’s about two orders of magnitude better than any man-made system can distinguish right now.
@Lester Caine – it occurs to me that if a meeting is scheduled in UTC (or in some European timezone), the only people whose ability to work around the meeting will be affected will be the people who just changed their clocks. The fact that the clocks just changed is fresh in my mind for at least a week after each shift. Do the people at the Middle East sites not know that their time might change on short notice?
@Random832
> That is EXACTLY the feature you are proposing to abolish. To interpret it as local time you have to store it in local time (not in UTC).
You are incorrect. This sort of thing is stored as a generator function parameterized with a start date time (usually in zulu) and a repeat definition. If you are storing your daily status meeting as a list of literal date-times with no end date, that iCalendar file is going to be mighty big.
> This sort of thing is stored as a generator function parameterized with a start date time (usually in zulu) and a repeat definition.
Wait, why would you put the original start date/time in UTC when the generator function needs it in local?
The file I was looking at was one with an oddball repeat type that Outlook couldn’t import and was generated by Lotus Notes, so it’s probably not representative. But all the times in the list were given in a local timezone.
If I have a recurring appointment at 19:30 Mondays, it needs to be able to:
– schedule it for 00:30 UTC Tuesdays when daylight saving time is not in effect
– schedule it for 23:30 UTC Mondays when daylight saving time is in effect
– reschedule future instances to be an hour earlier or later if new rules are put in place for daylight saving transitions
– reschedule future instances (to whatever time, but probably an hour later) if the time zone of the location the meeting is held in is changed
– not reschedule past instances in any of these cases.
In order to do this, even if the start date/time is _stored_ as 00:30 UTC Tuesday for god-knows-why reason, the generator has to basically immediately convert it to the timezone/location specifier stored along side it and carry out all further calculations knowing that what that actually means is 19:30 local Monday.
@Daniel Speyer:
>Also, if we’re fixing unix time, handling leap seconds cleanly should be high priority.
Can leap seconds even be handled cleanly? I suppose as long as leap seconds remain a thing we have to try, but I’d prefer to get rid of the leap second and split the second into two units: The atomic second and the calendar second, with the atomic second being a precise physical length of time, and the calendar second being precisely 1/86400 of a day, however long that day happens to be.
I’d like to second Jessica Boxer and others when they say time zone is not part of a time value and UTC is the correct format for a time value.
All other time formats besides UTC can be converted to/from local time zone, remote time zone, gregorian, julian, whatever, but UTC is better to calculate in.
Also, floating-point math has gotchas as I’m sure you know. Using it for something as important as time scares me.
@Random832 on 2014-10-03 at 23:44:03 said:
> Wait, why would you put the original start date/time in UTC when the generator function needs it in local?
OK we can go round and round on this a million times. There are a number of reasons why this would be the correct choice. But let me put it this way: the people who write these specs and write the code that do this kind of thing, do it the way that I am telling you. Go ahead and do it differently if you like, but RFCs encapsulate best practice, which I presume is what Eric is going for, and this seems to be best practice.
If you actually want to understand some of the astonishing subtitles of this stuff rather than just argue with me I’d recommend you read the RFC itself. It seems redundant to rehearse them here over again.
@Random832
Oh, or better yet, read the RFC then actually code it. Because as you know you don’t really fully grok the details of something like this until you have squeezed the last bug out of your unit tests.
Write it for Windows and sell it, you’ll make some good bank, because last time I checked there was only one version available in Windows and it was insanely slow. I talked to the author. He told me it was slow because recurring appointments are so complex.
Long ago, when writing a program to display orbital trajactories, I made everything revolve around Julian days (note that this is not the same thing as the Julian calendar).
The two key functions were:
double convert_calendar_to_julian(int year, int month, int day,
int hours, int minutes, int seconds);
Calendar_Date *convert_julian_to_calendar(double jdate);
The struct, used for display purposes and keyboard input ONLY, was:
typedef struct {
int year;
int month;
int day;
int hour;
int minute;
int second;
} Calendar_Date;
But the actual date was simply stored in a double. The integer part represented days in the Julian calendar, which lets any date for many millions of years be represtend. The hour, minute, and second were stored as fractions of a day.
For the RFC, I would keep the idea of Julian days as the fundamental, universal measure of a date, but store it in an int. Then add a second int to represent seconds within that day. Two ints can represent anything you need down to the second level. If you need to go to smaller time intervals, add another int that can be ignored if divisions that fine are unneeded.
struct TimeStamp {
int julian_day;
int seconds_of_day;
int microseconds; /* or nanoseconds, etc. */
}
Then have some simple functions that go back and forth easily between a date stored as a date in a single double (Julian date and fraction of day), the struct listed above, and any other forms needed with timezone, etc.
But for storage and interchange purposes, never, never store a timezone or a date/time in anything other than Julian days plus seconds (or fractions of seconds if needed).
> Go ahead and do it differently if you like, but RFCs encapsulate best practice, which I presume is what Eric is going for, and this seems to be best practice.
The iCalendar RFC has plenty of stuff about times with timezone information attached, I don’t know what RFC you read.
And the reason it has that stuff? Because user expectations (which theoretical best practices don’t actually trump for consumer software) are that when they put 7:00 then it means 7:00 – i.e. the time that the clock on their wall hits 7:00 on that day, no matter WHAT the state or federal government has them do to that clock between now and the day of the appointment.
The question of what UTC time is N days after X moment is location-dependent, because some days in some locations are 23 hours long and some are 25 hours long. And that is the question that a recurring appointment has to answer.
Off topic: ESR. it would be interesting to hear your take on gamergate.
>Off topic: ESR. it would be interesting to hear your take on gamergate.
I’m staying out of that one. What little I’ve learned of it put me in mind of Henry Kissinger’s comment about the Iran/Iraq war: “It’s a shame both sides can’t lose.”
I guess I don’t get why you’d use quad float seconds. As long as you’re using a whole 128 bits, why not, say, use signed integral zeptoseconds from the epoch? That keeps you in integer math (and thus avoiding any float compatibility issues on odd hardware, or sub-second arithmetic rounding problems), a range of 5,391,560,000 years on either side of the epoch, and twelve orders of magnitude finer discrimination than nanoseconds.
@Random832 “The fact that the clocks just changed is fresh in my mind for at least a week after each shift. Do the people at the Middle East sites not know that their time might change on short notice?”
The problem arises mainly where the venue is the one which is changing due to DST but no-one tells the other sites or overseas delegates. And yes the shift can be quite short notice. Some changes only get posted on TZ within a few days of the event, which is too short notice to get it through the current distribution system. Hence the need for tzdist …
Main fix is to avoid meetings in the problem periods, but that can leave quite a gap is the available schedule periods.
@Cathy “Long ago, when writing a program to display orbital trajactories, I made everything revolve around Julian days”
That is the one Cathy … with a simplification to get back to a single 64 bit timestamp of using a millisecond count for time.
@Random382 “The question of what UTC time is N days after X moment is location-dependent, because some days in some locations are 23 hours long and some are 25 hours long. And that is the question that a recurring appointment has to answer.”
The only way the display routine can display a 23 or 25 hour day is if the underlying base is UTC. It’s the only way to handle the two identical hours. BUT there are three elements involved here … Date, Time and Timestamp. A recurring event only has a Time but in association with a location. Only when it is associated with a Date does it turn into a timestamp and the UTC base becomes relevant. Which is why seconds are simply wrong for dates!
All historic material has a fixed timestamp, but events yet scheduled to happen may require adjustment if there is a local change of offset for whatever reason. If there is a change to the UTC value stored then it flags that there may be knock on effects to the change which may be missed if ONLY a local time is being recorded.
@esr lol. I understand. It would have been interesting to compare what is happening in the SF world with the gaming world.
@random
Because user expectations (which theoretical best practices don’t actually trump for consumer software) are that when they put 7:00 then it means 7:00 – i.e. the time that the clock on their wall hits 7:00 on that day, no matter WHAT the state or federal government has them do to that clock between now and the day of the appointment.
I’ll have to read the iCal rfc but I’d store alarms datetimes as local and alarm them when the UTC to local conversion said it was time to go off. That way it always works as expected as an alarm clock.
If the user wanted it to go off at an absolute time then I would make them input it as UTC.
A recurring event only has a Time but in association with a location.
No, there should be no location associated. If I set my alarm to go off at 7am every day it should always go off at 7am even if I set it in DC on Monday and am in Dubai on Friday.
Or even if I set it for 7 am on Friday. I mean Friday where I am then, not where I am now.
“user expectations (which theoretical best practices don’t actually trump for consumer software) are that when they put 7:00 then it means 7:00 – i.e. the time that the clock on their wall hits 7:00 on that day, no matter WHAT the state or federal government has them do to that clock between now and the day of the appointment.”
That’s fine for user input and output, but not for data storage and communication. Fundamentally, a given time should have one unique representation at the base level. If you allow timezones and such, there are many ways of representing times that are actually simultaneous.
Multiple ways to display a time/date, or to accept input — OK. Multiple ways to represent times as internal data that are actually the same — very bad. I should be able to do time1 == time2, or if not both ints, then (time1 – time2) < EPSILON, and know whether they match. This also simplifies comparison of two time/dates; (time1 – time2) is meaningful with little or no compound arithmetic and opportunity for error.
All you have to do is get the conversion routines between user input/display and the internal representation right once, in one library, and everything everywhere else becomes simple.
@nht “No, there should be no location associated. If I set my alarm to go off at 7am every day it should always go off at 7am even if I set it in DC on Monday and am in Dubai on Friday.
Or even if I set it for 7 am on Friday. I mean Friday where I am then, not where I am now.”
In that case the location is ‘mobile phone’ and THAT will use the information you set it to. If you only have a 24 hour clock all of this is relevant, but if you set your mobile only to show your home time then that is what you get. None of that changes the fact that you have asked for a time and the device needs to work out how to interpret that time based on your location … and you have to decide just what time you are working to as you change location …
This whole project and design discussion just gives me a warm fuzzy feeling inside! I wish I had some significant content to add, but lacking that, I’ll just toss out that it would be nice to add a C++ library to wrap this and give it an even easier to use interface (i.e. state that accepts configuration parameters and provides stream operators for printing&parsing, etc)
@ No, there should be no location associated. If I set my alarm to go off at 7am every day it should always go off at 7am even if I set it in DC on Monday and am in Dubai on Friday.
A wake-up alarm and an appointment are not the same thing, and you’ve just highlighted the main difference in how they need to be handled.
> Multiple ways to display a time/date, or to accept input — OK. Multiple ways to represent times as internal data that are actually the same — very bad.
But they’re not the same, because either one of them needs to change if the location it is defined in changes timezone.
Basically, a time scheduled in the future in a timezone is NOT actually a UTC time. The UTC time associated with it is a prediction of what UTC moment that timezone is likely to have that time, but it’s not a 100% reliable prediction. As it gets closer, it becomes less likely to have to be changed, but you need to retain that information at least until the time is in the past.
@ESR
Im no hacker, not even remotely. Would like to learn some C though. Ive been trying to digest all of this as much as i can. What is the goal here? Is it to write a non-standard C library for date and time? What ESR suggests as an alternative appears to not conform to http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf , not sure that is a relevant source since i dont have a copy of ISO/IEC 9899:2011. Reading at page 338 about ‘time.h’ it is stated that “The
tm structure shall contain at least the following members,
in any order. …” followed by the usual suspects all being int type and none being a floating point.
Should the goal of all this be to build a better library, even if it means introducing floats and not really caring about what the standard has to say about it, i would like to know before i continue to put effort into reading more docs.
Best regards.
> What is the goal here? Is it to write a non-standard C library for date and time?
Yes. The standard API is broken in ways that cannot be repaired in a upward-compatible manner.
It occurs to me that for certain purposes a time/date should be flaggable as “local/absolute”. If I have a weekly dinner reservation, I want it to be at 7PM Wednesday local time, and I don’t care what the absolute time is. Conversely, a filestamp should be in absolute time.
This would be a directional flag. For a “local” time/date, the nominal value is constant, and the absolute value is derived from it under whatever rules apply at that moment. For an “absolute” time/date, the absolute value is constant, and the nominal value is derived from it.
There needs to be a library which contains all the rules for conversions at various places and times through history.
Is this a useful idea?
>Is this a useful idea?
The existing, rather broken API already distinguishes between local and absolute (UTC) time.
It just occured to me that the reference in your RFC http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/time.h.html describes time.h in very much the same way.
@Rich Rostrom “There needs to be a library which contains all the rules for conversions at various places and times through history.”
Start with a download of the geonames database and then add all the missing historical movements that have not even made it into TZ yet …
This needs to be service, but even the tzdist service will only scratch the surface when it finally appears.
As a non-hacker type I can’t provide any technical help. But I can provide an “amusing” story.
Back in February, I made an appointment for August. I live north of Chicago, the other person lives in the Caribbean, and the appointment was for a meeting in Florida. The unstated assumption between the two of us was that the time of the appointment was to be in then-local-Florida time: Eastern Daylight Time. Unfortunately, the other person’s smartphone (running android IIRC, but I couldn’t swear to that) did not share this assumption and showed the meeting to be at 8am rather than 10am when it arrived in Florida. And the other person believed our meeting was two hours earlier, because that’s what the smartphone said on its scheduling app. Fortunately, we encountered each other the evening before, noticed the problem, and were able to straighten it out by reference to the original exchange of e-mails.
At the time, we put it down to the smartphone app being “helpful” but now I wonder if it was just confused.
That’s certainly true if you measure speed in instructions per second or other measures of computing power. But a narrower claim, that Moore’s law peters out on processor-clock frequency, is no longer a statement about the future. It’s a statement about history. Clock frequencies have been stagnant for over a decade now. They’ve never been stuck that long before 2003, and there are good physical reasons to bet they’ll stay stuck in the future.
Power dissipation, the reason they got stuck in the first place, rises linearly with clock speed, quadratically with the voltage. (And there are physical limits to decreasing the voltage because you need it for moving the transistors’ electrons between their “on” and “off” states fast enough.) These problems with power dissipation are unlikely to go away because they’re mostly constraints in the physics rather than the engineering.
But even if they did go away, further physical limits on clock speed would be lurking just around the corner. Remember that a clock rate of 10 Gigahertz corresponds to a cycle time of 100 picoseconds. Light travels at most 3 centimeters in 100 picoseconds. That’s the size of a typical chip (which, in turn, has been more or less constant ever since at least the Intel 4004 fourty years ago, for reasons to do with minimizing the rate of manufacturing defects). So the first thing that will happen is that the processor chip as a whole becomes an antenna for its own dissipated electromagnetic radiation, unpredictably screwing around with its own transistors’ “on” and “off” states.
But that’s not all. Somewhere between 10 and 100 GHz, you would run into problems keeping the processor in synch with itself, as clock signals sent during one cycle reach their destination during another, with some randomish phase shift. Again, these problems are mostly physical rather then technical. I wouldn’t expect them to have a technical fix.
To be sure, all of the above is just my opinion. Although it’s informed by my general education as a physicist, I was never a specialist in this corner of applied physics. You want to double-check with a material physicist or an electrical engineer who is. But with this disclaimer out of the way, I believe it’s safe to bet that we won’t be seeing terahertz clock rates for as long as people code in C.
I think that what is really broken is civil time. If we could do away with that, we could stick to the mostly solved problem of globally distributed ticks. (Though doing that securely is hard, particularly when secure includes distributed.)
Any software that attempts to implement civil time is going to be broken, in much the same way that a phlogiston-based CFD simulation of a jet engine would be broken, and for many of the same reasons. Though I agree that it would sure be nice if it were somewhat less broken than it is now.
Whether a time refers to the civil time or to the monotonic ticker can often be implied, but certainly must allow the user to override it. You may want your alarm clock to go off at 7:00 AM, civil time, every day, regardless of which civil time you find yourself in when the moment comes, but you don’t necessarily want medication dose intervals growing and shrinking. (I’m one of those guys that interprets “Take this pill 3 times a day” as “Take this pill every 8 hours”, and not as “Take these at random intervals that average out to about 3 in 24 hours”.)
Also agreed on preferring integer math over floating point, for reasons stated by many others above. I personally find myself spending more time on tiny systems as I get older, and also running on battery power. If we are finally going to have a good time system, I’d rather it be as widely available as possible. Moore’s law says that it won’t be an issue forever, but I wouldn’t be surprised if 20 years from now FPUs are still not universal in embedded systems.
Oh, and I’m not entirely convinced that there is much need for a data structure that handles both geological timespans (or worse) and picoseconds (or worse). Who needs to set an alarm that will go off at 14:37:59.912 459 912 555 on July 4th, 25025 AD? And who would believe even a millisecond timestamp from 1492? Simulations of the big bang don’t need to get the date right either, whether 4004 BC or -x billion years.
(I understand the arguments in favor of combining them into a single unified time structure, I’m just not convinced by them.)
Spelling error
tzset(3) is requred by POSIX to be thread-safe, but access to the globals it sets is intrinsically thread-unsafe.
requred
Wow, you’ve actually made – you’ve managed to come up with time API even uglier than old unix. That’s impressive, never thought it’s actually possible outside of php world.
Really – sober up and get rid of this nonsense with floating seconds: it sounds ridiculous when you’re trying to cater for embedded systems with limited resources and hope for arbitrary precision fpu at the same time.
Just do it the sane way – one more int for sub-second and one more byte for granularity indication:
0001 is for nano
0010 is for atto
1000 is for femto and so on… here, got your “future-proof” api without floating-point rubbish.
I find timezone issues fascinating. When dealing with local time conversion to GMT/UTC there are two edge cases that have to be dealt with. First is there are local times that don’t exist. Not to hard – just through an error. The second is more problematic – there are local times that are ambiguous – what’s the answer then?
And I have to mention my favorite Windows quirk. Under Windows the conversion to local time is done using the current timezone offset and not the offset at the time under consideration. The result is that when you are looking at timestamps in explorer or event viewer half the times are always incorrect – those file times or events that occurred under the offset not currently in force are flat out wrong.
I had to look up the SPOT rule as written by Brian Kernighan:
“… Also, the information appears only once, a ‘single point of truth’ from which other code is generated so there is only one place to keep information up to date. If instead there are multiple places, it is inevitable that they will get out of sync sometime. …”
So here we are now. We are likely to sit on data with ambiguous timestamps and we cant correct them since we dont know how they occurred. March 1st of year 2000 was a wednesday, how do we handle a timestamp that claims it was a tuesday? Back when time.h was standardized there was probably lots of data too with timestamps that didnt make sense. Should we really think “day of week” is meant to carry a value that is always related to the other members of the struct i a predictable way, or is it just partially related? We cant just modify a timestamp of data that comes from an external source, can we?
Struct tm fails to be an unambiguous representation of time. But can we really use that as an excuse for introducing floats. “I think this is tolerable in calendar dates, where subsecond arithmetic is unusual thing to do to them.” That statement implies a restriction of the use cases. APIs are forever and someone will eventually want to do some subsecond arithmetics.
Best regards.
> So here we are now. We are likely to sit on data with ambiguous timestamps and we cant correct them since we dont know how they occurred. March 1st of year 2000 was a wednesday, how do we handle a timestamp that claims it was a tuesday?
As far as I know, on Unix (and the history of C and Unix are deeply intertwined), struct tm was never intended for data storage usage. Why store in eighteen bytes what you can store in four? The most likely case for ambiguous timestamps in old data is going to be in local times stored without timezone information at all, not in storing redundant (and wrong) day-of-week and day-of-year fields. Too little information rather than too much.
Tyvm Random832. I realise i goofed there. Right now im reading up as much as i can on this. I dont know how to code. I dont know astronomy. I thought i knew the clock ;) , but the more you read the less you know. Im beginning to understand some of the problems here. Clean sheet redesign in order to remove the SPOT violation has to be the way forward. As for the question of integral floats and subsecond arithmetics… of course it has to be possible to safely do subsecond arithmetics. What response to prepare for the first complaint? Youre holding it wrong?
We have a lot of different things to do with time. What you’re doing should determine how you do it.
For logging events, file last-modified timestamps, etc., use UTC, period.
For future events (appointments), do not use any kind of struct; use ISO-8601 rules to format a string that unambiguously describes, say, that meeting I have scheduled that includes a company in Australia that’s doing some work for us (where CST/CDT/AEST/AEDT transitions make the time difference 15/16/17 hours, and we have to know whether the appointment follows our DST changes or theirs, which go in opposite directions). Putting the meeting time as “08:00 Australia/Sydney” makes that clear.
For those who think a TZ can be specified by a single “offset from GMT”, you are sadly mistaken. I will not move to a different timezone later this month when DST ends. My timezone is America/Chicago, not “CDT” (or worse, “CST” as many people insist on calling CDT) today and “CST” after the “fall back”.
Avoid the use of the float, because it adds a lot of overhead for little benefit. Sub-second precision can be handled by storing the sub-day time as the number of picoseconds elapsed since midnight UTC. 64 bits more than covers that, replacing the HH:MM:SS.ssssss with a single value. Alternatively, use 32 bits for the number of milliseconds elapsed since midnight and a separate 32 bits for the picoseconds, so that applications that don’t care about sub-millisecond precision can ignore it (especially embedded apps using 32-bit processors that would be forced to deal with double-precision arithmetic in software since the hardware doesn’t support it) and every calculation is upon integers, .
> especially embedded apps using 32-bit processors that would be forced to deal with double-precision arithmetic in software since the hardware doesn’t support it
Depends on what 32-bit processors. For example, the x86 architecture’s divide instruction technically divides edx:eax by the argument – in normal 32/32 usage edx has to be zeroed or sign-extended. Since the divisor is constant, there are also clever tricks you can do to avoid the divide instruction (though you have to be a lot more clever than on a 64-bit system where this can be done with a single multiplication.)
> Depends on what 32-bit processors.
Well, of course. That’s the whole point, really. A hardware-independent spec ought not to depend on any non-universal feature.
“Well, of course. That’s the whole point, really. A hardware-independent spec ought not to depend on any non-universal feature.”
But you don’t need a full-featured emulation, all you need to do is divide by a single constant, which can be done in three multiply instructions on a 32-bit system.
this is probably mostly orthogonal, but I’ve found that a lot of things about time programming are clarified by separating things into “point” and “interval” types and establishing rules for arithmetic among and between them. in the general case, a point is seconds/nanoseconds/whatever since the epoch, and an interval is a value in the same units representing the difference between two points. one thing that is immediately apparent is that only certain arithmetic operations make sense (at least in the case of ordinary time programming):
interval+interval -> interval
interval-interval -> interval
interval*interval -> error
interval/interval -> scalar
interval+scalar -> error
interval-scalar -> error
interval*scalar -> interval
interval/scalar -> interval
point+interval -> point
point-interval -> point
point*interval -> error
point/interval -> error
point+point -> error
point-point -> interval
point*point -> error
point/point -> error
point anything scalar -> error
this separation becomes particularly important when dealing with point types of other precisions, such as time-of-day-without-date — keeping track of whether “02:00” means “two AM” or “two hours” is a lot easier to do with some help from the language, and is one of the places Excel really falls down
for a vaguely similar idea, see ptrdiff_t
i read a blog post a year or two ago talking about doing directional math in a game in similar terms, separating “angle you’re looking” and “angle you’re turning through” into two different types. iirc, it drew some connections to the math of affine spaces, but that’s way beyond me
@Aaron Davies Why shouldn’t “interval” just be a scalar with a defined unit? Your model is too simplistic for calendar math (i.e. adding a number of months to a date)