[LEAPSECS] Leap seconds ain't broken, but most implementations are broken

Fri Jan 6 23:09:19 EST 2017

On 2017-01-06 11:33 AM, Martin Burnicki wrote:
> Brooks Harris wrote:
>> On 2017-01-05 05:56 AM, Tony Finch wrote:
>>> Martin Burnicki<martin.burnicki at burnicki.net>  wrote:
>>>> Please note that NTP servers not necessarily need to be providers for
>>>> leap second files. There are some well known sites which provide this
>>>> file, and the NTP software package from ntp.org comes with a script
>>>> which can be used to update the file automatically.
>>> I was thinking more that an NTP client or server would use its
>>> leapseconds
>>> file for validating LI bits from servers and for determining when they
>>> should leap.
>>>
>>> My thinking is that routine software patching and security updates happen
>>> often enough that maybe NTP can get leap second more reliably out-of-band
>>> instead of using in-band leap indicators from upstream servers.
>>> Lower-stratum devices could use their own leap second information to
>>> correct for operational or implementation errors upstream.
>>>
>>>> The potential approach with tzdist or special DNS allowed for a
>>>> distributed system, where the special DNS can only provide leap second
>>>> warning and the current TAI offset, while tzdist also provides the leap
>>>> second history, and a way to update time zone rules, so it could be
>>>> generally used to keep also conversion to local time correct.
>>> Oh, I forgot about the DNS publication scheme. That would also help a lot
>>> if it were implemented. And maybe better than relying on sufficiently
>>> frequent software updates.
>> So, thing is, since 1972, no common and official way to automatically
>> obtain the Leap Seconds information has been adopted. Its an obvious
>> missing link, missing for 4 decades! I'd find this just incredible
>> except I've now come accept this sort of frustration where timekeeping
>> is concerned. This has been discussed many times on LEAPSECS.
> Hm, I think there was no such requirement when NTP was introduced. Folks
> were happy that they could at all forward leap second announcements,
> which was sufficient then.
>
> The leap second file was introduced by Judah Levine in 2000, AFAIK, see:
> http://tycho.usno.navy.mil/ptti/2000papers/paper34.pdf
>
> This was probably because it became more important to know the history
> of leap seconds, and thus the TAI offset.
Right, that's where I say its incredible nothing was done since 1972. 
The Leap Second history is fundamental, and it seems to me an obvious 
missing link. How could it have gone ignored all this time?

I guess it comes from a point of view of distributing "now". That was 
always the first objective, to simply drive a clock - from there, humans 
could read the calendar and clock and apply their knowledge to calculate 
appointments - that was good enough, and the means to do more 
automatically didn't really exist. But that ignores the modern needs of 
applications to calculate durations, a fundamental requirement.

I've always seen the need to be able to accurately represent the entire 
UTC timescale (since UTC1972), both as seconds-since-UTC1972 and as 
YMDhms(UTC). This provides accurate historical timestamps (back to 
UTC1972) and so, durations. It also provides prediction, or scheduling, 
out to the next announced Leap Second or expiration.

It also provides means of testing algorithms against known values in 
development. The "corner cases", like Leap Seconds and DST shifts are 
the tricky bits, and you only know the values for the past. You haven't 
solved the general problem if you can't represent the entire timescale. 
Its straight forward *if* you have the Leap Seconds table. Oh, and if 
everybody is treating it exactly the same way.

Helping develop a clear and official way to get the Leap Second table 
should be a long term objective, I think.

> Ntpd's "autokey" feature was
> (mis-)used to be able to pass the TAI offset down to clients.
>
> As I wrote in another email, when autokey is obsoleted by NTS then
> eventually another new NTP extension field is required to be able to do
> this via the NTP protocol.
>
> Anyway, it's quite a number of years.
>
>> Ideally there would be one, and only one, TAI-UTC table, in some very
>> well specified form, residing somewhere in cyberspace, administered and
>> maintained by official rules and regulations by the IERS. There then
>> could be many API's via many technologies to access the information.
> Agreed.
>
>> Today, there are many ways to get it, but they are all in different
>> forms, not always so well specified, and all require some human
>> intervention or oversight somewhere between the IERS announcements and
>> the distribution. None of them are particularly "fast", imposing too
>> much overhead on the receiver in some circumstances. And "many ways to
>> get it" does not inspire confidence that each implementation will get
>> everything the same.
>>
>> I like the DNS publication scheme because you could imagine that IANA
>> could take responsibility for maintaining it, especially if there were
>> an official way to keep it automatically updated from a hypothetical
>> official IERS source.
> Yes, but if you have a greater view of the system then this fixes only
> the leap table issue, which may be sufficient for many purposes.
Yes, important point. Fixing Leap Seconds is necessary but not 
sufficient for a complete and practical world-wide timekeeping system. 
Humans need *local time*.
> The tzdist protocol, however, has the advantage that it can also
> distribute updated time zone descriptions, so you can update everything
> from the kernel/TAI up to user spaces applications dealing with local
> time in a single service.
Yes, generally I agree.

Leap Seconds and local time parameters are both dynamic metadata sets 
required. However, they may be updated on different schedules; Leap 
Seconds by Earth and the IERS, local time parameters by anybody 
anywhere, anytime. I think the two need to be available separately, 
probably with different mechanisms for notifications, polling, etc.

> You could build up a tzdist server hierarchy similar to NTP stratums or
> DNS root and secondary servers to distribute the information, and the load.
Yes. One might wish for:
Some mechanism for notification - "an update is available for Tz 
Database", "an update is available for Leap Seconds"
Some API call to obtain changes - "please send recent changes to Tz 
Database", "please send recent changes for Leap Seconds"
Some API call to obtain complete data sets - "please all Tz Database 
data", "please Leap Seconds history"

There could be, should be, various ways to do this, via typical internet 
technologies and faster, lighter, binary protocols.

IETF RFC 7808, Time Zone Data Distribution Service is a step in the 
right direction, I think.

>
>> One could imagine, and it would be straight forward, to have NTP servers
>> provide the TAI-UTC table, announcements, and expiration via the same
>> IPC transport mechanisms used by NTP. Again, hopefully, updated from a
>> hypothetical official IERS source.
>>
>> I've had the thought that Block-chain would be a good way to do it. It
>> would have all the purported anonymity, security, and persistence
>> qualities of Block-chain. In such a scheme, only IERS could make updates
>> to it and everybody else could read it.
>>
>> It makes sense that the Leap Second Table be combined with time zones,
>> as it is in Tz Database, because you really need all the local time
>> information together with Leap Seconds to achieve comprehensive
>> timekeeping. It occurs to me Tz Database could also be maintained as
>> Block-chain.
>>
>> The typical methods used in NTP, GPS, and PTP of distributing only the
>> upcoming Leap Second announcement has always seemed fragile and
>> incomplete to me.
> I don't fully agree here. The question is what you need.
>
> tzdist is the latest protocol which can distribute the full leap second
> history, if you need it.
>
> PTP and GPS are older but provide at least the TAI offset and warning of
> an upcoming leap second.
>
> NTP by far the oldest protocol which basically only provides a leap
> second warning, but has optional (not so obvious, and soon obsolete)
> extensions which can distribute TAI offset to its clients.
Well, as you have pointed out, not all sources are reliable all the time 
- you have to pick and choose and decide who's most right. When things 
don't match, or a server serves something wrong, what to do?

If a client had access to reliable official information through other 
channels it could just ignore the Leap Second info in GPS, PTP, or NTP 
and process from its own data, including applying history and upcoming 
for durations and scheduling. That doesn't eliminate all possible 
errors, but it could help mitigate them.

>
>> The lack of reliable automatic information from IERS
>> means some human intervention must occur in the announcements, and the
>> information is incomplete, only communicating the immediate upcoming
>> current Leap Second. Many systems will need to go elsewhere to get the
>> full table and expiration, and this leads to possible mismatches between
>> information sources, as noted in this thread.
> Human inventions will always been required to update the leap second
> information. This can be a person at IERS, or some other folks.
Yes, but it would ideally be an authorized person or persons at IERS 
operating under a known set of rules with cross-checks, and 
verifications, that is, a single source of reliable official 
information. Right now its "some other folks" and while they may be 
experts and well intentioned there's too many chances for mistake or 
mismatch somewhere.
> The IERS just started to provide an own leap second file a few years
> ago, after I had contacted them and asked them to do so since they are
> the authoritative source for leap second decisions. Before that they
> weren't even aware how important this stuff is.
>
>> Meantime, all this is happening in an environment where the underlying
>> specifications are difficult to understand, in some respects possibly
>> controversial, and the de facto standards of Tz Database are unofficial
>> and don't match Microsoft's world view.
> Ouch. I don't know why TZ should adopt to Microsoft's world view.

I didn't mean to suggest anybody should adopt Microsoft's world view. 
Far from it - what, do you suppose, is Microsoft's world view? Anybody 
want to guess?

I meant everybody needs to do the same thing(s) if we are to have a 
reliable system. Today Windows local time parameter list is more or less 
a subset of Tz Database, and the two are in decidedly different data form.

I'm not sure how anything or anybody can convince Microsoft of anything, 
let alone changing their local time parameters, so there's that reality 
to consider. The only thing that might move them is market force, like 
the EU demanding some specified behavior for banking transactions, or 
the US government saying "it will work like this or we won't buy it".

But note, see:

Sources for time zone and daylight saving time data
https://www.iana.org/time-zones/repository/tz-link.html

Other tz-based time zone software
Microsoft Windows 8.1 and later has tz data and CLDR data (mentioned 
below) used by Windows Runtime classes such as DateTimeFormatter. 
Exploring Windows Time Zones with System.TimeZoneInfo describes the 
older, proprietary method of Microsoft Windows 2000 and later, which 
stores time zone data in the Windows Registry. The Zone → Tzid table or 
XML file of the CLDR data maps proprietary zone IDs to tz names.

There is hope. But no universally adopted system yet. And its a mess of 
code.

>   AFAIK
> TZ recently had its 30th anniversay, it was invented in 1986.
>
> The first Windows versions were also available at that time, but just as
> a frontend to DOS which knew nothing about time zones and UTC.
>
> Windows NT was the first Windows version where the system time ran in
> UTC, and AFAIK NT was released in 1993, when TZ already existed for a
> couple of years.
I think Tz Database has become the only source that can be considered 
because its in the public domain and its well developed, well known, and 
widely used. Its essentially a de facto standard. If its workings were 
clearly defined and specified in a recognized due-process standard it 
might find its way into all implementations. IANA's involvement lends 
weight.

-Brooks

>
> Martin
>
> _______________________________________________
> LEAPSECS mailing list
> LEAPSECS at leapsecond.com
> https://pairlist6.pair.net/mailman/listinfo/leapsecs
>
>