SSAC Lightning Talks (1 of 2)

MTS-AV: Thank you, Scott. Hello, and welcome to the first of two SSAC Lightning Talk sessions. My name is Kathy, and I'm the participation manager for this session. Please be advised that this session is being recorded and is governed by the ICANN Community Participant Code of Conduct, the ICANN Expected Standards of Behavior, and the ICANN Community Anti-Harassment Policy.

MTS-AV: Regarding participation today, this session is designed for internal discussion among SSAC members on current security, stability, and resiliency topics. Observers are welcome to watch, but this session is not open for observer participation unless the moderator chooses to do so.

MTS-AV: For SSAC members, all members have been promoted to panelist status in Zoom. To join the speaking queue, use the raise hand feature, even if you're physically present in the room. With that, I'm happy to hand the floor over to Barry Lieba.

MTS-AV: Okay, this is Barry Liebel with SSAC, and those of you who were not in the room unfortunately missed the sparkling speech I made before the recording was started. Sorry about that.

MTS-AV: But we have the lightning talks, and we'll start with Raffaella, and then pass it on to Peter. So, Raffaella, go ahead.

MTS-AV: Thank you so much. Let me share the screen.

MTS-AV: Hey.

MTS-AV: Then, for some reason or another, PowerPoint doesn't want to present anymore.

MTS-AV: Come on.

MTS-AV: Okay, give me a second.

MTS-AV: That's always

MTS-AV: Okay.

MTS-AV: Okay, welcome, everyone.

MTS-AV: So this first lightning talk will be about a paper that a student of mine, master student of mine, wrote.

MTS-AV: About the reusual of domain validation token at scale from certificate authority, and this paper has been

MTS-AV: Just accepted that WTMC 2026. So I'm very happy. And it's a workshop co-located with CCS. And I'm going to give you a bit of a gist of what this paper is about and the result that we find out.

MTS-AV: So all of you have probably read the news that some hackers obtained counterfeit TLS certificate for Google and other large services by hijacking three top level domains through CCTLD.

MTS-AV: Now, the question that I would like to bring you today is that imagine that if the attacker can keep the control and can keep issuing the certificate even after the Ajax event happened.

MTS-AV: Is this possible in certificate world, and how?

MTS-AV: And…

MTS-AV: The most common model that we have in our head when we think of, okay, someone is doing a domain verification check for a domain name, when requesting a certificate, is this one. This is from a Let's Encrypt paper, describing basically the architecture of Let's Encrypt of a couple of years ago.

MTS-AV: to use Znik Security, where basically a domain owner using the ACME protocol request

MTS-AV: to, the registration authority of the certificate authority, certificate, the certificate authority checks with the validation authority, that is basically the server that they use.

MTS-AV: to validate if the domain owner, basically has the right control over the DNS server, in the case they use a DNS validation, or the web server, in case they use… and the DNS server, of course, in case they use HTTP validation.

MTS-AV: The reality, of course, is not that simple, because we live in a nasty world where people can do nasty stuff. So we have BGP hijacking that can happen, and you don't want to have a certificate authority fall in the trap of BGP hijacked.

MTS-AV: So, what they do, really, is they have multiple validation servers all across the world, because they want to measure the characteristic of this domain web server and this domain DNS server, and check that the records are consistent all across the world to see that, indeed, the result is always the same, and you don't have, like, hijacked path in the middle.

MTS-AV: But there is a problem, that if you start to do this kind of validation in a repeated way, you may overload the domain infrastructure or the CA validation server with catastrophic consequences. I mean, it starts to DDoS, like, infrastructure of.

MTS-AV: Certificate authority by just requesting millions and millions of certificate for a domain.

MTS-AV: Two common mitigations to this. The first one is rate limiting. It's often done by doing a PSL-based suffix, so you are allowed to request up to a certain amount of certificate per minute per TLD.

MTS-AV: Sorry. So per registered domain based on the public suffix list.

MTS-AV: And the second one is doing domain validation caching, the domain control validation caching. That basically means that you cache the information that you retained out of the fact that the domain owner was owning that domain.

MTS-AV: domain.

MTS-AV: In this presentation, we focus on caching and the security risk introduced by reusing these validation results.

MTS-AV: How the caching works, basically, the same picture as before, is that when the second time the domain owner requests the certificate for his domain.

MTS-AV: Instead of doing again all the validation, you have like a phase that it's the DCV caching. Everything is that was previously validated gets stored somewhere. And you can reuse that information for how long you can reuse that information.

MTS-AV: Well, you can reuse that information for, in the past, up to 398 days. Nowadays, it's going down. Now it's… the current is 200 days, because this has been reduced by the CA browser forum, and there is a plan for a gradual reduction of this validation, caching

MTS-AV: Up to 10 days in 2029. The important part is that 96.4% of the domain, certificates out there are domain-validated certificates, so these apply to most of the, certificates out there in the wild.

MTS-AV: What's the problem of doing validation caching? The problem is that domains out there expires.

MTS-AV: They are deleted, they are suspended for phishing or abuse.

MTS-AV: And when they expire, of course, their registration and delegation disappear. And in case you use, basically, a cached information for verifying the ownership of the domain.

MTS-AV: they may not reflect anymore the current controller of the domain name. And if you combine that with, for example, a BGP AJAC, you could get an attacker to obtain a certificate for a domain name that it really doesn't own anymore, and end up with intercepting TLS traffic.

MTS-AV: It's a fairly complicated attack, it's not… it's hard to do in the… in the… in the wild, but it's a potential risk.

MTS-AV: So how do we find how often this happens?

MTS-AV: And the way we find often this happens is that we want to find certificate.

MTS-AV: that have been logged in certificate transparency log.

MTS-AV: With no contemporaneous DNS evidence or registration evidence.

MTS-AV: So, how many of these… how many of these cases we find, how many of these cases have something that is compatible with the reuse of a possible cached token, because there was a certificate in the past that was cached.

MTS-AV: And, how these cases varies across CAs. So, are there CAs that reuse more validation token than others?

MTS-AV: So, to do this, we used a couple of data sources. The first thing, we used Zonestream. Zonestream is a project we run on the OpenIntel, where we basically measure in real time… match in real time all the domain names we learned from CityLog per certificate, that are basically the requests for certificates that are sent to the CityLog.

MTS-AV: against the daily gTLD leads on files, and we do that for detecting newly registered domain names, so we do that with the aim of basically creating a stream of newly registered domain names.

MTS-AV: And to confirm that if they are newly registered, we checked with RTOP, and we monitor also the DNS query for the next 48 hours for this newly registered domain name.

MTS-AV: We also use in these

MTS-AV: pipeline DNS Coffee, that's basically an historical overview of the zone files published in CZDS, for checking, for example, if the domain name was active in the 3 days prior or 3 days after the certificate issuance.

MTS-AV: Because certificate log has this problem of maximum merge delay, where the certificate may be requested, but then you may have, like, up to 48 hours for the certificate to get… the certificate request to get inserted into the log. So you need to account for a bit of caveats of timing.

MTS-AV: To account for these caveats of timing, we checked that the first measurement we are able to issue, the NS measurement we are able to issue is within 5 minutes of the issuance certificate request, from the,

MTS-AV: certificate owner, so we are sure that at least when we measure something, we are measuring that 5 minutes after the person clicked on, I want a new certificate under the ACME protocol.

MTS-AV: So, we looked at the entire 2025, and out of 14 million, potentially new domain, we found that

MTS-AV: around 1,800,000 were responding with 404 RTAP. That means that they were not existing, so…

MTS-AV: They were not new domain, they were just not existing domains.

MTS-AV: 284,

MTS-AV: Thousands of these were consistently in NX domain for the next 48 hours. We did measurement every 10 minutes for the next 48 hours, and they always respond with NX domain, so the domain was not existing.

MTS-AV: We filter out the one that were within 5 minutes of… from first measurement from the certificate issuance that brought us to 169,000

MTS-AV: Domains, and we also looked if there were

MTS-AV: appearing, like, in the range of plus-minus 3 days from the Zoom file, and we ended up with a number of 121.

MTS-AV: a thousand domains.

MTS-AV: So, what are these 121,000 domains belonging to 142 certificates? These are potentially certificates that have reused a previously cached validation token out there.

MTS-AV: So, 121 unique domains that has certificate without contemporaneous DNS evidence. And this is, like, a lower bound. This is just one where we can see that there may have been, like, a reusal of a cached validation token.

MTS-AV: Just because there was no DNS evidence. Reusal of validation tokens may also happen in case the domain still exists and still is in control of the user, but we don't have any visibility into that because the certificate

MTS-AV: Authorities do not publish if the certificate was issued because the domain was checked, or was checked by using a cache token.

MTS-AV: So, out of these, 121,000 domains, 111 were, did not reappear in the zone file, and just 10,000 were subsequently re-registered. We don't know if they were re-registered by different people, because, of course, RDAP doesn't give us any information about the registrant.

MTS-AV: But these are… could potentially be re-registered by different people, raising basically this problem of the ownership transfer.

MTS-AV: And…

MTS-AV: you may wonder, okay, can we explain all these cases with the TCP caching? The interesting part is that 91… probably yes, because 91% of these domains had the predecessor certificate from the same issuer within the 398-day reuse window, so there was a certificate in the past

MTS-AV: that had an active domain, an active DNS record that was issued, and then when the domain basically expired, they probably reused that validation token that they collected at the first time back in, when the domain was active.

MTS-AV: We are not 100% sure this is just an empirical observation, because that's what we can observe from big data, but this is probably the most likely explanation.

MTS-AV: If we look at which are the CAA that perform this kind of caching, we try to do that by comparing to their

MTS-AV: actual volume of certificate emission, not just by looking at the number of the domain in general. And we did that because we want to understand more if

MTS-AV: you know, if this caching is just dependent on the number of certificates you issue in the wild, or if it's really something that you try to do more as certificate authority, what we found interesting is that DigiCert and GoDaddy are apparently reusing more and more this caching mechanism compared to the other.

MTS-AV: So why this problem exists at all is that this problem exists at all because basically certificate authority reused domain validation evidence long after the original check.

MTS-AV: And,

MTS-AV: We don't have very historical visibility into DNS, because DNS is queried on demand, and we don't have visibility in temporary zone changes or short-lived

MTS-AV: zone changes inside the DNS.

MTS-AV: And, some CAA may operate, and this is just a speculation on

MTS-AV: private evidence of domain registration. If you are a registrar, you may have information regarding the fact that the domain is still in the control of the previous owner, and you may use that information to issue a certificate if you are both a certificate authority and a registrar, but that's, again, just a speculation.

MTS-AV: And the main question is, shall CA trust

MTS-AV: Get us a toll in, issuing certificates, because we have been… set as the

MTS-AV: The trust anchor for certificate authority, but we never ask for that.

MTS-AV: Luckily, on the CA side, something is moving. In March 2026, there was a ballot, the ballot SC081, where the reuse period was reduced to 200 days.

MTS-AV: And there are further reduce, reduction plan to bring, basically, the validation reuse closer to certificate lifetime that are also being shortened. Now we're talking of certificate of 45 days, at maximum, and then in the future, probably will be 9 or 10 days, and instead the target of

MTS-AV: These domain validation caching will be 10 days by March 2029.

MTS-AV: And with this, I conclude my presentation, and I'm open for questions, or we handle questions after.

MTS-AV: We'll take questions now.

MTS-AV: So Any questions for Rafaela?

MTS-AV: John.

MTS-AV: Yeah, thanks. This is fairly interesting. And I guess the question I was going to ask — Tell everybody recording who you are. Oh, I'm John Lemon, NSSAC member.

MTS-AV: I guess the question I was going to ask, I think, is mostly answered by your last slide, but my understanding is that if you can get the certificate lifetimes down to under a month, which is typically how long the DNS goes dead when a thing expires, you'd think this problem would go away.

MTS-AV: Is that realistic, or do you think there's still going to be CAs that leave stuff around for longer because they want to charge money or something?

MTS-AV: I think this is really sad. I think the main problem here was the initial

MTS-AV: requirement that was set of 398 days. That was completely insane.

MTS-AV: The… setting that to a timeline where you have, like, at least it aligns with the period that usually a registry and registrar keep domain out of the, for example, the possibility of resettling the domain will be enough to avoid this kind of situation.

MTS-AV: We have a question online from Jeff Huston.

Geoff Huston: Geoff Huston, hi, thanks for this. Look, we've come down from 398 days, progressively down to 10.

Geoff Huston: But it may not stop there. Where does it stop? One hour?

Geoff Huston: 10 seconds.

Geoff Huston: What's the logical point?

Geoff Huston: where you can cache material and be assured that that cached material is still valid. Because it strikes me that

Geoff Huston: This really is something that has no logical endpoint, and maybe we're just doing the wrong thing.

MTS-AV: So, to answer to that, I think one of the reasons why they performed… I tried to dig up why this caching even existed, and one of the reasons why

MTS-AV: One likely explanation I found is that really to prevent, Abuse of them…

MTS-AV: certificate infrastructure to do DDoS attack.

MTS-AV: That probably will…

MTS-AV: rate limiting will suffice, but… and even, like, a smaller period will suffice. I really don't know why they started from this very high number.

Geoff Huston: But my question is more, Where does it stop? Because 10 days seems totally artificial as a number.

Geoff Huston: What's wrong with one day? What's wrong with, you know, an hour?

MTS-AV: I think 10 days may be, like, a good compromise between the fact that the main name doesn't get immediately put out for reselling after 10 days, if it gets taken down or if it expires.

MTS-AV: I mean, it's an arbitrary, but I agree with you, it's an arbitrary number, it's an arbitrary decision that…

MTS-AV: they are taking, and I don't have, like, a convincing argument why it should be, like, 10 days, or 1 hour, or… probably it should be, like, the TTL of the… of the…

MTS-AV: The record they are caching, but the… the truth is that they may be abused.

MTS-AV: Times is right.

Geoff Huston: Probably a deeper conversation, but yeah, thank you.

MTS-AV: Sorry, Geoff, can you repeat?

Geoff Huston: I said it's probably a deeper conversation that also touches upon.

Geoff Huston: default certificate lifetimes, and the whole issue of whether a certificate infrastructure is a valid commentary on the current and short-term future of a domain name.

Geoff Huston: or whether the DNS has outlived and just gone past the utility of certificates at all.

Geoff Huston: But as I said, I think that's a deeper conversation than where we are here. It just strikes me as…

Geoff Huston: somewhat bizarre that the response to a problem is just to shorten the lifetime.

Geoff Huston: But when you do that, there's no endpoint. You can keep on shortening it until you get down to no lifetime at all, in which case you wonder why you're doing it.

MTS-AV: Lauren.

MTS-AV: Can you go back to the slide that listed the different?

MTS-AV: times.

MTS-AV: Back another one.

MTS-AV: Board 1.

MTS-AV: No, the one that had the table, the different CAs. Oh, this one. Yeah. So… For those…

MTS-AV: Do you have anything that shows how long the validation was reused for?

MTS-AV: Unfortunately, no. And the reason is we were unable to do this at scale because certificate logs are massive and are very hard to process. We really wanted to go back and.

MTS-AV: Look at when it was the previous certificate issued, and basically what was the certificate issued then.

MTS-AV: It was…

MTS-AV: paying to do this at scale. We really want to get that number. Probably that'd be extremely interesting.

MTS-AV: But the…

MTS-AV: High. No, I mean, this happens at least at the level where it's not a certificate was issued like the day before and the day after you see like the validation caching reuse is a there is at least like a month or something. Okay. Because the Google CA will do a maximum of eight days. Okay.

MTS-AV: And it seems like the numbers.

MTS-AV: If you have a look at what people who do 390 days and 8 days, you'd see a much wider disparity.

MTS-AV: I think if you do only each 8 days, you see here a lower number, because we do that 3 days filtering, so you already cut off some of the certificates we do cache.

MTS-AV: That's probably why Google then shows up with lower amount per number of certificates issued compared to the others.

MTS-AV: That may be a good, likely explanation.

MTS-AV: Do you have these officially somewhere?

MTS-AV: Yep.

MTS-AV: I can try and find the actual link, but that's under the Google Cloud Security blog. Okay.

MTS-AV: But I'm also just kind of wondering about the methodology, because it feels like some of it is not… anyway, we can chat more later.

MTS-AV: I think we have, time for a question or two from a non-SSAC, Questioner, if we have any?

MTS-AV: Andrew?

MTS-AV: We got one online.

MTS-AV: Andrew, go ahead.

MTS-AV: Hi, Andrew Campting. Thanks for the presentation, Raphael. Very interesting.

MTS-AV: Should we really be looking more at certificate revocation? Are we trying to solve the problem using the wrong lever?

MTS-AV: The main issue of certificate revocation is consuming certificate revocation.

MTS-AV: This, how do you push certificate revocation to the edge?

MTS-AV: That has been an unsolved problem for sure. I mean.

MTS-AV: In an ideal world, yes. In an ideal world, you don't want to have certificates dangling for domain names that are not under the control of the same person anymore, or are expired. Because that certificate would have been established on a chain of trust that is not there anymore.

MTS-AV: In an ideal world, yes.

MTS-AV: The practical world that may be challenging.

MTS-AV: Who's next?

MTS-AV: I have a comment on that. So I think,

MTS-AV: He said the practical problem with certificate revocation is,

MTS-AV: How to push the revocation information to the client?

MTS-AV: And that is essentially the same question as how short you make the certificate lifetime. It's, of course, a different data structure, it's a list for all domains, sort of, that are

MTS-AV: That have their certificates revoked, and the certificate is persitive.

MTS-AV: It's per domain, so you have to get that in a different way. But the fundamental question that you have some sort of near real-time delivery requirement.

MTS-AV: is the same. So I think it boils down to the same kind of problem, even

MTS-AV: If it's structurally different.

MTS-AV: And if we find a way to do this efficiently for certificate revocation lists, probably we also know how to do it for certificates themselves.

MTS-AV: Yeah, I mean, a lot of things have shown revocation just doesn't work, right? Either you have massive big CRLs, or you have OSCP, which leaks huge amounts of privacy stuff and doesn't scale.

MTS-AV: But the reason I sort of started wondering about the methodology is if you add up all of the numbers, you got to 92.8% or something. I think it was the next slide over.

MTS-AV: Yeah, go back one.

MTS-AV: Nope, you don't have slide numbers, unfortunately, so I didn't write them down. I think it was forward, forward, forward, forward.

MTS-AV: So yeah, go back. You just passed it.

MTS-AV: Nope. Next one.

MTS-AV: Next one?

MTS-AV: Next one.

MTS-AV: You had a thing where you added up all of the numbers and you got 98% or 92% of.

MTS-AV: These ones. Yes, that. So 92 point blah blah blah had cases of active delegation when they start

MTS-AV: Yeah, so then you're claiming that there are almost 3%, or a little bit more than 3%, of certificates that were issued with no DNS working at the time? Yeah, things that…

MTS-AV: This is an empirical evidence. It's just what you can observe from the external world. Sometimes you may have, like, certificates that have been issued for a domain name that existed less than a day.

MTS-AV: And in that case, it's extremely hard to

MTS-AV: find prior DNS evidence, because if it was existing for less than a day, then you have a certificate that was issued with the correct DNS evidence, but…

MTS-AV: the domain was not in the zoom file. That that is our historical view, and that means that there was no

MTS-AV: prior DNS evidence, according from an external observer.

MTS-AV: I mean, it's just the quirks of the… how the… what you can observe from the outside, basically. Yeah, I mean, it's also… is it possible that you're not getting all of the DNS information that you think you are? Right? Like, you're looking in…

MTS-AV: Some of these are.

MTS-AV: like it's

MTS-AV: It's unclear where you got all the, does the domain exist information from. I'm guessing some of the CZDS? CZDS zone file. Yeah, it's daily CZDS zone file. Okay, so if the CZDS file hasn't updated, because sometimes that happens, that could push it over. Sometimes they don't update for a couple of weeks. Okay. Yeah, so that case is a little mixed, basically.

MTS-AV: Hey, Kathy, is there anybody else in the queue? No.

MTS-AV: Okay, thanks, Rafaela.

MTS-AV: Thank you.

MTS-AV: Peter.

MTS-AV: Thank you. I think Kathy's gonna share the slides until I get to the demo.

MTS-AV: All right, so, DNSSEC timeline for TLDs sounds boring, but I'm not aware of any, Public.

MTS-AV: timeline that, gets updated every day or something. So, yeah, this is about this topic. Next slide, please.

MTS-AV: Recently, my colleague Barbara was writing an article about common beliefs about DNSSEC put to the test.

MTS-AV: It's been published on Tuesday, and some of those people here in the room have endorsed this.

MTS-AV: And, overall, roughly 60.

MTS-AV: And, we tried to, look at some…

MTS-AV: of the opinions and facts, or maybe non-facts circulating about DNSSEC and checking them against reality. So, for example, how much it's government-controlled, or whether it's cryptographically weak.

MTS-AV: And so, at the time, we were looking for data, and, we…

MTS-AV: found various things, Victor Duchovny's, and Wes Hardiker's DNSIC tools, for example.

MTS-AV: But we didn't find such a DNSSEC,

MTS-AV: TLD timeline, so that's what motivated this.

MTS-AV: When we were drafting the article in June, All right, next slide.

MTS-AV: So here's, the other stuff that we found. There's the IANIX outage list, which is well known. It is a curated list of ENSAC, cost.

MTS-AV: Validation or resolution outages, and

MTS-AV: Yeah, it's, there's a lot of incidents from the past, and recently there's less. It's unclear why that is. Then there's the ones from Victor and Wes that I just mentioned, and there is also, a presentation from just yesterday by John Christoph at VeriSign.

MTS-AV: He presented at PLNOG yesterday, and they did, I think, a SACSpider-based measurement of DNSSEC availability according to some weighting, and they came up with a very high availability. Unfortunately, the slides aren't online yet, at least as of yesterday.

MTS-AV: I'm looking very much forward to what exactly they did to figure that out.

MTS-AV: But still, we didn't find this DNSSEC timeline, so we built that, and, it does, An open record of…

MTS-AV: Essentially, what the status of… TLDs is every day with respect to DNSSEC.

MTS-AV: Next slide.

MTS-AV: Hmm.

MTS-AV: So what do we do? Every day in a GitHub action, we fetch the root zone and look at all the TLDs.

MTS-AV: Then we query the TLD's SOA record using a local unbound that's in a Docker container. It's configured to do, its own validation, it supports ETE.

MTS-AV: And, we then look at whether the AD bit was, set in the response… the unbound resolver's response. We record all kinds of… all kinds of other response details, like the R code and EDE, so that…

MTS-AV: If we have a classification bug, for example, we can reclassify later based on what the response actually said. And then we transform this into a JSON that gets pushed into a web app on GitHub pages. And so there is no separate infrastructure that we operate for this.

MTS-AV: And the web app, has various user interface widgets to slice and dice the data. We'll get to that in the demo.

MTS-AV: Yeah, so there is,

MTS-AV: Categories by GTLD and CCTLD, and by IDN and non-IDN, and various other

MTS-AV: things, and yeah, so one fun fact is, at the bottom right, you see the diagram of, GitHub Action runtime delay after it was scheduled, so it's scheduled every day at 3.30 UTC.

MTS-AV: And usually, it starts, like, half an hour later, or two hours, or something. In the worst case, it starts half a day later. We don't really care, it's still once a day, but it's just an interesting.

MTS-AV: Side insight, I think.

MTS-AV: All right. We started doing this in June.

MTS-AV: Next slide.

MTS-AV: So there's different statuses that come out of the classification. Secure is, you know, DNSSEC works fine. Insecure is when the TLD

MTS-AV: doesn't have a DS record, or has a DS record whose algorithm is unsupported, but that is currently not the case for any TLD.

MTS-AV: There is BOGUS when DNSSEC validation failed, and we record the EDE, extended DNS error codes, that indicate what exactly went wrong. The unbound resolver returns this, and

MTS-AV: Candidate errors are, like, signature expiration, or missing keys, and stuff like that.

MTS-AV: There's unreachable, which is when we don't get an answer. Of course, this could also be because our,

MTS-AV: Runner is, disconnected for some reason, so we do multiple attempts, and we only

MTS-AV: use this verdict if we confirm that we can actually talk to the root, but not to the TLE.

MTS-AV: And otherwise, we classify it as, unmeasured, which is error. And the error category normally is hidden in the display, because it's not an interesting.

MTS-AV: insight about DNSSEC configuration, it's more like a measurement artifact if something is an error.

MTS-AV: Okay, next slide.

MTS-AV: Yeah, so here's, a bit more about what we do, for unreachable. So when there is no response, we try again 2 minutes later, and we process other TLDs in the meantime.

MTS-AV: And then we check whether we can actually talk to the root service, which…

MTS-AV: Probably we can if we, process other TLEs in the meantime, but there is no communication between the different threads, so we just, you know, check the route anyway.

MTS-AV: Because there's no awareness, so to speak. And if we can talk to the root, but not to the TLD's name service, then we do unreachable. We also record unreachable if the unbound resolver itself

MTS-AV: Returns that, that can happen, for example, when, none of the IP addresses of the name servers.

MTS-AV: actually are reachable, but I mean, the IP addresses could be fetched, but they don't respond.

MTS-AV: Hmm.

MTS-AV: Yeah, so this, was a bit of engineering, not very complicated, but still, we didn't immediately see it was necessary.

MTS-AV: It did become necessary, because once we.

MTS-AV: You know, before we had that, there were,

MTS-AV: time windows within the measurement of a few seconds where there were connectivity issues for some reason. It's probably some rate limits that, you know, for outgoing DNS queries on GitHub runners, or I don't know.

MTS-AV: But we had stretches of,

MTS-AV: of sequential TLDs that were, you know, we processed them alphabetically, and there were sequences of, like, 3 or 5 or something that would, together.

MTS-AV: appear unreachable even though they were operated by different people. And that seemed implausible, so we thought it…

MTS-AV: Probably is some measurement issue, and it turned out to go away if we do this.

MTS-AV: It's not really clear, I wasn't able to find any documentation about GitHub rate limits, on this.

MTS-AV: sort of thing. And it seemed, like, random. I mean, it's… we do a few thousand queries, and then…

MTS-AV: Even if we slowed it down, it didn't go away, so I don't know.

MTS-AV: Next slide, please.

MTS-AV: All right, so at the top is, the timeline, so to speak. I mean, not the results yet. We are the yellow part on the right.

MTS-AV: from June 20… sorry, June 17 this year until today. All the other data from 2012 on is from Jaap Akerhus.

MTS-AV: Who used to be in the SSAC, and he has a similar system that, runs on his private equipment, or maybe an LNET lab's equipment, I don't know, but anyway. It's not a publicly available, thing. He presented that at ICANN55.

MTS-AV: And, he handed over his data to us, and, we included that in our displays, so we have

MTS-AV: data from back in 2012 until now.

MTS-AV: And, there were some

MTS-AV: format conversions. He also does two measurements a day, and not just one. And he only looked at records, sorry, at DLDs that had a DS record, where we also started looking at insecure domains, so that you can have nice graphs about that, too.

MTS-AV: Anyway, so there was some sort of adoption necessary.

MTS-AV: M.

MTS-AV: We started including insecure domains in August only because we noticed we would like to have these graphs. And we don't have any insecure historic data.

MTS-AV: But I think it can probably be reconstructed from historic zone files from the root, so we might import that later so that the graph doesn't jump.

MTS-AV: Next slide.

MTS-AV: All right, so it's demo time. I will switch to…

MTS-AV: Good luck, yeah. I mean, it all depends on GitHub.

MTS-AV: Anyway, so I will share my screen, and then I hope I don't mess up.

MTS-AV: What I'm doing…

MTS-AV: Dip, do, do, do, do.

MTS-AV: Give me one second.

MTS-AV: I think I wanted to show one more slide, actually, but I will show it on my own machine.

MTS-AV: Okay, so this is, essentially the display that the website gives, except that it's sliced yearly, which the website doesn't…

MTS-AV: do, and this is, easier to digest, so I will start with this. This shows, how signing grew. You can see that in 2012, this is always January 1st, there were just a few under 100

MTS-AV: TLDs that were assigned and most of them were CC TLDs.

MTS-AV: And, I don't know how many, maybe 50. There's now roughly 170, I think, signed CCTLEs, or 150.

MTS-AV: And they were by far overtaken by the GTLDs, of course, because of the 2012 round, which got things delegated around 2016.

MTS-AV: And you can see that the maximum is in 2018, and after that, a bunch of brand TLDs died.

MTS-AV: So it went down.

MTS-AV: And yeah, so that's the general behavior. And then in the middle of it, there is the IDNs, which are a small fraction.

MTS-AV: Okay.

MTS-AV: So… Here.

MTS-AV: So this is our beautiful GitHub URL. We might actually —

MTS-AV: Switch to a nicer domain at some point.

MTS-AV: Let me… Pull up my notes, what I wanted to show. So how signing grew is essentially,

MTS-AV: The absolute counts, you can have absolute counts here. Otherwise, you have log scale — sorry, percentage counts. When you load the page.

MTS-AV: The percentage count actually zooms in.

MTS-AV: Because otherwise, you wouldn't be able to see anything. If you go to 100% here…

MTS-AV: Right? Then, the failures are down here. It's not very useful.

MTS-AV: So we use percentage display by default, but we zoom in, and the top is roughly 3%.

MTS-AV: In the beginning, the failure rate was quite high.

MTS-AV: Which,

MTS-AV: improved over time, but the main reason for that is that there is a lot of GTLDs now which are required to operate the UNSAC in a solid way, and they are the large majority of the green.

MTS-AV: Okay, what else did I want to show?

MTS-AV: Yeah, so the absolute counts are essentially what we just saw in the other slide, but they are in logarithmic scale, because otherwise you would not be able to see the small numbers at all. So each.

MTS-AV: Tick here on the y-axis is a factor of 10.

MTS-AV: And that's why it looks a little skewed, but actually, it's sort of magnifying the bottom part of it.

MTS-AV: You can also see that there are some measurement errors.

MTS-AV: The… Double click to reset. Oh, okay. So, the measurement errors are usually

MTS-AV: Not displayed, so if we add them… We get in 2021.

MTS-AV: 2… Days where there were measurement errors.

MTS-AV: Roughly 50% of the measurements failed. This is from YARP's historic data. There was some connection issue or what, I don't know.

MTS-AV: And…

MTS-AV: yeah, we figured it's best to keep them in the data set, but don't display them by default. I don't think we can retroactively fix it.

MTS-AV: And so, if you… Go to percentage display and ignore them, then,

MTS-AV: Then you will not see them.

MTS-AV: Yeah.

MTS-AV: Well, oh, in fact, you do, because percentage is, to the full number.

MTS-AV: Okay, alright, then let's reload this to go back to the… initial display.

MTS-AV: I want to, share one specific day, January 3rd, 2020, so we can, go here. So, first of all, you see, for any day, what's happening. There is, like, on this day.

MTS-AV: July 13th, 2017, there were 1,394 secure TLDs and two bogus ones, which is 0.1%, and so we can now drag, here in the.

MTS-AV: Dataset?

MTS-AV: And zoom in.

MTS-AV: And ignore the secure ones, and then you see…

MTS-AV: that on the 3rd of January, the number of bogus domains went from 2 to 8.

MTS-AV: And so you can click on that, and then you get the day detail at the bottom.

MTS-AV: Okay, so the… Green stuff is hidden.

MTS-AV: And what we have here are the bogus ones. Those are all IDNs from India?

MTS-AV: And, they all had signature expiration.

MTS-AV: going on, so the EDE code was 7, and you can see here, signature expired. And you see which IP address the response came from.

MTS-AV: And there's,

MTS-AV: a null number of Ds records, because I think the data from Yap didn't record that number.

MTS-AV: I think we record since June how many DS records there are.

MTS-AV: Anyway, so these are the ends.

MTS-AV: were new IDNs from new scripts, and they were from the 2017 batch, and not open for registration. So apparently it was testing.

MTS-AV: And India had older IDNs before that.

MTS-AV: And they stayed secure, so they did not have an issue.

MTS-AV: And they also have the .in domain, obviously, and that also did not have an issue. So this is not really a production outage.

MTS-AV: It is more like, an easily visible… Testing artifact, it seems.

MTS-AV: And then after 5 days, that went away again.

MTS-AV: You can also, I don't know which day this is, I selected a random day.

MTS-AV: You can see here, like, when you… Mouse over… Which of the things, or what status?

MTS-AV: And then when you click on it, maybe let's here, click on this… Click on one of them.

MTS-AV: Line highlights, so when the… List is longer, it's closer to that position.

MTS-AV: Okay, so let's look at another story. There is the… Country of Jordan.

MTS-AV: They have an IDN.

MTS-AV: So you can filter by IDN. So you can add one or more or even other TLDs. Xn dash dash.

MTS-AV: MGBAYH.

MTS-AV: 7GPA. Can anyone do that, like, in their minds? Like, some people can do hex.

MTS-AV: But does anyone do puny code?

MTS-AV: Anyway, so you can filter by that.

MTS-AV: And then you can see that this CCTLD was introduced in 2023.

MTS-AV: And that most of the time it has been bogus, and

MTS-AV: Yeah, so that's really weird. But also it seems like nobody's really using that.

MTS-AV: Because if people were using that, then it would… failed pretty hard.

MTS-AV: Hum… Yeah, so this is from…

MTS-AV: I guess what we measured, roughly this period.

MTS-AV: And before that, it was YARPS data.

MTS-AV: And, the interesting thing is that.jo, which is the Jordan ASCII TLD, It's not even signed.

MTS-AV: So the IDN variant is signed but broken, and the non-IDN variant is not signed. So, it doesn't seem like they're really into doing DNSSEC so far.

MTS-AV: And I would also suggest that that is probably testing.

MTS-AV: So that raises the question whether ccTLDs are different when it comes to IDNs and DNSSEC.

MTS-AV: So that, gets us to a different selector, so we can look at,

MTS-AV: How do we do this?

MTS-AV: Absolute numbers may be… And, only bogus… And… Dip, dip.

MTS-AV: Let's see, let's compare… Well, the GTLDs, they… So maybe…

MTS-AV: I want to look at the GTLD. So, that's a very small number of failures, and if you only look at the ASCII variants.

MTS-AV: IDNs, okay.

MTS-AV: Anyway, so if you look at… whoops, I didn't know that disappears completely.

MTS-AV: The CCT LD ASCII versions.

MTS-AV: On some days, have one failure, or two.

MTS-AV: On one day, which is,

MTS-AV: June, 2 years ago. June 11.

MTS-AV: If you look at the IDN variants of it.

MTS-AV: They have longer stretches of that, and sometimes more in one day. This is the India thing that we just looked at.

MTS-AV: And you have to consider that the CCTLD IDNs are only

MTS-AV: Yeah, 3% of signed TLDs are IDNs, CCTLDs, so 1 30th of the signed TLDs are CCTLD IDNs, but they make up for 80% of all the bogus events in this database since 2012, and they make up 100% of them since we started taking over the measurement.

MTS-AV: So that really suggests that CCTLD IDNs, for some reason.

MTS-AV: Are not as solidly maintained in regards of DNSSEC stability.

MTS-AV: All right. And then finally, we can look at who is still unsigned.

MTS-AV: So, we can do insecure. We just started adding that in… Umm.

MTS-AV: August, so there is, like, a jump here, and we might backfill the other stuff.

MTS-AV: So we can select that, and then we can look at any random day here, and we can go down, and we see the list.

MTS-AV: And, so those are currently still insecure. Those, I think, are 15 IDNs, and the rest of them, like, 71 or something.

MTS-AV: Our ASCII ccTLDs and all the gTLDs, however, are secure, because that's mandated.

MTS-AV: Okay. Recently signed were Dot DJ, which is, I think, a nice.

MTS-AV: Suffix?

MTS-AV: and also Pk, Pakistan.

MTS-AV: I'll hand it back to… Kathy, after this demo…

MTS-AV: Okay, you can skip, like, a few slides, this is in case the demo wouldn't work.

MTS-AV: Yeah, exactly. So here we are. And so what can't the timeline show? So we have, for example, the outage of DOT DE in May this year.

MTS-AV: This was still when Jaap's measurements were what fed our database, and he has runs in UTC time in the morning at 6 and at 6 PM.

MTS-AV: And the DE outage was, between those windows. So, the 3-hour disruption was not visible, and that's why it isn't there. So you can't really use this timeline to spot Every outage is more like a trend indicator or something like that.

MTS-AV: And, yeah, so even if his run had been there, there's a chance of not spotting it, because not everything was wrong.

MTS-AV: So, some stuff did actually work.

MTS-AV: And also, it's a quite limited view, right? There's the GitHub, restriction, you know, whatever they have in their environment might affect the measurement. It's only one vantage point, it's one resolver.

MTS-AV: And yeah, so it's good as a base indicator of stuff, but it's not — something to…

MTS-AV: Objectively tell whether something is really the case or not.

MTS-AV: Globally.

MTS-AV: All right, next slide.

MTS-AV: So that's, the final slide.

MTS-AV: The code is open source, it's on GitHub here. Everybody, every TLD or people elsewhere can look up their data. I don't know if you spotted that during the demo, but the address bar actually changes as you select things and different filters, so you can click on these things that are appended in the.

MTS-AV: The URL bar, if you look at the PDF file here, or even when you found a nice display for something, you could just share it with somebody else.

MTS-AV: And yeah, thanks again to Jaap for sharing the data with us and happy to take any questions.

MTS-AV: Warren.

MTS-AV: So first off, thank you. This was awesome. Really, really good work.

MTS-AV: But as you noted, you can't really use it as an outage detector, because you do a single… How hard would it be, and how resource-intensive would it be to run this every, like, 2 or 3 minutes? Like, how much hardware would you need?

MTS-AV: I don't think you would need very much hardware, so,

MTS-AV: Jaap's measurements ran for 20 minutes and ours run between 6 and 16 minutes.

MTS-AV: Our main uncertainty is when they run, because the GitHub scheduling, but that could be fixed with an internal.

MTS-AV: So, with an external trigger. And also, it doesn't have to run on GitHub, right? There is other ways of running it. And if we did some, better optimization of.

MTS-AV: how the concurrent requests are done, or maybe shuffled the order to evade rate limits or whatever, then probably the runtime could, be reduced without introducing failure. So I think once every 10 minutes probably is realistic.

MTS-AV: We haven't looked into it because we,

MTS-AV: Mainly started from making Jaap's thing public, sort of.

MTS-AV: But there are certainly opportunities to change that.

MTS-AV: Are you suggesting to donate hardware? Yeah, possibly. Okay, cool.

MTS-AV: Mary, I don't have no further questions.

MTS-AV: Any questions from outside of SSAC are welcome.

MTS-AV: Okay, we, I guess we give you all 25 minutes back?

MTS-AV: Thank you all for coming, and we have another lightning talk session coming up in a couple of days, so come back for that.

MTS-AV: Thanks, Peter.

MTS-AV: Scott, please stop the recording.