Follow

DO NOT run scheduled requests to a public server at the top of the hour, pick a random minute. The worst victims of this problem are community NTP servers - at every hour and especially 00:00:00 UTC, the traffic spike is just impressive.

Also consider rate-limiting and exponential backoff in retry loops. Otherwise the results can be quite spectacular.

Some Internet folklore from China: In 2009, a random guy paid for DDoS to attack a competing game server and disabled its DNS service. Just a regular day, right? Following that, a popular video player with 1 billion installations, appropriately named Storm, all entered an endless retry loop to phone home, DNS requests flooded China Telecom's backbone and caused a nationwide network outage. The attackers suddenly found themselves to the state's enemies.

@niconiconi systemd unit options, in case one uses a systemd timer:

[Timer]
AccuracySec=10 seconds
RandomizedDelaySec=5 minutes

these two options may be adjusted depending on how much (or little) variance there should be.

@niconiconi (common for doing shit like syncing an arch repository)

@niconiconi Pro tip: If you run an NTP server, shut it down for five minutes around midnight for maintenance.
That will teach them.

@wakame@tech.lgbt @niconiconi@cybre.space Just return a slightly off time, so next time they will fetch it not at the peak. 👉😎👉

@wakame
No, it will not teach anything. This has been discussed to some extend in the public NTP server community. My take-away: There is nothing short-term that can be done to rectify the situation.

There's a "kiss of death" packet defined in the protocol, which you can send to badly programmed rogue clients (it contains no usable time info), or you can ignore their requests. Either will cause many to simply send more requests without exponential backoff.
@niconiconi

@dj3ei @wakame Yep, KoD will never be respected by legacy clients already deployed on millions of zombie-like embedded & networking devices that never receive any update in their lifetimes. We have to deal with them for as long as Internet exists...

@niconiconi @dj3ei It was more meant as a humorous remark, not an actual solution.

I personally know too many people who would rather copy/paste fifteen bad stackoverflow examples than to read even the synopsis of a standards document...

But I just head a great idea for a cyberpunk work of fiction where covert members of the FOSS army sneak into companies and other places housing old hardware and destroy (or even update) them. :blobcataww:

@niconiconi This is known as the #ThunderingHerdProblem

(Especially in system-configuration contexts, but also others.)

@niconiconi OpenBSD has a neat crontab(5) extension for this

man.openbsd.org/crontab.5

The ~ character is used to indicate that a random value between a range (or without a range, the full range of the field) should be used as the time to execute the job.

Sign in to participate in the conversation
Cybrespace

cybrespace: the social hub of the information superhighway jack in to the mastodon fediverse today and surf the dataflow through our cybrepunk, slightly glitchy web portal support us on patreon or liberapay!