kɜ:ʳ @chr@cybre.space

@quat @Corax our media provider is having issues right now, sorry :( https://status.digitalocean.com/incidents/crjwvq6b7lqw

@packetcat @halcy thinking about this, it may just be that the synapse federation traffic is too much for one process. i'll also try spinning up another worker for that.

@packetcat @halcy right now none of the sockets are in that status. also, the netdata alerts indicate that it's the accept queue overflowing rather than the syn queue

(although, that being said, seeing that cloudflare bumped their somaxconn to 16384 gives me the confidence to do the same, so maybe that will help)

@packetcat @halcy i have ufw enabled, but nothing limiting connections, just flat allows on the ports i need for the applications running on it

@packetcat @halcy on reading the man page, those numbers are in bytes, so probably not very large after all. there's also a bunch of sockets in `TIME_WAIT`, which seems relevant, but no program/PID information for those.

@packetcat @halcy recv-Q and send-Q are mostly zeroes, except there's quite a few sockets open on the synapse federation sender worker with what i assume are fairly large recv-Q sizes. guess i'll take a look at that! thanks for the tip

@packetcat @halcy i assume you meant `netstat`? lots of info here, what should i be looking for?

@halcy i already bumped up somaxconn to 1024 and tcp_max_syn_backlog to 8192 so whatever is happening is not just bursty traffic (or, it's really bursty really often)

@AFresh1 @aschmitz i've actually got nginx in front of it, both on the client API port and on the federation port. not sure how to check nginx's queue size. netdata has some metrics for nginx out of the box but nothing related to tcp queues that i can see.

netdata on the cybrespace matrix server has been reporting the following alarm at a steadily increasing rate over the last couple months: ip.tcp_accept_queue, "the average number of times the TCP accept queue of the kernel overflown, during the last minute"

this seems like the symptom of another problem, but i can't find much information about what it might be a symptom *of* (other than, possibly, a slow application not accepting connections fast enough).

can anyone advise? what should i be looking for to further diagnose the cause? what, if anything, could i do to mitigate this at the system level?

@geekylou yep, since a while ago

@rick_777

@witchfynder_finder ...etc check the "potions" section of the emoji picker

@galuade prolly not gonna mass import them but i do take requests

@ky0ko which instance(s) are you seeing a 4h delay with?

@witchfynder_finder getting sidekiq'd in the snout every time i try to have a relaxing instance administration day

@witchfynder_finder

cybre.space users: apologies for recent federation delays, it seems during times of peak activity we've outgrown the number of sidekiq workers we've been using. i've scaled them up and hopefully the instance will perform better under load, but i'll be keeping an eye on things over the next few days to be sure.

@ben @nightpool still no idea why people are seeing urls on this domain since we switched to raw CDN urls for media but i went ahead and set up that domain for automatic renewal anyway