kɜ:ʳ @chr@cybre.space

the cybre.space matrix server is down today as our database filled up its storage volume; currently working to free up space to get it back up and running, and submitted a request to our cloud provider to increase the disk size. don't have an ETA but i hope to have it back up by this evening.

hello! it's been a while since i've posted here other than to respond to specific inquiries, but since there's been a steady number of signups to cybre.space through community member invites and word of mouth: welcome!

if you're new here or if you just haven't seen it before, we have a patreon and liberapay which are used to pay for the costs of running the mastodon¹, matrix², and gitea³ services that we provide here.

¹https://cybre.space
²https://matrix.cybre.space
³https://cybre.tech

if you're using these cybre.space services, please consider supporting us through one of these means:

https://www.patreon.com/cybrespace
https://liberapay.com/cybrespace

we do not accept cryptocurrency donations due to environmental, social, and technological concerns. if you were going to reply to this post asking if you could donate crypto, please consider divesting yourself of your cryptocurrency resources and supporting an environmental or social justice initiative instead.

thank you!

@Alamantus @nightpool looked into this. i added the header to our media CDN configuration and it looks like audio is playable now. thanks for the heads up.

@OberstKrueger if it wasn't corrected shortly after the fix, it probably won't be corrected, but missing posts can still be fetched by loading threads that they may have been a part of and pasting direct URLs into the search bar

if you run mastodon on ubuntu or debian with jemalloc and you notice sidekiq crashing frequently, it may be related to this issue with FFI: https://github.com/mastodon/mastodon/issues/15751

the fix in our case was to add LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so to the systemd unit files, so that all native (non-ruby) code could use the same memory allocation system

just fixed an issue with cybre.space's sidekiq worker that was causing it to frequently (30-60s) restart, interrupting longer-running tasks and possibly causing posts to get lost from feeds. hopefully things should be a bit more stable now

and then suddenly its fine?? didnt get a response to my support ticket but i'll take it

@haskal digitalocean is definitely failing their SLA, but hetzner has been basically fine til now aside from an occasional short outage that kicked something of ours over. we are unfortunately getting what we're paying for, there, though.

well, this sucks

server slowness appears to be due to degraded disk performance on the host, following up with them to see if there's anything to be done about it

@TheGibson yeah we're at less than 50%

@TheGibson we seem to be good on memory and the sidekiq queues are empty, weirdly. current theory is it might be a slow db connection, possibly due to disk i/o

cybre.space users: this appears to be due to the cybre.space web server taking too long to respond and returning a 504 / timeout error. if you see this happen, it might mean the server is just processing slowly and will eventually submit the post! (looking into why it's taking so long, but this should help you avoid accidentally submitting duplicates in the meantime)

looks like cybre.space might be having intermittent problems with duplicating new posts made by users on here, looking into it now. (in the meantime, if someone sends you multiples of the same reply, it's probably not their fault!)