Show newer

@venko it would probably take a while normally but it might be worse than normal because there's some job backlog right now

@gargron yeah its pull, running a query in the rails console to figure out what all the jobs are

@nightpool @violet @haskal okay i patched to just log and re-raise that error. let's see if it reduces the number of posts with unavailable media

technical details 

@nightpool @violet @haskal good point. the ProcessingWorker job would have failed instead, and gotten retried by sidekiq. might be best to just restore that behavior tbh.

@violet @haskal @nightpool
long story short is in 3.3 we're now seeing posts with missing images because mastodon is handling errors from the media store, whereas before it would have forced the remote server to retry.

the real solution would be to reduce our error rate from our media store, but a temporary solution is to immediately retry media store if they fail. i'll try that and see if that helps.

technical details 

@violet @haskal @nightpool i think the problem's down to this commit: (which is new in mastodon 3.3). i added some logging to cybrespace to confirm we're hitting the case on line 255

if our storage backend errors out while we're caching a remote image, it'll hit this Seahorse::Client::NetworkingError, which will cause it to just give up on fetching the image.

before 3.3, it would have thrown all the way up to where the request came in, which would probably have resulted in the remote server seeing a 500 error from us and failing to deliver the post. it then would have gone into retry logic and probably come in successfully a few minutes later.

@violet @nightpool not sure if its correlated with the update, but this usually means that at the time received the post, it couldnt cache the media, either due to a problem with our media backend or the remote server’s copy of it being unavailable. not much we can do about this unless it turns out it was an issue with mastodon not fetching stuff correctly or something, i’m afraid

@patience @nightpool should be back now and hopefully not die again within a few days...

@patience @nightpool looks like the search server has outgrown our available resources, i'm looking into possible workarounds

@violet @vantablack should be back up now, i'll try to work out why it's going down and see if i can stabilize it

@niconiconi should be back up now, i'll try to work out why it's going down and see if i can stabilize it

kɜ:ʳ :dragnmlem: relayed

Question about #OrishaLand. Can any of you find a single article about it that isn't written by a cop when you search online? Very concerned about this, given that it's gotten a lot more attention from fascists, including local police, in the last few days. Can someone who's not fascist write about it? Can people work on a wikipedia page or something?

kɜ:ʳ :dragnmlem: relayed

A group of Black revolutionaries have started an autonomous zone named #OrishaLand in #Austin, #Texas. They've already done a lot to fight #gentrification by organizing material support & political education for the local Black community. It's a real glimmer of hope for what has otherwise been a shitty #BlackHistoryMonth.

For more about their recent efforts:

On social media:
Instagram: @400and1
Twitter: @_400and1
Facebook: /400and1

@devurandom should be back now, thanks for the heads up

@niconiconi should be back now, thanks for the heads up is now on mastodon 3.3! there were quite a few changes since 3.1, you can read about them here:
and here:

there are a few known issues in the fork, which i did not have time to address tonight, but should get to by the end of the week. please feel free to ping @ me or create new issues if you see anything else wrong after the update:

Show thread

okay after some, uh, brief jankiness due to a dependency needing an update, we should be fully back now

Show thread
Show older

cybrespace: the social hub of the information superhighway jack in to the mastodon fediverse today and surf the dataflow through our cybrepunk, slightly glitchy web portal support us on patreon or liberapay!