supakaity
Now
•
100%
Poor li'l blighter got attacked by a bin chick'n and was limping away last we saw em... after goin' the bin juice drinkin' irrit of course.
supakaity
Now
•
100%
Apparently I was wrong, it's decided it's done and working again now.
supakaity
Now
•
100%
The pict-rs upgrade is ongoing.
From what I can tell it'll be about another 5 hours. I'm going to have to go to bed and check on it in the morning.
Unfortunately the stock-standard lemmy-ui doesn't like it that pict-rs is migrating to a new version of database and not serving images, so it's stubbornly just not working at all.
Hey all! Our lemmy.blahaj.zone has been updated to v0.19.5. Let us know if you notice any issues with the upgrade!
supakaity
Now
•
100%
Our best haj, Shonky (they/them) is available now, over at their own Github repository for use when referring to Blåhaj Lemmy.
I'm guessing we'll need them to make an appearance for the Canvas template.
supakaity
Now
•
100%
So this was at the very start of the s02e10 episode - "Pegasus".
supakaity
Now
•
100%
Not that I am aware of, and not that it's not possible, but the protocols are quite different, the Lemmy API, the Misskey API (with various dialects for Sharkey, Iceshrimp and Firefish) and the Mastodon API (with Akkoma, Pleroma, etc) are all very different from each other and even making a third party UI that completely understands and implements a single API / dialect is an enormous undertaking.
supakaity
Now
•
100%
I think it's because a lot of her fans were like "hang on, she's just misunderstood, she's not really transphobic, just doesn't want those fake transes going into women's spaces", and Rowling kept it pretty vague, open for misinterpretation, not really ever able to be pinned down as actually being provably transphobic in the minds of many.
But something happened recently and it tipped her over the edge, from keeping her donations low key and her transphobia aimed at the theoretical "not really a woman 'trans-identified-male' boogeyman-woman" (who doesn't really exist), to outright making open donations to anti-trans groups and attacking real trans women.
All the sudden her supporters are like... Wait, this is not who we thought you meant… Where are these scary men? The ones invading women's spaces that you were talking about?
supakaity
Now
•
100%
Wow, okay that's weird. It's doing it for me too, must be some debugging option
I've been using alexandrite and haven't noticed.
I'll take a look tonight.
supakaity
Now
•
100%
For a desktop client, it actually works pretty nice on mobile too.
supakaity
Now
•
100%
okay, i'll take a look... probably something to do with the media proxy or the pictrs config.
supakaity
Now
•
100%
Yeah, you just need to login again and put in your totp code this time. Kinda annoying that it doesn't provide a field for your totp code on the first page.
Hi all our lovely users, Just a quick post to let you all know that along-side the upgrade to 0.19.3, we've also added a couple of alternate UIs to the Blåhaj lemmy for you. Obviously the default lemmy-UI at https://lemmy.blahaj.zone still exists and has been updated to 0.19.3 alongside the lemmy server update. There's also now an Alexandrite UI at https://alx.lemmy.blahaj.zone which is a more modern, smoother UI, written in svelte, by [sheodox](https://sheodox.com/). And then for those who are nostalgic for reddit days of yore, and memories of when PHP websites last ruled the earth, there's MLMYM (courtesy of [rystaf](https://github.com/rystaf)) at https://mlmym.lemmy.blahaj.zone. Please enjoy, and I hope the upgrades work well for you.
supakaity
Now
•
90%
I also agree.
While I couldn't care less (it's their body and bodily autonomy is super important to me), I'd be a little hurt if they didn't tell me about it.
In that case I'd be more interested in talking to my partner to try and understand why they felt that they couldn't talk to me about it and needed to keep it a secret.
supakaity
Now
•
100%
I asked my AI and that was its thought also.
supakaity
Now
•
100%
So I've completed migration of the pict-rs server to our new infrastructure. I'm not sure what the issue was you were having, as I am unable to replicate.
Based on the father ambiguous error message, I presume it was returning a "Request timed out" or something.
Anyhow, after the migration, I'd be interested if you still see errors. Let me know.
supakaity
Now
•
100%
Migration has been completed!
We're currently in the process of migrating our pict-rs service (the thing responsible for storing media/images/uploads etc) to the new infrastructure. This involves an additional step of moving our existing file-based storage to object storage, so this process will take a little time. New images/uploads may not work properly during this migration, however existing images should continue to load. We expect this migration to take about an hour. [EDIT] Migration has completed. 685,271 files / 153.38 GB were migrated. Copying to object storage took about 1.5 hours. Starting service back up on new server and debugging took another 30 minutes. Timeline: - Migration started at 2023-10-01 22:43 UTC. - [+1h32m] Objects finished uploading to object storage at 2023-10-02T00:15 UTC. - [+2h06m] Migration was completed at 2023-10-02 00:46 UTC.
supakaity
Now
•
100%
Okay, the new media cache server was locked up. Fixed the code and redeployed it. Images should come good after a page reload.
Blåhaj Lemmy will be down for database migration to the new servers in approximately 1.5 hours from now (06:00 UTC). Downtime is estimated at under an hour. I will have more details on the maintenance page during the migration and update the status as the migration progresses.
The server will be briefly down while we install a new updated version of lemmy and restart it. The maintenance window is 15 minutes, but should be much shorter.
So it's been a few days, where are we now? I also thought given the technical inclination of a lot of our users that you all might be somewhat interested in the what, how and why of our decisions here, so I've included a bit of the more techy side of things in my update. **Bandwidth** So one of the big issues we had was the heavy bandwidth caused by a massive amount of downloaded content (not in terms of storage space, but multiple people downloading the same content). In terms of bandwidth, we were seeing the top 10 single images resulting in around 600GB+ of downloads in a 24 hour period. This has been resolved by setting up a frontline caching server at pictrs.blahaj.zone, which is sitting on a small, unlimited 400Mbps connection, running a tiny Caddy cache that is reverse proxying to the actual lemmy server and locally caching the images in a file store on its 10TB drive. The nginx in front of lemmy is 301 redirecting internet facing static image requests to the new caching server. This one step alone is saving over $1,500/month. **Alternate hosting** The second step is to get away from RDS and our current fixed instance hosting to a stand-alone and self-healing infrastructure. This has been what I've been doing over the last few days, setting up the new servers and configuring the new cluster. We could be doing this cheaper with a lower cost hosting provider and a less resiliant configuration, but I'm pretty risk averse and I'm comfortable that this will be a safe configuration. I woudn't normally recommend this setup to anyone hosting a small or single user instance, as it's a bit overkill for us at this stage, but in this case, I have decided to spin up a full production grade kubernetes cluster with a stacked etcd inside a dedicated HA control plane. We have rented two bigger dedicated servers (64GB, 8 CPU, 2TB RAID 1, 1 GBPS bandwidth) to run our 2 databases (main/standby), redis, etc on. Then a the control plane is running on 3 smaller instances (2GB, 2 CPU each). All up this new infrastructure will cost around $9.20/day ($275/m). **Current infrastructure** The current AWS infrastructure is still running at full spec and (minus the excess bandwidth charges) is still costing around $50/day ($1500/m). **Migration** Apart from setting up kubernetes, nothing has been migrated yet. This will be next. The first step will be to get the databases off the AWS infrastucture first, which will be the biggest bang for buck as the RDS is costing around $34/day ($1,000/m) The second step will be the next biggest machine which is our Hajkey instance at Blåhaj zone, currently costing around $8/day ($240/m). Then the pictrs installation, and lemmy itself. And finally everything else will come off and we'll shut the AWS account down.
blahaj.zone
Discussion of the current situation with the Blåhaj instances, and upcoming maintenance.
Our lemmy is now running the 0.18.2 release version, which should fix some lingering issues we've been having. Let [@ada](https://lemmy.blahaj.zone/u/ada) or [myself](https://lemmy.blahaj.zone/u/supakaity) know if there's any issues!
Hi everyone, I'll begin migrating the lemmy blåhaj database to the new server this morning in about 20 minutes. Expected duration is about 1 hour for this migration. There will be a maintenance page up during the migration and I will be updating the status as we go to keep you updated on the process. Later today I'll also be upgrading the software to the latest release as well.
I have a process which copies emojos from the hajkey blahaj.zone site to our lemmy. Last night (for me) it copied an invalid emojo into the database and it broke lemmy for hours while Ada and I were sleeping. I've found the issue and fixed it now, so everything should be purring along once again, but for those who were left without the instance to use, I sincerely apologise. 
Hi everyone. Lemmy had an XSS vulnerability and we were one of the instances attacked through this vulnerability. We had our homepage replaced by a youtube video. The lemmy devs were quick to create a PR for this issue and we have patched our instance to fix it, Also while I do not believe any login tokens were compromised, we have taken the additional measure of rotating our secret key, so everyone will have to login to the site again as your existing credentials will no longer be valid. If login fails to work properly, you may have to clear your cookies for the site... it seems to get confused about whether you're logged in or not. Sorry that this happened, if you have any questions please contact [@ada@lemmy.blahaj.zone](https://lemmy.blahaj.zone/u/ada) or myself. 
All our Blåhaj zone Hajkey emojo's are now cloned here on Lemmy! 
Hey everyone, I'm your local server godess. I usually sit in the background doing things, while my lovely partner [@ada](https://lemmy.blahaj.zone/u/ada) does her thing moderating the server, but I do pop in every now and then to let you know when important things happen like the server gets an upgrade etc. So earlier today we were running a slightly old version of lemmy on the same hardware as the database server, which started causing problems as our instance has grown significantly over the last few days. So tonight we're running a freshly upgraded lemmy 0.17.4, moved onto its own dedicated server instance. Hopefully this should resolve the issues we were having and let me or Ada know if you notice any issues or glitches. Also Ada has relented and setup a [ko-fi donations page](https://ko-fi.com/blahajzone) for those that want to donate AND can afford it. We want to stress that we are running these instances to support our community, so the last thing we want is to add financial stress to the members of that community. If you're privileged enough to be able to help with a donation without causing yourself financial stress, you're welcome to help cover some of the running costs. But if you're not, please do not feel any obligation to do so as that would undermine the goal behind creating these spaces in the first place. We'd rather keep covering the costs ourselves than pull funds away from the members of our community that need it. Lots of love! Kaity (and Ada).
Hey all! The lemmy software running on Blåhaj Lemmy has been updated to the latest version v0.17.1, which should result in a few bugs disappearing, however I believe there's still a few known bugs remaining. Let me know if you notice any problems!
