General Lemmy.Cafe
The upgrade has gone through smoothly and everything seems to be running smoothly. The performance looks to be better on the backend, time will tell if the memory leak issue is actually solved. So far, though - so good!
www.timeanddate.com
### Lemmy Cafe will be having its database upgraded. #### Reasons - Pict-rs is expecting PostgreSQL 16. It's running fine now, but it might not be at some future point. - PostgreSQL 15 has a bug that requires `jit` to be turned off - otherwise DB keeps consuming all the memory available on the system and then some. In the end it gets culled by the kernel. This has performance as well as reliability issues. While turning `jit` off has remedied the constant failure, it has also made the database a bit slower. I prefer squeezing out as much performance as possible :) #### Plan - Point `nginx` to the maintenance page - Shut down PostgreSQL 15 - Run the upgrade tool - Start up PostgreSQL 16 - Point `nginx` to lemmy #### Expected downtime About an hour, if things go well. More if not so. Will try to keep the maintenance page updated. Here's the [timezone converter](https://www.timeanddate.com/worldclock/converter.html?iso=20240202T100000&p1=1440&p2=tz_et&p3=tz_pt&p4=236)
The process went through smoothly. I have also used the opportunity to split up a singular lemmy container into individual tasks - this has enabled a seemless upgrade process with no downtime, bar a few process quirks I need to work out. There have been some federation fixes merged into this release, so the situation should definitely be improving overall! I will make a more detailed write up of the whole setup later on, other admins might find it useful. Or not.
May your families grow strong and your presents plentiful!
The upgrade has reset the JWT, so you'll have to login again on your device as well as set up 2FA again. Enjoy the [new features](/post/1847683)!
Pun intended 😁 Just wanted to reiterate, my stance hasn't changed since [last declaration](https://lemmy.cafe/post/112513). If facebook starts funny business with extra domains - those will be added asap.
It's been a long one, but finally the database is clean. In the end the cleanup took <20 minutes, but getting to it took me most of the weekend. Sorry about flaky performance, it should be back to normal.
Migrated email into a dedicated self-managed solution, but it was at the end of the month, haven't included it here. **Income** - LiberaPay: £11.41 ___ **Expenses** - Servers: £28.85 ___ **Month balance** - -£17.44 ___ **Previous balance** - -£44.66 ___ **Balance to date** - -£62.1 ::: spoiler Previous reports ___ [August 2023](https://lemmy.cafe/post/561304) ___ [July 2023](https://lemmy.cafe/post/561283) ___ [June 2023](https://lemmy.cafe/post/561205) :::
Minor update in prep for the biggin. Pictrs was also pushed up to 0.4.4
The main feature being a fix the really nasty libwebp vulnerability.
I've just received an email that has links to what looks like some polish url redirect site. I suspect this is limited to admins as, by default, lemmy does not come with an email service. Needless to say - don't click random links! Stay safe, peeps!
I'm chuffed to bits to have finally had enough time to set it up externally. Google is no longer outright rejecting us! And by externally I don't mean succumbing to the behemoths - it's all self managed! I've seen a post recently about someone struggling hosting Lemmy on DigitalOcean and sending out verification emails. If you need help with config or would like to simply use lemmy.cafe's setup - feel free to reach out!
First donations! Also migrated pictrs from file to object storage, but have not been invoiced, yet. I suspect it's the low amount (<$1) and will probably keep carrying over until some threshold is reached. **Income** - LiberaPay: £15.26 ___ **Expenses** - Servers: £24.75 ___ **Month balance** - -£9.49 ___ **Previous balance** - -£35.17 ___ **Balance to date** - -£44.66
Signed up on LiberaPay. Stripe is slow to activate the account. Lemmy storage was growing quite quickly on the small VPS nodes, had to upscale a bit. **Income** - None ___ **Expenses** - Servers: £19.18 ___ **Previous balance** - -£15.99 ___ **Balance to date** - -£35.17
Decided to finally do a write up financials. **Income** - None ___ **Expenses** - Domain: £8.46 - Servers: £7.53 ___ **Balance to date** - -£15.99
Sorry about it, it seems like after having nuked the CSAM incident community lemmy-ui has freaked out again about the site logo. Fixed it for now, will keep a close eye on the web UI. Apps were not affected.
Following the recent CSAM spam the upload size has been further reduced to 50kB. This might impact community/profile banners, in which case I propose getting in touch and we'll sort it out on a case by case basis.
Apologies for the downtime.
DigitalOcean has been experiencing a network outage and as such Lemmy Cafe is also affected. [Incident page](https://status.digitalocean.com/incidents/q4b09b022nmh) suggests it should be resolved now.
Current infrastructure costs are ~30 GBP a month. Not crazy expensive, but something that has potential to grow in the future. Expense breakdown: - Lemmy UI+backend: 14GBP/month - Lemmy DB: 14 GBP/month - Domain: ~2 GBP/month - Traffic is not reaching the limits at the moment, no extra cost [LiberaPay link](https://liberapay.com/Illecors/donate) It is also present in the sidebar.
Everything appears to have gone through quite well, although the ambiguous UI error during the database migration was confusing.
Storage was getting full, it was time to upgrade.
https://feddit.de/c/ich_iel and https://lemmy.cafe/c/ich_iel@feddit.de show completely different posts. Shouldn't both sites have the same posts?
This addresses the issue some instances had today with the XSS attack.
There has been a vulnerability discovered in Lemmy. I have no reason to believe lemmy.cafe itself has been breached. We do no have any custom emojis, which appears to have been the culprit of some XSS attack. As a safety precaution, however, I have applied the suggested fix and rotated the JWT token, which will have invalidated everyone's session.
It looks like several Lemmy instances were exploited last night. It doesn't appear that much damage has been done, but users may have to [log out and back in if they use mobile apps](https://lemmy.cafe/post/132158). [@Illecors@lemmy.cafe](https://lemmy.cafe/u/Illecors) have you found any indication that lemmy.cafe may have been targeted too?
Right now there appears to be a bug where if `english` is the only language selected in the instance settings, Jerboa will never succeed posting, commenting, messaging, etc. Adding `undetermined` back to the list has fixes it.
Backend 0.18.1-rc.10 UI: 0.18.1-rc.11
I have added threads to the blocklist. It does not show up on [instance list](/instances), yet as I believe the server is simply unreachable due to disabled federation on Threads' end. As for the reason - I don't feel comfortable providing data to Meta. At the moment Lemmy software is very trusting and every instance syncs quite a bit of data about users from other instances.
Due to `lemmy.world` DoSing us whenever their servers decide it's our turn to get all their data I started looking into ways to create headroom for `Lemmy Cafe` without increasing the bill as under normal circumstances there are plenty of resources. This has lead me to the blocklist that blackholes all kinds of scrapers and known bad actors.
This is a new domain on an IP that was fairly recently assigned to it. All main email providers blacklist anything that is not gmail, outlook or aws by default. Nothing I can do about it, other than becoming part of the problem and paying them to host this instance's email.
Both backend and frontend. Also updated pict-rs to 0.4.0-rc.10
It has proven to just be spam
General Lemmy.Cafe
!general@lemmy.cafeGeneral site