Hello everyone!
It’s time for another long-overdue update on how Fedecan and our various sites are doing. It’s been just over two years since the great Reddit migration, and in that time we’ve made some solid progress:
- Registered Fedecan.ca as a non-profit
- Moved from OVH to our own hardware in a Vancouver datacenter
- Started collaborating with the team behind sh.itjust.works
- Launched Pixelfed.ca
- Launched Piefed.ca
Finances
Here’s a look at our bank balance since we began accepting donations:
We’re currently sitting at around $2,900, with a monthly burn of about $200, which gives us roughly a year of runway. We have some additional annual costs (like domain renewals and non-profit registration), but overall we run very lean.
Fedecan still owes:
- TruckBC: $1,980
- Shadow (me): $525
These were out-of-pocket hosting and non-profit registration costs from 2023/2024. It’d be great to get those covered, but we want to keep at least a year of operating expenses in reserve.
If you’re a regular user and value what we’re doing, please consider donating! We have multiple ways to donate, you can find the comparison and donation links on our website: https://fedecan.ca/en/donate
Sh.itjust.works
Nothing major to report here - we’ve all been a bit busy lately, but collaboration is continuing slowly behind the scenes.
Fediverse Growth
We’re seeing a healthy volume of posts and communities on lemmy.ca, surging with each Reddit drama:
Infrastructure
Our server is a Dell R7515 with an EPYC 7763, 1 TB ram and 4x 7.68tb nvme data disks, which is hosted in a datacenter in Vancouver, BC.
I spun up victoriametrics + victorialogs a few weeks ago and have been ingesting all of our data, giving us the ability to put together some nice grafana dashboards.
Everything is running great on the infrastructure side of things. Our server is barely working up a sweat and we shouldn’t have to worry about scaling for a long time.
Lemmy.ca still comprises almost all of our traffic:
Lemmy.ca
Our over provisioned stack is performing well, handling the occasional lemmy / lemmy-ui dropout:
Similarly the DB is mostly running out of ram:
Our object storage is slowly climbing as expected, but we’ve got several years of capacity to figure out a long term solution:
I’m also doing some limited analytics on our web logs. As expected, lemmy.world makes up the majority of our federation traffic:
One interesting thing to see from the user-agent data is the breakdown of traffic by the different mobile clients:
The “dart” UA is just a common web library, Thunder reports as this and I suspect other clients do too. If you’re a client developer, please set your user-agent!
Out of the alternative web clients we support, tesseract is the most popular although the overall traffic volume is still low:
We only store 7 days of logs but I’m hoping to get these pulled out into metrics soon, since it would be interesting to track which clients / interfaces people use over time.
Pixelfed.ca
Not much to say on this one, due to using local storage it currently runs on a single VM without redundancy.
Piefed.ca
Piefed runs on a pair of VMs with its own database and object storage backends.
Cloudflare
If you want to compare against previous data posts, here’s our same cloudflare graphs for lemmy.ca
As always, feel free to reach out if you have any questions or ideas. Thanks for being a part of the Fediverse!
You must log in or register to comment.
As requested, here’s some additional info about donation methods / sizes / volume!
Thanks, that’s quite informative. I take it the Stripe option is new. If those trends continue, it looks like things are currently financially secure, but not necessarily paying things back soon or building much of a reserve for replacement hardware. Hopefully the added awareness gets a few more donors. Given the low costs we’re seeing, even $10/year will make a notable difference.
No we launched with stripe, but if you look at our site you’ll see we don’t push it heavily since we pay more fees there.
PS: are the recurring ones per month or per year?
Monthly. There were a couple yearly that I just normalized down to monthly.
Thank you so much! If I understand correctly it’s around 80 donors, counting recurring and one-time together?
ish
Any thoughts on adding Liberapay as a donation provider? It would allow you to use Stripe as a recurring platform.
Thank you so much for your transparency and for encouraging this community. Will be happy to do my part by engaging in meaninful conversation, posting quality content, and will make a donation.
…and to be clear, try to minimize the arguments I find myself in but hey can’t promise 99.99 uptime on that one ;)
Finally got off my ass to donate. Thanks for all you’re doing!
How did we get so lucky as to get a competent team willing to put in this much work…thanks so much, really.
My thoughts exactly, thank you!
I have some questions 😄. What’s being used for operating systems and are these just connected straight to the internet with 1:1 NATs, or is there most balancing or firewalls involved?
What kind of virtualization technology is being used?
Are all these hosted on the same physical server?
Single physical host running proxmox. Mix of Ubuntu vms and lxc depending how much I trust the app and if I need direct access to the underlying zfs.
Opnsense + haproxy in front, nothing connected directly.
♥️ opnsense and zfs
I need to show people I work with how little hardware you can use to serve this much traffic. We do around 500/second but our monthly EC2 spend is horrendous for the scale of traffic we get. But rails is what it is…
I think you have it all covered, but are there ways people can contribute with time? I’m mostly k8s focused these days though.
Not really, the tech is stable and I honestly put very little time into it. Occasionally I get bored and do something like make dashboards, but ongoing maintenance is just the occasional software update.
I’d like to move it all to k8s at some point, but it’s a really low priority. Probably doesn’t make sense until we start scaling to multiple machines.
Thank you for keeping this instance alive, Also nice transparency very appreciated
Is there any regular offsite backup of data?
Yes, shipped to a server at my house.
That’s really cool that admins can tell which clients folks are using, that would be very helpful for developers to gauge user satisfaction.
I’m a sysadmin by day and… Only one host? And it needs 1TB of RAM? Seems excessive. Probably not though. IDK how optimized Lemmy is.
There’s a lot I could say here but I’ll limit myself to this: the proof is in the pudding. Lemmy.ca is working well. You all should be very proud of what you’ve done.
It doesn’t need it. That hardware is just what was available.
Two hosts would cost more in colo fees and we don’t need the resources. The redundancy would be nice, but this is enterprise grade hardware covered under Dell prosupport. I also have an identical box running my home systems, so I could steal parts temporarily if we had an issue.
Yeah the ram is just what the hosts came with. It’s way more than we need and so everything is very over provisioned.
All fair. I’m not trying to say you’re doing it wrong at all, quite the opposite.
And yes, redundancy is nice, but it really depends on the importance of the data on the system and the budget.
To be blunt: if Lemmy.ca goes down for any length of time, that would suck for everyone here, but nobody will die, there won’t be any loss of profits or whatever… In business talk, the risk of what could be lost due to an outage is less than the cost of the hardware to prevent an outage.
I understand your position and an in warranty Dell server system isn’t cheap.
What you’re currently going is clearly working. So I don’t have any complaints.
Request for info to be added to fundraising page: which fundraising payment options are processed by American companies. If they’re taking fees, I’d like those fees to go to Canadian firms.
Fuck it, sending an e-transfer. ;)
E: sent $100. The equivalent of two one-year reddit subscriptions. I cancelled my subscription to move over here. Good luck with the fundraising :)
There’s a reddit subscription? Whatever for??
Before the enshittification took root, reddit was where I interacted the most with people online. You could buy a subscription to get an ad-free experience. It came with a bunch of Reddit gold you could hand out. Basically, help pay for server time, and get an ad-free experience without adblocking.
The amount of time I spent on Reddit I would compare to a Netflix subscription or similar.
In the ramp up to their IPO when they started to fuck with everything, my goodwill evaporated. I stopped moderating and abandoned my subs. Whenever I had new content I wanted to post, I instead post to the fediverse. So, there ya go 😅
Thanks for taking the time to answer!
And wow, you mod a bunch of communities here. I just joined a few of them that piqued my interest! I’ve been meaning to learn how to play DnD with my kids.
I actually locked the D&D community on lemmy.ca that I created once the ttrpg.network D&D community took off. I recommend that one. I could reactive the lemmy.ca if someone went sideways with that instance.
I also recommend the lemmyverse.net website for finding communities to subscribe to. Copy and paste the community name – should look something like !dndnext@ttrpg.network – into the search box and then subscribe.
Also, try to subscribe to communities that aren’t on lemmy.world, even if one exists on lemmy.world. Let’s avoid putting all of our eggs in one basket ;)
Thanks again for the great advice!
Good idea, they all do except interac since everything routes through stripe. It’s absurd the amount of transaction data they get. Thanks!
Sent you something in addition to the recurring to help with that repaying. ☺️
Thank you! I got here at the great Reddit migration, all this time lemmy.ca has been extremely reliable. I’ve had a great time. Just want to say I appreciate the time, knowledge and effort you put in it.
