Here’s a few things that are happening that I feel should be addressed:
-196 is reopening in reddit
-our community is causing excessive load on the blahaj.zone servers
-i finally figured out how to add newlines in Lemmy markdown
Two of these things are more important than the third but I figured they should all be addressed.
First things first, the subreddit. Even though the og subreddit is reopening now, I do not plan to close the community. At this point 196 on Lemmy and 196 on Reddit are both separate yet similar communities and this lemmy community has a sizable enough user base and is active enough I feel closing it would be stupid.
Speaking of a sizable user base, the blahaj.zone servers are under excessive load which has been causing the admins to incur fees upwards of $3,000 a month. This is in no small part due to 196 being the fourth largest community on Lemmy.
I’m not sure what the future solution of this will be. It seems like they’re going to switch to a cheaper service provider which will also cause the entire website to be slower, at a lower cost.
For this reason, I think that a future migration to a different Lemmy server may be in order. I will be reaching out to major admins of other communities as well as our own ada to see if the pre-existing community can be transferred, or if we’ll have to start from scratch if we choose to move. This is very much a developing situation, and I will keep you all posted.
Finally, as you can see. I finally figured out how to add newlines to plaintext meaning that all of my future postings will be much more professional looking. Check out our new and improved rules and regulations pinned post to see just how much of a difference it makes.
You guys are the community, so I’d like to see what you think. Please discuss in the comments section of this post, and I will try to base my actions off of whatever you all conclude on.
Also please point out any typos I may have made I voice typed this on my phone
There’s no way $3k/month is an accurate number to run a single community. Lemmy.world (and mastodon.world etc) combined takes ~1k/month to run, and they’re far larger and more active.
Also, anyone who goes back to Reddit doesn’t belong with the 196 community. No bootlickers allowed.
To be clear, $3k is an accurate, but unacceptable amount.
As in that’s what it’s actually costing us, but it’s not what it should be costing. I’d imagine more like $250 is what we should be paying if I wasn’t using AWS in the silly way I am.
I’m admitting up front that I’ve been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.
Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.
And so I didn’t notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that’s what got me.
About half that $3k is just data transfer costs.
Anyhow the notice was just to let our users know what is going on and that there’ll be some maintenance windows in their future so it doesn’t surprise anyone.
We have a plan and it will all work out.
Don’t panic or have any kneejerk reactions, it’s just an FYI.
Thanks for the clear and concise update. Best of luck with your future migrations.
They’re paying for Amazon web services instead of using more professional hosting alternatives, and things are nowhere near as optimized as lemmy.world. also they are hosting a Mastodon instance in addition to the Lemmy instance. also I’m pretty sure that ruud has dedicated hardware for lemmy.world, whereas this is being done entirely off the cloud.
It really seems like that should be fixed before blaming 196 for their infrastructure. If they don’t know how to fix it then yeah it’s probably best to move 196 away so they don’t bleed out.
Just want to say, I don’t blame anyone else but myself.
I certainly don’t blame anyone at 196.
I hope I’m really clear about that. It’s one of the reasons I specifically didn’t name 196 in my announcement.
We’ve got a solution planned, we’ve already started to implement it and have the image transfer issue solved already.
We can afford to cover this ridiculous AWS bill, I just need to do some maintenance work so this doesn’t continue because I can’t continue to line Jeff Bezos’ pockets like this indefinitely.
Thank you 🙏
Yeah
They’re paying for Amazon web services instead of using more professional hosting alternatives, and things are nowhere near as optimized as lemmy.world. also I’m pretty sure that ruud has dedicated hardware for lemmy.world, whereas this is being done entirely off the cloud. in addition to all of that, they’re hosting a mastodon instance on the same website and the costs are for both
This was a throw away instance of two or three people before the Reddit wave. We were not planning on growing from three users to six thousand :)
Y’all are champs for suddenly taking on thousands of users at your own expense on what was meant to be a throwaway instance. It went from a small private server to hosting one of the most active communities on Lemmy in a short time, and you and your team of volunteer tech sorceresses have been doing the best you can to keep lemmy.blahaj.zone alive as it grows
I’m running my Lemmy instance in AWS as well for just a few people and trying to keep costs low. I can definitely imagine it’s rough trying to scale up and control costs.
are you planning on shifting to other vps providers?
We have already begun moving, and this announcement was to warn our users about future downtime as we migrate more critical services like databases etc.
ya know, it’s not hard to setup a totally independent instance on the same hypervisor as another.
I have servers, you have people, we could make independent lemmy instances on my two Dell R720xd and one Dell R730xd servers. They’re not overloaded with people like lemmy.world yet, that would put way way more stress on that server.
The cloud do be expensive
@yote_zip@pawb.social to be clear, $3k is an accurate, but unacceptable amount.
As in that’s what it’s actually costing us, but it’s not what it should be costing. I’d imagine more like $250 is what we should be paying if I wasn’t using AWS in the silly way I am.
I’m admitting up front that I’ve been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.
Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.
And so I didn’t notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that’s what got me.
About half that $3k is just data transfer costs.
Anyhow the notice was just to let our users know what is going on and that there’ll be some maintenance windows in their future so it doesn’t surprise anyone.
We have a plan and it will all work out.
Don’t panic or have any kneejerk reactions, it’s just an FYI.