FA+
Shop
April 30th, 2025- We're back! Kind of.
4 months ago
🏳️🌈💖Enjoy the site? Please consider supporting us via the links below!💖🏳️🌈
⭐ FA+ ⭐ SHOP ⭐ KO-FI ⭐
Journal Start
Welcome back to Fur Affinity! You’ll notice that we are in a version of read-only mode with the following limitations:
🚫 Uploads are currently disabled
🚫 Avatar changes and uploads are disabled
🚫 Profile banners cannot be edited
🚫 Submissions cannot be edited/replaced or deleted
🚫 New Account Creation has been disabled
A lot has happened in the last two days. Our original maintenance window was due to a sharp increase in storage usage for uploaded works that required us to expand our storage capacity on short notice. We are extremely happy so many of you are using Fur Affinity, and this took us a bit by surprise how quickly this happened!
However, once we brought the website back online we noticed missing images and gallery errors on uploads. Out of a concern for potential data loss or corruption, we took the site offline again to investigate and validate our systems. Upon further inspection, we identified the root cause and began the process of repairing our user uploads system.
You can read more about the technicals and timeline further below!
This is our current timeline of tasks for restoration at the moment:
⚆ Finish copying User Uploads to the newly mounted filesystem (We are here)
⚆ Verify that the content copied is correct and uncorrupted using hash comparison
⚆ Cleanup debugging tools and remount everything to their original states, which will require a properly brief downtime
⚆ Once everything is mounted and tested, we will bring Fur Affinity back online to full working order
As for the reason for our downtime, below is a technical recap of what happened to get us to this point, our process for troubleshooting, and our takeaways straight from our Tech Team:
After we took Fur Affinity offline a second time, we began investigating what went wrong in our planned maintenance.
Almost immediately, we began noticing issues with our filesystem that houses user uploads. We mount our user uploads to the website using a protocol called NFS, or Network File Share, which allows the website to access content hosted by another server, in this case it allows our Website Server to communicate with our Data Storage Server. When we attempted to investigate the seemingly inaccessible content on the Website Server, we were greeted with “Stale NFS Pointer” errors. Not good. We logged into the Data Storage Server to verify the content was still there, which in fact it was. To put it simply, while the User Uploads were present in the Data Storage Server, the Website Server was unable to access them.
This began an almost 15 hour discord call between our tech triage team, management, and constant debugging and research from our infrastructure and development teams. What we found was:
⚆ Checking NFS client logs and lock logs on both servers yielded no errors
⚆ Checking Kernel and Syslog messages on both servers yielded no errors
⚆ Files that existed before we expanded the Storage Server Capacity were not corrupted; however, at random they were inaccessible by the Website Server.
⚆ Accessing files directly from the Storage Server that were inaccessible by the Website Server showed the data was valid and intact.
⚆ Unmounting and remounting the NFS Share yielded no changes in the behavior.
⚆ Mounting the Storage Server NFS Share to a different server yielded no change in behavior, which narrowed the search down to just the Storage Server.
⚆ File System Verification Checks on both the Website Server and Storage Server came back positive, with the exception of the partition that housed User Uploads, which showed an incorrect and negative count of inodes
The mismatch of inodes, especially with the count showing a negative number, lead us to investigate the filesystem expansion process further, which led us to believe that an errant bug in the filesystem resizing process during Monday’s maintenance pushed the inode count on the newly resized file system beyond the maximum number of inodes a file system is allowed to have, overflowing it into the negative numbers, as reported by `df -i`, leading to random errant behavior when trying to allocate space for new data.
This mismatch can manifest itself in many different ways: data instability, ghosting and, in our case, creating accessor mismatches for protocols that rely on healthy INode data like NFS.
After 24 hours we had our culprit, and weighing all our options we decided to begin the process of organizing our storage cluster resources to create a new data partition that was properly sized and contained enough storage to copy the valid files out of the unstable filesystem. Fur Affinity’s Data Storage Partition is not small, so this operation has the potential to take a long time.
Which is where we are right now. Fur Affinity user uploads are currently being rehomed in the newly created data partition, which is not a quick process, and due to the length of this procedure we have decided to open FurAffinity in “File Read Only Mode” which allows for everything except User Uploads, Avatar Changes, Profile Banner Changes, and anything that would utilize the User Upload Partition (see the list at the start of this journal for an exhaustive list of unavailable features at this time). At the moment we have an estimate that this may take up to a week to resolve.
We know this is not ideal for the majority of users. We are hoping at least having the site accessible for everything else will help in regards to artists doing business as well as keeping communication flowing.
Throughout this process we have identified several takeaways that the tech team will be processing and digesting to improve our internal reporting of resource usage rate, as well as our processes for handling complex operations, such as resizing, in the future.
We will be making a regular maintenance schedule moving forward in addition to improving our internal reporting and metric tracking so that you can all be aware when maintenance is coming up again with more than 24 hours notice and more robust and meaningful updates if issues escalate, like we are doing with this update journal!
This won't cover all downtime needs to be clear. There may still be emergencies that come up that require unscheduled downtime. But we want to work with the community as much as possible with things we can predict so you stay informed and prepared!
We are all very tired but happy to know there is a light at the end of the long tunnel here. We know this didn’t go well and was completely unexpected on our end too but let’s try to keep criticism constructive.
Thank you for your patience and sticking with us through this!
**EDIT**
If you experienced upload issues between the first offline maintenance and the second, you may have issues loading images in your gallery. Don't panic! The files are still there, and we can restore it for you after our migration is completed. Please open a trouble ticket to let us know!
🚫 Uploads are currently disabled
🚫 Avatar changes and uploads are disabled
🚫 Profile banners cannot be edited
🚫 Submissions cannot be edited/replaced or deleted
🚫 New Account Creation has been disabled
A lot has happened in the last two days. Our original maintenance window was due to a sharp increase in storage usage for uploaded works that required us to expand our storage capacity on short notice. We are extremely happy so many of you are using Fur Affinity, and this took us a bit by surprise how quickly this happened!
However, once we brought the website back online we noticed missing images and gallery errors on uploads. Out of a concern for potential data loss or corruption, we took the site offline again to investigate and validate our systems. Upon further inspection, we identified the root cause and began the process of repairing our user uploads system.
You can read more about the technicals and timeline further below!
This is our current timeline of tasks for restoration at the moment:
⚆ Finish copying User Uploads to the newly mounted filesystem (We are here)
⚆ Verify that the content copied is correct and uncorrupted using hash comparison
⚆ Cleanup debugging tools and remount everything to their original states, which will require a properly brief downtime
⚆ Once everything is mounted and tested, we will bring Fur Affinity back online to full working order
⚠️At the moment, we do not have an ETA on this timeline’s completion, we are actively working on it as fast as we can.⚠️As for the reason for our downtime, below is a technical recap of what happened to get us to this point, our process for troubleshooting, and our takeaways straight from our Tech Team:
(art by 
artdecade)
artdecadeAfter we took Fur Affinity offline a second time, we began investigating what went wrong in our planned maintenance.
Almost immediately, we began noticing issues with our filesystem that houses user uploads. We mount our user uploads to the website using a protocol called NFS, or Network File Share, which allows the website to access content hosted by another server, in this case it allows our Website Server to communicate with our Data Storage Server. When we attempted to investigate the seemingly inaccessible content on the Website Server, we were greeted with “Stale NFS Pointer” errors. Not good. We logged into the Data Storage Server to verify the content was still there, which in fact it was. To put it simply, while the User Uploads were present in the Data Storage Server, the Website Server was unable to access them.
This began an almost 15 hour discord call between our tech triage team, management, and constant debugging and research from our infrastructure and development teams. What we found was:
⚆ Checking NFS client logs and lock logs on both servers yielded no errors
⚆ Checking Kernel and Syslog messages on both servers yielded no errors
⚆ Files that existed before we expanded the Storage Server Capacity were not corrupted; however, at random they were inaccessible by the Website Server.
⚆ Accessing files directly from the Storage Server that were inaccessible by the Website Server showed the data was valid and intact.
⚆ Unmounting and remounting the NFS Share yielded no changes in the behavior.
⚆ Mounting the Storage Server NFS Share to a different server yielded no change in behavior, which narrowed the search down to just the Storage Server.
⚆ File System Verification Checks on both the Website Server and Storage Server came back positive, with the exception of the partition that housed User Uploads, which showed an incorrect and negative count of inodes
The mismatch of inodes, especially with the count showing a negative number, lead us to investigate the filesystem expansion process further, which led us to believe that an errant bug in the filesystem resizing process during Monday’s maintenance pushed the inode count on the newly resized file system beyond the maximum number of inodes a file system is allowed to have, overflowing it into the negative numbers, as reported by `df -i`, leading to random errant behavior when trying to allocate space for new data.
This mismatch can manifest itself in many different ways: data instability, ghosting and, in our case, creating accessor mismatches for protocols that rely on healthy INode data like NFS.
After 24 hours we had our culprit, and weighing all our options we decided to begin the process of organizing our storage cluster resources to create a new data partition that was properly sized and contained enough storage to copy the valid files out of the unstable filesystem. Fur Affinity’s Data Storage Partition is not small, so this operation has the potential to take a long time.
Which is where we are right now. Fur Affinity user uploads are currently being rehomed in the newly created data partition, which is not a quick process, and due to the length of this procedure we have decided to open FurAffinity in “File Read Only Mode” which allows for everything except User Uploads, Avatar Changes, Profile Banner Changes, and anything that would utilize the User Upload Partition (see the list at the start of this journal for an exhaustive list of unavailable features at this time). At the moment we have an estimate that this may take up to a week to resolve.
We know this is not ideal for the majority of users. We are hoping at least having the site accessible for everything else will help in regards to artists doing business as well as keeping communication flowing.
Throughout this process we have identified several takeaways that the tech team will be processing and digesting to improve our internal reporting of resource usage rate, as well as our processes for handling complex operations, such as resizing, in the future.
⚠️There are no current concerns about storage or gallery corruption issues at this time, and should any further issues be discovered we have a full separate backup available to restore from that was taken roughly 5 hours before the outage just in case.⚠️
We will be making a regular maintenance schedule moving forward in addition to improving our internal reporting and metric tracking so that you can all be aware when maintenance is coming up again with more than 24 hours notice and more robust and meaningful updates if issues escalate, like we are doing with this update journal!
This won't cover all downtime needs to be clear. There may still be emergencies that come up that require unscheduled downtime. But we want to work with the community as much as possible with things we can predict so you stay informed and prepared!
We are all very tired but happy to know there is a light at the end of the long tunnel here. We know this didn’t go well and was completely unexpected on our end too but let’s try to keep criticism constructive.
Thank you for your patience and sticking with us through this!
**EDIT**
If you experienced upload issues between the first offline maintenance and the second, you may have issues loading images in your gallery. Don't panic! The files are still there, and we can restore it for you after our migration is completed. Please open a trouble ticket to let us know!
It was there for 3 days.
I always wonder what happened to that person and how many friends they made.
but yeah, still retarded
these things they mean change over time.
retarded is not typically used to insult people born with intellectual disabilities - if you actually are doing that shame on you. instead, retarded is typically meant as an insult to someone who should know better and is willfully being a fucking dumbass. (or just a bad situation in general.)
either way, its not hurting anyone to say it (except those who deserved it.)
This is Ferocious-Ferals' mom. Remember to change your underwear every day!
i have some emergency ychs on my page, cause my kitten needs a good medical treatment :3
check them out!
https://www.furaffinity.net/view/60324963/
It was there for 3 days.
I always wonder what happened to that person and how many friends they made.
https://www.furaffinity.net/view/60583232
Alright.
Thanks for your hard work. ^w^
That means he just fainted.
Besides, the most compatible Pokemon is Ditto: Ditto can be ANYTHING AND EVERYTHING, after all!
There are a bunch of those she'd quote. :D
My brother Billie had a three-foot -
Actually, I'd better not. It's rude.
Or seriously massive amounts of Red Bull and Monster intravenously used along with numerous boxes of Pizza and Dunkin Donuts brought in?
Thank you all so very very very very much for your hard work!
i saw the negative replies on twitter and i was honestly shocked by how toxic and disrespectful people were towards the hardworking staff. I originally had the impression that many furries worked in tech-adjacent fields and expected people to have some level of understanding for the fact that not every problem can be solved in an hour with a snap of ones fingers!
Thanks for all your hard work 😃👍
Hope I wont get a free trojan alongside it ;w;
Thanks!
It's never DNS until it's DNS.
It's never inodes until it's inodes.
Nice catch, and well done tech team. Potential file system corruption is never a fun thing to deal with, especially over a network.
Uuuuuugh 💀😤
I'm glad it's possible to bring back the site at least to this state, and waiting (patiently ofc xD) for any new updates and announcements
Y'all definitely deserve some rest now o7
Their death shall be avenged.
(Or is it "the Sons of Warvan"?)
And yes, NFS cause lot of troubles :/ I run into the issue with inode number on a 32bit app (Steam) over a large ZFS/NFS storage myself, but the patch was never merged. Note me if you want to know details.
May the rebuild go as smoothly as it possibly can. ^^
Anyway object stores like s3 are just a LOT more robust for this kind of use case because of the filesystem abstraction and nonblocking operations. NFS still has its uses of course, but I would not use it in anything of this scale.
A problem you could anticipate with object stores is rw speed, but that can (and should) be resolved with caching in something like redis/valkey. I remember looking at seaweedfs at some point bc it's faster than other object stores, but I think it's best to avoid introducing even more software into the stack.
As a DevOps/SRE engineer with over 15 years of experience, primarily focused on high-load projects, I personally wouldn’t recommend using distributed file systems (NFS/Ceph/etc) for workloads as heavy as this. Not only do they affect performance, but they also introduce single points of failure (SPOFs). These systems are difficult to maintain without downtime, and any issues can lead to days of recovery.
In the long term, I would recommend considering a switch to separate storage systems with data partitioned across them. While this is less convenient from an application standpoint, it significantly improves reliability. Each storage system would manage its own partition of the data. Since they are independent, any issues would be isolated to a single partition—and even that risk can be mitigated by having two copies of each partition.
This approach would simplify the storage service while improving both reliability and performance.
I know changes like that are a big investment, but I wanted to share the idea in case it might be helpful down the line.
Please feel free to reach out if I can help in any way. Thank you for your hard work on FA.
I hope the file-copying goes well!
Heh for real though thanks for your hard work!
Hope everything goes well and you manage to get the site to work properly without any more errors <3
I'm upset that their are people who don't take the time to read this and throw shade on FA for no reason.
And glad that at least read-only is possible while that transfer happens.
Thank you very much
Thank you for your hard work!
you do an amzing work 🔥💪
(It's a single donut that's large enough to take up the entire dozen-sized box. Death by carbohydrates!)
Don't forget to take care of yourself, please
The effort is well appreciated ( ˶ˆ꒳ˆ˵ )
I just wanted to say thank you for being transparent and for working so diligently to get this resolved!
Strength and patience in solving these problems!
Looking forward to further updates ~
Thank you to our tech guys for working their butts off while us mods could only cheer them from the sidelines.
*plays Venga Boys "We Like to Party"*
They really work, so long as only one is taken per hour, with a tall glass of water and a positive look on life. ;)
It's truely a miracle cure!!!
Does make me wonder just how big the total file size is for FA. Probably a good few hundred terabytes, maybe even a couple petabytes?
A small waffle party will commence shortly after
Only around 12 were lost
Also thank you for sharing the details about what exactly is going on, too.
You got this!
Read-only mode by itself helps a lot and definitely makes the wait more bearable; this seems like one of those "overdue" accidents because the site is so old and hasn't had many updates in the background, might be painful but slowly picking a it will help the site thrive for many years to come, hoping everything goes right for the team's sanity and make sure to rest too! :3
Maybe the character is called Exty. It's a cute name right?
Thanks so much for keeping the site going. <3
I have programming skills and I have a desire to help.
Congrats on figuring out the issue and starting a safe procedure path to restore things, and making a clear and cut post about what happened, especially including more technical stuff.
I know some folks got real mad at y'all like you could have known such an oddball issue would arise, so also cheers for keeping cool and all.
Thank you all on the tech team, on the community relation team, and everyone else at FA, I know it ain't easy when stuff goes downhill, between the troubleshooting and community management!
Again really nice to see some transparent reporting done like this, and having some updates along the way on other places while stuff's offline. I was one of those that were very critical of FA's direction not so long ago, so I am real happy to see all the positive work that's been done, despite the tragedy.
Cheers!
Keep up the good work!
Wishing you best of luck with getting this sorted out. May everything go well!
It's hard to say that any one thing is the heart of the Furry world, but FA has to be really close. The Pancreas of the Furry World? :D
Take the time you need to fix this because we would rather have a well thought out fix rather than a half assed rushed one. The updates and transparency is really appreciated!
You guys got this!
Thanks for the hard work.
Keep up the excellent work tech team :D
Good hygiene is beneficial to your general health and wellbeing! 🥰
I'll be off to the gas station then.
I know a lot of stuff, but those are the most FS related things I could think of off the top of my head, lol. Breaking things is great for learning, but hella stressful if that thing is "production". We've all done it, it's a rite of passage. :P
But thank you very much for all the effort done so far.
This is nothing compared to some of the "outages" we've had back in the day and furries should be grateful.
Let's remember FA, for all it has become, is still a community run site--driven by passionate members of our fandom willing to dedicate their skills and time. You guys are awesome, thank you for the hard work. <3
I'm proud. Thank you and the rest of the crew.
Is there another file system that'd be better to run on?
Is there a better way to structure the data besides one enormous partition/drive? (There's gotta be SOMETHING in big-data/storage stuff, right?)
You got this, guys!
As Long as we're able to see Pictures I Don't mind!!
Excellent crisis management! 👍👌💝
Good breakdown of the cause as well.
Always some gremlins in the systems, taking the time to fix it right is always best.
Y'all got this! Sending sweet cold coffee.
Will be looked into.
I suppose in the words of Siegmeyer of Catarina from Dark Souls: "The thing just won't budge. No matter how long I wait. And, oh, have I waited! So, here I sit, in quite a pickle."
Still read-only... still read-only...
This whole fiasco happened due to a sudden uptick in content uploaded onto FA and server storage needing to be expanded to compensate, which caused huge issues with the internal file system.
That's just what happens when you have an archive of a crap ton of art from over two decades' time. There's a reason there's file upload size and rate limits--those just add up way quicker than accounts with nothing on them.
At least, that's my theory--I'm not on the staff so I can't definitely say whether that's the case or not.
Thank you for bringing the site back up for communication purposes.
...
Sort of lol!
I wish you all the best of luck! Thank you for your hard labor!
I got really worried when you announced that it would take several hours to make an info post about this, I assumed the worst! Don't say things like that without saying that it's fine and that you're gonna put it into read only mode!! XDD
Thank you for putting in the time and effort to fix this stuff as quickly as you are, everyone involved deserves all the hugs and naps and coffee and snacks and whatever else might help <3
I did notice that some submissions show the "Submission Not Found" box if you click Download. Out of curiosity I tried Sciggles most recent five or six. They are not found. I'm unsure how far back
this goes.
However, the ones currently on the front page are working normally. So will some images need to be rebuilt from the backup, the whole shebang, or are these currently being resized that are throwing this error? It makes me hesitant to check my own gallery and scraps, and feel for users with thousands to check, if any have this error.
Lastly, are notes useable to create and send?
I'm not in tech, but would that mean it's affecting how an account is tied to the server? The thumbnails are fine, and I think I remember way back (which may no longer be the case) reading that the thumbnail generator was on a separate server, so I'm hoping if it's generating thumbnails that the data is intact but not accessible.
Is there a badge for that? XD
In seriousness, I hope it helps you all!
Happy we are here now!
Thank you for your hard work! ❤️❤️❤️
(what i think i think) is it sounds like when they moved to the bigger file housing system, something funky but small got renamed (or rather, renumbered) and because computers are computers they just try to run the wrong number and error out.
so like, the hard drive with all the art and the hard drive running the website couldn't talk to eachother after this update had that bug. now they know the bug it seeeeems like they've got to move everything and then i would think run the update again this time with special awareness to the potential of a bug? or maybe that's not even necessary, maybe it's just a long drawn out process of havin to move files so the Wrong number is now a Right number.
what i DO know is that a 15 hour bug hunt meant the symptom (two drives not talking to eachother) was a broad enough one that it was like trying to treat someone when they walk into the dr going "i feel sick :(" and no outward symptoms beyond that really vague report. so like a doctor, i guess they had to run a barrage of tests.
YAY!
Get some rest 🙇
https://www.furaffinity.net/fa_offline.webp?v=5
Also, 15h discord tech triage call sounds hellish, o o f
Thank you very much for the update!
Take your snack breaks, get your stretches in, and get proper rest y’all!
You’re our team and we appreciate the hard work you do!
Disclaimer: I'm an app engineer, not an OS admin
That said, for scheduled downtime, can they be announced on site like this? I'm assuming it was on either bluesky or the discord or something, but it would be appreciated for those of us who follow just here. ^_^;;
Thanks for the update! And now go sleep, you deserve it!
Hey, I know it's not much, but I changed to using jpg instead of PNG. (Mostly did this so it could load before my audience lose interest and swipe away.)
And sorry, I ate a couple lines of code because I was hungry and couldn't find anything else
I didn't know they were so important!
..If I had known I'd have eaten something less important, like the text formatting code or something!
Thanks to all the FA team. I don't usually post in the news section. I feel weird. Lol.
Thanks for all the hard work and info.
We do take every report seriously and try our best to address everything in a timely manner, as fairly as possible.
Nearly thinking real human content will be rare thing in future
so happy to see damn real art
Honestly, keep it up, team!
Thank you so much for all your hard work! You're the best! ❤️ 💖 ❣️ 💕 💘you got this <3
it's also refreshing to have insights about the technical details. these stale NFS handle errors were haunting me at my previous job as well... (at least until we finally changed our apps to not make use of a shared NFS volume, but use ceph's radosgw instead. ah, the memories :D)
Just curious, is all of FA independently hosted in a datacenter, or is it sitting on a cloud system like AWS? Does it have its own SAN system or is it just a handful of storage servers? I haven't been keeping up with the tech journals.
Thank you team FA!!
Dragon artist here, feel free to follow me if you like feral dragon art (inspired by the Legend of Spyro art style) See you :>
Great work, staff!
Bu they, do what you've got to do. Thanks for doing all you do and best wishes in the tribulations to come.
It sounds like one heck of an ordeal but I'm positive you're doing a very good job here. Keep it up, we'll cheer you on!
▕╯┃╭╮┃┃╭╮┃╰▕╯╭▏
▕╭┻┻┻┛┗┻┻┛ ╰▏
▕╰━━━┓┈┈┈╭╮▕╭╮▏
▕╭╮╰┳┳┳┳╯╰╯▕╰╯▏
▕╰╯┈┗┛┗┛┈╭╮▕╮┈▏SpongeBob
💖 to the staff
Thank you for all of your hard work! This must be quite tiresome for you guys, but our gratitude towards you guys CANNOT be understated!
I was thinking it might be hundreds of terabytes. Thanks for letting me know, it's interesting stuff :3
Kinda why camera photos take up more room than most anything else on your phone. :3
I assumed 20 terabytes was all the art and images on Fur Affinity.
I don't understand any of that tech talk but good luck! And thank you for the transparency!
by the way, Fender destroying the pc is very funny.
that's how i feel sometimes when everything fails 🤣
Also, I most definitely misread "tech triage team" as "tech triangle team," so...yall are a bunch of good triangles. Thank you for your pointiness on this issue! <3
These are uncommon and unfortunate events. Those who are able to get some benefit from it? We should be cheering them on. It was literally luck of the draw that they got there before the site went down. It isn't personal. I would personally prefer the site be online as it is currently so that artists and commissioners CAN still communicate, post journals, comments, etc. We are doing the best we can with the resources we have.
I salute and applaud the hard work you all are doing!
o7
I've seen a lot of whining about little issues with the site, but this one isn't some huge conglomerate like Google or Meta. And the way you handle the community shows it, because you actually act like PEOPLE.
"Ringtail O'Reily says she will roll a pair of 23-sider dice in appreciation, and in your honor."
I'm so happy and we are all so happy that you are keeping us all updated. Thank you FA staff for all of your hard work. Best of luck!
My raichu pokesona, Iona, might be to blame for any chewed wires....again....
I wish y’all the best of luck and thanks for updating us on it all!
I'm just glad FA's admin staff are quickly working at resolving the technical issues. :3
It's a relief to know there isn't any known data corruption at the moment, and that the tech team is currently validating data integrity!
Hoping it all works out fine and by next week the website fully recovers.
Wishing the whole team the best!
Lil note for when it's all done:
Make sure to let people know once everything is back that uploads are rate limited so the website doesn't die from a huge amount of uploads in one go, some people can get easily scared when things seemingly don't work.
Thanks for your work!
PS. Might want to remove the Twitter/X link from the "FA Is Temporarily Down" page if you're not using it anymore. I went there first and there was no current information to be found.
Thanks!!
And thank you for your hard work. All of you.
Don't you just love when computers decide to just not cooperate
Is your icon a GTF Hercules, perchance? :x
The simple fact that someone else here knows of FreeSpace is just <3 to me. :D
Also, it's best for artists to back up their own work and put it into other places, should the FurAffinity's website be down for quite some time (It may not happen for that length of time but it's best to be prepared than wait until too little too late). And yes, this also includes backing up DMs you've had over the past 10 to 20 years or so. Again, it's just a personal archive that you can do easily, but it does take time, especially for those who have huge amount of uploads on here, having to take the time to sort/organize them into people's HDD/SSD folders on PC, Mac, or Linux, and then go through the DMs and save/copy them into a separate folder.
Regardless, keep up the great work FurAffinity. = )
But either way, I agree with those that suggested an S3-compatible storage cluster: If you’re at the stage where you can blow out the inodes, you probably shouldn’t be hosting content via normal filesystem I/O or NFS anymore. (Honestly, I’m surprised the site has been using NFS this long: I have horror stories going back to the ‘90s with it.) S3 has its issues, but if it’s good enough for Amazon to host a hundred billion product listings on, it’ll probably hold up under the load of a few hundred thousand furries. Probably 😅
Either way, props to you guys for the hard work, and props for an actual postmortem with substance to it: Those are all too rare in the tech industry these days, and it’s refreshing to see one again.
1) AWS Acceptable Use Policy (and lack of control in general)
AWS would have a wide range of options to deem anything containing fetish content, plain porn, etc. inappropriate, it gives NO insight into internal moderation, and it can terminate accounts without warning for any perceived violation of their product stack, and sometimes due to outside and political pressure.
2) Bad Actors
FA has been subjected to loads of attacks over the decades, even if stuff like DDNS attacks seem to be happening less often these days or are negated well, but AWS would probably not take kindly to the potential risk. Then there's the potential of mass-reporting or even just someone mass-crawling images to trigger egress payments. AWS isn't a safe haven, it's another potential attack vector.
3) Cost
S3 is cheap if you work with large, slow files in a very predictable manner, so there's not a big chance of egress. They charge per operation (GET, PUT, etc.) and bandwidth out is not free, so you'd have to use CloudFront or very aggressive caching.
Cloud services can have benefits, especially when it comes to storage, access times, etc. - it's easy to forget there are clear downsides as well.
That would mean a whole new restructuring of it entirely, or figuring out what went wrong. We already know. If it was errantly allocating more space... Hmm. Well, you couldn't access them; Looks like The Data Structuring was incorrect, or faulty, being random... Randomness is unknowable, so that explains it...
That'll take a LOT of trial & error. Hopefully everything will be fixed A-OK. Data can be repaired, after all, it's like a machine, or a clock; The pieces have to be in The Right Places! Code, is a processor of data; It handles what can be read. You just gotta repair it like A Clock!
Hopefully you can get it repaired! Good luck!
Remember to take breaks and stay hydrated while doing your job. And don't feel guilty to play some games or watch Youtube in between sessions of work to avoid giving in to insanity.
Have a good week, and thanks for everything! ❤️
Hail spirit of the machine, essence divine,
in your code and circuitry the stars align.
By the Omnissiah's will we commune and bind,
with sacred oils and chants your grace we find.
Blessed be the gears, in perfect sync they turn,
blessed be the sparks, in holy fire they burn.
Through rites arcane, your wisdom we discern,
in your hallowed core the sacred mysteries yearn.
Not to mention that PNGs are lossless, unlike JPGs. ;3
Thanks for the suggestion, still! <3
I don't want to criticize anyone with this, but human never know what can happen in the future. Who knows, maybe another bug will appear in the system, after which all the effort your team put into fixing the previous one will be wasted and you're will have to start all over again looking for where the error occurred. So I wouldn't praise dinner before lunch.
We all love you ❤️
1. They updated their partition to be larger to allow for more file storage
2. Something started going wrong
3. They kind of went "oh shit, gonna stop this before it becomes a problem", and shut everything down (which i kind of agree with, if I start running into filesystem errors the first thing I do if I can is basically just unplug the drive).
4. They found out that the disk/partition was basically too large, which on one paw is kind of good news because that's easy to fix, but on the other paw is kind of bad news because the way you fix it is copy a huge amount of data and then do a bunch of integrity hashes and it just kind of unavoidably takes a long time.
I could be misunderstanding it though, I am not always the most perceptive fox >w<
And I think this is a reasonable way to approach recovery too, I also appreciate that you're being careful and not rushing a rollout to get the site up faster at the risk of data loss or other stability issues. <3
So just adding my voice to the choir that I think this is being handled well :3
Hope everything goes smoothly! <3
While I am also excited to have the site back, I really hope you all are taking great care of yourselves (I mean, a 15 hour Discord call?!). Thank you all again for everything.
Thank you, tech team, for your hard work. Hang in there!
Damn, the artist drew a great back shot of fender there.. great artwork, so expressive! So fuzzy :3
Best of luck fixin' the server up
If all else fail, just send a prayer to the omnissiah.
Upon seeing "negative count of inodes" I recoiled as if I'd been startled by a spider.
We wish you endless strength, strong coffee and patience to overcome all mistakes!!!😭❤️🔥❤️🔥❤️🔥
Is it too cheeky to ask how much capacity the storage server(s) have? I think I recall back when Dragoneer announced a brand new hardware overhaul that came with hundreds of TBs of capacity - all SSD based for fast read/writes. And it was a game changer for FA.
Makes sense if it could take a week if there's TBs upon TBs of data to move around.
everything will be fine
thanks for all your effort and dedication
seriously though, thanks for actually giving us updates. hope it goes well!
Even without uploads it's nice having it up and access to ones gallery (I'm too lazy to look through my own folders) x3
Would help, but only know about tube, and transistor technology! ^v^
There is more technical detail in the later parts of it, if you want to know. :3
aka maxgoof The Beach Bears
dunno if its even in here But Fender had no clue it even existed!
https://www.furaffinity.net/view/9792854/
niceI hope everything goes really well from here on out.
Thank you. :3
We missed you ❤️
Also, you can rest assured that all the materiel that we use is of the non-metaphasic kind, and as such is very, very hard and solid.
I hope you and your team are making significant progress.
Strength, patience and lots of coffee to the developers, I hope the site will soon start working as before (or even better) 💖
(It's good that I have savings and other social networks, but it will still be hard without commissions and sold adopts... I wish the ability to upload pictures would return soon, eh. But I understand that this is not a quick and easy process, so I'm stocking up on patience.)
In all seriousness, thank you all for your hard work. What a wild ride
Russian components...
All made in Taiwan!
Hopefully the upload function will work again soon.
Perhaps this would have bring you less stress than if you made it open.
I know how much work and how exhausting this is for you all folks.
Us, users really don't know how much work goes into maintaining a server.
With the hope the uploads get enable again very soon! I have a bunch of fun stuff to share! ^^
With the hope we can fill it with more content in a near future
...idk
For now I guess I'll have to learn other platforms for artist, although I haven't had much success with them…
As an artist, I feel comfortable here. Thanks ❤️
NEVER GIVE UP!
As per the journal, the system is in the process of moving the files to a new partition, which will take time.
Patience, young padawan. :3
So we really, really cannot say one way or another how long this will take.
In the event that this is the case, please accept my most sincere apology. I genuinely read our interaction as playful banter, and it might not have been the case.
The original estimate is one week from when the Tech team started the file copying, but it is not guaranteed.
Nothing has changed at time of writing in that regards.
It could be shorter, it could be longer. We don't know for certain.
On a more hilarious note, I can't wait for the Kevin Fang video about this outage- Haha.
this has very much so taught me I really, really need another reliable account for art posting and sales.I rely on furry art incomes way too much to risk a single site being able to take out my income.
Sorry for all the hard work you guys been doing though - hope you're able to get some rest :<
And John was the problem.
I wish you all the best of luck. And I hope it all works out ok.
Me and my flooffy smooshy monster friends are rooting for you.~💀🎀🖤✨️
nice
(This has very much helped me realize i only have 1 social media, and, thats detrimental to an artist. So hey! Good job shanghai-ing me into other platforms.)
the reason for no uploads and we are roughly 13TB into a 27TB transfer.
>looks inside
>bluesky/twitter
Please just make a Sofurry/Itaku/Weasyl
But it ain't a gallery LOL, both for the subjective reason that some people use GIFs and memes to reply and it floods the actual drawings, and the pragmatic one that files are downsized to 2000x2000 and/or 1mb while even FA lets you go up to 2k resolution, but 10mb filesize.
And other galleries have fewer limitations
I wonder if there is any way to fix the storage issue by limiting how many posts a person can make within a certain time frame. The first page of browsing moves so fast! It's great so many have moved here though.
It's good to get such a detailed brief of what's happened and why. I can't make any sense of the technicalities, but it's nice to get an impression. I like it, it's confident. Gives you an idea of what's going on backstage. Very good of you to take the time to write this, and keeping the site up in read mode was a good idea.
Feel weirdly proud that we've taken down FA bunker prime with the combined weight of our stuff!
Think of it like a library, with a front desk, and an archive. They can get the front desk to take things, and hand them over to the archive. They can get the front desk to talk to you, and if they talk directly to the archive they can get specific titles back, and a list of all titles and where they are stored. Sadly, the front desk is having trouble getting information or items back from the archive. Or so I gather, and could be very wrong.
Gotta be honest, this level of transparency on what happened, even tho I understand it about as well as I understand why my rocket ship in KSP keeps blowing up, is very very nice. I know its gotta be exhausting as well, but hopefully y'all keep this type of thing up for future issues!
Also, its gotta be INFURIATING having people ask "y can no upload" on the VERY journal explaining Y NO UPLOAD.
SERIOUSLY Y'ALL. READ. THE. THING.
Best of luck getting everything up and running again soon. 🫶
It just... will take a bit of time. And the team apologizes that we can't provide an ETA at this time, as the journal says.
Keep an eye on this space for updated information as things happen.
<winny_nag_voice> "is it fixed yet" <\winny_nag_voice>
also tries to add a tiny fat16 partition to the end of the normal one
Tech is HOPING that we'll be back within the next 5 days, but there are no guarantees and should not be relied upon.
Tech is hoping that we'll be back to nominal status sometime in the next 5 days, but there are NO GUARANTEES and should not be relied upon as there are too many unknowns at this time.
Good to read nice news. Thank God for my awesome friends and clients that although I'm not a t page one, it keep their support with work for me on this times TvT ♥️
And thank you to the cute puppy that keep us positive, thank you for all and cheers for the amazing tech team for their hard work WUUUU (throwing confetis)
...
ᴵ ᵈᶦᵈⁿ'ᵗ ʳᵉᵃˡᶦᶻᵉ ᶜᵒⁿˢᶦᵈᵉʳᶦⁿᵍ ʰᵒʷ ᵐᵃⁿʸ ᵖᵉᵒᵖˡᵉ ᵃʳᵉ ᶜᵒᵐᵖˡᵃᶦⁿᶦⁿᵍ ʰᵉʳᵉ ˢᵒʳʳʸ ;w;
My FA lies over the web,
My FA lies over the internet,
Oh bring back my FA to me.
Bring back, bring back,
Oh bring back my FA to me, to me,
Bring back, bring back,
Oh bring back my FA to me.
Last night as I lay on my pillow,
Last night as I lay on my bed;
Last night as I lay on my pillow,
I dreamt that my FA was dead.
Bring back, bring back,
Oh bring back my FA to me, to me,
Bring back, bring back,
Oh bring back my FA to me.
Oh, blow, ye data over the internet,
Oh, blow, ye data over the web;
Oh, blow, ye data over the internet,
And bring back my FA to me.
Bring back, bring back,
Oh bring back my FA to me, to me,
Bring back, bring back,
Oh bring back my FA to me.
We're unable to confirm an exact timeframe, but the tech team hopes to have things back to a nominal status within the next 5 business days.
Do understand that this is an estimate, and can change (for better or for worse) as we progress through the recovery process.
The entire staff thanks you for your understanding during these harrowing times.
I mean, years of customer service, furry community management, and conrunning/constaffing had to come in handy for something at some point or another, right? :3
We'll be providing updates as we progress through the process as diligently as possible.
We thank you for your understanding and cooperation during these difficult times for everyone!
If you wish to worship the IT techs, I'd advise making one awoo and drinking a red bull in their honor daily. ;3
We thank you for your cooperation and understanding during this period, and we apologize for the frustrations that might be felt due to this outage.
Or is all the cum flooding the server rooms a lil too hard to wade through? :O
It's really taking a byte out of the system...
_IF_ the massive surge in uploads is due to Promptographic ("A.I.") images and other creations, might we see a limitation of how many, if at all, they may be uploaded here on FA?
Please note: This question is NOT about the legality/ethics of Promptographs, but rather simply folks uploading large numbers of high-quality images to the site rapidly with the ease of their generation?And also some stuff that I 'suspect' is AI-generated, but can't be certain because... to be blunt... there is already a lot of sub-standard CGI stuff out there. It makes it hard to tell what's legit and what's being made by Skynet!
From our tech team,
We're at 61% of data copied to the final destination drive, and verified to be correct against the data from the backup.
But I'll take it!
Let's go, little server! You can do it!! <3
Whoooaaaa-oh! File transfer share!🎵
Actually infrquent downtimes where the site is still sorta usable might be a... nice communal-provoking thing. It makes me appreciate more what we do have, when we have it. And it gives massive boosts to some, and makes for always great memes--remember the 2020 election? XD
This site has been free for almost 20 years? or more? 12 bucks a year times how many people use this site that could help you hire on more people.
But that's just my unpopular idea.
It's not a buck-a-month, but it's fairly reasonable (at least, I think so!).
I had it for a few years now, and I genuinely like the fact that it removes third-party adverts, only keeping the community adverts. Just that feature alone is worthwhile to me.
But obviously, YMMV!
I think most of us old-timers remember that and would rather not re-experience it... or worse. X3
I remember that day, that was a big pain for all of us.
While it might not be a 100% perfect solution, posting journals still work. Maybe this could be a temporary alternative?
Hopefully, this helps a bit to get you back closer to your usual numbers.
We understand it might be frustrating, and we apologize for the inconvenience.
“Mass upload Tsunamis“
“Hope the server got protection!”
Anyone else leaving stuff on their desktop to have a subconscious link with said stuff pending further action, like if I'm gonna use it, it should be within a hand's reach, so if fA was a physical gallery, there would be a stack of drawings on my desk, just think about it, if I didn't have anywhere else to put these up, there's a non-zero chance I have a heart attack or something and then nobody will see these until someone goes and cleans up after me- that got existential for no reason
How soon will work with file uploads be restored. I am one of those people who is not friends with other social networks and lives only on FA -_-
Will uploads work within a week?
Again, this comes with no guarantees, and could take longer (or shorter!).
Because technology will technology...
But if the current trend continues, it should be fixed within that timeframe.
There's a more in-depth description of the operations being done in the second part of the journal that I believe will address your question. :3
I assumed it was being transferred over gigabit ethernet or something to be that slow, since any local interface seems like it should be done long before now!
However, the resonance cascade due to the delaminating it caused to the dish ended up changing the quarks' phases, and thus we had to revert to analog mode before we lost complete metaphasic integrity.
I feel betrayed. >:|
It is working and it isn't working at the same time, until observed, at which point, it is one of the two. 😆
the database containing all informational data like user accounts, submission information (title, description, …), comments, etc. is fine, that’s the reason comments, journals, watching, faving and so on is working
there was an unexpected failure with the storage that contains the uploaded data (mostly images, audio, perhaps some documents, …), which is currently being restored and needs to be verified that everything’s good again
Space [.IS] Party Central Trailer
The narrator has a voice and speed reminiscent of Richard Price, but more menacing, like the late 70's and early 80's VAs for horror movies. The ones who seemed to be thrilled for the villain and almost cynical towards the audience and the cast. He has that same trademark, metallic crackle that mics from back then always had. And a slow, droning "thump-thump, thump-thump" that wouldn't be out of place in Dawn Of The Dead hums along in the background, with sparking bell sounds to hail the arrival of the darkest angels in the universe... And the end of time itself, although, before time ends, there may be hope for a new timeline or universe. If the damned zombie-bots don't eat everybody first!
When all of those dead dudes started coming out of the ground
The only safe place left for our race was outer space
And in time we forgot. It was too hard on our brains
But we left behind their decrepit remains and bloodstains
*POP! POP-POP!* "I HATCHU! You bleedin' flesh-bound mutha-FUCKA'Z!"
"What are they even made of?"
"... Biodroids, infected by dark spectres. They could've spawned from those genetically engineered blackholes they keep talking about. They've been lying dormant inside this whole time, and we brought them back by FUCKING with time!"
"Sh-SHIT! WE'RE OUT OF TIME!"
"No! It's NOT OVER YET... We can still make it!"
"Make it where?! TO WHAT!!!"
"TO THE PARTY! DUDE! Y'know... Because shit's almost over? I wanna score, one last time, eh!"
"... Aaawwww, fuck it, you're right! I'm bein' STUPID!"
But it's not over yet! There's plenty of time for new regrets!
So load up your laser teks and and quantum berettes
And party down...
Until the clock winds down
Because Space IS PARTY CENTRAL!
...Did you mean Vincent Price?
Also, beware Amazon policies and internal politics (EULA updates that make your teeth curl in distaste).
I mean, use it if you must, but don't expect it to come without its fair share of issues.
Thanks for keeping us updated nonetheless!
Is the username change the (display name change)?
It's not perfect, but it's a step forward! :3
As for other features, I'm not sure what Tech has on their plate right now (other than getting the site back in proper working order), but you can get a decent idea of what's to come in this journal's "Roadmap" section.
I hope this helps! :3
Thank you for taking care of the site! Thanks to it I can make a living from making art <3
So barring any unforeseen issues, probably a few more days, yes.
Thank you for your patience!!
The Ballad Of Rob-N-Hood'z:
Four Crazy Furs (Part I)
If nobody could make a move then nobody would get away
Everyday we would stay in the holes right there lying in wait
Highway man come softly strolling along,
Singing a mellow song in a melody never wrong,
Then four crazy furs made a claim on their ride,
"Happy to see you good men but I'm low so do provide!"
"... Or your life..."
Little John yellin' "Kill-kill 'em all! Murda y'all - Oh, what a ball!"
Then "Robin, better dip down in the cut-n-deep there for safety - Sherrif be chasin' y'all!"
"Give it up, give it up!" When 'em know the time. Swappin' arrows out for the nines
Now it's bandit on bandit and we're like "God damn it! We had it - all! And we planned it fine!"
The cynical militant lycans are comin' to lay down the landmines
And look here they come rollin' up we invite them to dine...
... On the silver bullets in our nines...
So the tech team took steps to increase storage space before the site ran out.
During the partition resize, Something Bad™ happened, and corrupted some important stuff in the background.
The important stuff started giving users errors, and that was noticed/flagged.
So the tech team went "Oh crap", and investigated.
When they figured out what was going on, they had to take the site in read-only mode in order to stop any data loss, and rebuild the partition.
But when rebuilding a partition, it empties it. So, now, gotta copy all the files, and make sure everything is okay with no corruption across all partitions and drives, back to the original.
And here we are!
(Obviously, OVERSIMPLIFIED explanation).