Feature : add third-party cloud as mass-storage for users (nextcloud) #476

Closed
opened 2023-12-31 07:47:12 +01:00 by Gobtous · 10 comments

Hi all,

As a main storage solution we already have a small allocated mass storage for our file.

I think we can expand it by connecting it with our own cloud solution as nextcloud for exemple. It avoid using up the server memory.

But there several problem :

it mays be inequal as people having an external cloud are rich. They paid the server, domain...or not. But that's not fair.

Moderator can't verify its content, same problem as the ability to post hotlink image (issue #441). To solve it, i suggest we let :

  • the admin decide to enable it on its instance.
  • the moderator activate it per user after several verification(ID, activity, ip, account creationkdate) so it ease up our moderation job

@AntoineD also talked me about S3 protocole

Hi all, As a main storage solution we already have a small allocated mass storage for our file. I think we can expand it by connecting it with our own cloud solution as nextcloud for exemple. It avoid using up the server memory. But there several problem : it mays be inequal as people having an external cloud are rich. They paid the server, domain...or not. But that's not fair. Moderator can't verify its content, same problem as the ability to post hotlink image (issue #441). To solve it, i suggest we let : * the admin decide to enable it on its instance. * the moderator activate it per user after several verification(ID, activity, ip, account creationkdate) so it ease up our moderation job @AntoineD also talked me about S3 protocole
AntoineD added the
enhancement
discussion
labels 2023-12-31 11:54:02 +01:00

Are you suggesting that users can add their own cloud as a media source or that admins can use them as a data backend?

Are you suggesting that users can add their own cloud as a media source or that admins can use them as a data backend?
Member

And getting something like Backblaze B2 is not an option, because...? Like, I absolutely do not understand what do you mean, why nextcloud of all things? What's the deal with such verification? Why should it be per user only?

And getting something like [Backblaze B2](https://www.backblaze.com/cloud-storage) is not an option, because...? Like, I absolutely do not understand what do you mean, why nextcloud of all things? What's the deal with such verification? Why should it be per user only?
Author

Are you suggesting that users can add their own cloud as a media source or that admins can use them as a data backend?

I'm suggesting that user add their own cloud solution so it free iceshrimp's memory and reduce server cost. And let admin activate it per user for moderation issue.

> Are you suggesting that users can add their own cloud as a media source or that admins can use them as a data backend? I'm suggesting that user add their own cloud solution so it free iceshrimp's memory and reduce server cost. And let admin activate it per user for moderation issue.
Author

And getting something like Backblaze B2 is not an option, because...? Like, I absolutely do not understand what do you mean, why nextcloud of all things? What's the deal with such verification? Why should it be per user only?

Because most intance on the fediverse lack moderators or moderation tool. And from my experience as a moderator on lemmy we had to remove porn, lolicon, spam...it was a very difficult experience and it propagate very quickly on the fediverse. So my idea is to check user's identity and decide whether you trust them or not.

So letting admin/modo activate it per users is a safety measure because you won't be able to check the content of third party cloud, nor nuke them. We can go to prison if we don't moderate properly. And, futhermore it's all up to the admin to decide its model. I find it safer this way.

If there is problem the admin can remove it and it will decrease his/her workload.You can't expect a moderator to check it everyday, we also need some free time. So i wanted this way so admin can scale it to its instance management. That's purely from moderation point of view.

You can activate it per user or not. That's up to admin.

I don't know blackhaze, if it's good, then it's good. I trust ya.

i'm suggesting nextcloud because i know it and use it. I manage a nextcloud server for a translation team. it's an alternative to google drive, used in murena os, a degoogled os with lot privacy features.

And its strong point are its app as collaborative writing, expense management, gps for trekking, mail, agenda, task, quizz, trello like...i use it and it sync all data on phone, desktop and is well integrated on linux.

You can also see mastodon post on nextcloud dashboard.

> And getting something like [Backblaze B2](https://www.backblaze.com/cloud-storage) is not an option, because...? Like, I absolutely do not understand what do you mean, why nextcloud of all things? What's the deal with such verification? Why should it be per user only? Because most intance on the fediverse lack moderators or moderation tool. And from my experience as a moderator on lemmy we had to remove porn, lolicon, spam...it was a very difficult experience and it propagate very quickly on the fediverse. So my idea is to check user's identity and decide whether you trust them or not. So letting admin/modo activate it per users is a safety measure because you won't be able to check the content of third party cloud, nor nuke them. We can go to prison if we don't moderate properly. And, futhermore it's all up to the admin to decide its model. I find it safer this way. If there is problem the admin can remove it and it will decrease his/her workload.You can't expect a moderator to check it everyday, we also need some free time. So i wanted this way so admin can scale it to its instance management. That's purely from moderation point of view. You can activate it per user or not. That's up to admin. I don't know blackhaze, if it's good, then it's good. I trust ya. i'm suggesting nextcloud because i know it and use it. I manage a nextcloud server for a translation team. it's an alternative to google drive, used in murena os, a degoogled os with lot privacy features. And its strong point are its app as collaborative writing, expense management, gps for trekking, mail, agenda, task, quizz, trello like...i use it and it sync all data on phone, desktop and is well integrated on linux. You can also see mastodon post on nextcloud dashboard.
Author

And getting something like Backblaze B2 is not an option, because...? Like, I absolutely do not understand what do you mean, why nextcloud of all things? What's the deal with such verification? Why should it be per user only?

Because most intance on the fediverse lack moderators or moderation tool. And from my experience as a moderator on lemmy we had to remove porn, lolicon, spam...it was a very difficult experience and it propagate very quickly on the fediverse. So my idea is to check user's identity and decide whether you trust them or not.

So letting admin/modo activate it per users is a safety measure because you won't be able to check the content of third party cloud, nor nuke them. We can go to prison if we don't moderate properly. And, futhermore it's all up to the admin to decide its model. I find it safer this way.

If there is problem the admin can remove it and it will decrease his/her workload.You can't expect a moderator to check it everyday, we also need some free time.

You can activate it per user or not. That's up to admin

I don't know blackhaze, if it's good, then it's good. I trust ya.

i'm suggesting nextcloud because i know it and use it. I manage a nextcloud server for a translation team. it's an alternative to google drive, used in murena os, a degoogled os with lot privacy features.

And its strong point are its app as collaborative writing, expense management, gps for trekking, mail, agenda, task, quizz, trello like...i use it and it sync all data on phone, desktop and is well integrated on linux.

You can also see mastodon post on its dashboard.

> And getting something like [Backblaze B2](https://www.backblaze.com/cloud-storage) is not an option, because...? Like, I absolutely do not understand what do you mean, why nextcloud of all things? What's the deal with such verification? Why should it be per user only? Because most intance on the fediverse lack moderators or moderation tool. And from my experience as a moderator on lemmy we had to remove porn, lolicon, spam...it was a very difficult experience and it propagate very quickly on the fediverse. So my idea is to check user's identity and decide whether you trust them or not. So letting admin/modo activate it per users is a safety measure because you won't be able to check the content of third party cloud, nor nuke them. We can go to prison if we don't moderate properly. And, futhermore it's all up to the admin to decide its model. I find it safer this way. If there is problem the admin can remove it and it will decrease his/her workload.You can't expect a moderator to check it everyday, we also need some free time. You can activate it per user or not. That's up to admin I don't know blackhaze, if it's good, then it's good. I trust ya. i'm suggesting nextcloud because i know it and use it. I manage a nextcloud server for a translation team. it's an alternative to google drive, used in murena os, a degoogled os with lot privacy features. And its strong point are its app as collaborative writing, expense management, gps for trekking, mail, agenda, task, quizz, trello like...i use it and it sync all data on phone, desktop and is well integrated on linux. You can also see mastodon post on its dashboard.

I honestly believe this is not a viable option.

  1. Nextcloud is build as a private cloud alternative and was never meant to act as a high frequent storage provider.
    A proper implementation with iceshrimp will be probably difficult and even if we manage it, there is no practical experience how well Nextcloud behaves when hundreds of remote servers are trying to pull a image, you just posted, at the same time.
    Considering the fact that Nextcloud is pushing everything trough a php based webdav server, I assume it will rather poorly.

  2. Most Nextcloud instances are usual run privately on either lowend hardware or with a not so great internet connection. So loading times will be slower. Also the overall reliability is lower, which can cause issues for other users of the instance. (eg. images are not loading because nextcloud instance is down, or does not exist anymore etc).

  3. I actually don't see the economical benefit here.
    Running a Nextcloud instance is more resource heavy and more expensive than using something like S3.
    The only difference is that you "outsource" the responsibility to your users, which is, I assume, not the goal of iceshrimp or most fedi instances. The goal is usually to provide a platform for them, so they don't have to do it themself.

  4. As you already mentioned, it requires additional moderation effort

Soo, overall I don't think it's a good idea and even if we resolve some of the mentioned issues, it would still be not worth the effort to actually implement it, considering the user base that wants serve images from their Nextcloud instance, but don't run a own iceshrimp instance is probably rather low.

I honestly believe this is not a viable option. 1. Nextcloud is build as a private cloud alternative and was never meant to act as a high frequent storage provider. A proper implementation with iceshrimp will be probably difficult and even if we manage it, there is no practical experience how well Nextcloud behaves when hundreds of remote servers are trying to pull a image, you just posted, at the same time. Considering the fact that Nextcloud is pushing everything trough a php based webdav server, I **assume** it will rather poorly. 2. Most Nextcloud instances are usual run privately on either lowend hardware or with a not so great internet connection. So loading times will be slower. Also the overall reliability is lower, which can cause issues for other users of the instance. (eg. images are not loading because nextcloud instance is down, or does not exist anymore etc). 3. I actually don't see the economical benefit here. Running a Nextcloud instance is more resource heavy and more expensive than using something like S3. The only difference is that you "outsource" the responsibility to your users, which is, I assume, not the goal of iceshrimp or most fedi instances. The goal is usually to provide a platform for them, so they don't have to do it themself. 4. As you already mentioned, it requires additional moderation effort Soo, overall I don't think it's a good idea and even if we resolve some of the mentioned issues, it would still be not worth the effort to actually implement it, considering the user base that wants serve images from their Nextcloud instance, but don't run a own iceshrimp instance is probably rather low.
Author

Thank replying and giving those technical information. You are right. Let's do like you suggested, i don't know how server network is managed :)

A last comment :
i run my nextcloud on infomaniak server and gandi. The bandwidth is unlimited so i don't understand how it can break its use, i guess it's related to task, ram, cpu...something like that.

Well, i havent meet any problems since 5 years. Currently, i run it for 15 people on a low hardward with 3 websites hosted and several devices that sync with nextcloud. It hold pretty well.

But that's not the same scale as the fediverse.

As for the price, when you run any social network, let's say you have around 800 peoples. 80 peoples post 1 picture per days, taken with their smartphone. those pictures weight 1mb. So 80mb per day and it increase every day. And who pay the server ? The admin. Are donation enough ?

So that's why i suggested a cloud storage solution to externalize the memory cost. Currently, i delete old picture to free up space so we are back to the same problem you mentionned in point 2 : viewing picture.

The posted picture will diseaspar because i wanted to post new picture, so i deleted them or the admin decided to remove old post from the last year to free up space on his instance.

That's all, let's close the topic :)

Aaand hank a lot for developping iceshrimp. ;)

Thank replying and giving those technical information. You are right. Let's do like you suggested, i don't know how server network is managed :) A last comment : i run my nextcloud on infomaniak server and gandi. The bandwidth is unlimited so i don't understand how it can break its use, i guess it's related to task, ram, cpu...something like that. Well, i havent meet any problems since 5 years. Currently, i run it for 15 people on a low hardward with 3 websites hosted and several devices that sync with nextcloud. It hold pretty well. But that's not the same scale as the fediverse. As for the price, when you run any social network, let's say you have around 800 peoples. 80 peoples post 1 picture per days, taken with their smartphone. those pictures weight 1mb. So 80mb per day and it increase every day. And who pay the server ? The admin. Are donation enough ? So that's why i suggested a cloud storage solution to externalize the memory cost. Currently, i delete old picture to free up space so we are back to the same problem you mentionned in point 2 : viewing picture. The posted picture will diseaspar because i wanted to post new picture, so i deleted them or the admin decided to remove old post from the last year to free up space on his instance. That's all, let's close the topic :) Aaand hank a lot for developping iceshrimp. ;)

Maybe you really should look into S3 storage.
I pay roughly 6-7 Euro/Month per TB.

Considering we take your 80MB/Day value as a basis, you would need like 35 years to fill up 1TB of space.
80MB/Day is probably really low, depending on the instance size, but even we take 800MB/Day as a value you still need 3.5 years to fill up 1TB of space.

So, I think storage space is not really an issue. At least compared to the other resources you would need to run a 800 user instance.

Maybe you really should look into S3 storage. I pay roughly 6-7 Euro/Month per TB. Considering we take your 80MB/Day value as a basis, you would need like 35 years to fill up 1TB of space. 80MB/Day is probably really low, depending on the instance size, but even we take 800MB/Day as a value you still need 3.5 years to fill up 1TB of space. So, I think storage space is not really an issue. At least compared to the other resources you would need to run a 800 user instance.
Author

Well, its Amazon ? Plenty issue. :)

Well, its Amazon ? Plenty issue. :)
Member

S3 is not just Amazon, There's Wasabi, Backblaze who also offer S3-compatible buckets which are cheaper than AWS, you could also selfhost it using Minio, if you really want that

S3 is not just Amazon, There's Wasabi, Backblaze who also offer S3-compatible buckets which are cheaper than AWS, you could also selfhost it using Minio, if you really want that
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: iceshrimp/iceshrimp#476
No description provided.