Currently in Beta
ShardS3 is still unfinished and under active development. Version 1.0 will be available soon.
ShardS3 is a storage service built for backing up data via Amazon S3 API. It uses multiple storage backends to store objects.
Data is split among multiple backends similar to RAID using reed-solomon encoding. This allows for faster up and downloads and most importantly adds redundancy in case a backend fails or data gets deleted.
Disclaimer: This is only a test project. Using services like Discord for backup and data storage may be against their policies.
A violation of these policies could lead to penalties. Including banned accounts, blocked phone numbers, data loss or other consequences.
Use this software at your own risk.
The following diagram shows how an S3-Objects is split and stored: In this example there are three backends:
- Backend 1. Allows file size of up to 40MiB
- Backend 2. Allows file size of up to 25MiB
- Backend 3. Allows file size of up to 200Mib
Since the smallest allowed size if 25MiB, each encoded shard must be 25MiB. The number of allowed backends to fail is one, which is why there are three parity shards. If any of the backends fail, the data is still available. If more than one backend fails, the data is not recoverable.
Final Result:
For a detailed explanation of the program you can checkout the docs
Work In Progress
ShardS3 can be deployed via Helm to a Kubernetes Cluster.
podman compose -f .\container\compose.yaml up
- Telegram
- Discord
- Local File (for testing only)
- youtube
- ...
The following S3 Operations are supported:
- ListObjectsV2
- HeadObject
- GetObject
- PutObject
- DeleteObject
- DeleteObjects
- GetObjectAcl
- GetObjectTagging
- ListBuckets
- HeadBucket
- GetBucketLocation
- GetBucketEncryption
- GetBucketAcl
- SigV4 authentication
- Presigned URLs
- CreateMultipartUpload
- UploadPart
- CompleteMultipartUpload
- AbortMultipartUpload
- ListMultipartUploads
- ListParts
- Caching
- Multipart Upload
- Versioning
- Tagging