Go S3

Open-source Go projects categorized as S3
Edit details
Topics: Golang Go AWS Storage Minio
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 23 Go S3 Projects

  • minio

    The Object Store for AI Data Infrastructure

    • Project mention: Montrez patte blanche : tuez des mutants ! | dev.to | 2024-05-16

  • rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

    • Project mention: Supabase Storage: now supports the S3 protocol | dev.to | 2024-04-18

    rclone: a command-line program to manage files on cloud storage.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    • Project mention: S3 Is Showing Its Age | news.ycombinator.com | 2024-05-22

    > When it gets too out of hand, people will paper it over with a new, simpler abstraction layer, and the process starts again, only with a layer of garbage spaghetti underneath.

    I'm pretty happy that there are S3 compatible stores that you can host yourself, that aren't insanely complex.

    MinIO: https://min.io/

    SeaweedFS: https://github.com/seaweedfs/seaweedfs (this one's particularly nice and is permissively licensed, in contrast to everything else)

    There was also Zenko, but I don't think they gained a lot of traction for the most part: https://www.zenko.io/

    Of course, many will prefer hosted/managed solutions and that's perfectly fine, but at least when you run software yourself, you are more in control over it and for the most part can also make the judgement on how hard it is to operate and keep operational (e.g. similar to what you'd experience when running PostgreSQL/MariaDB/MySQL or trying to run Oracle).

  • thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    • Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10

    Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

  • litestream

    Streaming replication for SQLite.

    • Project mention: Uber Migrates 1T Records from DynamoDB to LedgerStore to Save $6M Annually | news.ycombinator.com | 2024-05-20

    Litestream is open source.

    https://github.com/benbjohnson/litestream

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    • Project mention: Low-Cost Read/Write Separation: Jerry Builds a Primary-Replica ClickHouse Architecture | dev.to | 2024-05-30

    To solve our pain points, we chose JuiceFS due to the following reasons:

  • SFTPGo

    Full-featured and highly configurable SFTP, HTTP/S, FTP/S and WebDAV server - S3, Google Cloud Storage, Azure Blob

    • Project mention: What you guys are hosting instead of Nextcloud? I'm sick of it. | /r/selfhosted | 2023-11-29

    EDIT: Thanks for the recommendations from all of you!! I've chose to use the below: - Files: sftpgo - Calendar: baikal - Notes: memos (But beware, it sends opt-out telemetry) - Network folder: webdav on sftpgo

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    • Project mention: Is Posix Outdated? | news.ycombinator.com | 2023-10-19

    The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

    Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.

    So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

    Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

    That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

    https://github.com/s3fs-fuse/s3fs-fuse

    https://github.com/kahing/goofys

    https://www.cuno.io/

  • wal-g

    Archival and Restoration for databases in the Cloud

    • Project mention: WAL-G 3.0.0 – fast disaster recovery for Postgres | news.ycombinator.com | 2024-03-17

  • mc

    Simple | Fast tool to manage MinIO clusters :cloud:

  • S3Scanner

    Scan for misconfigured S3 buckets across S3-compatible APIs!

  • s5cmd

    Parallel S3 and local filesystem execution tool.

    • Project mention: GitHub issues from top Open Source Golang Repositories that you should contribute to | dev.to | 2024-01-15

    s5cmd - Extended character support for s3 compatible backend

  • minio-go

    MinIO Go client SDK for S3 compatible object storage

  • docker-volume-backup

    Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage

    • Project mention: I have a question about | /r/paulsdevblog | 2023-11-29

    I am interested in coming up with a backup plan before I get too invested in this setup. I found the docker-volume-backup project that looks like it might be a possible solution. However I'm not sure how to implement it using docker swarm since I am new to all of this. I would be interested in learning what backup solution you use for your docker swarm servers.

  • rill

    Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code. (by rilldata)

    • Project mention: Governments on GitHub | news.ycombinator.com | 2023-06-09

  • clickhouse-backup

    Tool for easy ClickHouse backup and restore using object storage for backup files.

  • s3-benchmark

    Measure Amazon S3's performance from any location.

    • Project mention: S3 Benchmark: Measure Amazon S3's performance from any location | news.ycombinator.com | 2024-03-23

  • csi-s3

    A Container Storage Interface for S3

  • ftpserver

    Golang based autonomous FTP server with SFTP, S3, Dropbox, and Google Drive connectors.

  • helm-s3

    ⎈ Helm plugin that adds support for AWS S3 as a chart repository.

    • Project mention: Installing multiple helm charts in one go [Approach 2 - using helmfile] | dev.to | 2023-12-26

    $ helm plugin list NAME VERSION DESCRIPTION diff 3.8.1 Preview helm upgrade changes as a diff helm-git 0.12.0 Get non-packaged Charts directly from Git. s3 0.14.0 Provides AWS S3 protocol support for charts and repos. https://github.com/hypnoglow/helm-s3 secrets 4.1.1 This plugin provides secrets values encryption for Helm charts secure storing

  • storage

    A vendor-neutral storage library for Golang: Write once, run on every storage service.

  • mort

    Storage and image processing server written in Go

  • Walrus

    🔥 Fast, Secure and Reliable System Backup, Set up in Minutes. (by Clivern)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Go S3 related posts

  • Uber Migrates 1T Records from DynamoDB to LedgerStore to Save $6M Annually

    5 projects | news.ycombinator.com | 20 May 2024

  • JuiceFS 1.2 Beta 1: Gateway Upgrade, Enhanced Multi-User Permission Management

    2 projects | dev.to | 5 May 2024

  • How (and why) to run SQLite in production

    2 projects | news.ycombinator.com | 27 Mar 2024

  • S3 Benchmark: Measure Amazon S3's performance from any location

    1 project | news.ycombinator.com | 23 Mar 2024

  • SQLite3 Replication: A Wizard's Guide🧙🏽

    2 projects | dev.to | 27 Feb 2024

  • Show HN: Stree – tree view of S3 bucket

    1 project | news.ycombinator.com | 27 Feb 2024

  • A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes

    2 projects | news.ycombinator.com | 22 Feb 2024
  • A note from our sponsor - SaaSHub
    www.saashub.com | 1 Jun 2024
    SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source S3 projects in Go? This list will help you:

Project Stars
1 minio 44,785
2 rclone 44,311
3 seaweedfs 21,323
4 thanos 12,652
5 litestream 10,163
6 juicefs 9,905
7 SFTPGo 8,363
8 goofys 5,065
9 wal-g 3,075
10 mc 2,725
11 S3Scanner 2,417
12 s5cmd 2,377
13 minio-go 2,310
14 docker-volume-backup 1,638
15 rill 1,405
16 clickhouse-backup 1,162
17 s3-benchmark 781
18 csi-s3 747
19 ftpserver 574
20 helm-s3 548
21 storage 527
22 mort 501
23 Walrus 478

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com