A BNOC data share?

Data
1

I’ve been thinking about a good solution for a Bitcoin data share where we can upload and share e.g. multiple TB of historical debug logs (Offer: detailed Bitcoin Core node debug.log's (2024-...); potentially automated uploads each night) and ad-hoc data collection results like the one requested in Request: Bitcoin and/or Lightning packet captures.

Requirements:

  • 10+ TB of space
  • no server/disk maintaince required for me
  • good connectivty
  • ideally, cheap or free :slight_smile:
  • ideally, some data redundancy - but it’s a share, not a backup.

Happy to hear ideas.

2

One thing that catched my attention was Google offering 100TB of Google Drive storage for non-profits.

I spoke to someone from a Bitcoin non-profit and they mentioned it would be an option to use this Google Drive space for something like this.

I’m not sure how I feel about giving even more data to Google. However, XORing/encrypting the files with a well-known secret might be enough for it to be automatically ingested into the depth of Google..

I’m not sure how easy it is to download files via e.g. wget from Google Drive…

1 Like
3

Maybe self-hosted S3 options worth considering: garage or seaweedFS.
Community members can contribute storage nodes to the cluster. (it’s something longterm…)

This sidesteps Google Drive’s privacy concerns and stays open-source.

2 Likes
4

It seems like rclone has Google Drive support:

But my read of the docs is that that’s mostly for a single user (I’ve used it successfully that way). Not sure how it could be used for people to download from a public / shared folder though; a quick web search suggests its possible?

5

There also seems to be GitHub - wkentaro/gdown: Google Drive public file downloader when curl/wget fails. · GitHub.

1 Like
6

Brink.dev was so kind a set up a “Shared Drive” thing using their Google Drive subscription (thanks Brink!). We probably have about 10 TB or so there. I have yet to test this, but will do so the next time I need to share some data. Happy to provide write access if someone needs it to share files, but likely only to trusted people, since I obviously don’t want to get Brink into problems with Google by hosting random, suspicous files.

Here’s a public debug log from one of the demo peer-observer logs for testing: https://drive.google.com/file/d/16H_30cM33plIImIQhGT_e8gDYtz3pl-D/view

There’s now also a static, non-Google file viewer here: BNOC data-share

3 Likes