Самое полное руководство по работе с TweetDeck :: (12) Oleg Michman 25% �� 146% ☢️ ☣️⚠️ (@CASBT_UA) / Твіттер

 

BTFS – mount any .torrent file or magnet link as directory

32-40 minutes

---

This is effectively what IPFS is, but IPFS takes it to the next level by using the equivalent of torrent files for subdirectories within the mounted torrent files, so you can "traverse" entire networks of the pretty-much-a-torrent files without first having to download and mount them. And like, you could implement similar functionality here by having torrent files that contain files named .torrent have those files be treated as directories, but at that point IPFS is going to be much more feature complete; even like, if you have some giant file, on IPFS that might be represented by trees of hashes as opposed to merely a giant array of hashes so you can incrementally and efficiently access the parts you need. So like, while this is awesome work, if you find it inspiring, maybe channel that inspiration into learning about IPFS and extending that ecosystem, rather than doubling down on trying to build on torrent files.

¿Por qué no los dos? How is it "doubling down", they could exist side by side. Your comment kind-of shows the mentality I've seen with the IPFS project before. The IPFS project has doubled down on it's effort of being more "advanced", "innovative" and "efficient" than BitTorrent, losing what actually makes people like torrents and distributed systems. Unfortunately right now the IPFS project is totally ignoring that it's effectively nothing and takes nothing to the next level. I think what I mentioned before, is the root cause. Zero ounce of compatibility with the existing BT swarm. It is simply a massive amount of wasted potential to ignore the thousands of terabytes of content the torrent system hosts, the work and services provided and spent for the ecosystem. So much could be done that would strengthen both systems at the same time. Just from the IPFS point of view we could right now have embedded resources from torrents, tens of different clients available that could just be CDNs, existing seedboxes strengthening both swarms and probably much more. That would be the mix of torrents and web people want, not what IPFS is currently. IPFS is basically reinventing the wheel, but throwing out the round part. Maybe the IPFS project should channel more work on building next to giants rather than doubling down and throwing away everything BT has to offer?

To be clear, I am not involved in the IPFS project and in fact don't like key aspects of the project and think it is doomed to long-term failure when it gets replaced by something better. That said, the core and pretty fundamental limitation with BitTorrent, however, is that the swarms exist "per torrent", which kind of makes a non-starter for a general purpose filesystem? As such, I don't quite disagree with what you'd like to see done, but that work should go in IPFS... like, trying to extend the way BitTorrent works "up" to something that works for this use case is going to involve a much more fundamental set of low-level changes than even the WebTorrent shift (which is where I'd argue "if only BitTorrent understood the value of working with existing browser-based systems and opening up powerful use cases they would understand that not doing this is like throwing away the round part of a wheel... etc." ;P WebTorrent is the future of BitTorrent as far as I'm concerned, and anything failing to embrace that is already kind of doomed?...) than the other direction, which is really just "go into IPFS and add a little adapter that lets it work with torrent files (at the fundamental loss of efficiency and awkward lack of resource sharing that such usage implies).

The primary difference I see here is that IPFS is rather centralized in terms of development, it is much much easier for them to write a BT compat layer of some sort than it is for the BT community. A core set of BT is really rather ossified thanks to the amount of implementations out there. Just as an example there are quite a few BEPs out there that implement some features that IPFS has, unfortunately those features haven't gotten any mainstream adoption so far. > That said, the core and pretty fundamental limitation with BitTorrent, however, is that the swarms exist "per torrent", which kind of makes a non-starter for a general purpose filesystem? Well, ipfs nodes also tend to pin only certain content. I've seen IPFS links stop working mentioned here on HN even. I see no functional difference in that sense. Content is kept available by those willing to spend resources storing and uploading the specific content. To address the claimed performance/efficiency problems, IPFS peers could share the same files using IPFS itself, but still also seed them for BT clients. I totally get though that it wouldn't be super easy to implement, but IMHO the increase in IPFS usefulness, size and resiliency thanks to the integration seems to outweigh the downsides I can see. Though a while ago I did try sharing the same set of files on two systems by running the two in parallel. It kind-of worked but manually updating the file locations in the torrent client and having an overview became too annoying, it should be built-in. E.g. IPFS sees `x.torrent` and `x`, then assumes `x` is downloadable from BT using that torrent file. > WebTorrent is the future of BitTorrent as far as I'm concerned, and anything failing to embrace that is already kind of doomed? In some use-cases, absolutely.

What are the current best practices for hosting dynamically updated content on IPFS? Something like a blog that gets updated once a day or a Hacker News clone that gets multiple updates per minute. Last time I checked it was DNS, has something changed sice then?

HN is too frequently updated, but for blogs or personal web pages it's okay. You can either use IPNS or update a DNS record each time you update your content. I chose the latter as years of experiences proved it more robust than IPNS (also, my registrar provides a very simple API that I can use to automatically update the appropriate DNS record, so this is done by the same Python script that update my IPFSite). The downside is that hosting an IPFS node is quite heavy on the network. I have a reasonably good connection at home (a bit less than 1MB/s up, a bit more than 10MB/s down) and if I run the IPFS daemon locally I can't really use my internet connection for anything else. I used to rent a little VPS for that, but now I use the free-tier of temporal.cloud and it is sufficient for my usage.

Something that's as dynamic as HN seems to me like a bad fit for IPFS. A blog seems fine for it; I make my own blog accessible through IPFS. DNS seems like the best and currently most popular option for letting you have an address with an update-able IPFS link. The nice thing about using DNS is that you can use the same address for people accessing via IPFS and people accessing via standard browsers.

Are there any design changes that could create organic adoption of IPFS? Or, conversely, what design decisions do you think have hindered adoption?

I've had a breif look at IPFS, but not enough to say anything with any amount of confidence. That said, the thing that immediately put me off was that everything seemed to want to be browser based, like starting some service that makes a local web server that acts as a GUI - there was even a chrome extension that tried to make this easy for you. A feature that might exist, but I haven't discovered, that would make IPFS more appealing to me, would be a way to mount the IPFS like it were an FTP server, and navigate it with whatever file manager you prefer and not have to worry about managing the backend stuff. That is to say, I'd like BTFS for IPFS.

Given the other reply to your comment was someone who seriously thinks IPFS's core feature--being able to be used as a filesystem--doesn't exist, I guess I'll have to go with "better onboarding documentation" ;P.

The lack of standard dns glue, i.e. a domain which turns to local client if you have it, web gateway like ipfs.io otherwise.

And for ones who are after a product which can already do all those written above can go for Sia Skynet instead of waiting IPFS.

The source code is far smaller that I would've expected, tiny in fact - no seriously, its probably <2000 lines of actual code and looks extremely simple to audit and fix bugs. It seems the only real external dependency is libtorrent. Honestly, if your not already amazed if not at the simplicity of this project, than you should be by the simplicity of fuse filesystems and the capabilities it exposes.

If you think that's excellent and are interested in fuse, you're probably intereted in 9front which takes the concept to another level. Ftpfs does what it says, but webfs handles http and forms and the like, upas/fs can show your imap inbox, it's splendid and simple and allows your tooling to be the familiar nix-like utilities like awk and sed. Even capture your screenshot by catting /dev/screen to a file

Which lib/approach would you recommend if i need run a remote batch process using local files? For example, on the local system: remote-execute gcc localfile.c -o localfile.o And on the remote system, there is a gcc and a remote agent which receives and retransmits the files

You would use rcpu(1) which essentially imports the remote cpu. In other words the remote processor "sees" the local file structure/namespace. 9front is full of "free carrots". If you want a remote vpn, just import another servers /net ontop of your process namespace and you're all set.

It also stays at "looks very interesting", it's not a usable OS on a normal machine for everyday tasks.

SSH for job running and management, NFS for filesystem access. Have remote systems mount the local file system. If NFS isn’t available, have the remote job pull files over rsync from your local system, then rsync back the resulting output.

Ugh! Thanks but no thanks I dont want to mount and expose nfs, or allow a full ssh login. This will be a custom agent/daemon on both ends

I wouldn't recommend NFS over the open internet but there's a few options based around SFTP (rsync, scp, sshfs) and being based on SFTP means they can run without granting that user full SSH login access while still taking advantage of the security benefits that SSH brings. For job execution you could write your own agent but doing likely wouldn't be any more secure than SSH. Just make sure you have disabled password logins (use keys instead) and fail2ban or equivalent running to auto blacklist attacks. You could probably use Chef or SaltStack if really wanted to avoid a remote shell but if you're not already running config management then you have to ask yourself if you're over-engineering a solution. An alternative solution would be to run an OpenVPN tunnel and then you can SSH to your hearts content. But even here, unless you have multiple machines you want to connect to, I can't help thinking you're just making life harder for yourself without getting any realistic gains. This is all based on the very high level spec provided so I accept there might be some currently undisclosed detail that renders the above suggestions moot.

NFS works great over the open Internet, as long as you do it through a secure tunnel. I've been doing this for years as a way of increasing the size of the available storage in a VPS.

I've used both vtun and WireGuard for remote NFS. On a good day, I get 100MB/s to the NFS filesystem on my San Jose AWS instance (from Vegas) via WireGuard. Note: That was before CoVid-19. CenturyLink/Quest has since (stealthily) throttled my bandwidth down from 1Gbps to ~750Mbps.

It's really no different to running any other service over VPN or SSH tunnel. It does work well with NFSv3 but never tried with NFSv4.

It's only 2000 lines because it's in C++. I wrote a simple Fuse system to mount zip and rar files (read-only like this) in <300 lines of Python. Fuse is pretty easy to make useful things with.

This is cool. I can mount most of the Internet Archive as a directory with this. Would love this to grow into what your consumer cloud storage services are but for torrents (a global file system you can mount locally, using torrent data as inodes, with visual representations for availability status similar to the Dropbox green check boxes). Perhaps even with support for web seeding from a storage system of last resort (Backblaze, S3, etc).

The cool part is you can have S3 serve a torrent, and then disable access to the object with permissions once it reaches an adequate seed level. You’re no longer charged for outbound data, but S3 will continue to act as a tracker for the swarm.

Just checked Amazon's S3 page linked to by the PP. Question - how do the following "Notes" that Amazon has on it's page affect that? "Amazon S3 does not support the BitTorrent protocol in AWS Regions launched after May 30, 2016." "You can only get a torrent file for objects that are less than 5 GBs in size."

FWIW the S3 BitTorrent support is very primitive and limited. It hasn't been updated since 2006 from what I can tell.

While you're here I have to thank you for your go torrent library, it's really nice ! I have never used S3's torrent capability, but at this point bittorrent is pretty much "done" -- there's not much to add to make the protocol better without breaking compatibility, which means there's not much support to maintain a working software. But I'm not sure about it, maybe they don't participate in the DHT or don't use PEX, so that could be something they can improve.

Well, I wrote a device driver for Windows 8 that would mount a .torrent file as a virtual disk way back in 2011. We pitched the intellectual property around to several large companies. The only interest we had in the technology was video game companies... where we could mark file pieces as 'high priority' and allow the player to play the game even when the game disk was partially downloaded. The IP was sold to a company developing their next-generation console video game system as part of a 24 million dollar package and never saw the light of day. I suspect what they really wanted was the associated patents.

I wonder how intelligently this handles piece download order. Torrent clients usually use rarest-first download algorithm, but if you've mounted a movie and you're playing it with VLC, you'd probably want it to download sequentially so there's as little time spent buffering as possible. You could download the pieces on demand (whatever piece the application is trying to read), but I suspect that will provide a suboptimal experience, especially if your download speed is barely above the bitrate of the movie.

I have been using btfs for years so I can answer. It can handle videos and other streamable formats if there are enough seeds: the connection is not usually a problem. The pieces are not downloaded strictly sequentially but using a sliding window[1], so if a piece is not immediately available btfs will not waste time and bandwidth waiting for it but just keep downloading other pieces nearby. The problem is until enough pieces have been assembled, applications reading the file may literally hang: I guess not much software was made with slow I/O in mind. [1]: https://github.com/johang/btfs/issues/7#issuecomment-1916443...

There are all kinds of reasons a filesystem could be slower to serve certain bytes: striping across multiple heterogeneous physical disks, network filesystems like nfs/cifs/btfs, even IOp starvation. Video players can gracefully handle streaming media despite spotty connections; is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long? They could default to wanting all the bytes, but have a setting that instructs it to treat the filesystem as if it were a network source.

> is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long? AFAIK vlc has that functionality. It allows you to buffer the video n seconds in advance.

Setting a modest buffer size helps but does not solve this entirely. For example, turning on the subtitles or switching audio track will usually hang the player for a while.

Is there any way to get it to immediately start downloading the files, then prioritize the bits requested by applications reading the file? Also, very curious, how have you been using btfs? What have you been using it for?

I think this should be the default behavior: as soon as you mount the torrent btfs start fetching it like any other torrent client. I use it mainly for streaming videos: btfs comes with a small python script "btplay <magnet>" that will mount the torrent and start your player, very handy. Another cool thing you can do is downloading and extracting a file simultaneously, just mount a torrent and call "unzip".

I wrote https://github.com/anacrolix/torrent specifically to address the on-demand and streaming requirements of some products I created. It also doesn't make assumptions about how you store the data as you stream, so there's no requirement that the files exist on disk as they're described in the metainfo, like other clients.

libtorrent has repeatedly ignored requests for sequential download because of the silly spec. There's a patch out there that quite easily enables it. It's a bit absurd to me because there's a very good use case for sequential download and it's not clear there are any actual side effects.

Maybe, but probably less relevant for seeds that are not new. Also if people are streaming via torrents it's a net benefit if all peers have early bits not just random data. In short I do not have empirical evicidence but I do believe the decision to do so should not be made by a library and I also believe the intial assumption that random is better holds true today.

Yes, but only 2nd order... If all users downloaded in order, then users downloading the file cannot exchange chunks with other downloaders. That "exchange" is necessary to encourage users not to limit their upload speeds. In today's world where the vast majority of users just use the default settings for their torrent client, it probably doesn't matter much.

Libtorrent-rasterbar has supported sequential downloading for a long time. Are you sure you're not thinking of the other libtorrent (libtorrent-rakshasa)?

I'm not sure clients actually download rarest-first. It's supposed to be random. From The BitTorrent Protocol Specification, http://bittorrent.org/beps/bep_0003.html: > Downloaders generally download pieces in random order, which does a reasonably good job of keeping them from having a strict subset or superset of the pieces of any of their peers. However clients can certainly choose to download in whatever order they wish. Like Popcorn Time.

Can someone explain why I might use this instead of downloading a select group of files using a torrent file/magnet link? Is the advantage that it selectively downloads files on an as-needed basis? Or, is the product niche somehow related to the fact that the torrent file/magnet link is mounted as a filesystem, instead of only as a collection of files and folders?

The latter - it lets you navigate the file structure of a torrent as well as read (stream) file content using any of your usual applications, without having to download the files first.

Okay. I'm afraid to ask. What happens in the case of a) a torrent with no seeders or b) Has seeders with enough data to get a directory listing but either the seeders "drop off" afterwards or are incomplete Other experiences with network filesystems (CIFS/NFS) on multiple platforms sometimes have things getting hairy if the connection to the server drops while the filesystem is mounted or in use

I would be way more interested in this project if it downloaded normally in the background immediately after the torrent is added, and then downloaded specific bits as programs requested them. I haven't looked into the code base for this exactly but it says nothing about it in the Readme. Otherwise, like you're describing, you could add a file and then go to read it a week later and be met with no seeders and thus no file when it could have been grabbed somewhere in between. But maybe that extends beyond the capabilities of a file system?

This is a brilliant concept of a tool. I’ve never worked with monorepos before, but something like this with rw capacity would be an absolutely amazing pipeline, if every developer in the office and beyond are full nodes in the system.

Let's create a chain of immutable blocks with back pointers on top of this to create new versions of the data. Maybe even call it a... blockchain.

Neat idea, but it seems like abuse of the torrent protocol. If peers don't grab chunks randomly then it partially defeats the purpose of bittorrent.

Every torrent client I've used recently has an "enable sequential download" option. The only torrents this would realistically affect are maybe ones where the seeder may go offline when a bunch of clients have been doing sequential downloads and no one has the final parts of the content.

No. Hyperdrive is not like the official BEP-conform "mutable DHT item"-based updatable torrents. They essentially say "download the most recent version signed by this pubkey", which ia notably distinct from hyperdrive's append-only log.

"download the most recent version signed by this pubkey" is literally how hyperdrive works. The append-only log is hypercore, one of the building blocks of hyperdrive. Hyperdrive is a layer above and gives you a view of how the archive is at any point in time, with preferential access to the latest version. TBF what I understood from the request of "mutable torrents" wasn't the literal feature, but a way to have decentralized nodes exchanging updatable content. Bittorent with the Mutable Torrents BEP does work, but it's nowhere near as developed as what hyperdrive is today. And as sad as it may sound I don't think it will ever be.

Isn't that just IPFS? And forget piracy; it's great for ready access to any data that you want mirrored to a lot of servers at once - imagine if you mounted /var/cache/yum over BTFS:) (...maybe with an overlay filesystem in the middle; I forget the internals of yum's cache...)

If you make your own torrent files, how to prevent it accidentally share with the world? Is there anything in the torrent files that can prevent this? (Program and configuration can make mistake.)

Yes, there are BEPs that describe not announcing or accepting connections in certain circumstances. There are also peer discovery mechanisms that are designed to work on local networks. Of course none of this matters if you mess up (and someone externally cares enough to take your data), or if your client doesn't support any of this stuff.

“Why this project updated yesterday and not the project last updated 3 years ago?” Unless you mean a different one since you didn’t link source

It also depends on which client ID it exposes and if that one is whitelisted on trackers. But that's not really a technology problem.

That answer probably lies more so with the tracker's software and approved clients than this project's implementation.

Well that is neat, albeit an unfortunate name (1-char off from BTRFS). Wonder what performance is like - does it try to preemptively download? Cache? I would expect it to take a while to run `cp ~/mnt_btfs/big_file ~/Downloads/`, but how about `find ~/mnt_btfs`? If it's performant, it could be really nifty:)

I'm on such a BTRFS kick right now that I reflexively up voted. Oops. Cool project though; I regret nothing!

It looks like the BTFS in the OP was the "original", with its first commit in July of 2015. Could be wrong, because the GitHub interface doesn't make it easy to get to the beginning of the 13k commits for go-btfs.

Yes, OP's btfs was definitely first - in Tron's git history you'll also see that it's a fork of go-ipfs, so the vast majority of commits is from IPFS people. However it's good to see that Tron are no longer trying to hide the fact it's a fork. For a long time they didn't provide the source code of their btfs.

At least you can always clone and then just ask git... EDIT: Actually, since this was a fork, it would probably be a pain to hunt down, even with a local checkout. But it usually works.

9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac 2014-06-26 Big things have small beginnings. (I have an alias specifically for finding that out) Personnally, I don't care about any of the crytpo-mining-coining and I think the two are sufficiently distinct.

That first commit is to the repo it is forked from, which was go-ipfs... so it didn't get the name until the fork at a later time.

The TRON Project is also a 1980-1990s project in Japan to develop IoT infrastructure. They even made their own CPUs and developed 3 different flavours of Operating Systems: - an embedded real-time kernel ITRON - an soft real time single user desktop OS called BTRON - CTRON a server operating systems with real time guarantees for packet/phone line switching In addition they also developed an true international character encoding before Unicode. It wasn't ASCII backwards compatible which is why it wasn't picked by Unicode committee. A good resource to learn about it is here: http://tronweb.super-nova.co.jp/homepage.html

There's also an amazing full length Daft Punk music video that sometimes is considered a sequel to the 80s film, but you'll likely want to watch the spin-off cartoon for a lot of the necessary plot. In the plot of the cartoon the titular character (program) Tron is assumed dead, and then corrupted by a corrupt system. I can only assume this is why his name is being bad mouthed by some sort of crypto-currency scheme, as Tron was a good program, a security program built to defend the users and would never stand silent on for-profit exploitation of users.

If you are only using torrents for "stolen archives", then you don't know about OS distribution via torrents and archive.org? And we use it internally to distribute large files.

> With BTFS, you can mount any .torrent file or magnet link and then use it as any /read-only/ directory in your file tree.

At first I wasn't sure if you were being too serious or just literal when that wasn't needed, but then I realized that you didn't read my full comment.

? At this writing, the full comment is: > Awesome! People can now use standard tools for adding malware into stolen archives! /s But the most charitable reading I can think of is that you could use BTFS to pirate something, and very slightly reduce the number of steps when modifying it? I guess? But that's no different than just using a normal torrent client and then modifying the files once downloaded. It's just a different interface. So... what context am I missing?

Generally, /s in comments is shorthand for /sarc, a closing tag that indicates the prior comment is sarcasm. Thanks for being a good sport about it!

Просмотры: