(12-02-2017, 12:07 AM)phaz0n Wrote:

(12-01-2017, 07:05 AM)Ender Wrote:

(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlanetary_File_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-02-2017, 12:01 AM)phyrrus9 Wrote:

(12-01-2017, 11:45 PM)phaz0n Wrote: IPFS doesn't use a blockchain, actually. You seem to confuse IPFS with conventional blockchain protocols like BTC and Ethereum which use hash trees, and generally don't scale well to handle mass storage as a result of their architecture.
IPFS on the other hand does not use a strict tree but more of a generalized directed graph model. The graph is necessarily acyclic as long as the properties of the hash function used will hold true. Furthermore, like you described in your design, full files are addressed by their hash and refer to individual blocks of content which are distributed across the network. On another note: IPFS is accessible through FUSE as well.
DHTs form the basis of pretty much every peer to peer network protocol in one form or another, so that's nothing special really. Even blockchain systems like Ethereum use DHTs for node discovery. They are trivial to implement because most of the code is already there. So no, it won't be difficult to get the protocol itself working as it is pretty simple itself - my claim is that because the files being stored on the network are encrypted, stored with the intention of the owner being the sole retriever thereof, there is not enough incentive for nodes to handle storage of other nodes' data. It is simply more cost-effective, and arguably more practical for each person to use a centralized filesystem with parity-based redundancy to store the data in an encrypted format, rather than to offload it on to a distributed network and have to handle the storage of parts of others' encrypted data in order to ensure data redundancy.

That said, I am happy to help and contribute to this project for research purposes.

Again, I understand your desire to play yourself up here, but we aren't talking about IPFS. I'm not generally interested in constantly reading the wikipedia entry for IPFS. We don't want blockchain, period. The point is to be entirely self sustaining, firewall independent, fully encrypted, and completely decentralized. These are features that IPFS doesn't offer, and using IPFS defeats the entire purpose of the project (kernel development).

I'm just saying there is no blockchain in IPFS, and your concept has some non-technical flaws which undermine the practicality of its usage.. I would also appreciate if you avoided using subtle ad-hominems in place of an argument against my claim. That is all.

(12-02-2017, 12:14 AM)phyrrus9 Wrote:

(12-02-2017, 12:07 AM)phaz0n Wrote:

(12-01-2017, 07:05 AM)Ender Wrote: Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-02-2017, 12:01 AM)phyrrus9 Wrote: Again, I understand your desire to play yourself up here, but we aren't talking about IPFS. I'm not generally interested in constantly reading the wikipedia entry for IPFS. We don't want blockchain, period. The point is to be entirely self sustaining, firewall independent, fully encrypted, and completely decentralized. These are features that IPFS doesn't offer, and using IPFS defeats the entire purpose of the project (kernel development).

I'm just saying there is no blockchain in IPFS, and your concept has some non-technical flaws. I would also appreciate if you avoided using subtle ad-hominems in place of an argument against my claim.

The "technical flaws" you refer to are easily solved with engineering. You would incentivize in the same way that bt works. The more blocks you are willing to store, the more likely another will be willing to store your blocks, and balance the table so that you store a minimal amount of your own blocks.

What I don't appreciate is you've been off topic this whole time. The thread is about building a filesystem, not finding an existing one. The end result has nothing to do with the filesystem created and everything to do with kernel development, as I've stated. It could be a completely useless dummy project and it would still be valid. I've never heard of you before this thread, and then all of the sudden you come in quoting wikipedia and suggesting I use bitcoin technology, I'm not interested in that. If you want to be constructive to the original topic, I welcome it, but if you're just going to debate why your opinions are better than mine then you should find a different thread.

(12-02-2017, 12:31 AM)phaz0n Wrote: 6. Making a FUSE driver doesn't make it kernel development. A kernel manages hardware. A file system manages data, and uses the kernel for the hardware part of it. You said you wanted to make a distributed encrypted file system, which has nothing to do with kernel development.

(12-02-2017, 12:59 AM)Ender Wrote:

(12-02-2017, 12:31 AM)phaz0n Wrote: 6. Making a FUSE driver doesn't make it kernel development. A kernel manages hardware. A file system manages data, and uses the kernel for the hardware part of it. You said you wanted to make a distributed encrypted file system, which has nothing to do with kernel development.

Ignoring FUSE for a second, the filesystem is usually implemented in the kernel.

FUSE abstracts some of the stuff that is done when building from scratch, but one can still call it kernel development and not be wrong.

(12-02-2017, 01:13 AM)phaz0n Wrote: Mixing in p2p networking and distributed systems makes it even further apart from kernel dev, and calling userspace filesystem implementation combined with those things "kernel development" would be an insult to kernel development.

(12-02-2017, 12:31 AM)phaz0n Wrote:

(12-02-2017, 12:14 AM)phyrrus9 Wrote:

(12-02-2017, 12:07 AM)phaz0n Wrote: I'm just saying there is no blockchain in IPFS, and your concept has some non-technical flaws. I would also appreciate if you avoided using subtle ad-hominems in place of an argument against my claim.

The "technical flaws" you refer to are easily solved with engineering. You would incentivize in the same way that bt works. The more blocks you are willing to store, the more likely another will be willing to store your blocks, and balance the table so that you store a minimal amount of your own blocks.

What I don't appreciate is you've been off topic this whole time. The thread is about building a filesystem, not finding an existing one. The end result has nothing to do with the filesystem created and everything to do with kernel development, as I've stated. It could be a completely useless dummy project and it would still be valid. I've never heard of you before this thread, and then all of the sudden you come in quoting wikipedia and suggesting I use bitcoin technology, I'm not interested in that. If you want to be constructive to the original topic, I welcome it, but if you're just going to debate why your opinions are better than mine then you should find a different thread.

1. They are non-technical flaws. The solution you have devised is correct, but does not change the fact that there is a more effective means of achieving the same end.
2. Yes, you can reinvent the wheel if you want. It will be a waste of code and time, yet a valuable learning experience and lots of fun.
3. You don't need to have heard from me, I am simply another user and I would like to share my point. This is a forum, and people will criticize your idea. Get used to it.
4. I am not quoting wikipedia - I am citing the IPLD specification located here: https://github.com/ipld/specs/tree/master/ipld
5. I haven't been arguing that my opinion is better than yours, I am helping you by giving you insight on potential design issues that may cause problems for real world use. Of course this is not an issue if it is a dummy project as you had mentioned, but then the initial "kernel development" point you made, which by the way was not mentioned at all in your original post, becomes void.
6. Making a FUSE driver doesn't make it kernel development. A kernel manages hardware. A file system manages data, and uses the kernel for the hardware part of it. You said you wanted to make a distributed encrypted file system, which has nothing to do with kernel development.

(12-02-2017, 01:13 AM)phaz0n Wrote:

(12-02-2017, 12:59 AM)Ender Wrote:

(12-02-2017, 12:31 AM)phaz0n Wrote: 6. Making a FUSE driver doesn't make it kernel development. A kernel manages hardware. A file system manages data, and uses the kernel for the hardware part of it. You said you wanted to make a distributed encrypted file system, which has nothing to do with kernel development.

Ignoring FUSE for a second, the filesystem is usually implemented in the kernel.

FUSE abstracts some of the stuff that is done when building from scratch, but one can still call it kernel development and not be wrong.

Yes, support for filesystems is implemented in the kernel in most cases. FUSE, on the other hand, is a software interface for userspace filesystems drivers, which make it not kernel dev. However, while filesystem support is a kernel feature, I would not call it a part of kernel development. File system design is a very different beast, only tangentially related to the kernel in the sense that the kernel can manage files on a block device according to the specification of the FS. Mixing in p2p networking and distributed systems makes it even further apart from kernel dev, and calling userspace filesystem implementation combined with those things "kernel development" would be an insult to kernel development.

(12-02-2017, 01:24 AM)phyrrus9 Wrote: Additionally, we don't need an OSdev lecture, we both have a lot of experience with that sort of stuff. I don't think you understand what kernel dev is. If it's part of the kernel, it's kernel dev. The kernel can run in userland and still be part of the kernel (do some research on BSD, XNU, and NT). We aren't trying to put you down, but you're trying to assert yourself here and it's not the place. I was like you when I was young (look at some of my posts from 5 years ago, you'll catch the tail end of my high school days).

(12-02-2017, 05:51 AM)phaz0n Wrote:

(12-02-2017, 01:24 AM)phyrrus9 Wrote: Additionally, we don't need an OSdev lecture, we both have a lot of experience with that sort of stuff. I don't think you understand what kernel dev is. If it's part of the kernel, it's kernel dev. The kernel can run in userland and still be part of the kernel (do some research on BSD, XNU, and NT). We aren't trying to put you down, but you're trying to assert yourself here and it's not the place. I was like you when I was young (look at some of my posts from 5 years ago, you'll catch the tail end of my high school days).

I don't need to hear minute exceptions about microkernels and hybrids that perform much of their functions through userspace servers and drivers (which themselves are not part of the kernel), that's irrelevant to the current case so you're just spouting shit.
You're not even correct about the BSD kernel, which itself is monolithic.

Furthermore, you seem to enjoy making incorrect assumptions about my age and using this as part of your argument. That's weak. In a previous post you accused me of being off topic in this thread by using examples of other p2p networks to make a comparison (while also erroneously claiming IPFS uses a blockchain database). Nonetheless, you continue to make off-topic arguments that generally target myself rather than my claim. I am unsure whether this is a tactic intended to spite me or simply a result of your pretentiousness, but it's highly unproductive. I am not here to assert myself. I have bigger things to worry about than my reputation on some scrappy tech forum I visit occasionally. I am here to make an argument, and the discussion has been mostly derailed from the technology in favor of passive-aggressive ad-hominems regarding my age, simply because I criticized a facet of your project.
It's poor form to turn to targeting a person in an argument, even more so if you lack the information needed to make such statements. For the number of instances of comments regarding maturity you have made, it seems you have a way to go.
Dealing with this low level of discussion isn't worth the time anymore.

(12-02-2017, 05:51 AM)phaz0n Wrote: I have bigger things to worry about than my reputation on some scrappy tech forum I visit occasionally. I am here to make an argument

(12-01-2017, 11:45 PM)phaz0n Wrote: That said, I am happy to help and contribute to this project for research purposes.

(12-01-2017, 01:55 AM)phyrrus9 Wrote: I want to build a distributed file system with multilayer encryption built in from the ground up.