Login Register






Poll: Good idea?
You do not have permission to vote in this poll.
Fuck yeah, count me in!
100.00%
2 100.00%
This isn't for me, sorry.
0%
0 0%
I don't understand this
0%
0 0%
Total 2 vote(s) 100%
* You voted for this item. [Show Results]

Thread Rating:
  • 0 Vote(s) - 0 Average


Want to make a filesystem. filter_list
Author
Message
Want to make a filesystem. #1
So, after having some talks about MacOS security about encryption and how it should always be on, but it's usually not; along with my desire to make a distributed file system (think tor but for files and with native OS support). I've come up with an idea. I want to build a distributed file system with multilayer encryption built in from the ground up.

What does this mean?
It means your files won't necessarily be stored on your machine, but rather they'd be broken up into small segments and distributed over the network. This adds both security, and the ability to have large disks available wherever you are.
It also means that your files will be secure, with encryption built in from the ground up, the files on disk will always be encrypted so nobody can get at them.
Lastly, it will have encryption built into transfers. My idea is that the file as a whole is encrypted with YOUR key, and then each individual shard is encrypted with the public key of it's storage node. When the files are sent back to you, they're encrypted with your public key so that nobody can intercept the shards.

Obviously there would need to be redundancy built in (shards stored on more than one node for both speed and recovery), but I think it's a cool concept. It closely resembles a filesystem built on the concept of torrenting.

Here is an example framework I was considering using that uses FUSE as tha backbone, allowing our filesystem driver to run in userland (and work on other operating systems). What do you guys think? Would you be willing to take part in either the dev or setting up the original network of nodes?

Reply

RE: Want to make a filesystem. #2
Reminds me of STORJ and Sia (I have money in both)

Sounds like a cool project, I've wanted to do something like this before.


(11-02-2018, 02:51 AM)Skullmeat Wrote: Ok, there no real practical reason for doing this, but that's never stopped me.

Reply

RE: Want to make a filesystem. #3
(12-01-2017, 02:02 AM)Ender Wrote: Reminds me of STORJ and Sia (I have money in both)

Sounds like a cool project, I've wanted to do something like this before.

That's exactly where I got the "distributed" idea from. Theirs was crap and written in Node though, and didn't have crypto.

[+] 1 user Likes phyrrus9's post
Reply

RE: Want to make a filesystem. #4
https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.
Keep in mind that in distributed systems are much more complicated than their centralized counterparts. I am sure you will have no problem implementing the filesystem, but there is the entire p2p networking stack to worry about as well.
If the purpose is simply to store your own encrypted files securely, there is not enough incentive for other nodes to bear the cost of storing your encrypted data, which is useless to them. At this point you might as well just devise a centralized FS built for data redundancy and use of encryption.
Where distributed file systems really shine is not as a vault for selfish encrypted data storage but rather a cryptographically authenticated public lookup table of content, wherein each file is usually addressed by its hash. This allows nodes that share a file(s) of interest to efficiently store parts of the content, and share said parts among themselves.

In other words, there's no point distributing data you want to keep to yourself, especially since there is nothing to gain from someone storing your encrypted data for you. If you want the redundancy/fault tolerance then just write a normal FS for that IMHO.

Reply

RE: Want to make a filesystem. #5
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed. The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this. He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once. Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible. In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.
(This post was last modified: 12-01-2017, 07:05 AM by Blink.)


(11-02-2018, 02:51 AM)Skullmeat Wrote: Ok, there no real practical reason for doing this, but that's never stopped me.

Reply

RE: Want to make a filesystem. #6
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.
Keep in mind that in distributed systems are much more complicated than their centralized counterparts. I am sure you will have no problem implementing the filesystem, but there is the entire p2p networking stack to worry about as well.
If the purpose is simply to store your own encrypted files securely, there is not enough incentive for other nodes to bear the cost of storing your encrypted data, which is useless to them. At this point you might as well just devise a centralized FS built for data redundancy and use of encryption.
Where distributed file systems really shine is not as a vault for selfish encrypted data storage but rather a cryptographically authenticated public lookup table of content, wherein each file is usually addressed by its hash. This allows nodes that share a file(s) of interest to efficiently store parts of the content, and share said parts among themselves.

In other words, there's no point distributing data you want to keep to yourself, especially since there is nothing to gain from someone storing your encrypted data for you. If you want the redundancy/fault tolerance then just write a normal FS for that IMHO.

That's exactly the opposite of what I'm talking about. What I'm proposing is using distributed hash tables, not a blockchain. I'm aiming for decentralized storage (there's no central ledger). Distributed hash tables are actually very simple, it's the network resolution of them that gets difficult, luckily TCP/IP and ARP exists and I don't have to do any of that sort of stuff. The point of making this decentralized is so that there is a near unlimited size of the disk, and it's very difficult for an attacker to piece together any part of it let alone a single file from the user. As for files referenced by hashes, this is what I was talking about. It's a distributed hash table. You can take a look at this project on my github about how hash maps work, DHTs are just a reduntant and decentralized superset of hash maps. The redundancy isn't a requirement of having a filesystem, it's a requirement of this specific type of file system. If one node goes offline, you would lose the data that's stored on that node, but if you had redundant nodes, no data is lost and another node replicates the missing data for future redundancy. It's there as a safety measure. Think of it like a RAID array. Each drive doesn't contain all of the data, but rather a piece of it (stored on a stripe). If one drive fails (assuming there's no parity), you would've lost the data on those stripes, but you wouldn't have lost the whole file. This isn't the best example because RAID makes use of parity (which would be a security vulnerability for this scenario), but the general concept is the same.

I understand the points you're trying to make, but none of them are really valid for this case. File systems, peer networking, graph theory, and kernel development are all particularly advanced topics, and I applaud you for joining in this conversation. You should track this project as time goes on, you could really pick up a lot of useful knowledge

(12-01-2017, 07:05 AM)Ender Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed. The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this. He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once. Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible. In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

No, Dijkstras wouldn't be used here. It's a very inefficient algorithm that's very simple. It makes a great teaching tool for basic graph theory but it's not particularly useful in practice. We'd be using a variant of least cost routing and a tuned hash algorithm to determine the fastest route to nodes as well as the optimal placement for redundant and primary nodes.

[+] 2 users Like phyrrus9's post
Reply

RE: Want to make a filesystem. #7
(12-01-2017, 07:05 AM)Ender Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-01-2017, 07:14 AM)phyrrus9 Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.
Keep in mind that in distributed systems are much more complicated than their centralized counterparts. I am sure you will have no problem implementing the filesystem, but there is the entire p2p networking stack to worry about as well.
If the purpose is simply to store your own encrypted files securely, there is not enough incentive for other nodes to bear the cost of storing your encrypted data, which is useless to them. At this point you might as well just devise a centralized FS built for data redundancy and use of encryption.
Where distributed file systems really shine is not as a vault for selfish encrypted data storage but rather a cryptographically authenticated public lookup table of content, wherein each file is usually addressed by its hash. This allows nodes that share a file(s) of interest to efficiently store parts of the content, and share said parts among themselves.

In other words, there's no point distributing data you want to keep to yourself, especially since there is nothing to gain from someone storing your encrypted data for you. If you want the redundancy/fault tolerance then just write a normal FS for that IMHO.

That's exactly the opposite of what I'm talking about. What I'm proposing is using distributed hash tables, not a blockchain. I'm aiming for decentralized storage (there's no central ledger). Distributed hash tables are actually very simple, it's the network resolution of them that gets difficult, luckily TCP/IP and ARP exists and I don't have to do any of that sort of stuff. The point of making this decentralized is so that there is a near unlimited size of the disk, and it's very difficult for an attacker to piece together any part of it let alone a single file from the user. As for files referenced by hashes, this is what I was talking about. It's a distributed hash table. You can take a look at this project on my github about how hash maps work, DHTs are just a reduntant and decentralized superset of hash maps. The redundancy isn't a requirement of having a filesystem, it's a requirement of this specific type of file system. If one node goes offline, you would lose the data that's stored on that node, but if you had redundant nodes, no data is lost and another node replicates the missing data for future redundancy. It's there as a safety measure. Think of it like a RAID array. Each drive doesn't contain all of the data, but rather a piece of it (stored on a stripe). If one drive fails (assuming there's no parity), you would've lost the data on those stripes, but you wouldn't have lost the whole file. This isn't the best example because RAID makes use of parity (which would be a security vulnerability for this scenario), but the general concept is the same.

I understand the points you're trying to make, but none of them are really valid for this case. File systems, peer networking, graph theory, and kernel development are all particularly advanced topics, and I applaud you for joining in this conversation. You should track this project as time goes on, you could really pick up a lot of useful knowledge

(12-01-2017, 07:05 AM)Ender Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

No, Dijkstras wouldn't be used here. It's a very inefficient algorithm that's very simple. It makes a great teaching tool for basic graph theory but it's not particularly useful in practice. We'd be using a variant of least cost routing and a tuned hash algorithm to determine the fastest route to nodes as well as the optimal placement for redundant and primary nodes.

IPFS doesn't use a blockchain, actually. You seem to confuse IPFS with conventional blockchain protocols like BTC and Ethereum which use hash trees, and generally don't scale well to handle mass storage as a result of their architecture.
IPFS on the other hand does not use a strict tree but more of a generalized directed graph model. The graph is necessarily acyclic as long as the properties of the hash function used will hold true. Furthermore, like you described in your design, full files are addressed by their hash and refer to individual blocks of content which are distributed across the network. On another note: IPFS is accessible through FUSE as well.
DHTs form the basis of pretty much every peer to peer network protocol in one form or another, so that's nothing special really. Even blockchain systems like Ethereum use DHTs for node discovery. They are trivial to implement because most of the code is already there. So no, it won't be difficult to get the protocol itself working as it is pretty simple itself - my claim is that because the files being stored on the network are encrypted, stored with the intention of the owner being the sole retriever thereof, there is not enough incentive for nodes to handle storage of other nodes' data. It is simply more cost-effective, and arguably more practical for each person to use a centralized filesystem with parity-based redundancy to store the data in an encrypted format, rather than to offload it on to a distributed network and have to handle the storage of parts of others' encrypted data in order to ensure data redundancy.

That said, I am happy to help and contribute to this project for research purposes.

Reply

RE: Want to make a filesystem. #8
(12-01-2017, 11:45 PM)phaz0n Wrote:
(12-01-2017, 07:05 AM)Ender Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-01-2017, 07:14 AM)phyrrus9 Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.
Keep in mind that in distributed systems are much more complicated than their centralized counterparts. I am sure you will have no problem implementing the filesystem, but there is the entire p2p networking stack to worry about as well.
If the purpose is simply to store your own encrypted files securely, there is not enough incentive for other nodes to bear the cost of storing your encrypted data, which is useless to them. At this point you might as well just devise a centralized FS built for data redundancy and use of encryption.
Where distributed file systems really shine is not as a vault for selfish encrypted data storage but rather a cryptographically authenticated public lookup table of content, wherein each file is usually addressed by its hash. This allows nodes that share a file(s) of interest to efficiently store parts of the content, and share said parts among themselves.

In other words, there's no point distributing data you want to keep to yourself, especially since there is nothing to gain from someone storing your encrypted data for you. If you want the redundancy/fault tolerance then just write a normal FS for that IMHO.

That's exactly the opposite of what I'm talking about. What I'm proposing is using distributed hash tables, not a blockchain. I'm aiming for decentralized storage (there's no central ledger). Distributed hash tables are actually very simple, it's the network resolution of them that gets difficult, luckily TCP/IP and ARP exists and I don't have to do any of that sort of stuff. The point of making this decentralized is so that there is a near unlimited size of the disk, and it's very difficult for an attacker to piece together any part of it let alone a single file from the user. As for files referenced by hashes, this is what I was talking about. It's a distributed hash table. You can take a look at this project on my github about how hash maps work, DHTs are just a reduntant and decentralized superset of hash maps. The redundancy isn't a requirement of having a filesystem, it's a requirement of this specific type of file system. If one node goes offline, you would lose the data that's stored on that node, but if you had redundant nodes, no data is lost and another node replicates the missing data for future redundancy. It's there as a safety measure. Think of it like a RAID array. Each drive doesn't contain all of the data, but rather a piece of it (stored on a stripe). If one drive fails (assuming there's no parity), you would've lost the data on those stripes, but you wouldn't have lost the whole file. This isn't the best example because RAID makes use of parity (which would be a security vulnerability for this scenario), but the general concept is the same.

I understand the points you're trying to make, but none of them are really valid for this case. File systems, peer networking, graph theory, and kernel development are all particularly advanced topics, and I applaud you for joining in this conversation. You should track this project as time goes on, you could really pick up a lot of useful knowledge

(12-01-2017, 07:05 AM)Ender Wrote: Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

No, Dijkstras wouldn't be used here. It's a very inefficient algorithm that's very simple. It makes a great teaching tool for basic graph theory but it's not particularly useful in practice. We'd be using a variant of least cost routing and a tuned hash algorithm to determine the fastest route to nodes as well as the optimal placement for redundant and primary nodes.

IPFS doesn't use a blockchain, actually. You seem to confuse IPFS with conventional blockchain protocols like BTC and Ethereum which use hash trees, and generally don't scale well to handle mass storage as a result of their architecture.
IPFS on the other hand does not use a strict tree but more of a generalized directed graph model. The graph is necessarily acyclic as long as the properties of the hash function used will hold true. Furthermore, like you described in your design, full files are addressed by their hash and refer to individual blocks of content which are distributed across the network. On another note: IPFS is accessible through FUSE as well.
DHTs form the basis of pretty much every peer to peer network protocol in one form or another, so that's nothing special really. Even blockchain systems like Ethereum use DHTs for node discovery. They are trivial to implement because most of the code is already there. So no, it won't be difficult to get the protocol itself working as it is pretty simple itself - my claim is that because the files being stored on the network are encrypted, stored with the intention of the owner being the sole retriever thereof, there is not enough incentive for nodes to handle storage of other nodes' data. It is simply more cost-effective, and arguably more practical for each person to use a centralized filesystem with parity-based redundancy to store the data in an encrypted format, rather than to offload it on to a distributed network and have to handle the storage of parts of others' encrypted data in order to ensure data redundancy.

That said, I am happy to help and contribute to this project for research purposes.

Again, I understand your desire to play yourself up here, but we aren't talking about IPFS. I'm not generally interested in constantly reading the wikipedia entry for IPFS. We don't want blockchain, period. The point is to be entirely self sustaining, firewall independent, fully encrypted, and completely decentralized. These are features that IPFS doesn't offer, and using IPFS defeats the entire purpose of the project (kernel development).

Reply

RE: Want to make a filesystem. #9
(12-01-2017, 07:05 AM)Ender Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-02-2017, 12:01 AM)phyrrus9 Wrote:
(12-01-2017, 11:45 PM)phaz0n Wrote:
(12-01-2017, 07:05 AM)Ender Wrote: Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-01-2017, 07:14 AM)phyrrus9 Wrote: That's exactly the opposite of what I'm talking about. What I'm proposing is using distributed hash tables, not a blockchain. I'm aiming for decentralized storage (there's no central ledger). Distributed hash tables are actually very simple, it's the network resolution of them that gets difficult, luckily TCP/IP and ARP exists and I don't have to do any of that sort of stuff. The point of making this decentralized is so that there is a near unlimited size of the disk, and it's very difficult for an attacker to piece together any part of it let alone a single file from the user. As for files referenced by hashes, this is what I was talking about. It's a distributed hash table. You can take a look at this project on my github about how hash maps work, DHTs are just a reduntant and decentralized superset of hash maps. The redundancy isn't a requirement of having a filesystem, it's a requirement of this specific type of file system. If one node goes offline, you would lose the data that's stored on that node, but if you had redundant nodes, no data is lost and another node replicates the missing data for future redundancy. It's there as a safety measure. Think of it like a RAID array. Each drive doesn't contain all of the data, but rather a piece of it (stored on a stripe). If one drive fails (assuming there's no parity), you would've lost the data on those stripes, but you wouldn't have lost the whole file. This isn't the best example because RAID makes use of parity (which would be a security vulnerability for this scenario), but the general concept is the same.

I understand the points you're trying to make, but none of them are really valid for this case. File systems, peer networking, graph theory, and kernel development are all particularly advanced topics, and I applaud you for joining in this conversation. You should track this project as time goes on, you could really pick up a lot of useful knowledge


No, Dijkstras wouldn't be used here. It's a very inefficient algorithm that's very simple. It makes a great teaching tool for basic graph theory but it's not particularly useful in practice. We'd be using a variant of least cost routing and a tuned hash algorithm to determine the fastest route to nodes as well as the optimal placement for redundant and primary nodes.

IPFS doesn't use a blockchain, actually. You seem to confuse IPFS with conventional blockchain protocols like BTC and Ethereum which use hash trees, and generally don't scale well to handle mass storage as a result of their architecture.
IPFS on the other hand does not use a strict tree but more of a generalized directed graph model. The graph is necessarily acyclic as long as the properties of the hash function used will hold true. Furthermore, like you described in your design, full files are addressed by their hash and refer to individual blocks of content which are distributed across the network. On another note: IPFS is accessible through FUSE as well.
DHTs form the basis of pretty much every peer to peer network protocol in one form or another, so that's nothing special really. Even blockchain systems like Ethereum use DHTs for node discovery. They are trivial to implement because most of the code is already there. So no, it won't be difficult to get the protocol itself working as it is pretty simple itself - my claim is that because the files being stored on the network are encrypted, stored with the intention of the owner being the sole retriever thereof, there is not enough incentive for nodes to handle storage of other nodes' data. It is simply more cost-effective, and arguably more practical for each person to use a centralized filesystem with parity-based redundancy to store the data in an encrypted format, rather than to offload it on to a distributed network and have to handle the storage of parts of others' encrypted data in order to ensure data redundancy.

That said, I am happy to help and contribute to this project for research purposes.

Again, I understand your desire to play yourself up here, but we aren't talking about IPFS. I'm not generally interested in constantly reading the wikipedia entry for IPFS. We don't want blockchain, period. The point is to be entirely self sustaining, firewall independent, fully encrypted, and completely decentralized. These are features that IPFS doesn't offer, and using IPFS defeats the entire purpose of the project (kernel development).

I'm just saying there is no blockchain in IPFS, and your concept has some non-technical flaws which undermine the practicality of its usage.. I would also appreciate if you avoided using subtle ad-hominems in place of an argument against my claim. That is all.
(This post was last modified: 12-02-2017, 12:11 AM by phaz0n.)

Reply

RE: Want to make a filesystem. #10
(12-02-2017, 12:07 AM)phaz0n Wrote:
(12-01-2017, 07:05 AM)Ender Wrote:
(12-01-2017, 06:56 AM)phaz0n Wrote: https://en.wikipedia.org/wiki/InterPlane...ile_System

This seems to do a lot of what you are talking about.

Close, but not exactly.
Blockchain-based systems keep everything in one chain, which is distributed.  The problem with this in currencies, is that there is the possibility of a 51% attack.

Earlier, I was talking to @"phyrrus9" about this.  He is planning on using a distributed hash table for this, which is somewhat similar to a blockchain, but is split up into multiple parts, with no single node having all of it at once.  Then an algorithm (he pointed me to Dijkstra's algorithm when talking about cryptocurrencies, not sure if that's what's being used here though) is used to distribute the data in the fastest way possible.  In cryptocurrencies, this adds an extra layer of protection, since it's extremely hard to figure out how to make a fake valid transaction that gets routed to your specific node, and takes more computational power than is needed in Bitcoin and other systems.

(12-02-2017, 12:01 AM)phyrrus9 Wrote:
(12-01-2017, 11:45 PM)phaz0n Wrote: IPFS doesn't use a blockchain, actually. You seem to confuse IPFS with conventional blockchain protocols like BTC and Ethereum which use hash trees, and generally don't scale well to handle mass storage as a result of their architecture.
IPFS on the other hand does not use a strict tree but more of a generalized directed graph model. The graph is necessarily acyclic as long as the properties of the hash function used will hold true. Furthermore, like you described in your design, full files are addressed by their hash and refer to individual blocks of content which are distributed across the network. On another note: IPFS is accessible through FUSE as well.
DHTs form the basis of pretty much every peer to peer network protocol in one form or another, so that's nothing special really. Even blockchain systems like Ethereum use DHTs for node discovery. They are trivial to implement because most of the code is already there. So no, it won't be difficult to get the protocol itself working as it is pretty simple itself - my claim is that because the files being stored on the network are encrypted, stored with the intention of the owner being the sole retriever thereof, there is not enough incentive for nodes to handle storage of other nodes' data. It is simply more cost-effective, and arguably more practical for each person to use a centralized filesystem with parity-based redundancy to store the data in an encrypted format, rather than to offload it on to a distributed network and have to handle the storage of parts of others' encrypted data in order to ensure data redundancy.

That said, I am happy to help and contribute to this project for research purposes.

Again, I understand your desire to play yourself up here, but we aren't talking about IPFS. I'm not generally interested in constantly reading the wikipedia entry for IPFS. We don't want blockchain, period. The point is to be entirely self sustaining, firewall independent, fully encrypted, and completely decentralized. These are features that IPFS doesn't offer, and using IPFS defeats the entire purpose of the project (kernel development).

I'm just saying there is no blockchain in IPFS, and your concept has some non-technical flaws. I would also appreciate if you avoided using subtle ad-hominems in place of an argument against my claim.

The "technical flaws" you refer to are easily solved with engineering. You would incentivize in the same way that bt works. The more blocks you are willing to store, the more likely another will be willing to store your blocks, and balance the table so that you store a minimal amount of your own blocks.

What I don't appreciate is you've been off topic this whole time. The thread is about building a filesystem, not finding an existing one. The end result has nothing to do with the filesystem created and everything to do with kernel development, as I've stated. It could be a completely useless dummy project and it would still be valid. I've never heard of you before this thread, and then all of the sudden you come in quoting wikipedia and suggesting I use bitcoin technology, I'm not interested in that. If you want to be constructive to the original topic, I welcome it, but if you're just going to debate why your opinions are better than mine then you should find a different thread.

Reply







Users browsing this thread: 2 Guest(s)