Any file system that allows access to files from multiple hosts sharing via a computer network making it possible for multiple users on multiple machines to share files and storage resources.
Questions tagged [distributed-filesystem]
56 questions
2
votes
0 answers
BeeGFS very slow after update
I am running a cluster with 1 head node, 40 compute nodes, 2 data and 2 meta nodes. All nodes are running Ubuntu 20.04, kernel version 5.4.0-100-generic. They are all connected via 10 Gbit/s interfaces expect for the data and meta nodes, which are…
Targon
- 153
- 4
2
votes
1 answer
IPFS search file mechanism
I am using IPFS(Inter Planetary File System) to store documents/files in a decentralized manner.
In order to search a file from the network, is there a record of all the hashes on the network(like leeches)?
How does my request travel through the…
Chamin Wickramarathna
- 1,672
- 2
- 20
- 34
2
votes
1 answer
Alluxio with/without HDFS
I have a cluster with HDFS as an under storage distributed file system, but I've just read about alluxio that is fast and flexible. So, My question is: Should I use Alluxio with HDFS or Alluxio is alternative for HDFS? (I see in their site that…
DAVID_ROA
- 309
- 1
- 3
- 18
2
votes
1 answer
Can Flink output be sinked to a NFS or GPFS file system?
Flink has a RollingFileSink which per the documentation works with HDFS (Hadoop). Are there similar classes which can be used to sink data to file systems such as NFS, ext4 or…
victtim
- 790
- 5
- 17
2
votes
0 answers
Network usage of file open and seek in distributed filesystem like ceph
When I open a file stored in another node in a distributed filesystem and just read the 100 bytes. Does the filesystem try to "prefetch" more data to my node? E.g., The network traffic for sending the file is more than 100 bytes.
Another question is…
Helin Wang
- 4,002
- 1
- 30
- 34
2
votes
2 answers
is hadoop DistributedFileSystem thread safe?
I am using hadoop for writing data I scrape.
I have a spring service that is called from multiple threads to write some content to the HDFS.
@Service
public class WriteService
{
public void write(String path, String content)
{
…
user1002065
- 595
- 1
- 8
- 19
2
votes
1 answer
Implementing blob storage
I'm looking for a way to implement (provide) blob storage for an application I'm building.
What I need is the following:
Access is done using simple keys (like primary keys; I don't need a hierarchy);
Blobs with sizes will be from 1KiB to 1GiB.…
Pieter van Ginkel
- 29,160
- 8
- 71
- 111
2
votes
1 answer
3Phase commit protocol - Distributed System
I got asked this question in one of my interviews. I am curious about the answer for this question:
What will happen if in 3 phase commit protocol after sending pre-commit message, the co-ordinator receives positive response from all the resource…
AlienOnEarth
- 747
- 7
- 14
2
votes
1 answer
Inotify-like feature in a distributed file system
As the title goes, I want to trigger a notification when some events happen.
A event above can be user-defined, such as updating specified files in 1-miniute.
If files are stored locally, I can easily make it with the system call inotify, but the…
xunzhang
- 2,838
- 6
- 27
- 44
2
votes
1 answer
Is the filesystem on Azure Websites distributed?
I know that other storage solutions are preferred for virtually any non-static data.
However, I still see some people using the filesystem provided in the Azure Websites service, which currently amounts to 10GB of space, particularly for die-hard…
tne
- 7,071
- 2
- 45
- 68
2
votes
1 answer
Distributed file system for android
I'm trying to develop a serverless distributed file system for android OS(a feature like Dropbox but serverless). I'm looking for suggestions on how I approach it.
I have seen about FUSE on android but didn't get a clear picture on how it would help…
macman
- 21
- 1
1
vote
0 answers
Read / Write large amounts of files daily
We receive around 10 million images per day ranging in size from 3kb to 200kb. At peak times it is around 400 images per second. It is an average of around 30kb per image.
At the moment all these images come into a single server with a 1TB NVMe SSD…
Mudders
- 127
- 2
- 12
1
vote
1 answer
How does IPFS store small files (less than 1KB)
The white paper states:
Small values (equal to or less than 1KB) are stored directly on the DHT. For values larger, the DHT stores references, which are the
NodeIds of peers who can serve the block.
What does mean: "stored directly"?
Does it mean…
Gomo
- 47
- 6
1
vote
0 answers
what interfaces should a file system provide to make it be supported by Spark?
I have developed a distributed file system which provide interfaces like standard Linux file system. Now I want it to be supported by Spark which means Spark can read files from it and save files to it just like HDFS. Since I am not familiar with…
hexiecs
- 312
- 1
- 12
1
vote
1 answer
Why do small files create hot spots in the Google File System?
I don't understand this from the Google File Systems Paper
A small file consists of a small number of chunks, perhaps just one. The chunkservers storing those chunks may become hot spots if many clients
are accessing the same file.
What…
Abhirath Mahipal
- 938
- 1
- 10
- 21