Stack Overflow
Stack Overflow

Questions tagged [sparse-file]

A sparse file is a file where only the non-zero data are stored in data blocks and all zero-data are generated by the filesystem when the file is read. As such, the sparse file uses disk space efficiently by storing only metadata about the zeros, rather than storing the zeros physically on storage. When the sparse file is later read, the filesystem generates the zeros dynamically.

62 questions
1
vote
0 answers

How do I seek for holes and data in a sparse file in golang

I have a sparse file in linux and I wish to seek to a location containing data within that file. In C unistd.h gives me access to the constant SEEK_DATA for the whence flag of lseek. However I can't find an equivalent constant in golang. I've…
Q the Platypus
  • 825
  • 1
  • 7
  • 13
1
vote
1 answer

Backup LMDB database from Hyperledger Sawtooth

I have a sawtooth 1.1 dockerized network, and i'm trying to backup the database from the validators so i can put down every container and then recover in case of a disaster. Trying to achieve this i proceed as followed: Stopped the all…
rbn.araujo
  • 11
  • 3
1
vote
1 answer

Allocate file without zeroing and without creating a sparse file in linux

My goal is to instantly allocate a lot of space to a file without making a sparse file. Reading from the file should output the garbage left in free space, rather than 0s. Both truncate and fallocate make sparse files. Is this possible?
Daffy
  • 841
  • 9
  • 23
1
vote
1 answer

How to copy sparse file from ESXi quickly?

I have ESXi 6.0, from which I need to copy a sparse file of size ~900GB and its actual size 5GB to another Linux machine. I started with SCP and it takes very longer time, as it is unware of sparse file. Next I tried, "tar -S" to tar the sparse…
Vicky
  • 431
  • 1
  • 6
  • 12
1
vote
0 answers

Sparse file taking all fallocate()d space at once

I'm trying to create a sparse file (for a QEMU HDD image). Both qemu-img and fallocate are proving confusing. $ fallocate -l 100M disk.img $ ls -lsh disk.img 101M -rw-r--r-- 1 i336 users 100M Jul 22 12:03 disk.img Note the 101M. strace shows a…
i336_
  • 1,813
  • 1
  • 20
  • 41
1
vote
0 answers

How to convert Svmlight data to a database

I am using data in Svmlight format (you may know it as libsvm) this is the format I tried to create a SQL database to store it, the issue is that the format is sparse, and in a regular database it will be very consuming to store the data, and if i…
thebeancounter
  • 4,261
  • 8
  • 61
  • 109
1
vote
1 answer

How to create sparse file on Macos

I'm trying to understand the difference between the sparse file and no-sparse using the following article http://extrabright.com/blog/2010/03/30/how-to-know-if-a-file-on-linux-is-sparse/ I also created the sparse file…
Ratatouille
  • 1,372
  • 5
  • 23
  • 50
1
vote
2 answers

What are the limitations on size and number of allocated ranges in a Sparse File?

I'm working on a piece of software that writes data to NTFS sparse files and I can't find any documentation on what the limits of an NTFS sparse file are. I have seen reference to the fact that limitations exist, but not to what those limitations…
ChrisPatrick
  • 984
  • 7
  • 19
1
vote
0 answers

Why does LibSVMLoader produce an OutOfMemoryError in Weka?

I am trying to load a dataset in LibSVM Format with weka. Here the code: LibSVMLoader svmLoader = new LibSVMLoader(); svmLoader.setSource(new File("data_with_libsvm_format")); Instances svmData =…
Duong Trung Nghia
  • 310
  • 4
  • 13
1
vote
1 answer

What do I need to get SEEK_HOLE to work with sparse files?

I am trying to work more efficiant with sparse files. I have read about the SEEK_HOLE functionallity in newer Linux Kernels. According to other people, this should be in Kernel version 3.1 and later. However, as seen bellow, there must be something…
Daniele Testa
  • 1,538
  • 3
  • 16
  • 34
0
votes
0 answers

List all the hole and data segments in a sparse file

I am trying to implement a program which can print all the hole and data segments in a regular sparse file using lseek(2) and its arguments SEEK_DATA and SEEK_HOLE, which is something like: $ ./list_hold_and_data_segs sparse_file This file has 100…
Steve Lau
  • 658
  • 7
  • 13
0
votes
1 answer

retrieve sparse file from tar created without -S flag

I created a backup for a file then compressed it and store it using tar. At the time I didn't know it was a sparse file, So I didn't use the -S flag. Now I am trying to retrieve the data, but I can't since when I extract I get a non sparse file. Is…
user1928596
  • 1,503
  • 16
  • 21
0
votes
1 answer

How to convert a sparse matrix to a dense matrix from .mat file in Python?

import xlrd, xlwt import scipy as sp import scipy.io import pandas as pd import matplotlib.pyplot as plt import pylab from scipy.sparse import csc_matrix %matplotlib inline FileString =…
akif
  • 1
  • 1
0
votes
1 answer

Output numpy.ndarray as csr file

I'm trying to output a numpy ndarray as a CSR file (this is an intermediate stage, I'm trying to use a program that requires CSR format as input). So far, I've tried using scipy.sparse.coo_matrix() and writing out to an ijv file with the following…
Brendan O'Connell
  • 1
  • 1
0
votes
0 answers

Enlarging a Sparse File

I am designing a filesystem based queue and it impacts performance to scan the queue to compute its size when it is very large. I was looking for a way to count the task completions from many different processes. One idea I had was to create a…
SapphireSun
  • 9,170
  • 11
  • 46
  • 59
1 2
3
4 5