Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
1
vote
1 answer

Solr indexing of a large data set

I have content that is about 50 TB large. The number of documents in this set is about 250 million. The daily increment to this is not very large nay my be about 10000 documents of varying sizes totaling under 50 MB. The current indexing effort is…
user2754571
  • 251
  • 3
  • 14
1
vote
1 answer

SAP RFC heavy upload - 3MB txt file produces 150MB upload

I have a problem with startRFC.exe that produces much bigger network-traffic than required. startRFC has 3 parameters = 3 internal tables = 3 CSV files. Total size of these files that are sent to SAP is 3MB, but it takes 15minutes and totally is…
Racky
  • 1,173
  • 18
  • 24
1
vote
2 answers

Sorting and making "genes" in output bitstrings from a genetic algorithm

I was wondering if anybody had suggestions as to how I could analyze an output bitstring that is being permuted by a genetic algorithm. In particular it would be nice if I could try to identify patterns of bits (I'm calling them genes here) that…
Skyler
  • 410
  • 4
  • 18
1
vote
4 answers

Need some help calculating percentile

An rpc server is given which receives millions of requests a day. Each request i takes processing time Ti to get processed. We want to find the 65th percentile processing time (when processing times are sorted according to their values in increasing…
mousey
  • 11,601
  • 16
  • 52
  • 59
1
vote
1 answer

Optimizing BufferedReader for large input in java

I am trying to read a single line which is about 2 million characters long from the standard input using the following code: BufferedReader in = new BufferedReader(new InputStreamReader(System.in)); s = in.readLine(); For the aforementioned input,…
KingSweeney
  • 11
  • 2
1
vote
3 answers

Large maintenance php script. How to print debug string while the script is executing?

I have a very large php maintenance script (basically it recreates thumbnails for an internal archive), it takes 10 to 20 minutes to complete and I noticed that php only displays "echos" when the whole script has finished parsing. Is there any way…
0plus1
  • 4,475
  • 12
  • 47
  • 89
1
vote
1 answer

Issue with cloning and large structure processing

My Perl script have weird behaviour which I don't understand. I'm processing large structure stored as array of hashes which is growing while processing. The problem is that structure has about max 8mb when I store it on hdd, but while it is…
jesper
  • 879
  • 8
  • 21
1
vote
0 answers

Very Large MySQL Table Design

We want to store a very large table in MySQL. The workload's spec is like this: Around 1 billion rows. Each row has just two fields: a long int as the key and a blob with average size of 20KB as the value. High rate of inserts, around 1000 per…
Saeed Shahrivari
  • 815
  • 1
  • 9
  • 16
1
vote
2 answers

Different databases or tables sets for each user

I'll have to create database(s) to store very large amounts of data but being able to extract data fast enough using MySQL. I was wondering if it will help if I create a new database or a new tables set for each user instead of using a single large…
Samuil Banti
  • 1,735
  • 1
  • 15
  • 26
1
vote
2 answers

Large 3D volume bad_alloc

I'm developing an application that creates a 3D Voronoi Diagram created from a 3D point cloud using boost multi_array allocated dynamically to store the whole diagram. One of the test cases I'm using requires a large amount of memory (around…
Ishida
  • 277
  • 3
  • 13
1
vote
1 answer

Asking for suggestions on database table design base on this described scenario

It might be a weird situation, but it just come to my mind... Imagine I have a database table which takes 1 million new rows everyday. There are 3 columns in the table: id, value, date. What I would like to do with the rows is to load all rows…
Tony Fantacy
  • 65
  • 7
1
vote
0 answers

Big sql tables and entity framework

I am trying to find out better way deal with large volume of records within single table. Given that a table have 50M records and records are inserted about ~1.5M records a day. Which options are available to avoid further performance struggling…
Vladimirs
  • 8,232
  • 4
  • 43
  • 79
1
vote
4 answers

SQL Server 2005 proper index to filter 30,000,000 registers

I have a problem with a stored procedure of a transactional table, the user have a web form to find transactions by several values. The process is taking too long and I don't know how to set proper index. here is my stored procedure: CREATE…
jmpena
  • 1,399
  • 6
  • 19
  • 26
1
vote
3 answers

Doing large updates against indexed view

We have an indexed view that runs across three large tables. Two of these tables (A & B) are constantly getting updated with user transactions and the other table (C) contains data product info that is needs to be updated once a week. This product…
user217136
  • 11
  • 2
1
vote
1 answer

Create a 350000 column csv file by merging smaller csv files

I have about 350000 one-column csv files, which are essentially 200 - 2000 numbers printed one under another. The numbers are formatted like this: "-1.32%" (no quotes). I want to merge the files to create a monster of a csv file where each file is…
rwms
  • 313
  • 1
  • 5
  • 15
1 2 3
…
20 21