Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
2
votes
1 answer

How to efficiently filter objects out of an (initially) large list of objects

I need to filter a large list of complex (20+ properties) objects into multiple sub lists. To create the sub-lists, I have a list of filter specifications. Requirements are: a) An item is not allowed to be part of two sub lists and b) it must be…
Björn Boxstart
  • 1,098
  • 1
  • 12
  • 25
2
votes
3 answers

How to split data over MySQL tables

I have a website with members who message each other. There are getting to be a few members and they like to send messages - I'm sure you can see where this is going. Currently I have said messages stored in a nicely relational table cunningly…
da5id
  • 9,100
  • 9
  • 39
  • 53
2
votes
3 answers

Getting started with massive data

I'm a mathematician and occasionally do some statistics/machine learning analysis consulting projects on the side. The data I have access to are usually on the smaller side, at most a couple hundred of megabytes (and almost always far less), but I…
Max
  • 21
  • 1
2
votes
3 answers

Performing Aggregate Functions on Multi-Million Row Tables

I'm having some serious performance issues with a multi-million row table that I feel I should be able to get results from fairly quick. Here's a run down of what I have, how I'm querying it, and how long it's taking: I'm running SQL Server 2008…
Dan Short
  • 9,598
  • 2
  • 28
  • 53
2
votes
4 answers

Graphing large amounts of data

In a product I work on, there is an iteration loop which can have anywhere between a few hundred to a few million iterations. Each iteration computes a set of statistic variables (double precision), and the number of variables can be up to 1000…
Walter Williams
2
votes
1 answer

How to efficiently store a large but EXTREEMLY sparse amount of data in a 3D representation?

I ran into this problem in the code I an working on: I need to be able to store a large amount of data in RAM, and be able to access it using three integer i,j,k (similar to x,y,z coordinates). The problem is that if I try to use a 3D array, then…
Vladimir
  • 225
  • 1
  • 11
2
votes
4 answers

Avoid an "out of memory error" in Java(eclipse), when using large data structure?

OK, so I am writing a program that unfortunately needs to use a huge data structure to complete its work, but it is failing with a "out of memory error" during its initialization. While I understand entirely what that means and why it is a problem,…
gnomed
  • 5,483
  • 2
  • 26
  • 28
2
votes
1 answer

Displaying large sorted lists in WPF QUICKLY?

I am developing a program that should be able to display long (up to 500 items) lists of data that need to be resorted when their contents change. Essentially, I have a viewmodel with an observable collection that contains classes with observable…
bluebit
  • 2,987
  • 7
  • 34
  • 42
2
votes
4 answers

Efficient File I/O and Conversion of Strings to Floats

I have some gigantic (several gigabyte) ASCII text files that I need to read in line-by-line, convert certain columns to floating point, and do a few simple operations on these numbers. It's pretty straightforward stuff, except that I'm thinking…
dsimcha
  • 67,514
  • 53
  • 213
  • 334
2
votes
6 answers

Read from one large file and write to many (tens, hundreds, or thousands) files in Java?

I have a large-ish file (4-5 GB compressed) of small messages that I wish to parse into approximately 6,000 files by message type. Messages are small; anywhere from 5 to 50 bytes depending on the type. Each message starts with a fixed-size type…
Rudiger
  • 6,634
  • 9
  • 40
  • 57
2
votes
1 answer

MySQL - Handling 200M+ records per day

I have a MySQL table that will need to store about 200 million records on a daily basis. The data comes in flat files that I will be bulk uploading using "LOAD DATA INFILE". Most of the data elements are IDs, so not much text. One flat file for a…
user333746
  • 2,605
  • 3
  • 18
  • 11
2
votes
4 answers

Select Count(*) over large amount of data

I want to do this for a Report but i have 20,000,000 of records in my table and it causes an TimeOut in my application. SELECT T.transactionStatusID, TS.shortName AS TransactionStatusDefShortName, count(*) AS qtyTransactions …
jmpena
  • 1,399
  • 6
  • 19
  • 26
2
votes
5 answers

how to handle large lists of data

We have a part of an application where, say, 20% of the time it needs to read in a huge amount of data that exceeds memory limits. While we can increase memory limits, we hesitate to do so to since it requires having a high allocation when most…
Edward Q. Bridges
  • 16,712
  • 8
  • 35
  • 42
2
votes
1 answer

serialize list of huge composite graphs using protobuf-net causing out-of-memory-exception

I am trying to serialize an object containing a list of very large composite object graphs (~200000 nodes or more) using Protobuf-net. Basically what I want to achieve is to save the complete object into a single file as fast and as compact as…
mda
  • 53
  • 5
2
votes
5 answers

Is there a way to maintain a 200MB immutable data structure in memory and access it from a script?

I have a list of 9 million IPs and, with a set of hash tables, I can make a constant-time function that returns if a particular IP is in that list. Can I do it in PHP? If so, how?
konr
  • 1,173
  • 12
  • 26
1 2 3
…
20 21