Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
3
votes
6 answers

What's the best way to transfer a large dataset over an ASMX web service?

I've inherited a C# .NET application which talks to a web service, and the web service talks to an Oracle database. I need to add an export function to the UI, to produce an Excel spreadsheet of some of the data. I have created a web service…
Malvineous
  • 25,144
  • 16
  • 116
  • 151
3
votes
1 answer

How to query, then process a large quantity in somewhat short time frame

I have a large table of orders. At a set time, I need to send an SMS message (different content for each one unfortunately) to a large chunk of them (based on if they've opted for the message and if they included a phone number). It could be…
Dave
  • 28,833
  • 23
  • 113
  • 183
3
votes
6 answers

java efficient de-duplication

Lets say you have a large text file. Each row contains an email id and some other information (say some product-id). Assume there are millions of rows in the file. You have to load this data in a database. How would you efficiently de-dup data (i.e.…
mnquasar
  • 31
  • 1
  • 2
3
votes
2 answers

Best database solution for managing a huge amount of data

I have to design a traffic database which includes data from different towns (8 towns) 2mb in a period of 10 min for each town 24h. The incoming data is the same for all Town. So my first question is what is better on the performance side: design…
Younes
  • 51
  • 6
3
votes
3 answers

Handling large records in a Java EE application

There is a table phonenumbers with two columns: id, and number. There are about half a million entries in the table. Database is MySQL. The requirement is to develop a simple Java EE application, connected to that database, that allows a user to…
craftsman
  • 15,133
  • 17
  • 70
  • 86
3
votes
5 answers

Fast conversion of numeric data into fixed width format file in Python

What is the fastest way of converting records holding only numeric data into fixed with format strings and writing them to a file in Python? For example, suppose record is a huge list consisting of objects with attributes id, x, y, and wt and we…
pico
  • 1,349
  • 2
  • 10
  • 15
3
votes
3 answers

SQL Server table structure for storing a large number of images

What's the best practice for storing a large amount of image data in SQL Server 2008? I'm expecting to store around 50,000 images using approx 5 gigs of storage space. Currently I'm doing this using a single table with the columns: ID:…
James Cadd
  • 12,136
  • 30
  • 85
  • 134
3
votes
1 answer

Processing large image volumes

I have a huge 3D volume, which sizes above 2GB. I want to load this data, process it and display in my application as a 3D volume, in which zooming and rotation of volume cn be performed. I saw applications which loads this huge data and displays…
rps
  • 59
  • 5
3
votes
2 answers

FileStream Seek fails on large files at second call

I'm working with large files , beginning from 10Gb. I'm loading the parts of the file in the memory for processing. Following code works fine for smaller files (700Mb) byte[] byteArr = new byte[layerPixelCount]; using (FileStream fs =…
VladL
  • 12,769
  • 10
  • 63
  • 83
3
votes
3 answers

Update column from another table in large mysql db (7 million rows)

Description I have 2 tables with the following structure (irrelevant columns removed): mysql> explain parts; +-------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra…
s3v3n
  • 8,203
  • 5
  • 42
  • 56
3
votes
2 answers

STXXL equivalent in Java

I'm searching a collection framework designed for huge datasets in Java that behaves transparently, like STXXL does for C++. It should transparently swap to disk, but in a much more efficient manner than plain OS-based VM swapping. A…
Steve Schnepp
  • 4,620
  • 5
  • 39
  • 54
2
votes
2 answers

SQL Error adding constraint to table, ORA-01652 - unable to extend temp segment

I've got this table with millions of rows that I loaded via the append hint. Now I go to turn the constraints back on, I get the following: 2012-03-23 01:08:53,065 ERROR [SQL] [main]: Error in executing SQL: alter table summarydata add constraint…
Jon Bristow
  • 1,675
  • 3
  • 27
  • 42
2
votes
1 answer

Plotting a graph in Linux (Ubuntu)

I have a file containing large amount of data (3.5GB, about 100M records, one repord per line). Format of a record is:
Timofey Gorshkov
  • 4,987
  • 6
  • 41
  • 66
2
votes
2 answers

Reading large volume of data from oracle database and export it as .dat file using C#

We have a query that will be executed on a monthly basis and returns data of size 1GB. Query used here is just a select query with inner joins, no cursor involved. Currently they are executing this query in Toad and exporting the data from output…
Gayathri
  • 21
  • 5
2
votes
2 answers

Large data processing technology & books

I am looking for good resources on how to query large volume of data efficiently. Each data item is represented as many different attributes such as quantity, price, history info, etc. The client will provide different query criteria but without…
q0987
  • 34,938
  • 69
  • 242
  • 387
1 2 3
…
20 21