Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
3
votes
5 answers

Appropriate data structure for faster retrieval process (data size: around 200,000 values all string)

I have a large data set of around 200, 000 values, all of them are strings. Which data structure should i use so that the searching and retrieval process is fast. Insertion is one time, so even if the insertion is slow it wouldn't matter much. Hash…
Elvis
  • 125
  • 9
3
votes
3 answers

SELECT a varchar field on one entry in an 8.2 million entry table - performance help

I have a table with 8.2 million entries in a SQL Server 2005 database. This table stores basic past customer details (referrer, ip, whether they entered via an advertisement, etc) for every customer we have had come to the site. Unfortunately,…
Cody Mays
  • 31
  • 1
3
votes
5 answers

Large dataset (SQL to C#), long load time fix

I have a site I'm building, it's an application that creates mail merges (more or less...) based on a couple of user preferences. It can generate Cartesian joins worth of data without a problem, but in comes the needs of enterprise to make life a…
C Bauer
  • 5,003
  • 4
  • 33
  • 62
3
votes
3 answers

Efficient MySQL schema with partitioning for huge dataset (7.300.000.000 rows and roughly 80 GB of data)

This is a follow up to my question "Efficiently storing 7.300.000.000 rows" (Efficiently storing 7.300.000.000 rows). I've decided to use MySQL with partitioning and the preliminary schema looks like this: CREATE TABLE entity_values ( entity_id…
knorv
  • 49,059
  • 74
  • 210
  • 294
3
votes
2 answers

Removal of 30-40 millions record without affecting performance

I have a SQL Server database running in Full recovery. I need to remove data (around 30-40 million records) but I cannot take the database offline as it's in constant use. I also cannot switch it to Simple recovery mode incase anything happens and…
Neil Knight
  • 47,437
  • 25
  • 129
  • 188
3
votes
3 answers

Python - Best data structure for incredibly large matrix

I need to create about 2 million vectors w/ 1000 slots in each (each slot merely contains an integer). What would be the best data structure for working with this amount of data? It could be that I'm over-estimating the amount of processing/memory…
Georgina
  • 311
  • 4
  • 11
3
votes
2 answers

dumping a mysql table to CSV (stdout) and then tunneling the output to another server

I'm trying to move a database table to another server; the complication is that the machine currently running the table has little to no space left; so I'm looking for a solution that can work over the net. I have tried mysqldumping the database…
Ben Novakovic
  • 581
  • 7
  • 17
3
votes
1 answer

How to store and query very large data sets (beyond relational DB's)

We are currently facing a problem of how to effectively store and retrieve data from very large data sets (into the billions). We have been using mysql and have optimized the system, OS, raid, queries, indexes etc, and are now looking to move on. I…
jW.
  • 9,280
  • 12
  • 46
  • 50
3
votes
2 answers

Big sites showing less data

I look after a large site and have been studying other similar sites. In particular, I have had a look at flickr and deviantart. I have noticed that although they say they have a whole lot of data, they only display up to so much of it. I persume…
David
  • 16,246
  • 34
  • 103
  • 162
3
votes
2 answers

Draw line graph in Objective C with large number of data and show them in collection view

I have large volumes of data (items/row) stored in a file with extension .rec (just like .text) I need to show the vertices in a line graph supporting pinch zoom and dragging. I can draw the line graph with CoreGraphics. But it doesn't work well…
kallol
  • 319
  • 1
  • 13
3
votes
5 answers

How do I allow the user to easily choose how much memory to allocate in a Java Swing app?

We have a Swing app that processes relatively large amounts of data. For instance we currently process CSV files with millions of rows of data. For the reasons of performance and simplicity we just keep all of the data in memory. However…
Jay Askren
  • 10,282
  • 14
  • 53
  • 75
3
votes
1 answer

How can i create a powershell script to move thousands of log files from one location to another?

I have a folder on a server that contains thousands of log files. New files are being populated every second and the folder continues to grow. Once a week I want to take those files, copy them and paste them into another folder. I will then run a…
mattk
  • 93
  • 1
  • 1
  • 7
3
votes
6 answers

Large MySQL tables

For a web application I'm developing, I need to store a large number of records. Each record will consist of a primary key and a single (short-ish) string value. I expect to have about 100GB storage available and would like to be able to use it…
Snukker
  • 1,963
  • 3
  • 18
  • 18
3
votes
3 answers

Handling 100's of 1,000,000's of rows in T-SQL2005

I have a couple of databases containing simple data which needs to be imported into a new format schema. I've come up with a flexible schema, but it relies on the critical data of the to older DBs to be stored in one table. This table has only a…
Matt W
  • 11,753
  • 25
  • 118
  • 215
3
votes
1 answer

insert large volume of data in mysql

I want to insert atleast 500,000 fresh records in one shot. For which I have used while loop inside procedure. My query is working fine but it is taking alot of time to execute. So, I am looking for a solution using which I can make the process of…
Amrit Shrestha
  • 1,620
  • 20
  • 25
1 2 3
…
20 21