Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
0
votes
0 answers

Amazon EC2 Force umount Volume does not work

I am trying to detach an Amazon EC2 Volume, but It get stuck some where, here is the strace output, I tried fuser -km /mnt/web but none of the option is working. I also tried ls /mnt/, and deleted the entry from /etc/fstab but it still won't…
Aftab Naveed
  • 3,652
  • 3
  • 26
  • 40
0
votes
0 answers

node.js express.js app not working for large traffic (600/min)

I am working on node.js express.js app which is used for user data collection (in get request) and sends back only 200 ok response. After getting data in request I am checking for existing entry in database dynamodb and redshift and for newer…
Sandip Pingle
  • 647
  • 2
  • 14
  • 37
0
votes
1 answer

MySQL: How to handle regular high volume of inserts/updates?

I have table where I track various statistics about site usage. Once a week, I plan to use these statistics and information from other tables to calculate various key indicators for use in multiple reports. This is in order not to have compute the…
Tom
  • 30,090
  • 27
  • 90
  • 124
0
votes
0 answers

Sql Server large data

I have sql server 2012 installed on a Virtual machine. With quad core processor (2.0 GHz) and 32 GB RAM. The Data volume is about 400 GB and there is a Table with 824 million records in it. The Records are date wise. I have to pull data from…
user3518425
  • 1
0
votes
1 answer

Pythonic way to process 200 million element data set?

I have a directory of 1,000 files. Each file has many lines where each line is an ngram varying from 4 - 8 bytes. I'm trying to parse all files to get the distinct ngrams as a header row, then for each file, I want to write a row that has the…
user3517765
  • 1
0
votes
1 answer

Updating large volume of records

I have a table having more than 10 mil records and I have to update multiple columns of this table from the data which I get from few other tables which again have large number of records. All the tables are indexed I tried doing it in batches and…
chapter473
  • 15
  • 7
0
votes
3 answers

Theoretical large volume issue, can't use collection to sort in .NET

Excuse the title of this post, but I can't really think of a more creative title. I am calling a 3rd party web service where the authors are ordering transaction results from most recent. The total transaction count is greater than 100 000. To make…
JL.
  • 78,954
  • 126
  • 311
  • 459
0
votes
2 answers

Algorithm for calculate all difference values from a large list

​I have a question that now I have a list of 3 million records and I want get all difference values between every two records. A simple nested loop may take forever. Could anyone suggest me any algorithm which is capable of handling this problem?
DingMartin
  • 3
  • 2
0
votes
1 answer

MySql search efficiency

So I am trying to get some geographical data at the moment from the some 34 thousand records we have. We stored a records of Person, which include the basics like firstname, surname, address and postcode. I am wanting to get information about the…
Lars
  • 59
  • 1
  • 2
  • 8
0
votes
1 answer

Matrix calculation gets slower after each iteration in matlab

I have a 1024*1024*51 matrix. I'll do calculations to change some value of the matrix within for loops (change the value of matrix for each iteration). I find that the computing speed gets slower and slower and finally my computer gets into…
lee891031
  • 1
  • 2
0
votes
2 answers

Improving select speed - mysql - very large tables

Newbie to MySQL and SQL in general - so please be gentle :-) I have a table with a very high number of rows. The table is: create table iostat ( pkey int not null auto_increment, serverid int not null, datestr char(15) default 'NULL', esttime …
opsdog
  • 13
  • 4
0
votes
1 answer

Command line script or software tools to label 3d point cloud dataset

How can i label a 3d point cloud dataset? is there a software which can load a text file containing x,y,z values and then visualize it , so that I can label it ?
iceman
  • 4,211
  • 13
  • 65
  • 92
0
votes
1 answer

Display ALL countries in an autocomplete form

First time posting on here because Google is yielding no results! So, I have a website that is based around travelling and locations. Everytime someone enters content into the site, they select a location and that content then has lat and long,…
Benny McLennan
  • 1
  • 1
0
votes
1 answer

Statistics on large table presented on the web

We have a large table of data with about 30 000 0000 rows and growing each day currently at 100 000 rows a day and that number will increase over time. Today we generate different reports directly from the database (MS-SQL 2012) and do a lot of…
Addeladde
  • 777
  • 2
  • 9
  • 28
0
votes
2 answers

Search a large amount of data in DynamoDB

I have a use case, in which my data is stored in DynamoDB with hashkey as UniqueID and range key as Date. The same data is also present in simple storage service of Amazon(S3). I want to search all the data based on time range. I want this to be…
Nitish
  • 31
  • 1
  • 4
1 2 3
…
20 21