Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
2
votes
1 answer

Large dataset, ProcessPoolExecutor issues

PROBLEM - ProcessPoolExecutor hasn't increased speed. Confirmed by tqdm Learned enough about python to copy and/or write a program that works. each file takes ~40 seconds to load->filter->write. I have ~6,800 files to work through and want a better…
Yusuf
  • 23
  • 4
2
votes
1 answer

Can I break down a large-scale correlation matrix?

the correlation matrix is so large (50000by50000) that it is not efficient in calculating what I want. What I want to do is to break it down to groups and treat each as separate correlation matrices. However, how do I deal with the dependence…
angielovesmath
  • 31
  • 2
2
votes
3 answers

How can we handle large matrices in matlab(larger than 10000x10000)

In my program I am faced with some matrices that are larger than 10000x10000. I cannot transpose or inverse them, how can this problem be overcome? ??? Error using ==> ctranspose Out of memory. Type HELP MEMORY for your options. Error in ==>…
Abolfazl
  • 453
  • 4
  • 8
  • 19
2
votes
2 answers

Speed Up Oracle Select by Cache or partition

I've got a oracle MV with 100+ million rows. Much of the our code calls against this view, so I would like to keep the syntax intact and make sure these calls are very fast. Because of business logic I know that less than 1 million rows would be…
Peter Frey
  • 361
  • 4
  • 17
2
votes
2 answers

How do I design a table which will store very large data?

I need to design a table in Oracle, which will store 2-5 TB of data in a day. It can grow to 200TB, and records will purged when it crosses 200 TB. Is it a feasible choice to keep it in OLTP, or do I need to shift it to data warehouse DB? Please…
paseena
  • 4,207
  • 6
  • 32
  • 51
2
votes
0 answers

Keras : Dealing with large image datasets

I am trying to fit a model using a large image datasets. I have a memory RAM of 14 GB, and the dataset have the size of 40 GB. I tried to use fit_generator, but I end up with a method that does not delete the loaded batchs after using theme. If…
Khelifi Aymen
  • 132
  • 1
  • 10
2
votes
2 answers

VBA for excel; editing very large files

I have a very large set of log files with AIS(shipping) data. Since these log files are about 200Mb per day, I'm trying to size them down for archiving. The files look like this: 244630075;under way ;128°';…
jefti
  • 23
  • 2
2
votes
1 answer

Adwords API BulkMutateJobService Fetch Global Monthly Search Volume For Multiple Keywords

I've just gotten into the Adwords API for an upcoming project and I need something quite simple actually, but I want to go about it the most efficient way. I need code to retrieve the Global Monthly Search Volume for multiple keywords (in the…
Adam
  • 1,962
  • 2
  • 17
  • 30
2
votes
1 answer

Efficient retrieval of large data sets in Mongo mapper?

I am storing a large amount of Twitter data, and would like to retrieve about 500k records for data processing at a time. I have a TwitterTweet mongo document that contains basic tweet data, and try to retrieve it as follows: weekly_tweets =…
bluebit
  • 2,987
  • 7
  • 34
  • 42
2
votes
2 answers

Send 5MB of data over a socket?

Hi I want to send a fixed amount of data (say 5MB) from server to Android client over TCP using the Java programming language. The data doesn't matter it will be dropped at the client I am only doing this to do performance measurements on the phone.…
Mike
  • 35
  • 1
  • 5
2
votes
3 answers

What is the maximum recommended number of rows that a SQL 2008 R2 standalone server should store in a single table?

I'm designing my DB for functionality and performance for realtime AJAX web applications, and I don't currently have the resources to add DB server redundancy or load-balancing. Unfortunately, I have a table in my DB that could potentially end up…
Giffyguy
  • 20,378
  • 34
  • 97
  • 168
2
votes
5 answers

One reader thread, one writer thread, n worker threads

I am trying to develop a piece of code in Java, that will be able to process large amounts of data fetched by JDBC driver from SQL database and then persisted back to DB. I thought of creating a manager containing one reader thread, one writer…
Adalbert27
  • 73
  • 4
2
votes
2 answers

Laravel/DataTable fails at displaying ~80,000 rows

I'm trying to use DataTables with server side processing to display 80,000 rows (paginated), but Laravel crashes with the error: PHP Fatal error: Maximum execution time of 300 seconds exceeded in…
f7n
  • 1,476
  • 3
  • 22
  • 42
2
votes
1 answer

Transfer large messages with Apache CXF

I'm writing a CXF WS to upload some large files - up to 1GB. In most cases they won't be >10-15MB, but the problem is that it is ineffective to load the file and send it as regular byte[] using the standard binding. For that reason a custom…
munch
  • 2,061
  • 2
  • 16
  • 23
2
votes
1 answer

SQLite view across multiple databases. Is this okay? Is there a better way?

Using SQlite I have a large database split into years: DB_2006_thru_2007.sq3 DB_2008_thru_2009.sq3 DB_current.sq3 They all have a single table call hist_tbl with two columns (key, data). The requirements are: 1. to be able to access all the…
Pete
  • 409
  • 4
  • 8
1 2 3
…
20 21