Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
0
votes
2 answers

MySQL Query mutates on huge instances:

I am currently calculating values to fill a database of 15 milion records. The first 7 mill went just fin,e however now my update query starts giving problems : Now & then a random letter changes into some jibberish. In java I generate the query…
FireFox
  • 472
  • 2
  • 4
  • 14
0
votes
1 answer

Ruby : The best way to manage a large 3d array

I would like to know what is the best way to manage a large 3d array with something like : x = 1000 y = 1000 z = 100 => 100000000 objects And each cell is an object with some amount of data. Simple methods are very loooooong even if all data are…
MARTIN Damien
  • 996
  • 2
  • 15
  • 36
0
votes
1 answer

How to solve the aggregate query with condition in Google App Engine

Suppose, In my app, I ask users to input some string. A user can input string multiple times. Whenever any user inputs a string, I log it in the database along with the day. Many strings can be same, even though inputted by different users. In the…
Anil Padia
  • 513
  • 1
  • 6
  • 12
0
votes
1 answer

Using OR and HAVING vs. using AND in MySQL

I'm working on a Coldfusion8/MySQL query in which I'm scanning a database for A-B pairs, for example: S=2, M=2, L=2, XL=2 I'm trying to improve the script that originally handled this, which limited entries to 4 pairs and because I'm wondering,…
frequent
  • 27,643
  • 59
  • 181
  • 333
0
votes
2 answers

How to use for each loop to help load large dataset

I'm trying to load a large dataset from SQL Server 2008 in SSIS. However, it's too slow for Visual Studio load everything at once. Then I decide to use for-each loop to just load part of the table each time. E.g. If there are 10 million records, I…
Echo
  • 1,117
  • 4
  • 22
  • 43
0
votes
3 answers

Reading large sets of data in Java

I am using Java to read and process some datasets from the UCI Machine Learning Repository. I started out with making a class for each dataset and working with the particular class file. Every attribute in the dataset was represented by a…
The Mitra Boy
  • 764
  • 1
  • 6
  • 13
0
votes
2 answers

Loading millions of DB row in wicket table through a list causing outofmemory error

I'm loading million lines of data by sortabledataprovider .. the query returns a list(Arraylist) as I sent it to a Wicket ajax enabled table and pagination enable table. So the problem is that - If there are concurrent queries - the application…
Imtiaz Mirza
  • 591
  • 1
  • 7
  • 26
0
votes
1 answer

Postgres DB query retrieval efficiency with django

I have 7 tables in PostgreSql which was integrated with Django 1.3 application.Out of the 7 tables we will access based on the name which is existing in all tables with different column data.Some particular name contains 1,00,000 rows.So, My problem…
Nava
  • 6,276
  • 6
  • 44
  • 68
-1
votes
1 answer

Architecture behind large-scale social sites like Twitter - help with a new startup.

I apologize in advance for this not being a specific programming question, but I believe this is the best place for such an inquiry. Basically, I have an idea for what I believe to be an interesting social networking site (not just a clone of…
KeithComito
  • 1,387
  • 1
  • 13
  • 24
-1
votes
1 answer

Plotly scatter large volume geographic data

I tried to write a code that creates a visualization of all forest fires that happened during the year 2021. The CSV file containing the data is around 1.5Gb, the program looks correct for me, but when I try to run it, it gets stuck without…
Souad
  • 13
  • 5
-1
votes
1 answer

Python: Program running Slooooow

I've got an issue with my Pandas data analysis running very slowly and wondered if someone could help please. I have 2 data frames I'm trying to compare if the IP address in one frame is part of the network in another frame: VFFrame contains 10…
GarryU
  • 1
  • 1
-1
votes
2 answers

Add delimiters while writing into a text file

I am writing an array into the text file, which I later use to read in excel for plotting. The data file is split in such a way that after 1000000 steps (approximately), the file closes and starts writing to another file. However, my text file is…
user1234
  • 257
  • 2
  • 13
-1
votes
2 answers

Which supervised classifiers in scikit-learn are recommended for large datasets?

There are many supervised classifier algorithms available in scikit-learn but I couldn't find any information about their scaalbility regarding large datasets. I know that for instance, support vector machines don't behave well with huge datasets,…
zlatko
  • 596
  • 1
  • 6
  • 23
-1
votes
1 answer

File diff of large size files

I need to code this task in java. I have 2 large files around 5GB each containing text data of multiple rows. Each row is a line of comma separated fields, for example "name,empId,designation,address,...,so on up to 30 fields". I need to read these…
user6604211
  • 1
  • 1
-1
votes
1 answer

Web API returning null JSON objects C#

I have a web API returning 117k JSON objects. Edit: The API is calling MySQL to fetch 117k rows of data, putting them into a IEnumerable and sending them through JSON All I see is…
lzc
  • 1,645
  • 5
  • 27
  • 41
1 2 3
…
20
21