Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
0
votes
1 answer

How to build a wordlist

So now I want to make Estonian wordlist ~about 20m unique words in lowercase. To get input for wordlist, corpus of Estonian can be used. Corpus files are in Text Encoding Initiative (TEI) format. I tried using regex to find the words. This is what…
Margus
  • 19,694
  • 14
  • 55
  • 103
0
votes
1 answer

How to select/query effectively 70,000 json data?

I'm now developing javascript library which consists 70,000 more villages in Indonesia (accessible at http://bisbak.com/regina/) and I build a data browser widget. Everything is fine in Safari and Firefox. But when using Chrome, it always takes long…
Muslax
  • 1
  • 1
0
votes
1 answer

Looking for example for plotting Volumetric Slice in Ilnumerics

is there any example for plotting volumetric slice in Ilnumerics use community version. This is an example I got from matlab website: Volumetric slice image example of matlab I have array X, Y, Z as posistions and V (velocity) as value for color…
Cas
  • 63
  • 5
0
votes
1 answer

entity-framework Can not run count command when data is amount of records

My code var numberOfStudents = _context.Student.Count(); In the database , about 4.000.000 records. I tried var numberOfStudents = _context.Student.Count(); but "Timeout expired" error still occur. I setted command timout to 5 mins. But this error…
Cuong_BG
  • 11
  • 3
0
votes
3 answers

PHP cURL 'Fatal error: Allowed memory size' for large data sets

I know about the option to set the internal memory ini_set("memory_limit","30M"); But I wanted to know if there is a better approach for querying data? I have a WHILE LOOP that checks to see if I need to query for another 1000 records. using the…
Phill Pafford
  • 83,471
  • 91
  • 263
  • 383
0
votes
2 answers

psycopg2 COPY using cursor.copy_from() freezes with large inputs

Consider the following code in Python, using psycopg2 cursor object (Some column names were changed or omitted for clarity): filename='data.csv' file_columns=('id', 'node_id', 'segment_id', 'elevated', 'approximation', 'the_geom',…
Adam Matan
  • 128,757
  • 147
  • 397
  • 562
0
votes
1 answer

SQL COUNT(*) a JOIN-statement instead of COUNT(*) inside a nested SELECT-statement for better Performance

Based on http://www.plumislandmedia.net/mysql/haversine-mysql-nearest-loc/ The following SQL Query returns an array of cities nearest to $lat/$lon pair within a given $radius. I added another join-statement (c.) to display the code3l-column from…
TNT_Larsn
  • 73
  • 8
0
votes
1 answer

Displaying a large collection in a DataGrid

The amount of items in collection: ~100k The amount of field displayed in columns: 4-10 The problem itself - the collection is taken from a database using EntityFramework. It takes about 10-12s on dev computers to load and materialize all the…
Jefim
  • 3,017
  • 4
  • 32
  • 50
0
votes
0 answers

Storing massive amounts of images. Storage schema and consequences? or is there a better way?

I am working on an auction listing project. Part of that project allows users to upload photos of the items(sometimes hundreds of items) they will have for sale on a given date. So I have multiple questions that all lead to how to do this…
user3154948
  • 137
  • 2
  • 11
0
votes
2 answers

Can't iterate over huge number of DataStore records

I'm trying to iterate over a huge number of datastore records, currently about 330,000. Conceptually, each record has a row, a column, and a value, and I'm iterating over the records and constructing a matrix, which I'll then use for calculations.…
Bruce Dov Krulwich
  • 1
  • 2
0
votes
1 answer

SQL table with many specific datetimes. How do I make this more scalable?

I have a rather troublesome table that looks like this: EventTimeLog Id (bigint) | Time (datetime) | LogId (FK to Log tables Id column) This table outlines the times a Log event occurred. This way to only write one unique log event and times those…
mBrice1024
  • 790
  • 5
  • 25
0
votes
1 answer

Docker mongodb - how are data only containers portable

How are data only containers portable when they ultimately store data on the host under /var/lib/docker ? Let's say I have this data container on my machine that stores entries into a database. And now let's say someone else 500 miles away wants to…
Kingamere
  • 9,496
  • 23
  • 71
  • 110
0
votes
1 answer

mysql - the right way to do eavy transofrmation

often when you handle "back office" and "front office", you need two different database structure the front office beeing a transformation of the backoffice's. My predecessors have always used a "batch processing" approach (whe have tousand of batch…
Bruno
  • 1,088
  • 1
  • 13
  • 31
0
votes
0 answers

How to create a dynamic marks filling table

I have to create SQL server database table for student performance system. IN this admin have the authority to create a feature list and marks on which student will be judge. this feature list can have many layers(at the starting it have 3…
sonal agrawal
  • 11
  • 4
0
votes
2 answers

Export large amount of data from Oracle 10G to SQL Server 2005

I need to export 100 million data rows (avg row length ~ 100 bytes) from Oracle 10G database table into SQL server (over WAN/VLAN with 6MBits/sec capacity) on a regular basis. So far, these are the options that I have tried and a quick summary. Has…
user274816
  • 83
  • 2
  • 5
1 2 3
…
20 21