Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
0
votes
2 answers

Large Volume Data Searches

I have been asked to look into developing a solution for logging for a large organisation. I was thinking having some sort of service that has a msg queue of logs and inserts them into some sort of database but there is lie's the rub. Oracle and…
Joshy
  • 657
  • 8
  • 20
0
votes
0 answers

Oracle Export Very Large Tables

I try to export a table with 1.8 Billion Rows and 1.5TB os size not sucess for now. exp xxx/xxxx@server file=tabela_grande.txt direct=y tables=('TB_DM_HORI_CLUSTER_CANAL_01') I'm trying EXP command but it's taking too long. About 10 hours and not…
Rafael Colvara
  • 13
  • 3
0
votes
1 answer

Extract points from very large girds

I have 10 grids (currently stored as ascii grids from a GIS), each of them with about 4.5GB uncompressed. In addition I have about 100,000 location with an x and y coordinate. I need to extract the grid value at each of this location. I am currently…
johannes
  • 14,043
  • 5
  • 40
  • 51
0
votes
2 answers

Sitemaps structure for large App Engine site

I'm thinking on the best way to structure a large App Engine site (+1M urls). I need a sitemaps.xml file in the root path of the domain file that links to sitemap[n].xml files. The sitemaps.xml file can link up to 1000 sitemap[n].xml files and each…
ana
  • 1,565
  • 4
  • 17
  • 22
0
votes
0 answers

Finding Free large-scale Databases to work with SQL Server Express or Developer

I would like to improve my overall programming proficiency with SQL Server Express/Developer. My programming has so far been effective only with "baby" databases with tables with fewer than 200 rows. Anyone know of free sources for databases with…
gary
  • 129
  • 10
0
votes
1 answer

wxpython: Is there a quicker method for inserting large quantities of text into text controls?

I'm coding a GUI in wxpython right now and one of its features is a text control. This text control often takes 10's of thousands of numbers of varying length inserted into it. When it is being filled with the data, it takes a long time (30 seconds…
Ampig
  • 179
  • 1
  • 10
0
votes
0 answers

Using cv.glmnet on a very large data-set

I use the cv.glmnet() function to estimate a penalized multinomial logit model. Since the datset is far too large to make things reproducable, I show the call of the function: cvfit = cv.glmnet(x= X, y=as.numeric(dat$choice_t) , …
Jogi
  • 304
  • 4
  • 15
0
votes
3 answers

What should i do for accommodating large scale data storage and retrieval?

There's two columns in the table inside mysql database. First column contains the fingerprint while the second one contains the list of documents which have that fingerprint. It's much like an inverted index built by search engines. An instance of a…
consumer
  • 709
  • 3
  • 7
  • 19
0
votes
1 answer

Slowly Increasing Save Time When Saving As in Excel

I created a macro enabled Excel file that performs the following actions (high level): User selects template file (which is itself macro enabled) via file dialog User selects data files (which are not macro enabled) also via file dialog Macro steps…
apynn
  • 11
  • 3
0
votes
0 answers

VBA Write line at once in CSV format for large volume

Good day! In VBA I have to write into a CSV file a large array. To increase speed of writing I want to write a line at once to avoid looping over each field. The solution with looping works. The solution with line at once inserts more or less quotes…
itsergiu
  • 662
  • 6
  • 11
0
votes
0 answers

How to deal with large merModLmerTest

I try to use R on my laptop to run the HLM regression of a large dataset of about 2GB(500,000 lines), and the format of this dataset is spss(.sav).Sorry I could not share the data, as required by my professor, but I would try my best to provide as…
exteral
  • 991
  • 2
  • 12
  • 33
0
votes
1 answer

Docker - Feed data from read-only directory on host into container

I'm a relative novice to Docker, so please bear with me if the answer is obvious. I'm trying to give my docker container access to a shared data directory on the host, which has specific group permissions and is read-only for the group of the…
Val
  • 6,585
  • 5
  • 22
  • 52
0
votes
3 answers

Faster Access Version of ArrayList?

Does anyone know of something similar to ArrayList that is better geared to handling really large amounts of data as quickly as possible? I've got a program with a really large ArrayList that's getting choked up when it tries to explore or modify…
Cambot
  • 257
  • 2
  • 13
0
votes
1 answer

More than 10 000 000 sesssions in Universal Analytics

We have a resource of our client with more than 10 000 000 sessions (not hits) per month. Is it possible that they will contact our client soon and ask him to use a paid account in Universal Analytics (or to reduce the data volume)?
Nadezda Kantur
  • 1
0
votes
1 answer

speed up slow count in a 2 millions rows table

i have a Mysql table which has 2 millions rows. The size is 600Mb. this query take 2 seconds. I don't know how to speed it up. The table is already in a Myisam format. I don't know if i reached the limit of the slowness of a select count. SELECT…
Frederic CS
  • 89
  • 1
  • 6
1 2 3
…
20 21