Stack Overflow
Stack Overflow

Questions tagged [data-processing]

Data Processing concerns the converting of raw data to machine-readable form and its subsequent processing (as storing, updating, rearranging, or printing out) by a computer.

Data Processing concerns the converting of raw data to machine-readable form and its subsequent processing (as storing, updating, rearranging, or printing out) by a computer.

More Info

909 questions
0
votes
1 answer

When to apply Data whitening

Data Whitening (features scaling and mean normalization) is very useful when we use features that represent different characteristics and are on very different scales (eg number of rooms in a house and house price). What about the case when the…
teaLeef
  • 1,879
  • 2
  • 16
  • 26
0
votes
1 answer

What is a Data warehouse in this use case

I'm trying to figure out the difference (between tools/services/programs) between Data Warehouse, Clustered Data Processing and the tools/infrastructure for querying a Data Warehouse So Let's say I have the following setup to perform some data…
user6123723
  • 10,546
  • 18
  • 67
  • 109
0
votes
1 answer

Speeding up document processing and loading into database

I have a few million documents. What I am trying to do is simple, process the documents to extract the information I need and load it into a database. I am doing it in Python and using SQLAlchemy. Also I am using multiprocessing to make use of all…
y2p
  • 4,791
  • 10
  • 40
  • 56
0
votes
1 answer

Appropriate data processing design pattern?

I'm looking for an appropriate design pattern to accomplish the following: I want to extract some information from some "ComplexDataObject" (e.g. an Image) and save the relevant information in a more suitable format, let's call it a…
Mikael Call
  • 75
  • 1
  • 8
0
votes
1 answer

Running code (loop) server side and retrieving output later on

I am trying to do a simple program that keeps track of some internet data. I can get the data from a public JSON object, so that's not really the problem. I would like to automize the process as much as possible. I need to use HTML code and run it…
OFRBG
  • 1,653
  • 14
  • 28
0
votes
0 answers

Trying to process raw string (rank of countries by GDP) with python for other uses

I'm pretty new to this so sorry if this is a dumb question. I'm trying to sort some data. Here's a rank of countries by GDP for example, that I'd like to find percentages of, add up certain amounts top down (like a CDF), etc. How would I make a…
Alexander Kleinhans
  • 5,950
  • 10
  • 55
  • 111
0
votes
1 answer

Calculate min/avg/max/std-dev for ICMP time stamp data from hping

What's the best way to calculate min/avg/max/std-dev for some random data in shell? What if one has several columns per line, and needs to calculate the statistics for each one? Sample input (based on processing of the hping output), with the…
cnst
  • 25,870
  • 6
  • 90
  • 122
0
votes
2 answers

Tools to do data processing from Java

I've got a legacy system that uses SAS to ingest raw data from the database, cleanse and consolidate it, and then score the outputted documents. I'm wanting to move to a Java or similar object oriented solution, so I can implement unit testing, and…
dwjohnston
  • 11,163
  • 32
  • 99
  • 194
0
votes
1 answer

How do I perform koyck lag transformations in PMML?

I'm using PMML to transfer my models (that I develop in R) between different platforms. One issue I often face is that given input data I need to do a lot of pre-processing. Most times this is rather straightforward in PMML but I cannot figure out…
Dr. Mike
  • 2,451
  • 4
  • 24
  • 36
0
votes
2 answers

How to change long gene names to abbreviated in some automatic way (microarray data processing)?

Is there any automatic way to convert a list of long gene names (like Cadherin_3453) to its abbreviations, like CDHRN_3453? Are there any abbreviation name convention in Genomics, Bioinformatics? Sorry, no code herein
KvasDub
  • 281
  • 7
  • 16
0
votes
3 answers

How to add a postfix in incremental order to a printed string in Ruby?

For microarray data processing, I need to make a list of gene names from 1 to 654, like Gene_1 ... Gene_654. My simple Ruby code produces the following: 1.upto(654).each { |i| print "Gene" } The result…
KvasDub
  • 281
  • 7
  • 16
0
votes
1 answer

Archiving whole result of a Twitter Hashtag serach

Is there a data processing tool that allow to countinuously download the result of a Hashtag search in https://search.twitter.com, or that just allow to download the whole tweets of the specific hashtag (data scraper)? 'The Archivist Desktop' is the…
Naourass Derouichi
  • 773
  • 3
  • 12
  • 38
0
votes
1 answer

Data processing and updating of selected records

Basically, the needed job is for large amount of records on a data base, and more records can be inserted all the time: Select <1000> records with status "NEW" -> process the records -> update the records to status "DONE". This sounds to me like…
Liran
  • 144
  • 2
  • 6
0
votes
2 answers

How do you create a dependency structure in php?

I have a lot of status and I'm building functions for each status. I'm trying to figure out how I can revert changes based on status. The be a bit more clear: Say you have 5 status: START INSTALLED PROC COMPLETE UPLOADED In order to go from status…
user962449
  • 3,743
  • 9
  • 38
  • 53
0
votes
3 answers

php parsing array and validating data

I'm having trouble manipulating and array to filter some data. Here's an example of the array: Array ( [0] => Array ( [0] => Array ( [accounts] => Array ( …
user962449
  • 3,743
  • 9
  • 38
  • 53
1 2 3
…
60 61