Stack Overflow
Stack Overflow

Questions tagged [data-processing]

Data Processing concerns the converting of raw data to machine-readable form and its subsequent processing (as storing, updating, rearranging, or printing out) by a computer.

Data Processing concerns the converting of raw data to machine-readable form and its subsequent processing (as storing, updating, rearranging, or printing out) by a computer.

More Info

909 questions
-1
votes
1 answer

Dependent data processing pipelines where files arrive asynchronously

I have several data dependent tasks/pipelines of which some depend on the completion of another. What makes it even harder is that the data can arrive asynchronously, meaning that certain tasks need to wait until all the files or tasks in the…
WJA
  • 6,676
  • 16
  • 85
  • 152
-1
votes
1 answer

Google Cloud data processing options for parsing of files that are delivered asynchronously

I am looking for the best Google Cloud solution to stream & batch process files into a clean database on BigQuery. I could use Dataflow or Cloud functions to achieve this result. However, I am struggling a bit with the architecture/design and the…
WJA
  • 6,676
  • 16
  • 85
  • 152
-1
votes
1 answer

Understanding the percent change effect after applying MinMaxScaler?

I wanted to know if the percent change between two values remains the same after scaling the data using MinMaxScaler from scikitlearn. If not which scaling method should I use in order to make sure that the percentage change remains the same after…
Jash
  • 55
  • 8
-1
votes
1 answer

Troubles Training Tensor flow Neural Network, how can I fix this problem?

I am currently training an image classification model with three categories of vehicles (Vans/SUVs, Cars and Trucks). I have 1800 training images and 210 validation images. When I try to plug in the data. I pre-process data with…
Oliver Wesche
  • 49
  • 2
  • 4
-1
votes
1 answer

How many ways to normalize data?

I am curious about how many ways can we normalize data in data processing step before we use it to train machine learning model, deep learning model and so on. All I know is Z-score normalization = (data - mean)/variance. Min-Max normalization =…
Ronakrit W.
  • 693
  • 4
  • 8
-1
votes
2 answers

Find the missing values in data frame using python

A1 A2 A3 B1 B2 B3 aa 1.0 4.0 9 NaN NaN 2.0 bb NaN NaN 9 9.0 2.0 2.0 cc 3.0 2.0 7 1.0 3.0 2.0 dd 4.0 5.0 7 NaN 4.0 5.0 ee 5.0 NaN 1 5.0 1.0 5.0 ff 3.0 5.0 2 5.0 2.0 NaN I need to fill the…
sp29
  • 363
  • 4
  • 11
-1
votes
2 answers

Is there a R package to assist in large data processing?

I am processing a large dataset (after being cleaned). The data set is then processed to create an adjacency matrix, which is passed a logicEval to id obs that contain the uniqueID. 5 When running the code snippet to create adjacency matrix, the…
OctoCatKnows
  • 399
  • 3
  • 17
-1
votes
1 answer

Python Print function adding unwanted new line

I have a script that reads a file. The file contains lines of text data, each line corresponds to a player and each attribute of a player is separated by a tab. My script breaks apart the line into an array for each player, then simply reconstructs…
pho_pho
  • 672
  • 11
  • 30
-1
votes
1 answer

add my function data column into original data in python

I am trying to add my function value into my dataset column. I have eight columns which are: 'DATE','Max_R','Total_R','Avg_R','MAX_T','TOTAL_T','AVG_T' Then I divided my DATE column into three columns as Day, Month and year respectively. Here is my…
Nayan
  • 55
  • 1
  • 4
  • 9
-1
votes
1 answer

How to load (.gz) dataset to google cloud (using python)

I would like to load the huge amount of data which is compressed (.gz) and I don't know how handle with it. My dataset it is pageviews from wikipedia. My goal is to provide basic statistic measures to analyse them. I found this article where is used…
Merix
  • 63
  • 1
  • 9
-1
votes
2 answers

Coerce Large List of Lists to Data Frame Keeping List Property

I have a large list of lists full of IDs with the following properties list1 <- c(145540,145560, 157247, 145566) list2 <- c(166927, NA, NA, NA) list3 <- c(145592, 145560, 145566, NA) list <- list(list1, list2, list3) Now I would like to coerce this…
quantmarketing
  • 37
  • 6
-1
votes
1 answer

Applying neural network algorithms on Encrypted data

I have encrypted text dataset and i want to classify it using neural network algorithm. I know that there is a pattern in the encrypted data. example of my input data : diss%^ghghE(t dffd$#KL*vb xod@#:n>did .... My questions is should i treat…
user780975
  • 13
  • 1
  • 6
-1
votes
3 answers

Read Data from 1.dat File using JavaScript

This is my first question for JavaScript as I want to read data from "1.dat" file into respective text-fields / boxes. The is structure of "1.dat" file 3 "Caroline Martin","2007 E.L Road","Los Angeles CA 87457",43 "Brian Green", "210…
NimishK
  • 21
  • 1
  • 5
-1
votes
1 answer

Impute class values based on map

I want to impute marker classes (either class A or class B), based on proximity of known marker classes. So for example if I know M1 and M4 are class A, then all markers positioned in the map between M1 and M4 can also be classified as A. If I know…
Alpesh Querer
  • 3
  • 1
-1
votes
1 answer

Choice of script language for processing and transposing csv data

I have been provided with some data in an SQL table, which needs to be transformed according to some rules. However, this transformation requires dynamically addressing column names, i.e., the names of columns in which values should be stored are…
JHH
  • 8,567
  • 8
  • 47
  • 91
1 2 3
…
60 61