Questions tagged [data-engineering]
69 questions
0
votes
1 answer
How to copy folders+files as per the folder names passed as parameter in Azure data factory
I have an on-prem file storage, there I have a folder named 'India'. Inside 'India', there are 36 folders with all the state/union teritory names. I want to build a copy data pipeline that takes a parameter for state names, and while copying it will…
0
votes
1 answer
how do i parse a date field column effectively for a machine learning model
I am try to predict the stock price of IBM. but i have gottchas on handling the date column field for model training in a linear regression algorithm.
this is how my dataset looks like:
Date Open High Low Close Adj…
geek
- 307
- 2
- 10
-1
votes
1 answer
MSSQL Bulk insert file path : Cannot bulk load because the file
CSV to SQL server data ingestion using python getting issue in file path.
"/Users/raj/Desktop/csv_to_mssql/file_name.csv" could not be opened. Operating system error code 3(The system cannot find the path specified.).
Running this from local MacBook…
PRAVIN MASKE
- 31
- 4
-1
votes
1 answer
trying to organize ETL pipelines
I have written several data pipelines in SSIS and Python. This are starting to become hard to track and schedule. There is an increasing need for them to be automated to run automatically at night when the the OLTP systems are least used.
I have…
David Mwendwa
- 1
- 2
-1
votes
1 answer
Is it necessary to have same format of json for flatting the file?
you can see a different response from API
Explanation:
In the above-shared screenshot, I compared two different files. On the left side is the data of
web page no 1 while on the right side, I get the data of web page no 1080.
You can see that on web…
junaidbilal
- 7
- 3
-2
votes
0 answers
i m not able to crawl https://www.chictr.org.cn/ due to its cookies
I'm not able to crawl https://www.chictr.org.cn/ due to its cookies.
This is an chinese website. We have captcha which is fairly simple actually only need to slide but it detects if you are bot.
On every new requests they have TraceID.
If we don't…
-2
votes
1 answer
Is there any data scaling methods except for Min-Max Normalization and Quantile transformation that keeps the range between [0,1]?
I have been working on a machine learning project and I have been trying to scale the features before feeding it to my model. I know Min-Max Normalization and Quantile Transformation scales down the features in the range of 0 and 1. I was wondering…
Jacob Issac
- 5
- 2