Questions tagged [data-import]

The import and export of data is the automated or semi-automated input and output of data sets between different software applications.

The import and export of data is the automated or semi-automated input and output of data sets between different software applications. It involves "translating" from the format used in one application into that used by another, where such translation is accomplished automatically via machine processes, such as transcoding, data transformation, and others. True exports of data often contain data in raw formats otherwise unreadable to end-users without the user interface that was designed to render it.

Import and export of data shares semantic analogy with copying and pasting, in that sets of data are copied from one application and pasted into another. In fact, the software development behind operating system clipboards (and clipboard extender apps) greatly concerns the many details and challenges of data transformation and transcoding, in order to present the end user with the illusion of effortless copy and paste between any two apps, no matter how internally different. The "Save As" command in many applications requires much of the same engineering, when files are saved as another file format.

The ability to import and export data (or lack of such ability) has large economic implications, because it can be resource-intensive to input data in non-automated ways (such as manual rekeying), and because lack of interoperability between systems unable to import or export data between each other causes stovepiping and lack of opportunity and efficiencies such as those seen in, for example, mash-ups.

Source: Wikipedia.org (Access date: 2016-Feb-17) https://en.wikipedia.org/wiki/Import_and_export_of_data

369 questions
57
votes
5 answers

'utf-8' codec can't decode byte 0x92 in position 18: invalid start byte

I am trying to read in a dataset called df1, but it does not work import pandas as pd df1=pd.read_csv("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv",sep=";") df1.head() Here are huge errors from the…
Tuyen
  • 977
  • 1
  • 8
  • 23
54
votes
7 answers

Can you automatically create a mysqldump file that doesn't enforce foreign key constraints?

When I run a mysqldump command on my database and then try to import it, it fails as it attempts to create the tables alphabetically, even though they may have a foreign key that references a table later in the file. There doesn't appear to be…
Tai Squared
  • 12,273
  • 24
  • 72
  • 82
51
votes
9 answers

Variable 'sql_mode' can't be set to the value of 'NO_AUTO_CREATE_USER'

I am using MySQL Workbench 8.0. I am trying to dump test data to DB including all the tables, stored procedures and views with data. When I try to import it's says import finished with one error and the error is Variable 'sql_mode' can't be set to…
Rajeswari ML
  • 549
  • 1
  • 5
  • 12
13
votes
1 answer

How to skip an unknown number of empty lines before header on pandas.read_csv?

I want to read a dataframe from a csv file where the header is not in the first line. For example: In [1]: import pandas as pd In [2]: import io In [3]: temp=u"""#Comment 1 ...: #Comment 2 ...: ...: #The previous line is empty ...:…
bmello
  • 1,864
  • 3
  • 18
  • 23
9
votes
1 answer

how to import parquet files from s3 to postgresql rds

I'm trying to load a parquet file from s3 to AWS postgresql RDS. The data import using aws_s3.table_import_from_s3 is working fine for csv file but when I tried the similar way for parquet file, I get the below error: ERROR: invalid byte sequence…
swetha
  • 91
  • 1
  • 3
9
votes
3 answers

Power Bi PostgreSQL data import error - The remote certificate is invalid according to the validation procedure.'"

I am trying to connect PostgreSQL database to Power Bi. Database is hosted on Heroku. I am getting the following error - An error happened while reading data from the provider: 'The remote certificate is invalid according to the validation…
shubham.a
  • 91
  • 1
  • 1
  • 3
9
votes
0 answers

Input data into R from .dat and .sps files

I've been trying to import data into R using a .dat and .sps files. The .dat file has no headers and varying lengths of columns which are of course included in the .sps file. It also contains missing values. I've tried using spss.fixed.file from…
Juan
  • 121
  • 1
  • 3
8
votes
2 answers

Is there a way other than below to load a json.rows file into RStudio?

I have a json.rows file -> instances.json.rows with approximately 223k rows I tried using jsonlite and came up with instancesfile <- fromJSON("instances.json.rows") But i kept getting an error Error in parse_con(txt, bigint_as_char) : parse error:…
ak95
  • 85
  • 1
  • 6
8
votes
2 answers
8
votes
1 answer

How can I import .OWL file into Neo4J?

I have been playing with Neo4J to get a feel on how easy it is to work with related data. so far I have work with the Neo4J community browser to create nodes and build relationship between them. I have developed an ontology using protege a link!.…
kxmehdi
  • 81
  • 1
  • 2
8
votes
4 answers

Fastest way to import CSV files in MATLAB

I've written a script that saves its output to a CSV file for later reference, but the second script for importing the data takes an ungainly amount of time to read it back in. The data is in the following…
Doresoom
  • 7,398
  • 14
  • 47
  • 61
7
votes
1 answer

Import txt file in R ignoring first few lines

downloaded data from the MET office on rainfall for Scotland. First few lines: Scotland Rainfall (mm) Areal series, starting from 1910 Allowances have been made for topographic, coastal and urban effects where relationships are found to…
Shery
  • 1,808
  • 5
  • 27
  • 51
7
votes
2 answers

Importing data from an XML file into R

I want to import an XML file from polarpersonaltrainer.com that stores heartrate data into a R data.frame. Is there a package that makes importing XML easy or do I have to write my own parser?
Christian
  • 25,249
  • 40
  • 134
  • 225
7
votes
4 answers

How can I do indexing .html files in SOLR

The files I want to do indexing is stored on the server(I don't need to crawl). /path/to/files/ the sample HTML file is
5
votes
4 answers

Import data from txt to Mathematica

Consider the following list in mathematica: a = { { {0, 0, 0}, {1, 0, 0}, {1, 1, 0} }, { {0, 0, 1}, {1, 0, 1}, {1, 1, 1} } }; Now, invoke: Export["test.dat", a] and then b = Import["test.dat"] You will see that at the end…
Dror
  • 12,174
  • 21
  • 90
  • 160
1
2 3
24 25