0

I am new to DataBricks and Apache Spark. I would like to use Python for my notebook.

I am using data bricks and I can confirm I have created:

  • A Cluster (test) along with a attached python based notebook (test-notebook)
  • A Table from JSON File (employee_info)
  • A Notebook.

The Sample Table I have is from a JSON which I have uploaded as test file is as below:

{ 
 "FirstName": "John",
 "LastName": "Mark",
 "MiddleName": "Lewis",
 "username": "johnlewis2",
 "Id ": "29103-1201-E",
 "YearJoined": "2014",
 "MonthJoined": "7",
 "DayJoined": "23",
 "TimeJoined": "14:32",
 "Manager": "Larry Hins",
 "Position": "Web Developer",
 "Building": "Blue Building",
 "Floor": "2"
 }

If I have uploaded this table, how can I (Assuming I am using Python for the Notebook):

  • Access the table
  • Duplicate a whole row
  • Save the table with a duplicated column as a new table

Thanks.

timotree
  • 1,325
  • 9
  • 29
Techno04335
  • 1,365
  • 6
  • 22
  • 43
  • What do you mean by "A whole row"? – timotree Dec 05 '16 at 18:11
  • @timotree, If I wanted to copy the credentials via "John", "Mark", "Lewis" and so on and make another row of its copy. Resulting in an example column: "firstname:" "John" "John' – Techno04335 Dec 05 '16 at 20:15
  • So you want each label (left-hand side) to have two values? – timotree Dec 05 '16 at 20:18
  • @timeotree, yes or in a better viewed manner. The Left side would be the columns and a row would consist of the actual right side credentials, I would like to create another duplicate row. – Techno04335 Dec 05 '16 at 20:27
  • I understand what you mean now. You're thinking of JSON as being like a grid or spreadsheet. I will give you a soluting for making more 'columns' but this is not the way JSON is designed to do more than one of something. – timotree Dec 05 '16 at 20:29
  • @timotree, thanks! I do not really have to duplicate it but what I wanted to do is to test transformation of data using spark. – Techno04335 Dec 05 '16 at 20:32
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/129836/discussion-between-timotree-and-techno04335). – timotree Dec 05 '16 at 20:36

0 Answers0