1

I'm using PowerShell to concatenate social media log files and Facebook, in their infinite wisdom, have added a SECOND row to their header line.

I currently use this:

Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv $FileOut -NoTypeInformation -Append

to concat Twitter logs which works great. Is there a 'simple' way to simply skip the second line in all the files during concatenation? I know I can do some data work after the fact to remove the second row (since it's static) from the resulting file, but I'd rather see if I can just get rid of it from the get go.

I can share the first two lines if anyone is curious, but it's exceptionally long and annoying =D

ETA: Ideally, I either need to figure out a) how to take that concatenation line and have it know that the header is two lines long and then I can just remove the second line from the resulting file OR b) how to get the concatenation process to skip the 2nd line in each file without lots of work.

Here's the first two lines of the file (all that's really relevant, since the second line is the one I want to skip):

"Post ID",Permalink,"Post Message",Type,Countries,Languages,Posted,"Audience Targeting","Lifetime Post Total Reach","Lifetime Post organic reach","Lifetime Post Paid Reach","Lifetime Post Total Impressions","Lifetime Post Organic Impressions","Lifetime Post Paid Impressions","Lifetime Engaged Users","Lifetime Matched Audience Targeting Consumers on Post","Lifetime Matched Audience Targeting Consumptions on Post","Lifetime Negative Feedback from Users","Lifetime Negative Feedback","Lifetime Post Impressions by people who have liked your Page","Lifetime Post reach by people who like your Page","Lifetime Post Paid Impressions by people who have liked your Page","Lifetime Paid reach of a post by people who like your Page","Lifetime People who have liked your Page and engaged with your post","Lifetime Organic views to 95%","Lifetime Organic views to 95%","Lifetime Paid views to 95%","Lifetime Paid views to 95%","Lifetime Organic Video Views","Lifetime Organic Video Views","Lifetime Paid Video Views","Lifetime Paid Video Views","Lifetime Average time video viewed","Lifetime Video length","Lifetime Post Stories by action type - like","Lifetime Post Stories by action type - share","Lifetime Post Stories by action type - comment"
,,,,,,,,"Lifetime: The number of people who had your Page's post enter their screen. Posts include statuses, photos, links, videos and more. (Unique Users)","Lifetime: The number of people who had your Page's post enter their screen through unpaid distribution. (Unique Users)","Lifetime: The number of people who had your Page's post enter their screen through paid distribution such as an ad. (Unique Users)","Lifetime: The number of times your Page's post entered a person's screen. Posts include statuses, photos, links, videos and more. (Total Count)","Lifetime: The number of times your Page's posts entered a person's screen through unpaid distribution. (Total Count)","Lifetime: The number of times your Page's post entered a person's screen through paid distribution such as an ad. (Total Count)","Lifetime: The number of unique people who engaged in certain ways with your Page post, for example by commenting on, liking, sharing, or clicking upon particular elements of the post. (Unique Users)","Lifetime: The number of people who matched the audience targeting that clicked anywhere in your post on News Feed. (Unique Users)","Lifetime: The number of clicks anywhere in your post on News Feed from the user that matched the audience targeting on it. (Total Count)","Lifetime: The number of people who have given negative feedback to your post. (Unique Users)","Lifetime: The number of times people have given negative feedback to your post. (Total Count)","Lifetime: The number of impressions of your Page post to people who have liked your Page. (Total Count)","Lifetime: The number of people who saw your Page post because they've liked your Page (Unique Users)","Lifetime: The number of paid impressions of your Page post to people who have liked your Page. (Total Count)","Lifetime: The number of people who like your Page and who saw your Page post in an ad or sponsored story. (Unique Users)","Lifetime: The number of people who have liked your Page and clicked anywhere in your posts. (Unique Users)","Lifetime: Number of times your video was viewed to 95% of its length without any paid promotion. (Unique Users)","Lifetime: Number of times your video was viewed to 95% of its length without any paid promotion. (Total Count)","Lifetime: Number of times your video was viewed to 95% of its length after paid promotion. (Unique Users)","Lifetime: Number of times your video was viewed to 95% of its length after paid promotion. (Total Count)","Lifetime: Number of times your video was viewed for more than 3 seconds without any paid promotion. (Unique Users)","Lifetime: Number of times your video was viewed for more than 3 seconds without any paid promotion. (Total Count)","Lifetime: Number of times your video was viewed more than 3 seconds after paid promotion. (Unique Users)","Lifetime: Number of times your video was viewed more than 3 seconds after paid promotion. (Total Count)","Lifetime: Average time video viewed (Total Count)","Lifetime: Length of a video post (Total Count)","Lifetime: The number of stories created about your Page post, by action type. (Total Count)",,
John Eddy
  • 51
  • 5
  • 1
    Please share a sample csv that you are importing. – Daniel Sep 08 '21 at 20:04
  • 2
    If you want to skip some lines you can use `Select-Object -Skip [int]` ... in PowerShell version 7+ you can even use `Select-Object -SkipIndex 2,4,5` or something like this – Olaf Sep 08 '21 at 20:04
  • 1
    Please share some sample data and maybe even some desired output data (formatted as code in original formatting please) – Olaf Sep 08 '21 at 20:08
  • I'll add the data in the above (its too long to fit in a comment), tho, it's really not relevant. It is always, literally, the second line I want to skip. Tho if I could simply not concatenate lines where the first two fields are null, that should actually work since actual data rows would not have that be blank. – John Eddy Sep 08 '21 at 20:36
  • 1
    It seems not to be valid CSV. If it's just the second line of the file you want to get rid of why not using PowerShell version 7 and the `-SkipIndex` parameter of `Select-Object`? – Olaf Sep 08 '21 at 20:46
  • Curious why you don't think it's valid CSV? – John Eddy Sep 08 '21 at 21:08
  • 1
    Because I get an error when I try to import it. ;-) `The member "Lifetime Organic views to 95%" is already present.` – Olaf Sep 08 '21 at 21:10
  • 1
    Have you considered using `Select-Object -SkipIndex 1`? – Olaf Sep 08 '21 at 21:11
  • a) Well, I mean, if I had known that -Skip/-SkipIndex existed, I wouldn't have had to ask the question in the first place. b) Ah. Invalid CSV for Import-Csv. That's not exactly the same thing as an invalid CSV file. Also: WTF Facebook. Why do you want to hurt me like this. I'll go see if I can figure out what they're doing and try the skipindex/skip. – John Eddy Sep 08 '21 at 21:39

2 Answers2

0

For 5.1, you can remove specific indexes from an arraylist. The method doesn't output to the pipeline, so it requires a loop:

foreach ($file in (Get-ChildItem -Filter *.csv).FullName) {
  $csv = [System.Collections.ArrayList](Import-Csv $file)
  $csv.RemoveAt(1)
  $csv | Export-Csv $FileOut -NoTypeInformation -Append
}
Cpt.Whale
  • 4,784
  • 1
  • 10
  • 16
0

-Skip/-SkipIndex should work just fine, however Facebook, in their infinite wisdom, have decided to make the items in the first row not unique across the board, so I'm going to have to not go the easy route of just deleting the second row and instead removing the first two lines and adding my own headers so I can work with the data when it's all done.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
John Eddy
  • 51
  • 5