0

In windows I'm measuring the different times between launching a command from within cygwin and launching it from powershell, and I get huge time differences:

Trying to count the lines in a 2.7GB file using wc -l from within cygwin takes 5 secons, but when done from powershell Get-Content file.txt|Measure-Object -line takes EONS, I just got bored after a few minutes and canceled the command.

EDIT: To be fair I tried cat file|wc -l and the result takes 10 seconds nowhere near the (minutes/hours) times on powershell.

Am I doing something wrong or is it just that powershell is useless?

YoMismo
  • 103
  • 3
  • Short description of the dupe : read the file by chunks (100/1000/whatever lines per read) and increment a variable by the number of read lines – Cid Jul 15 '21 at 07:33
  • Well, it's not useless, you are telling it to do something different then you probably expect it to do. Basically, you are first loading the whole file (2.7GB) in memory. Which is not the same as `wc` is doing. – Michaël Hompus Jul 15 '21 at 07:34
  • 2
    the `wc` command is a specialized util for doing a very very very narrow range of things. powershell does a LOT more ... even when you don't want it. [*grin*] for instance, the `Get-Content` cmdlet grabs each line, builds an object, _and adds lots of details about each line to that object._ when you need to deal with large files ... the _native_ PoSh cmdlets are poor choices. **_however, you can access dotnet stuff from PoSh that can read files quite quickly._** something like `streamreader` is one option. – Lee_Dailey Jul 15 '21 at 07:45
  • It is definitely not useless, but requires an deeper understanding of the (dis)advantages and how to use the [PowerShell pipeline](https://learn.microsoft.com/powershell/module/microsoft.powershell.core/about/about_pipelines) – iRon Jul 15 '21 at 07:46
  • @Cid yes it does, it takes 32 seconds, 3 times the `cat` and 6 times more than 'wget' alone. I don't know if this is a duplicated question though – YoMismo Jul 15 '21 at 07:56
  • @Lee_Dailey, thanks for the info. I'm probably used to Linux, I do not want a command to do "more stuff" than what I need, specially when I'm doing batch stuff with huge (2.7G is not much) files. If I need to use .net stuff then I wouldn't need shell stuff, I may not be able to see the potencial of powershell, but linux commands appear to be way more flexible (just because each command does one thing fast). I was trying to add every 1000 lines a text, with linux awk command it takes seconds, but I could't find a fast way to do it on powershell, maybe .net will do but thats beyond shell IMO. – YoMismo Jul 15 '21 at 08:09
  • See also: [PowerShell scripting performance considerations](https://learn.microsoft.com/powershell/scripting/dev-cross-plat/performance/script-authoring-considerations) – iRon Jul 15 '21 at 08:22
  • Thanks @iRon, I'll check those 2 links out. – YoMismo Jul 15 '21 at 09:16
  • @YoMismo - you are welcome! [*grin*] ///// if you have access to 'nix AND you are dealing with just plain text ... then stay with the 'nix stuff. if you can't use 'nix, then look at calling dotnet from inside PoSh. ///// from what i can tell, you likely otta stick with 'nix utils if at all possible. – Lee_Dailey Jul 15 '21 at 11:27
  • @Lee_Dailey I wish, but that's the problem. I'm sadly wording on Windows now :(, things that took a couple of seconds and were made on an single line now appears to need hours or .net and a block of code :/. Cid's comment helped to speed up what I posted, but what I really needed was inserting lines every 1000 lines or so, awk does that in seconds (less than a minute) in that file, and using Cid's link solution still takes hours. – YoMismo Jul 15 '21 at 16:01
  • @YoMismo - ouch! [*frown*] if you are allowed, i would install wsl and use that. Install WSL on Windows 10 | Microsoft Docs — https://learn.microsoft.com/en-us/windows/wsl/install-win10 – Lee_Dailey Jul 15 '21 at 23:16
  • @Lee_Dailey, thanks, I used cygwin in my machine because it didn't need system install privileges. On servers I'll have to ask for what you suggested and see if there's luck ;) Thanks. – YoMismo Jul 16 '21 at 14:21
  • @YoMismo - you are most welcome! glad to have helped a little bit ... [*grin*] – Lee_Dailey Jul 17 '21 at 04:15

0 Answers0