From unix time in seconds to datetime in ISO-8601

Question

TL;DR. How to go from unix time in seconds to datetime in ISO-8601 format ? This looks more like a library question than a language question.

I would like to be able to interpret the ping timestamp (unix time + microseconds) as a datetime to be able to visualize it or perform date and time operations on it. I have a long ping trace (thousands of lines) where each line indicates the success or failure of the ping.

I already managed to make a toy example using a regular expression to get the unix time (without the microseconds) but I didn't find how to use datetimes in APL. I found the APLtree project on Github by the APLteam and their sub-project called DateAndTime but I couldn't intall it. I don't know how to proceed.

Note that I know that I already have the datetime written explicitly in ISO-8601 format in front of the ping timestamp but for the sake of learning the basics of APL I would like to manipulate datetimes. I could also perform operations on the unix times and then try (but how) to convert them to a datetime in ISO-8601 format.

The trace is generated on Linux with the command:

ping -n -i 30 -O -D my.domain.name | while read pong; do echo "[WAN] $(date --iso-8601=seconds): $pong"; done

The output trace is:

[WAN] 2020-01-31T18:02:35+0100: [1580490155.323878] 64 bytes from 8.8.8.8: icmp_seq=1037 ttl=234 time=720 ms
[WAN] 2020-01-31T18:03:34+0100: [1580490214.630890] no answer yet for icmp_seq=1038

The APL program:

      v←('\[(\d+)\.'⎕S'\1')⊃⎕NGET'/Users/ludo/Desktop/so/ping.txt' 1
      v
 1580490155  1580490214
      ⍴v
2
      DISPLAY v
┌→──────────────────────────┐
│ ┌→─────────┐ ┌→─────────┐ │
│ │1580490155│ │1580490214│ │
│ └──────────┘ └──────────┘ │
└∊──────────────────────────┘

I have another question: why the rank difference between the usage of [] and ⊃ ? Do we have a vector in one case and a simple string in the other one (scalar) ? Why is ⍴⍴v[1] zero ? See the example below.

      v←⎕NGET'/Users/ludo/Desktop/so/ping.txt' 1
      v
  [WAN] 2020-01-31T18:02:35+0100: [1580490155.323878] 64 bytes from 8.8.8.8: ic
      mp_seq=1037 ttl=234 time=720 ms  [WAN] 2020-01-31T18:03:34+0100: [1580490
      214.630890] no answer yet for icmp_seq=1038   UTF-8-NOBOM  13 10
      DISPLAY v
┌→───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ┌→─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ┌→──────────┐ ┌→────┐ │
│ │ ┌→───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ┌→──────────────────────────────────────────────────────────────────────────────────┐ │ │UTF-8-NOBOM│ │13 10│ │
│ │ │[WAN] 2020-01-31T18:02:35+0100: [1580490155.323878] 64 bytes from 8.8.8.8: icmp_seq=1037 ttl=234 time=720 ms│ │[WAN] 2020-01-31T18:03:34+0100: [1580490214.630890] no answer yet for icmp_seq=1038│ │ └───────────┘ └~────┘ │
│ │ └────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ └───────────────────────────────────────────────────────────────────────────────────┘ │                       │
│ └∊─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘                       │
└∊───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
      ⍴v
3
      v[1]
  [WAN] 2020-01-31T18:02:35+0100: [1580490155.323878] 64 bytes from 8.8.8.8: ic
      mp_seq=1037 ttl=234 time=720 ms  [WAN] 2020-01-31T18:03:34+0100: [1580490
      214.630890] no answer yet for icmp_seq=1038
      ⊃v
 [WAN] 2020-01-31T18:02:35+0100: [1580490155.323878] 64 bytes from 8.8.8.8: icm
      p_seq=1037 ttl=234 time=720 ms  [WAN] 2020-01-31T18:03:34+0100: [15804902
      14.630890] no answer yet for icmp_seq=1038
      ⍴v[1]

      ⍴⊃v
2
      ⍴⍴v[1]
0
      ⍴⍴⊃v
1

Maybe `https://stackoverflow.com/q/58300764/452614` which uses `dfns` could be the way to go. I will check this. — Ludovic Kuty, Feb 02 '20 at 07:30
Something like this might work : `date¨ (days 1970 1 1)+seconds÷86400` where `seconds` is `2⊃¨⎕VFI¨ ('\[(\d+)\.'⎕S'\1')⊃⎕NGET'/Users/ludo/Desktop/so/ping.txt' 1`. `⎕VFI` is used to convert from string to number. Datetime are in UTC. — Ludovic Kuty, Feb 02 '20 at 08:05
I should mention that Dyalog 18.0 will have a built-in, `⎕C` to do the `date`/`days` conversion, and possibly a tool to do the ISO formatting too. — Adám, Feb 02 '20 at 08:19

score 1 · Accepted Answer · answered Feb 02 '20 at 08:12

How to go from unix time in seconds to datetime in ISO-8601 format ?

      'date' 'days'⎕CY'dfns'
      unix0←days 1970 1 1
      sPerDay←24×60×60
      UnixToTs←date unix0+÷∘sPerDay
      TsToIso←{'--T::'@(2+3×⍳5)∊⍕¨(⊃,100+1∘↓)6↑⍵}   ⍝ modified from aplcart.info?q=iso8601
      UnixToIso←TsToIso UnixToTs
      UnixToIso 1580490155
2020-01-31T17:02:35

Try it online!

For your convenience, here's the reverse:

      IsoToTs←⍎¨∊∘⎕D⊆⊢
      TsToUnix←sPerDay×unix0-⍨days
      IsoToUnix←TsToUnix IsoToTs
      IsoToUnix '2020-01-31T17:02:35'
1580490155

Try it online!

using a regular expression to get the unix time

If your file is consistent, I recommend using APL to extract the data, as it can make a huge speed difference. For example, if all the lines are using this exact format:

      v←(⍎¯10↑43∘↑)¨⊃⎕NGET'/Users/ludo/Desktop/so/ping.txt' 1
      DISPLAY v
┌→────────────────────┐
│1580490155 1580490214│
└~────────────────────┘

why the rank difference between the usage of [] and ⊃ ?

[…] never discloses ("opens up") elements it extracts from an array, so nested array's elements stay enclosed.

Do we have a vector in one case and a simple string in the other one (scalar) ?

Yes, although it is better to phrase it as a vector in one case and an enclosed vector in the other.

Why is ⍴⍴v[1] zero ?

Because the rank of a scalar (even if it contains a higher-rank array) is zero.

Thanks for the quick answer. It contains a lot of useful information which will provide food for thoughts in the days to come. For the speed, I indeed noticed a 1 or 2 seconds delay to process 5500 lines with the regex. — Ludovic Kuty, Feb 02 '20 at 08:17

From unix time in seconds to datetime in ISO-8601

1 Answers1