0

I have a CSV file like this:

2021-08-09 15:50:44      38962 part-00000-6baa0883-5212-49f7-9ba2-63a352211fdd-c000.snappy.parquet
2021-08-09 16:50:44      38962 part-00000-6baa0883-5212-49f7-9ba2-63a352211fdd-c000.snappy.parquet

I'd like to extract all the timestamps into one list so that I can perform the evaluation function below (ie evaluating if check_timestamps_updated is true).

Problem is also taking the date into account, not just the time. What's the most efficient way of combining the two separate columns (date and time) from the csvreader object so that it can be compared with control_time?

from datetime import datetime as dt

control_time = str(str(dt.now()))
reader = csv.reader(results, delimiter=" ")
        time_column = list(zip(*reader))[1]
        check_timestamps_updated = all(i >= control_time for i in time_column)
PianoTiger
  • 41
  • 7
  • If you just want to compare the time then probably want to set the control time as `control_time = str(dt.now().time())` so your only taking the time part from dt not the date – Chris Doyle Aug 09 '21 at 21:41
  • That's the thing - I had that before and it worked. Then it gave a false positive because the date also matters. Hence, I need to include the date as well... – PianoTiger Aug 09 '21 at 22:22

1 Answers1

0

As far as I understand what you want to do can be implemented as below,

import csv
from datetime import datetime as dt

check_timestamps_updated = True
control_time = dt.now().timestamp()
with open('example.csv', newline='\n') as f:
    reader = csv.reader(f, delimiter=" ")
    for line in reader:
        date = dt.strptime(f'{line[0]} {line[1]}', '%Y-%m-%d %H:%M:%S').timestamp()
        if date >= control_time:
            check_timestamps_updated = False
print(check_timestamps_updated)

You asked the most efficient way to merge two columns but I think it depends on what you are mentioning as efficiency. If the csv file is too big and if there is a chance to have a memory problem what I implemented above can work without an issue. But if you are mentioning time this is still a good one.

Cenk Bircanoglu
  • 499
  • 4
  • 10