Data Science Cookie Cutter: How to keep multiple versions of raw files

Question

Using the data science cookiecutter, how is it possible to keep multiple versions of raw files?

More precisely, if I have two files

data/raw/scrape_2017_06.json
data/raw/scrape_2017_04.json

How can I assure that make always uses the newest version?

Are we talking about a specific Makefile or just Makefile in general? — akond, Jun 20 '17 at 10:24

score 2 · Accepted Answer · answered Jun 20 '17 at 13:01

As long as you can relay on file names and sorting those files in raw/data by names in lexical order is equivalent of sorting them by their age, following solution would be enough:

DATAFILE:=$(lastword $(sort $(wildcard data/raw/*.json)))

all:
    @echo The latest datafile is $(DATAFILE)

If not, you need to rely on operating system commands. For Unix system many implementations (certainly, the GNU one) support -t to sort by modification time. So it would be:

DATAFILE:=$(firstword $(shell ls -t data/raw/*.json))

Can I use DATAFILE as the dependency of a target? – David Jun 20 '17 at 13:49 — David, Jun 20 '17 at 13:49

Data Science Cookie Cutter: How to keep multiple versions of raw files

1 Answers1