The python data access and cataloguing project "Intake", https://intake.readthedocs.io/en/latest/
Questions tagged [intake]
25 questions
1
vote
1 answer
Intake Cache specify filename/location
I'm trying to use intake and the intake-xarray to open and store remote files. I have a minimized catalog file here:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/ice_sheet_reconstructions.yaml
It looks like…
pgierz
- 674
- 3
- 7
- 14
1
vote
1 answer
Moving data from a database to Azure blob storage
I'm able to use dask.dataframe.read_sql_table to read the data e.g. df = dd.read_sql_table(table='TABLE', uri=uri, index_col='field', npartitions=N)
What would be the next (best) steps to saving it as a parquet file in Azure blob storage?
From my…
Ray Bell
- 1,508
- 4
- 18
- 45
1
vote
1 answer
Persisting only part of a data source
I'm using intake to access the catalog catalog.ocean.GFDL_CM2_6.GFDL_CM2_6_control_ocean_surface.
At the moment I only work with small patches of that data, but accessing that data every single time is still quite costly (it's on Google Cloud…
ArthurPGB
- 43
- 4
0
votes
0 answers
How to filter a catalog by date when importing intake in Python?
I have the following code
import intake
import pandas as pd
# Importing json file
catalog_file = "../my/path/path.json"
#using `intake` to manage the data
cat = intake.open_esm_datastore(catalog_file)
# Furthering filtering the data
hits =…
jei L
- 33
- 6
0
votes
1 answer
Inake-xarray: AttributeError: 'list' object has no attribute 'startswith'
I try to run this code, which ought to work...
import xarray as xr
import intake
import hvplot.xarray
import dask
catalog_url =…
Jonathan Roy
- 405
- 1
- 6
- 18
0
votes
1 answer
Hiding secrets in intake catalog for remote access (S3/MinIO)
I'm trying to build an intake catalog for my team. The datasets are on a shared MinIO server for which each user should have their own service account, and therefore a key/secret pair.
When creating the first catalog entry like this:
source =…
Adrian
- 755
- 9
- 17
0
votes
2 answers
Handle environmental variables in config options
I have snakemake command line with configuration options like this:
snakemake --config \
f1=$PWD/file1.txt \
f2=$PWD/file2.txt \
f3=/path/to/file3.txt \
...more key-value pairs \
--directory /path/to/output/dir
file1.txt and…
dariober
- 8,240
- 3
- 30
- 47
0
votes
2 answers
Relative paths in config.yaml for Snakefile
How can I use relative paths in my configuration file so that users do not need to change USER in the paths for output directories?
I have this:
config.yml
proj_name: H1N1_rhesus
contact:
email: user.edu
person: user
01-preprocess:…
Noah_Seagull
- 337
- 5
- 18
0
votes
1 answer
Add parameters to Python Intake LocalCatalogEntry
I'm trying to build a LocalCatalogEntry for Python's Intake package (as part of a larger catalog, which might have multiple entries, one of which I'm trying to create here). However, I can't seem to figure out how to feed it user parameters to…
Jessica
- 505
- 1
- 3
- 11
0
votes
1 answer
'import intake' in google colab generates ContextualVersionConflict
!pip install intake-esm
install the latest version of intake-esm (2020.12.18) in google colab
but the import intake statement generates the following error:
ContextualVersionConflict: (requests 2.23.0 (/usr/local/lib/python3.6/dist-packages),…
aparnadeepak101
- 19
- 4