-1

I've been trying to run pandas using python 2.7 on a macbook pro and keep getting the following error:

File "/Users/Hofstadter/anaconda/lib/python2.7/site-packages/pandas/io/common.py", line 376, in _get_handle f = open(path_or_buf, mode)

IOError: [Errno 13] Permission denied: 'datasets/cats_0.8_0.6_0.4_0.2/target.csv'

It looks like for some reason the folder being created for the files below, including target.csv, has restricted permissions. Here's what that code looks like:

def get_tables(df):
    categorical_cols = [col for col in df.columns if col.endswith('_cat')]
    train_table = df[categorical_cols]
    for col in categorical_cols:
        train_table = pd.concat(
            [
                train_table, pd.get_dummies(
                    train_table[col],
                    prefix=col,
                    prefix_sep='_',
                    dummy_na=False).astype(int)
            ],
            axis=1,
            join='inner')
        train_table.drop(col, axis=1, inplace=True)

    print('Tables Created :)')
    return train_table

The tables are created without issue but then I get a permission error when trying to save them as below.

def save_tables(data_path,
                df,
                top_quant,
                mh_quant,
                ml_quant,
                low_quant,
                train=True):

    df = categorize_features(df, top_quant, mh_quant, ml_quant, low_quant)
    X = get_tables(df)

    os.makedirs(data_path, True)

    x_path = '{}/tournament_table.csv'.format(data_path)

    if train:
        x_path = '{}/train_table.csv'.format(data_path)
        y = df['target'].to_frame()
        y.columns = ['target']
        y.to_csv('{}/target.csv'.format(data_path), index=False)
    else:
        ids = df['id'].to_frame()
        ids.columns = ['id']
        ids.to_csv('{}/ids.csv'.format(data_path), index=False)

    X.to_csv(x_path, index=False)
114
  • 876
  • 3
  • 25
  • 51
  • 2
    The error arises because you are trying to use some file you don't have the permissions to, not because `/Users/Hofstadter/anaconda/lib/python2.7/site-packages/pandas/io/common.py` doesn't have the right permissions. – juanpa.arrivillaga Jun 05 '17 at 00:04
  • @juanpa.arrivillaga Hmm, okay but why wouldn't I have access to part of the standard pandas library? Is there a way I can reinstall pandas so that I do have the correct permission? I'm using Anaconda if that helps. – 114 Jun 05 '17 at 00:08
  • If you are `Hofstadter` and the file belongs to you, then you must use `u+r` or `a+r`, not `o+r`, to make it readable. But what is the value of `mode`? – DYZ Jun 05 '17 at 00:24
  • 1
    No, you aren't getting me, I'm saying *the problem isn't the pandas source*, it is some file you are trying to open with `pandas`. – juanpa.arrivillaga Jun 05 '17 at 00:36
  • Please show the _complete_ error message. – DYZ Jun 05 '17 at 00:39
  • @juanpa.arrivillaga Right, sorry about that, I see what you mean now. – 114 Jun 05 '17 at 00:48
  • @DYZ `IOError: [Errno 13] Permission denied: 'datasets/cats_0.8_0.6_0.4_0.2/target.csv'` where target.csv is created as in my edited post. – 114 Jun 05 '17 at 01:02
  • Then it's `target.csv` that has wrong permissions or perhaps does not even exist. – DYZ Jun 05 '17 at 02:28
  • @DYZ Hmm but target.csv should be created using the code above, so why would a newly created file have the wrong permissions? – 114 Jun 05 '17 at 13:05

1 Answers1

0

I suspect your issue is not the call to_csv() but where you are calling os.makedirs.

Calling os.makedirs(data_path, True) is interpreting the True as the parameter for mode. See:

makedirs(name [, mode=0o777][, exist_ok=False])

If you want to use the default mode for the new directories but ignore existing, your call should be os.makedirs(data_path, exist_ok=True)

Steve
  • 19
  • 3
  • Thanks, but if I make that change I get the following error: File "get_dataset.py", line 62, in save_tables os.makedirs(data_path, exist_ok=True) TypeError: makedirs() got an unexpected keyword argument 'exist_ok' – 114 Jun 05 '17 at 13:09
  • I see. You are using Python 2.7. `os.makedirs()` is different. See the [docs](https://docs.python.org/2.7/library/os.html#os.makedirs). Just call it like `os.makedirs(data_path)` but you will have to manually handle the exception case if the directories already exist. The `exist_ok` options is only available in Python 3. – Steve Jun 05 '17 at 23:24
  • Thanks so much, I should have looked at that. So now surprisingly the file is created without any permissions issues. However I now get the following error (I've confirmed that the folder did not exist before running the program and that it did afterwards): `File "/Users/Hofstadter/anaconda/lib/python2.7/os.py", line 157, in makedirs mkdir(name, mode)` `OSError: [Errno 17] File exists: '/Users/Hofstadter/path/cats_0.8_0.6_0.4_0.2'` – 114 Jun 07 '17 at 14:48