How to extract (import) data from a mainframe dataset to excel table

Question

I want to build a little application that calculates the critical batch of a batch flow. As input I need to use a Mainframe dataset. If possible, being dynamic, that is, I can choose the fields that apply at the time.

I've searched the internet about that but found nothing that suited what I wanted to do. Is there a way to do that?

Answer 1

I have a dataset in a mainframe library and I want to ftp that file to Excel.

Convert the file to CSV on the mainframe (for example, via a REXX exec, a z/OS UNIX shell script, or a Lua4z program), and then insert that CSV file into Excel via FTP.

You do not need to transfer the CSV file to your PC's file system and then, as a separate step, open it in Excel.

Instead, you define the FTP (or HTTP) URL for the CSV as a data source in Excel. One advantage of this technique is that you can refresh the data from that URL without having to reapply formatting in Excel.

There are various tutorials on the web for doing this.

In brief:

1. Create a new blank workbook (I'm using Excel 2010).
2. Select the first cell in the empty worksheet (this step is unnecessary - the cell is already selected - if you've only just created the workbook).
3. On the Data tab, click From Text

4. In the File name text box of the Import Text File dialog, enter the FTP URL of the CSV file. For example:

   ftp://zos1//u/me/data.csv

   (This assumes that your mainframe is configured to allow FTP using this path.)

   The two consecutive slash (/) characters following the host name (zos1) indicate that the path refers to a z/OS UNIX file (/u/me/data.csv).

   The CSV file must be in a z/OS UNIX path. The FTP client does not accept MVS-style (dsname) paths such as 'me.csv(data)' (even when URL-encoded; that is, with the single quotes escaped as %27); by contrast, cURL accepts such paths just fine.

   The CSV file on the mainframe must be ASCII encoded, not EBCDIC. (Here, I'm using the term ASCII imprecisely: the precise character encoding you want depends on your PC's settings. You probably want Windows-1252.) This is because the FTP client sets the default transfer type to binary.

5. Enter your user name and password (your z/OS TSO user ID and password).
6. Wait for the data to load.
7. Format the cells. For example, set the format of any columns containing date/time values.
8. On the Data tab, click Connections, select the connection (that Excel created when you specified a URL for the file name), and clear the check box Prompt for file name on refresh.

To refresh the data, replacing the current data with the results of a new FTP request: on the Data tab, click Refresh All. The data is replaced; the cell formatting remains intact.

Converting an EBCDIC-encoded CSV file to ASCII

(Strictly speaking, I mean ISO-8859, not ASCII.)

Suppose you have JCL that generates a CSV file encoded in EBCDIC. You want to make that CSV file available to Excel via FTP as an ASCII-encoded z/OS UNIX (zFS) file.

Replace your existing DD statement for the output CSV file with the following DD statement:

//OUTCSV      DD PATH='/u/me/data-ebcdic.csv',       
//            PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
//            PATHDISP=(KEEP,DELETE),          
//            PATHMODE=(SIRUSR,SIWUSR,SIRGRP), 
//            FILEDATA=TEXT                    

Replace the ddname OUTCSV with your ddname, and the zFS file path /u/me/data-ebcdic.csv with the path that you want to use.

Thanks to the FILEDATA=TEXT parameter, the resulting CSV file will have a X'15' byte at the end of each line.

Append the following step to your JCL:

//ICONV   EXEC PGM=IKJEFT01
//SYSTSIN DD *
BPXBATCH sh iconv -f IBM-037 -t iso8859-1 +
 /u/me/data-ebcdic.csv +
 > /u/me/data-ascii.csv
/*
//SYSPRINT DD  SYSOUT=*
//SYSTSPRT DD SYSOUT=*

In case you're wondering why I'm calling iconv as a shell command via BPXBATCH, the following:

//ICONV    EXEC PGM=EDCICONV
//         PARM=('FROMCODE(IBM-037),TOCODE(iso8859-1)')

didn't quite work: it left the X'15' bytes as is, whereas running iconv as a shell command correctly converted them to X'0A'. (z/OS 2.2.)

Answer 2

You've got some good information in the comments, consensus appears to be conversion to CSV (or TSV to avoid commas embedded in your data) is the easiest route. Here is a bit more information, copied from another answer...

I would strongly suggest you get the files into a text format before transferring them to another box with a different code page. Trying to deal with mixed text (which must have its code page translated) and binary (which must not have its code page translated but which likely must be converted from big endian to little endian) is harder than doing the conversion up front.

The conversion can likely be done via the SORT utility on the mainframe. Mainframe SORT utilities tend to have extensive data manipulation functions. There are other mechanisms you could use (other utilities, custom code written in the language of your choice, purchased packages) but this is what we tend to do in these circumstances.

Once you have your flat files converted such that all data is text, you can transfer them via FTP or SFTP or FTPS.

...and thanks for coming back and adding more information. Hopefully the people here have provided enough information to help you solve your problem.

Answer 3

XML would be another possible text oriented solution. It would take more effort to create, but you could design your spreadsheet in Excel and save as an XML document, then write a program to generate the xml text using the data from your mainframe dataset. While this would be more difficult to implement than a simple CSV or TSV file, it has the advantage of implementing the spreadsheet formulas and attributes that a CSV file can not do. Another advantage, you can attach the XML document to an SMTP email note and deliver the document in "spreadsheet format" to your client.