2

I am trying to sort a data set and writing into a new data set, Every time i am executing the job, its abending (ABEND=U0016).

message form SYSOUT.

    WER276B  SYSDIAG= 24646562, 29667262, 29667262, 27500165            
    WER164B  307,288K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,
    WER164B     0 BYTES RESERVE REQUESTED, 307,272K BYTES USED          
    WER036B G=5174,B=1,BIAS=99 
    WER162B 75 PREALLOCATED SORTWORK TRACKS, 3,750,000 DYNAMICALLY 
             ALLOCATED, 
    WER162B 26,721,480 ACQUIRED IN 2,230 SECONDARY EXTENTS, 0 RELEASED, 
             TOTAL 30,471,555 TRACKS USED
    WER046A  SORT CAPACITY EXCEEDED              
    WER493I  ZIIP PROCESSOR USED                 
    WER211B  SYNCSMF  CALLED BY SYNCSORT; RC=0000
    WER449I  SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE
    WER066A  APROX RCD CNT        30430982 

Here is my code:

    //STEP50  EXEC SORTD                                             
    //SORTIN   DD DSN=FILEXYZ(0),                          
    //            DISP=SHR,DCB=BUFNO=192                             
    //SORTOUT  DD DSN=FILE2XXY,                            
    //            DISP=(NEW,CATLG,DELETE),RETPD=365,VOL=(,,,99),     
    //            DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),    
    //            UNIT=(TAPE,2)                                      
    //SYSOUT   DD SYSOUT=*                                           
    //SYSPRINT DD SYSOUT=*                                           
    //SYSIN    DD *                                                  
           SORT FIELDS=(19,5,PD,A,8,6,PD,A,50,2,ZD,A)                  
           OUTREC IFTHEN=(WHEN=(70,18,CH,EQ,C' encoding="IBM037"'),    
                         OVERLAY=(70:C'  encoding="UTF-8"'))             
           OPTION DYNALLOC=(SYSDA,255)                                 
    /*  

Here is code from SORTD from the jcl:

       4 //STEP40  EXEC SORTD                                                  
       5 XXSORTD   PROC CYLS=1,SYSFICH='*',                                    
         XX     DMPCLAS='1,DEST=ABNORMAL'                                      
         XX*                                                                   
         XX*    REMOVED SEP PARAMETER 89/07/20                                 
         XX*                                                                   
       6 XXSORTD    EXEC PGM=SORT                                              
       7 //SYSOUT   DD SYSOUT=*                                                
         X/SYSOUT   DD SYSOUT=&SYSFICH                                         
         IEFC653I SUBSTITUTION JCL - SYSOUT=*                                  
       8 XXSORTWK01 DD DSN=&WORK1,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK1,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
       9 XXSORTWK02 DD DSN=&WORK2,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK2,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      10 XXSORTWK03 DD DSN=&WORK3,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK3,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      11 XXSORTWK04 DD DSN=&WORK4,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK4,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      12 XXSORTWK05 DD DSN=&WORK5,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK5,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      13 XXSYSABEND DD SYSOUT=&DMPCLAS                                         
         IEFC653I SUBSTITUTION JCL - SYSOUT=1,DEST=ABNORMAL                    
      14 XXSYSUDUMP DD SYSOUT=&DMPCLAS                                         
         IEFC653I SUBSTITUTION JCL - SYSOUT=1,DEST=ABNORMAL                    
      15 //SORTIN   DD DSN=FILEXYZ(0),                               
         //            DISP=SHR,DCB=BUFNO=192                                  
      16 //SORTOUT  DD DSN=FILE2XXY,                                 
         //            DISP=(NEW,PASS,DELETE),RETPD=365,VOL=(,,,99),           
         //            DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),         
         //            UNIT=TAPE                                               
      17 //SYSPRINT DD SYSOUT=*                                                
      18 //SYSIN    DD *     

Please suggest me some hints to over come this issue. Thank you.

NITISH SINGH
  • 55
  • 1
  • 9
  • I'll take a look later this morning. My initial thought is you'll need to adjust the BUFNO down ... As indicated, your system config will affect this parm. – Hogstrom Aug 14 '18 at 12:12
  • Ok thanks, even without BUFNO its abending with same message. Sort capacity exceeded. – NITISH SINGH Aug 14 '18 at 12:14
  • Can you show us the contents of SORTD – SaggingRufus Aug 14 '18 at 14:04
  • Are there other messages from sysout / sysprint? Normally there is a hint as to what resource is limited – Hogstrom Aug 14 '18 at 14:27
  • @Hogstrom Posted sysout messages in the question body. Thanks. – NITISH SINGH Aug 14 '18 at 17:13
  • Nitish, can you provide the JCL from the SORTD procedure ? You can get this from the job. What I'm specifically interested is the requested region size but all of it will be helpful in case there are other questions. Basically the same request as @s – Hogstrom Aug 16 '18 at 01:16
  • 1
    Sorry, was researching and ran out of my 5 minutes to edit the comment. Completing the above comment here ... Basically the same request as @SaggingRufus. You'll see it in the job output as lines that start as XX which simply means this JCL is expanded from a PROC. – Hogstrom Aug 16 '18 at 01:29
  • @Hogstrom posted code from SORTD from jcl. Thanks. – NITISH SINGH Aug 17 '18 at 11:20
  • This is helpful. Thanks. I don't have access to the Synscsort manuals (and I'm not asking as they are considered proprietary). Can you do the following and see if this helps. In line with the previous example, BUFNO depends on the environmental resources. Reduce BUFNO to BUFNO=10 (its a modest increase over the default of 5). For the SORTOUT DD BLKSIZE can you confirm what BLKSIZE is being assigned? Not knowing your drive type you might try a blksize of 180300 for the SORTOUT DD. It seems like you are very I/O bound and so you'll need to make the I/O more efficient. – Hogstrom Aug 17 '18 at 14:00
  • One other thought @NITISHSINGH is the file format. Your using FB with a fairly large LRECL. If you can manage it I think a VB file would be better assuming the XML documents are variable length. With fixed length your moving a huge amount of data that you don't need to unless the doc sizes are truly the same size. – Hogstrom Aug 18 '18 at 00:44
  • 1
    Have you tried a larger region size (or REGION=0)? Your SORTD proc has a CYLS parameter - you might try invoking it with a much higher value than the default of 1: use CYLS=1000 or so. Try making SORT allocate bigger work files too...you get 1000 cyls on the dynamically allocated ones - there's probably an option to raise this. Passing estimated record count is a good idea, as phunsoft suggests below. Switching to VB records is also a good idea - possibly even spanned (VBS). And BUFNO is probably ignored in favor of an internally computed value, so it might not help in this specific case. – Valerie R Aug 18 '18 at 15:28
  • What is the size of your input file? You really need this info to calculate an appropriate amount of work space. – Rich Jackson Aug 20 '18 at 05:52
  • _I/O File info_ : SORTIN : RECFM=FB ; LRECL= 30050; BLKSIZE= 30050 SORTIN : DSNAME=InputDataset.G00$$V00 OUTREC RECORD LENGTH = 30050 SORTOUT : RECFM=FB ; LRECL= 30050; BLKSIZE= 30050 SORTOUT : DSNAME=ouput Dataset **NoofRecords** : RCD IN 24362971, OUT 24362971 **File Size** FILESIZE 732,107,278,550 BYTES **Tracks Usage** TOTAL 30,471,555 TRACKS USED All the above info is for STEP-1 @RichJackson – NITISH SINGH Aug 20 '18 at 07:33
  • @NITISHSINGH The info in the job step only shows what was processed up to the point of the abend. What does ICF or your tape management system say about the size of the input file? Or how many records are in it (est: #ofRecs * LRECL)? – Rich Jackson Aug 20 '18 at 20:31

3 Answers3

4

I'll go out-on-a-limb here and assume that the files in question here are the same as previous question with same file attributes by same OP at: How can I reduce CPU in SORT operation

With 80,000,000 records at an LRECL of 30050, your input file is approximately 2.2TB.

A generally accepted rule-of-thumb for sort work space is 1.3x the input file size. So, you’ll need to target ~2.85TB (I.e ~3.7 million CYLS or ~55 million tracks) of sort work space.

With SyncSort’s maximum of 255 SORTWK DDs, you’ll need to acquire ~14,500 CYLs across each of the supported 255 DDs to accommodate your space requirements. This is a bit extreme, and will likely present a number of challenges, but there are a few options that could allow you to achieve this.

Disclaimer - I have not personally used SyncSort myself, but after perusing the docs, these concepts look to be consistent with other sort products that I do have experience with.

Rely on DYNALLOC

You have specified DYNALLOC=(SYSDA,255) and this is generally the recommended approach. However, there are a couple of issues with your current setup:

  • By using the SORTD PROC, you are robbing yourself of 5 DDs that could have been more appropriately utilized. The hardcoded SORTWK DDs in the PROC with SPACE=(CYL,(1)) limit the total work space you can acquire because DYNALLOC now only has 250 DDs to work with. As @SaggingRufus suggests, calling SYNCSORT directly would avoid the 1-CYL allocations of the PROC DDs and allow DYNALLOC to work with all 255 DDs.

  • SyncSort is not projecting the necessary work space accurately. Based on the output that you provided, SyncSort initially acquired 75 tracks (1 CYL for each of 5 hardcoded DDs in the SORTD proc… in other words, 5 DDs at 1 CYL * 15 Tracks-per-CYL = 75) explicitly, then dynamically acquired an additional 3,750,000 tracks (1,000 CYLs (or 15,000 tracks) * 250 DDs). It then acquired an additional 26.7 million tracks (1.78MM CYLs) across 2,230 secondary extents, which indicates an average of 799 CYLs (or 11,983 tracks) per secondary extent and an average of 8.75 secondary extents per volume/DD. All of this results in a total of ~1.6TB (~2MM Cyls) being allocated for sort work space, which is way below the estimated work space requirements for a 2.2TB input file. Even if we apply the DYNALLOC attributes to the hardcoded SORTWK DDs from the PROC, that only adds about 32GB to the total work space and is still woefully inadequate. So, as @phunsoft suggests, adding the SIZE=Ennnnnnnn parameter may be necessary for SyncSort to project work space appropriately.

  • Additional note - The fact that you're getting an average of <10 secondary extents per DD/volume indicates that you're exhausting the available free space on volumes in the work space pool for your environment. You should coordinate with your storage admins to ensure that enough disk space is available for your work space.

So, bypassing the PROC to provide access to all 255 DDs and providing an estimated record count could allow SyncSort to more reliably calculate the necessary work space, but this amount of data in a single sort is likely an outlier and default calculations may not be sufficient. Also, the available free space in your work space pool may be a limiting factor.

Hardcode SORTWK DDs

If you hardcode the SORTWK DDs in your JCL, you bypass the DDs in the PROC and eliminate some of the uncertainty of the DYNALLOC calculations. This is actually the approach that I, personally, would take first in this scenario… hardcode each SORTWK DD with the best possible options to acquire ~14,500 CYLS on each DD. So:

  • Hardcode each of 255 DDs (SORTWK01 - SORTWKFF) in job

  • Include SPACE=(CYL,(4000,1000)) on each DD (even though SyncSort will supposedly calculate secondary allocations itself)

  • Include DSNTYPE=LARGE on each DD to allow greater than 4G per volume (each SORTWK is restricted to 1 volume) The aggregate of these DDs should provide sufficient work space for your sort requirements.

Each of the above options assume that sufficient vols/extents/overall-space are available in the work space pool for your shop. I’d recommend coordinating with your storage management team to ensure availability of resources

Use MAXSORT

SyncSort provides the MAXSORT capability to accommodate extremely large data sets. It sorts sections of an input file into intermediate files, and re-uses SORTWK space for subsequent sections, until the intermediate files for each section can be combined into a single output file. There are particular requirements for using MAXSORT. As referenced earlier, I don’t have access to SyncSort, so I can’t test this capability myself. You'll have to research the SyncSort doc yourself for the details of this capability.

Rich Jackson
  • 301
  • 1
  • 5
0

Since your sort input is read from tape, sort cannot reliably calculate the sort work space needed. Try adding the SIZE=Ennnnnnn parameter to the SORT statement. nnnnnnn is the number of records to be read. E tells sort this is an estimate.

phunsoft
  • 2,674
  • 1
  • 11
  • 22
  • The `SORTIN` is not necessarily read from tape ... `SORTOUT` is written to tape per the original post. – Hogstrom Aug 15 '18 at 01:54
  • @Hogstrom: You're right. It doesn't say here, but I silently understood this to be a continuaion of "How can I reduce CPU in sort operation" the OP started on July 28. Should have mentioned this; sorry I did not. – phunsoft Aug 15 '18 at 04:20
0

try calling SYNCSORT directly instead of calling this proc. This PROC only has 5 workfiles set up. It is usually best to let SYNCSORT allocate as may workfiles as needed rather than telling it how many it can use.

I believe it is the workfiles that have run out of space. Try this:

//SORT10 EXEC PGM=SYNCSORT,                                   
//             PARM=('INCORE=OFF,DYNALLOC=(SYSDA,255)',EQUALS)
//SORTIN   DD DSN=FILEXYZ(0),                          
//            DISP=SHR,DCB=BUFNO=192                             
//SORTOUT  DD DSN=FILE2XXY,                            
//            DISP=(NEW,CATLG,DELETE),RETPD=365,VOL=(,,,99),     
//            DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),    
//            UNIT=(TAPE,2)
//SYSOUT   DD  SYSOUT=*
//SYSPRINT DD  SYSOUT=*
//SORTMSG  DD  SYSOUT=*
//SYSIN    DD *                                                  
       SORT FIELDS=(19,5,PD,A,8,6,PD,A,50,2,ZD,A)                  
       OUTREC IFTHEN=(WHEN=(70,18,CH,EQ,C' encoding="IBM037"'),    
                     OVERLAY=(70:C'  encoding="UTF-8"'))             
       OPTION DYNALLOC=(SYSDA,255)                                 
/*
SaggingRufus
  • 1,814
  • 16
  • 32