0

I have a JCL that is sorting millions of records (record length of 28,704). We noticed that we were running out of workfiles (the default of 32). we recently added the parameter to override the default value of 32 work files to 255.

We found a couple of tips of the syncsort website that told us how many cylinders each workfile would use and doing out the math, it looked like we would need at least 200 workfiles. Another recommendation said to have 130% space available for maximum performance. This lead to us specifying the max amount of workfiles (255). We are aware of the extra overhead involved by adding the extra file.

To the point AKA the question

will using these parameters in our sort:

//SORT10 EXEC PGM=SYNCSORT,COND=(8,LT),                        
//            PARM=('INCORE=OFF,DYNALLOC=(SYSDA,255)',EQUALS)

cause it to always allocate 255 workfiles, or will it allocate the minimum that is required up to a max of 255.

SaggingRufus
  • 1,814
  • 16
  • 32
  • 1
    Syncsort manual? – Steve Ives Apr 20 '17 at 18:18
  • @SteveIves that's just crazy enough to work! Unfortunately, I don't have access to one. However based on this comment and Bill's answer, I will contact the guys at our site that deal directly with SyncSORT. I figured that this would be an easier that everyone knew except for me! Also, our people who deal with the software vendors will not contact them unless we can show we did how research and tried other means. – SaggingRufus Apr 21 '17 at 10:09
  • Which doesn't make sense to me because I assume we are paying for the support that the vendor provides. Oh well, I don't get paid to question these things. – SaggingRufus Apr 21 '17 at 10:10
  • There's just too many products with too many options for everyone to know all of them. Looking things up in the manual is preferable as it (usually) gives you a quicker answer than going to the vendor, but the vendor's support people are absolutely there to support you and answer any usage questions you might have and should not turn down a request just because the answer is in the manual, although it is assumed you looked first. – Steve Ives Apr 21 '17 at 10:43
  • I'll have to look into getting soft copies of some manuals, because I would love to be able to look up the answers to my question without coming here and asking questions that I should be able to answer on my own. – SaggingRufus Apr 21 '17 at 10:58
  • Do you have a good reason for INCORE=OFF in your parameters? I'd think that would slow things down quite a bit, as well as requiring much more I/O to the work files. – Valerie R May 25 '17 at 15:43
  • @ValerieR I will be honest, I don't even know what that parameter does. I inherited the JCL with the parameter already there – SaggingRufus May 25 '17 at 16:02
  • Most modern sorts try to do as much as possible with their data in memory - I believe INCORE=OFF tells Syncsort to do less in memory and more on the sort work files. In days gone by, many systems were memory constrained, and big sorts in memory would actually be slower than sorts using work files. In a memory-constrained situation, paging I/O is probably slower than Syncsort's I/O. If your "INCORE=OFF" is just a relic of some old situation, you might try removing it and playing with different region sizes to see what works best on your current system. – Valerie R May 25 '17 at 21:09
  • @ValerieR that actually makes a lot of sense.... I thought those sorts were super slow. Paging would explain that – SaggingRufus May 26 '17 at 21:30

1 Answers1

2

The Manual will be your friend (if you don't have one, you should be able to obtain one). It would also be useful to know the options that SyncSORT at your site has been installed with.

There is probably some "overhead" in establishing 255 as a default. However, even with 255, only work data sets that are required will actually be opened and used in any given step.

It would be common for a fairly small number (say six) to be established as the default, with overrides for steps which are known to require more workspace.

The best suggestion I can make is to make two queries to SyncSORT support: how to establish a reasonable number for DYNALLOC for 99.999% of our SORTs; what to specify for work space for xyz step which has this many records of this size.

If you provide to them the information they want, they will provide specific information that relates to how your site is using SyncSORT (they may even make other suggestions) and to your data.

You are paying for support. So use it. Any time you have something exceptional or which isn't behaving in reasonable way. Contact SyncSORT.

It is probably not possible for you to individually contact SyncSORT without permission from the person who has that as part of their role. So you have to find that person. If you do get nowhere with with, their website is obvious, and support email address available - but they will need a license number, and the person at your site who didn't want to make the contact, may find out about it :-)

Expect substantial resource-savings and increased robustness after considering their advice and applying what you are able to.

Anyone who is at a site which is licensed to use SyncSORT can obtain a free copy of the documentation. It will require the supply of some information. There are Terms and Conditions to its use (once you have your own copy, you are not allowed to distribute it, for instance). You want the Programmer's Guide. There are some further publications you can request at the same time: Installation Guide; Exploiting MFX: SortWriter Data Utilities Guide; Exploiting MFX: MAXSORT; Exploiting MFX: JOIN.

One thing they don't document is SyncTOOL. If you contact SyncSORT support about this, you will be referred to IBM's DFSORT documentation for ICETOOL, which is in the DFSORT Application Programming Guide.

There are over 40 options which can be set to establish how SyncSORT operates (and your Sysprogs will be dealing with tens or hundreds of IBM and ISV packages, each with their own installation/customisation and with multiple methods of applying changes). Many of the operations relating to the technicalities of operation interact with each other. It won't necessarily be clear to someone who is not extremely experienced with the innards of SyncSORT exactly what is best for your site, with the hardware, software and policies your site is running, for a particular unusual SORT.

What is an "unusual SORT"? Any SORT that isn't usual. SORTing millions of 28000-byte records is unusual. It's not something you want to take six time to get it to run, and then find it is slow (which means expensive).

Although I have a certain amount of experience, I would not hesitate to contact SyncSORT over your unusual SORT. They are far more likely to get it right than I am, in far less time.

I would hope that the installation options were chosen with support from SyncSORT. A small number of DYNALLOC files is normal and will be sufficient for most SORT steps. If something changes (new hardware, software, taking on a mass of new data, etc), and you suddenly have many steps failing, SyncSORT support is a very good choice, even after upping the DYNALLOC and all being fine. It may be fine, but 3% less efficient, which is 3% more expensive :-)

Although support contracts can have elements of individuality, I've never heard of one from SyncSORT which pays for support calls. Maybe I just don't get out enough.

One point about support calls is the amount of information you may have to provide. That can be "fun", especially for suspected bugs. But it will pay off. That is perhaps more likely the reason your technical staff are wary.

SyncSORT support will take very minor queries: trivial question directed by SyncSORT Technical Architect to SyncSORT Support.

Bill Woodger
  • 12,968
  • 4
  • 38
  • 47