The OOTB multi-select actions for the Document Library can be found in the share-documentlibrary-config.xml file.
<multi-select>
<action type="action-link" id="onActionDownload" icon="document-download" label="menu.selected-items.download" />
<action type="action-link" id="onActionCopyTo" icon="document-copy-to" label="menu.selected-items.copy" />
<action type="action-link" id="onActionMoveTo" icon="document-move-to" permission="Delete" label="menu.selected-items.move"/>
<action type="action-link" id="onActionAssignWorkflow" asset="document" icon="document-assign-workflow" label="menu.selected-items.assign-workflow"/>
<action type="action-link" id="onActionDelete" icon="document-delete" permission="Delete" label="menu.selected-items.delete" notAspect="hwf:hybridWorkflow,sys:undeletable" />
<action type="action-link" id="onActionCloudSync" asset="document" icon="document-cloud-sync" permission="CreateChildren" label="menu.selected-items.cloudSync" notAspect="sync:syncSetMemberNode" syncMode="ON_PREMISE" />
<action type="action-link" id="onActionCloudSyncRequest" icon="document-request-sync" label="menu.selected-items.cloudRequestSync" hasAspect="sync:syncSetMemberNode" syncMode="ON_PREMISE" />
</multi-select>
If you want additional multi-select actions to be displayed then you will want to augment this configuration attribute. Rather than editing this file directly you'll want to provide an extension to this particular configuration section (you can find some documentation of this here)
It would be worth reviewing how the existing multi-select actions are implemented to ensure that your custom action is compatible as both a single item action and a multi-select action