0

MarkLogic version - 9.0-6.2 DHF Version : 2.0.6 (id 'com.marklogic.ml-data-hub' version '2.0.6')

Trying to deploy changes through 'gradle mlReloadModules' command but noticed that the REST extension billing.sjs is not being deployed. This file (billing.sjs) is saved at \data-hub\EODH\plugins\entities\billing\harmonize\REST\services and I executed 'gradle mlReloadmodules' command several times from \data-hub\EODH directory

I went to data-hub-MODULES db and browsed /marklogic.rest.resource/billing/assets/resource.sjs to see that the old code is still present there.

I also did 'rm .tmp/user-modules-deploy-timestamps.properties' before reloading modules.

Below is a selected section of output from 'gradle mlReloadmodules -d' command

23:17:34.455 [DEBUG] [com.marklogic.com.marklogic.client.ext.modulesloader.impl.EntityDefModulesFinder] Finding modules in baseDir: G:\data-hub\EODH\plugins\entities\billing 23:17:34.455 [DEBUG] [com.marklogic.com.marklogic.client.ext.modulesloader.impl.EntityDefModulesFinder] Finding modules in baseDir, which was modified to be: file:/G:/data-hub/EODH/plugins/entities/billing/ 23:17:34.455 [DEBUG] [com.marklogic.com.marklogic.client.ext.modulesloader.impl.EntityDefModulesFinder] Finding Entity Def at path: file:/G:/data-hub/EODH/plugins/entities/billing/.entity.json 23:17:34.455 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing] 23:17:34.456 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Searching directory [G:\data-hub\EODH\plugins\entities\billing] for files matching pattern [G:/data-hub/EODH/plugins/entities/billing/.entity.json] 23:17:34.456 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/.entity.json] to resources [] 23:17:34.457 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] Loading modules from base directory: G:\data-hub\EODH\plugins\entities\billing\harmonize\REST 23:17:34.470 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.PropertiesModuleManager] Loading properties from: G:\data-hub\EODH.tmp\user-modules-deploy-timestamps.properties 23:17:34.470 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding modules in baseDir: G:\data-hub\EODH\plugins\entities\billing\harmonize\REST 23:17:34.470 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding modules in baseDir, which was modified to be: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/ 23:17:34.470 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding namespaces at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/namespaces/.* 23:17:34.471 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\namespaces] 23:17:34.471 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Skipping [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\namespaces] because it does not exist 23:17:34.471 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/namespaces/.] to resources [] 23:17:34.471 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding options modules at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/options/. 23:17:34.471 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\options] 23:17:34.471 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Searching directory [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\options] for files matching pattern [G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/options/.] 23:17:34.472 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/options/.] to resources [file [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\options\BillingInfoList.json]] 23:17:34.472 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding REST properties file at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/rest-properties.json 23:17:34.472 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding service modules at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/.xq 23:17:34.472 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services] 23:17:34.472 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Searching directory [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services] for files matching pattern [G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/.xq] 23:17:34.473 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/.xq] to resources [] 23:17:34.473 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding service modules at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/.sjs 23:17:34.473 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services] 23:17:34.473 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Searching directory [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services] for files matching pattern [G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/.sjs] 23:17:34.474 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/.sjs] to resources [file [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services\billing.sjs], file [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services\billing360.sjs], file [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services\billingReg.sjs]] 23:17:34.474 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding transform modules at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/transforms/.xq* 23:17:34.474 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\transforms] 23:17:34.474 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Skipping [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\transforms] because it does not exist 23:17:34.474 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/transforms/.xq] to resources [] 23:17:34.474 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding transform modules at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/transforms/.xsl 23:17:34.474 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\transforms] 23:17:34.475 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Skipping [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\transforms] because it does not exist 23:17:34.475 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/transforms/.xsl] to resources [] 23:17:34.475 [DEBUG] [com.marklogic.appdeployer.command.modules.AllButAssetsModulesFinder] Finding transform modules at path: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/transforms/.sjs 23:17:34.475 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Looking for matching resources in directory tree [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\transforms] 23:17:34.475 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Skipping [G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\transforms] because it does not exist 23:17:34.475 [DEBUG] [org.springframework.core.io.support.PathMatchingResourcePatternResolver] Resolved location pattern [file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/transforms/.sjs] to resources [] 23:17:34.475 [INFO] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] Loading BillingInfoList query options from file BillingInfoList.json 23:17:34.489 [DEBUG] [com.marklogic.client.impl.OkHttpServices] Putting /config/query/BillingInfoList 23:17:34.529 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.DefaultExtensionMetadataProvider] Unable to build metadata from resource file: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/billing.sjs; cause: G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services\metadata\billing.xml (The system cannot find the file specified) 23:17:34.530 [INFO] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] Loading billing resource extension from file billing.sjs 23:17:34.540 [INFO] [com.marklogic.client.impl.ResourceExtensionsImpl] Writing resource services source for billing 23:17:34.540 [DEBUG] [com.marklogic.client.impl.OkHttpServices] Putting config/resources/billing 23:17:34.552 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.DefaultExtensionMetadataProvider] Unable to build metadata from resource file: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/billing360.sjs; cause: G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services\metadata\billing360.xml (The system cannot find the file specified) 23:17:34.553 [INFO] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] Loading billing360 resource extension from file billing360.sjs 23:17:34.560 [INFO] [com.marklogic.client.impl.ResourceExtensionsImpl] Writing resource services source for billing360 23:17:34.560 [DEBUG] [com.marklogic.client.impl.OkHttpServices] Putting config/resources/billing360 23:17:34.576 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.DefaultExtensionMetadataProvider] Unable to build metadata from resource file: file:/G:/data-hub/EODH/plugins/entities/billing/harmonize/REST/services/billingReg.sjs; cause: G:\data-hub\EODH\plugins\entities\billing\harmonize\REST\services\metadata\billingReg.xml (The system cannot find the file specified) 23:17:34.577 [INFO] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] Loading billingReg resource extension from file billingReg.sjs 23:17:34.583 [INFO] [com.marklogic.client.impl.ResourceExtensionsImpl] Writing resource services source for billingReg 23:17:34.583 [DEBUG] [com.marklogic.client.impl.OkHttpServices] Putting config/resources/billingReg 23:17:34.609 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] shutdownTaskExecutorAfterLoadingModules is set to false, so not shutting down taskExecutor 23:17:34.609 [DEBUG] [com.marklogic.client.ext.modulesloader.impl.DefaultModulesLoader] Finished loading modules from base directory: G:\data-hub\EODH\plugins\entities\billing\harmonize\REST

Mike Gardner
  • 6,611
  • 5
  • 24
  • 34
Bhanu
  • 427
  • 2
  • 8
  • I think DHF v2 is looking at a different place for rest extensions. Did you try putting it under src/main/ml-modules/services/? – grtjn Mar 19 '19 at 09:21
  • Thanks a lot. We have been deploying the same way all the time. We have not upgraded DHF in months. It worked just until yesterday. However I will try placing REST extension in the suggested folder. – Bhanu Mar 19 '19 at 10:56
  • It is not working from src/main/ml-modules/services/ either. Also, we are able to push the code successfully from a different machine (by keeping the code at plugins\entities\billing\harmonize\REST\services. Looks like my gradle cache is messed up and not deploying the code (even after 'rm .tmp/user-modules-deploy-timestamps.properties'). Appreciate any suggestions. – Bhanu Mar 19 '19 at 14:22

2 Answers2

1

Gradle task names are case insensitive. Property names are case sensitive. The case sensitivity of file paths - which may be the issue in that ticket - is determined by the OS.

One thing would be to verify that none of the directories in your deployment path are set to case sensitive. Linux Tools run by Windows (Windows Subsystem for Linux - Bash shell) will create folders that are case sensitive. This default behavior can be changed by editing the wsl.conf

You can check for Case Sensitivity by using the following command:

fsutil.exe file queryCaseSensitiveInfo C:\folder

Case sensitivity is set per directory, and does not affect sub-directories, so each directory in the file path may need to be checked.

It's also possible you may be encountering a known bug in the 4.0.x code DHF does not deploy REST extensions. It has since been fixed in 4.1.0 and later.

If this is the issue you are facing, would suggest moving to the latest version of Data Hub Framework, as there were a lot of bug fixes and enhancements to the deployment process.

If that is not an option at this time, a workaround is detailed in the issue comments:

https://github.com/marklogic/marklogic-data-hub/issues/1574#issuecomment-441785549

Mike Gardner
  • 6,611
  • 5
  • 24
  • 34
  • Thanks Michael. We are using version 2.0.3. I noticed that the lower case 'mlreloadmodules' is not working but the mixed case 'mlReloadModules' is working fine. Our MarkLogic is deployed on windows (community edition) and I am running from powershell, so we did not see case sensitivity issues earlier, so this is a surprise behavior for us. I will keep an eye on the deploys and update if I find anything new. – Bhanu Mar 20 '19 at 15:17
0

We are using version 2.0.3. I noticed that the lower case 'mlreloadmodules' is not working but the mixed case 'mlReloadModules' is working fine. Our MarkLogic is deployed on windows (community edition) and I am running from powershell, so we did not see case sensitivity issues earlier, so this is a surprise behavior for us. I will keep an eye on the deploys and update if I find anything new.

Thanks everyone, for your responses.

Bhanu
  • 427
  • 2
  • 8