Patching Dspace for DS-2542 (XOAI date granularity) results in no change of behaviour

Question

I am trying to apply a patch to our Test environment running Dspace 5.1 (with some additional modules) to fix a OAI harvesting issue due to improper date granularity. The issue is DS-2542 XOAI does not support non granular YYYY-MM-DD harvesting properly

Environment

• Dspace 5.1 (inherited, and subsequently upgraded from 4.2)
• Tomcat version 6
• Oracle Java 1.7 update 79
• Red Hat Enterprise Linux version 6.6

How I applied the patch

First I need to grab the patch for Dspace (dspace-oai), from https://github.com/DSpace/DSpace/pull/912. That gives me the following patch (as a diff), which I put in /tmp/

diff --git a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
index 1995fc0..cdb17d9 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
@@ -50,6 +50,11 @@ public boolean isShown(DSpaceItem item)
     public SolrFilterResult buildSolrQuery()
     {
         String format = dateProvider.format(date).replace("Z", ".999Z"); // Tweak to set the millisecon
+        if (format.substring(11, 19).equals("00:00:00"))
+        {
+            format = format.substring(0, 11) + "23:59:59" + format.substring(19);
+        }
+
         return new SolrFilterResult("item.lastmodified:[* TO "
                 + ClientUtils.escapeQueryChars(format) + "]");
     }
diff --git a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
index e968414..b73955d 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
@@ -31,8 +31,7 @@ public static String format(Date date)
     }
     public static String format(Date date, boolean init)
     {
-       SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'000Z'");
-       if (!init) sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'999Z'");
+        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
         // We indicate that the returned date is in Zulu time (UTC) so we have
         // to set the time zone of sdf correct.
         sdf.setTimeZone(TimeZone.getTimeZone("ZULU"));

I have applied the patch using the following sequence of commands:

sudo su - builder
  cd /usr/local/src/
    git clone 'https://github.com/lyncode/xoai.git' xoai
    cd xoai/
      git branch -a
      git checkout 3.x
      less pom.xml    <-- verify that com.lyncode version is 3.2.10-SNAPSHOT
      mvn package install
      cd ..
    cd dspace-src/
      patch -p1 < /tmp/473f2faaba99671b55372bcca1604aea2acf9601.diff    # (needed tweaking)
      vim dspace-oai/pom.xml   # change <xoai.version> from 3.2.9 to 3.2.10-SNAPSHOT
      mvn package -U -P \!dspace-jspui,\!dspace-lni,\!dspace-rest,\!dspace-swordv2,\!dspace-rdf,\!dspace-xmlui-mirage2
      exit
cd /usr/local/src/dspace-src/dspace/target/dspace-installer/
  ant update
  chown -Rc tomcat:tomcat /usr/local/dspace/
  service tomcat6 restart

Testing methodology

By post-patching results are the same as my pre-patching. Using 2015-04-07 still fails, but should pass after patching. 2015-04-07T00:00:00Z works the same and passes.

$ curl 'https://test-dspace.example.com/oai/request?verb=ListRecords&metadataPrefix=oai_dc&until=2015-04-07'
...<error code="badArgument">Invalid date given in until parameter</error></OAI-PMH>

$ curl 'https://test-dspace.example.com/oai/request?verb=ListRecords&metadataPrefix=oai_dc&until=2015-04-07T00:00:00Z'
... completeListSize="3731" cursor="0">oai_dc//2015-04-07T00:00:00.000Z//100</resumptionToken></ListRecords></OAI-PMH>

I have seen, and read multiple times, a DSpace - Tech email entitled 'Building DSpace after bug fix (DS-2542)', and although I have done everything as I should have according to that email trail (including both the "proper" -- edit your POM -- and "quick and dirty" -- replace the jar -- methods listed therein), the problem remains.

Things I've tried (all to no avail)

• Ensured tomcat was properly stopped, then started.
• Repeated the full build process (mvn -U clean package)
• Ensured there were no xoai-*.jar around other than xoai-3.2.10-SNAPSHOT.jar in /usr/local/dspace/lib/ and /usr/local/dspace/webapps/oai/WEB-INF/lib/
• Enable DEBUG logging for com.lyncode.xoai and org.dspace.xoai
• Replaced the jar -- thinking that perhaps the build process (mvn or ant) of dspace was replacing it with some other.
• Removed ~/.m2/repository and build again from scratch (including all modules).
• added -verbose:class to Tomcat startup parameters to see where things are loaded (see additional appendix)

What I'm currently thinking / questioning

I'm confident that the jar is being created as expected, and is well installed, and that there is no old jar in play.

I'm not confident that the code I think should be run is actually getting run, or if something else is overshadowing the code... and I don't know how to diagnose that in Tomcat.

Is there a command (presumably dspace dsrun ...) I could use to replicate this test outside of Tomcat?

I should point out that I'm a Systems Engineer, not a Java Developer.

Appendices

log settings

Added at the end of the appender section for dspace.log

log4j.logger.com.lyncode.xoai=DEBUG, A1
log4j.logger.org.dspace.xoai=DEBUG, A1

This let me see logs such as:

2015-07-28 02:34:07,222 DEBUG org.dspace.xoai.services.impl.solr.DSpaceSolrServerResolver @ Solr Server Initialized

2015-07-28 02:34:07,233 DEBUG com.lyncode.xoai.dataprovider.OAIRequestParameters @ RootParameterMap 'until' = '2015-04-07'

and if I edit /usr/local/src/xoai/src/main/java/com/lyncode/xoai/dataprovider/OAIDataProvider.java, I can turn that into

2015-07-28 02:50:15,374 DEBUG com.lyncode.xoai.dataprovider.OAIRequestParameters @ CAMERON-RootParameterMap 'until' = '2015-04-07'

but adding CAMERON to be beginning of the seemingly only place where the error text is generated does not show a change in the test output (this in OAIDataProvider.java from that same directory):

throw new BadArgumentException("CAMERON Invalid date given in until parameter");

git diff of dspace-src directory (relative to pre-patching)

# git diff
diff --git a/build.properties b/build.properties
index 844cb93..09cb5c3 100644
--- a/build.properties
+++ b/build.properties
@@ -166,6 +166,7 @@ http.proxy.port = 3128
 loglevel.other = WARN
 # loglevel.other: Log level for other third-party tools/APIs used by DSpace
 # Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
-loglevel.dspace = INFO
+#loglevel.dspace = INFO
+loglevel.dspace = DEBUG
 # loglevel.dspace: Log level for all DSpace-specific code (org.dspace.*)
 # Possible values (from most to least info): DEBUG, INFO, WARN, ERROR, FATAL
diff --git a/dspace-oai/pom.xml b/dspace-oai/pom.xml
index 6e56a52..136a10f 100644
--- a/dspace-oai/pom.xml
+++ b/dspace-oai/pom.xml
@@ -16,7 +16,7 @@
         <!-- This is the path to the root [dspace-src] directory. -->
         <root.basedir>${basedir}/..</root.basedir>
         <spring.version>3.2.5.RELEASE</spring.version>
-        <xoai.version>3.2.9</xoai.version>
+        <xoai.version>3.2.10-SNAPSHOT</xoai.version>
         <jtwig.version>2.0.1</jtwig.version>
     </properties>

diff --git a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
index 1995fc0..cdb17d9 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/filter/DateUntilFilter.java
@@ -50,6 +50,11 @@ public class DateUntilFilter extends DSpaceFilter
     public SolrFilterResult buildSolrQuery()
     {
         String format = dateProvider.format(date).replace("Z", ".999Z"); // Tweak to set the millisecon
+        if (format.substring(11, 19).equals("00:00:00"))
+        {
+            format = format.substring(0, 11) + "23:59:59" + format.substring(19);
+        }
+
         return new SolrFilterResult("item.lastmodified:[* TO "
                 + ClientUtils.escapeQueryChars(format) + "]");
     }
diff --git a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
index e968414..b73955d 100644
--- a/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
+++ b/dspace-oai/src/main/java/org/dspace/xoai/util/DateUtils.java
@@ -31,8 +31,7 @@ public class DateUtils
     }
     public static String format(Date date, boolean init)
     {
-       SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'000Z'");
-       if (!init) sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.'999Z'");
+        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
         // We indicate that the returned date is in Zulu time (UTC) so we have
         // to set the time zone of sdf correct.
         sdf.setTimeZone(TimeZone.getTimeZone("ZULU"));

Git log from /usr/local/src/xoai (github.com/lyncode/xoai checkout of branch 3.x)

commit f6721be7ef5bf75d790e220ee81821e8eb70986e
Merge: dd9ef83 71ae14e
Author: João Melo <jmelo@lyncode.com>
Date:   Thu Oct 9 18:33:57 2014 +0100

    Merge pull request #34 from kosarko/dates_fix

    from and until granularity

commit dd9ef830a16209e624bd03c3e91e7c5d7bdbd449
Merge: 8e778c2 c04b0bc
Author: João Melo <jmelo@lyncode.com>
Date:   Thu Oct 9 18:33:29 2014 +0100

    Merge pull request #33 from kosarko/filter_fix

    fixed typo

commit 71ae14ed5ee9e6a5a6e3d6253424c212ac98081a
Author: Ondřej Košarko <kosarko@ufal.mff.cuni.cz>
Date:   Thu Oct 9 19:05:03 2014 +0200

    from and until granularity

    Regardles of the repository setting the from and until parameters must
    handle day granularity. Error message fixed

commit c04b0bcd177007f618f18008e53a61a81aabdb06
Author: Ondřej Košarko <kosarko@ufal.mff.cuni.cz>
Date:   Thu Oct 9 19:01:08 2014 +0200

    fixed typo

Instances of xoai-*.jar

$ sudo updatedb
$ locate xoai- | grep '\.jar$' | grep -v bak | xargs ls -l
-rwxr-xr-x 1 tomcat  tomcat   321775 Jul 28 03:04 /usr/local/dspace/lib/xoai-3.2.10-SNAPSHOT.jar
-rwxr-xr-x 1 tomcat  tomcat   321775 Jul 28 03:04 /usr/local/dspace/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder  321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace/modules/oai/target/oai-5.1/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder  321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace-oai/target/dspace-oai-5.1/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder  321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace/target/dspace-installer/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder  321832 Jul 28 01:34 /usr/local/src/dspace-src/dspace/target/dspace-installer/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder  321775 Jul 28 03:04 /usr/local/src/xoai/target/xoai-3.2.10-SNAPSHOT.jar
-rw-rw-r-- 1 builder builder 1571092 Jul 28 03:04 /usr/local/src/xoai/target/xoai-3.2.10-SNAPSHOT-javadoc.jar
-rw-rw-r-- 1 builder builder  222305 Jul 28 03:04 /usr/local/src/xoai/target/xoai-3.2.10-SNAPSHOT-sources.jar

Verifying that tomcat should see the same (in case of broken links, of which there appear to be none)

$ find -L /usr/share/tomcat6 -type f -name 'xoai-*.jar' -print | xargs ls -l
-rwxr-xr-x 1 tomcat tomcat 321775 Jul 28 03:04 /usr/share/tomcat6/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar

Tomcat -verbose:class findings

All com.lyncode.xoai. classes come from file:/usr/local/dspace/webapps/oai/WEB-INF/lib/xoai-3.2.10-SNAPSHOT.jar

Many many thanks for reading,

Cameron

Answer 1

Have you tried clearing the oai response cache? I don't know whether this type of thing gets cached, but it's the only thing I can think of that you haven't mentioned yet.

[dspace]/bin/dspace oai clean-cache

https://wiki.duraspace.org/display/DSDOC5x/OAI+2.0+Server#OAI2.0Server-OAIManager%28SolrDataSource%29