3

I am working on getting lucene indexing working on Google App Engine. I am using a ramdirectory to make the index and then serializing it (the ramdirectory object) to memcache and blobstore for persistent storage. http://code.google.com/appengine/docs/java/blobstore/overview.html#Writing_Files_to_the_Blobstore For search I just deserialize it and use in my searches.

I am facing a null pointer exception when I close the indexwriter.

I think that might have something to do with the fact that only the following libraries are supported in google app engine. http://code.google.com/appengine/docs/java/jrewhitelist.html

I am using lucene 3.5.0 and app engine java version 1.6.1

The following is the stack trace which i get 

java.lang.NullPointerException
at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:103)
at org.apache.lucene.store.DataOutput.writeStringStringMap(DataOutput.java:189)
at org.apache.lucene.index.SegmentInfo.write(SegmentInfo.java:623)
at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:394)
at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:872)
at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4601)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3453)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3524)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1879)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1822)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1786)

The code works properly on my local machine (I haven't added much of a code , just added some sample documents and did a indexwriter.close())

Has someone faced this problem before ?? and if so is there a workaround for it ??

The code where I am finding the problem is simple 

RAMDirectory dir = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer(Version.LUCENE_35));
IndexWriter writer = new IndexWriter(dir,config); 
Document doc;
doc = new Document();
doc.add(new Field("text","mary had a little lamb", Store.YES, Index.ANALYZED));
writer.addDocument(doc)
writer.close();

the exception is thrown when i am trying to close the writer in the last line

sdey
  • 33
  • 3
  • when exactly does it fail? can you explain how to reproduce this behavior (for example, you index some stuff, run a search, try to close the index)? – milan Jan 23 '12 at 00:52
  • @milan gaelucene is read only, i am trying to build the index on app engine here. I can create an index somewhere else , serialize it and move it to app engine for searching later on. What I am having difficulty is indexing on the appengine. – sdey Jan 23 '12 at 00:52
  • @milan I have added the code excerpt for you to reproduce. – sdey Jan 23 '12 at 00:55
  • I have exactly the same issue when doing commit on RAMDirectory in Lucene. Locally everything works great - but on GAE it doesn't. Were you able to resolve the problem? – jdevelop Feb 09 '12 at 11:49
  • @jdevelop nopes sorry, i haven't been able to figure this out. I think everything else needed for a fully functional small scale lucene indexing and search system is working, your indexes can be made in ramdirectory, stored as persistent objects either in memcache or in blobstore and read right back for searching later. But the part about commit/close has been a problem for a long time, I am trying to debug the same using lucene source , lets see if that works – sdey Feb 10 '12 at 12:48
  • @sdey okay, I saw that this question was asked in Lucene mailing list, did anybody respond till now? – jdevelop Feb 12 '12 at 11:27

1 Answers1

3

the problem is that for some reason Lucene tries to store os.version and os.arch in index.

I don't know why, however the solution is adding the properties to your appengine-web.xml:

<system-properties>
    <property name="os.version" value="1.0.GAE whatever" />
    <property name="os.arch" value="GAE whatever" />
</system-properties>

and it will work for you. Hope that helps :)

jdevelop
  • 12,176
  • 10
  • 56
  • 112
  • thanks a lot, this solution works :) This is going to help me a lot in deploying my search architecture to google app engine . Thanks again – sdey Feb 19 '12 at 19:47