Netbean not responding when using Saxp read and hibernate save a lage database

Question

My project parse xml file from dblp , it about 1Gb to save in ORM database , so i use SAXP for reading imformation like (paper'author, paper'infomation ......) and use Hibeanate to save data in to database :

Step 1: Use SAXP read a element like this:

<article key="journals/cs/BhaskarAS11" mdate="2011-11-09"><author>Data Ram Bhaskar</author><author>Kasim Karam Abdalla</author><author>Raj Senani</author><title>Electronically-Controlled Current-mode second order Sinusoidal Oscillators Using MO-OTAs and Grounded Capacitors.</title><pages>65-73</pages><year>2011</year><volume>2</volume><journal>Circuits and Systems</journal><number>2</number><ee>http://dx.doi.org/10.4236/cs.2011.22011</ee><url>db/journals/cs/cs2.html#BhaskarAS11</url></article>

To get : Author'name , publicaiton, publisher, ... of publication

Step 2: Check deplicate Author, publisher, ... in database . Step 3: Save publication in database End: go to next element until end database.

When i run file dblp about 10MB it run ok (11 minute) - it input about 21000 publicaiton in to database. But when run full dblp file (1GB) , when a saw in database it about 75000 pub (about 1 day) the Netbean throw not respond - i can't get any error or message from netBean - and it ca't input any pub in to database (but in dblp have about 1,7 milion pub).

Sorry for the long question, but I'd very much appreciate any help any of you can give me! the problem SAXP ? or Hibernate ....?

Here some code :

Hibernate config:

<hibernate-configuration>
  <session-factory>
    <property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
    <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
    <property name="hibernate.connection.url">jdbc:mysql://localhost:3306/cspublicationcrawler</property>
    <property name="hibernate.connection.username">root</property>
    <property name="hibernate.connection.password">root</property>
    <property name="hibernate.transaction.auto_close_session">false</property>
    <property name="hibernate.transaction.flush_before_completion">true</property>

    <property name="hibernate.connection.pool_size">50</property>
    <property name ="hibernate.jdbc.batch_size">500</property> 
    <property name="hibernate.current_session_context_class">org.hibernate.context.ThreadLocalSessionContext</property>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Magazine.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Reviewer.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Conference.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Author.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Paper.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/RankAuthorKeyword.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Publisher.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Comment.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Domain.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/Journal.hbm.xml"/>
    <mapping resource="uit/tkorg/cspublicationtool/entities/PaperType.hbm.xml"/>
  </session-factory>

Hibernate Util

public HibernateUtil() throws Exception {
    sessionFactoryConfigPath = "";
    sessionFactory = new Configuration().configure().buildSessionFactory();
}

public HibernateUtil(String sessionFactoryConfigPath) {
    this.sessionFactoryConfigPath = sessionFactoryConfigPath;
    sessionFactory = new Configuration().configure(sessionFactoryConfigPath).buildSessionFactory();
}

/**
 * Begin a transaction
 */
protected void beginTransaction() {
    session = sessionFactory.getCurrentSession();
    session.beginTransaction();
}

/**
 * Commit transaction and close session
 */
protected void commitAndClose() {
    if (session != null) {
        for(int i=0;i<10000;i++) {
            if ( i % 500 == 0 ) { //50, same as the JDBC batch size
                                 //flush a batch of inserts and release memory:
                    session.flush();
                    session.clear();
            }               
        }
        session.getTransaction().commit();
        if (session.isOpen()) {
            session.close();
        }
    }
}

SAXP

  public void endElement(String uri, String localName, String qName) throws SAXException {
    try {
        if(!recordTag.equals(WWW)&&!recordTag.equals(PROCEEDINGS)){
            super.endElement(uri, localName, qName);
            if(this.str != null){
                this.value = this.str.toString();
            }
            if (qName.equals(AUTHOR) || qName.equals(EDITOR)) {
                 try {                        
                    String temp = value.replaceAll("'","");
                    author = this.authorBO.checkExitAuthor(temp);
                    if (author ==null)
                    {
                        author = new Author();
                        author.setAuthorName(value);
                        authorBO.addNew(author);
                    }
                     authors.add(author);
                     author=null;
                return;
                } catch (Exception ex) {
                    Logger.getLogger(CSPublicationSAXEventHandler.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
            if(qName.equals(TITLE)){
                this.paper.setTitle(value);
                return;
            }

            if(qName.equals(BOOKTITLE)){
                if(recordTag.equals(INPROCEEDINGS)){
                    String temp = value.replaceAll("'","");
                    conference = conferenceBO.checkExitConference(temp);
                    if(conference == null)
                    {
                        conference = new Conference();
                        conference.setConferenceName(value);
                        conferenceBO.addNew(conference);
                        this.paper.setConference(conference);
                        conference=null;
                        return;
                    }
                }else {
                    this.paper.setTitle(value);
                    return;
                }
            }

            if(qName.equals(PAGES)){
                this.paper.setPages(value);
                return;
            }

            if(qName.equals(YEAR)){
                this.paper.setYear(Integer.parseInt(value));
                return;
            }

            if(qName.equals(ADDRESS)){
                this.paper.setAdress(value);
                return;
            }
            if(qName.equals(JOURNAL)){
                try {
                    String temp = value.replaceAll("'","");
                    journal = this.journalBO.checkExitJournal(temp);
                    if (journal ==null)
                    {
                        journal = new Journal();
                        journal.setJournalName(value);
                        journalBO.addNew(journal);
                    }

                    this.paper.setJournal(journal);
                    journal =null;                        
                    return;
                } catch (Exception ex) {
                    Logger.getLogger(CSPublicationSAXEventHandler.class.getName()).log(Level.SEVERE, null, ex);
                }
            }

            if(qName.equals(VOLUME)){
                this.paper.setVolume(value);
                return;
            }

            if(qName.equals(NUMBER)){
                this.paper.setNumber(value);
                return;
            }

            if(qName.equals(MONTH)){
                this.paper.setMonth(value);
                return;
            }

            if(qName.equals(URL)){
                this.paper.setUrl(value);
                return;
            }

            if(qName.equals(EE)){
                this.paper.setEe(value);
                return;
            }

            if(qName.equals(CDROM)){
                this.paper.setCdrom(value);
                return;
            }

            if(qName.equals(CITE)){
                this.paper.setCite(value);
                return;
            }

            if(qName.equals(PUBLISHER)){
                 try {
                    String temp = value.replaceAll("'","");
                    publisher =publisherBO.checkExitPublisher(temp);
                    if (publisher ==null)
                    {
                        publisher = new Publisher();
                        publisher.setNamePublisher(value);
                        publisherBO.addNew(publisher);
                    }

                    this.paper.setPublisher(publisher);
                    publisher=null;                        
                    return;
                } catch (Exception ex) {
                    Logger.getLogger(CSPublicationSAXEventHandler.class.getName()).log(Level.SEVERE, null, ex);
                }
            }

            if(qName.equals(CROSSREF)){
                this.paper.setCrossref(value);
                return;
            }

            if(qName.equals(ISBN)){
                this.paper.setIsbn(value);
                return;
            }

            if(qName.equals(SERIES)){
                this.paper.setSeries(value);
                return;
            }

            if(qName.equals(SCHOOL)){
                this.paper.setSchool(value);
                return;
            }

            if(qName.equals(CHAPTER)){
                this.paper.setChapter(value);
                return;
            }

            if (qName.equals(recordTag)) {
                this.paper.setAuthors(authors);
                this.paperBO.addNew(paper);
                if(this.authors != null){
                    this.authors = null;
                }
                if(this.paper != null){
                    this.paper = null;
                }
                if(this.str != null){
                    this.str = null;
                }
                return;
            }
        }
    } catch (Exception ex) {
        Logger.getLogger(CSPublicationSAXEventHandler.class.getName()).log(Level.SEVERE, null, ex);
    }
}

@Override
public void endDocument() throws SAXException {
    super.endDocument();
}

@Override
public void startDocument() throws SAXException {
    super.startDocument();
    try {
        str = new StringBuffer();
        this.authorBO = AuthorBO.getAuthorBO();
        this.conferenceBO = ConferenceBO.getConferenceBO();
        this.journalBO = JournalBO.getJournalBO();
        this.publisherBO = PublisherBO.getPublisherBO();
        this.paperTypeBO = PaperTypeBO.getPaperTypeBO();
        this.paperBO = PaperBO.getPaperBO();
    } catch (Exception ex) {
        Logger.getLogger(CSPublicationSAXEventHandler.class.getName()).log(Level.SEVERE, null, ex);
    }
}

@Override
public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
    super.startElement(uri, localName, qName, attributes);    
    this.str = new StringBuffer();
    if ((attributes.getLength()>0) && (attributes.getValue("key")!=null)) {
        recordTag = qName;
        this.paper = new Paper();            
        this.authors = new HashSet <Author>();       
        this.paper.setDblpKey(attributes.getValue("key"));            
        if(!recordTag.equals(WWW)&&!recordTag.equals(PROCEEDINGS))
        {
            papertype = this.paperTypeBO.checkExitPaperType(qName);
            if (papertype ==null)
            {
                try {
                    papertype = new PaperType();
                    papertype.setNameType(qName);
                    paperTypeBO.addNew(papertype);
                } catch (Exception ex) {
                    Logger.getLogger(CSPublicationSAXEventHandler.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
            this.paper.setPaperType(papertype);
            papertype=null;
        }
        return;
    }
}

Just a tiny memory and time saver: `TreeSet < Author >` i.o. `HashSet`. — Joop Eggen, Dec 27 '11 at 05:21

score 2 · Accepted Answer · edited May 23 '17 at 12:20

The reason netbeans (or any other tool that runs your code) crashes is simple : on large datasets, hibernate's stateful sessions take a lot of memory up.

The Solution : Use a "stateless" hibernate session - you are supposed to use stateless sessions (which don't track the state of your objects) for large hibernate ETL jobs.

//edit the session creation method in HibernateUtil as follows.
    StatelessSession session = sessionFactory.openStatelessSession();

Why "StatelessSessions" are important for ETL style hibernate jobs : Since typical JVMs run with < 1g ram, you must be careful that youre not cacheing objects in a session when loading 10 gigs of data via hibernate.

Other optimizations :

1) Run your hibernate program on an extremely high performance server, or one that is local to the database server.

2) Disable any foreign-key-checks and indexing in your database server when the importer is running.

3) Look into other hibernate java/GC optimizations, for example, reducing garbage collection duties or increasing JVM memory. Java Application Server Performance

In order to fix this issue, you will need to change the session type - and make some minor edits to the way you insert, commit, and flush.

Why we use different Session types :

When hibernate slows down or stops responding, it probably means your are pushing the JVm too it's Limit in terms of memory (that is, you've created alot of objects that are being managed in JVM memory). Hibernate's excellent web-app performance is due to it's ability to manage the state and life cycle of beans that it queries or updates in the database--- usually, we reap the benefits of this in standard web apps without realizing the hidden cost.. So when we do large, ETL style database programming , we want no cache, no state- just writes at high speed. This means using a stateless session, which does not attempt the fancier, web-optimized hibernate stateful session management.

I have proplem with import into table mapping betwent to table exaple : author_paper (idauthor, idpaper) . If i run with session it auto import when have data in author column and paper colim --> i config ringt . But when i chage seseion by use StatelessSessions i can't import data ? — tiendv, Jan 11 '12 at 07:17

score 0 · Answer 2 · answered Dec 27 '11 at 04:47

If you're going to load that much data, you should consider skipping hibernate, and just used JDBC with batched inserts on a prepared statement. Also make certain that your sax parser is not holding on to items after they've been inserted into the database, you will run out of memory quickly.

Netbean not responding when using Saxp read and hibernate save a lage database

2 Answers2