1

When I try bulk query in nodejs with promise.all I got ECONNREFUSED error after some number of query resulted in success.

I am building node server for my web application. I made some queries resulted from AJAX requests from client. They all worked fine till now.

I tried to load data from excel to some number of tables (i use pg module). I wrote a code that insert records after reading if there is not a same record. I used promise to make queries. But it made some queries and then start to get ECONNREFUSED error.

  • I have changed max_connections to 10000 and shared_buffers to 25000MB. And restart postgres server
  • I have changed max_connections to 1000 and shared_buffers to 2500MB.And restart postgres server
  • I have changed max_connections to 300 and shared_buffers to 2500MB. And restart postgres server
  • I have changed my code from POSTGRESQL Pool to POSTGRESQL Client

I have omitted some query in promise array

but nothing changed. Almost 180 records were inserted at all. Then got en error.


    function loadData(auditArrayObject){
        return new Promise(function (resolve, reject) {
            let promises=[
                loadAuditItem(auditArrayObject.audit,
                             auditArrayObject.auditVersion),
                loadProcesses(auditArrayObject.processArray),
                loadControlAims(auditArrayObject.controlAimArray),
                loadCriterias(auditArrayObject.criteriaArray),
                loadParameters(auditArrayObject.parameterArray),
            ]
            Promise.all(promises)
            .then(objectsWithId=>{
              ......
            }
    }

    function loadProcesses(processArray){
        return new Promise(function (resolve, reject) {
            let promises=[];
            for(let i=0;i<processArray.length;i++){
                let process= new Process(null,processArray[i],false)
                let promise= postGreAPI.readProcessByName(process.name)
                             .then(resultProcess=>{
                                if (!resultProcess) {
                                    postGreAPI.createProcess(process)
                                    .then(createdProcess=>{
                                        resolve(createdProcess)
                                    })
                                    .catch(err=>{
                                        reject({msg:"createProcess 
                                               hata aldı",err:err})
                                    })
                                } else {
                                    return (resultProcess)
                                }
                             })
                             .catch(err=>{
                                reject({msg:"readProcessByName 
                                        hata aldı",err:err})
                             })
                promises.push(promise)
            }
            Promise.all(promises)
            .then(processArray=>{
                resolve({key:"Process",value:processArray})
            })
            .catch(err=>{
                reject({msg:"Processlerden birisi insert edilemedi",err:err})
            })
        });
    }

    postGreAPI.readProcessByName:

    var readProcessByName = function (name){

            return new Promise(function (resolve, reject) {
                let convertedName=convertApostrophe(name)
                let query = "SELECT * FROM process WHERE name='" 
                          + convertedName + "'"
                queryDb(query)
                .then(result=>{
                    if (result.rows.length>0){
                        let process = new Process(result.rows[0].id,
                                                  result.rows[0].name,
                                           result.rows[0].isactive);
                        resolve(process)
                    }else{
                        resolve(null)
                    }
                })
                .catch(err=>{
                    reject(err)
                })
            })

        }

    queryDb:

    var queryDb2 = function (query,params) {
            return new Promise(function (resolve, reject) {
                let pool = new PostGre.Pool(clientConfig);
                pool.connect(function(err, client, done) {
                    if(err) {
                        return reject (err);
                    }
                    client.query(query,params, function(err, result) {
                        done();
                        if(err) {
                            return reject (err);
                        }
                        resolve(result)
                    });
                });
            })
        }

And error is :

     Error: connect ECONNREFUSED 127.0.0.1:5432
        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1113:14)
      errno: 'ECONNREFUSED',
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 5432 }

Actually i have succed that load before my laptop configuration changed. Before there was a windows 7 but now windows 10.

Bergi
  • 630,263
  • 148
  • 957
  • 1,375
Alper Bilgil
  • 21
  • 1
  • 6
  • Are you maybe creating a new connection for each row and then immediately closing it again? In that case this might be a Windows problem: Windows keeps closed ports open (reserved?) for some time. So if you constantly (and quickly) open and close connections, this might mean that Windows runs out of (client) ports. Can you refactor the code to only open and use a **single** connection for the whole process? (Btw: it's either Postgres or PostgreSQL, but [not "postGre"](https://wiki.postgresql.org/wiki/FAQ#What_is_PostgreSQL.3F_How_is_it_pronounced.3F_What_is_Postgres.3F)) –  Jan 09 '19 at 11:45
  • Tanks for answer. It seems fixed. When i create pool once for whole process. – Alper Bilgil Jan 10 '19 at 14:12

1 Answers1

0

Your code seems to be in bits and pieces, So cannot point exact line for the error, But the error is coming because of one reason, connection pooling.

Your code is attempting to create a connection with POSTGRES every time you query the database hence for initial runs some data is entered in the database and then starts failing, you need to remove every time connecting part from the query section and use a single instance to deal with the query and also close connection once completed.

You also mentioned you have upgraded from Win 7 to Win 10, their is no problem with windows version, but your hardware might have got higher configuration too (Ram and number of cores), the way event loop works you some time don't get this error with low configuration systems but with larger config systems you get these errors.

r7r
  • 1,440
  • 1
  • 11
  • 19