0

I have more than 1000000 record how to speed up search in sheet. I normally search for 20s how to improve ? (sheet include 20 column and 10000 record)

var ss =  SpreadsheetApp.openByUrl(urldb);
var ws =  ss.getSheetByName("Account");
var data = ws.getDataRange().getValues();

for(var i = 0; i < data .length; i++){

    if(data [i][1] == "ID998724"){

      Logger.log("found you" + data [i][1]);

    }
};

return data[i][1];
Zenoo
  • 12,670
  • 4
  • 45
  • 69
Naoa Lee
  • 135
  • 2
  • 19
  • 3
    Since you're only looking in column 2 for your data, instead of using `.getDataRange()` have you tried using `.getRange()` for just column 2 and passing the amount of rows your sheet has? – ross Jun 19 '19 at 08:31
  • A million records should probably be in a database, not a spreadsheet, but anyway, could you do the fetching of the data asynchonously? Request about 1,000 or 10,000 records at a time, and as the next batch downloads search the previous. (And is this a better fit for [codereview.se]?) – Ken Y-N Jun 19 '19 at 08:40
  • @loc dinh Did my answer show you the result what you want? Would you please tell me about it? That is also useful for me to study. If this works, other people who have the same issue with you can also base your question as a question which can be solved. If you have issues for my answer yet, I apologize. At that time, can I ask you about your current situation? I would like to study to solve your issues. – Tanaike Jun 25 '19 at 22:47
  • Thanks all for answering my question ! I want speed up find also get data row it – Naoa Lee Jun 26 '19 at 07:15

1 Answers1

11
  • You want to search the value of ID998724 form the column "B" on the sheet of "Account" in Spreadsheet using Google Apps Script.

If my understanding is correct, how about these 3 sample scripts? Please think of this as just one of several answers.

Sample script 1:

In this script, I used Class TextFinder for this situation. This was added in a recent update of Google.

var urldb = "###"; // Please set this.
var searchValue = "ID998724";

var ss = SpreadsheetApp.openByUrl(urldb);
var ws = ss.getSheetByName("Account");
var f = ws.createTextFinder(searchValue).findAll();
if (f.length > 0) {
  for (var i = 0; i < f.length; i++) {
    if (f[i].getColumn() == 2) {
      Logger.log("found you" + f[i].getValue())
    }
  }
}

Sample script 2:

In this script, the values are retrieved from the column "B". This is also mentioned at ross's comment. And also from the result of benchmark, I used filter() for this situation.

var urldb = "###"; // Please set this.
var searchValue = "ID998724";

var ss = SpreadsheetApp.openByUrl(urldb);
var ws = ss.getSheetByName("Account");
var data = ws.getRange(1, 2, ws.getLastRow(), 1).getValues();
var f = data.filter(function(e) {return e[0] == searchValue});
if (f.length > 0) {
  for (var i = 0; i < f.length; i++) {
    Logger.log("found you" + f[i])
  }
}

Sample script 3:

In this script, I used Query Language for this situation.

var urldb = "###"; // Please set this.
var searchValue = "ID998724";

var ss = SpreadsheetApp.openByUrl(urldb);
var ws = ss.getSheetByName("Account");
var query = "select * where B='" + searchValue + "'";
var url = "https://docs.google.com/spreadsheets/d/" + ss.getId() + "/gviz/tq?gid=" + ws.getSheetId() + "&tqx=out:csv&tq=" + encodeURIComponent(query);
var options = {headers: {'Authorization': 'Bearer ' + ScriptApp.getOAuthToken()}};
var csv = UrlFetchApp.fetch(url, options);
var f = Utilities.parseCsv(csv);
if (f.length > 0) {
  for (var i = 0; i < f.length; i++) {
    Logger.log("found you" + f[i][1])
  }
}

Note:

  • In your script, I think that an error occurs at return data[i][1]. Because i is the same with data.length. If you want to retrieve the value by return data[i][1], for example, please put break after Logger.log("found you" + data [i][1]).

References:

If I misunderstood your question and these sample scripts were not the results you want, I apologize.

ross
  • 2,684
  • 2
  • 13
  • 22
Tanaike
  • 181,128
  • 11
  • 97
  • 165
  • You can explain more Sample script 3 to me ? I do not understand about (" var options = {headers: {'Authorization': 'Bearer ' + ScriptApp.getOAuthToken()}}; var csv = UrlFetchApp.fetch(url, options); var f = Utilities.parseCsv(csv); ") what to do ??? – Naoa Lee Jun 26 '19 at 04:40
  • 1
    @loc dinh Thank you for replying. 1.``var options = {headers: {'Authorization': 'Bearer ' + ScriptApp.getOAuthToken()}}`` is used for [``UrlFetchApp``](https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app#fetch(String,Object)). It is required to use the access token for this method. 2. ``var csv = UrlFetchApp.fetch(url, options); var f = Utilities.parseCsv(csv)`` is used for parsing the retrieved values from UrlFetchApp. [Ref](https://developers.google.com/apps-script/reference/utilities/utilities#parsecsvcsv) Because the result is returned as CSV data. – Tanaike Jun 26 '19 at 05:14
  • @Tannaike Thanks you so much. But i have a question I find ID but how can i get record it (row data). Because speed ​up the search ID finally for get data it. In your way only get ID – Naoa Lee Jun 26 '19 at 07:14
  • 1
    @loc dinh If you are using sample 3, ``csv`` of ``var csv = UrlFetchApp.fetch(url, options)`` is the values including the rows which has ``searchValue``. So when you want the rows including ``searchValue``, please modify ``Logger.log("found you" + f[i][1])`` to ``Logger.log("found you" + f[i])``. By this, you can see the list of rows. Is my understanding for your replying correct? – Tanaike Jun 26 '19 at 07:29
  • I change Logger.log("found you" + f[i][1]) to Logger.log("found you" + f[i]) it run well but length of it is 26 --> i think this is 26 column but my sheet have 5 column. And i have more question if we parse to csv so find id in csv is faster ? why parse to csv – Naoa Lee Jun 26 '19 at 07:47
  • @loc dinh I deeply apologize for my poor English skill. In the sample 3, the values are not searched from CSV data. ``searchValue`` is searched using the Query Language. In order to use this, UrlFetchApp is used. In this case, the searched result is returned as CSV data. I used CSV parser for showing the ID. If you want only the result, you can directly retrieve ``csv`` of ``var csv = UrlFetchApp.fetch(url, options)``, and the scripts below ``var f = Utilities.parseCsv(csv)`` are not required. In the current stage, in my environment, the sample 3 that Query Language is used is fastest of all. – Tanaike Jun 26 '19 at 07:53
  • @loc dinh And when you want to retrieve the column "A" to "E", please modify the query from ``var query = "select * where B='" + searchValue + "'"`` to ``var query = "select A,B,C,D,E where B='" + searchValue + "'"``. – Tanaike Jun 26 '19 at 07:59
  • No problem, me too. Thanks for your enthusiasm, my English skills are also very bad. In example 3 are you sure that it will improve speed ? @Tanaike tell me why ? – Naoa Lee Jun 26 '19 at 08:06
  • Tks you again, I modify query and i run it run well, returns the result as I want. So you can tell me why your way faster than me ? – Naoa Lee Jun 26 '19 at 08:17
  • @loc dinh Thank you for replying. When a value is searched from Spreadsheet, I think that there are largely divided into 2 methods. One is that at first, all values are retrieved, and then the value is searched from the retrieved values. Another is that the value is directly searched without retrieving the values from Spreadsheet. I think that retrieving all values at the latter might be done with inside of Google. – Tanaike Jun 26 '19 at 12:05
  • @loc dinh The sample 3 uses the latter method. I think that the reason of the speed is due to this. The internal process of Google might be much faster than that of the run of Google Apps Script. If my speculation was not what you want, I apologize. – Tanaike Jun 26 '19 at 12:05
  • @Tannaike No, you are really good ! Thanks you so much. You have shared knowledge that is very useful to me. You are very interesting. Do you use social networks like facebook ? I want to be friends with you. If possible, please give me the information sent to me via gmail : stack.quyetle95@gmail.com . Again, Thanks you very much for helping me ! – Naoa Lee Jun 27 '19 at 01:40
  • @loc dinh Thank you for replying. I'm glad your issue was resolved. Unfortunately, I have not facebook. I deeply apologize for my situation. – Tanaike Jun 27 '19 at 08:56
  • @Tannaike. Thanks you so much ^^ and i have new ask question for new problem, can you help me ? – Naoa Lee Jun 27 '19 at 09:23
  • @loc dinh Thank you for replying. If you have new question, can you post it as new question to Stackoverflow? By this, your question can be thought by many users including me. If you can cooperate to resolve your issue, I'm glad. – Tanaike Jun 27 '19 at 12:12
  • I have posted it, Thank you for spending time with me ! ^^ – Naoa Lee Jun 28 '19 at 01:22
  • Tanaike this is link of new my question https://stackoverflow.com/questions/56785787/how-to-get-gmail-of-user-use-my-app-script . I think you can help me solve this problem. Please help me ! – Naoa Lee Jun 28 '19 at 08:35
  • @loc dinh Thank you for replying. If I could correctly understand about it and find the solution, I would like to answer it. – Tanaike Jun 28 '19 at 13:07
  • hi @Tanaike, I have 1 more question ? In 3 Sample script, what is the fastest way? – Naoa Lee Sep 19 '19 at 04:39
  • @Naoa Lee "Sample script 3" is the fastest of all. 2nd one is "Sample script 1". The process costs of "Sample script 3" and "Sample script 1" are much lower than that of "Sample script 2". These carries out retrieving and searching values in the internal server side. I think that this is the reason of the differences. – Tanaike Sep 19 '19 at 04:48
  • thank you very much. But in "Sample script 3" if data have column datetime type or specal type is data return to fail. How to improve it run alway well ? – Naoa Lee Sep 19 '19 at 06:19
  • @Naoa Lee Thank you for replying. Although I'm not sure about the detail of your new issue of `return to fail`, for example, how about using "Sample script 1"? If you still have the issue, can you post it as new question by including the detail information? Because I have no information about your current issue and I would like to think of your new issue by correctly understanding your current situation, and also your new issue is different from your this question which search the value of `ID998724` form the column "B". If you can cooperate to resolve your issue. I'm glad. – Tanaike Sep 19 '19 at 08:12
  • @Tanaike I usually want to find the index (row position) where the value is found. I usually use var hVals=sht.getRange("H:H").getValues(); var rowFound=hvals.findIndex(row=>row[0]=="xx"). Can the above 3 script examples give me the row number where the match was found? Will it save time? – arul selvan Jul 19 '23 at 10:39
  • @Tanaike I found method 1 is able to give me the row number where the value is found by f[i].getRow(). It is slightly slower than method 3, but much faster than getvalues and findIndex. Thank you – arul selvan Jul 19 '23 at 11:18
  • @arul selvan About your questions, I would like to support you. But, I have to apologize for my poor English skill. Unfortunately, I cannot understand your questions. So, can you post it as a new question by including more information? By this, it will help users including me think of a solution. If you can cooperate to resolve your questions, I'm glad. Can you cooperate to do it? – Tanaike Jul 19 '23 at 11:51