Adding exactly 100 values to database using ajax

Question

What is the easiest/fastest way to add 100 points to a database? Please assume all writes will not work due to duplicates, bad data, etc.

I'm trying to update a database with exactly 100 values.

Once I have a good piece of data, I need to add it to the database and I use a function called updateDB.

This function just writes a lat/lng coordinate to the database. If there is a duplicate or the write fails, I send "error" from php and the loop should continue collecting data until I have exactly 100 points to the database. Here's the function I'm using.

cct is used for xss prevention, please ignore it, this works fine.

////more above this
if(100-completed > dispatched)
    dispatched++;
    updateDB(lat,lng);
/// more junk and then this function
function updateDB(lat,lng)
{
    var cct = $("input[name=csrf_mysite]").val();
    $.ajax({
        type: "POST",
        url: "/form",
        data: { 
            'lat': lat,
            'lng': lng,
            'id_set': id_set,
            'csrf_complexity': cct },
        success: function(result) {
            var obj = jQuery.parseJSON(result);
            if( obj.status === "OK" )
            {
                completed++;
                var marker = new google.maps.Marker(
                {
                    icon: markerIcon,
                    position: new google.maps.LatLng(lat, lng),
                    map: map
                });
                $( "#progressbar" ).progressbar( "option", {
                    value: completed,
                    max: 100
                });
                $("#amount").text("Getting image " + completed + " of 100");
            }
        },
        error: function(data){
            //alert(data.responseText);
            },
        complete: function(data){
            if(completed == 100)
                window.location = "/start/curate";
            dispatched--;
        }
    });
}

This function does not work. So any idea why?

It should work simply. Call updateDB until it either reaches 100 added values and only call updateDB when there is no possibility that there will be extra calls. Dispatch does not decrement properly so I'm assuming complete isn't called on every event.

Ideas? Or any other way to do this would be awesome.

score 4 · Answer 1 · answered Mar 28 '11 at 09:12

Why don't you have the server code deal with the counting and send only one (or a few) queries? E.g. construct an array of 100 data points first, send the data to the server in a single query, and have it respond with how many more it needs, then send back that many, and do it again until it's got 100.

If the overhead of obtaining each piece of data on the client is very low, and it's OK to get data you don't need, then just send (say) 110 at first. With some knowledge of the failure rate you should be able to optimize this easily.

You can only have (I believe) two simultaneous async queries at once, anyway, so async or not, it's going to take a long time to do this. I can't think of any reason not to group the data as much as possible and cut the number of queries down to 1 or a handful. Even if you still run 100 database queries at the server, the time to do that is inconsequential compared to the overhead of an HTTP post/response.

the only reasonable answer thus far – Uku Loskit Apr 10 '11 at 19:58 — Uku Loskit, Apr 10 '11 at 19:58

score 3 · Accepted Answer · answered Apr 12 '11 at 21:08

Write your server side script to respond with a figure - Then in the first request you sent 100 data points:

Browser             Server
  |                   |
  |---> 100 data ---> |
  |<- send 20 more <- |
  |                   |
  |--->  20 data ---> |
  |<- send  1 more <- |
  |                   |
  |--->  1  data ---> |
  |<- send  0 more <- |
  |                   |
  v                   v

Like that you don't get the kind of sync trouble that comes from counting at the client end how much information has been processed at the server end - the server processes the data, excludes the duplicates, ill-formed etc., and the server counts what it needs.

All the client has to do is send all it can and poll the server for how much more is needed. Also you want to send multiple points at once because of the overheads of an Ajax request, but you don't want to send much more than is actually needed, because that too is wasteful.

Hope this helps.

score 1 · Answer 3 · edited Apr 10 '11 at 19:55

1

if(100-completed > dispatched) should be:

while(100-completed > dispatched)

Assuming completed is 0 to begin with.

edited Apr 10 '11 at 19:55

ThiefMaster

310,957
84
592
636

answered Mar 28 '11 at 07:57

markmnl

11,116
8
73
109

That would work if this were synchronous, but it doesn't work for async processes. If I have 99 completed, often times it will have outstanding threads. By the time the page redirected, often it will have completed 2-3 more threads, meaning 102-103 are in the database. – Phil Salesses Mar 28 '11 at 08:06
im not much a php/jquery person - is there some method you have to call after instantiating your ajax request such as: begin_send()? – markmnl Mar 28 '11 at 10:00
of course you still need while, if will only evaluate once - I have updated my answer – markmnl Mar 28 '11 at 10:02
I have to say - sending 100 requests seems like madness - why?! – markmnl Mar 28 '11 at 10:03

score 0 · Answer 4 · answered Mar 28 '11 at 08:12

0

You'll want to add async:false to your .ajax call and increment/decrement according to the success/error methods you have written in.

If you're meaning for this to be a sequential, iterative process, where the user enters 100 values one at a time and your program stops requesting information once it has 100 values, you may want to do a simple check server side and update your completed/dispatched values that way. However, setting async:false within your .ajax call will block for you and wait for the call's response before continuing forward.

answered Mar 28 '11 at 08:12

Black Box Operations

257
1
5

This also it wayyy to slow for the process I'm doing. Turning async off means the page takes about 2 minutes to add 100 points. With async on it's done in about 20 seconds (but I get 100-107 points sometimes) – Phil Salesses Mar 28 '11 at 08:17
Would adding a simple check in your PHP where no values are allowed to be inserted if SELECT COUNT(*) FROM TABLE returns >= 100 work? – Black Box Operations Mar 28 '11 at 08:33

score 0 · Answer 5 · answered Apr 12 '11 at 07:22

It's strange. I've been able to load about 1000 DB loaded map markers in less than one second.

You shoud load all 100 server side and add them in one request. The request cost is usually quite big (for simple task like connect to db and load one item it may be like 70% time spent on request and 30% on actual work).

score 0 · Answer 6 · answered Apr 14 '11 at 04:15

0

Maybe it's my personal love for JSON and serialising form data, but have you thought of simply serialising a data-packet on the client side, sending the data and requesting a pass/fail response?

answered Apr 14 '11 at 04:15

Mitch Malone

882
6
17

score 0 · Answer 7 · answered Apr 15 '11 at 07:28

I think you have a bug in the code but I am not sure if it is just a typing error.

if(100-completed > dispatched) dispatched++; updateDB(lat,lng);

Based on your indentation you want to call updateDB in the if clause, but you didn't use curly braces so you call it even if the statement is true or not. Without this bug although your implementation would be better if you used the method that @jamietre suggested, your code should work.

Reese · Answer 8 · 2011-04-15T16:58:22.737

Edit: nevermind this answer, the missing braces in the 'if' statement threw me off, updateDB is only called when 100-completed > dispatched right?

Assuming I'm understanding your problem, it looks like it might be a timing issue. For example, when completed = 99, the updateDB function could be called 4 more times before the first time the success callback is executed for when completed was 99, and then it still has 3 more queued up (depending on the code surrounding the 'if' statement). So, it would be slower, but if you can make sure that updateDB is not called again until the 'completed' callback (and/or success/error, if you think completed isn't being called every time), do you think that might fix it?
This is the order I'm imagining the lines being executed, if you set breakpoints on two lines updateDB(lat,lng); and completed++:

updateDB(lat,lng); (where completed=99)
updateDB(lat,lng); (where completed=99)
updateDB(lat,lng); (where completed=99)
updateDB(lat,lng); (where completed=99)
completed++; (completed now=100, 3 more ajax calls still executing)
completed++; (completed now=101)
completed++; (completed now=102)
completed++; (completed now=103)

My first idea would be to add a flag in the completed callback like so:

complete: function(data){
ok_to_run_again_flag=true;
dispatched--;
if(completed == 100)
window.location = "/start/curate";
};

and

if((100-completed > dispatched) && ok_to_run_again_flag)
{
ok_to_run_again_flag=false;
dispatched++;
updateDB(lat,lng);
}

I'm also wondering why you say "Dispatch does not decrement properly". What is its value, compared to what you are expecting?

Adding exactly 100 values to database using ajax

8 Answers8