2

Goal:

To slightly slow down the sending of requests when multiplexing with libcurl, possibly by introducing small time delays in between the sending of each of the HTTP/2 request to a server. The multiplexing program needs to listen out for any changes from one webpage for around 3 seconds at a set time once a day. However, the multiplexing program finishes execution in under a second even when setting the variable num_transfers to the thousands (variable seen in the code further below).

It would be useful if there was a way to introduce for example, a 3 millisecond delay in between the transmission of a group of multiplex requests. This would mean the program could still send requests asynchronously (so it be won't blocked / won't have to wait for a response from the server before sending the next request) but at a slightly slower rate.

A definition of multiplexing taken from this resource:

Multiplexing is a method in HTTP/2 by which multiple HTTP requests can be sent and responses can be received asynchronously via a single TCP connection. Multiplexing is the heart of HTTP/2 protocol.

Ideal outcome:

An ideal program for this situation would be one that could send a few non-blocking/ multiplex requests every approx. 3 milliseconds. The program would run for around 3-4 seconds in total.

Current problem:

Currently the program is too fast when multiplexing, meaning a few thousand requests could be sent and received within around 350 milliseconds which can lead to the sending IP address to be blocked for a few minutes..

Please note it is not an option in this scenario to use a synchronous / blocking approach - a requirement of this program is that it must not be forced to wait for a response to be returned before sending another request. The issue lies in the fact that the program is too fast at sending a high number of requests.

Attempts at solving:

In the DO...WHILE loop seen in the code below, an attempt was made to introduce some artificial time delays at various locations within the loop using usleep(microseconds) from unistd.h, but this introduced a time delay either before or after all of the requests were sent, rather than an interleaved time delay between sending requests.

Current code:

#include <iostream>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <chrono>

#include <string>

/* somewhat unix-specific */
#include <sys/time.h>
#include <unistd.h>

/* curl stuff */
#include <curl/curl.h>
#include <curl/mprintf.h>

#ifndef CURLPIPE_MULTIPLEX
#define CURLPIPE_MULTIPLEX 0
#endif

struct CURLMsg *msg;

struct transfer {
    CURL *easy;
    unsigned int num;
    //FILE *out;
    std::string contents;
};

struct MemoryStruct {
    char *memory;
    size_t size;
};

struct MemoryStruct chunk;

#define NUM_HANDLES 1000

static size_t WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp) {
    transfer *t = (transfer *)userp;
    size_t realsize = size * nmemb;    
    t->contents.append((const char*)contents, realsize);
    return realsize;
}

static void setup(struct transfer *t, int num)
{
    CURL *hnd;

    hnd = t->easy = curl_easy_init();

    curl_easy_setopt(hnd, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
    curl_easy_setopt(hnd, CURLOPT_WRITEDATA, (void *)t);

    /* set the same URL */
    curl_easy_setopt(hnd, CURLOPT_URL, "https://someurl.xyz");

    /* HTTP/2 please */
    curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);

    /* we use a self-signed test server, skip verification during debugging */
    curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
    curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);

    #if (CURLPIPE_MULTIPLEX > 0)
    /* wait for pipe connection to confirm */
    curl_easy_setopt(hnd, CURLOPT_PIPEWAIT, 1L);
    #endif
}

int main() {
    struct transfer trans[NUM_HANDLES];
    CURLM *multi_handle;
    int i;
    int still_running = 0; /* keep number of running handles */
    int num_transfers = 3;

    chunk.memory = (char*)malloc(1);
    chunk.size = 0;

    /* init a multi stack */
    multi_handle = curl_multi_init();

    for(i = 0; i < num_transfers; i++) {
        setup(&trans[i], i);

        /* add the individual transfer */
        curl_multi_add_handle(multi_handle, trans[i].easy);
    }

    curl_multi_setopt(multi_handle, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
//  curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, 1L);

    // Main loop
    do {

        CURLMcode mc = curl_multi_perform(multi_handle, &still_running);

        if(still_running) {
            /* wait for activity, timeout or "nothing" */
            mc = curl_multi_poll(multi_handle, NULL, 0, 1000, NULL);
        }

        if(mc) {
            break;
        }

        // Get response
        do {
            int queued;
            msg = curl_multi_info_read(multi_handle, &queued);
            if ((msg) && (msg->msg == CURLMSG_DONE) && (msg->data.result == CURLE_OK)) {

                // Get size of payload
                curl_off_t dl_size;
                curl_easy_getinfo(msg->easy_handle, CURLINFO_SIZE_DOWNLOAD_T, &dl_size);

                    for (int i = 0; i < num_transfers; i++) {
                        std::cout << trans[i].contents;
                    }
                    std::cout << std::flush;
            }
            
        } while (msg);

    } while (still_running);

    for(i = 0; i < num_transfers; i++) {
        curl_multi_remove_handle(multi_handle, trans[i].easy);
        curl_easy_cleanup(trans[i].easy);
    }

    free(chunk.memory);
    curl_multi_cleanup(multi_handle);
        
    return 0;
}

Summary question:

Is there a way to send a group of multiplexed requests to a single URL approximately every 3 milliseconds? Another idea to attempt to solve this was to wrap the entire functionality contained in main() within a FOR loop, and putting a time delay at the end of each iteration of the FOR loop.

p.luck
  • 646
  • 2
  • 9
  • 34
  • https://en.cppreference.com/w/cpp/thread/sleep_for Keep in mind that the system most likely will sleep longer than you wanted it to. Depending on your use case that may not be a problem. – George Aug 25 '21 at 15:41

0 Answers0