Sending subsequent curl requests after receiving response for initial request

Question

Before I write the details, here's what I want to achieve.

1. Search for some images on pixabay (searching for yellow flowers in this case)
2. When I post this query, I will get json array with details of images.
3. Parse the data and store array data.
4. Now send subsequent curl request to retrieve/download images using URLs present in the array data.

Current code :

I was able to achieve 1, 2 and 3. 3rd implementation I have done (as of now) by store data locally in a file and parsing it.

I am stuck at 4th point.

Posting one of the element from array data :

   "hits":[
      {
         "id":2295434,
         "pageURL":"https://pixabay.com/photos/spring-bird-bird-tit-spring-blue-2295434/",
         "type":"photo",
         "tags":"spring bird, bird, tit",
         "previewURL":"https://cdn.pixabay.com/photo/2017/05/08/13/15/spring-bird-2295434_150.jpg",
         "previewWidth":150,
         "previewHeight":99,
         "webformatURL":"https://pixabay.com/get/gc323739b5570ab1afac9cff34f0ed431beffcf004e0660fcaff96a7b1780933e31c982b26cdb3234c4a757a9e7e8b824bda6059340ead3c6b755f1265f7ace52_640.jpg",
         "webformatWidth":640,
         "webformatHeight":426,
         "largeImageURL":"https://pixabay.com/get/g58a377c4cffe13bffd15cb7b455ecec08329352a443689296c5f15565b3a7f11bcd5ec6e4d2945bb3d32b2a63ee33d6c9a9925119944d128cab4bca663f87620_1280.jpg",
         "imageWidth":5363,
         "imageHeight":3575,
         "imageSize":2938651,
         "views":488125,
         "downloads":261832,
         "collections":1911,
         "likes":1846,
         "comments":221,
         "user_id":334088,
         "user":"JillWellington",
         "userImageURL":"https://cdn.pixabay.com/user/2018/06/27/01-23-02-27_250x250.jpg"
      }
]

main.cc

#include "baseurihandler/base_uri_handler_pixabay.h"

#include "json_parser.h"
#include "download_image.h"

#include <stdio.h>
#include <iostream>
#include <curl/curl.h>

#include <iostream>
#include <chrono>
#include <string>
#include <thread>
#include <vector>

void JsonParser();

int main() {
    CURL *curl;
    CURLcode res;

    std::vector<std::pair<std::string, std::string>> query_param;
    query_param.push_back(std::make_pair("q", "yellow+flowers"));
    query_param.push_back(std::make_pair("image_type", "photo"));

    std::string s;

    baseuri::PixabayURIHandler pixabay_uri_handler;

    pixabay_uri_handler.SetQueryParameters(query_param);

    //auto str = pixabay_uri_handler.GetURI();
    auto str = std::string("https://pixabay.com/api/?key=xxxxxxx-xxxxxxxxxxx&q=yellow+flowers&image_type=photo");
    std::cout << str << std::endl;

    curl = curl_easy_init();
    if(curl) {

        curl_easy_setopt(curl, CURLOPT_URL, str.c_str());

        /* example.com is redirected, so we tell libcurl to follow redirection */
        curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, baseuri::PixabayURIHandler::CurlWrite_CallbackFunc_StdString);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &pixabay_uri_handler);

        /* Perform the request, res will get the return code */
        res = curl_easy_perform(curl);
        /* Check for errors */
        if (res != CURLE_OK)
            fprintf(stderr, "curl_easy_perform() failed: %s\n",
                    curl_easy_strerror(res));

        /* always cleanup */
        curl_easy_cleanup(curl);
    }

    //std::cout << pixabay_uri_handler.GetResultJsonString() << std::endl;
    std::cout << "calling jsonparser.. " << std::endl;
    JsonParser(); //This will return vec of URLs - TBD

    char *jpg_test = "https://pixabay.com/get/g58a377c4cffe13bffd15cb7b455ecec08329352a443689296c5f15565b3a7f11bcd5ec6e4d2945bb3d32b2a63ee33d6c9a9925119944d128cab4bca663f87620_1280.jpg";
    if (!download_jpeg(jpg_test))
    {
        printf("!! Failed to download file!\n" );
        return -1;
    }

    return 0;
}

Implementation of download_jpeg:

#include <stdio.h>
#include <curl/curl.h>

size_t callbackfunction(void *ptr, size_t size, size_t nmemb, void* userdata)
{
    FILE* stream = (FILE*)userdata;
    if (!stream)
    {
        printf("!!! No stream\n");
        return 0;
    }

    size_t written = fwrite((FILE*)ptr, size, nmemb, stream);
    return written;
}

bool download_jpeg(char* url)
{
    FILE* fp = fopen("out.jpg", "wb");
    if (!fp)
    {
        printf("!!! Failed to create file on the disk\n");
        return false;
    }

    CURL* curlCtx = curl_easy_init();
    curl_easy_setopt(curlCtx, CURLOPT_URL, url);
    curl_easy_setopt(curlCtx, CURLOPT_WRITEDATA, fp);
    curl_easy_setopt(curlCtx, CURLOPT_WRITEFUNCTION, callbackfunction);
    curl_easy_setopt(curlCtx, CURLOPT_FOLLOWLOCATION, 1);

    CURLcode rc = curl_easy_perform(curlCtx);
    if (rc)
    {
        printf("!!! Failed to download: %s\n", url);
        return false;
    }

    long res_code = 0;
    curl_easy_getinfo(curlCtx, CURLINFO_RESPONSE_CODE, &res_code);
    if (!((res_code == 200 || res_code == 201) && rc != CURLE_ABORTED_BY_CALLBACK))
    {
        printf("!!! Response code: %d\n", res_code);
        return false;
    }

    curl_easy_cleanup(curlCtx);

    fclose(fp);

    return true;
}

What is the correct way of forming the subsequent URL for downloading images? Instead of directly sending largeImageURL to download_jpeg(), I tried adding api key as query parameter together with largeImageURL value, but did not help -- received 400 response code.

There's nothing in the pixabay documentation that is helping.

Answer 1

curl easy handles do not share session states without a special additional actions. But each individual one keeps a session state and can reuse it until curl_easy_cleanup() is called. You use another curl easy handle in download_jpeg - this is the reason of the error "This URL is invalid or has expired."

The easiest fix is reusing the curl easy handle.

bool download_jpeg(CURL* curl, const char* url)
{
    FILE* fp = fopen("out.jpg", "wb");
    if (!fp)
    {
        printf("!!! Failed to create file on the disk\n");
        return false;
    }

    curl_easy_setopt(curl, CURLOPT_URL, url);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, callbackfunction);
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);

    CURLcode rc = curl_easy_perform(curl);
    if (rc)
    {
        printf("!!! Failed to download: %s\n", url);
        return false;
    }

    long res_code = 0;
    curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &res_code);
    if (!((res_code == 200 || res_code == 201) && rc != CURLE_ABORTED_BY_CALLBACK))
    {
        printf("!!! Response code: %d\n", res_code);
        return false;
    }

    fclose(fp);

    return true;
}

int main() {
    CURL *curl;
    CURLcode res;

    std::vector<std::pair<std::string, std::string>> query_param;
    query_param.push_back(std::make_pair("q", "yellow+flowers"));
    query_param.push_back(std::make_pair("image_type", "photo"));

    std::string s;

    baseuri::PixabayURIHandler pixabay_uri_handler;

    pixabay_uri_handler.SetQueryParameters(query_param);

    //auto str = pixabay_uri_handler.GetURI();
    auto str = std::string("https://pixabay.com/api/?key=xxxxxxx-xxxxxxxxxxx&q=yellow+flowers&image_type=photo");
    std::cout << str << std::endl;

    curl = curl_easy_init();
    if(curl) {
        curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "");
        curl_easy_setopt(curl, CURLOPT_URL, str.c_str());

        curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, baseuri::PixabayURIHandler::CurlWrite_CallbackFunc_StdString);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &pixabay_uri_handler);

        /* Perform the request, res will get the return code */
        res = curl_easy_perform(curl);
        /* Check for errors */
        if (res != CURLE_OK)
            fprintf(stderr, "curl_easy_perform() failed: %s\n",
                    curl_easy_strerror(res));
    }

    //std::cout << pixabay_uri_handler.GetResultJsonString() << std::endl;
    std::cout << "calling jsonparser.. " << std::endl;
    JsonParser(); //This will return vec of URLs - TBD

    const char* jpg_test = "https://pixabay.com/get/g58a377c4cffe13bffd15cb7b455ecec08329352a443689296c5f15565b3a7f11bcd5ec6e4d2945bb3d32b2a63ee33d6c9a9925119944d128cab4bca663f87620_1280.jpg";
    const bool rv = download_jpeg(curl, jpg_test);
    if (!rv)
        printf("!! Failed to download file!\n" );

    /* always cleanup */
    curl_easy_cleanup(curl);

    return rv ? -1 : 0;
}

The original download_jpeg() has the issues:

•  curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
  

  must pass long 1L

   curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
  

• Memory leaks - curl_easy_cleanup(curlCtx); is not called if error occurs, due to eraly return. The new version does not create new curl easy handler, thus there is no a memory leak.

The main has been updated:

• curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "") activates the cookie engine.
• curl handle is passed to download_jpeg and reused there.
• curl_easy_cleanup(curl); has been moved to the end.
• String literals are const char arrays in C++, thus const char* jpg_test = ""; must be used.

---

If you need use several curl easy handles in a single session, follow the manual Sharing between easy handles, or use a single curl multi handle with many curl easy handles.