Fetching data using other runtimes (python, .NET, Java) as well as cURL works fine, but on NodeJS, returns Error 404

Question

So I was trying to fetch data by reverse engineering Twitter's API. Fetching the data using cURL or even in other languages and runtimes such as .NET (using RestSharp), python (using requests) works fine and returns the tweets as expected.

When I do the same in NodeJS, irrespective of the library (axios, https, requests), it always returns Error 404

The request I'm using is:

cURL Code

curl --location --request GET 'https://api.twitter.com/2/search/adaptive.json?q=(from%3Aelonmusk)&query_source=typed_query&count=40' --header 'sec-ch-ua: "Not_A Brand";v="99", "Microsoft Edge";v="109", "Chromium";v="109"' --header 'x-twitter-client-language: en' --header 'x-csrf-token: <csrf_token>' --header 'sec-ch-ua-mobile: ?0' --header 'authorization: Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA' --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.61' --header 'x-twitter-auth-type: OAuth2Session' --header 'x-twitter-active-user: yes' --header 'sec-ch-ua-platform: "Windows"' --header 'Accept: */*' --header 'host: api.twitter.com' --header 'Cookie: <cookie>'

NodeJS Axios Code

var axios = require('axios');

var config = {
  method: 'get',
  maxBodyLength: Infinity,
  url: 'https://api.twitter.com/2/search/adaptive.json?q=(from%3Aelonmusk)&query_source=typed_query&count=40',
  headers: { 
    'sec-ch-ua': '"Not_A Brand";v="99", "Microsoft Edge";v="109", "Chromium";v="109"', 
    'x-twitter-client-language': 'en', 
    'x-csrf-token': '<csrf_token>', 
    'sec-ch-ua-mobile': '?0', 
    'authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA', 
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.61', 
    'x-twitter-auth-type': 'OAuth2Session', 
    'x-twitter-active-user': 'yes', 
    'sec-ch-ua-platform': '"Windows"', 
    'Accept': '*/*', 
    'host': 'api.twitter.com', 
    'Cookie': '<cookie>'
  }
};

axios(config)
.then(function (response) {
  console.log(JSON.stringify(response.data));
})
.catch(function (error) {
  console.log(error);
});

The response I'm getting is: AxiosError: Request failed with status code 404

If I use node-libcurl library to fetch data instead of axios or https, and disable ssl verification while using node-libcurl library, the data is fetched as intended and I am getting the list of tweets.

Also, executing the same request in Postman works alright too.

The problem only occurs in NodeJS

Answer 1

Bearer token has expiry, after expired time, it will not works.

So you need to call get token first with API key/secret, Detail for get access-token in here

Then search API will remove time dependency. detail about search in here

This code will works

const axios = require("axios")

const getToken = async () => {
    try {
        const API_KEY = '<your API Key>'
        const API_KEY_SECRET = '<your API SECRET>'

        const response = await axios.post(
            'https://api.twitter.com/oauth2/token',
            '',
            {
                params: {
                    'grant_type': 'client_credentials'
                },
                auth: {
                    username: API_KEY,
                    password: API_KEY_SECRET
                }
            }
        );
        return Promise.resolve(response.data.access_token);

    } catch (error) {
        return Promise.reject(error);
    }
}

const searchTweet = async (token, query, limit) => {
    try {
        const siteUrl = `https://api.twitter.com/2/tweets/search/recent?tweet.fields=created_at,geo&query=${query}&max_results=${limit}`
        const response = await axios.get(siteUrl, {
            headers: {
                'Accept-Encoding': 'application/json',
                Authorization: `Bearer ${token}`
            }
        });
        return Promise.resolve(response.data);

    } catch (error) {
        return Promise.reject(error);
    }
}

getToken().then(token => {
    // console.log("access token: " + token)
    searchTweet(token, 'elonmusk', 10).then(tweets => {
        for (tweet of tweets.data) { 
            console.log(JSON.stringify(tweet))
        }
    }).catch(error => console.log(error));
}).catch(error => console.log(error));

Result

$ node get-tweet.js
{"created_at":"2023-02-01T16:29:26.000Z","edit_history_tweet_ids":["1620821639058563072"],"id":"1620821639058563072","text":"RT @ThierryBreton: I take note of the path that Twitter is committed to take in Europe to comply with #DSA rules.\n\nNext few months will be…"}
{"created_at":"2023-02-01T16:29:26.000Z","edit_history_tweet_ids":["1620821638102278145"],"id":"1620821638102278145","text":"@Tessa430 @elonmusk He explained that. He wants to see how the code effects his account personally"}
{"created_at":"2023-02-01T16:29:26.000Z","edit_history_tweet_ids":["1620821637305352192"],"id":"1620821637305352192","text":"@kucoincom It's OK to have your eggs in one basket as long as you control what happens to that basket” ~ @ElonMusk; Listen to him and invest in #TechTrees #TTC  $TTC"}
{"created_at":"2023-02-01T16:29:26.000Z","edit_history_tweet_ids":["1620821636370014210"],"id":"1620821636370014210","text":"@Hauwah_omar0 @elonmusk plz remove this kind of bitchsss"}
{"created_at":"2023-02-01T16:29:26.000Z","edit_history_tweet_ids":["1620821636332290049"],"id":"1620821636332290049","text":"RT @GermanosGhassan: @GermanosPeter @Twitter After acquiring twitter, Elon musk promised to block spams and fakes. Twitter for Lebanese pol…"}
{"created_at":"2023-02-01T16:29:25.000Z","edit_history_tweet_ids":["1620821635262464000"],"id":"1620821635262464000","text":"@elonmusk @TaraBull808 Don’t worry, twitter is dropping faster than a nuclear bomb."}
{"created_at":"2023-02-01T16:29:25.000Z","edit_history_tweet_ids":["1620821634511949828"],"id":"1620821634511949828","text":"@elonmusk @ElijahSchaffer AKA hard-coded."}
{"created_at":"2023-02-01T16:29:25.000Z","edit_history_tweet_ids":["1620821634436464640"],"id":"1620821634436464640","text":"RT @SidneyPowell1: Thank you @elonmusk \nI'm back!! Freedom of speech is crucial to our survival as the beacon of freedom for the world. I &…"}
{"created_at":"2023-02-01T16:29:25.000Z","edit_history_tweet_ids":["1620821632918118401"],"id":"1620821632918118401","text":"@elonmusk "}
{"created_at":"2023-02-01T16:29:25.000Z","edit_history_tweet_ids":["1620821632615940096"],"id":"1620821632615940096","text":"@libsoftiktok @elonmusk How do we change to private?"}