Curl works in insomnia but does not work in Postman or Mac Terminal

Question

I am trying to execute the following curl in Postman, Insomnia and Local Mac Terminal. It works with Insomnia but does not work with Postman and my local terminal.

curl --request GET \
  --url 'https://www.lowes.com/rnr/r/get-by-product/5013699701/pdp/prod?offset=10' \
  --header 'accept: application/json, text/plain, */*' \
  --header 'accept-language: en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4' \
  --header 'authority: www.lowes.com' \
  --header 'dnt: 1' \
  --header 'referer: https://www.lowes.com/pd/Whirlpool-21-4-cu-ft-Side-By-Side-Refrigerator-with-Ice-and-Water-Dispenser-and-Can-Caddy-Fingerprint-Resistant-Stainless-Steel/5013699701' \
  --header 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \
  --header 'sec-ch-ua-mobile: ?0' \
  --header 'sec-ch-ua-platform: "macOS"' \
  --header 'sec-fetch-dest: empty' \
  --header 'sec-fetch-mode: cors' \
  --header 'sec-fetch-site: same-origin' \
  --header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'

What might be the issue?

With Postman and Local Mac terminal, I am getting the following output.

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;lowes&#46;com&#47;rnr&#47;r&#47;get&#45;by&#45;product&#47;5013699701&#47;pdp&#47;prod&#63;" on this server.<P>
Reference&#32;&#35;18&#46;b56e3f17&#46;1679172086&#46;19ace9d1
</BODY>
</HTML>

In insomnia, I am getting a valid json as response.

Also why does the following python code not work

import requests

url = "https://www.lowes.com/rnr/r/get-by-product/5013699701/pdp/prod?offset=10"

payload={}
headers = {
  'accept': 'application/json, text/plain, */*',
  'accept-language': 'en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4',
  'authority': 'www.lowes.com',
  'dnt': '1',
  'referer': 'https://www.lowes.com/pd/Whirlpool-21-4-cu-ft-Side-By-Side-Refrigerator-with-Ice-and-Water-Dispenser-and-Can-Caddy-Fingerprint-Resistant-Stainless-Steel/5013699701',
  'sec-ch-ua': '"Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"macOS"',
  'sec-fetch-dest': 'empty',
  'sec-fetch-mode': 'cors',
  'sec-fetch-site': 'same-origin',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

HELP!

Tried everything that i could

No. not authorization issue. otherwise why would it work insomnia. — user8087010, Mar 19 '23 at 11:04
How does insomnia do authorization? How can i do the same in python? — user8087010, Mar 19 '23 at 12:10

score 0 · Answer 1 · answered Mar 18 '23 at 21:10

It worked for me using requests

Code:

import requests
import json

url = "https://www.lowes.com/rnr/r/get-by-product/5013699701/pdp/prod?offset=10"

my_headers = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
    "Connection": "keep-alive",
    "DNT":  "1",
    "Host": "www.lowes.com",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent":   "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0"

}

r = requests.get(url, headers=my_headers)
json_obj = r.json()
print(json.dumps(json_obj, indent=4))

Output Sample (as its too large for post:

{
    "Limit": 10,
    "Offset": 10,
    "TotalResults": 8484,
    "Locale": "en_US",
    "Results": [
        {
            "Id": "249181136",
            "SourceClient": "whirlpoolbrand",
            "ProductId": "5013699701",
            "ContentLocale": "en_US",
            "IsFeatured": false,
            "TotalInappropriateFeedbackCount": 0,
            "IsSyndicated": true,
            "SyndicationSource": {
                "ClientId": null,
                "LogoImageUrl": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86YXR0cmlidXRpb25sb2dvMg/fbc9a2c4-83a6-4d7f-9be3-bd2c30a88e41",
                "ContentLink": "https://www.whirlpool.ca/fr_ca/kitchen/refrigeration/refrigerators/side-by-side/p.33-inch-wide-side-by-side-refrigerator-21-cu.-ft.wrs321sdhz.html",
                "Name": "whirlpool.com"
            },
            "TotalClientResponseCount": 0,
            "TotalCommentCount": 0,
            "Photos": [
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/70482ed3-61c0-56a0-876e-24ed59f0f6f7"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/70e5b4c2-e6a4-50f9-ba62-efcf3097ab72"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/460f2b76-b7da-55d7-a23c-f36ab7a718bf"
                        }
                    },
                    "Id": "7195283",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": "Shipping damage"
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/1ad532bf-3cfa-5194-8c67-d58599631221"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/a6e71d93-463f-5bb2-b64e-f7edcc3fb7ee"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/5233ed7f-b03d-5c32-8bfa-17664a7fbcc9"
                        }
                    },
                    "Id": "7195284",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": "Non adjustable shelves"
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/06131e04-dee6-59df-9f10-91d687d36e9d"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/c12ea116-5382-5f51-8101-6ac7700b5acf"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/4e274ab7-a0fb-5e27-b661-b729922a6dc9"
                        }
                    },
                    "Id": "7195285",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": "Under cabinet perfect fit for my house"
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/70bb2765-61d2-5926-9854-0f49ce605bdc"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/e3306817-8971-5694-9c43-1f9afe787e1f"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/26a3cbc1-8bf4-5522-a9e0-9ec76972b1b5"
                        }
                    },
                    "Id": "7195286",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": "In door milk stowage"
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/32d22fd9-1aa0-55d5-b1b3-3ca24d0d8cce"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/b87d6764-da35-5a4e-a3a7-3ed8ef60c790"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/77f22ccf-a6fa-5cfd-b07a-41297bb888df"
                        }
                    },
                    "Id": "7195287",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": "Narrow width freezer side"
                }
            ],
            "Rating": 3,
            "IsRatingsOnly": false,
            "IsRecommended": false,
            "SubmissionTime": "2021-08-24T13:58:23.000+00:00",
            "ReviewText": "[This review was collected as part of a promotion.] Bought it from lowes about 3 weeks ago out of neccessity. It was dropped in the wearhouse. I found a ding inthe sheetmetal and when h20 was hooked up  the icemaker did not work i am currenly awaiting new part from whirlpool via warrenty work ticket. Not recommend for a large family for the shelves are not adjustable and will not hold a gallon size of milk. Freezer capaicity is limited will not hold frozen flat without tipping vertical.",
            "Title": "Right size below cabinets",
            "UserNickname": "Rejhron66",
            "ClientResponses": [],
            "TotalPositiveFeedbackCount": 0,
            "SecondaryRatings": {},
            "TotalFeedbackCount": 0,
            "RatingRange": 5,
            "CampaignId": null,
            "TotalNegativeFeedbackCount": 0,
            "ContextDataValues": {},
            "Videos": [],
            "SecondaryRatingsOrder": [],
            "Helpfulness": null,
            "Badges": {},
            "IsLowesLoopBadge": false,
            "IsRecievedProductForFree": false,
            "IsExpertReviewer": false
        },
        {
            "Id": "249654294",
            "SourceClient": "whirlpoolbrand",
            "ProductId": "5013699701",
            "ContentLocale": "en_US",
            "IsFeatured": false,
            "TotalInappropriateFeedbackCount": 0,
            "IsSyndicated": true,
            "SyndicationSource": {
                "ClientId": null,
                "LogoImageUrl": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86YXR0cmlidXRpb25sb2dvMg/fbc9a2c4-83a6-4d7f-9be3-bd2c30a88e41",
                "ContentLink": "https://www.whirlpool.ca/fr_ca/kitchen/refrigeration/refrigerators/side-by-side/p.33-inch-wide-side-by-side-refrigerator-21-cu.-ft.wrs321sdhz.html",
                "Name": "whirlpool.com"
            },
            "TotalClientResponseCount": 0,
            "TotalCommentCount": 0,
            "Photos": [
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/5c51935f-3644-528d-adf3-95f5e18979a0"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/91f9222a-1f73-5a7d-9fad-ae5c96845d66"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/57ca9753-771d-584c-92d5-6faa1f0d7c65"
                        }
                    },
                    "Id": "7237985",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": null
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/7d7e890f-3071-54c8-9710-e0d6706ab4b7"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/7f433902-d722-56c9-aace-7f4098dc6d85"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/e10cd3f3-ddad-56f9-a20b-038ec2f5b13b"
                        }
                    },
                    "Id": "7237986",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": null
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/ed0b6b76-65a6-5ca7-b284-a13d188e7aa6"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/8cec022a-ebbd-596d-8489-f3a9fdb80342"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/ac340a0b-b146-548e-a4b9-d68fddafb003"
                        }
                    },
                    "Id": "7237987",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": null
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/702685e9-3097-5229-b300-865434d1dfda"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/f50efd4c-4d71-5c67-ae4c-8c93b4228685"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/0d47e61e-8dbf-52ce-8ac4-7c1899ac909b"
                        }
                    },
                    "Id": "7237988",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": null
                },
                {
                    "Sizes": {
                        "normal": {
                            "Id": "normal",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/29fe5b00-9c11-5807-b34b-13f536e11f81"
                        },
                        "thumbnail": {
                            "Id": "thumbnail",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/4480aad8-afb8-56a5-ba7c-e0bed023f754"
                        },
                        "large": {
                            "Id": "large",
                            "Url": "https://photos-us.bazaarvoice.com/photo/2/cGhvdG86d2hpcmxwb29sYnJhbmQ/69c8a37c-ef1b-57df-8d07-1df0c73373f1"
                        }
                    },
                    "Id": "7237989",
                    "SizesOrder": [
                        "normal",
                        "thumbnail",
                        "large"
                    ],
                    "Caption": null
                }
            ],
            "Rating": 4,
            "IsRatingsOnly": false,
            "IsRecommended": true,
            "SubmissionTime": "2021-09-11T17:56:38.000+00:00",
            "ReviewText": "[This review was collected as part of a promotion.] This is a nice size fridge for a couple or a family of 3. The shelves in the fridge are not moveable, but the ones in the freezer are. It gets really cold. I only have an issue with the loud humming noise it make. If you can ignore the humming it's a great fridge.",
            "Title": "Love the stainless steel",
            "UserNickname": "Panzie",
            "ClientResponses": [],
            "TotalPositiveFeedbackCount": 0,
            "SecondaryRatings": {},
            "TotalFeedbackCount": 0,
            "RatingRange": 5,
            "CampaignId": null,
            "TotalNegativeFeedbackCount": 0,
            "ContextDataValues": {},
            "Videos": [],
            "SecondaryRatingsOrder": [],
            "Helpfulness": null,
            "Badges": {},
            "IsLowesLoopBadge": false,
            "IsRecievedProductForFree": false,
            "IsExpertReviewer": false
        }

Your python code did not work for me. Strange! Getting 403 status code — user8087010, Mar 19 '23 at 11:00
I have posted a similar question. Can you please answer that too? https://stackoverflow.com/questions/75784328/why-does-the-following-curl-works-but-equivalent-python-code-does-not Thanks! — user8087010, Mar 19 '23 at 19:31
is the python code really working you? We need to see why is it not working for me. — user8087010, Mar 19 '23 at 20:12
Perhaps [this](https://stackoverflow.com/a/70315217/13684789) is worth a try; that is, restarting your router/modem. — Übermensch, Mar 19 '23 at 20:33

Curl works in insomnia but does not work in Postman or Mac Terminal

1 Answers1

Linked