How to get a list of website (url) cookies with Ruby

Question

I'd like to know if there's a clean way of getting a list of cookies that website (URL) uses?

Scenario: User writes down URL of his website, and Ruby on Rails application checks for all cookies that website uses and returns them. For now, let's think that's only one URL.

I've tried with these code snippets below, but I'm only getting back one or no cookies:

url = 'http://www.google.com'
r = HTTParty.get(url)
puts r.request.options[:headers].inspect
puts r.code

or

uri = URI('https://www.google.com')
res = Net::HTTP.get_response(uri)
puts "cookies: " + res.get_fields("set-cookie").inspect
puts res.request.options[:headers]["Cookie"].inspect

or with Mechanize gem:

agent = Mechanize.new
page = agent.get("http://www.google.com")
agent.cookies.each do |cooky| puts cooky.to_s end

It doesn't have to be strict Ruby code, just something I can add to Ruby on Rails application without too much hassle.

score 1 · Accepted Answer · answered May 31 '18 at 08:32

1

You should use Selenium-webdriver:
you'll be able to retrieve all the cookies for given website:

require "selenium-webdriver"


@driver = Selenium::WebDriver.for :firefox #assuming you're using firefox

@driver.get("https://www.google.com/search?q=ruby+get+cookies+from+website&ie=utf-8&oe=utf-8&client=firefox-b-ab")

@driver.manage.all_cookies.each do |cookie|
    puts cookie[:name]
end

answered May 31 '18 at 08:32

Gsk

2,929
5
22
29

Works like a charm! Please only add you need to install gecko driver after installing Selenium-webdriver. Thanks – JohnDoeTheOne May 31 '18 at 09:14

score 1 · Answer 2 · answered Oct 09 '20 at 06:57

 #cookie handling functions
    def add_cookie(name, value)
      @driver.manage.add_cookie(name: name, value: value)
    end

    def get_cookie(cookie_name)
      @driver.manage.cookie_named(cookie_name)
    end

    def get_all_cookies
      @driver.manage.all_cookies
    end

    def delete_cookie(cookie_name)
      @driver.manage.delete_cookie(cookie_name)
    end

    def delete_all_cookies
      @driver.manage.delete_all_cookies
    end

score 0 · Answer 3 · answered May 31 '18 at 08:30

0

With HTTParty you can do this:

puts HTTParty.get(url).headers["set-cookie"]

Get them as an array with:

puts HTTParty.get(url).headers["set-cookie"].split("; ")

answered May 31 '18 at 08:30

keoghpe

406
5
18

As stated above, this doesn't work well.. Not sure why not. – JohnDoeTheOne May 31 '18 at 09:01
1

@JohnDoeTheOne I don't see why you say the code I've given here doesn't work well. In your original question your checking the request headers - that's why you're not finding the cookies. In my answer I'm checking the headers of the response. Using selenium to do this is overkill since you're going to have to spin up a web browser every time you want to check the cookies. – keoghpe May 31 '18 at 09:59
Its true, but try it yourself and you will see that with your solution not all cookies are getting listed (like 3rd party cookies like Google Analytics), might be because it doesn't actually load the whole webpage with scripts, but it only get cookies upon request. I still can create background job with Sidekiq to run Selenium and then asnyc the result back once its ready.. but I need the whole thing :) – JohnDoeTheOne May 31 '18 at 10:36

How to get a list of website (url) cookies with Ruby

3 Answers3