How can I use openssl to get results from HTTP GET requests?

Question

I need to use openssl to perform some HTTP GET requests in a shell script. The line I'm using to do this right now is shown below. This is parsing the content of an XML response of the following formats.

<Result>success</Result>

<Result>failure</Result>

echo -e "GET /test HTTP/1.1\r\nHost:$(hostname)\r\n\r\n" | openssl 2>&1 s_client -quiet -connect server-url:443 | grep -o -P --color '(?<=Result\>).*(?=\</Result)'

This works and returns the string 'success' or 'failure' accordingly. The problem I'm facing is that the openssl command does not terminate after doing the GET request but instead sits there waiting on more input. I believe this is due to the implicit -ign_eof which prevents automatic termination caused by the -quiet option. I've tried using the -no_ign_eof option but that causes the openssl command to terminate before the GET request has received a response so I can't get the content of the response if I use that.

How can I modify this command so I can pass the GET request through stdin (required as I want to put this in a loop) but have the openssl command terminate after each request?

Why are you trying to abuse `openssl` for this, rather than using something more appropriate like `curl`? — Michael Hampton, Oct 23 '15 at 16:18
Because curl isn't available on the system and I don't have permission to install it — conorgriffin, Oct 23 '15 at 16:18
Do yourself a favor, and go have a chat with the person who _does_ have permission to install it. — Michael Hampton, Oct 23 '15 at 16:19
I intend to but that doesn't help me get this workaround up and running... — conorgriffin, Oct 23 '15 at 16:20
If you can't get `CURL` there is `wget` you could abuse bit better for it you can use the info at http://stackoverflow.com/questions/12120935/wget-output-document-and-headers-to-stdout — Martin Barker, Oct 23 '15 at 16:30

score 6 · Accepted Answer · answered Oct 23 '15 at 16:43

6

What you really should be doing is using a tool designed for fetching Web resources, such as curl, wget, or libwww-perl's GET command. If nothing is available, you should have your system administrator install something appropriate.

With that out of the way...

The openssl command does not terminate because the web server didn't close the connection.

Remember that by default HTTP keeps connections open after each request as a performance optimization. Once one request finishes, another request can be sent over the same connection, rather than closing and reopening a new connection.

If you want to instruct the server to close the connection instead, you can send the Connection: close HTTP header.

answered Oct 23 '15 at 16:43

Michael Hampton

244,070
43
506
972

As mentioned in the comments on the question, unfortunately it is proving difficult to get open source tools installed on this AIX system and `openssl` seems to be the only tool available to me to do this right now. I've tried your suggestion and it works. Thanks – conorgriffin Oct 23 '15 at 16:48
2

As a _technical_ matter it's very easy to install the tools. I expect your roadblocks are entirely _political_ in nature. – Michael Hampton Oct 23 '15 at 16:49
yes they are, IBM even supply a list of open source tools on their website that includes curl http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/alpha.html – conorgriffin Oct 23 '15 at 16:50
2

Remember, and remind anyone who is in a position to listen, that workarounds like this ultimately _cost_ money. – Michael Hampton Oct 23 '15 at 16:51
Note: if you use POST, don't forget to include the request body (duh!) – Nickolay Aug 25 '19 at 15:14

Scott Centoni · Answer 2 · 2023-04-07T18:44:10.827

IBM uses Perl for bootstrapping downloads, and I would recommend using Perl until you can get approval to install curl properly, because it is bundled with the operating system. Specifically, their dnf_aixtoolbox.sh script uses lwp-download to download the rpm.rte package in lines like this:

    LDR_CNTRL=MAXDATA=0x80000000@DSA /usr/opt/perl5/bin/lwp-download https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/dnf_bundle_aix_73.tar

While you're not trying to download a file, documentation on the LWP library provides a number of different examples for different uses.

score 0 · Answer 3 · answered Oct 30 '18 at 16:59

0

Another simple (but probably worse) solution would be to use HTTP/1.0 instead of HTTP/1.1.

answered Oct 30 '18 at 16:59

JepZ

109
1

It might work, but new-ish http servers sometimes can't handle 1.0. – kubanczyk Oct 30 '18 at 17:04
1

@kubanczyk I have yet to come across an http server that cannot handle HTTP/1.0. A much more likely problem would be an http server that requires a `Host` header and a client that only sends the `Host` header when using HTTP/1.1. But ultimately the reason using HTTP/1.0 would help here is that it has different default behavior regarding closing the connection compared to HTTP/1.1. Adding a `Connection` header to the request would address that. – kasperd Oct 30 '18 at 17:59

score -1 · Answer 4 · edited Jul 19 '21 at 16:49

There is a new scenario that brings up several of the points discussed in this thread previously.
One is related to curl, the other related to tlsv1.3

First, I was using curl and found that curl will interpret a '#' as the start of a new URL. I have found others who have identified this same issue with curl using other special characters. In this case, the URL is truncated and curl fails.

Heading

The precise problem is using nonconforming characters to extend the URL name. Web developers are adding data after the ".html". My encounter is with the ---------.html#<date>.
As entered, curl will truncate the URL at the #. If I encode it -------.html%23<date>, then paste the encoded URL into the website via a browser, it encodes the encoded characters again with results in "no page found."

Web developers are attaching data to URLs so instead of sending a new webpage each time a user requests a table change (frequently a table) the webpage can be updated by sending just the table data via JSON and not the entire webpage.

I don't see getting curl changed anytime soon. Thus, using OpenSSL to bring up URLs is necessary if such features are to be tested. I was able to find an unsecure website that happens to use # in the server paths. I tested this using OpenSSL and it completed successfully.

echo -e "GET /#/Methods HTTP/1.1\r\nHost: eu.httpbin.org\r\nConnection: Close\r\n\r\n" | openssl s_client -quiet -state -connect eu.httpbin.org:443

Therefore, it can conclude that OpenSSL does not treat '#' as a special character in a URL. Curl parses input as HTML where '#' is a special character related to a HTML anchor.

The second issue, which is somewhat related to the initial topic, is using this same OpenSSL technique with tlsv1.3. There are no errors reported by OpenSSL, but it stops after the second "read R Block."

echo -e "GET / HTTP/1.1\r\nHost: cmegroup.com\r\nConnection: Close\r\n\r\n" | openssl 2>&1 s_client -CAfile firefoxCertBundle.pem -cert privClientCrt.pem -key privClient.key -tls1_3 -ciphersuites TLS_AES_256_GCM_SHA384 -quiet -state -connect cmegroup.com:443

    SSL_connect:before SSL initialization
    SSL_connect:SSLv3/TLS write client hello
    SSL_connect:SSLv3/TLS write client hello
    SSL_connect:SSLv3/TLS read server hello
    SSL_connect:TLSv1.3 read encrypted extensions
    depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global 
    Root CA
    verify return:1
    depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
    verify return:1
    depth=0 C = US, ST = Illinois, L = Chicago, O = Chicago Mercantile Exchange Inc, 
    CN = www.cmegroup.com
    verify return:1
    SSL_connect:SSLv3/TLS read server certificate
    SSL_connect:TLSv1.3 read server certificate verify
    SSL_connect:SSLv3/TLS read finished
    SSL_connect:SSLv3/TLS write change cipher spec
    SSL_connect:SSLv3/TLS write finished
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSLv3/TLS read server session ticket
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSLv3/TLS read server session ticket

The problem here is that `#` **is a special character**. Nothing after the # is intended to be sent to the server. It is interpreted client side only. It does not make sense to attempt to send it to the server. — Michael Hampton, Jul 11 '21 at 03:00
please check RFC3986, you'll find that [fragments (portion of URI that's after the #)](https://datatracker.ietf.org/doc/html/rfc3986#page-24) is dereferenced solely by the useragent (e.g. browser). second, try adding `Connection: keep-alive` and `Accept: text/html;q=0.8` on your `openssl s_client` request, you'll find the server replies with 301. — mforsetti, Jul 11 '21 at 07:49

How can I use openssl to get results from HTTP GET requests?

4 Answers4

Heading