What is the proper way to escape a URL for URLConnection.getInputStream()?

Question

I'm having a really bizarre problem with URLConnection.getInputStream() when I have a space (' ') in the query string portion of a URI. Specifically, I have one URL that works and another that does not, when I think they should both fail or both succeed, additionally, its every time.

Working URL: http://minneapolis.craigslist.ca/search/sss?catAbb=sss&query=iPhone+sprint&sort=date&srchType=A&format=rss Failed URL (exception below) : http://winnipeg.craigslist.ca/search/sss?catAbb=sss&query=iPhone+sprint&sort=date&srchType=A&format=rss

conn.getInputStream() throws the IO exception: "Illegal character in query at index 67: http://winnipeg.en.craigslist.ca/search/sss?catAbb=sss&query=iPhone sprint two&sort=date&srchType=A&format=rss"

It appears openConnection can't get the space (which I've already replaced with a '+' as I'd expect to have to with a 'URL', I've also tried '%20' with the same results.

Additionally, URL.toString() reports the URLS as I printed above, with the '+' not the space.

Code is as follows, searchUrl is a 'URL' instance.

        URLConnection conn = null;
        conn = searchUrl.openConnection();
        conn.setConnectTimeout(CONNECT_TIMEOUT);
        conn.setUseCaches(true);
        conn.setAllowUserInteraction(false);

        ByteArrayOutputStream oStream = new ByteArrayOutputStream();
        InputStream istream = conn.getInputStream();
        int numBytesRead, numBytesWritten = 0;
        byte[] buffer = new byte[8 * 1024];
        while ((numBytesRead = istream.read(buffer, 0, 8 * 1024)) > 0) {
            oStream.write(buffer, numBytesWritten, numBytesRead);
            numBytesWritten += numBytesRead;
        }

Any ideas on where to deal with this? I'm about to pitch URLConnection and go another route...

Thanks Kenny.

I think you have made a mistake in your question. The URLs both search for "iPhone sprint" but the error message says that the URL is "iPhone sprint two". Please correct the question with the **real** URLs and the **real** error message. — Stephen C, Oct 01 '12 at 05:37
The problem was hidden in that the " " was in the redirect URL NOT in the one that I originally encoded. I intercepted the redirect and encoded the parameters in the result. So essentially the problem was with the web site I was accessing creating a redirect that wasn't correct. — Kenny, Oct 24 '12 at 14:30

score 1 · Answer 1 · answered Oct 01 '12 at 05:51

There is something wrong with your question (see my comment).

However, the fundamental problem here is that a URL with a space character in the query part is not a legal URL ... not withstanding that a typical web browser will accept it. The exception is therefore correct.

Your example URLs seem to show that the space is escaped with a '+'. This is HTML form escaping not proper URL escaping. You seem to be saying that you get the same result is you use %20 ... which would be correct escaping.

So my theory is that you are actually passing this URL to your code via a route that is removing the escapes ... not-withstanding what your traceprints seem to be telling you. (If I could see an SSCE we'd be able to test this theory ...)

FWIW, fixing the problem by calling UrlEncoder.encode as some of the other answers have suggested is a bad idea. The problem is that it is likely to "encode" other characters that shouldn't be encoded.

score 0 · Answer 2 · answered Oct 01 '12 at 08:39

The URL itself is best encoded with new URI(null, url, null).toASCIIString().

Each key and value in the query string can be separately encoded with URLEncoder.encode(). According to RFC 2936 this isn't correct and the whole thing should be encoded as for the URL itself, but I've never seen it fail.

score -1 · Answer 3 · answered Oct 01 '12 at 05:11

-1

Did you try URLEncoder.encode(string, "UTF-8")

Following is the example:

Replace

String url = "http://somesite.com/page?user=" + user;

with

String url = "http://somesite.com/page?user="
+ URLEncoder.encode(user, "UTF-8");

answered Oct 01 '12 at 05:11

Anshu

7,783
5
31
41

janenz00 · Answer 4 · 2012-10-01T05:50:53.110

-1

String url= URLEncoder.encode("your URL without http or your query string part here");
URL searchUrl = new URL("http://" + url);
URLConnection conn = null;
conn = searchUrl.openConnection();

edited Oct 01 '12 at 05:50

answered Oct 01 '12 at 05:41

janenz00

3,315
5
28
37

What is the proper way to escape a URL for URLConnection.getInputStream()?

4 Answers4