Downloading Xcode with wget or curl

Question

I am trying to download Xcode from the Apple Developer site using just wget or curl. I think I am successfully storing the cookie I need to download the .dmg file, but I am not completely sure.

When I run this command:

wget \  
   --post-data="theAccountName=USERNAME&theAccountPW=PASSWORD" \  
   --cookies=on \  
   --keep-session-cookies \  
   --save-cookies=cookies.txt \        
   -O - \  
   https://developer.apple.com/ios/download.action?path=/ios/ios_sdk_4.1__final/xcode_3.2.4_and_ios_sdk_4.1.dmg > /dev/null

A file called cookies.txt is created and contains something like this:

developer.apple.com FALSE / FALSE 0 XXXXXXXXXXXXXXXX  XXXXXXXXXXXX
developer.apple.com FALSE / FALSE 0 developer.sessionToken

I'm not completely certain, but I think there should be more to it than that (specifically, an alphanumeric string after sessionToken).

When I try to do the same thing with curl using this:

curl \  
   -d "theAccountName=USERNAME&theAccountPW=PASSWORD" \
   -c xcode-cookie \ 
   -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" \
   https://developer.apple.com/ios/download.action?path=/ios/ios_sdk_4.1__final/xcode_3.2.4_and_ios_sdk_4.1.dmg

I get a file called xcode-cookie that contains the same information as the cookies.txt file wget gives me, except that the lines are reversed.

I then tried to download the .dmg file.

Using wget:

wget \
   --cookies=on \
   --load-cookies=cookies.txt \
   --keep-session-cookies \
   http://developer.apple.com/ios/download.action?path=/ios/ios_sdk_4.1__final/xcode_3.2.4_and_ios_sdk_4.1.dmg

This gives me a file called login?appIdKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&path=%2F%2Fios%2Fdownload.action?path=%2Fios%2Fios_sdk_4.1__final%2Fxcode_3.2.4_and_ios_sdk_4.1.dmg , which is just an HTML page containing the login form for the developer site.

Using curl:

curl \
   -b xcode-cookie \
   -c xcode-cookie \
   -O -v \
   -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" \
   https://developer.apple.com/ios/download.action?path=/ios/ios_sdk_4.1__final/xcode_3.2.4_and_ios_sdk_4.1.dmg

Which prints basically the same thing as wget (minus the HTML).

I want to say it has to do with the sessionToken not being in the cookie, but like I said before I am not sure. I even tried exporting the cookies from my browser and following the instructions in the blog post I linked below and several other sites I found while searching for help.

I must be doing something wrong, unless Apple has changed something since Oct. 10 because this guy seems to be to do something right.

Thanks in advance!

Answer 1

For Chrome,

1. Install cookies.txt Chrome extension
2. Login to Apple Developer site and get the url for downloading
3. Run cookies.txt extension and download cookies.txt file

4. From the cookies.txt download directory, load cookies into wget and start resumable download. For example, to download Xcode_7.dmg, you would run:

   wget --load-cookies=cookies.txt -c http://adcdownload.apple.com/Developer_Tools/Xcode_7/Xcode_7.dmg

Answer 2

Maybe this is the easiest way to use curl:

• open Google Chrome.app;
• goto site developer.apple.com;
• press CMD+SHIFT+J or click top-right Menuicon -> Tools -> Developer Tools;
• click Network panel;
• now click Xcode download link at apple.com;
• you will see one or more request records in the Network panel;
• right click the latest record, then click Copy as cURL;

Now, you got the curl command for this download link with cookies and other http-requeset-fields, just paste to your terminal and add -o xxx.dmg at the end.

Answer 3

Here is a script that uses curl instead of wget, so it will work on a stock Mac. All you need to set is the path to the xcode DMG file. The script will ask you for your username and password and it will figure out the ACTION and WOSID values for you.

#!/bin/sh

# Change this line to the URI path of the xcode DMG file.
XCODE_PATH="/ios/ios_sdk_4.2__final/xcode_3.2.5_and_ios_sdk_4.2_final.dmg"

echo "Enter your Apple Dev Center username."
read -p "> " USERNAME
echo "Enter your Apple Dev Center password."
read -p "> " PASSWORD

curl \
        -L -s -k \
        --cookie-jar cookies \
        -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5" \
        https://developer.apple.com/devcenter/ios/login.action \
        -o login.html

ACTION=$(sed -n 's/.*action="\(.*\)".*/\1/p' login.html)
WOSID=$(sed -n 's/.*wosid" value="\(.*\)".*/\1/p' login.html)
echo "action=${ACTION}"
echo "wosid=${WOSID}"

curl \
        -s -k --cookie-jar cookies --cookie cookies \
        -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5" \
        -e ";auto" "https://daw.apple.com${ACTION}?theAccountName=${USERNAME}&theAccountPW=${PASSWORD}&theAuxValue=&wosid=${WOSID}&1.Continue.x=0&1.Continue.y=0" \
        > /dev/null

curl \
        -L --cookie-jar cookies --cookie cookies \
        -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5" \
        -O https://developer.apple.com/ios/download.action?path=${XCODE_PATH}

rm login.html
rm cookies

Answer 4

I've just been down the same road after failing repeatedly to download the latest version of Xcode directly to my Mac dev machine. I also couldn't get Daniel's approach to work exactly as written.

This is what did work for me:

1. Used Firefox to log in to the iOS dev center

2. Used Cookie Exporter 1.5 to save the cookies to a text file, say "cookies.txt"

3. FTP'd cookies.txt to a staging server of mine that has a nice fat pipe onto the Internet

4. Used wget from the staging server to download the file (about 5 minutes)

5. Used FTP to transfer the DMG file back to my dev machine (about 2 hours)

Here is the wget command line:

wget --cookies=on --load-cookies=cookies.txt --keep-session-cookies --save-cookies=cookies.txt http://adcdownload.apple.com/ios/ios_sdk_4.1__final/xcode_3.2.4_and_ios_sdk_4.1.dmg

The trick here is to NOT use the download link from the page, but rather to start the download then use the context menu in the download list to "Copy Download Link".

Answer 5

Not optimal but works for me:

If you know direct URL for download (e.g. you started download) then you can log-in on developer portal and run the following JS in console:

copy('wget -c http://adcdownload.apple.com//Developer_Tools/xcode_6_beta_5_za4gu6/xcode_6_beta_5.dmg --header="Cookie:' + document.cookie + '"')

This will copy wget command in clipboard with all necessary cookies for wget to work. You can restart this command anytime to continue.

Answer 6

The answer by @petert used to work for me in the past, but now it shows the error 403 (Forbidden).

I ended up using the app Downloader-for-Apple-Developers, which is open source and has a graphic interface, to login to the Apple's developer website and download the Xcode .xip file.

This is how the graphical interface looks like: The lates version can be downloaded from here: https://github.com/vineetchoudhary/Downloader-for-Apple-Developers/releases

Answer 7

So I've seemed to figure out the answer to my own question. Here's how you can download Xcode using curl.

First, run this:

curl \
-L -s -k \
--cookie-jar cookies \
-A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5" \
https://developer.apple.com/devcenter/ios/login.action \
-o login.html

Open the login.html file that's created and look for these 2 things:

• The action attribute of the login form. It should be on/around line 54.
• The input field named wosid. This should be on/around line 129.

Copy the value of the action attribute as well as the value of the value attribute of the input field. You'll need these for the next step.

Here's the next curl command:

curl \
-s -k --cookie-jar cookies --cookie cookies \
-A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5" \
-e ";auto" "https://daw.apple.com{ACTION}?theAccountName={USERNAME}&theAccountPW={PASSWORD}&theAuxValue=&wosid={WOSID}&1.Continue.x=0&1.Continue.y=0" \
> /dev/null

• Replace {ACTION} with the action value you saved
• Replace {USERNAME} and {PASSWORD} with your Apple Dev Center credentials
• Replace {WOSID} with the wosid value you saved
• Run the command

You should now have the cookie that will allow you to download the .dmg file. If you haven't noticed by now, the cookie is stored in a file called cookies.

Now just run this last curl command and the Xcode image should begin downloading:

curl \
-L --cookie-jar cookies --cookie cookies \
-A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5" \
-O https://developer.apple.com/ios/download.action?path=/ios/ios_sdk_4.1__final/xcode_3.2.4_and_ios_sdk_4.1.dmg

I've tried this on 2 different machines and works on both.

Answer 8

Here's a little bash script to wget Xcode :

#!/bin/bash

export ID=YourAppleID
export PW=YourPW

[ -f cookies ] && rm cookies && touch cookies

2>Header \
wget \
-S \
-O R1 \
http://developer.apple.com/ios/download.action?path=/ios/ios_sdk_4.2__final/xcode_3.2.5_and_ios_sdk_4.2_final.dmg

tac Header | grep Location
LOCATION=$(grep Location Header | sed -E 's/^ *Location: ([^/]+:\/\/[^/]+)\/.*$/\1/')
[ -z "$LOCATION" ] && { echo "Bad day for LOCATION...";exit;} || echo "LOCATION=$LOCATION"
rm Header

ACTION=$(grep action R1 | sed 's/^.*action="//;s/".*$//')
[ -z "$ACTION" ] && { echo "Bad day for ACTION...";exit;} || echo "ACTION=$ACTION"

POST=$( grep input R1 | sed 's/<input/\
<input/g' | grep input | sed 's/^.*name="//' | sed 's/".*value="/=/;s/".*$//' | sed '/=/!s/$/=/' | sed '/theAccountName/s/$/'$ID'/;/theAccountPW/s/$/'$PW'/' | sed '/=$/d' | sed -n '1h;1!H;${x;s/[[:space:]]/\&/g;p;}' | sed  's/$/\&1.Continue.x=0\&1.Continue.y=0/')
[ -z "$POST" ] && { echo "Bad day for POST...";exit;} || echo "POST=$POST"

2>Header \
wget \
-S \
--save-cookies cookies \
--keep-session-cookies \
-O R2 \
--post-data="$POST" \
$LOCATION/$ACTION

URL=$( grep -i REFRESH R2 | sed 's/^.*URL=//;s/".*$//' )
[ -z "$URL" ] && { echo "Bad day for URL...";exit;} || echo "URL=$URL"

wget \
-S \
--load-cookies cookies \
$URL &

sleep 1; rm R1 R2 Header cookies

Comments are the same as for joe's Solution - Thnx Joe ;o) The good idea is to start the traffic analysis from the download url of Xcode.