Download file/folder from sharepoint using Curl/Wget automatically

Question

I have been trying to use Curl and wget to download file from Sharepoint. I am planning to make it as Script which runs automatically everyday and download the file from URL.

I tried using CURL with following command

curl -O --user Myusername:Mypassword https://OurDomain.sharepoint.com/_XXX&file=IPS_cleaned.xlsx&action=default

But it gave me error about SSL connection. I got to know that there is some existing bug in CURL 7.35 So i downgraded it to 7.22. But still gives me same error.

I also tried using Wget

wget --user=Myusername --password=MyPassword --no-check-certificate https://OurDomain.sharepoint.com/_XXX&file=IPS_cleaned.xlsx&action=default

But it still gives me error -- Unable to establish SSL connection

Can someone please let me know how i can accomplish my task

UPDATE

I was able to resolve the error in CURL. Below is the command that i gave

curl -O -L --sslv3 -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13" --user Myusername:Mypassword 'https://OurDomain.sharepoint.com/_%7BB21r-9CA2-345DEF%7D&file=IPS_cleaned.xlsx&action=default'

Now what it downloads is a file, which when i open it shows me Login page of Sharepoint. It does not download the actual excel file.

Any reason?

"But it gave me error about SSL connection" - you need to do better than that. What is the actual error returned? What do you get when you run `echo "GET / HTTP/1.1" | openssl s_client -connect :`? — jww, Jul 24 '14 at 12:39
@jww - I was able to resolve that error. But stuck at other location. Question updated — Neil, Jul 25 '14 at 02:43

score 21 · Answer 1 · answered Apr 02 '20 at 13:08

21

Another potential solution to this involves taking your sharepoint link and replacing the text after the '?' with download=1:

This: https://my.sharepoint.com/:u:/g/XXX/XXXX-bunchofRandomText?e=kRlVi

Becomes this: https://my.sharepoint.com/:u:/g/XXX/XXXX-bunchofRandomText?download=1

Now, you can just:

wget https://my.sharepoint.com/:u:/g/XXX/XXXX-bunchofRandomText?download=1

*Note, this example used a single file and a link where anyone with the link could access the file (no credentials required)

answered Apr 02 '20 at 13:08

LucasBoatwright

1,456
1
16
20

Thank you for this. For whatever reason, wget seems to not get the actual binary of the file without `?download=1` at the end. With it, my xlsx comes down and can be opened with openpyxl – Russ Schultz Jun 21 '22 at 15:01
3

I get error 403 Forbidden – jjrr Oct 19 '22 at 13:00
2

@jjrr The permissions for the link must allow anyone with the link to access the file – LucasBoatwright Oct 19 '22 at 15:19
Additionally, to the `download=1`, in my case, quotes (`'`) were the cure: `wget ''`. :) – imy Jan 26 '23 at 13:43

Sijo M Cyril · Answer 2 · 2021-03-17T14:34:32.093

Please use rclone

Download and install the latest one from https://rclone.org/downloads

First option: Use OneDrive to access SharePoint sites/personal folder. This option will help you to upload large files.

1.create rclone configurations using the rclone config command

2.Select New remote and give a name

3.Select cloud storage OneDrive

4.Leave client ID and secret as blank

5.Edit advanced config: n

6.Remote config: Use auto-config: y

7.Open the URL on the browser and give access to rclone

8.Select personal/shared site URL option

8a.Shared site URL option you have to give the site URL. ie; https://sharepoint.com/sites/SiteName

9.Select personal/Documents drive. Documents drive will show if you selected the shared site URL option in the 8th step

Save config and quit

And the configuration file contents will be like the following. If you selected the Personal option drive type will be personal.

[onedrive]
type = onedrive
token = 
drive_id = 
drive_type = documentLibrary

Second option: In this option, you can upload up to 2 GB-sized files.

1.create rclone configurations using rclone config command

2.Select New remote and give a name

3.Select cloud storage WebDAV

4.Give site URL, username and password

5.Save and quit

And the configuration file contents will be like the following. Password will be in an encrypted format.

vim /root/.config/rclone/rclone.conf

[sharepoint]
type = webdav
url = https://sharepoint.com/sites/SiteName/Documents
vendor = sharepoint
user = 
pass =

Download a file from SharePoint.

rclone copy --ignore-times --ignore-size --verbose sharepoint:SourceFolder/file.txt DestFolder

Thank you for this. A quick note is that during the interactive config one needs to set `webdav` as the type of `storage` (currently no. 34) and (after entering the URL to the sharepoint instance) one can set the Webdav service to `sharepoint`. — somada141, Nov 12 '20 at 16:58

Sijo M Cyril · Answer 3 · 2019-07-16T08:51:29.367

8

Firefox plugin that captures the link with session ID etc.. and it provides a command you could paste in the console for curl or wget.

If anyone has a better suggestion please let me know.

It gives you a curl or wget command with headers, cookies and all, with a copy to clipboard button, right on the download dialogue.

Download URL: https://addons.mozilla.org/en-US/firefox/addon/cliget

Reference: https://superuser.com/questions/27243/how-to-find-out-the-real-download-url-on-download-sites-that-use-redirects/1239026#1239026

edited Jul 16 '19 at 08:51

answered May 24 '19 at 09:13

Sijo M Cyril

660
1
8
14

1

This is the easiest way to download files from SharePoint or anywhere else using the terminal. Thanks man! – Vaibhav S May 31 '20 at 16:03

score 7 · Answer 4 · answered Mar 22 '15 at 11:06

Struggled with the same issue myself, and had my not-so-automatic-but-man-so-convenient way, with a daily log-in.

logged into Sharepoint with a browser,
exported the cookie,

run the following command.

wget --cookies=on --load-cookies cookies.txt --keep-session-cookies --no-check-certificate -m https://yoursharepoint.com

And files were downloaded just fine.

score 7 · Answer 5 · answered Dec 14 '20 at 13:13

For anyone using CURL to download a file on Sharepoint with an "Anyone with the link" download option. Below are the steps I had to follow to download. Essentially you have to use the cookie from the share link, and then download the file from a different download link they don't provide easily for you.

When sending the CURL command for the “share link” it returns a 302 message, a forward link, and a cookie. If we save that cookie and use it to hit a “download” link I am able to download the file. Essentially, Microsoft uses the initial “share link” to send the cookie to the browser, and then redirect to their “View File” website. On that website you need to use the cookie provided (authentication), and select your next function (On screen view, print, download, etc). When you click the download button you hit a different link. I was able to find this link by going to the "view page" website for the file/link, turning on developer tools, and watching the link the browser follows when hitting download. You can then replicate that link for each file. If we use that download link along with the cookie, we can download the file.

curl -i -c cookies.txt SHARE LINK

curl -o docsdownloaded.pdf -b cookies.txt DOWNLOAD LINK

Share Link Ex: https://tenant.sharepoint.com/:b:/s/Folder/EdNUf4xAVzFJgBoO0MqkfppR5tgobxLrmCnRqU4LFJQ?e=rOGNSD

Download Link Ex:https://tenant.sharepoint.com/sites/Folder/_layouts/15/download.aspx?SourceUrl=%2Fsites%2FFolder%2FShared%20Documents%2FGeneral%2FBig%2Dfile%2Epdf

Thanks this worked for me using curl to download an "Anyone with the link" on Google Colab. — Nick Fernandez, Jun 30 '22 at 12:51

Hugo Abreu · Answer 6 · 2022-05-29T03:54:31.017

4

Use wget adding &download=1 at the end of the link.

wget "<yourlink>&download=1"

it will be download with <yourlink> string as name, then just mv with the correct name after.

edited May 29 '22 at 03:54

answered Jul 13 '20 at 17:54

Hugo Abreu

109
1
7

1

Only works if the site administrator allows it. – Colin 't Hart Jun 19 '21 at 15:12

score 3 · Answer 7 · answered Mar 31 '16 at 19:57

Similar to the answer Zyglute gave, using cURL:

You can export your login cookie using the cookies.txt Chrome extension: https://chrome.google.com/webstore/detail/njabckikapfpffapmjgojcnbfjonfjfg

Then use the following code:

curl -b cookie.txt https://OurDomain.sharepoint.com/_XXX&file=IPS_cleaned.xlsx&action=default

At some point your Sharepoint session will expire (not sure how long that takes), and you will need a new cookie file.

EDIT: If a malicious user gets a hold of your cookie.txt, they could get into your SharePoint account, so be sure to keep it safe.

user1747935 · Answer 8 · 2023-07-24T08:38:29.303

My scenario: user shared a Sharepoint share link which required a password but not a login to download it Link looked like this https://testsite.sharepoint.com/:u:/s/folderloc/lOtsOfDiffErenTLetTers_tg?e=s0m3str1Ng

I tried a lot of different methods, finally using https://stackoverflow.com/a/37780143/1747935 this comment found a way to do it

Steps

In your browser, open the sharepoint link and enter the password so that the download link shows up and you're able to click the download link to start the download (you don't have to actually download it)

Open your developer tools (f12) and Open the network tab

In your browser click the download link to start the download

There will be a document type resource call that shows up in the form of download.aspx (you can cancel the download after it shows up)

Right click on the link and copy as Curl

Copy the text to a text editor and add --output <filename you're trying to download>

Execute the curl command on the box/server

It should run okay! I used checksum to make sure the file was correct (was able to download the file quickly to my local laptop)

Download file/folder from sharepoint using Curl/Wget automatically

8 Answers8

Linked