1

I'm trying to upload a public video file from Amazon S3 to Dailymotion using a NodeJS Lambda function. AWS Lambda functions have a storage limitation: I can't download the whole video file locally on the serverless function and then send it to Dailymotion.

I searched for a way to read the file on parts and send them with HTTPS requests to Dailymotion, and I found the urllib library that seems helpful!

To upload a video file to Dailymotion, you can send a POST request to an UPLOAD_URL that should be generated using another authenticated request:

curl -X POST \
     -F 'file=@/path/to/your/video.mp4' \
     '<UPLOAD_URL>'

The steps are clearly mentioned on this document page.

What I need is a way to stream the file from a URL like this https://cdn.example.com/videos/filename.mp4 to Dailymotion through my Lambda function.

Anyone with good experience can help with a clear example of how to do it.

Jafar Abazeed
  • 142
  • 2
  • 11
  • 2
    FYI: [AWS Lambda Now Supports Up to 10 GB Ephemeral Storage | AWS News Blog](https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage) – John Rotenstein Sep 02 '22 at 11:04
  • Video files have different sizes and most of them are less than `512MB`. Reserving extra unnecessary storage size will lead to an increment in the cost that you can avoid. That's why it's better to use the Lambda function as a middle mean to read and upload the file in chunks. Thank you @JohnRotenstein! – Jafar Abazeed Sep 02 '22 at 11:18
  • I found this excellent article which proposes a good solution to allocate "semi-dynamic" ephemeral storage for Lambda functions: [Serverless Compute with Dynamic Ephemeral Storage](https://medium.com/storyboard-podcasts/storyboard-dev-blog-serverless-compute-with-dynamic-ephemeral-storage-8fd38c0df93d), and it comes with a fully deployable example on [GitHub](https://github.com/Storyboard-fm/lambda-dynamic-ephemeral-storage-example). – Jafar Abazeed Sep 08 '22 at 05:42

1 Answers1

0

I found a solution using request and form-data libraries. Just:

  1. Created a new form.
  2. Appended the file to the form as a stream using request.
  3. Submit the form.
  4. Wait for the upload to end to send the callback back to the caller with the response.

Here's a working copy of my Lambda function code:

import request from 'request';
import FormData from 'form-data';

export const handler = (event, context, callback) => {
  const form = new FormData();
  form.append('file', request('https://cdn.mysite.com/video.mp4'));
  form.submit({
    host: 'upload-XX.dcX.dailymotion.com',
    path: '/upload?uuid=XXXXX&seal=XXXXX&extra=XXXXX'
  }, (error, response) => {
    if (error) throw error;
    let body = '';
    response.on('data', chunk => {body += chunk.toString()});
    response.on('end', () => {callback(null, JSON.parse(body))});
    response.resume();
  });
};

Update

Another solution using the https native module of Nodejs instead of the deprecated request library:

import https from 'https';
import FormData from 'form-data';

export const handler = (event, context, callback) => {
  const form = new FormData();
  https.get('https://cdn.mysite.com/video.mp4', response => {
    form.append('file', response);
    form.submit(
      {
        host: 'upload-XX.dcX.dailymotion.com',
        path: '/upload?uuid=XXXXX&seal=XXXXX&extra=XXXXX'
      },
      (error, response) => {
        if (error) callback(error);
        let body = '';
        response.on('data', chunk => body += chunk.toString());
        response.on('end', () => callback(null, JSON.parse(body)));
        response.resume();
      }
    );
  });
};
Jafar Abazeed
  • 142
  • 2
  • 11