Can My Users Of Website Upload Video To S3 Bucket

In web and mobile applications, it's common to provide users with the ability to upload data. Your awarding may allow users to upload PDFs and documents, or media such as photos or videos. Every modern web server technology has mechanisms to allow this functionality. Typically, in the server-based environment, the procedure follows this menstruation:

Application server upload process

The user uploads the file to the application server.
The application server saves the upload to a temporary space for processing.
The application transfers the file to a database, file server, or object store for persistent storage.

While the process is simple, it can have significant side-effects on the functioning of the web-server in busier applications. Media uploads are typically large, so transferring these can represent a large share of network I/O and server CPU time. Yous must also manage the state of the transfer to ensure that the unabridged object is successfully uploaded, and manage retries and errors.

This is challenging for applications with spiky traffic patterns. For example, in a web application that specializes in sending vacation greetings, it may feel nearly traffic only effectually holidays. If thousands of users try to upload media around the aforementioned fourth dimension, this requires you to scale out the application server and ensure that in that location is sufficient network bandwidth available.

By directly uploading these files to Amazon S3, yous tin avoid proxying these requests through your application server. This can significantly reduce network traffic and server CPU usage, and enable your application server to handle other requests during busy periods. S3 as well is highly available and durable, making it an platonic persistent shop for user uploads.

In this blog post, I walk through how to implement serverless uploads and show the benefits of this approach. This pattern is used in the Happy Path spider web application. Yous can download the code from this web log post in this GitHub repo.

Overview of serverless uploading to S3

When you lot upload directly to an S3 saucepan, you must offset request a signed URL from the Amazon S3 service. Y'all can then upload straight using the signed URL. This is two-footstep process for your awarding front end:

Serverless uploading to S3

Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda function. This gets a signed URL from the S3 bucket.
Directly upload the file from the application to the S3 bucket.

To deploy the S3 uploader example in your AWS account:

Navigate to the S3 uploader repo and install the prerequisites listed in the README.doc.
In a terminal window, run:
git clone https://github.com/aws-samples/amazon-s3-presigned-urls-aws-sam
cd amazon-s3-presigned-urls-aws-sam
sam deploy --guided
At the prompts, enter s3uploader for Stack Name and select your preferred Region. Once the deployment is complete, note the APIendpoint output.The API endpoint value is the base URL. The upload URL is the API endpoint with /uploads appended. For example: https://ab123345677.execute-api.us-west-ii.amazonaws.com/uploads.

CloudFormation stack outputs

Testing the application

I show two ways to test this application. The first is with Postman, which allows you to directly phone call the API and upload a binary file with the signed URL. The second is with a basic frontend application that demonstrates how to integrate the API.

To test using Postman:

First, re-create the API endpoint from the output of the deployment.
In the Postman interface, paste the API endpoint into the box labeled Enter request URL.
Choose Send.
After the request is complete, the Body section shows a JSON response. The uploadURL attribute contains the signed URL. Re-create this attribute to the clipboard.
Select the + icon adjacent to the tabs to create a new request.
Using the dropdown, modify the method from Get to PUT. Paste the URL into the Enter asking URL box.
Choose the Trunk tab, then the binary radio button.
Choose Select file and cull a JPG file to upload.
Cull Send. You see a 200 OK response after the file is uploaded.
Navigate to the S3 console, and open up the S3 bucket created by the deployment. In the bucket, y'all see the JPG file uploaded via Postman.

To examination with the sample frontend application:

Copy index.html from the example's repo to an S3 bucket.
Update the object's permissions to brand information technology publicly readable.
In a browser, navigate to the public URL of index.html file.
Select Choose file then select a JPG file to upload in the file picker. Cull Upload paradigm. When the upload completes, a confirmation message is displayed.
Navigate to the S3 panel, and open the S3 bucket created past the deployment. In the bucket, you see the second JPG file yous uploaded from the browser.

Understanding the S3 uploading process

When uploading objects to S3 from a spider web application, yous must configure S3 for Cantankerous-Origin Resource Sharing (CORS). CORS rules are defined equally an XML document on the saucepan. Using AWS SAM, you can configure CORS as part of the resource definition in the AWS SAM template:

                      S3UploadBucket:     Type: AWS::S3::Bucket     Properties:       CorsConfiguration:         CorsRules:         - AllowedHeaders:             - "*"           AllowedMethods:             - GET             - PUT             - HEAD           AllowedOrigins:             - "*"

The preceding policy allows all headers and origins – it's recommended that you lot use a more restrictive policy for production workloads.

In the first step of the process, the API endpoint invokes the Lambda office to make the signed URL request. The Lambda function contains the post-obit code:

          const AWS = crave('aws-sdk') AWS.config.update({ region: process.env.AWS_REGION }) const s3 = new AWS.S3() const URL_EXPIRATION_SECONDS = 300  // Master Lambda entry signal exports.handler = async (event) => {   render look getUploadURL(event) }  const getUploadURL = async function(event) {   const randomID = parseInt(Math.random() * 10000000)   const Primal = `${randomID}.jpg`    // Get signed URL from S3   const s3Params = {     Bucket: process.env.UploadBucket,     Key,     Expires: URL_EXPIRATION_SECONDS,     ContentType: 'epitome/jpeg'   }   const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)   return JSON.stringify({     uploadURL: uploadURL,     Cardinal   }) }

This function determines the proper name, or key, of the uploaded object, using a random number. The s3Params object defines the accepted content type and also specifies the expiration of the fundamental. In this example, the key is valid for 300 seconds. The signed URL is returned as part of a JSON object including the key for the calling application.

The signed URL contains a security token with permissions to upload this single object to this bucket. To successfully generate this token, the lawmaking calling getSignedUrlPromise must have s3:putObject permissions for the saucepan. This Lambda function is granted the S3WritePolicy policy to the bucket by the AWS SAM template.

The uploaded object must match the aforementioned file proper name and content type equally defined in the parameters. An object matching the parameters may be uploaded multiple times, providing that the upload procedure starts before the token expires. The default expiration is fifteen minutes but yous may want to specify shorter expirations depending upon your use instance.

In one case the frontend application receives the API endpoint response, it has the signed URL. The frontend application so uses the PUT method to upload binary data directly to the signed URL:

          permit blobData = new Blob([new Uint8Array(assortment)], {type: 'image/jpeg'}) const result = look fetch(signedURL, {   method: 'PUT',   body: blobData })

At this point, the caller application is interacting directly with the S3 service and not with your API endpoint or Lambda function. S3 returns a 200 HTML status code one time the upload is complete.

For applications expecting a large number of user uploads, this provides a simple way to offload a large amount of network traffic to S3, away from your backend infrastructure.

Adding hallmark to the upload process

The electric current API endpoint is open, available to any service on the internet. This ways that anyone can upload a JPG file once they receive the signed URL. In most production systems, developers want to use authentication to command who has access to the API, and who can upload files to your S3 buckets.

You can restrict access to this API by using an authorizer. This sample uses HTTP APIs, which support JWT authorizers. This allows you to command access to the API via an identity provider, which could be a service such equally Amazon Cognito or Auth0.

The Happy Path awarding only allows signed-in users to upload files, using Auth0 as the identity provider. The sample repo contains a 2d AWS SAM template, templateWithAuth.yaml, which shows how you can add an authorizer to the API:

                      MyApi:     Type: AWS::Serverless::HttpApi     Properties:       Auth:         Authorizers:           MyAuthorizer:             JwtConfiguration:               issuer: !Ref Auth0issuer               audience:                 - https://auth0-jwt-authorizer             IdentitySource: "$asking.header.Authorization"         DefaultAuthorizer: MyAuthorizer

Both the issuer and audience attributes are provided by the Auth0 configuration. By specifying this authorizer as the default authorizer, information technology is used automatically for all routes using this API. Read part i of the Inquire Around Me serial to learn more than well-nigh configuring Auth0 and authorizers with HTTP APIs.

After hallmark is added, the calling web awarding provides a JWT token in the headers of the request:

          const response = await axios.get(API_ENDPOINT_URL, {   headers: {     Authorisation: `Bearer ${token}`         } })

API Gateway evaluates this token earlier invoking the getUploadURL Lambda office. This ensures that only authenticated users can upload objects to the S3 saucepan.

Modifying ACLs and creating publicly readable objects

In the current implementation, the uploaded object is non publicly accessible. To brand an uploaded object publicly readable, y'all must set its access control listing (ACL). There are preconfigured ACLs available in S3, including a public-read choice, which makes an object readable by anyone on the net. Set up the appropriate ACL in the params object before calling s3.getSignedUrl:

          const s3Params = {   Bucket: process.env.UploadBucket,   Key,   Expires: URL_EXPIRATION_SECONDS,   ContentType: 'paradigm/jpeg',   ACL: 'public-read' }

Since the Lambda function must accept the advisable saucepan permissions to sign the asking, you must besides ensure that the office has PutObjectAcl permission. In AWS SAM, you can add the permission to the Lambda role with this policy:

                      - Statement:           - Outcome: Allow             Resources: !Sub 'arn:aws:s3:::${S3UploadBucket}/'             Action:               - s3:putObjectAcl

Conclusion

Many spider web and mobile applications let users to upload data, including large media files like images and videos. In a traditional server-based application, this tin can create heavy load on the application server, and also apply a considerable amount of network bandwidth.

By enabling users to upload files to Amazon S3, this serverless pattern moves the network load away from your service. This can make your application much more scalable, and capable of treatment spiky traffic.

This blog post walks through a sample application repo and explains the procedure for retrieving a signed URL from S3. Information technology explains how to the test the URLs in both Postman and in a web application. Finally, I explain how to add authentication and make uploaded objects publicly accessible.

To acquire more than, run across this video walkthrough that shows how to upload directly to S3 from a frontend spider web application. For more serverless learning resources, visit https://serverlessland.com.