Comparing two methods for uploading a file to a server

This post shows two ways you can upload a file to a Spring web application. I do not show the full code but parts of it.

Method 1 – using multipart/form-data

server code:

@PostMapping("/fileUpload")
public ResponseEntity<String> handleMultipartFileUpload(@RequestPart("file") MultipartFile file) {
    if (file.isEmpty()) {
        return ResponseEntity.badRequest().body("File is empty");
    }

    try {
        long t0 = System.currentTimeMillis();
        // Process the file here
        byte[] bytes = file.getBytes();
        String checksum = Utils.calculateMD5(bytes);
        logger.info("received {} bytes with checksum {} in {} ms", bytes.length, checksum, System.currentTimeMillis() - t0);
        
        return ResponseEntity.ok("File uploaded successfully");
    } catch (Exception e) {
        e.printStackTrace();
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error uploading file");
    }
}

client code:

with open('sample_file.txt', 'rb') as f:
    files = {'file': f}
    response = requests.post(file_upload_url, files=files)

# Printing the response
print(response.status_code)
print(response.text)

this method corresponds to following OpenAPI spec:

requestBody:
    required: true
    content:
        multipart/form-data:
        schema:
            type: object
            properties:
            file:
                type: string
                format: binary

Method 2 – using application/octet-stream

server code:

@PostMapping(value = "/binaryUpload",
consumes = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity<String> handleBinaryUpload(HttpServletRequest httpServletRequest) {
    try {
        long t0 = System.currentTimeMillis();
        var inputStream = httpServletRequest.getInputStream();
        Checksum checksum = Utils.calculateMD5(inputStream);
        logger.info("received {} bytes with checksum {} in {} ms", checksum.length(), checksum.checksum(), System.currentTimeMillis() - t0);
        return ResponseEntity.ok("File uploaded successfully");
    } catch (Exception e) {
        e.printStackTrace();
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error uploading file");
    }
}

client code:

with open('sample_file.txt', 'rb') as f:
    response = requests.post(binary_upload_url, data=f, headers={'Content-Type': 'application/octet-stream'})

# Printing the response
print(response.status_code)
print(response.text)

this method corresponds to following OpenAPI spec:

requestBody:
    required: true
    content:
        application/octet-stream:
        schema:
            type: string
            format: binary

Comparison

Method 1 gave me following error when I used a large file:

org.springframework.web.multipart.MaxUploadSizeExceededException: Maximum upload size exceeded

It can be fixed by adding following to application.properties [1]:

spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=10MB
server.tomcat.max-swallow-size=100MB

With this I was able to upload my test file:

MainController          : received 5646730 bytes with checksum 715a149541e7cf0995b341a2b18d3bfe in 12 ms

However how big is too big? where should the cap be?

Method 2 works as well:

MainController          : received 5646730 bytes with checksum 715a149541e7cf0995b341a2b18d3bfe in 11 ms

in addition it does not require tweaking maximum file upload limit. It seems superior and better choice for sending very large blobs to a server. On the other hand sending a file as multipart/form-data will allow you to access metadata about the file on the server [1]. Also checkout this question on SO.

This answer re: multipart/form-data on SO says:

It does not split huge files into parts.

With Method 2 I was able to upload 4GB file to the server without errors:

MainController          : received 3848008288 bytes with checksum 8c22ba24008740228b73731969546db8 in 6529 ms

I think this method will also lend itself to the case when the client is NOT uploading a file from disk but rather connecting to a BLOB storage like AWS S3 and wants to ingest a blob from there (i.e., upload the blob to your server without having to save it to local disk). TODO: test it.

This entry was posted in Computers, programming, Software. Bookmark the permalink.

Leave a comment