PageBlob upload as multiple chunks: x-ms-range invalid

Question

This is a follow-up question to my previous question. What I realized is I can only upload 4 mega bytes max to Azure. So I was trying to modify the code to chunk upload to pageblob. But now I get an error trying to upload the first chunk.

sas_uri = '<SAS URI>'
uri = urlparse(sas_uri)

conn = http.client.HTTPSConnection(uri.hostname, port=uri.port, timeout=3000)

file_path = r"C:\Users\user\Downloads\npp.Installer.exe"

def chunk(msg, n):
    for i in range(0, len(msg), n):
        yield msg[i:i + n]


with open(file_path, 'rb') as reader:
    file = reader.read()

    file_size = len(file)
    block_size = file_size
    boundary = block_size % 512
    if boundary != 0:
        padding = b'\0' * (512 - boundary)
        file = file + padding
        block_size = block_size + 512 - boundary    # needed to make the file on boundary

    headers = {
        'Content-Type': 'application/octet-stream',
        'Content-Length': 0,
        'x-ms-blob-type': 'PageBlob',
        'x-ms-blob-content-length': block_size
    }

    # Reserve a block space
    conn.request('PUT', sas_uri, '', headers)
    res = conn.getresponse()
    data = res.read()
    print(data.decode("utf-8"))

    CHUNK_MAX_SIZE = int(4e+6) # 4 mega bytes
    index = 0
    for chunk in chunk(file, CHUNK_MAX_SIZE):
        chunk_size = len(chunk)
        headers = {
            'Content-Type': 'application/octet-stream',
            'Content-Length': chunk_size,
            'x-ms-blob-type': 'PageBlob',
            'x-ms-page-write': 'update',
            'x-ms-range': f"bytes={index}-{index + chunk_size - 1}"
        }
        # Upload the file
        conn.request('PUT', sas_uri + '&comp=page', file, headers)
        res = conn.getresponse()
        data = res.read()
        print(data.decode("utf-8"))

        index = index + chunk_size

Error:

<?xml version="1.0" encoding="utf-8"?>
<Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:c312a91d-401c-0000-44e9-95bd08000000
Time:2021-08-21T17:33:47.2909476Z</Message><HeaderName>x-ms-range</HeaderName><HeaderValue>bytes=0-3999999</HeaderValue></Error>

Update: I corrected the CHUNK_MAX_SIZE so now the first chunk gets uploaded without error but subsequent chunks result in this error:

!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">Bad Request\r\n

Bad Request - Invalid Verb

---

HTTP Error 400. The request verb is invalid.

Answer 1

The problem is with the following line of code:

CHUNK_MAX_SIZE = int(4e+6) # 4 mega bytes

It gives you 4000000 which is not 4MB. 4MB is 4194304 bytes.

Please try by changing your code to:

CHUNK_MAX_SIZE = 4 * 1024 * 1024 # or 4194304

and your code should work fine.

UPDATE

Please change the following line of code:

conn.request('PUT', sas_uri + '&comp=page', file, headers)

to

conn.request('PUT', sas_uri + '&comp=page', chunk, headers)

and you should be able to upload the file. Essentially I changed file to chunk in your code as you would want to just upload the chunk and not the entire file.