Error with uploading small file sizes versus successful large file sizes with Python SDK

SDKs & Tools Python
1

Hello, I am currently building an integration that downloads & uploads files from Box using the Box Python SDK.

I am having issues using the upload_stream function (https://github.com/box/box-python-sdk/blob/main/docs/usage/files.md#upload-a-file) with an IOBase stream.
I am doing the below (taken from the docs):

new_file = parent_folder.upload_stream(object_bytes, file_name)

The error I get it is:

TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
File "upload_to_box", line 174, in upload_small_file
    new_file = parent_folder.upload_stream(object_bytes, file_name)
  File "/usr/local/lib/python3.10/site-packages/boxsdk/util/api_call_decorator.py", line 63, in call
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/boxsdk/object/folder.py", line 322, in upload_stream
    file_response = self._session.post(
  File "/usr/local/lib/python3.10/site-packages/boxsdk/session/session.py", line 100, in post
    return self.request('POST', url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/boxsdk/session/session.py", line 138, in request
    response = self._prepare_and_send_request(method, url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/boxsdk/session/session.py", line 348, in _prepare_and_send_request
    network_response = self._send_request(request, **kwargs)

I tried digging through the stack trace and beyond in the Python SDK code to see where a subtraction operand was occurring but couldn’t find it easily.

This same IOBase stream works fine with the chunked uploader system (https://github.com/box/box-python-sdk/blob/main/docs/usage/files.md#chunked-upload) below:

upload_session: UploadSession = parent_folder.create_upload_session(file_size, file_name)
chunked_uploader = upload_session.get_chunked_uploader_for_stream(object_bytes, file_size)
new_file = chunked_uploader.start()

Exact line in Github for the stacktrace: https://github.com/box/box-python-sdk/blob/main/boxsdk/session/session.py#L348

Wondering if anyone has any inkling as to why/where this error is actually occurring. That could potentially help us debug on our side if there is something ultimately wrong with our IOBase stream.

For more context:
The object_bytes IOBase stream was recently converted to be chunked to support a fully chunked system. Essentially we are trying to do chunked downloads from a 3rd part source resulting in our own IOBase that we use to chunk upload to Box when the file is >=20MB or do the in memory upload if it’s <20MB (as per the docs). It was working before when our object_bytes IOBase stream was fully loaded into memory.

2

We are currently trying to do a workaround where we take our chunked object_bytes and write it to a local disk file and then try using the upload_stream with a open read stream of the local disk file. Will report back if that works. Still would be curious about where the original error is coming from though…

3

Got a workaround that is successful for our situation currently. Though any help on figuring out the original source error would be extremely appreciated! That way we can hopefully fix our implementation and get rid of the workaround…

The workaround is to write our chunked byte stream to a standard BytesIO stream and then give that to the upload_stream function from the Box SDK:

def upload_small_stream_to_box(parent_folder: Folder, object_bytes: IOBase, file_name: str):
    chunk_size = 256 * 1024
    upload_bytes = BytesIO()

    while True:
        chunk = object_bytes.read(chunk_size)
        if not chunk:
            break
        upload_bytes.write(chunk)
    
    new_file = parent_folder.upload_stream(upload_bytes, file_name)

    return new_file

It’s inefficient, but it at least gets us unstuck… Thankfully it’s only for files <20MB.

4

As an added note of context, it looks like the upstream code that is giving me the object_bytes stream is coming from a GCP storage blob.

image

Wonder if that has anything to do with it?

5

Thanks for posting! What version of Python SDK are you using? I wasn’t able to pinpoint the stack trace to the source code of our latest version (v3.9.2). Also, please consider using GitHub - box/box-python-sdk-gen: Repository for generated Box Python SDK which is our new generation of Python SDK.

6

Hello mhagmajer, I am using 3.9.2 as far as I am aware here is what my dependency looks like in my pyproject.toml.

dependencies = [
    'boxsdk == 3.9.2'
]

Here is what it looks like venv wise in VS Code:
image

I had started with the generated Box Python SDK, but found it fell rather short of the SDK I am using…

7

Here is the exact line of code the error stacktrace is referring too…

Apparently I can’t post links to the github sdk in a comment…
https://github.com/box/box-python-sdk/blob/main/boxsdk/session/session.py#L348