How to handle timeouts when downloading a file async

[LuchiLucs]

I wrote an async function to download a propetary binary file like so:

async def download_file(session, timeout, filename, url, headers, params) -> str:
    async with session.get(url, headers=headers, params=params, timeout=timeout) as response:
        if response.status == 200:
            async with aiofiles.open(filename, "wb") as f:
                await f.write(await response.read())
            return filename

Sometimes, the client crashes due to "asyncio.TimeoutError". So I tried to increase the timeout of the GET request for each file downloaded by this function but I wonder if I can design this better. To date, I create an async session and I run n-th async tasks with this function to download all required files in parallel. Some are small, others are very large. Is there a way to download what is left of file when some error occurs?

[Dreamsorcerer]

I'm not really sure what is included to support such requests, but if the server supports range requests, then, in theory, you'd be able to use that to resume a download: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#range_requests

[LuchiLucs]

I created this modified snippet. Would you mind telling me if you see something I could change to be robust or if the logic is right here? It is my first time dealing with requests.

@async_timer
async def download_file(session, timeout, url, headers, params, filename) -> str:
    start = 0
    for count in range(HTTP_RETRY_COUNT):
        try:
            headers['Range'] = f'bytes={start}-'
            async with session.get(url, headers=headers, params=params, timeout=timeout) as response:
                if response.status == 206:  # Partial Content
                    mode = 'ab' if start != 0 else 'wb'
                    async with aiofiles.open(filename, mode) as f:
                        while True:
                            chunk = await response.content.read(HTTP_CHUNK_SIZE)
                            if not chunk:
                                break
                            await f.write(chunk)
                            start += len(chunk)
                    logger.info(f"{filename} downloaded with success")
                    return filename
                else:
                    pass #TODO
        except TimeoutError:
            logger.warning(f"Timeout of {filename}: restart #{count} from bytes {start}")
            pass #TODO
    else:
        logger.error(f"Impossible to download {filename} after {HTTP_RETRY_COUNT} retries")

[Dreamsorcerer]

At a glance, looks fine to me if it's working. You could maybe change the loop condition so that it will always retry if it successfully downloaded some data in the previous attempt though (and it was a 206).