Timeout for certain tables (Regionaldatenbank)

[sjockers]

Requesting certain tables from regionsalstatistik.de takes several minutes, eventually causing a timeout.

I've encountered this with the following tables: 41120-12-01-4, 41141-03-01-4, 41141-03-02-4-B – but other tables may have the same issue.

# Requesting the table 41120-12-01-4 takes several minutes and will eventually cause a timeout 
t = pystatis.Table(name="41120-12-01-4") 
t.get_data()

Result:

ReadTimeout: HTTPSConnectionPool(host='[www.regionalstatistik.de](https://file+.vscode-resource.vscode-cdn.net/Users/sjockers/Projects/swr/agrar_regionalstatistik/www.regionalstatistik.de)', port=443): Read timed out. (read timeout=300)

Requests for other tables (from regionsalstatistik.de) run through and have much shorter request times, including some from the same statistics as the problematic ones, e.g. 41141-03-02-4, 41141-03-02-4-B.

[pmayd]

I am currently looking into the issue, but I have some general troubles with the Regionalstatistik API: Currently I am not able to make any request against the API, I am always getting a Code 6 in return telling me that I am executing too many parallel requests, which is not the case...

[pmayd]

@sjockers I investigated more and it only happens for fomat=ffcsv which is very bad for us as we only support this format. And the problem is not even the background job, actually the initial request against the tablefile endpoint never finishes for the ffcsv, so the API does not even return a response and also does not start a background job like in the past. So it seems like the whole system is broken currently, probably because they want to switch from background jobs to the archive format, but I don't understand why this happens without a new API version, so in general the whole API is very unstable right now and a lot of things do not work with Regionalstatistik....the csv download works, I tried it with Postman, but I don't think it makes much sense for us to also support this format as it requires a different approach in parsing...

[sjockers]

Thanks for looking into this, Michael! I agree that it would not make sense to support the CSV-format instead, as it would make parsing much harder (and would probably cause more issues down the line).

I have contacted regionalstatistik.de (IT.NRW) about the problem, lets see what they say.

[sjockers]

Update from regionalstatistik.de: This is apparently a known issue that concerns certain tables with complex headers: These tables cannot currently be rendered as ffcsv, causing the present bug.

Upgrading to a newer version of GENESIS should fix the problem, which will happen "in the coming months". There's no concrete date for the migration, though.

I'd suggest to ignore this bug until regionalstatistik.de is migrated to 'new' Genesis.