Skip to content

Commit

Permalink
PI-2330 Fix truncated allocations report due to async timeout (#4039)
Browse files Browse the repository at this point in the history
The exception is a "Socket interrupted" error exactly 30 seconds after the request is started, while reading data from the database in the Oracle JDBC driver.  The error is logged but because the response headers have already been sent, the status is still 200 OK.  The interruption happens in Oracle's TimeoutSocketChannel class.  When this class opens a DB connection, or starts reading data, it schedules an interruption to occur after a given socket timeout (soTimeout).  This sent me down the rabbit hole that the JDBC driver must have been causing the error after 30 seconds, however the socket timeout defaults to 0 (meaning no timeout), so it wasn't that.

Turns out actually Spring was interrupting the thread after its async timeout, which defaults to 30 seconds.  I definitely did look into this earlier on, but there must have been something wrong with my testing as I'm sure changing that value didn't fix it at first... 

I've also added a sanity check to the cron job, to ensure the report contains records from January 2024 which will be the last records to be returned due to ordering on the SQL query - and retry if not.

---

*For future reference*
Other timeouts I tried adjusting, when it seemed like a DB timeout:
```yaml
server:
  tomcat:
    connection-timeout: 5m
    keep-alive-timeout: 5m
spring:
  datasource:
    hikari:
      data-source-properties:
        oracle.net.CONNECT_TIMEOUT: 300_000
        oracle.net.READ_TIMEOUT: 300_000
        oracle.jdbc.ReadTimeout: 300_000
        oracle.jdbc.defaultConnectionValidation: LOCAL
        socketTimeout: 300
  jpa:
    properties:
      jakarta.persistence.query.timeout: 300_000
```
  • Loading branch information
marcus-bcl authored Jul 17, 2024
1 parent 64c94b1 commit 1afd938
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ data:
echo Downloading report...
curl -fsSL -H "Authorization: Bearer $hmpps_auth_token" http://workforce-allocations-to-delius/initial-allocations.csv > "/tmp/$filename"
echo Downloaded report with $(wc -l < "/tmp/$filename") rows
echo Validating report...
if grep -q '01/2024' "/tmp/$filename"; then echo 'Report is valid'; else echo 'Report is incomplete' >&2; exit 1; fi
echo Starting file upload...
file_details=$(curl -fsSL -XPOST -F "token=$SLACK_TOKEN" -F "filename=$filename" -F "length=$(wc -c < "/tmp/$filename")" https://slack.com/api/files.getUploadURLExternal)
Expand Down Expand Up @@ -81,5 +83,5 @@ spec:
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
restartPolicy: Never
restartPolicy: OnFailure
{{- end -}}
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ interface InitialAllocationRepository : JpaRepository<Person, Long> {
join district on district.district_id = team.district_id
join borough on borough.borough_id = district.borough_id
join probation_area on probation_area.probation_area_id = allocation.probation_area_id
order by allocation.allocation_date desc
""",
nativeQuery = true
)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Default config
server.shutdown: graceful
spring:
mvc.async.request-timeout: 5m # for report download
jackson:
default-property-inclusion: non_null
jpa:
Expand Down

0 comments on commit 1afd938

Please sign in to comment.