fhir.resources.R4B appears to hang GCP (apache beam) data flow jobs

[bmcd77]

This is just a "has anyone else had this problem" query?

I'm working on generating fhir resources into the Google Healthcare API Fhir store using the fhir.resources.R4B library. I'm doing this in an apache beam datarunner job which you can build then once you are happy run as a Job in GCP (google cloud processing).

It worked 100% running locally and I could build fhir and post to the fhir store just fine. I'd run my apache beam job with data runner which is allows you to run the job on your local pc.

When I run it in GCP using the requirements.txt below the python installs the library fine (google logging indicates the install worked and lib was retrieved from Pypi ok) and the GCP job start fine but just doesn't pick up any data. We're reading from a pub/sub. Remove the fhir.resources.R4B library and the job runs and picks up data!

Requirements.txt:

setuptools>=65.5.1
apache-beam[gcp]
google-cloud-pubsub
google-cloud-storage
google-api-python-client
hl7apy~=1.3.5
fhir.resources

Just wondered if:

• Anyone has used this library successfully with dataflow in google cloud?
• Anyone can suggested libraries that I could add to my installs to possibly help (is this a dependency issue?)

Lastly i did a pip freeze on my local machine which gives you ALL the libraries you have in your virtual environment and installed all of them in GCP Dataflow in the hope it was a missing lib that I wasn't being told about but no joy. The pip freeze library list was long

Pip freeze output is a bit long but here it is:

aiodns==3.0.0
aiohttp==3.8.4
aiomultiprocess==0.9.0
aiosignal==1.3.1
annotated-types==0.6.0
apache-beam==2.56.0
argcomplete==3.1.1
async-timeout==4.0.3
attrs==22.2.0
bc-detect-secrets==1.4.29
bc-jsonpath-ng==1.5.9
bc-python-hcl2==0.3.51
beautifulsoup4==4.12.2
boolean.py==4.0
boto3==1.26.155
botocore==1.29.155
cached-property==1.5.2
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
checkov==2.3.303
click==8.1.3
click-option-group==0.5.6
cloudpickle==2.2.1
cloudsplaining==0.5.1
colorama==0.4.6
ConfigArgParse==1.5.3
contextlib2==21.6.0
crcmod==1.7
cryptography==41.0.5
cyclonedx-python-lib==3.1.5
decorator==5.1.1
deep-merge==0.0.4
dill==0.3.1.1
dnspython==2.6.1
docker==6.1.3
dockerfile-parse==2.0.0
docopt==0.6.2
docstring_parser==0.16
dpath==2.1.3
email_validator==2.1.1
fastavro==1.9.4
fasteners==0.19
fhir.resources==7.1.0
frozenlist==1.3.3
gitdb==4.0.10
GitPython==3.1.31
glob2==0.7
google-api-core==2.19.0
google-api-python-client==2.129.0
google-apitools==0.5.31
google-auth==2.29.0
google-auth-httplib2==0.2.0
google-cloud-aiplatform==1.51.0
google-cloud-bigquery==3.22.0
google-cloud-bigquery-storage==2.25.0
google-cloud-bigtable==2.23.1
google-cloud-core==2.4.1
google-cloud-datastore==2.19.0
google-cloud-dlp==3.17.0
google-cloud-language==2.13.3
google-cloud-pubsub==2.21.1
google-cloud-pubsublite==1.10.0
google-cloud-recommendations-ai==0.10.10
google-cloud-resource-manager==1.12.3
google-cloud-spanner==3.46.0
google-cloud-storage==2.16.0
google-cloud-videointelligence==2.13.3
google-cloud-vision==3.7.2
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.63.0
grpc-google-iam-v1==0.13.0
grpc-interceptor==0.15.4
grpcio==1.63.0
grpcio-status==1.62.2
hdfs==2.7.3
hl7apy==1.3.5
httplib2==0.22.0
idna==3.4
igraph==0.10.4
importlib-metadata==6.7.0
iniconfig==2.0.0
isodate==0.6.1
Jinja2==3.1.2
jmespath==1.0.1
Js2Py==0.74
jsonpickle==3.0.4
jsonschema==4.17.3
junit-xml==1.9
lark==1.1.5
license-expression==30.1.0
lxml==4.9.3
Markdown==3.4.3
MarkupSafe==2.1.3
multidict==6.0.4
networkx==2.6.3
numpy==1.26.4
oauth2client==4.1.3
objsize==0.7.0
openai==0.27.8
orjson==3.10.3
overrides==7.7.0
packageurl-python==0.11.1
packaging==23.0
pluggy==1.0.0
ply==3.11
policy-sentry==0.12.6
policyuniverse==1.5.1.20230608
prettytable==3.8.0
proto-plus==1.23.0
protobuf==4.25.3
pyarrow==11.0.0
pyarrow-hotfix==0.6
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycares==4.3.0
pycep-parser==0.4.1
pycparser==2.21
pydantic==2.7.1
pydantic_core==2.18.2
pydot==1.4.2
Pygments==2.16.1
pyjsparser==2.7.1
pymongo==4.7.2
pyparsing==3.1.0
pyrsistent==0.19.3
pytest==7.2.2
python-dateutil==2.8.2
pytz==2024.1
pywin32==306
PyYAML==6.0
rdflib==6.3.2
redis==5.0.4
regex==2023.6.3
requests==2.31.0
rsa==4.9
s3transfer==0.6.1
schema==0.7.5
semantic-version==2.10.0
shapely==2.0.4
six==1.16.0
smmap==5.0.0
sortedcontainers==2.4.0
soupsieve==2.4.1
spdx-tools==0.7.1
sqlparse==0.5.0
tabulate==0.9.0
termcolor==2.3.0
texttable==1.6.7
toml==0.10.2
tomlkit==0.12.1
tqdm==4.65.0
typing_extensions==4.6.3
tzdata==2024.1
tzlocal==5.2
unidiff==0.7.5
update-checker==0.18.0
uritemplate==4.1.1
uritools==4.0.1
urllib3==1.26.16
wcwidth==0.2.6
websocket-client==1.6.0
xmltodict==0.13.0
xq==0.0.4
yarl==1.9.2
yq==3.2.2
zipp==3.15.0
zstandard==0.22.0

Any comments or suggestions appreciated

thanks.