Skip to content

Commit

Permalink
Switch Statsd host in Toolforge environment to cloudmetrics1003.
Browse files Browse the repository at this point in the history
Since f45589b, the application is instrumented and sends
metrics to the Wikimedia Cloud Services statsd endpoint.

It was initially sent to cloudmetrics1002,
then switched to cloudmetrics1001 (6de2308).

Since a few weeks, logs report issues reaching that endpoint [1].

This started on 2022-11-20 according to the otherwise fine Shell logs.
(Not sure exactly when it started for the Python webservice,
as the uwsgi logfile ballooned to 31 million lines,
making precise investigation a tad difficult).

It turns out cloudmetrics1002 (as well as cloudmetrics1001)
were decommissioned on November 16th (see T297444).

It also turns out the pystatsd library plain crashes
if the statsd host is unreachable [2].

This replaces the host with the new cloudmetrics1003.

Bug: T325936

[1] Python traceback:
```
Traceback (most recent call last):
  File "/data/project/integraality/www/python/src/app.py", line 10, in <module>
    from pages_processor import PagesProcessor, ProcessingException
  File "./pages_processor.py", line 16, in <module>
    from property_statistics import PropertyStatistics, QueryException
  File "./property_statistics.py", line 16, in <module>
    from statsd.defaults.env import statsd
  File "/data/project/integraality/www/python/venv/lib/python3.7/site-packages/statsd/defaults/env.py", line 17, in <module>
    maxudpsize=maxudpsize, ipv6=ipv6)
  File "/data/project/integraality/www/python/venv/lib/python3.7/site-packages/statsd/client/udp.py", line 35, in __init__
    host, port, fam, socket.SOCK_DGRAM)[0]
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
```

[2] jsocol/pystatsd#130
  • Loading branch information
JeanFred committed Dec 26, 2022
1 parent e8f045a commit c90f9ef
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion bin/defaults.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ echo_time() {
: ${TOOLFORGE_PATH:=$HOME/www/python/}
: ${VIRTUAL_ENV_PATH:=$TOOLFORGE_PATH/venv}
: ${APP_PATH:=$TOOLFORGE_PATH/src}
: ${STATSD_HOST:=cloudmetrics1001.eqiad.wmnet}
: ${STATSD_HOST:=cloudmetrics1003.eqiad.wmnet}
: ${STATSD_PORT:=8125}
: ${STATSD_PREFIX:=toolforge.integraality}
2 changes: 1 addition & 1 deletion deploy/hosts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
login-buster.toolforge.org

[toolforge:vars]
statsd_host=cloudmetrics1001.eqiad.wmnet
statsd_host=cloudmetrics1003.eqiad.wmnet
statsd_port=8125
statsd_prefix=toolforge.integraality

0 comments on commit c90f9ef

Please sign in to comment.