demo course failure

[thraxil]

I'm putting this here for now. It may not be a Sultan issue; might just be generic edx/devstack. But I'm going to log it here until I can come back later to investigate in more detail.

$ sultan config init
# minimum config edits
$ sultan instance create

That gets through most of it until eventually (after some collectstatic stuff):

2020-11-04 09:52:58,681 INFO 1158 [ecommerce.courses.models] /edx/app/ecommerce/ecommerce/ecommerce/courses/models.py:244 - Course seat product stock record with certificate type [verified] for [course-v1:edX+DemoX+Demo_Course] does not exist. Instantiated a new instance.
2020-11-04 09:52:58,685 INFO 1158 [oscar.alerts] /edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/oscar/apps/customer/alerts/utils.py:59 - Sending alerts for 'Seat in edX Demonstration Course with verified certificate (and ID verification)'
2020-11-04 09:52:58,688 INFO 1158 [oscar.alerts] /edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/oscar/apps/customer/alerts/utils.py:134 - Sent 0 notifications and 0 messages
Created audit and verified seats for [course-v1:edX+DemoX+Demo_Course]
2020-11-04 09:52:58,726 ERROR 1158 [ecommerce.courses.publishers] /edx/app/ecommerce/ecommerce/ecommerce/courses/publishers.py:114 - Failed to publish commerce data for [course-v1:edX+DemoX+Demo_Course] to LMS.
Traceback (most recent call last):
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1151, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1196, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1147, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 950, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 893, in send
    self.connect()
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9442f8c438>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='edx.devstack.lms', port=18000): Max retries exceeded with url: /oauth2/access_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9442f8c438>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/ecommerce/ecommerce/ecommerce/courses/publishers.py", line 101, in publish
    commerce_api_client = site.siteconfiguration.commerce_api_client
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/edx/app/ecommerce/ecommerce/ecommerce/core/models.py", line 482, in commerce_api_client
    return EdxRestApiClient(self.build_lms_url('/api/commerce/v1/'), jwt=self.access_token)
  File "/edx/app/ecommerce/ecommerce/ecommerce/core/models.py", line 415, in access_token
    token_type='jwt'
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 142, in get_oauth_access_token
    return get_oauth_access_token(url, client_id, client_secret, token_type=token_type)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 72, in get_oauth_access_token
    'User-Agent': USER_AGENT,
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/edx/app/ecommerce/venvs/ecommerce/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='edx.devstack.lms', port=18000): Max retries exceeded with url: /oauth2/access_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9442f8c438>: Failed to establish a new connection: [Errno 111] Connection refused',))
An error occurred while attempting to publish [course-v1:edX+DemoX+Demo_Course] to LMS
+ echo -e '\033[0;32m Provisioned ecommerce.\033[0m'
 Provisioned ecommerce.

more stuff succeeds

Then

2020-11-04 09:56:35,626 INFO 268 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:153 - Executing Loader [http://edx.devstack.lms:18000/api/courses/v1/]
2020-11-04 09:56:35,631 ERROR 268 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:28 - CoursesApiDataLoader failed!
Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1151, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1196, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1147, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 950, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 893, in send
    self.connect()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f2741ccd080>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='edx.devstack.lms', port=18000): Max retries exceeded with url: /oauth2/access_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2741ccd080>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 25, in execute_loader
    loader_class(*loader_args).ingest()
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 31, in __init__
    self.username = self.get_username_from_client(self.api_client)
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 41, in get_username_from_client
    token = client.get_jwt_access_token()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 270, in get_jwt_access_token
    self._ensure_authentication()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 254, in _ensure_authentication
    timeout=self._timeout,
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 170, in get_and_cache_oauth_access_token
    timeout=timeout,
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 112, in get_oauth_access_token
    timeout=timeout
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='edx.devstack.lms', port=18000): Max retries exceeded with url: /oauth2/access_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2741ccd080>: Failed to establish a new connection: [Errno 111] Connection refused',))
2020-11-04 09:56:35,810 INFO 268 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:153 - Executing Loader [http://edx.devstack.ecommerce:18130/api/v2/]
2020-11-04 09:56:35,814 ERROR 268 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:28 - EcommerceApiDataLoader failed!
Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1151, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1196, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1147, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 950, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 893, in send
    self.connect()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f2741cd5128>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='edx.devstack.lms', port=18000): Max retries exceeded with url: /oauth2/access_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2741cd5128>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 25, in execute_loader
    loader_class(*loader_args).ingest()
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/api.py", line 295, in __init__
    super(EcommerceApiDataLoader, self).__init__(partner, api_url, max_workers, is_threadsafe, **kwargs)
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 31, in __init__
    self.username = self.get_username_from_client(self.api_client)
  File "/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 41, in get_username_from_client
    token = client.get_jwt_access_token()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 270, in get_jwt_access_token
    self._ensure_authentication()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 254, in _ensure_authentication
    timeout=self._timeout,
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 170, in get_and_cache_oauth_access_token
    timeout=timeout,
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 112, in get_oauth_access_token
    timeout=timeout
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='edx.devstack.lms', port=18000): Max retries exceeded with url: /oauth2/access_token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2741cd5128>: Failed to establish a new connection: [Errno 111] Connection refused',))
CommandError: One or more of the data loaders above failed.
2020-11-04 09:56:38,924 INFO 279 [algoliasearch_django.registration] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/algoliasearch_django/registration.py:74 - REGISTER <class 'course_discovery.apps.course_metadata.algolia_models.AlgoliaProxyProduct'>
2020-11-04 09:56:40,561 INFO 279 [course_discovery.apps.core.utils] /edx/app/discovery/discovery/course_discovery/apps/core/utils.py:53 - ...index [catalog_20201104_095640] created.
Indexing 0 People
Indexing 0 courses
Indexing 0 course runs
Indexing 0 programs
-e Adding demo program to Discovery...
2020-11-04 09:56:43,706 INFO 292 [algoliasearch_django.registration] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/algoliasearch_django/registration.py:74 - REGISTER <class 'course_discovery.apps.course_metadata.algolia_models.AlgoliaProxyProduct'>
Traceback (most recent call last):
  File "/edx/app/discovery/discovery/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 86, in handle
    exec(options['command'])
  File "<string>", line 42, in <module>
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 408, in get
    self.model._meta.object_name
course_discovery.apps.course_metadata.models.DoesNotExist: Course matching query does not exist.
Makefile:123: recipe for target 'dev.provision.services' failed
make: *** [dev.provision.services] Error 1

I don't know much about the demo course and discovery service stuff. I do know that we often disable them on our deploys, so perhaps we need to do that here as well.

I do end up with a working devstack so this may just be a case of edx/ansible being overly verbose about inconsequential errors.

[iamjazzar]

@thraxil Is this still an issue?