rasa train is very slow - using DIETClassifier

[salila28]

I am running the command rasa train and it is very very slow(10% after I let it run the whole night) . It seems to be using most memory and resources on my computer (32G RAM but total virtual memory of 65.8G). Although the model is only 217MB in size and the dataset is not that big. I am trying to figure ways to speed up this process. I tried to run on a VM with 16G and it choked with OOM errors.
My config.yml looks like below:

Configuration for Rasa NLU.

https://rasa.com/docs/rasa/nlu/components/

language: en

pipeline:

No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.

If you'd like to customize it, uncomment and adjust the pipeline.

See https://rasa.com/docs/rasa/tuning-your-model for more information.

- name: WhitespaceTokenizer

• name: SpacyNLP
  model: "en_core_web_sm"
• name: SpacyTokenizer
• name: SpacyEntityExtractor
• name: SpacyFeaturizer
• name: RegexFeaturizer
• name: LexicalSyntacticFeaturizer
• name: CountVectorsFeaturizer
• name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 4
• name: DIETClassifier
  epochs: 200
• name: EntitySynonymMapper
• name: ResponseSelector
  epochs: 100
• name: FallbackClassifier
  threshold: 0.3
  ambiguity_threshold: 0.1

Configuration for Rasa Core.

https://rasa.com/docs/rasa/core/policies/

policies:

# No configuration for policies was provided. The following default policies were used to train your model.

# If you'd like to customize them, uncomment and adjust the policies.

# See https://rasa.com/docs/rasa/policies for more information.

- name: MemoizationPolicy

- name: TEDPolicy

max_history: 5

epochs: 100

name: RulePolicy
batch_size=10

[hsm207]

what is the output of rasa --version and what is the training time like on the bot from rasa init?

[salila28]

rasa --version

2.0.3

It seems that the examples I am passing in the intents are the content of an entire worddoc on a single line and the memory is not handling so many chars together - If I break the lines down, it moves a little faster but not sure if breaking an intent example of multiple line is advisable or ok to do

[hsm207]

1. What is the distribution of the number of words in your intent examples i.e. min, max, mean?
2. Have you tried upgrading rasa to the latest version?
3. Does training on the rasa init bot take a long time too?

[salila28]

1. around 2000 words(12k chars)
2. using 2.0.3 since last model used that version. I will try to upgrade - any versions advised?
3. rasa init completes within few mins versus the project training takes around > 5 hrs even after I split up the words on seperate lines to shorten the lenth of the line

[hsm207]

around 2000 words(12k chars)

I assume you are saying that the total number of words for all your examples are around 2,000 words.

What I meant to ask was the average number of words in an example, the longest example and the shortest example.

using 2.0.3 since last model used that version. I will try to upgrade - any versions advised?

I advise upgrading to rasa 2.8.5

split up the words on seperate lines to shorten the lenth of the line

I don't understand what you mean by this. Did you have an example where it has 50 words and now you created 2 examples with 25 words each? It would be helpful if you can share a before and after of a training example to show how you did the split.

[salila28]

Actually the total number of words for a single example is 2000 words. I think this is probably not a idea but it is a document we are uploading directly from customers that will shows as an example of the intent based on the document.

I upgraded to 2.8.5 and don't see much progress

The splitting on seperate lines was to identify whether the huge example was casuing memory issue and it proved correct. I am struggling how to use an entire document content as a single example

[hsm207]

Actually the total number of words for a single example is 2000 words.

DIET Classifier does not impose a limit on the length of an example but the memory required to train the model will grow as the input length gets longer. The OOM error and long training time is expected since your training example is very long. Note that DIET Classifier is based on the transformers architecture.

If you really want to use DIET Classifier to classify a large document, then you need to truncate it before it gets passed to DIET Classifier for training and/or simplify the config.yml e.g. reduce number of layers, featurizers, etc . Alternatively, you can write a custom action that will classify the document by calling a model that can be trained on large documents.