Replies: 1 comment
-
|
Thanks for the report. I'll go over it in order. Openshift AI recommendationsWe are mostly implementing these already. Notably the smaller context window is why we use drain, we have a synthetic dataset [0], and finetuning produced [1]. The CodeLlama is something I've considered, but ultimately didn't embrace. The reason is the considerable syntactic difference between data used to train code models and that we are consuming. Simply put, code models were trained on source code of various software, not on log files produced by set software. They are capable of replicating that source code, annotating it (because source includes comments) and even synthesizing it. Prepending, appending, and inserting new code into old is their strength. Anything else .. not so much. Instruct tuned code models are slightly better. They were also trained with prompts including some natural language instructions about what the code should look like and how should it work. This distinction is well seen when you compare output of CodeLlama with that of CodeLlama-instruct when given a following prompt:
Too few extracted linesThis is a feature, without positive connotations that word normally carries, of language models. LLama3 performanceThis is interesting, and I think it may be a good idea to switch to llama3 as our default. There is only one drawback, License. Different output llama provides between calls is unfortunately another "feature" of LLMs, their stochastic nature just can't be avoided. The best we can hope for is some sort of reasonable median response. ExtractorsLLM based extractor should be massive improvement on accuracy, as it would actually be able to pick up minute differences between log snippets that would otherwise get ignored by drain. Unfortunately it doesn't really scale, at least not with current gen HW. The problem lies in following When The time it takes to complete analysis must be less than the time it would take to write a reasonably long report and post it on a mailing list or similar. Or at least not longer by that much. Otherwise we won't save any time. We could try for a hybrid approach though. The drain is set to drill down to The extractor code is fine, in general terms, but it could deal better with memory. Fine tuningSadly you can't fine tune entire LLM, even the small 7B scale, on consumer HW. Even with smallest batch size the model is just too big. The best thing you can do are variations of LoRa, as I did. These are less impressive in results than "proper" fine tuning. But it's often enough to make a difference. [0] https://huggingface.co/datasets/fedora-copr/autoannotated_snippets_mistral |
Beta Was this translation helpful? Give feedback.
-
Log Detective Experiments
For context: I'm currently learning how to do packaging and was given some sample projects to attempt to package so I could learn to use fedpkg and make specfiles.
I was packaging warehouse: https://flattool.github.io/warehouse/
I talked to one of the engineering managers from Openshift AI who gave me some recommendations to try:
The TLDR of this document is:
Different Models
Mistral7b
When doing the packaging I left out the %files section.
When using the mistral7b model, these were the results produced:
LLama3
The formatting in this section is not perfect as the reply that was produced included some code blocks.
LLama3 produced:
The error message is not very informative, but it's clear that the build of warehouse-1.6.2-1.fc40.noarch failed.
Running
rpmbuild -bbcommand with--nodepsoption to disable dependency checking and see if there are any other errors:This error message is a bit more informative. It indicates that the command
/usr/bin/systemd-nspawnfailed, but it doesn't provide much information about what went wrong.Checking for unpackaged file(s):
This error message indicates that some files were installed but not packaged. These files include various Python scripts, CSS file, GResource file, icons, and other resources.
Analysis:
The error message indicates that there are installed (but unpackaged) files found during the build process. The list of files includes various Python scripts, CSS file, GResource file, icons.
Explanation:
The failure occurred because some files were not included in the RPM package. These files include Python scripts, CSS file, GResource file, icons. This is a common issue when packaging software that has multiple components. To fix this issue, you need to include these files in the RPM package using the
%filessection of the RPM spec file.To resolve this issue, add the missing files to the
%filessection of the RPM spec file:This will ensure that all necessary files are included in the RPM package. Rebuilding the package should resolve the issue.
I am a bit confused, but I understand now. The error message is indicating that there are some files installed during the build process that were not included in the RPM package.
To fix this issue, you need to include these files in the
%filessection of the RPM spec file. This will ensure that all necessary files are included in the RPM package and can be properly packaged and distributed.Here is an example of how you could add these files to the
%filessection:By adding these files to the
%filessection, you ensure that they are properly packaged and can be installed correctly.Please let me know if this makes sense. I am happy to help further!
LLama3 Attempt 2
On a newer version of logdetective (rather than providing the download link for the .gguf file, we can provide the model name) I seem to have different results with the same model
Something that I noticed was a common occurrence was that I would often get some form of "2.
RPM build errors:- This indicates that there were issues during the RPM build process." in my output.I had missed one of the Friday meetings but in the notes i noticed :
"drain may not be the best for test results where there are multiple lines that are relevant
I was under the impression that drain3 had the constraint of only handling line by line pieces of the log. I decided to play around with extractors but did not have much success.
Extractors
I played around with asking ChatGPT to modify our current LLMExtractor to chunk our input to try and compare it vs the current DrainExtractor. I had no success even running this, as the kernel in my Jupyter Notebook would constantly crash. I had it run once without the kernel crashing, but eventually I got the pop up saying that Chrome is out of memory. I ignored the error and my laptop crashed...
I didn't want to spend a huge amount of time looking into getting this working without the go ahead of others on the team. I do have a RTX 3080 in my personal computer that I attempted to use and had the same issue. It may just be a jupyter thing.
Fine Tuning attempt and using CodeLlama
CodeLlama
I wanted to see the types of responses that CodeLlama would give, so I decided to run it on my machine based on how it works here: https://ollama.com/library/codellama
I took what I would imagine to be the "interesting snippet of the log" and gave it the prompt that we currently use in LogDetective:
and the result was.. not very useful!
I also tried to use it with LogDetective, but it took 30 minutes or so to run and had a huge output:
Attempt to fine tune CodeLlama
I downloaded our data and tried to format it in a way to be used as a data set to train the CodeLlama model. Most of the code was generated from ChatGPT in an attempt to get something basic working and adjust it later. Similar to the Extractor attempts my machine could not handle this...
I attempted to format the data with this:
I'm not sure on the best way to present the data that we have gathered in a way that it can be used to train a model. I attempted to use this to train CodeLlama with the following:
I tried multiple version of this - receiving errors saying to add the pad_token, as well as attempting to reduce the batch sizes but had no luck in the efforts.
Beta Was this translation helpful? Give feedback.
All reactions