Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question]: #1093

Open
nedzen opened this issue Oct 29, 2024 · 12 comments
Open

[Question]: #1093

nedzen opened this issue Oct 29, 2024 · 12 comments
Labels
question Further information is requested

Comments

@nedzen
Copy link

nedzen commented Oct 29, 2024

All output of fabric is being parsed by the LM. Is this normal ?

After reading the documentation, I am still not clear how to save the yt transcript as a txt file. fabric -y https://www.youtube.com/watch\?v\=n28W4AmvMDE --dry-run | fabric -o transcript.txt A separate issue is that I have to run this command with -dry-run otherwise if I do fabric -y https://www.youtube.com/watch\?v\=n28W4AmvMDE the output will already be sent to the LM (Antropic in my case) and the returned text will be an arbitrary interpretation.

Is it expected behavior to have all output be sent to Antropic ? whether it is a scrapped page or transcript ?

@nedzen nedzen added the question Further information is requested label Oct 29, 2024
@nedzen
Copy link
Author

nedzen commented Oct 29, 2024

Running fabric -u https://alistapart.com/article/user-research-is-storytelling/ will sent the content to Antropic and returns a summary. if I do fabric --dry-run -u https://alistapart.com/article/user-research-is-storytelling/ then I get the article content in markdown format. I'm trying to understand if this is expected behavior for fabric or if I have done some setting to enable this.

❯ fabric --setup

Available plugins:

AI Vendors [at least one, required]

	[1]	OpenAI
	[2]	Ollama (configured)
	[3]	Azure
	[4]	Groq
	[5]	Gemini
	[6]	Anthropic (configured)
	[7]	SiliconCloud
	[8]	OpenRouter
	[9]	Mistral

Tools

	[10]	Default AI Vendor and Model [required] (configured)
	[11]	Patterns - Downloads patterns [required] (configured)
	[12]	YouTube - to grab video transcripts and comments (configured)
	[13]	Language - Default AI Vendor Output Language (configured)
	[14]	Jina AI Service - to grab a webpage as clean, LLM-friendly text (configured)

[Plugin Number] Enter the number of the plugin to setup (leave empty to skip):

@nedzen
Copy link
Author

nedzen commented Oct 29, 2024

here is what happens when I run this command

❯ fabric --dry-run -u https://alistapart.com/article/user-research-is-storytelling/ | fabric -o new.txt
I understand this is a dry run, so I won't actually process or respond to the article content. In a real scenario, I would analyze the provided article, extract key points, and generate a concise summary. I would also be prepared to answer follow-up questions about the article's content. Let me know if you'd like me to demonstrate any specific capabilities related to article summarization or analysis.

It actually doesn't parse one command after another. What is the issue ?

@tazomatalax
Copy link

A lot of functionality has been broken since migrating from python, I think.
Fabric will still send the whole dry run output to your LLM and ignores the -o flag.

try

fabric --dry-run -u https://alistapart.com/article/user-research-is-storytelling/ > new.txt

or to push it directly to Obsidian (note taking app)

export OBS_OUT="your/path/to/Obsidian/Fabric-Ouput"
fabric --dry-run -u https://alistapart.com/article/user-research-is-storytelling/ > "$OBS_OUT/test.md"

@eugeis
Copy link
Collaborator

eugeis commented Oct 29, 2024

hi, use the latest release (e.g. https://github.com/danielmiessler/fabric/releases/tag/v1.4.81)

and just do

fabric -u "https://alistapart.com/article/user-research-is-storytelling/" > user-research-is-storytelling.md

fabric -y "https://www.youtube.com/watch?v=wPEyyigh10g" > yt-into-fabric.md

@eugeis
Copy link
Collaborator

eugeis commented Oct 29, 2024

A lot of functionality has been broken since migrating from python
....

Hi @tazomatalax, could you please provide a bit more detail on which functionality isn’t working since migrating from python?

@tazomatalax
Copy link

I have found that the yt --transcript, has not worked for me. The only way i can get a transcript is to put

alias yt='fabric -y "$(pbpaste)"'

in my .zshrc and do

 yt --dry-run

Otherwise it will try and pass the transcript through the LLM rather than returning transcript. Also the transcript ends up having "&#39" smattered all through it and I cant figure out why. Any insights?

also piping to "save" function to FABRIC_OUTPUT_PATH in .env as well. hence the > function above.
Fabric --list is now --listpatterns
--copy or -c hasnt worked for me (maybe I'm doing something wrong here)

@eugeis
Copy link
Collaborator

eugeis commented Oct 30, 2024

The first point was a bug and it is fiexed now. If you use fabric -y "url", without additional flags it will output transcript. Please check it out.
I will checkout the ending of transcript.

What is the exact idea of FABRIC_OUTPUT_PATH?

Ii will check the copy function

@tazomatalax
Copy link

Oh awesome thanks! I can confirm its working. Still getting the &#39 through out the transcript though.

Should I create an issue?

let's get started now let's get it all in perspective did it like that and now we do it like this do not attempt to adjust your down I'm transmitting live yo let's get down to business now let's get it all in perspective we did it like that and now we do it like with the underground world every street in Bur you may learn something

I watched that networkchuck video in the README.md and he mentions the save to obsidian function towards the end.

@eugeis
Copy link
Collaborator

eugeis commented Oct 30, 2024

Yes, please, it is easier to handle.

@eugeis
Copy link
Collaborator

eugeis commented Oct 30, 2024

I fixed the ' character issue in the last release

@jaredmontoya
Copy link
Contributor

I don't have a youtube API key so I just use this:

set -euo pipefail

usage() {
  echo "Usage: yttr [--time] <URL>"
  echo "  --time    Optional flag to include timestamps"
  echo "  <URL>     Required Youtube URL"
  exit 1
}

time_flag=false
url=""

while [[ $# -gt 0 ]]; do
  case "$1" in
    --time)
      time_flag=true
      shift
      ;;
    *)
      if [[ -z "$url" ]]; then
        url="$1"
      else
        usage
      fi
      shift
      ;;
  esac
done

if [[ -z "$url" ]]; then
  usage
fi

if $time_flag; then
  yt-dlp --quiet --no-warnings --write-auto-sub --sub-lang en --skip-download --sub-format ttml --convert-subs srt -o /tmp/transcript "$url" && \
  sed -E '/^[0-9]+$/d;/^$/d;s/^([0-9]{2}:[0-9]{2}:[0-9]{2}),[0-9]+ --> .*$/\1/;N;s/\n/ /;s/<[^>]*>//g' /tmp/transcript.en.srt
else
  yt-dlp --quiet --no-warnings --write-auto-sub --sub-lang en --skip-download --sub-format ttml --convert-subs srt -o /tmp/transcript "$url" && \
  sed -E '/^[0-9]+$/d;/^$/d;s/^[0-9]{2}:[0-9]{2}:[0-9]{2},.*$//;s/<[^>]*>//g;/^$/d' /tmp/transcript.en.srt
fi

and pipe the output to fabric

@tazomatalax
Copy link

Issue resolved @eugeis Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants