-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrate black forest labs FLUX.1 models into Galaxy #1496
Changes from all commits
6255005
cf930ce
41e8175
8100223
22cf4f2
5056445
3946bb2
5ec0a2d
899ee1f
3cfe7d3
a75758b
389d3bf
3305e3a
0ed1cba
2429e5c
add5ac7
416152c
4dc2456
84da1ee
a4a361f
d8f617b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
name: black_forest_labs_flux | ||
owner: bgruening | ||
description: Using the black forest labs FLUX.1 models to generate image based on user input. | ||
long_description: | | ||
This tool uses the black forest labs FLUX.1 models to generate image based on user input. | ||
Users can specify the model (dev,schnell),and provide prompt. | ||
The tool will then generate and return the corresponding image based on the input provided. | ||
remote_repository_url: https://github.com/bgruening/galaxytools/tree/master/tools/flux | ||
homepage_url: https://github.com/bgruening/galaxytools/tree/master/tools/cflux | ||
type: | ||
categories: | ||
- Machine Learning | ||
maintainers: | ||
- arash77 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
import sys | ||
|
||
import torch | ||
from diffusers import FluxPipeline | ||
|
||
model = sys.argv[1] | ||
|
||
prompt_type = sys.argv[2] | ||
if prompt_type == "file": | ||
with open(sys.argv[3], "r") as f: | ||
prompt = f.read().strip() | ||
elif prompt_type == "text": | ||
prompt = sys.argv[3] | ||
|
||
if model not in ["black-forest-labs/FLUX.1-dev", "black-forest-labs/FLUX.1-schnell"]: | ||
print("Invalid model!") | ||
sys.exit(1) | ||
|
||
|
||
pipe = FluxPipeline.from_pretrained(model, torch_dtype=torch.bfloat16) | ||
pipe.enable_sequential_cpu_offload() | ||
pipe.vae.enable_slicing() | ||
pipe.vae.enable_tiling() | ||
pipe.to(torch.float16) | ||
|
||
image = pipe( | ||
prompt, | ||
num_inference_steps=4, | ||
generator=torch.Generator("cpu").manual_seed(42), | ||
).images[0] | ||
|
||
image.save("output.png") |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
<tool id="black_forest_labs_flux" name="FLUX" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="23.0"> | ||
<description>text-to-image model</description> | ||
<macros> | ||
<token name="@TOOL_VERSION@">2024</token> | ||
<token name="@VERSION_SUFFIX@">0</token> | ||
</macros> | ||
<requirements> | ||
<requirement type="package" version="3.12">python</requirement> | ||
<requirement type="package" version="2.4.1">pytorch</requirement> | ||
<requirement type="package" version="0.19.1">torchvision</requirement> | ||
<requirement type="package" version="0.30.2">diffusers</requirement> | ||
<requirement type="package" version="4.44.2">transformers</requirement> | ||
<requirement type="package" version="0.34.0">accelerate</requirement> | ||
<requirement type="package" version="0.2.0">sentencepiece</requirement> | ||
<requirement type="package" version="4.25.3">protobuf</requirement> | ||
<requirement type="package" version="0.24.6">huggingface_hub</requirement> | ||
</requirements> | ||
<command detect_errors="exit_code"><![CDATA[ | ||
export HF_HOME='$flux_models.fields.path' && | ||
python '$__tool_directory__/flux.py' | ||
'$flux_models' | ||
'$input_type_selector' | ||
'$prompt' | ||
]]></command> | ||
<configfiles> | ||
</configfiles> | ||
<inputs> | ||
<param name="flux_models" label="Model data" type="select" help="contact the administrator of this Galaxy instance if you miss model data"> | ||
<options from_data_table="huggingface"> | ||
<filter type="static_value" column="4" value="flux"/> | ||
<filter type="static_value" column="5" value="1"/> | ||
</options> | ||
<validator message="No model annotation is available for FLUX" type="no_options"/> | ||
</param> | ||
<conditional name="input_type"> | ||
<param name="input_type_selector" type="select" label="Choose the type of input"> | ||
<option value="file" selected="true">File based input</option> | ||
<option value="text">Text based input</option> | ||
</param> | ||
<when value="file"> | ||
<param name="prompt" type="data" optional="false" format="txt" label="Prompt file" help="This data will be used as prompt"/> | ||
</when> | ||
<when value="text"> | ||
<param name="prompt" type="text" optional="false" label="Prompt" help="This text will be used as prompt" area="true"> | ||
<validator type="empty_field"/> | ||
</param> | ||
</when> | ||
</conditional> | ||
</inputs> | ||
<outputs> | ||
<data name="output" format="png" label="${tool.name} on ${on_string}" from_work_dir="./output.png"/> | ||
</outputs> | ||
<tests> | ||
<test expect_exit_code="1" expect_failure="true"> | ||
<param name="flux_models" value="unknown"/> | ||
<param name="input_type_selector" value="file"/> | ||
<param name="prompt" value="flux_test.txt"/> | ||
<assert_stdout> | ||
<has_text text="Invalid model!"/> | ||
</assert_stdout> | ||
</test> | ||
</tests> | ||
<help><![CDATA[ | ||
|
||
.. class:: infomark | ||
|
||
**What it does** | ||
|
||
FLUX is a text-to-image model that generates images from textual descriptions. | ||
|
||
Usage | ||
..... | ||
|
||
**Input** | ||
|
||
1. **Model data**: | ||
Select the model data from the dropdown list. The available models are dev and schnell. | ||
For using dev model, make sure to not use it for commercial purposes. | ||
|
||
2. **Input Prompt**: You can provide the input prompt in two ways: | ||
- **File based input**: Upload a txt file containing the prompt. | ||
- **Text based input**: Enter the prompt text in the text area. | ||
|
||
**Output** | ||
|
||
This response is saved in the `output.png` file. | ||
|
||
]]></help> | ||
<citations> | ||
<citation type="bibtex"> | ||
@misc{flux, | ||
author = {black forest labs}, | ||
title = {FLUX github repository}, | ||
howpublished = {\url{https://github.com/black-forest-labs/flux}}, | ||
year = {2024}, | ||
note = {Accessed: 2024-09-04} | ||
} | ||
</citation> | ||
</citations> | ||
</tool> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Extreme close-up of a spiral galaxy, direct frontal view, with highly detailed stars and nebulae. The image has sharp focus on texture and color, with ethereal cosmic lighting to capture the galaxy's natural shine and depth. The words 'Galaxy Project' are painted in large, white brush strokes with visible texture, centered across the image. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
|
||
#This is a sample file distributed with Galaxy that is used to define huggingface | ||
#models, using 7 columns tab separated | ||
#(longer whitespace are TAB characters): | ||
# | ||
#The entries are as follows: | ||
# | ||
#<unique_build_id> <display_name> <pipeline_tag> <domain> <free_tag> <version> <folder_base_path> | ||
# | ||
#pipeline_tag (see this URL: https://huggingface.co/models?pipeline_tag=text-to-image they call it that way) | ||
#domain (defined by the Galaxy community, e.g. image / sequence / text) | ||
#free_tag (not idea yet, but could be freely used by an admin to specify more filter options) | ||
#version (if available) | ||
#Your huggingface.loc file should include the huggingface cached model folder you have stored. | ||
#For example: | ||
# | ||
#black-forest-labs/FLUX.1-dev FLUX.1 [dev] is an open-weight, guidance-distilled model for non-commercial applications. text-to-image image flux 1 /path/to/huggingface/that-contains/hub/models--black-forest-labs--FLUX.1-dev | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @beatrizserrano @anuprulez @kostrykin maybe you are interested in this. We are trying to come up with a potential Galax data table for huggingface. So that Galaxy instance admin can download models deposit them locally and register them in such data tables. Different tools can then depend on that table and filter them according to a tiny bit of metadata. |
||
#black-forest-labs/FLUX.1-schnell FLUX.1 [schnell] is the fastest model, tailored for personal use. text-to-image image flux 1 /path/to/huggingface/that-contains/hub/models--black-forest-labs--FLUX.1-schnell |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
<!-- Paths are relative to the value of `tool_data_path` in galaxy.ini --> | ||
<tables> | ||
<table name="huggingface" comment_char="#" allow_duplicate_entries="False"> | ||
<columns>value, name, pipeline_tag, domain, free_tag, version, path</columns> | ||
<file path="huggingface.loc" /> | ||
</table> | ||
</tables> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it really need to contain path to the folder or the path to the model/file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks more like the file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be the folder of HF_HOME