Update README.md
Browse files
README.md
CHANGED
@@ -18,28 +18,48 @@ base_model: stabilityai/stable-diffusion-2-1
|
|
18 |
inference: true
|
19 |
---
|
20 |
|
21 |
-
<!-- This model card has been generated automatically according to the information the training script had access to. You
|
22 |
-
should probably proofread and complete it, then remove this comment. -->
|
23 |
|
24 |
|
25 |
# LoRA text2image fine-tuning - remi349/sd_trained_3D_lora
|
26 |
-
These are LoRA adaption weights for stabilityai/stable-diffusion-2-1. The weights were fine-tuned on the remi349/finetuning_dataset_for_3D_training dataset. You can find some example images in the following.
|
27 |
-
|
28 |
-
|
29 |
|
|
|
30 |
|
31 |
## Intended uses & limitations
|
32 |
|
|
|
|
|
|
|
33 |
#### How to use
|
34 |
|
35 |
```python
|
36 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
37 |
```
|
38 |
|
39 |
#### Limitations and bias
|
40 |
|
41 |
-
|
42 |
|
43 |
## Training details
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
44 |
|
45 |
-
|
|
|
18 |
inference: true
|
19 |
---
|
20 |
|
|
|
|
|
21 |
|
22 |
|
23 |
# LoRA text2image fine-tuning - remi349/sd_trained_3D_lora
|
|
|
|
|
|
|
24 |
|
25 |
+
These are LoRA adaption weights are for stabilityai/stable-diffusion-2-1. The weights were fine-tuned on the remi349/finetuning_dataset_for_3D_training dataset.
|
26 |
|
27 |
## Intended uses & limitations
|
28 |
|
29 |
+
This model aims at generating images of isolated objects, compatible with 2D_to_3D models like [Triposr](https://github.com/VAST-AI-Research/TripoSR) or [CRM](https://huggingface.co/Zhengyi/CRM).
|
30 |
+
It was finetuned in order to create after a pipeline of prompt-to-3D model.
|
31 |
+
|
32 |
#### How to use
|
33 |
|
34 |
```python
|
35 |
+
# first load the basic architecture and everything
|
36 |
+
import torch
|
37 |
+
import os
|
38 |
+
from diffusers import StableDiffusionPipeline
|
39 |
+
pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16)
|
40 |
+
|
41 |
+
# Then add the lora weights to the model stable diffusion 2
|
42 |
+
pipe.unet.load_attn_procs('remi349/sd_trained_3D_lora')
|
43 |
+
pipe.to("cuda")
|
44 |
+
|
45 |
+
# Then you can begin the inference process on a prompt and save the image generated
|
46 |
+
|
47 |
+
prompt = 'a rabbit with a yellow jacket'
|
48 |
+
image = pipe(prompt, num_inference_steps=30, guidance_scale=7.5).images[0]
|
49 |
+
image.save("my_image.png")
|
50 |
```
|
51 |
|
52 |
#### Limitations and bias
|
53 |
|
54 |
+
This model is a first try some hyperparameters tuning should be done, but for that we would need a solid automated benchmark.
|
55 |
|
56 |
## Training details
|
57 |
+
The model finetuned model is [Stable Diffusion 2](https://huggingface.co/stabilityai/stable-diffusion-2).
|
58 |
+
The data used to train this model is the dataset available on uggingface at 'remi349/finetuning_dataset_for_3D_training'.
|
59 |
+
you can download it thanks to the command
|
60 |
+
```python
|
61 |
+
from datasets import load_dataset
|
62 |
+
dataset = load_dataset("remi349/finetuning_dataset_for_3D_training", split = 'train')
|
63 |
+
```
|
64 |
|
65 |
+
This dataset is a subset of the dataset [Objaverse](https://objaverse.allenai.org/).
|