Young Poseidon Wrestling a Sea Dragon
Control Net Series
The workflow I used to create these images is a bit of a not-so-closely-guarded secret that many creators use: make a render in Midjourney to drive the image creation in Stable Diffusion.
The reason you might want to do this is because Midjourney is a truly imaginative engine. It has a lot of content restrictions, yes, but for a simple 2-3 line prompt, you can create really good compositions out of the box without a lot of tweaks.
When I talked to creators, I know that most of them use the image they create in Midjourney to do img2img, but personally I have found that it is far better to use control nets for this purpose. I have produced several paintings done using control nets before, and you can see them in the sets about wrestling:
The wrestling paintings started their lives with color paintings and photos, because that’s what the sources are. For “Grab,” it was based on an existing photo. For “Submission” and “Behind,” those were paintings I have made in Midjourney previously.
But if I were to start from scratch knowing that I intended to use the source only to drive composition, then my go-to is always the Line Art control net. Why? Because it gives the best results amongst all other control nets.
Before going with this flow, I actually did try text prompting inside SD only, but the poses are simply not very imaginary. Trying to get the figures to do an action pose is limiting using text prompts alone — but not so much inside Midjourney.
Additionally, I wanted to depict a young Poseidon with a face of a young man but other attributes as the Poseidon that you have come to known in classical paintings — long hair, thick beard, muscular.
Inside Midjourney, I prompt for a black and white pencil drawing of Poseidon fighting a sea dragon. I have other renders, but so that I could illustrate these with examples as closely as I could, I chose to show only one of them.
Images
I then wrote my description of what I wanted to depict inside txt2img, and put the Midjourney render inside the Line Art control net with these parameters:
- Weight: 0.8. I want some variations between images.
- Res: 1024. Always pre-process at 1024 minimum. You can go higher if the source is higher resolution, but it can be expensive memory wise.
- Start: 0. Start influencing from the first step.
- End: 1. Influence til the last step.
- Mode: Balanced. Highly dependent on the image, and depends on how much you want to change from the line art. I have other better examples of this, and I will post those examples in the future, where I used the text prompt to drastically change the image while keeping the pose.
These images were not edited extensive because I mainly wanted to show the flow. It would take a long time to manually edit all the hands and feet for posting. But I want to highlight the following:
- Poseidon wears a tshirt in the line art. These paintings follow the clothing described in my text prompt.
- In the lien art, Poseidon is looking down at the dragon. I asked SD to have the figure look at the camera instead.
- Colors can be changed drastically from line art. If you start with a color image, the control nets tend to pick up on color information even if you use a technically black and white process e.g. Soft Edge.
- The line art is flat, but the SD renders are all perfectly three-dimensional.
I hope that this gives you some ideas of how you could make use with the control net to make some great creations. If you don’t have Midjourney, SDXL is an excellent option as it is equally imaginative. I used Midjourney for these mostly because it is faster for me to generate ideas. I have a M2Max locally and it‘s simply not fast enough to generate SDXL images at a speed that’s comparable to Midjourney.
Line art: Text prompt in Midjourney v5.2
Paintings: Text prompt in Stable Diffusion, Automatic1111 v1.6.0
- 20 steps, Euler a, 512x512
- Denoising: 0.5
- Airfuck’s Brute Mix v1, Clip skip: 2
- CN0: Line Art inverted, Weight 0.8, Res 1024, Start 0, End 1, Balanced.
- Hires: 2x, 1024x1024, 10 steps, 8x_NMKD-Superscale_150000_G
- Post: Topaz Gigapixel HQ 4x, 4096x4096
- Post: Adobe Lightroom color correction
See Technical Detail for full tech specs.
Images
Images
Additional Images
Images
Technical Detail
- Steps: 20
- Sampler: Euler a
- CFG scale: 5
- Face restoration: CodeFormer
- Size: 512x512
- Model hash: 8ff8347bb0
- Model: airfucksBruteMix_v10
- VAE hash: 235745af8d
- VAE: vae-ft-mse-840000-ema-pruned.vae.ckpt
- Denoising strength: 0.5
- Clip skip: 2
- ControlNet 0:
- Module: invert (from white bg & black line)
- Model: control_v11p_sd15_lineart (43d4be0d)
- Weight: 0.8
- Resize Mode: Crop and Resize
- Low Vram: False
- Processor Res: 1024
- Guidance Start: 0
- Guidance End: 1
- Pixel Perfect: False
- Control Mode: Balanced
- Hires negative prompt: nipple
- Hires upscale: 2
- Hires steps: 10
- Hires upscaler: 8x_NMKD-Superscale_150000_G
- Lora hashes:
- add_detail: 7c6bad76eb54
- polyhedron_men_eyes: 63e74b4b613e
- bulgerk-dickprint: b1934ecdac9a"
- TI hashes:
- bad-hands-5: aa7651be154c
- Asian-Less-Neg: 22d2f003e76f
- negative_hand-neg: 73b524a2da12
- NegfeetV2: df90b1ff666d
- face_editor_enabled: True
- face_editor_face_margin: 1.6
- face_editor_confidence: 0.97
- face_editor_strength1: 0.3
- face_editor_strength2: 0
- face_editor_max_face_count: 20
- face_editor_mask_size: 0
- face_editor_mask_blur: 12
- face_editor_prompt_for_face: look at camera, brown hair, long thick beard, handsome greek face, - strong facial expression
- face_editor_apply_inside_mask_only: True
- face_editor_apply_scripts_to_faces: False
- face_editor_face_size: 512
- face_editor_use_minimal_area: False
- face_editor_ignore_larger_faces: True
- face_editor_affected_areas: Face
- face_editor_workflow: {"face_detector": "RetinaFace", "rules": {"then": {"face_processor": "img2img", "mask_generator": {"name": "BiSeNet", "params": {"fallback_ratio": 0.1}}}}}
- face_editor_upscaler: None
- face_editor_tilt_adjustment_threshold: 40
- Version: v1.6.0