Mini Shading and Real-Life/Toon Combination Tutorial (Using Photoshop)

Pics are numbered in the order they were done. The letters represent tools and windows of note.

1.
Get the photo and the color/ink layers for your picture. In my case I combined the inking and color into one layer and the photo was in another. I created a shading layer (on top of the color/ink layer) and then used the Magic Wand tool (B) on the ink/color layer to create a selection around the character, inverted the selection and created a layer mask for the shading layer. (Photoshop automatically turns the current selection bounds into a layer mask for you.) This way you don't have to worry about shading outside the character.

Use the paintbucket/flood fill tool (G) with contiguous mode turned off with the color black at some percentage of opacity. (Because this is a bright picture I used 30%, but its more common to use 50%.) The light rectangle in the middle of image 1 shows the difference between the pure colors and the darkened shading layer. You can see the results of the layers created in window J. This is your mid-shading point. I guess I should point out when you color the character try to use bright colors since you'll be darkening them at this step. If you need black try using a dark gray instead. (I think I was still too dark in my gray.... >_> )


2.
Because this image had a lot of harsh lighting I decided to make a layer showing the layout of how I want the highlights to run. This provides both a layout for the highlight layer and tells me where I need to remove the darkening I created in step 1 in the shading layer.


3.
First important step - set the history brush marker to where you are now in the document history (I). This is a lifesaver because if you mess up you can use the History brush (E) to get the messed up portion of the shading back to where it was without having to undo and redraw everything to the point you were at.

Now work the shading layer. Set the color to pure black. I use the brush tool (D) at 100% opacity with 1 or 2% flow. I also set the eraser tool (F) similarly. The heal brush ©, Blur/Smudge tool (H) I leave at 100% for both settings. The History brush (E) I set to 100% opacity and 4% flow. Brush sizes will vary depending on what parts of the character you are shading. For instance, for getting the large gradual shadings on the thighs, tail, and torso I used a 100 pt brush. For things like fur tufts and fingers, you'll need to go smaller. (Though you can still use a medium-large brush if all you're doing is shading the edges because the layer mask will keep everything inside.

Use the brush tool to darken, the erase to lighten. You can also use the history brush to lighten or darken if you feel you've gone too far in one direction.

I wish there was a magic way to teach how to shade the dark and the light to create the illusion of light casting its rays, but there isn't one I know of >_> <_<. I used the highlight layout layer as a guide here, and you can do the same. I also sometimes create a hard-shaded layer just to try out different shaded areas to see what looks right, then I create the real shading layer and use the temporary one as a guide. I also know setting the flow rate so low means it's going to take a lot of strokes to get any effect, but I find this is easier than making it too dark or too light.

HOWEVER, I've also found that when the flow is set really low, the brushes never reach fully opaque or fully transparent on the shading layers. So if you have been going over and over an area and it doesn't seem to be changing, try upping the flow rate temporarily and see if that makes a difference. You can use the Blur and/or smudge tool (H) to fix any harsh lines in the shading.

Shading interior appendages: You can either 1) Just shade it as is and go back with a small history brush to remove the newly-created shading from areas you don't want, or you can use the Magnetic Lasso (A) to trace over the outline of an arm, toe, or finger and then only that part will be shaded.

USE THE HEAL BRUSH ©! When you're shading a large area with smaller brushes, you'll sometimes get a lumpy effect since us biological beings aren't usually as precise as computers when putting down strokes. Just stroke an area you want to remove the lumps from and the heal brush will do a pretty good job of smoothing it all out. Then you can go back and add detail again. Also use the heal brush if you've shaded an appendage with a selection as described above and have a noticeable line where the new shading and old shading meets. If the healing brush is not working out (or you don't have one) you can also use the Smudge tool (H) to repair lumps.

If you start seeing a lot of noise in the shading, use the Blur tool to get rid of the noise.

Picture 3 is the result of the end of step 3. The highlight areas have been cleared of their shading, and most of the shading of the dark areas is done. However as you go along you'll probably need to tweak things a bit (as I always do) so don't be afraid to go back to the shading layer and make changes if things aren't looking as good as you hoped.


4.
Create a new layer for highlighting. Usually my smooth-shaded pics have only a little bit of highlights and most of the work is done my the shading layer. However because of the bright conditions, the highlights play a major role here. For the light fur areas I picked a color that matches bright sunlight. For the red areas I picked an orangish pink color.

The rules here are pretty much the same as the shading layer, but since I needed a glow effect outside of the character lines I didn't copy the layer mask into this layer so I needed to go with a smaller brush all around and then do cleanup with a small eraser brush (again, with flow set to 1%).

For this layer I pretty much used the lighting layout layer to guide my actions. It's a lot easier to preplan something than to have to go back and redo a layer because it didn't turn out the way you expected.

It's a bit of an art to figure out what is okay by simply erasing the shading layer completely vs. adding actual hightlights to a section. Most of the highlights here are added, but the highlight on the cheek under the eye closest to the viewer is simply the removable of the shading completely from that section. I had to set a higher-than-usal flow rate to get it to clear completely.

Picture 4 show the end results of step 4.


5.
Exterior shadows. In addition to the cast shadows that fall inside the character, you'll need to create a layer for the shadows the character casts onto its surroundings. Create the layer BELOW the color/ink layer so you don't have to worry about creating additional shadows inside the character. (You already finished with interior shadows in the shading layer, yes? ;) ) In this case it's the chair. Here I screwed up a bit. >_< I should have taken two identical photos - one with the sun shining through and the other with the sun blocked so I just erase the bright sun layer photo and have the blocked sun layer positioned below appear to act as my shadow. Instead I had to shade everything around the character like I did the character himself, then use the stamp tool to put the pattern back in areas were the utter brightness obscured it completely.

As you can see from picture 5, there isn't much to the exterior shadows. They mostly run around the bottom edges. The cast head shadow and the area around his back had the most subsantial work. I decided to include the shading and highlight layers in this view as well just to make it clearer how the exterior shadows interact. Remember, they're hidden behind the color layer so you're actually seeing a lot more of them than actually appear in the final picture.

If your character is interacting with the background in any way, you can tweak it's appearance on this layer if you like. For me, since it was a black chair this was convenient. I had to add some extra wrinkles to where his left foot is touching the chair to make it appear like it's pressed in. Also I added some highlights there to make the wrinkles more visible. If the interaction is greater, you may need to use the warping filter to actually warp the pixels in the photo to match up with your character's interaction.

6.
Final picture! Sign it and you're done! ^_^