Feb 20

Feb 20 What I learned when I let AI render my hand sketches (2024)

I frequently use both hand sketching and CAD modeling in my engineering design work. My general purpose for hand sketching is to explore a variety of concepts, while my purpose for CAD is to create a database of precise geometry that can be used for creating engineering drawings, digital manufacturing, and photo-realistic renderings.

A recent addition to the growing body of Artificial Intelligence (AI) tools leap frogs the CAD modeling and uses hand sketches to produce photo-realistic renderings of an idea. For example, even a rudimentary hand sketch and a text prompt can produce the following.

The total sketching, uploading, prompting and rendering time was less than 5 minutes for this image. Its worth noticing that the AI captures most of the detail, even the odd ones such as the slope of the frontal grill-headlight area, and the mis-orientation and size of the wheels. Also notice that AI adds some features, such the brake light, the detail down the center of the hood and a tiny passenger side mirror.

The AI output is interesting, more advanced than I expected, and surprisingly able to have interpreted my 1-minute sketch of a car correctly.

Trying to produce a great rendering with AI

The AI did a good job producing a nicely colored well-lit rendering using only a 1-minute car sketch, and basic text prompt. I iterated with the AI only once, first asking for a “black 1960’s sports car”, then asking for a red one. The AI output to human input ration was superb for that demonstration. But what happens when I give the AI a 15 or 30-minute sketch? And what’s the best I could get out of the AI system? Below are my results, compared to my CAD rendering that did not use AI (last rendering) .

For the 15-minute sketch, it is interesting to note that the AI struggled with what to do with my center line. Also it had some difficulty knowing that the hook at the top of the lantern was supposed to be solid. It did, however, generally do an amazing job knowing what was supposed to be opaque versus translucent, and it chose correctly to make the bulb glow.

When the AI used the 30-minute sketch, I don’t believe it produced anything better than when it was based on a 15-minute sketch. I dedicated more time to trying to produce a rendering that was more consistent with my vision, and for that, the rendering is closer to what I envisioned. But I couldn’t get red button to be opaque and not glowing. And I couldn’t get the bulb to glow, despite reasonable efforts.

While this particular example demonstrates how AI renderings compare to a human-made rendering, it does not capture the main role that I believe AI can play in the design process. Ultimately, I prefer my CAD rendering in this case, because it represents exactly what I wanted. Trying to get AI to give me what I had envisioned in my head was a generally frustrating experience.

Role of AI in co-Creation

The generative AI experience does not need to be frustrating, or low value as discussed above. Like all tools, the designer must know when to use it. I believe AI has four distinct roles in the design process.

To rapidly contextualize a human-made design. For example, consider the contextualized camping lantern rendering shown below. Compare this to the CAD rendering in a white studio above. Contextualizing the CAD render, as shown below was a 5 minute effort. I believe this is a valuable contribution by the AI.

To rapidly explore/represent a new idea. For example, consider the heated lunch box with removable battery shown below. From sketch to render was a 10 minute activity — including a good discussion with a potential user about heating food during the work day. I generally like the results, which I attribute partially to the fact that I was open to the details chosen by the AI. I had not already chosen what I wanted, as I had with the lantern. So I spent much less effort trying to get the AI to make it the way I wanted it to be, and instead accepted that the AI had design freedom it was working with.

To iterate or demonstrate iteration. Consider the drill iterations/variations shown below. The hand sketch is a 10 minute sketch, and the iterations took about 10 more minutes in total. I believe this level of variety is exceptional for 10 minutes of effort.

To bring sketches to life. For the examples below, I asked the AI to stay true to my sketch. As a result it essentially rendered my sketches, which I believe is a useful contribution by the AI.

Tips for using This kind of AI

To be clear, I used the AI program called Vizcom [1]. This impressive browser-based rendering software uses two designer inputs to generate a rendering; a sketch and a text prompt. The AI’s use of a hand sketch is what sets this software apart from other generative AI, and what makes it useful for product design. Like all interactive AI systems, constructing a useful prompt is essential. With Vizcom, the user decides how much influence the sketch has versus how much influence the text prompt has.

If I were using Vizcom for the first time, this is what I would want to know:

Don’t spent time trying to get AI to create what you envision in your head. Allow the AI some freedom, and accept that its value is in generating ideas and appearances that are new and interesting to you. If you have a specific render you want, you will likely find yourself frustrated by AI’s inability to give it to you.
Iterating through prompts and levels of sketch-influence can be both exhilarating and frustrating. It takes practice, skill, and time. I generated all the images in this article with very little practice and very little skill. All in all, generating the images in this article took more time than I had wanted it to — because of the number of iterations I carried out.
To get the best results, provide a clean sketch with easy to see lines, connected lines, and no other distractions on the page (no wrinkles, no scribbles, etc.). Construction lines will cause significant problems because the AI doesn’t know they’re construction lines. It interprets it as geometry as shown by the washing machine example below.
Shade your sketch to help the AI know what is solid material and what is open air.
Do not include any text. The AI simply can’t deal with it well (yet).
A well constructed prompt is important. The more information you provide the better. Vizcom weights the first words more heavily than the last words, and distinct ideas can be separated by commas.
Different parts of the sketch can be the focus of the generative design, including the background. This is one way to take a render you like, and tell the AI to generate a background, as I did for the camping lantern in the forest above.

Beyond 2024

I have added the year 2024 to the title of this article because I expect this topic to change significantly in the coming years. I believe generative design through AI can already be useful for early stage design exploration, design representation, and primarily as conversation starters with potential users, clients, and others. Particularly interesting is the opportunity to co-design with a client — to sketch together, write text prompts together, and through that process discover latent needs and preferences.

For the refinement that comes with later stage design, I don’t believe the AI is ready to make useful contributions to the design itself. Besides, in later stage design precise geometry is needed including a CAD model, which the AI cannot yet produce. That said, I do believe that the AI can take a rendering from a CAD model, such as the lantern, and contextualize in a very valuable way.