Shane Fu

0:00 [Applause]

0:07 hi everybody thank you it's nice that for inviting me and thank you everybody for coming in so today I'm going to talk about how to creatively direct your AIS and before I get to that here's my show real as you can see I I usually do a lot of vertical CGI animations these design for social media I've worked with a lot of clients in TCH fashion and Automotive Industries helping them with their social campaigns and also out of home digital billboard animations but you're probably wondering this is cool but this is CGI why am I here to talk about AI here today instead I guess it's what everybody's talking about this day but I think you know because I've also be I've I've always been a lazy designer so I learned to be resourceful and if a tool that can make the process faster and ELD better result I will use that tool instead and I think in the recent years generative AI has slowly matures as the tech and you know in my eyes becomes more and more as a designer's tool and I'm also working towards to become a creative director and I think those two Kit can really help me with my journey so I've been doing some experiments over the last couple month with stabil diffusion and here are essentially so so the process is essentially like this I take base animation that I made in 3D and I fed it into Sid diffusion.

2:02 And it spits out something magical this is example of to let AI figure out figure things out using the abstraction that I made but there's also example that you know it can also be more literal where AI serves as a really Advanced filtering some sense and uh along with those experiments came with a lot of attentions they are good attentions which is you know really moving comment and there's also not so good attentions so I think overall it's it's pretty divisive and I'm not sure who's the real keyboard warer here to be honest so so in order not to be like that uh you know I've made some design principles for myself to make me comfortable incorporating when incorporating a in my design process so you know some of the obvious one I want to sketch out the concept first either by hand joints or in 3D and I got to spend more time designing outside of AI than when using it. And I don't want to name any artists or Studio in the prompts I found that really unethical in doing so and I will never respond to rud comments on Instagram so with that in mind let's dive in how does it work exactly well the actual process usually looks like this this this this.

3:47 And then this so you know oh shoot sorry so those are actual screenshots by the way. And I don't want to bore Everyone by going over step by step for like 5 hours so I just want to tell you guys the general thinking process that I integrated in my into my own design and here's the list of the software that I used H since this is the the most commonly asked questions and it sounds like a bunch of gibberish so let's break down each components and in order to do that I'll be using the new animation. And this.

4:34 This is the one that made those people leave those comments so and I've always had this image in my mind that a hand basically tapping into a black hole a void and something magical a magical plant or Greenery element just comes out of it. And in in the style of like I guess retro animations so I made this sketch forgive me for the disproportionate human hand anatomy it's not the best joint you know I that do but I think this is good to give a mental image right of what I want to achieve so I get to work at first I made this hand animation I I used a riged hand animated it.

5:21 And then rended it out in realistic with realistic 3D textures right. And this is essentially the foreground and for the background I want to make this like really magical plant and floral elements just pops out from one point and outwards and I go so since I'm a lazy designer I I I basically go back to a old project I used that that I did and took the grass animation out which is essentially a hair simulation and and then I pop back in with the same perspective with the hand and this is the animation I'll be using for the base animation all right.

6:07 So now we have the animation ready what's the next step now we have to basically let stabil diffusion which is the AI to understand where to draw and how to draw it. And I do that by using the second component which is called control net essentially I can extract from this animation with both depth map which is uh for those of you not familiar with 3D which essentially is a map where the brightest spot represents the the spot that's closest to camera and the dark spot that's like the Fest to camera and this will let AI know the perspective of the image it's dep the depth of the image and there's also the line art map which shows the outline and Contour of the image so basically step division will learn where to draw there instead so now we have that in place let's have some fun with it it's this is the the fun part right.

7:14 So we can fit it in and then use some prompt based on this image and is here is a dito made out of a donut made of dtos right.

7:21 But we need to focus not I don't we don't want to have too much fun right to focus. And we need to figure out how to make this magical plants right.

7:31 So we give it a simple prompt exotic plants growing we're getting close but this is like way too photorealistic for my liking I wanted to have like this really retro anime style right.

7:46 So I added a little bit more contexts and there we have the image on the right.

7:52 So we have the steel frame how do we want to animate it. And this is is essentially where the last component come in I used a motion module called anime diff and essentially within it I can key frame not only the Basse animation. But also the prompt and you can see the numbers on the left are the frame numbers and the the prompts on the right are essentially you know Fades into one another so you can have a whole story to be animated with and you know.

8:29 Now we have the key frame prompt but we can also have the static prompt and this is where you know we have all the sty Styles calization and more context coming that stays consistent throughout the animation so with all those in mind with all the the four components I talk about we can you know we can you know feed this and feed into Ai and then come out for into something like this yeah and before I show you the final animation I want to tell you some challenges I've encountered at first I used smoke simulation instead which yield good result in the other work you've seen but I guess in this case it's just not that good cuz I guess you know with the the AI Tech is current stage is sometimes just doesn't work for some for one reason or another there's a lot of uh reasons and I've also tried combining both you know both both the foreground and background together as one image and fit into it.

9:43 But also with the limitation of the tech right now you can't really specify context with different region of the image so that AI can really understand where exactly to draw and and plan so sometimes the result of the plant ring isn't ideal you know there's not enough floral element that comes out in this case and sometimes even more horrifying things happens where you can see the fingers are popping out on the plan ring which is kind of gross yeah. So so what's the solution well it's pretty simple I just took the one the the the animation that the hand that I like the most and essentially took the alpha mask from the CG render and I masked it out with only the hande element and then overlay it on top of the plan animation that I like the most so here is the result so yeah.

10:57 So yeah this is the this is the final animation and coming back to the software that I listed before only the last two really evolves in you know generative AI with prompting and all that stuff and if you think about it every one of them. Actually serves a unique purpose in the design process it's similar to how animators would would think while animating by hand but I'm not good at that.

11:30 So I came up with my own way so what's the take away from this are we I don't know are going to lose our jobs or something well I'm I'm not in a position to make the predictions but if there's a Ideal World that I will hope to live in I would like to think that AI will continue to exist and traditional art mediums will also continue to exist and if we use both both of them for good both will flourish I think it's always brilliant ideas that Prevail in art and design and AI is just another method to actualize your ideas so in a sense if you're are a designer or college student who want to become a creative director and now you have this leverage to make yourself one so thank you