Sofy Yuditskaya interviews Sneha Belkhale of Kiki Yago.
KìKí Yågø is an art collaboration of the musician and poetess Kiki Yago and 3D graphics engineers Codercat (Kirill and Sne). KìKí Yågø creates music and casts high-speed spells programmed to actively use the multidimensionality of our mind in order to achieve a complex harmonic balance of being in a rapidly developing TECHNO-society. Animated by the studio of computer enthusiasts Codercat, artists and designers creating experiences in the most advanced areas, from virtual and augmented reality to WebVR, procedural animation and generation, KìKí opens up its universe to you, in which the universe of humanity is reflected with an optical effect of fractal self-similarity. Our goal is to question physical reality and push the boundaries of what’s possible through technology and art.
Kiki Yago poetry translated from Russian by Sofy Yuditskaya:
Sofy: Could you walk me through your video making process? There’s three of you, and you all do different parts, and it exists in this hybrid virtual-physical space and there’s hybrid performance in it…
Sne: We met Kiki two years ago in St Petersburg. We had the idea to do this whole Kiki Yago project because we found out about metahumans and Unreal Engine. Metahumans are one of the reasons I decided to go into Unreal Engine in the first place. Before I was working in Unity and when I saw that you could have this realistic animated avatar in your scenes and do these crazy effects on them I was like « oh my God this is everything that I wanted » I spent all this time going really deep into Unreal Engine, it was a month of just being in a warehouse going deep.
https://www.unrealengine.com/en-US/metahuman
So we started collaborating with Kiki remotely. She downloaded an app on her phone called the Face Cap App (https://apps.apple.com/us/app/face-cap-motion-capture/id1373155478) and would basically sing with the iPhone just mounted in front of her while recording her face on a webcam so that we could match the the facial animation with how she actually looked. She would record her face singing these different tracks which are—you’ve heard them, they’re crazy tracks with 50 mile per hour words— she’s just spinning and rapping these words in front of her iPhone. When she was happy with the result, she would send us the animation data over Telegram which we imported to Unreal Engine and mapped onto the metahuman.
And that process is actually what the metahuman developers want you to be able to do, it’s called « virtual production » being able to to sing and just use the iPhone camera to record facial animation, this is a pipeline that they’ve built. We didn’t have to figure out these technical algorithms out but we did have to figure out how to set up our own pipeline in Unreal Engine. Once we got it running it was a really cool development pipeline because she could be in St Petersburg really far from us, but just using her iPhone she could send us an entire facial animation data set which previously required the whole team being in the same room and using a physical facial rig with the dots, tracking markers and cameras.
Now we can just do it with an iPhone. After she sends the animation data to us, we’d compare what the metahuman looks like to the webcam video that she sent us and do a bit of adjustment to the facial data mapping. There were some weird things with the mouth where it wouldn’t close fully, and the eyes would rotate too far. Then we’d send her the videos and she’d have some thoughts and comments, things that work, things that don’t, and then she would send us another recording. We repeated this loop until we got a recording of the whole track that we liked.
Sofy: wow that’s crazy so you weren’t even in the same place?
Yeah. We actually just did the craziest thing just a couple days ago. We found this program called Tails which allows multiple computers to think that they’re on the same network. So we downloaded that program to my computer and to her iPhone. Right now she’s in Sri Lanka, and we could see her metahuman avatar moving on our screen.
Sofy: wow
Sne: Yeah! She was basically live streaming her facial data to us in New York and that was so freaking cool because it was a really immersive way to talk to someone. You can scroll around the avatar, you can move, you can basically position your camera however you like.
Sofy: So then the body can be made out of weird jellyfish bodies and all that stuff you add.
Sne: So in that metahuman the face was completely coming from Dasha. If we wanted to do live tracking for the body, we’d need a Rokoko suit, like the one that Leilani has.
https://www.drome.media/leilani-franco
https://www.rokoko.com/products/smartsuit-pro
Dasha doesn’t have a motion capture suit so we did a combination of things. Mostly getting animations from Mixamo which is an online set of animations which has really basic animations, like « walking » and « idle » and this has been something that is useful for us because neither me or Kirill are animators. We don’t know how to do it by hand, we normally get someone with a suit or do something with AI. So we’ve been trying to come up with music video styles and things that don’t require having a lot of animation.
For that first music video, if you look at it more closely it’s just a walking loop and for the second, the Stream one in the forest, the Baba Yaga one, she has poses but the poses are static. Kirill hand posed the the avatar but there’s no animation so we just do these cuts— really fast cuts between different animation poses so we don’t have to animate between the poses.
Creating smooth human body animations by hand is so hard— it’s really easy to fall into an Uncanny Valley situation. So we came up with different styles, like this stop motion method to add diversity to the movement.
The body animations are mostly hacky like that for the Kiki Yago project, but there are other details we enjoy focusing on, like costume animations and other procedural things that are happening during the videos, that development is all mostly done in Houdini. I’ve been working on some different pipelines to bring geometries from Houdini into Unreal Engine. It’s been a very interesting process of actually starting with a metahuman in Unreal, exporting the body from Unreal to Houdini, and then developing in Houdini different costumes and hairstyles or just abstract things like wires coming off the body. I love creating these unique geometry layers which can be then exported back to Unreal, and they’re able to fit perfectly on the body because it was exported from the engine originally. And then in Unreal I apply materials and do more timing based things for the video.
Sofy: I’m just curious could you have Leilani dance and Dasha sing and then put them together?
Sne: Yeah there’s nothing stopping you from doing that. And now with that remote system I was just telling you about we can be on the same network. I could even have Dasha in Sri Lanka and Leilani and Hamburg with the Rokoko suit and they could be one avatar.
https://codercat.xyz/digital-circus/
https://codercat.xyz/neither-ever/
We did try the Rokoko suit once. We found the one person who had the Rokoko suit in Tiblisi, and he let us just hang and play with it one day and we recorded some fun movements. We didn’t record anything specifically for a video, we just recorded Dasha dancing but that was so fun. She was really hyped to feel her body or, the avatar’s body actually come to life because it’s really just her face most of the time.
Seeing her body motions mapped to the avatar was really exciting for her and me.
I really love this whole pipeline, it’s amazing when you can watch the avatar move and you can recognize the human behind it. People have these micro motions that you can recognize even on an abstract digital character. And then also there’s this cool feedback loop that happens when Dasha looks at what’s happening on the avatar’s face and starts changing and refining her expressions to work well on the avatar. And the avatar is obviously inspired by her, but having this feedback loop that occurs between digital and physical stuff is really cool. It’s really interesting to think about how in the future that’s how we will exist in certain places. We’ll have this sense of self that’s defined by these avatars, and Kiki definitely got a sense of self at some point.
https://codercat.xyz/kiki-yago-live/
I used to actually call Dasha Kiki, that was her name, I think she introduced herself to me as Kiki the first time we met. After we kept doing this project I couldn’t call Dasha Kiki anymore because Kiki was the avatar. The avatar was pulling that name to herself. I felt she was generating her own entity. So now I call Dasha Dasha and Kiki used to be called « avatar ». It was interesting, there was this moment when I was really confused what I should call the human person. I spent so much time with the avatar, I spent probably more time with the avatar at the beginning than Dasha. I told her about this and she said « just call me whatever you want ».
Sofy: I imagine a future where we’re in some virtual space and you see a crazy spider but you recognize it’s look and you’re like that’s… Nathan! [Laughter]
Sne: I guess we will have all these possibilities of how we could look on any certain day but it’s still your movement. Though you can put a scrambler on your movements too……
Sofy: Can you tell me about the imagery? There’s all this amazing forest imagery and these weird creatures. From a purely conceptual perspective what do all these things mean? Where did they come from, why those things and not other things?
Sne: The Stream music video was really inspired by Dasha and her hometown. She’s from four hours north of Omsk so somewhere deep, deep, in the forest of Siberia. She just visited her hometown last year and sent us a bunch of photos of the forest near her hometown. It was so surreal and strange just these really thin trees that are super tall but also really sparse. The trees look like little toothpicks coming out of the ground. She sent us all these photos of the forest by her house. Then Kirill got inspired by it also and he started modeling that forest in Unreal Engine. There’s also a whole story with the swamps and Kiki Yago coming out of the swamp. That’s something that Dasha came up with, it’s from Russian folklore.
There’s a whole story of how the name Kiki came up, it’s not entirely from he Kikimura mythos but I know Dasha is familiar with that story. She also had a song named Stream which was about the allegory of the swamp, and we tried to pull all of those things together in that visual. And some of the visuals there are random—the swamp creatures and stuff. I like creating procedural shapes so when I think of this deep dark creepy vibe I’m just trying to think of how I can embody that in a procedural way. That’s how those weird spiraling attacking plants came out. They were oversized and just very very creepy in the way that they move because they move behind trees and attack you from the corner of the frame, we thought of them as basically the Kikimora. Kiki Yago, she’s the keeper of that swamp area. She controls everything in this forest and so we’re thinking of geometries that would support that idea.
Sofy: Can you tell me about the other piece?
Sne: There’s a recent one with Dasha and Kiki speaking to each other in a white void. That was a really fun one to film because we were doing all this work remotely and then Dasha came and visited us in Tiblisi for three months. That’s when we did the live shows and finally got to do iterate on this process together. It was fun to do the face cap developing remotely but then having her actually in the room, the feedback loop is just so much better. And now we had the possibility of doing shows and more physical virtual production work. We wanted to have a scene where Dasha and Kiki are speaking to each other—it’s a whole story—we went around these ideas so many times just trying to imagine Kiki’s relationship to Dasha the physical human, because in all the previous clips Kiki was just by herself. We wanted to introduce them in the same space and explore what they would need from each other and their relationship.
Sofy: Did you find out?
Sne: Well we made a story which is funny but basically there’s an entity who wrote some of the lyrics for Kiki Yago and it’s walled in somewhere.
We had this idea that the entity is sitting in a void but then there’s this little bird that flies through the window and gives them a microchip and they’re able to somehow swallow or absorb it and it uploads them to the metaverse where they become a metahuman and that human is Kiki. So Kiki is then in metaverse and this is the whole story of Kiki. Also that is the reason none of the lyrics of her songs make sense or feel like it’s a lot of terminology and different languages mixed together.
Imagine a person that just got all the information of the world in a couple milliseconds this is how your brain would be— you wouldn’t be able to form coherent sentences but you have a lot of information that you want to express. That’s Kiki.
We’re thinking that the entity has been uploaded as Kiki but then Kiki needs to actually have a plan to free herself in the real world, she needs a free Dasha so she creates a human which is Dasha. She creates a human that looks like herself and then also trains the human or there’s a mutual training that happens where they’re are able to teach each other how to speak and how to operate on Earth. And then she spawns Dasha on Earth with the mission to free her. That training sequence that’s actually Kiki training Dasha.
Sofy: That’s the scariest thing I can imagine —-not being able to move around freely for that long.
Sne: Yeah, that’s the defining characteristic of our lives —moving around— that’s what we to do. I mean us specifically through virtual space and physical space and around the world so yeah, it definitely brings a really interesting dimension to everything, having someone who’s not moving around at all be part of it. And that for me is still a mystery because I’ve never met or spoken to this person but I know them in some way because I’ve just been reading their lyrics for the last year and there’s something that definitely comes through. And also just hearing about them from Dasha but it’s the mystery component for me of this whole project. Is she good? And who is they? And where are they? What part of the country might they be in…
Dasha: Starting with this video “Mother Taiga” a new member, joined our team. Alexander Doronin or She Monkey a multi instrumentalist, who collaborated with Dasha for years in a band named Weo. They wrote the song Mother Taiga 7 years ago when they just met. We find out this is perfect music for Kiki and the idea about her birth and language learning.
Sofy: Are you hiding any messages in the digital world and if so, what are they? What’s the secret hypnosis telling us to do?
Sne: I need to have some secret— somehow overlay my face on some objects in the background. I feel there’s a lot of voids and this is something that I don’t know if I put into things intentionally but it just keeps happening. I keep thinking of these feelings of going into nothingness just going through that spiral at the end of the Stream. There’s a Stream that comes and picks you up and then you’re just falling through this other Stream of squiggly avatars for too long [Laughter] for 30 seconds or something and even in the end of Siroi Xa! everything kind of disintegrates and you’re in this really long tunnel and then everything dissolves and Kiki’s clothes come off and she just dissolves into a black void with neon tubes.. I feel it’s some existentialism [Laughter] seeping through most of the music videos and clips.
It’s interesting how in this medium the existentialism is the « game over » idea, it’s you just turning off the TV. They blend together. Before, I never thought of it as existentialism I thought of it as finishing the game—but I can turn it back on on my end. For that world, the world ended. And we’re just that world and they’re other worlds and there’s generally another world. At the end of everything disintegrating there’s something else that you encounter, you have to go through the disintegration part and the feeling everything has just disappeared to find this other in the end of the Stream. It’s a big server room which I really liked. We just wanted something that was the most contrasting. There’s this beautiful, creepy forest that feels natural, and then we transitioned to the most contrasty walls of black servers—it’s computers dreaming. We were very inspired by a digital sound effect at the end of the track. It doesn’t matter if it doesn’t make sense this is going to be a server room. The Kikimura community also needs to go on the internet. They need to store their data somewhere.
It reflects our process— without requiring an entire film crew to make this work we can have a really refined aesthetic because it’s only three people. But the output is equivalent to what a film crew would do. Being able to do it with just the phone versus having a bunch of equipment is perfect for me and Kirill because we travel with just our laptops and that’s it. And with just a laptop and iPhone we can create a fully motion captured music video.
Sofy: The traveling thing is interesting because 3D work is a very heavy process [in terms of gear needed and computing power] but you have this traveling lifestyle. How did those two things come together?
Sne: The backpacking thing started six years ago. At that time I was doing computer vision and AR, but all my work was on the computer at that point and it was this realization that made us start this journey. We were both working in San Francisco and we were spending eight hours in an office, and in this office we were completely on our computers the entire day and me and Kirill and this other person we had this group realization.
At that time, with digital nomad culture just starting, Kirill and I decided to adventure and travel together with only backpacks and our laptops. At that point I only had a Macbook and I was doing more web development: 3JS, so still graphics, but it was on the browser. Back then I had a job with a VR company and I needed to buy a better laptop. I needed a Windows machine because I literally just couldn’t run the project if I didn’t have it. So for some time I was traveling with two laptops. I still had my MacBook because I just couldn’t let go of it. It’s just so lovable. It’s is my home, the home directory on my MacBook. And then I got the Windows and I was traveling with both of those computers and their chargers and the headset— my backpack was 90% technology.
About the interviewer:
Sofy Yuditskaya (@the_s0urce) is a site-specific media artist and educator working with sound, video, interactivity, projections, code, paper, and salvaged material. Her work focuses on techno-occult rituals, street performance, and participatory art. Sofy’s performances enact and reframe hegemonies, she works with materials that exemplify our deep entanglement with petro-culture and technology’s affect on consciousness. She has worked on projects at Eyebeam, 3LD, the Netherlands Institute voor Media Kunst, Steim, ARS Electronica, Games for Learning Institute, The Guggenheim (NYC), The National Mall and has taught at GAFFTA, MoMA, NYU, Srishti, and the Rubin Museum. She is a PhD Candidate in Audio-Visual Composition at NYU GSAS currently on her Fulbright year at TheISRO in Bangalore
