To Japanese readers: 日本語で読みたい方はこちらをどうぞ.
Since the end of 2017, I have been involved in a project with one prominent flamenco dancer. Why flamenco? It’s an attempt to see how AI can be used to extend flamenco as a form of dance. This project is led by YCAM (Yamaguchi Center for Arts and Media), Japan’s leading museum and research institute for media art. (Please note that this article is a review of the project from the author’s personal point of view. I do not represent YCAM.)
Flamenco dancer Israel Galván has been called a revolutionary in the world of flamenco. He is a choreographer, a dancer with style with minimalism and physicality that is very different from the traditional flamenco. (I’m sure I’m not the only one who has a stereotype of flamenco where a beautiful woman dances with a rose in her hand while clicking castanets…) You may find more similarity between his flamenco and modern dance or modern ballet than with conventional flamenco. You can see it well from the interspersed use of the word “controversial” in the Wikipedia article on him.
At first, I had no clue, “Why, flamenco? Why would I do that?” But when I met him for the first time, he told me that “flamenco is music,” and that changed my perspective. I had a strong impression of the elegant upper body movements in flamenco. According to Israel, the key to flamenco is the Zapateado (foot rhythm like tap dance). If the Zapateado and Palma (handclaps) are the rhythms, then the hand movements are the “visual melody.” It is flamenco that conveys emotions as multi-sensorial music that integrates the rhythm of Zapateade and Palma, the visual melody of the body movement, guitars, and songs. Once I realized it, my direction became more explicit: I could think of this project as a music generation project with AI.
He is also known for rarely dancing with other flamenco dancers. According to him, the reason for this is that when you dance with other dancers, you “tend to stay in the existing conventions.” Instead, he dances with a giant rocking chair as a fighting bull (above) or with an unstable chair with one leg cut off. In one of his pieces, “La Casa(2006)”, he incorporated the movement of the shelves in the house into his dance (video below). In order to break down the stereotype of gender, he even disguises himself as a woman.
While these objects physically respond to his movement in some way, of course, these objects don’t have any notion of flamenco rhythm patterns. Nor is it just exact echos of Israel’s rhythms. These physical objects react to Israel’s movement in somewhat predictable ways with some randomness and contingency.
What is important is the delicate balance between predictability and unpredictability, and the ambiguity that leaves open room for Israel to reinterpret.
It was in 2017 that YCAM began to approach him. YCAM didn’t have the idea of using AI from the beginning, but in the course of discussions, the use of AI was proposed in response to Israel’s desire to “dance with his alter ego” and “have a buddy like Sancho Panza for Don Quijote.”
Around that time, I was working on my AI DJ project at YCAM. It was very natural for me to join this dance project. Needless to say, there was a lot of overlap with AI DJ project, which started out with the idea of “playing DJ alongside with my alter ego DJ.”
The delicate balance between predictability and unpredictability, and the ambiguity that leaves open room for the dancer to reinterpret.
Nevertheless, once the project actually started, we faced a lot of different problems. Unlike common music, in this case, there is no digital data at all available for the training of AI. The first challenge for us was to make a custom flamenco shoes with sensors. YCAM InterLab, a research branch within YCAM, managed to build the shoes with piezo, gyro, and pressure sensors, and start collecting data on Israel’s Zapateado. The shoes can collect data at three points in his foot: the heel, toe, and belly of the foot.
I won’t go into detail here, but the creation of the shoes for this data collection was way more challenging than we initially thought. The first shoe we made broke in three seconds when we asked Israel to try because the pressure of Israel’s step was too intense (graph below). The second technical challenge was how to increase the temporal resolution (sampling frequency) to record his zapateado. It turned out that he taps more than 10 steps per second at its fastest. We also had to make the shoe entirely wireless to use them in the performance.
In flamenco, a compás, a 12-beat-long loop, is a basic unit of musical development. We had a hard time getting Israel to put on the data collection shoes and dance different compás over and over again.
We implement a model that generates zapateados by using the data collected over these painstaking efforts as training data. Using the wireless shoes to take Israel’s steps as input data, our zapateado generation model will predict and generate the next development leading to the input.
As for the generation model, we tried various models used in music generation, including one that uses RNN/LSTM to estimate the next step from the previous sequence of steps. Finally, however, we adopted an architecture called the Variational Autoencoder to generate two-compás-long steps(24 beats). The architecture of the model is similar to the one of MusicVAE of Google Magenta.
The model learns whether or not there is a hit in each grid of quantized beats (onsets), the strength of the hit (velocity), and the deviation of the hit timing from the grid (timing offsets).
In the performance, data from Israel’s last 2 compás is read by the shoes and fed to the model to generate the next coming comás. To make generated step patterns into audible sounds, we made simple machines that use solenoids to hit the wooden floor. We could use software synthesizers and samplers for this purpose, but we decided not to. We wanted to put physically moving machines on stage, so that the audience might feel the existence of invisible AI, without “anthropomorphizing” AI too much. (The solenoid machines were created by my old friend, Kanta Horio)
In the creation of this project, I went through the process of collecting data, training the Zapateado generation model, and validating the results over and over again. However, when I let Israel hear the results of the generation of the learned model, he always told me:
“It’s too flamenco-y.”
“I don’t need a degraded copy of myself.”
His comments had me tearing my hair! I trained the model with data of Israel, but at the same time, I needed to deviate from them. It was the biggest challenge throughout this project.
I tried various things: to mix traditional flamenco rhythms (e.g., Bleria and Sigiriya) and Israel’s freestyle rhythms in training / to take a generated pattern as a probability distribution and randomly sample from it / to emphasize or invalidate the deviations from the grid (time offset).
How to deviate moderately from flamenco-ness while learning the steps of flamenco
As for random sampling, you can think of an irregular roulette with big holes and small holes. If you roll roulette and throw a ball in, you’re naturally more likely to fall into a big hole. If you repeat it over and over again, the probability of a ball falling into that hole should be proportional to the size of the hole.
What the rhythm generation model does is similar to creating this uneven roulette. For example, the first beat of the compass has a very high probability of a hit, so it corresponds to a huge hole. If there are a total of 10 hits in the compás, then if you throw 10 balls, they should be distributed in the compás according to the size of the holes, i.e., the probability that there are hits as predicted by the generative model.
Here, what happens if we try to make the larger holes bigger and the smaller ones smaller? This should lead to a less surprising result. On the other hand, if we even out the difference between large and small, we can produce a highly unexpected result while following the prediction moderately. Completely eliminating the variations in the hole size will result in totally random outcomes. The parameter that controls this randomness is called temperature and is often used when using these generative models. We manipulated the temperature parameter as the scene unfolds during the performance.
Finally, our performance, “Israel & イスラエル” had its premiere on February 2, 2019 (イスラエル is Israel in Japanese). The title hints at the existence of the “two” Israels, Israel himself (Israel) and the AI alter ego of him trained and made in Japan (イスラエル).
After the first performance, Israel came to me with a satisfied look on his face and said to me:
“I felt like there was an unknown creature on stage, not a musician, not a flamenco dancer, not a human being in the first place. It was very inspiring.”
“I sometimes forgot I was dancing alone.”
It was the best compliment for the team and me, as the project was intended to stimulate his own creativity.
I imagine that he found the balance between predictability and unpredictability of our AI system unique and exciting, and it stimulated his instincts as a dancer.
I felt like there was an unknown creature on stage, not a musician, not a flamenco dancer, not a human being in the first place. It was very inspiring.
“I felt like there was an unknown creature on stage, not a musician, not a flamenco dancer, not a human being in the first place. It was very inspiring.”
During the rehearsal sessions, he was reluctant to dance with AI, which made the production team quite nervous. Now I understand why so. He wanted AI to “surprise” himself in the production, in front of the audience. I’m sure that he is confident enough to properly respond to anything coming out from the AI model.
I don’t care if I fail. I want you to surprise me.
Six months back from production, in the summer of 2018, I flew to his hometown of Seville with the staff of YCAM InterLab to visit his studio. The main objectives were to deliver the shoes to him for data collection and to facilitate discussions about the direction of the performance.
During our time off, we walked the streets of Seville. We even went to a show to get a better understanding of the flamenco culture. The show held at a special venue called” tablao” was great by itself, but what impressed me the most was a scene in a bar we came across by chance. A circle of singing and dancing flamenco spontaneously arose there. In the center of the circle, there was an old lady singing as she has been doing for decades. When the lady stopped singing, to my surprise, a 20-something girl in front of me stepped forward and started dancing.
The dance and song continued without any particular arrangement in advance. I witnessed a world that is only possible because of the unwritten rules shared by those born and raised in flamenco culture.
How difficult it is to bring a whole new set of values to the traditional flamenco world that is so firmly rooted in. It was the first moment I realized that, even though I was and am an outsider. I believe I caught a glimpse of Israel’s mindset that made him want to try dancing with machines.
This project is currently in progress: after the first performance in February 2019, we performed at the Maison de la Culture du Japon in Paris for three days in October. (Shout out to the generous support of the Maison.) I personally would like to continue to develop and update the generation model to reflect the current state of AI technology.
I hope that the current situation caused by the new coronavirus will be over as soon as possible.
I hope we can soon show new performance somewhere in the world with Israel, the members of his company, and all YCAM staff.
Credits:
Israel & イスラエル
Direction, Choreography & Dance
Israel Galván
Cast
Richi Owaki (YCAM), Rie Okada
R&D Direction
Takayuki Ito (YCAM), Richi Owaki (YCAM)
Technical Direction
Richi Owaki (YCAM), Junji Nakaue (YCAM), Pablo Pujol (Israel Galván Company)
Production Management
Clarence Ng (YCAM)
Machine Learning & Artificial Intelligence System Development
Nao Tokui (Qosmo, Inc.), Miyu Hosoi (Qosmo, Inc.)
Data Capturing System Design & Development
Mitsuhito Ando (YCAM)
Visual Development
Satoru Higa, Ryo Kanda
Device Design & Development
Kanta Horio, Alberto Boem, Keina Konno & Richi Owaki & Hoshiro Ando (YCAM), Pablo Pujol (Israel Galván Company)
Sound Engineering
Junji Nakaue (YCAM), Pedro Leon (Israel Galván Company), Mitsuhito Ando (YCAM)
Additional Compositions Extracts
Ikue Mori
Video Engineering
Richi Owaki (YCAM), Keina Konno (YCAM)
Stage Set
Hoshiro Ando (YCAM)
Lighting
Fumie Takahara (YCAM)
Lighting Program & Network Construction
Yohei Miura (YCAM)
Assistant Director & Script
Miguel Álvarez-Fernández
Hair Makeup
Sayano Tani
Coordination
Rie Okada, Asumi Kitahori (YCAM)
Interpreter & Translation
Rie Okada
Delegate Producer
Akiko Takeshita (YCAM)
Co-producer
Pilar Lopez (Israel Galván Company)
※The project was made collectively among the members above regardless of the credits.
Technical Support
Yano butai
Education Program
Daichi Yamaoka & Soma Ishii (YCAM)
Public Relations
Momoko Aoyagi & Nanami Hashimoto & Satomi Okazaki (YCAM), Tomoko Yotsumoto (Sankai Production, LLC)
Documentation
Tomoya Watanabe (YCAM)
Legal Consultant
Tasuku Mizuno (CITY LIGHTS LAW)
Administration
Kiyomi Arifuku & Yuriko Asamoto (YCAM), Rosario Gallardo (Israel Galván Company)
Co-production Coordination
Pilar Lopez (Israel Galván Company)
Organizer
Yamaguchi City Foundation for Cultural Promotion
Co-developed
YCAM InterLab
Association
Yamaguchi City, Yamaguchi City Board of Education
Support
the Agency for Cultural Affairs, Government of Japan, Japan Arts Council, Embassy of Spain in Japan
Produced by
Yamaguchi Center for Arts and Media [YCAM] & Israel Galván Company
Thank you for reading till the end. We are Qosmo, Inc. a collective of Artists, Designers, Engineers and Researchers. Read other Medium articles from Qosmo Lab and if you are so intrigued to find out more, get in touch with us from here. We are actively searching for new members, collaborators and clients who are passionate about pushing the boundaries of AI and creativity. Ciao!
