Visual Music & Machine Learning Workshop for Kids
This writeup is a summary of the thoughts & experiences we had along a workshop that we designed for kids, in order to transcend gaps between generations via playful knowledge sharing and understanding cutting edge technology through making instead of relying only on theoretical descriptions.
It all started with the 50th anniversary of the Leonardo Almanach. We’ve been asked by Nina Czegledy to make an event that makes it possible to share high level, advanced topics (science, technology) for kids in playful forms. Since Leonardo is an academic journal, it is an interesting experiment in itself: making such concepts available for people who were not involved in it before. If we have a look on the current (and near future) media art landscape, we see that there are a lot of developments on both the hardware side and the software side. Since current popular hardware advancements are so ephemeral, unreliable, and entertainment related (VR, AR & the like), we wanted to dive into the realms of something different. Something that can tell more about paradigm shifts, cognition, thinking and the world: the area of creative usage of machine learning.
We have large experience in audiovisual performances and fine arts — Andrea Sztojanovits, Lorand Szecsenyi-Nagy are both artists & researchers such as myself — so we decided to share concepts through creative ways of music creation. While drawing and building are ways of constructive expression, music is a form of communication using intuitive, sensual contexts, so if we go this way, kids can find connections and consequences in a more personal aspect as opposed to more regular, abstract ways that are operating only on the visual domain.
Embedded Experience
First we tried and played around with the possibilities of the human body: showing how conductivity can form electric circuits is always releasing a bunch of ideas that helps in leaving behind cognitive bias for the children. By letting electricity flow through each of us, interrupting & connecting the circuit, we immediately became part of a larger, distributed system. When children were touching each other, it became clear, that this is something that we can do together, everyone is equal and very important in the circuit. By letting current rise through all participants, concepts like trust, equality, experimentation, play, openness arise naturally, and these lead to constructive conversations, it also helps children to get comfortable with the situation, and initiate communication & sharing of subjective experience.
Trending AI Articles:
1. Paper repro: “Learning to Learn by Gradient Descent by Gradient Descent”
2. Reinforcement Learning for Autonomous Vehicle Route Optimisation
3. Best websites a programmer should visit in 2018 @code_wonders
We were adding & connecting fruits, environmental artefacts (gas tubes that we found in the room), any sort of conductive material to the system. This way we were showing the possibility of abstraction in a system, asking questions like if something works, what else can work, how can we extend existing concepts regarding to a problem (in this case, generating sound)? Objects, environment were followed by the introduction of conductive paint, so we could face challenges and inspiring aspects of drawing, painting & organizing sounds in a visual way.
Sounds
After making drawings of weird & surreal creatures, distant, never-seen before landscapes and newly created constellations, we connected the drawings with the computer, mapping accurate sounds to different regions and semantic components of the drawings. Musical structures (piano sounds, xylophone, marimba) and sampled sonic realms (bubbles, animal sounds, natural forces) both found their ways to appear along with the tangible space of the drawings. Since kids came from different backgrounds with different interests, we found ourselves in a very diverse set of sonic galaxy: some of them were more musical, some more tactile & open for strange sounds, some are just observing the whole scene.
These sounds, combined with the drawings were definitely creating a very unique way to understand abstract components and their correlations. Although it was purely intentional and experimental, children were tackling at the deep interrelations of interface & usability design, cognitive science, software ergonomics, system theory without touching any concept verbally & directly of these fields.
Machine Companion
After turning drawings into working models of playful sonic interfaces (connecting the cables, mapping the inputs), we started to try and discuss the possibilities of what an artificial machine can add (or take away) to extend play, expression & creativity to the system. First we started to add simple delay effects to the sounds. This is a concept where the sound is repeated with a certain amount of feedback. We could say, time is captured, and rotated backwards into the musical sequence. Time, repetition is the very basic of musical structures, so it is a good point to start with the concepts. Our second playful example was to add states for the repetition: we’ve been teaching Markov chains on the fly, to continue our recently added sound sequences. This system does no more than repeating some events we’ve already showed them: while we are training the chain, we teach the probability (of switching fro m one pitch to the next one) too, that makes possible to move along from one state to the next one, etc. This model is used within a lot of fields from creating content for games, composing music, or simulating conversations. It is much more advanced compared to our previous feedback-delay model, since it really takes care of the “state” that we pass to it.
The third example was about to introduce the companion of an artificial entity more further: we were playing together with a machine where the system was trying to catch up where we left off. By playing simple melodies to the system, it was trying to continue our “musical thought” by improvising in the same scale & mode, using a Recurrent Neural Network that is not only aware of current state, but is also time aware, so can track and notice changes during the evolving composition. We were using pre trained models from Magenta, implemented in deeplearn.js by Tero Parviainen.
What we’ve noticed, that kids grabbed immediately the constrains & possibilities of the first (delay-feedback) system. They started to play around with it, they changed their mood for playing, they were both acting & listening. What was more surprising is that playing together with the neural network was also very smooth: they understood that they have to wait for the next improvised “answers” from the system, they also integrated these answers into their own playing technique. Since it was very basic, we could observe it only through the dynamics and the speed (including breaks, longer silence) in their play.
Across generations
The workshop was followed by a very inspiring talk on Neural Art by Daniel Varga, that was investigating the visual aspects of current deep learning methods and artistic outputs. It also turned out as a relevant way to connect the gap between the children and their parents, describing more on the topics of GANs, Deep Dream and the role of those vision based convolutional approaches in our regular life.
At the end, we also saw some inspiring examples from the works of Mario Klingemann & Memo Akten who are investigating the edges and borderlines of current ML technologies from a very provocative and artistic point.
This event was part of the worldwide Leonardo 50 celebration year. It would have not been realized without the hosting of Prezi & the help of Adam Somlai Fischer.
