Neural Style Transfer

Surviving the Game Jam without artists

The Global Game Jam is a massive event held all over the world where developers have 48 hours to create a new game. Making a game is hard and fun and requires a lot of skillsets. This year we were lacking a vital one: we were three developers with the artistic talent of a dead horse. We had no one to draw some amazing graphics for our game. That’s when my friend Uri from Incineration Games suggested we use machine learning to generate the assets.

Neural Style Transfer

is a branch of machine learning which could be used to generate some content. Given a content image(C) and a style image(S) the neural network generates a new image(G) which attempts to apply the style from S to G. The loss function consists of three components:

• Content Loss: makes sure that G preserves the content from C
• Style Loss: evaluates the artistic similarity between G and S
• Variation Loss: smooths G so that no sudden shifts in color are present.

Top 3 Most Popular Ai Articles:

1. Google will beat Apple at its own game with superior AI

2. The AI Job Wars: Episode I

3. Introducing Open Mined: Decentralised AI

For a more in-depth look regarding the algorithm please check out the resources that follow. This blog post will focus on the results related to generating game art.

• Paper: A Neural Algorithm of Artistic Style
• Keras Implementation
• More resources
• Coursera course(week 4): Convolutional Neural Networks by Andrew Ng

The Game

Our game is a hot seat cooperative game where two cars are linked together with a transmission uplink which acts like a physical spring . You can use the uplink to kill the evil computer viruses, but you need to avoid the black holes.

Screenshot of the game. Use the tether to kill the viruses, but avoid the black holes. Download link: http://goo.gl/a6TR1S

Neural Style Transfer results

We used the keras implementation listed above. Most of the assets were generated with the default hyperparameters. In the actual game we use the images generated at every epoch to create an animation which makes the sprites more engaging.

Recovering the alpha channel

The regular neural style transfer algorithm completely ignores the alpha(transparency) channel. It is a VERY important part when it comes to assets, so I’ll outline some possible alpha recovery options:

• copy the alpha values from the original — that’s the one we used (see gist below)
• apply masking to incorporate the alpha channel in the model
• apply some kind of filtering or thresholding

So, what’s wrong with this method is that sometimes the neural style transfer algorithm produces some amazing artifacts. Consider the following case:

Car with additional background artifacts generated by the neural style transfer algorithm

Restoring the original alpha channel completely removes the additional content generated by the algorithm:

Same image as above with the alpha channel from the original image. Sadly, all of the background artifacts are removed.

It would be cool to come up with a strategy which preserves the style artifacts, but makes the rest of the image transparent.

Observations

During the course of the hackathon I tried a lot of different style reference images. I noticed the following:

• Using a style reference image which has similar dominant colors to the content reference image produces more stable results.
• Style reference images should contain some kind of pattern similar to the images posted in the results section. This seems quite reasonable — if the style in the reference image is random, then there is no style to copy except for noise.

The Future of Game Design

It’s exciting to think about the possible uses of machine learning in game design. One possibility is to use neural style transfer or something similar to automatically generate content based on a given reference. Think about all the hats!

Of course, there’s also the advent of the AI agents trained by reinforcement learning. This has the potential to make games even more challenging.

All the advancements in NLP(natural language processing) have left me excited about the opportunity to develop a smarter NPC. Imagine talking to a quest giver who can also freely chat about general events in the game.

The most important part of incorporating ML in games is to make sure that the game is still fun! I recently had a chat with a friend about developing smarter units with machine learning — supposedly, they would learn to auto retreat or auto cast spells or focus on a single enemy. However, he was totally sceptical towards this idea — controlling the units is the player’s job; the player should be in full control. If the control is dedicated to the machine, then the game is no longer fun.

Where do you think machine learning fits in with game development?