Face to face

2020 – the year when screens moved in between our face-to-face conversations.

Stuck behind a wall of technology, we attempt to interpret virtual backgrounds, read pixelated faces and solemnly nod at remarks made by our colleagues on mute.

How good are we at reading our conversation partner’s emotions through the Zoom curtain? 

But how do we make sense of partially transmitted, non-verbal communication in times of remote offices and flakey connections? How good are we at reading our conversation partner’s emotions through the Zoom curtain?

Your Face is Muted is an experiential experiment in the MENTAL: Head Inside exhibition at Melbourne’s Science Gallery.

The interactive experience pushes conversations through the filter of technology and explores both our desire and our ability to have meaningful conversations while flying empathetically blind.

Technology Fostering Meaningful Conversations

Connecting and empathising is the key to meaningful conversations – especially around critical topics like climate change, human rights and mental health advocacy. Meaningful conversations are generative – this means they contribute to our knowledge and understanding of each other and the world around us.

The ability to read non-verbal cues and empathise with each other determines the quality of our conversations. But what happens when we shift these conversations from in-person into the digital realm?

Your Face is Muted by Tilman Dingler, Zhanna Sarsenbayeva, Eylül Ertay, Hao Huang and Melanie Huang. 

Created by University of Melbourne students and researchers over Zoom from different sides of the globe during the lockdown in 2020, Your Face is Muted explores what we lose when we converse through patchy connections and how they affect our conversations.

The experience tests how much we rely on non-verbal cues by obfuscating parts of our faces.

In today’s media landscape, emotionally charged topics are often polarising and this can hinder a productive debate. Few topics can be simplified into black-and-white arguments, but rather fall on a nuanced spectrum. An effective way to discover this nuance is to engage with other people’s perspectives.

Your Face is Muted provides a space for debate and conversation.

The experience allows conversationalists to empathise with opposing viewpoints in order to understand the underlying debate, to look introspectively and develop an informed standpoint.

The curtain of technology, thereby, aims to initiate, facilitate and perhaps hinder the ongoing conversation. Its remote nature explores the comfort of disagreement through technology-disabled communication.

Your Face is Muted mimics a Zoom call setting, where two people sit across from each other with a wall and a screen between them. What they see of their conversation partner is a full, partial or modified projection of their face.

Your Face is Muted by Tilman Dingler, Zhanna Sarsenbayeva, Eylül Ertay, Hao Huang and Melanie Huang. 

The interlocutors experience the conversation with limited yet focused non-verbal cues. Two front-facing webcams record each person’s face, then facial recognition software crops the relevant parts.

The conversations are split into three parts.

First, there’s an icebreaker which aims to loosen up the conversation and create a playful environment with questions like “describe your past week in one word”.

Then there’s a “would you rather” question to get to know each other a bit, something like “Would you rather give up social media or eat the same dinner for the rest of your life?”.

And finally, there’s a mini-debate where interlocutors are encouraged to take a specific standpoint– so “social media brings more harm than good”, for example.

Each conversation comes from a pool of topics, so the experience can be undertaken multiple times. Depending on the visitor’s age, debating topics range from “cell phones should be banned in schools” to “climate change is a result of natural cycles”.

The topics aim to invoke empathetic responses in conversation partners when engaging with its nuances. The system then detects their emotions in real-time and quizzes them on their ability to read each other’s emotions.

Stuck behind a wall of technology, we attempt to interpret virtual backgrounds, read pixelated faces. 

While prior research has shown that people are quite capable of judging emotions both in face-to-face and online settings – it’s our face-to-face conversations that tend to be more satisfying and bring people closer together.

Your Face is Muted aims to identify specific aspects of remote communication and their role in people’s ability to read and react to emotions. The experience allows visitors to explore their connection with others through remote communication technology and invites them to reflect on the impact online communications have on our conversations with each other.

From a research point of view, the experience aims to uncover answers to the following questions: How much of our ability to empathise with each other depends on seeing different parts of each others’ faces?

And how do partial occlusion and technology-disabled communication influence our ability to empathetically debate and affect the overall quality of our conversations?

The project is created by researchers from the Human-Computer Interaction group at the University of Melbourne. The group designs technologies that detect users’ emotions, attention and cognitive load, a field also referred to as Affective Computing. The nature of being an experiential experiment allows the exhibit to evolve over time according to people’s feedback.

Your Face is Muted challenges us to read each other’s emotions through the curtain of technology.

As remote working and those all-too-familiar Zoom calls seem here to stay, at least for a while, having an understanding of how technology can and does change the way we converse and empathise with each other seems more vital than ever.

Part exhibition, part experiment, MENTAL is a welcoming place to confront societal bias and stereotypes about mental health. It features 21 works from local and international artists and research collaborators that explore different ways of being, surviving and connecting to each other.  This article was written by Dr Tilman Dingler, Dr Zhanna Sarsenbayeva, Eylül Ertay, and Hao Huang of the University of Melbourne.  It was published by Pursuit.