Big Life Fix


We worked with the Big Life Fix team (Inventor, Trevor Vaugh from Maynooth University, and Chiara Cavarra, a Digital Design Engineer at Xilinx) to help Motor Neuron Disease sufferer, Róisín, to communicate with her loved ones using a digitised version of her own voice.

Big Life Fix team and Marino team chat with Roisin

The challenge

Work with the Big Life Fix team to help to give MND sufferer Róisín Foley the ability to communicate in her own voice and help her to pass on her experience and wisdom to her children.

The solution

A smart soundboard app that allows Róisín to quickly and easily use her most used phrases that automatically updates based on who she is speaking with. To help Róisín to record messages for her children to access when she can no longer communicate with them we created an easy to use audio/video recorder.

The impact

The communication app allowed Róisín to communicate with family and friends even when attached to her breathing machine while the legacy app enabled her to comfortably record video and audio to leave behind for her loved-ones.

Róisín Foley was diagnosed with Motor Neuron Disease (MND) in September 2017, just one month after her 30th birthday. This rare illness is destroying her motor skills, making everyday tasks increasingly difficult. MND will eventually render her unable to speak or communicate with her three daughters and family.

RTÉ TV series, Big Life Fix, got in touch with us to see how we could help Róisín improve her quality of life.

“There’s only ten people my age in the country who have it. I’ve heard you have just as much a chance of winning the national lottery as you do of contracting MND.”
- Róisín

Challenges

We had two main challenges:

  • Give Róisín the ability to communicate in her own voice (Voice Banking)
  • Maintain her voice alive, pass on her experience and wisdom to her children (Voice Legacy)

Róisín’s health was deteriorating every day so we had to move fast. She was spending half of her time on her breathing machine so we needed to figure out how we mightintegrate our solution into her routine.

We wouldn’t have the time to build the "perfect" solution, so we asked, "what can we do in a very short space of time to bring value to her daily life?"

What we did

We worked in a very agile way - designers working very closely with developers, building a solution with the aim to get it to her as soon as possible. We could then iterate and adapt it to her needs and limitations as much as possible.

We used a combination of recordings and AI-generated sentences for Róisín’s digital voice. The aim of this was to be as close as possible to her own voice. Using an MVP approach, we built a solution she could start using on her tablet straight away.  

Our work included:

  • Research into current and analogous solutions
  • Categorisation of recorded sentences
  • New sentences generation through Artificial Intelligence (AI)
  • Accessible design and development
  • User testing and iteration on design
  • Contextual location content through beacons implementation 

Research - What’s out there?

We had access to a library of recorded phrases for when Róisín wasn’t able to use her own voice. We set out to figure out what could be a good way for her to access these.

After our initial brief, we did some research on soundboards, computer vision, eye gazing, beacons and AI, amongst others. We saw there were multiple solutions out there to help the voiceless, however most of them would use someone else’s voice. We wanted for Róisín to be able to speak in her own voice. 

mages of research boards


Hackathon

Róisín and her brother came to meet us at Marino HQ where we got to chat with her and hear about the main challenges she faces living with MND. She told us about her daily life, family, interests and her relation to technology. 

Róisín is a very sociable woman, who is not just the mother to her children but she is also the mammy to all her brothers and sisters. She told us there are people coming and going every day in their house and they have a lot of conversations around the kitchen table. She also told us about interests which she enjoyed until she was diagnosed such as meditation, music, going to gigs, hiking and going to the gym.

She was such an inspiration - we really wanted to do all we could to try to make her life even slightly easier.

The team chatting with Róisín

We used the "How Might We?" (HMW) method of note taking to uncover what we had to consider when designing for her. We had some concerns about mobility, accessibility and overall how to design the flow of a conversation to make it as natural as possible.

Sharing "How Might We?" notes

   

Organising Róisín's needs

We had plenty of unknowns but one thing we knew for sure at this stage is that we wanted to preserve Róisín’s voice and unique personality. 

Since our solution was going to allow Róisín to have conversations, we asked ourselves "how do we design a conversation?" Our starting point was that conversations usually have a clear structure - they have a start, middle and end. 

We started sketching out ideas around structuring conversations. What might she need to start with? Probably a greeting or a conversation opener such as ‘How is it going?’ or ‘How was school today?’. And, from there, how would the conversation develop? Predicting in which way conversations could go from this initial stage is extremely difficult - there are endless ways a conversation could follow. 

Designing a conversation

One initial assumption was that after the first conversational exchange Róisín might want go deeper and that words such as Why, What, Who, How and Where could be really handy for her to develop the conversation further.

Some rough ideas starting to take shape

There were multiple ideas brainstormed at this stage, even though we knew we’d have to settle with a simple solution for our MVP (Minimum Viable Product). An idea to speed up Róisín answers was the app reacting to external sound - such as offering predictive answers when a child asked if they could do something  (‘Can I watch TV?’, for example). 

Conversation flows

We also identified that conversations are also highly dependent on context - who you are talking to, where you are, the time of the day, how do you feel or even what is the weather like (especially in Ireland!), so we started thinking about how we might identify and use context within the design.

First working prototype 

At the end of our hackathon day, we had built the first working version of our soundboard. We could scroll through all the sentences we had thus far, but it was unclear how they might be categorised so that Róisín could easily find them?

A working prototype
A working prototype

Alongside this, we had a very first draft of what the interface might look like. This considered the first possible conversation step, who Róisín was talking to and gave her shortcuts to frequently used words.

Wireframing the interface

Keeping Róisín’s voice alive

Róisín recorded hundreds of sentences for us to use. It was amazing to have this material but how could we organise it for Róisín to easily access it? We all have different mental models so categorising these sentences was a particularly challenging task. 

"We all have different mental models so categorising these sentences was a particularly challenging task." 

We had a head start and found a way for Róisín to participate in this process, making sure sentence categories matched her own mental models.

On top of this, we used LyreBird.ai, an artificial intelligence tool, to analyse Róisín’s voice and allow her to create new sentences by mimicking her own voice. This technology was of course still far from perfect but at least gave Róisín the ability to generate new sentences on the spot.

Creating new phrases


Accessibility - More important than ever

Róisín was losing her hand strength so actions like holding a mobile phone were increasingly difficult for her. We decided on developing our main soundboard app on a tablet she could easily use on a stand.

“At the beginning I didn’t tell them a whole lot about it. My 12-year-old understands. They know that my hands don’t work properly. My eldest girl does all the hair and stuff for me. She does the ponytails going to school.”
- Róisín

We had accessibility in mind first and foremost. We were very mindful of these while designing the apps:

  • Touch areas - considering easiest to access screen areas when placing key actions
  • Fitts’s Law - minimising distance between objects and increasing their size for easier selection
  • Grid / Swype keyboard - predictive text
  • Colour and Contrast

We also did some research into eye gazing for future-proofing. We tested the Hawkeye app, that would allow Róisín to control the interface using her eye gaze only. The implementation of this was scheduled for a later stage.

User testing - The moment of truth 

Once we had our MVP prototype ready we gave it to Róisín to play around with it and give us her feedback. We wanted to learn about things like:

  • General usability and accessibility
  • Sentences categorisation 
  • Most frequently used sentences
  • Use of sentences in different contexts
  • Search
Using a basic but function app to get feedback from Róisín

Roisin’s movement had deteriorated since we last saw her so she had some difficulties hovering over buttons or accessing categories in their current placement. It was great to be able to observe her reaction and the difficulties she encountered, so we could take this and work on improving the app.

Overall Róisín was very happy with the soundboard and told us she saw how she could start using it straight away.

“I see how I could start using it straight away”
Róisín testing the prototype
The app prototype

Contextual location content through beacons implementation 

We wanted to make it easy for Róisín to access sentences quickly.

While categorising her sentences, we identified that some were specific to either children or adults also some were specific to certain people. We thought we could use beacons to identify when a person entered the room and allow Róisín to quickly access sentences she would be able to use in this context. A good example of these were sentences related to school or home tasks she could ask her children about. 

Higher fidelity prototype


Wearable beacons such as a bracelet or pendant for the children and closest family members could be used for this functionality. 

Her house was mapped out to test this approach.

Scale model of Róisín's house
A scale model of Róisín's house
Róisín's house model
Recreation of Róisín's house floor plan
Recreating Róisín's house for testing beacons
Beacons
Beacons used to detect proximity to people

After the test, we made some changes to the layout and positioning of actions to ensure it was easier for Róisín to tap on the screen.

We also changed the interface colours to a cleaner, muted palette based on Róisín's own colour preferences.

Updated app interface


Legacy / Video Recordings app

In addition to the main soundboard app, we built a Legacy app for Róisín to be able to record audio and videos and choose who she would like to receive these. 

This app was intended to keep her voice alive, pass on her experience and capture her wisdom for her children and family. 

Rough legacy app flow

Outcomes

The communication app helped Róisín to communicate with family and friends even when attached to her breathing machine while the legacy app gave Róisín a way to easily and comfortably record audio and video to leave behind for her loved-ones.

Working with Róisín was truly an inspiration for all involved. We aimed to positively impact her life and hopefully we have done that. What we didn't expect was the impact Róisín would have on us and how we work. Getting to know Róisín and her situation really heightened the importance for us , as an industry, to focus on accessibility and designing and developing for people of all needs. We plan to make this a key part of everything we do and continue to work on serving those with all levels of ability.

Accessibility Webinar

We're putting together a webinar on accessibility best practises for web and mobile. Register now to get access.

Let’s chat!

We’re ready to start the conversation however best suits you - on the phone at +353 (0)1 833 7392 or by email

Get in touch