Audio Intelligence Pilot
Senior Audio UX Designer - 2018
Deliver interaction updates for a Mixed Reality communication application on revolutionary hardware.
Role . Senior Audio Designer
Team . 4 Design, 5 Engineering, 3 PM
Date . March 2018 - November 2018
Brief
Use audio domain knowledge to translate the physical world to audio datasets for Machine Learning Model training. Establish key design strategy and communicate to stakeholders internally and white-glove deploy to an individual customer as a Proof of Concept.
Role . Senior Audio Designer
Team . 4 Design, 5 Engineering, 3 PM
Date . March 2018 - November 2018
Brief
Use audio domain knowledge to translate the physical world to audio datasets for Machine Learning Model training. Establish key design strategy and communicate to stakeholders internally and white-glove deploy to an individual customer as a Proof of Concept.
Background
Contraints .
The development team operated under constraints consistent with a typical incubation team.
The development team operated under constraints consistent with a typical incubation team.
- Resources were limited to the staff we had on hand
- Time frame was limited to demonstrate product market fit
- The technology being developed was being iterated on for the first time and had many limitations
- Datasets were being captured, labelled and validated by our team
- The hardware we were developing for was being developed internally by partner team
Goals .
- Scope work to deliver a prototype in allocated timeframe
- Research customer scenario with Thyssenkrupp to understand customer needs and resources available
- Learn and understand about Machine Learning model generation to plan and collect input for the model
- Deliver working Prototype to Thyssenkrupp Aerospace within the scoped time period
- Scope future product development areas
Role and Responsibilities
- Research machine learning and customer requirements
- Conduct team brainstorms
- Collaborate with data scientists to execute model hypotheses
- Communicate with Product Leadership on designs
- Communicate and conduct research with customers
- Communicate with partnered internal teams
Design Process
Research
Approaching a new area to apply my Audio UX skills, I needed to spin up on understanding AI and Machine Learning technologies, as well as our customer/user needs extremely quickly.
Approaching a new area to apply my Audio UX skills, I needed to spin up on understanding AI and Machine Learning technologies, as well as our customer/user needs extremely quickly.
Design
Our team conducted several brainstorms to identify opportunity areas (responding to the research generated) and selected three service concepts to explore. Our major, and deployed project, was a white glove partnership with Thyssenkrupp Aerospace that allowed us to capture audio datasets to generate a bespoke ML Model for Sound Identification work for them. My role in this was to take ownership of the audio dataset generation, consulting with the data scientists to establish the size of the datasets and the overall design of the classes to achieve the desired identification accuracy outcomes.
Our team conducted several brainstorms to identify opportunity areas (responding to the research generated) and selected three service concepts to explore. Our major, and deployed project, was a white glove partnership with Thyssenkrupp Aerospace that allowed us to capture audio datasets to generate a bespoke ML Model for Sound Identification work for them. My role in this was to take ownership of the audio dataset generation, consulting with the data scientists to establish the size of the datasets and the overall design of the classes to achieve the desired identification accuracy outcomes.
The second was a more generalised version of the Thyssenkrupp project, proposed to be a Microsoft Cognitive Service that businesses could adapt and deploy to identify a range of sounds. My role within this workstream was to identify core users, scenarios and user journeys, and to work with the Data Science team to validate the model's accuracy and overall performance.
The third was an individual prototype for a spatial audio UX platform that separated audio sources based on their semantic meaning and relative importance to the user. This concept was proposed to be deployed cross-platform across all Microsoft computing experiences. I was the sole designer exploring this concept, collaborating with a prototyping engineer to develop a Unity prototype for Virtual Reality.
|
Customer Validation
For our Thyssenkrupp project, development and customer validation was conducted in cycles with three visits. These visits allowed us to test incremental versions of the Machine Learning model we developed, with the final test culminating with a test of the model on the hardware device that was being developed for the Pilot program. |
|
Results
Audio Understanding Service
The workstream with Thyssenkrupp Aerospace was successfully piloted on site, and featured as a highlight at the Microsoft Build conference in 2018.
The workstream with Thyssenkrupp Aerospace was successfully piloted on site, and featured as a highlight at the Microsoft Build conference in 2018.
Audio UX Platform
The Audio UX Platform prototype was successfully tested internally, but was deprioritised to focus on the Audio Understanding customer pilot with Thyssenkrupp.
Outcomes and Next Steps
All work in this team was discontinued in November 2018 due to changing focus in group investment.
The Audio UX Platform prototype was successfully tested internally, but was deprioritised to focus on the Audio Understanding customer pilot with Thyssenkrupp.
Outcomes and Next Steps
All work in this team was discontinued in November 2018 due to changing focus in group investment.
