Olga Slizovskaia PhD thesis defense 21 October 2020

The complementary relationship between audio and video data drives multimodal studies across various domains and problems. Within that scope, we emphasise the important role that visual modality plays in music perception and conduct a study on multimodal data fusion techniques in the music information retrieval domain. In this talk, we outline challenges in audio-visual music information retrieval and present our work that addresses instrument classification and source separation problems in musical instrument performances. We discuss the efficiency of conditioning techniques being applied at different levels of a primary network and make use of two extra modalities of data, namely instrument labels and the corresponding visual stream data.

Audio-Visual Deep Learning Methods for
Musical Instrument Classification and Separation

PhD candidate: Olga Slizovskaia

Supervisors:

Dr. Emilia Gómez

Joint Research Centre, EC
Music Technology Group, UPF

Dr. Gloria Haro

Image Processing Group, UPF

Committee:

Dr. Xavier Giró-i-Nieto

Universitat Politècnica de Catalunya

Dr. Xavier Serra

Universitat Pompeu Fabra

Dr. Estefanía Cano

Agency for Science, Research and Technology,
A*STAR, Singapore

21/10/2020

Outline

I - Audio-Visual MIR

II - Audio-Visual Musical Instrument Classification

III - Audio-Visual Source Separation

IV - Conclusion

I - Audio-Visual MIR

First of all, audio-visual MIR. So, what is audio-visual learning and MIR and why do we study it? When I was 11 y.o., I somehow passed through a regional selection stage and my music teacher brought me to compete to a prestigious-like musical competition in my area. I did not proceed further but I remember the experience of whatching (watching, not listening) performances of other participant, especially the senior ones. It was a feast of performance techniques, both from a technical and artistic point of view, and I recall that we discussed for a couple of weeks everything that we saw and tried to adapt some tricks. That was my first immersive experience of how highly multimodal could be our perception of a music performance. But definitely not the last one. Many years have passed, but at any life concert, I watch maybe even more than I listen, especially if the scene is complex. You can narrow or widen the vision area and focus on a particular aspect of the performance. We (as a kind) are very efficient in aggregating information that comes from different perception channels in order to make sense of the word around us, and it is still unclear how we are doing it. And we should care about AV methods because while machine perception is trying to mimic human perception often with success, many problems remain open, especially for complex scenes that involve multi-sensory merging. Within the area of MIR, there are many tasks that humans do using the full range of perception capabilities and there are a number of tasks that can benifit from multi-modal approaches.

Why do we study Audio-Visual MIR?

Music is multimodal, we aim for better understanding and analysis
Humans are very good at merging different sources of information
Yet, ML algorithms are not as good as humans at multi-sensory merging
Various practical applications
- alignment
- transcription
- separation
- localization
- tagging

What are the challenges in Audio-Visual MIR?

Shortage of dedicated datasets
Low quality of available data, large diversity in data
Dimensionality mismatch problem
Data aggregation problem
Extra: limited computing power

Research goals

modern and efficient audio-visual methods for MIR

Research focus (I/II)

Where can we merge different data representations?

Early fusion

Where can we merge different data representations?

Late fusion

Where can we merge different data representations?

Hybrid fusion

Where can we merge different data representations?

Research focus (II/II)

How should we merge data from different sources?

II - Audio-Visual Musical Instrument Classification

Problem definition

* Video courtesy: YouTube-8M dataset, an actual training sample

For the classification setup, we have a set of objects $X \in \mathbb{R}^N$ and a set of class labels $C \in \{0,1\}^K$. We are interested in predicting a final subset of labels $y=(y_1, y_2, ..., y_K) \in C$ for a sample $x=(x_1, x_2, ..., x_N) \in X$ which is an $N$-dimensional representation of an object $x$. Therefore, we are looking for a function such as where $\theta$ are the function parameters and $ \hat{y} $ is a probability vector for the class estimates. A common mapping function $ f $ could be a complex non-linear neural network, where the final probability scores can be obtained via softmax function \[ \hat{y_j}(z) = \frac{e^{z_j}}{\sum_{i=1}^L e^{z_i}}, \] which takes a vector of arbitrary real-valued scores $z = (z_1, ... z_L) \in \mathbb{R}^L $ and transforms it to a vector of values between zero and one that sum to one. A common choice to learn the parameters $\theta$ is through a backpropagation algorithm where one of the common optimization techniques can be used to minimize a loss function. For multi-labels classification, the loss function can be defined as categorical cross-entropy between ground truth and estimated probability distributions of class labels:

Formal problem definition

set of objects $ \definecolor{signal}{RGB}{18,110,213} {\color{signal} X} \subset \mathbb{R}^N$, samples $ {\definecolor{signal}{RGB}{18,110,213} \color{signal}x}=({\color{signal} x_1, x_2, ..., x_N}) \in {\color{signal}X}$
set of class labels $\definecolor{categories}{RGB}{203,23,206} {\color{categories}C} \subset \{0,1\}^{\color{categories}K}$
final subset of labels $ \definecolor{classes}{RGB}{114,0,172} \definecolor{categories}{RGB}{203,23,206} {\color{classes} y } = ({\color{classes}y_1, y_2, ..., y_K}) \in {\color{categories} C }$ for a sample $ \definecolor{signal}{RGB}{18,110,213} \color{signal} x$

\[ \begin{aligned} \definecolor{classes}{RGB}{114,0,172} \definecolor{params}{RGB}{45,177,93} \definecolor{model}{RGB}{251,0,29} \definecolor{signal}{RGB}{18,110,213} \definecolor{probability}{RGB}{217,86,16} \definecolor{categories}{RGB}{203,23,206} {\color{probability} \hat{y}} = {\color{model} f}_{\color{params} \theta}({\color{signal} x}) \end{aligned} \] find parameters for a model that gives
class estimation probabilities for a sample

\[ \definecolor{loss}{RGB}{128,121,14} \definecolor{classes}{RGB}{114,0,172} \definecolor{probability}{RGB}{217,86,16} \definecolor{categories}{RGB}{203,23,206} { \color{loss} \mathcal{L} } ( {\color{classes} y}, {\color{probability} \hat{y}}) = -{\color{loss} \sum}_{\color{categories} i=1}^{\color{categories} K} {\color{classes}y_i} {\color{loss} * \log}( {\color{probability} \hat{y}_i }) \] for multi-labels classification, minimize categorical cross-entropy between ground truth and estimated probability distributions of class labels

First multimodal approach

Hybrid/Late fusion
Concatenation

Audio input Video input

Audio-based CNN Video-based CNN

First multimodal approach

MelSpec audio representation
Architectures
- Han et al., 1.5M params
- Choi et al., 2.4M params
- Xception, 9.6M params

Bag-of-frames RGB
Inception V3
Number of frames: 10-100
- FCVID: 20-100
- YouTube-8M: 10-20
Fine-tune or train from scratch

DATASETS

FCVID: Musical Performance With Instruments
12 classes, 5K videos, 260 hours

YouTube-8M: MusInstr-Normalized
46 classes, 60k videos, 4k hours

Summary of contributions

Proposed a new multimodal network for musical instrument classification
Demonstrated efficiency of the audio-visual approach
Shown that multi-modal fusion is beneficial overall
Concatenation might not be enough for controversial inputs

III - Audio-Visual Source Separation

Can you hear the difference?

Sample 1

Sample 2

Sample 3

* All examples are taken from the URMP dataset: https://doi.org/10.5061/dryad.ng3r749

Can you hear the difference now?

Separated 1	Separated 2	Separated 3

Problem definition

Single channel source separation (SCSS) consists of estimating the individual sources x_i given a mono mixture time-domain signal $y$ of $N$ sources: Instead of predicting time-domain signals, a general approach for solving SCSS involves the estimation of N masks for Short-Term Fourier transform (STFT) values of the mixture. In this case, we consider a time-frequency representation of the mixture Y and the sources Xi, and the goal of the source separation method is to learn a real-valued (or complex-valued) mask $M_i$ for each source $i$ . Let us denote by $|\boldsymbol{X_i}(\tau, \omega)|$ and $|\boldsymbol{Y}(\tau, \omega)|$ the magnitude of the STFT value, of $\boldsymbol{X_i}$ and $\boldsymbol{Y}$ respectively, at frequency $\omega$ and time frame $\tau$. In this work, we only consider two types of \textit{real-valued} masks, namely \textit{ideal ratio or soft} masks $M_i^{ir}$:

Formal problem definition

a mono mixture time-domain signal $\definecolor{mix}{RGB}{18,110,213} \color{mix} y(t)$
individual sources $ \definecolor{gt}{RGB}{203,23,206} \color{gt} x_i(t)$

\[ \definecolor{loss}{RGB}{128,121,14} \definecolor{predicted}{RGB}{217,86,16} \definecolor{gt}{RGB}{203,23,206} \definecolor{nsources}{RGB}{114,0,172} \definecolor{sum}{RGB}{251,0,29} \definecolor{mix}{RGB}{18,110,213} {\color{mix} y(t)} = {\color{sum} \sum_{\color{nsources}i=1}^{\color{nsources}N} {\color{gt} x_i(t)}} \] the mixture equals to the sum of all sources

\[ \definecolor{loss}{RGB}{128,121,14} \definecolor{predicted}{RGB}{217,86,16} \definecolor{nsources}{RGB}{114,0,172} \definecolor{model}{RGB}{251,0,29} \definecolor{gt}{RGB}{203,23,206} \definecolor{mix}{RGB}{18,110,213} \definecolor{param}{RGB}{45,177,93} {\color{predicted} \hat{x}_{\color{nsources}i}(t)} = {\color{model}f_{\color{params}\theta}^{\color{nsources}i}}({\color{mix}y(t)}) \] find parameters for a model that gives
individual sources estimation from the mixture

Formal problem definition

approaches to estimate individual sources from the mixture

direct waveform estimation

predicting time-domain signals $ \definecolor{nsources}{RGB}{114,0,172} \definecolor{predicted}{RGB}{217,86,16} \color{predicted} \hat{x}_{\color{nsources}i}(t)$

\[ \definecolor{nsources}{RGB}{114,0,172} \definecolor{loss}{RGB}{128,121,14} \definecolor{predicted}{RGB}{217,86,16} \definecolor{gt}{RGB}{203,23,206} {\color{loss} \mathcal{L}^{w} {\color{black}=} \sum_{\color{nsources}i=1}^{\color{nsources}N} \sum_{j=1}^{T} ({\color{gt} x_{\color{nsources}i}(j)} - {\color{predicted} \hat{x}_{\color{nsources}i}(j)})^2} \]

masking-based approach

predicting ratio masks $ \definecolor{nsources}{RGB}{114,0,172} \definecolor{predicted}{RGB}{217,86,16} \color{predicted} \hat{M}_{\color{nsources}i}^{r} $

$ \definecolor{mix}{RGB}{18,110,213} \color{mix} \boldsymbol{Y} $, $ \definecolor{gt}{RGB}{203,23,206} \definecolor{nsources}{RGB}{114,0,172} \color{gt} \boldsymbol{X_{\color{nsources} i}} $ STFT values of the mixture and sources

magnitude of the STFT value at frequency $\nu$ and time $\tau$
$\definecolor{gt}{RGB}{203,23,206} \definecolor{nsources}{RGB}{114,0,172} | {\color{gt} \boldsymbol{X_{\color{nsources}i}}(\tau, \nu) } |$, $\definecolor{mix}{RGB}{18,110,213} | {\color{mix} \boldsymbol{Y}(\tau, \nu)}|$

ideal ratio mask of source $\definecolor{gt}{RGB}{203,23,206} \definecolor{nsources}{RGB}{114,0,172} | {\color{gt} \boldsymbol{X_{\color{nsources}i}}(\tau, \nu) } |$ \[ \definecolor{gt}{RGB}{203,23,206} \definecolor{nsources}{RGB}{114,0,172} \definecolor{mix}{RGB}{18,110,213} \definecolor{predicted}{RGB}{217,86,16} {\color{predicted} \hat{M}_{\color{nsources}i}^{r}} = \frac{|{\color{gt} \boldsymbol{X_{\color{nsources}i}}(\tau, \nu) } |}{|{\color{mix} \boldsymbol{Y}(\tau, \nu)}|} \]

\[ \definecolor{loss}{RGB}{128,121,14} \definecolor{nsources}{RGB}{114,0,172} \definecolor{predicted}{RGB}{217,86,16} \definecolor{gt}{RGB}{203,23,206} {\color{loss} \mathcal{L}^{r} {\color{black}=} \sum_{\color{nsources}i=1}^{\color{nsources}N} \sum_{j=1}^{|T \times F |} ({\color{gt} M_{\color{nsources}i}^{ir}(j)} - {\color{predicted} \hat{M}_{\color{nsources}i}^{r}(j)})^2} \]

Metrics

Source to Distortion Ratio (SDR)
Source to Inferences Ratio (SIR)
Sources to Artifacts Ratio (SAR)
Scale-Independent Source to Distortion Ratio (SI-SIR)
Scale-Dependent Source to Distortion Ratio (SD-SIR)
Predicted Energy at Silence (PES)

Now I would like to shortly discuss the common approach to solve this problem and why it is difficult to solve. One of the problems is that sources often overlap and time and frequency. It's especially true for instruments playing in harmony and in unison, this is often the case for classical music. And this is also a problem for instruments of the same family, since they have very similar timbre as well. It has been shown that the complexity of the task increases with the number of sources, and in our work we tackle the problem not of just two-channel source separation, but we address multiple instrument separation where actual number of sources is not known. For many years standard common approach for this task had consisted in estimating binary or ratio masks which could be applied to a magnitude of mixture spectrogram in order to obtain the magnitude spectrogram of the sources, and then recostruct the original sound using either the mixture phase of phase-reconstruction algorithm. Thus we lose some information which can be potentially usefull for source separation and this is our primary motivation for waveform domain source separation. On the other hand, with all this complexity and the first example I've shown you in mind, we want to answer another question: So we took week information, which is instrument labels information which can be available from another classifier network from taken from a different modality, a decided to use them to condition our model.

Known challenges

Same-family instruments are similar to one another
Unknown in advance number of sources
Complexity increases with the number of sources
Overlap in time and frequency between sources

💪 Can we use extra information to improve separation?

Preliminary study

Multi-instrument Source Separation in Time Domain
Hybrid fusion
Multiplicative Conditioning on Instrument Labels

Conditioned Wave-U-Net Architecture

Experiments

URMP Dataset

13 instruments
44 videos -> 40 valid pieces
12 duest, 20 trios, 8 quartets
87 unique audio tracks
Train/Test split: 30/10 pieces

Video Demo: Trained on 30 videos

Summary of results

Method	SDR	SIR	SAR
InformedNMF	-0.16	1.42	9.31
Exp-Wave-U-Net	-4.12	-3.06	12.18
CExp-Wave-U-Net	-1.37	2.16	6.36

Summary of results

Take away note: it was inspiring but we lacked TPUs and scaling was not an option

Ablation and Speedup

Learning rate

TPUs

tf.float32 vs tf.float16

Total Speedup: x35.4

What we did next...

SOLOS

* Joint work with Juan Montesinos https://juanmontesinos.com/Solos

13 instruments matching the URMP dataset
Only solo performances / auditions
Semi-automatic and manual quality control
Mix-and-separate approach
755 individual recordings
Extra: valid timestamps and skeletons

Conditioned Wave-U-Net Architecture

Design Decisions

Conditioned Wave-U-Net Architecture

Design Decisions

Conditioned Wave-U-Net Architecture

Design Decisions

Conditioned Wave-U-Net Architecture

Design Decisions

Context vectors

Label conditioning

Visual conditioning

Visual-motion conditioning

Design Decisions

Conditioning Design

FiLM conditioning

Multiplicative conditioning

### Summary of results

|Method | ID | Cond | SI-SDR `$\uparrow$` | PES `$\downarrow$` | SDR `$\uparrow$` | SIR `$\uparrow$` | SAR `$\uparrow$` |
							| :--- | :-----: | :----: | :----: | :----: | :----: | :---: | :---: |
							| U-Net U-Net | 6 6-W`$_2$` | none none | `$ -2.1\pm 6.2 $` `$ -7.4\pm 9.9 $` | `$ -8.1\pm 5.9 $` `$ -24.4\pm 15.8 $` | `$ -0.77\pm5.76 $` `$ -2.27\pm6.85 $` | `$ 0.57\pm6.85 $` `$ 1.46\pm8.39 $` | `$ 10.88\pm3.21 $` `$ 5.74\pm5.23 $` |
							| FiLM-encoder Final-multiply | 9-W`$_2$` 11-W`$_2$` | labels labels | `$ \mathbf{-3.5\pm6.1} $` `$ -7.2\pm9.8 $` | `$ -4.4\pm6.8 $` `$ -18.6\pm20.3 $` | `$ \mathbf{-1.68\pm5.47} $` `$ -1.86\pm6.84 $` | `$ -0.39\pm6.48 $` `$ \mathbf{2.36\pm8.80} $` | `$ \mathbf{11.44\pm4.15} $` `$ 5.13\pm4.07 $` |
							| FiLM-bottleneck-1F Final-multiply-1F | 17-W`$_2$` 16-W`$_2$` | visual visual | `$ -8.7\pm10.8 $` `${-4.1\pm7.2}$` | `$ \mathbf{-25.9\pm23.9} $` `$ -15.4\pm14.3 $` | `$ -3.02\pm7.10 $` `$ \mathbf{-1.49\pm6.08} $` | `$ 1.17\pm8.95 $` `$ 0.98\pm8.39 $` | `$ 5.09\pm5.08 $` `$ 9.18\pm3.71 $` |							|FiLM-final-lstm-5F | 13-W`$_2$` | visual-motion | `$ \mathbf{-3.8\pm6.3} $` | `$ -9.4\pm12.2 $` | `$ -2.51\pm5.32 $` | `$ -1.43\pm6.38 $` | `$ \mathbf{12.44\pm4.72} $` |
							|SoP-unet5-Solos-3F | 21 | visual| `$-16.9\pm8.6$` | n/a | `$-2.9\pm4.7$` | `$-1.67\pm5.34$` | `$11.07\pm6.87$` |

Summary of results

* Exp. 6 - Baseline U-Net
* Exp. 16 - Final-multiply-1F conditioning

Summary of results

Results: Video Demo

## Ablation study results

|Method | ID | SI-SDR `$\uparrow$` | SD-SDR `$\uparrow$` | PES `$\downarrow$` | SDR `$\uparrow$` | SIR `$\uparrow$` | SAR `$\uparrow$` |
					| :--- | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
					| IRM | U | `$ 13.1\pm5.4 $` | `$ 12.7\pm6.4 $` | n/a | `$ 11.79\pm4.26 $` | `$ 19.94\pm5.59 $` | `$ 13.02\pm4.25 $` |
					| input mix | L | `$-3.7\pm5.7$` | `$-3.7\pm5.7$` | `$18.2\pm4.2$` | `$ -3.48\pm4.82 $` | `$ -3.20\pm4.95 $` | `$ 18.10\pm11.21 $` |
					| No filtering | 1 | `$ -2.1\pm 6.2 $` | `$ -15.2\pm 10.0 $` | `$ -8.1\pm 5.9 $` | `$ -0.77\pm5.76 $` | `$ 0.57\pm6.85 $` | `$ 10.88\pm3.21 $` |
					| Wiener (1 it.) | 1-W`$_1$` | `$ -3.9\pm 7.5 $` | `$ -16.0\pm 16.1 $` | `$ -15.1\pm 10.1 $` | `$ -1.01\pm6.20 $` | `$ 1.53\pm7.65 $` | `$ 7.70\pm3.81 $` |
					| Wiener (2 it.) | 1-W`$_2$` | `$ -7.6\pm 10.7 $` | `$ -21.4\pm 24.1 $` | `$ -25.4\pm 16.8 $` | `$ -2.56\pm7.10 $` | `$ \mathbf{1.72\pm8.52} $` | `$ 4.99\pm5.62 $` |
					| 1-W`$_2$` with CL | 2-W`$_2$` | `$ -8.2\pm 10.9 $` | `$ -24.1\pm 25.1 $` | `$ -26.2\pm 17.8 $` | `$ -3.06\pm7.06 $` | `$ 1.38\pm8.55 $` | `$ 4.59\pm5.72 $` |
					| 2-W`$_2$` but binary masks | 3-W`$_2$` | `$ -9.7\pm 15.2 $` | `$ -21.6\pm 27.4 $` | `$ \mathbf{-36.2\pm 18.4} $` | `$ -3.82\pm7.95 $` | `$ 0.09\pm7.10 $` | `$ 4.93\pm7.53 $` |
					| 1-W`$_2$` with noise | 4-W`$_2$` | `$ -7.6\pm 9.2 $` | `$-21.7\pm 20.4 $` | `$ -21.7\pm 15.6 $` | `$ -3.12\pm6.91 $` | `$ 0.21\pm8.34 $` | `$ \mathbf{6.04\pm5.32} $` |
					| 1-W`$_2$` no dB normalise | 5-W`$_2$` | `$ -10.1\pm 11.6 $` | `$ -25.3\pm 24.9 $` | `$ -27.0\pm 17.0 $` | `$ -3.88\pm7.63 $` | `$ 0.70\pm9.45 $` | `$ 4.63\pm6.13 $` |
					| 1-W`$_2$` linear-scale STFT | 6-W`$_2$` | `$\mathbf{-7.4\pm 9.9} $` | `$ \mathbf{-17.5\pm 19.8} $` | `$ -24.4\pm 15.8 $` | `$\mathbf{-2.27\pm6.85}$` | `$ 1.46\pm8.39 $` | `$ 5.74\pm5.23 $` |
					| 1-W`$_2$` but MHU-Net | 7-W`$_2$` | `$ -8.1\pm9.1 $` | `$ -23.7\pm20.4 $` | `$ -20.9\pm15.8 $` | `$ -3.29\pm6.35 $` | `$ 0.07\pm8.22 $` | `$ 5.75\pm4.53 $` |

## Label conditioning results

|Method | ID | SI-SDR `$\uparrow$` | SD-SDR `$\uparrow$` | PES `$\downarrow$` | SDR `$\uparrow$` | SIR `$\uparrow$` | SAR `$\uparrow$` |
					| :--- | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
					| w/o conditioning | 6-W`$_2$` | `$ -7.4\pm 9.9 $` | `$ -17.5\pm 19.8 $` | `$ -24.4\pm 15.8 $` | `$ -2.27\pm6.85 $` | `$ 1.46\pm8.39 $` | `$ 5.74\pm5.23 $` |
					| FiLM-bottleneck | 8-W`$_2$` | `$ -8.1\pm9.4 $` | `$ -19.1\pm18.1 $` | `$ -19.5\pm15.3 $` | `$ -2.77\pm6.64 $` | `$ 1.17\pm8.24 $` | `$ 4.95\pm4.84 $` |
					| FiLM-encoder | 9-W`$_2$` | `$ \mathbf{-3.5\pm6.1} $` | `$ -20.7\pm6.7 $` | `$ -4.4\pm6.8 $` | `$ \mathbf{-1.68\pm5.47} $` | `$ -0.39\pm6.48 $` | `$ \mathbf{11.44\pm4.15} $` |
					| FiLM-final | 10-W`$_2$` | `$ -10.1\pm11.7 $` | `$ -34.5\pm28.4 $` | `$ \mathbf{-33.9\pm28.6 }$` | `$ -3.50\pm6.79 $` | `$ 0.29\pm9.14 $` | `$ 5.88\pm4.97 $` |
					| Label-multiply | 11-W`$_2$` | `$ -7.2\pm9.8 $` | `$ \mathbf{-15.3\pm15.5 }$` | `$ -18.6\pm20.3 $` | `$ -1.86\pm6.84 $` | `$ \mathbf{2.36\pm8.80} $` | `$ 5.13\pm4.07 $` |

## Visual conditioning results

|Method | Frames | ID | SI-SDR `$\uparrow$` | SD-SDR `$\uparrow$` | PES `$\downarrow$` | SDR `$\uparrow$` | SIR `$\uparrow$` | SAR `$\uparrow$` |
					| :--- | :---: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: |
					|w/o conditioning | 0 | 6-W`$_2$` | `$ -7.4\pm 9.9 $` | `$ \mathbf{-17.5\pm 19.8} $` | `$ -24.4\pm 15.8 $` | `$ -2.27\pm6.85 $` | `$ \mathbf{1.46\pm8.39} $` | `$ 5.74\pm5.23 $` |
					|FiLM-encoder | 1 | 12-W`$_2$` | `$ -8.3\pm8.9 $` | `$ -26.1\pm17.5 $` | `$ -12.2\pm13.4 $` | `$ -3.38\pm5.54 $` | `$ -0.69\pm6.60 $` | `$ 5.91\pm4.25 $` |
					|FiLM-bottleneck | 1 | 17-W`$_2$` | `$ -8.7\pm10.8 $` | `$ -20.7\pm20.1 $` | `$ \mathbf{-25.9\pm23.9} $` | `$ -3.02\pm7.10 $` | `$ 1.17\pm8.95 $` | `$ 5.09\pm5.08 $` |
					|FiLM-final | 1 | 18-W`$_2$` | `$ -4.8\pm7.4 $` | `$ -18.4\pm13.5 $` | `$ -10.9\pm8.7 $` | `$ -2.24\pm6.22 $` | `$ 0.05\pm7.91 $` | `$ 8.87\pm4.90 $` |
					|Final-multiply | 1 | 16-W`$_2$` | `${-4.1\pm7.2}$` | `$ -19.0\pm13.9 $` | `$ -15.4\pm14.3 $` | `$ \mathbf{-1.49\pm6.08} $` | `$ 0.98\pm8.39 $` | `$ 9.18\pm3.71 $` |
					|FiLM-final-lstm | 5 | 13-W`$_2$` | `$ \mathbf{-3.8\pm6.3} $` | `$ -22.7\pm12.6 $` | `$ -9.4\pm12.2 $` | `$ -2.51\pm5.32 $` | `$ -1.43\pm6.38 $` | `$ \mathbf{12.44\pm4.72} $` |
					|FiLM-bottleneck-lstm | 5 | 14-W`$_2$` | `$ -4.7\pm6.9 $` | `$ -17.9\pm10.7 $` | `$ -10.3\pm12.7 $` | `$ -1.55\pm5.94 $` | `$ 1.02\pm7.96 $` | `$ 7.90\pm3.27 $` |
					|FiLM-bottleneck-mp | 5 | 15-W`$_2$` | `$ -7.0\pm7.8 $` | `$ -23.0\pm14.4 $` | `$ -12.1\pm15.0 $` | `$ -2.46\pm5.82 $` | `$ 0.54\pm7.66 $` | `$ 6.28\pm3.61 $` |
					|SoP-unet7 | 3 | 19 | `$-18.7\pm8.9$` | `$-21.1\pm9.4$` | n/a | `$-3.76\pm4.00$` | `$-1.45\pm4.68$` | `$7.56\pm3.13$` |
					|SoP-unet7-ft | 3 | 20 | `$-17.5\pm8.5$` | `$-20.3\pm9.3$` | n/a | `$-2.57\pm4.99$` | `$0.47\pm6.43$` | `$6.89\pm2.48$` |
					|SoP-unet5-Solos | 3 | 21 | `$-16.9\pm8.6$` | `$-18.7\pm8.9$` | n/a | `$-2.9\pm4.7$` | `$-1.67\pm5.34$` | `$11.07\pm6.87$` |

Summary of contributions

Proposed a conditioned model for time domain separation
Contributed to an audio-visual separation dataset
Demonstrated efficiency of the audio-visual approach
Conveyed a detailed study for fusion and conditioning methods

IV - Conclusion

Contributions

modern and efficient audio-visual methods for MIR

for MIR → classification, separation
methods → data fusion, conditioning
audio-visual → datasets (YouTube8m-MusInst, Solos)
modern and efficient → deep learning

Research focus

Where can we merge different data representations?

Late fusion and hybrid fusion for high-level concepts
Early fusion for continuous and strong conditioning

How should we merge data from different sources?

matching high-level concepts → concatenation / biasing
strong constraints → scaling
efficiency with dimension mismatch → FiLM

Publications

Audio-Visual MIR

O. Slizovskaia, E. Gómez, and G. Haro. (SMC, 2016) Automatic musical instrument recognition in audiovisual recordings by combining image and audio classification strategies.
https://github.com/Veleslavia/SMC2016
O. Slizovskaia, E. Gómez, and G. Haro. (ACM ICMR, 2017) Musical instrument recognition in user-generated videos using a multimodal convolutional neural network architecture.
https://github.com/Veleslavia/ICMR2017
O. Slizovskaia, E. Gómez, and G. Haro. (ISMIR-LBD, 2017) Correspondence between audio and visual deep models for musical instrument detection in video recordings.
O. Slizovskaia, E. Gómez, and G. Haro. (ICML-ML4M, 2018) A Case Study of Deep-Learned Activations via Hand-Crafted Audio Features.
O. Slizovskaia, L. Kim, E. Gómez, and G. Haro. (ICASSP, 2019) End-to-end sound source separation conditioned on instrument labels.
https://github.com/Veleslavia/vimss
J.F. Montesinos, O. Slizovskaia, and G. Haro. (IEEE MMSP, 2020) Solos: A Dataset for Audio-Visual Music Analysis.
https://github.com/JuanFMontesinos/Solos
O. Slizovskaia, G. Haro and E. Gómez. (IEEE/ACM TASLP, 2020) Conditioned Source Separation for Musical Instrument Performances (under review after major changes).
https://github.com/Veleslavia/conditioned-u-net

Collaborations

J. Pons, O. Slizovskaia, R. Gong, E Gómez, X. Serra. (EUSIPCO, 2017) Timbre analysis of music audio signals with convolutional neural networks.
https://github.com/Veleslavia/EUSIPCO2017
E. Fonseca, R. Gong, D. Bogdanov, O. Slizovskaia, E. Gómez, X. Serra. (DCASE, 2017) Acoustic scene classification by ensembling gradient boosting machine and convolutional neural networks.
R. Gong, E. Fonseca, D. Bogdanov, O. Slizovskaia, E. Gómez, X. Serra. (DCASE, 2017) Acoustic scene classification by fusing LightGBM and VGG-net multichannel predictions.
J. Serrà, D. Álvarez, V. Gómez, O. Slizovskaia, J.F. Núñez, and J. Luque. (ICLR, 2020) Input complexity and out-of-distribution detection with likelihood-based generative models.
D. Michelsanti, O. Slizovskaia, G. Haro, E. Gómez, Z.H. Tan, and J. Jensen. (INTERSPEECH, 2020) Vocoder-Based Speech Synthesis from Silent Videos.

Thank you!