Emotion Detection in Audio

Speech emotion recognition — predicting the emotion in a spoken clip — comparing two ways of turning raw audio into features for a classifier: hand-engineered MFCCs and learned representations from Wav2Vec2. The clips come from the Kaggle Audio Emotions set, sorted into seven labels (angry, sad, disgusted, fearful, happy, neutral, surprised) with an 80/20 train/test split. The task is classification, but the interesting choice is upstream, in how the waveform is represented.

MFCCs follow a fixed recipe. librosa.feature.mfcc windows the waveform into short frames, warps each frame's power spectrum onto the perceptual mel scale, log-compresses, and runs a discrete cosine transform, keeping 40 coefficients per frame. Averaging over time collapses each clip to a single 40-number vector, the coarse spectral envelope that carries timbre and vowel quality. It is compact and requires no training, but it discards whatever the fixed recipe does not capture.

Wav2Vec2 learns its features instead. The learned path loads the audio at Wav2Vec2's native 16 kHz and passes it through the wav2vec2-base-960h feature extractor. Wav2Vec2 is a transformer pretrained on large amounts of unlabeled speech with a self-supervised objective, so its activations already encode phonetic and prosodic structure. These sequences vary in length, so they are zero-padded to the longest clip before batching. Feeding them to the classifier brings knowledge from far more audio than the labeled emotion set alone: transfer learning in place of hand-engineering.

Either way, the head is a classifier: both feature paths feed a small PyTorch Lightning module trained with cross-entropy and Adam. The MFCC head is a 1-D convolution over the coefficient vector followed by a linear layer and softmax over the seven emotions; the Wav2Vec2 head is a linear projection down to the same seven logits. Holding the objective fixed and swapping only the features isolates the comparison between the engineered and the learned representation.

Collaborative project with Ivy (Aiwei) Zhang and Carlos Guerrero Alvarez.