Multimodal Urban Experience Analysis

Recording Urban Experiencewith Multimodal Data

Urban experience is shaped by our surroundings, but felt differently by each person. Does greener always mean better?

We captured these experiences through environmental data, field recordings, and self-reports.

Urban Environment

Background loudness
Surrounding greenness
Site Photos

Field Recording

Audio transcription
Multilingual sentiment classification

Emotion Ratings

7 Emotion Rating Scales

What is this place like?

How do participants describetheir experience?

How do participants ratetheir feelings?

Urban Environment

Background loudness
Surrounding greenness
Site Photos

What is this place like?

Field Recording

Audio transcription
Multilingual sentiment classification

How do participants describetheir experience?

Emotion Ratings

7 Emotion Rating Scales

How do participants ratetheir feelings?

Finding Patternsin the DataFinding Patternsin the Data

After collecting the data, we organized and preprocessed the different forms of record, then used Quantitative Analysis to examine the relationship between sentiment and urban environment.

Data Processing

Human Experience

HUMAN EXPERIENCE

Sentiment Classification

Automatic Sentiment Classification

TranscriptionMultilingual sentiment classification[-2, 2]

We used the multilingual sentiment classification model (tabularisai/multilingual-sentiment-analysis) to classify the transcriptions on a five-level scale, then mapped the results to [-2, 2].

Self-report Scales

Manual Sentiment Scoring

Self-reportDirection alignment[-2, 2]

We aligned the seven self-report emotion scales from negative to positive and normalized them to [-2, 2], placing self-reports and transcript sentiment scores on the same scale.

Urban Environment

URBAN ENVIRONMENT

Sonic Environment

LOUDNESS

AudioRMSLoudness

We calculated RMS from the recordings as a relative measure of Loudness.

Visual Environment

GREENNESS

Environment photosGreenfilterGreenness calculation

We calculated the proportion of green pixels in Site Photos as a measure of surrounding Greenness.

Diff1 = G − R · Diff2 = G − B · Diff = Diff1 × Diff2 (method from Li et al. (2015))

Quantitative Analysis

Sonic Environment × Emotion

LOUDNESS × SENTIMENT

We used linear regression to examine Loudness in relation to automatically scored Transcription sentiment and fused emotion.

Experiment 01Loudness × Transcription Sentiment · Transcription sentiment
Experiment 02Loudness × Fusion Sentiment · Fused emotion

Scatterplots and linear regressions comparing loudness with transcription sentiment and fusion sentiment

Higher Loudness tends to coincide with more negative emotion overall.

Compared with affective expression in free descriptions, emotion scores from structured Self-report scales varied more noticeably, with clearer positive and negative responses.

Visual Environment × Emotion

GREENNESS × SENTIMENT

We compared Greenness with overall emotion and individual Self-report dimensions through correlation analysis.

Analysis 01Greenness × Overall Sentiment · Overall emotion
Analysis 02Greenness × Self-report Sentiment · Self-reported emotion

Scatterplots and linear regressions comparing greenness with overall sentiment across locations

Greenness shows a more mixed relationship with emotion.

While the overall trend is positive, the relationship varies across locations. At Franz-von-Rinecker-Weg, higher Greenness coincided with more negative emotion.

From NumbersBack to People

Both analyses showed exceptions to the overall trend. We chose Hahnenhof for a closer look, returning to participants’ own words for context beyond the numbers.

LoudnessHahnenhofGreennessFranz-von-Rinecker-Weg

QUANTITATIVE EVIDENCE

Scatterplot of loudness level and sentiment score for participants at Hahnenhof — Loudness Level vs. Sentiment at Hahnenhof

CLOSE READING01 / 02

“This place looks really lively with lots of bars and restaurants around ... I can see lots of people around ... and this place also has trees ...”

— P16

WHAT WE FOUND

Positive doesn't mean uncomplicated.

A POSITIVE label doesn’t capture the full experience. The same environment can be perceived differently: lively or noisy, green or monotonous, depending on the person.

TAKEAWAY

Find the patternsSee the people

Quantitative AnalysisAnomalous ObservationClose Reading

← Back to projects