Societal Attitudes Toward AI

Public sentiment toward AI has shifted over the past two decades, and this project mines the technology-article dataset for how, and for the events that moved it. The corpus runs to 17,092 articles and 28 million words spanning 2000 to 2023, drawn from news outlets and AI labs. Three Jupyter notebooks carry the work: one profiles the dataset, one scores sentiment, and one runs the Procrustes comparison.

After Porter stemming and English stopword removal, a gensim LdaModel fit over a corpora.Dictionary of the articles extracts ten latent topics, each a distribution over words and each article a mixture over topics. The top topics name the recurring threads of the discourse: chatbots and privacy, deep learning and DeepMind, autonomy and security, the metaverse.

nltk's SentimentIntensityAnalyzer scores each article on four axes (positive, negative, neutral, and a compound aggregate), averaged by year. The positive, negative, and neutral bands stay fairly flat; the compound score is what jumps, and reading it per topic rather than over the whole corpus exposes swings the global average smooths away.

Comparing two years directly is the hard part. Each year's articles produce an LDA topic-distribution matrix, and a distribution is defined only up to rotation and scale, so two years cannot be lined up as they stand. scipy.spatial.procrustes standardizes both matrices and finds the orthogonal $R$ that best overlays one on the other,

solved in closed form from the SVD $B^{\top }A=U\Sigma V^{\top }$, giving $R=U{V}^{\top }$; the leftover residual is the disparity between the two years. Running it on consecutive years, and on every year against 2022, turns the drift of the AI conversation into a single number per pair. The disparities spike around the dot-com bust of 2000–2002, again through 2013–2016, and then climb sharply from 2018 onward, peaking across 2021–2023 as large language models entered the discourse.

The full report has the results. Collaborative project with Aimen Abdulaziz and Angelic McPherson.