AI / Tech Dataset

A web scraper written in Haskell that collected 17,000+ articles from technology publishers — DeepMind, MIT Technology Review, OpenAI, Singularity Hub, and TechCrunch — into an open dataset published on HuggingFace.

Extraction runs as an arrow pipeline. The parser (MyData.Parser) is built on HXT, whose arrows generalize a plain function $b\to c$ into a composable stage over an XML tree. loadPage fetches the page with simpleHttp, hands the bytes to readString [withParseHTML yes, withWarnings no], and runs arrows over the resulting DOM with runX. Each field is its own small arrow chain: getWords descends with //> and merges paragraph and heading nodes with the choice operator <+> (hasName "p" <+> hasName "h1" <+> ...), then deep (isText >>> getText) pulls their text; getLinks selects a nodes and reads getAttrValue "href"; getTitle and getYear do the same over their tags. The results assemble a WebPage record — title, year, links, and the body text as a prefix tree.

A Config record loaded from config.yml with Data.Yaml drives the run, keeping the tuning in configuration rather than code: domains are the seed URLs, targets are the keywords a page is scored against, limit caps the work queue, and wordcount sets how many keyword hits a page needs to be kept. Retargeting the scraper at a new publisher is a line in config.yml, not a change to the parser.

Main.iter walks each publisher's page graph outward from the seed URLs by breadth-first search, holding the frontier as a queue and the visited URLs as a Data.Set; set difference (\\) drops links already seen, and isAllowed keeps the crawl inside the seed domains. Each fetched page's body is folded into a Trie of words, and hasKeyWords accepts the page only when at least wordcount targets are present — the filter that kept the collection on-topic across a run of 17,000-plus articles.

The result is cleaned, titled, dated article text, released open-source for downstream mining. Collaborative project with Aimen Abdulaziz and Angelic McPherson.