🤖 Quickly scrape tweets without API or headless browser

#undefined

Writing a scraping tool is a boring process, you have to use headless browser or an API (but that wouldn't be called scraping, would it?).

It takes a lot of time to develop and run such a tool. Whenever possible, it's best to avoid writing a standalone application for that.

My goal was to gather links to some tweets that were listed under Twitter's search page. I went to the search page, put this little snippet that I wrote in the DevTools and started scrolling until I was satisfied with the results.

It will probably stop working in the near future, since it is fully based on text content of some DOM nodes, but you can of course take a look at Twitter's DOM and modify it to your needs.


const links = new Set();

window.addEventListener('scroll', () =>

[...document.querySelector('[aria-label="Timeline: Search timeline"').children[0].children].forEach((el) => {

const singleLink = el.querySelectorAll('a')[3];

if (singleLink) {

 links.add(singleLink.getAttribute('href'));

}

}),

);

console.log(links);

Short-term memory for faster AI agents

AI agents struggle with latency and context switching. Redis fixes it with a fast, in-memory layer for short-term context—plus native support for vectors and semi-structured data to keep real-time workflows on track.

Start building

Top comments (0)

🐯 🚀 Timescale is now TigerData: Building the Modern PostgreSQL for the Analytical and Agentic Era

We’ve quietly evolved from a time-series database into the modern PostgreSQL for today’s and tomorrow’s computing, built for performance, scale, and the agentic future.

So we’re changing our name: from Timescale to TigerData. Not to change who we are, but to reflect who we’ve become. TigerData is bold, fast, and built to power the next era of software.