Forem: Rehan Sayyed

I Tried Scraping LinkedIn Posts with Puppeteer and This Is What Actually Worked

Rehan Sayyed — Mon, 30 Mar 2026 18:58:25 +0000

I Tried Scraping LinkedIn Posts with Puppeteer and It Fought Back

I thought this would take 20 minutes.

Open Puppeteer.
Grab some text.
Save it to JSON. Done.

Instead, LinkedIn reminded me that modern web apps are not just pages.

They are systems.
They react.
They delay.
They break your assumptions.

And suddenly, a simple script turns into a late night debugging session.

This is what actually worked.

📚 Table of Contents

🚀 Setting up Puppeteer
🌐 Launching a real browser
🔐 Logging into LinkedIn
⏳ Handling the security check
🔗 Providing the post links
🔁 Visiting each post
📜 Scrolling to load content
🖼️ Waiting for images
🧠 Extracting content and images
📦 Storing the data
💾 Saving it to a file
🧩 Final thoughts
🧾 Full working script

🚀 Setting up Puppeteer

We start simple.

const puppeteer = require('puppeteer');
const fs = require('fs');

Puppeteer controls the browser
fs stores the data

Clean. Minimal. Enough to begin.

Then we wrap everything so we can use async and await properly.

(async () => {

🌐 Launching a real browser

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

I kept headless mode off on purpose.

Because LinkedIn is very sensitive to automation.

Headless browsers get flagged faster.
A visible browser behaves more like a real user.

And when you are debugging
visibility is everything.

🔐 Logging into LinkedIn

await page.goto('https://www.linkedin.com/login', {
  waitUntil: 'networkidle2',
  timeout: 60000
});

await page.type('#username', 'xxxx');
await page.type('#password', 'xxxxxx');
await page.click('button[type="submit"]');
await page.waitForNavigation({ timeout: 60000 });

At this point, everything felt done.

I thought I was finished.
I was not.

LinkedIn often throws:

security checks
captchas
verification screens

So even after login, we pause.

⏳ Handling the security check

console.log('Please complete the security check...');
await new Promise(resolve => setTimeout(resolve, 12000));

No hacks. No bypass.

Just wait.

Sometimes the most reliable solution
is the least clever one.

🔗 Providing the post links

const postLinks = [ ... ];

Instead of crawling everything, I used a fixed list of URLs.

Why?

Predictable input gives predictable output.

LinkedIn infinite scroll can get messy very quickly.

🔁 Visiting each post

for (const link of postLinks) {
  await page.goto(link, {
    waitUntil: 'domcontentloaded',
    timeout: 60000
  });

Notice I used domcontentloaded.

LinkedIn keeps making background requests forever.
Waiting for network idle can hang your script.

This small change
saved a lot of frustration.

📜 Scrolling to load content

This is where things started breaking.

await page.evaluate(async () => {
  let scrollHeight = document.body.scrollHeight;

  let scrollInterval = setInterval(() => {
    window.scrollBy(0, 1000);

    if (document.body.scrollHeight > scrollHeight) {
      scrollHeight = document.body.scrollHeight;
    }

    if (document.body.scrollHeight === scrollHeight) {
      clearInterval(scrollInterval);
    }
  }, 300);
});

LinkedIn loads content lazily.

No scroll.
No content.

So we simulate a real user:

Scroll
Pause
Repeat

Until nothing new loads.

🖼️ Waiting for images

await page.waitForSelector('img[src*="media.licdn.com"]', {
  timeout: 15000
});

Without this step, your data will be incomplete.

This ensures images are fully loaded before extraction.

🧠 Extracting content and images

const postData = await page.evaluate(() => {
  const contentElement =
    document.querySelector('[data-test-post-container] .break-words') ||
    document.querySelector('.feed-shared-update-v2__description') ||
    document.querySelector('.feed-shared-text__text-view') ||
    document.querySelector('span.break-words');

  const images = [];
  const imageElements = document.querySelectorAll('img[src*="media.licdn.com/"]');

  if (imageElements.length > 1) {
    images.push(imageElements[1].src);
  }

  return {
    content: contentElement
      ? contentElement.innerText.trim()
      : 'No content found',
    images: images.length > 0 ? images : ['No post image found']
  };
});

This is real world scraping.

There is no single selector that always works.

So we try multiple fallbacks until something works.

Also:

first image → profile
second image → actual post

Not perfect.
But practical.

📦 Storing the data

scrapedPosts.push({
  link,
  content: postData.content,
  images: postData.images
});

We collect everything step by step.

Simple. Clean. Reliable.

💾 Saving it to a file

fs.writeFileSync(
  'linkedInScrapedPosts.json',
  JSON.stringify(scrapedPosts, null, 2)
);

Now everything is structured and ready to use.

🧩 Final thoughts

This script is not perfect.

But it is real.

It reflects how scraping actually works:

You try something
It breaks
You tweak
You retry

And slowly, it starts working.

The real lesson?

Web pages are not static
They are systems reacting to user behavior

Once you understand that, everything changes.

⚠️ A small note

Always respect platform rules and use scraping responsibly.

🧾 Full working script

If you just want it to work, copy this and run it.

const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  await page.goto('https://www.linkedin.com/login', {
    waitUntil: 'networkidle2',
    timeout: 60000
  });

  await page.type('#username', 'YOUR_EMAIL');
  await page.type('#password', 'YOUR_PASSWORD');
  await page.click('button[type="submit"]');
  await page.waitForNavigation({ timeout: 60000 });

  console.log('Complete the security check if prompted...');
  await new Promise(resolve => setTimeout(resolve, 12000));

  const postLinks = [
    // add your links here
  ];

  const scrapedPosts = [];

  for (const link of postLinks) {
    await page.goto(link, {
      waitUntil: 'domcontentloaded',
      timeout: 60000
    });

    await page.evaluate(async () => {
      let scrollHeight = document.body.scrollHeight;

      let scrollInterval = setInterval(() => {
        window.scrollBy(0, 1000);

        if (document.body.scrollHeight > scrollHeight) {
          scrollHeight = document.body.scrollHeight;
        }

        if (document.body.scrollHeight === scrollHeight) {
          clearInterval(scrollInterval);
        }
      }, 300);
    });

    await page.waitForSelector('img[src*="media.licdn.com"]', {
      timeout: 15000
    });

    const postData = await page.evaluate(() => {
      const contentElement =
        document.querySelector('[data-test-post-container] .break-words') ||
        document.querySelector('.feed-shared-update-v2__description') ||
        document.querySelector('.feed-shared-text__text-view') ||
        document.querySelector('span.break-words');

      const images = [];
      const imageElements = document.querySelectorAll('img[src*="media.licdn.com/"]');

      if (imageElements.length > 1) {
        images.push(imageElements[1].src);
      }

      return {
        content: contentElement
          ? contentElement.innerText.trim()
          : 'No content found',
        images: images.length > 0 ? images : ['No post image found']
      };
    });

    scrapedPosts.push({
      link,
      content,
      images
    });
  }

  fs.writeFileSync(
    'linkedInScrapedPosts.json',
    JSON.stringify(scrapedPosts, null, 2)
  );

  console.log(scrapedPosts);

  await browser.close();
})();

I Tried Scraping LinkedIn Posts with Puppeteer and This Is What Actually Worked

Rehan Sayyed — Mon, 30 Mar 2026 18:58:25 +0000

I Tried Scraping LinkedIn Posts with Puppeteer and It Fought Back

I thought this would take 20 minutes.

Open Puppeteer.
Grab some text.
Save it to JSON. Done.

Instead, LinkedIn reminded me that modern web apps are not just pages.

They are systems.
They react.
They delay.
They break your assumptions.

And suddenly, a simple script turns into a late night debugging session.

This is what actually worked.

📚 Table of Contents

🚀 Setting up Puppeteer
🌐 Launching a real browser
🔐 Logging into LinkedIn
⏳ Handling the security check
🔗 Providing the post links
🔁 Visiting each post
📜 Scrolling to load content
🖼️ Waiting for images
🧠 Extracting content and images
📦 Storing the data
💾 Saving it to a file
🧩 Final thoughts
🧾 Full working script

🚀 Setting up Puppeteer

We start simple.

const puppeteer = require('puppeteer');
const fs = require('fs');

Puppeteer controls the browser
fs stores the data

Clean. Minimal. Enough to begin.

Then we wrap everything so we can use async and await properly.

(async () => {

🌐 Launching a real browser

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

I kept headless mode off on purpose.

Because LinkedIn is very sensitive to automation.

Headless browsers get flagged faster.
A visible browser behaves more like a real user.

And when you are debugging
visibility is everything.

🔐 Logging into LinkedIn

await page.goto('https://www.linkedin.com/login', {
  waitUntil: 'networkidle2',
  timeout: 60000
});

await page.type('#username', 'xxxx');
await page.type('#password', 'xxxxxx');
await page.click('button[type="submit"]');
await page.waitForNavigation({ timeout: 60000 });

At this point, everything felt done.

I thought I was finished.
I was not.

LinkedIn often throws:

security checks
captchas
verification screens

So even after login, we pause.

⏳ Handling the security check

console.log('Please complete the security check...');
await new Promise(resolve => setTimeout(resolve, 12000));

No hacks. No bypass.

Just wait.

Sometimes the most reliable solution
is the least clever one.

🔗 Providing the post links

const postLinks = [ ... ];

Instead of crawling everything, I used a fixed list of URLs.

Why?

Predictable input gives predictable output.

LinkedIn infinite scroll can get messy very quickly.

🔁 Visiting each post

for (const link of postLinks) {
  await page.goto(link, {
    waitUntil: 'domcontentloaded',
    timeout: 60000
  });

Notice I used domcontentloaded.

LinkedIn keeps making background requests forever.
Waiting for network idle can hang your script.

This small change
saved a lot of frustration.

📜 Scrolling to load content

This is where things started breaking.

await page.evaluate(async () => {
  let scrollHeight = document.body.scrollHeight;

  let scrollInterval = setInterval(() => {
    window.scrollBy(0, 1000);

    if (document.body.scrollHeight > scrollHeight) {
      scrollHeight = document.body.scrollHeight;
    }

    if (document.body.scrollHeight === scrollHeight) {
      clearInterval(scrollInterval);
    }
  }, 300);
});

LinkedIn loads content lazily.

No scroll.
No content.

So we simulate a real user:

Scroll
Pause
Repeat

Until nothing new loads.

🖼️ Waiting for images

await page.waitForSelector('img[src*="media.licdn.com"]', {
  timeout: 15000
});

Without this step, your data will be incomplete.

This ensures images are fully loaded before extraction.

🧠 Extracting content and images

const postData = await page.evaluate(() => {
  const contentElement =
    document.querySelector('[data-test-post-container] .break-words') ||
    document.querySelector('.feed-shared-update-v2__description') ||
    document.querySelector('.feed-shared-text__text-view') ||
    document.querySelector('span.break-words');

  const images = [];
  const imageElements = document.querySelectorAll('img[src*="media.licdn.com/"]');

  if (imageElements.length > 1) {
    images.push(imageElements[1].src);
  }

  return {
    content: contentElement
      ? contentElement.innerText.trim()
      : 'No content found',
    images: images.length > 0 ? images : ['No post image found']
  };
});

This is real world scraping.

There is no single selector that always works.

So we try multiple fallbacks until something works.

Also:

first image → profile
second image → actual post

Not perfect.
But practical.

📦 Storing the data

scrapedPosts.push({
  link,
  content: postData.content,
  images: postData.images
});

We collect everything step by step.

Simple. Clean. Reliable.

💾 Saving it to a file

fs.writeFileSync(
  'linkedInScrapedPosts.json',
  JSON.stringify(scrapedPosts, null, 2)
);

Now everything is structured and ready to use.

🧩 Final thoughts

This script is not perfect.

But it is real.

It reflects how scraping actually works:

You try something
It breaks
You tweak
You retry

And slowly, it starts working.

The real lesson?

Web pages are not static
They are systems reacting to user behavior

Once you understand that, everything changes.

⚠️ A small note

Always respect platform rules and use scraping responsibly.

🧾 Full working script

If you just want it to work, copy this and run it.

const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  await page.goto('https://www.linkedin.com/login', {
    waitUntil: 'networkidle2',
    timeout: 60000
  });

  await page.type('#username', 'YOUR_EMAIL');
  await page.type('#password', 'YOUR_PASSWORD');
  await page.click('button[type="submit"]');
  await page.waitForNavigation({ timeout: 60000 });

  console.log('Complete the security check if prompted...');
  await new Promise(resolve => setTimeout(resolve, 12000));

  const postLinks = [
    // add your links here
  ];

  const scrapedPosts = [];

  for (const link of postLinks) {
    await page.goto(link, {
      waitUntil: 'domcontentloaded',
      timeout: 60000
    });

    await page.evaluate(async () => {
      let scrollHeight = document.body.scrollHeight;

      let scrollInterval = setInterval(() => {
        window.scrollBy(0, 1000);

        if (document.body.scrollHeight > scrollHeight) {
          scrollHeight = document.body.scrollHeight;
        }

        if (document.body.scrollHeight === scrollHeight) {
          clearInterval(scrollInterval);
        }
      }, 300);
    });

    await page.waitForSelector('img[src*="media.licdn.com"]', {
      timeout: 15000
    });

    const postData = await page.evaluate(() => {
      const contentElement =
        document.querySelector('[data-test-post-container] .break-words') ||
        document.querySelector('.feed-shared-update-v2__description') ||
        document.querySelector('.feed-shared-text__text-view') ||
        document.querySelector('span.break-words');

      const images = [];
      const imageElements = document.querySelectorAll('img[src*="media.licdn.com/"]');

      if (imageElements.length > 1) {
        images.push(imageElements[1].src);
      }

      return {
        content: contentElement
          ? contentElement.innerText.trim()
          : 'No content found',
        images: images.length > 0 ? images : ['No post image found']
      };
    });

    scrapedPosts.push({
      link,
      content,
      images
    });
  }

  fs.writeFileSync(
    'linkedInScrapedPosts.json',
    JSON.stringify(scrapedPosts, null, 2)
  );

  console.log(scrapedPosts);

  await browser.close();
})();

My Git Commits Were Linked to the Wrong GitHub Account. Here’s Why.

Rehan Sayyed — Wed, 11 Mar 2026 03:34:27 +0000

Why My Git Commits Appeared Under the Wrong GitHub Account

Every developer eventually has this moment.
You push your commits, everything succeeds… and GitHub says someone else wrote them.

I had just finished pushing a few commits to a repository. The push succeeded without any errors. Git behaved exactly the way it always does.

Naturally, I opened GitHub to see the commits in the repository history.

And then something looked odd.

The commits were there.
The code was correct.
But the author was not me.

Instead of appearing under my GitHub profile, the commits were showing up under a different identity. For a few seconds I honestly wondered if I had somehow pushed from another machine or used the wrong account.

That confusion lasted longer than I want to admit.

Everything looked normal locally

The workflow had been completely standard.

git add .
git commit -m "feat: add reconciliation API"
git push origin main

No warnings. No strange output. Just a normal push.

Yet on GitHub the commit author was wrong.

At first it felt like one of those odd GitHub UI glitches that sometimes resolves after a refresh. I refreshed the page.

Nothing changed.

The commits existed, but they were linked to the wrong identity.

The important thing many developers forget

This is where the key detail about Git becomes important.

Git does not know anything about GitHub usernames.

When you create a commit, Git stores two pieces of author information inside that commit object

• user.name
• user.email

That is all.

Later, when GitHub receives the commit, it checks the email stored in the commit and tries to match it with a GitHub account.

If the email matches one of the emails on your GitHub account, the commit gets linked to your profile.

If it does not match, GitHub simply shows the commit as belonging to that email or as an unlinked author.

So the problem was almost certainly inside my local Git configuration.

Investigating the local configuration

The next step was checking what identity Git was actually using.

git config user.name
git config user.email

And there it was.

The email configured in my repository was not the same email connected to my GitHub account.

Git had been happily creating commits with the wrong identity the entire time.

From Git’s perspective everything was correct. The commit contained a name and an email. That is all Git cares about.

But GitHub could not associate that email with my account.

The annoying part

Changing the Git config only affects future commits.

The commits I had already pushed still contained the wrong author information.

Which meant the history itself needed to be rewritten.

This is one of those situations where Git’s powerful history editing tools become incredibly useful.

Rewriting the last three commits

Since the problem was only in the last few commits, interactive rebase was the cleanest solution.

git rebase -i HEAD~3

This command tells Git to open an interactive editor for the last three commits.

The editor shows something like this

pick a1b2c3 commit message
pick d4e5f6 commit message
pick g7h8i9 commit message

To modify the commits, each pick is changed to edit.

edit a1b2c3 commit message
edit d4e5f6 commit message
edit g7h8i9 commit message

After saving the file, Git pauses at each commit so it can be modified.

Fixing the author information

For each paused commit, the author can be reset using

git commit --amend --reset-author --no-edit

This command rewrites the commit with the correct identity taken from the current Git configuration.

The commit message stays the same because of --no-edit.

Once the commit is updated, the rebase continues.

git rebase --continue

Git then moves to the next commit and repeats the process.

The slightly scary message: detached HEAD

During this process Git often prints a message saying something like

“HEAD detached at …”

That message can look intimidating the first time you see it.

Normally the HEAD pointer references a branch such as main. During an interactive rebase, Git temporarily checks out individual commits while rewriting them.

When that happens, HEAD points directly to a specific commit instead of a branch.

This state is called a detached HEAD. It is completely normal during a rebase and simply means Git is modifying history step by step.

Updating GitHub after rewriting history

Once the rebase finished, the commits were fixed locally.

But the remote repository still had the old commits.

Because the commit history had changed, a normal push would be rejected. The remote branch needed to be updated with the rewritten history.

git push --force

After the force push, I refreshed the GitHub page.

This time the commits appeared exactly where they should have been.

Under my profile.

The lesson hiding behind this small bug

This entire issue comes from a simple misunderstanding about how Git works.

Git does not connect commits to GitHub usernames.

Git only stores a name and an email inside each commit.

GitHub later links the commit to a user account using that email.

Once you understand that model, this behavior suddenly makes perfect sense.

And the best way to avoid the problem entirely is to set your global Git configuration correctly once.

git config --global user.name "Your Name"
git config --global user.email "your@email.com"

From that point forward, every repository on your machine will use the correct identity.

It is a small configuration detail, but it quietly controls how your work appears across every Git repository you touch.

One of those subtle lessons that makes you understand Git just a little bit better.

Git is simple in principle but endlessly surprising in practice. Understanding how commits store identity can save you hours of confusion later.