Movies addict?! Prove it.

Alaa Desouky — Fri, 30 Aug 2019 00:16:51 +0000

Try it here: Movies Addict.

Movies Addict is a little online/(offline soon) game that tests your knowledge with movies quotes. Don't forget to share your score.

Tech used

React
Next

Current issues

Obviously not the best UI. 😅
Small Database.
Duplicate movie name, occasionally.

You can find the repository here: Movies Addict

Feel free to share your opinion, good or bad. Also, what other options could be added?

I scraped social media platforms and built an api with it, cause why not 🤷‍♂️

Alaa Desouky — Mon, 26 Aug 2019 05:05:09 +0000

First of all, I'm by no means a professional software engineer, so this won't be the cleanest code you'll see. I'm using this blog post to document my coding process, share my thoughts and the approaches I took to solve problems, also as feedback on how I did things wrong/right.

The inspiration for this project came from Wesbos's Twitter and Instagram scraping project.

You can find the repo here: status-scraper

So, what does it do exactly?

It's an api that accepts a social media flag and a username and returns the user status (eg. # of followers, following, posts, likes, etc...).

Endpoint is /scrape/:flag/:username, and currently the :flag can be any of the following:

t => twitter.com
r => reddit.com
g => github.com
b => behance.net
q => quora.com
i => instagram.com

So, a call for https://statusscraperapi.herokuapp.com/scrape/t/mkbhd would return the following response:

{
 user: "mkbhd",
 status: {
  twitterStatus: {
  tweets: "45,691",
  following: "339",
  followers: "3,325,617",
  likes: "25,255"
  }
 }
}

Tech used

Node
esm, an ECMAScript module loader
Express
Axios
Cheerio

Server configuration

// lib/server.js
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running on port ${PORT}`));


// lib/app.js
class App {
  constructor(app, routePrv) {
    this.app = express();
    this.config();
    this.routePrv = new Routes().routes(this.app);
  }

  config() {
    this.app.use(cors())
    this.app.use(helmet());
  }
}

export default new App().app;

Project structure

The app has three modules:

Module 1 - Router:

// lib/routes/router.js

// all routes have the same structure
export class Routes {
  routes(app) {
    ....
    // @route  GET /scrape/g/:user
    // @desc   log github user status
    app.get("/scrape/g/:user", async (req, res) => {
      const user = req.params.user;
      try {
        const githubStatus = await Counter.getGithubCount(
          `https://github.com/${user}`
        );
        res.status(200).send({ user, status: { githubStatus } });
      } catch (error) {
        res.status(404).send({
          message: "User not found"
        });
      }
    });
    ...
  }
}

Module 2 - Counter:

Acts as a middleware between the route and the acual scraping.
It gets the html page and pass it to the scraper module.

// lib/scraper/counter.js
class Counter extends Scraper {
  ...
  // Get github count
  async getGithubCount(url) {
    const html = await this.getHTML(url);
    const githubCount = await this.getGithubStatus(html);
    return githubCount;
  }
  ...
}

export default new Counter();

Module 3 - Scraper:

It's where all the work is done, and I'll be explaining each social network approach.
Let's start.

Twitter

Twitter response has multiple <a> elements that contain all data we want, and it looks like this:

<a class="ProfileNav-stat ProfileNav-stat--link u-borderUserColor u-textCenter js-tooltip js-nav" title="70 Tweets" data-nav="tweets" tabindex=0>
  <span class="ProfileNav-label" aria-hidden="true">Tweets</span>
  <span class="u-hiddenVisually">Tweets, current page.</span>
  <span class="ProfileNav-value"  data-count=70 data-is-compact="false">70</span>
</a>

The class ProfileNav-stat--link is unique for these elements.
With cheerio, we can simply get all <a> with the class, loop through them, and extract the data of the title attribute.
Now we have "70 Tweets", just split it and store as a key-value pair.

// lib/scraper/scraper.js

// Get twitter status
async getTwitterStatus(html) {
  try {
    const $ = cheerio.load(html);
    let twitterStatus = {};
    $(".ProfileNav-stat--link").each((i, e) => {
      if (e.attribs.title !== undefined) {
        let data = e.attribs.title.split(" ");
        twitterStatus[[data[1].toLowerCase()]] = data[0];
      }
    });
    return twitterStatus;
  } catch (error) {
    return error;
  }
}

Reddit user page has a <span id="profile--id-card--highlight-tooltip--karma"> on the right side with user's total karma, so it's very easy to get. But when hovered over, it displays post/comment karma.

Reddit response has a <script id="data"> that contains these two pieces of data nested inside an object.

window.___r = {"accountManagerModalData":....
...."sidebar":{}}}; window.___prefetches = ["https://www....};

Just extract the <script> data and parse 'em into json. But we need to get rid of window.___r = at the start, ; window.___prefetches.... at the end and everything after it.

This could be the laziest/worst thing ever :D
I split based on " = ", counted the #of characters starting from that ; -using a web app of course-, and sliced them out of the string. Now I have a pure object in a string.

// lib/scraper/scraper.js

  // Get reddit status
  async getRedditStatus(html, user) {
    try {
      const $ = cheerio.load(html);
      const totalKarma = $("#profile--id-card--highlight-tooltip--karma").html();

      const dataInString = $("#data").html().split(" = ")[1];
      const pageObject = JSON.parse(dataInString.slice(0, dataInString.length - 22));
      const { commentKarma, postKarma } = pageObject.users.models[user];

     return {totalKarma, commentKarma, postKarma};
    } catch (error) {
      return error;
    }
  }

It responded with status code 999! like, really linkedin.

I tried sending a customized head request that worked with everyone on stack overflow, but it did not work for me. Does it have something to do with csrf-token? I'm not really sure.
Anyways, that was a dead-end, moving on to the next one.

Github

This one was fairly easy, there are five <span class="Counter"> that displays the #of repositories, stars, etc.. Loop through 'em to extract the data, and with Cheerio I can get the element's parent, which is an <a> that has what these numbers represent. Store 'em as key-value pairs and we're ready to go.

// lib/scraper/scraper.js

 // Get github status
  async getGithubStatus(html) {
    try {
      const $ = cheerio.load(html);
      const status = {};
      $(".Counter").each((i, e) => {
        status[e.children[0].parent.prev.data.trim().toLowerCase()] = e.children[0].data.trim();
      });
      return status;
    } catch (error) {
      return error;
    }
  }

Behance

Also an easy one, a <script id="beconfig-store_state"> that has an object with all data required. Parse it into json and extract them.

Youtube - you broke my heart

Youtube's response is a huge mess, it has a punch of <script> tags that don't have any ids or classes. I wanted to get the channel's number of subscribers and total video views, both can be found in the About tab.

The desired <script> is similar to the Github one, I could use the same split, slice, parse thing and I'll be done.

But, these two simple numbers are nested like 12 levels deep within the object and there are arrays involved, it's basically hell.

So, I wrote a little helper function that accepts the large JSON/object and the object key to be extracted, and it returns an array of all matches.

// lib/_helpers/getNestedObjects.js

export function getNestedObjects(dataObj, objKey) {
  // intialize an empty array to store all matched results
  let results = [];
  getObjects(dataObj, objKey);

  function getObjects(dataObj, objKey) {
    // loop through the key-value pairs on the object/json.
    Object.entries(dataObj).map(entry => {
      const [key, value] = entry;
      // check if the current key matches the required key.
      if (key === objKey) {
        results = [...results, { [key]: value }];
      }

      // check if the current value is an object/array.
      // if the current value is an object, call the function again.
      // if the current value is an array, loop through it, check for an object, and call the function again.
      if (Object.prototype.toString.call(value) === "[object Object]") {
        getObjects(value, objKey);
      } else if (Array.isArray(value)) {
        value.map(val => {
          if (Object.prototype.toString.call(val) === "[object Object]") {
            getObjects(val, objKey);
          }
        });
      }
    });
  }

  // return an array of all matches, or return "no match"
  if (results.length === 0) {
    return "No match";
  } else {
    return results;
  }
}

As much as I was thrilled that getNestedObjects actually works -try it-, it didn't last for long though.
Somehow the received html didn't contain that <script>, and I have no idea why. I checked if it has the numbers, but a dead-end.
Thanks, youtube.

Quora

The response has multiple <span class="list_count">, and it's the exact same as Github.

Instagram

The response literarily has a problem from each one above:

✅ Multiple <script> tags with the same type="text/javascript"
✅ split, slice, parse
✅ The numbers are nested very deep with the object

 // Get instagram status
  async getInstagramStatus(html) {
    try {
      const $ = cheerio.load(html);
      // get the script containing the data
      let script;
      $('script[type="text/javascript"]').each((i, e) => {
        if (e.children[0] !== undefined && e.children[0].data.includes("window._sharedData =")) {
          return (script = e.children[0].data);
        }
      });

      // get json fromat string
      const dataInString = script.split(" = ")[1];

      // convert to json object
      const pageObject = JSON.parse(dataInString.slice(0, dataInString.length -1));

      // extract objects with status
      const [{ edge_followed_by }] = getNestedObjects(pageObject, "edge_followed_by");
      const [{ edge_follow }] = getNestedObjects(pageObject, "edge_follow");
      const [{ edge_owner_to_timeline_media }] = getNestedObjects(pageObject, "edge_owner_to_timeline_media");

      return {
        followers: edge_followed_by.count,
        following: edge_follow.count,
        posts: edge_owner_to_timeline_media.count
      };
    } catch (error) {
      return error;
    }
  }

At least I got to use the helper.

Wraping up

This was a cool project to make and I've learned a lot of stuff building it.
I've also created a frontend app with React and Next that interacts with the api, you can view it here: Status Logger
Maybe I'll write a blog post for it later.

In the meantime, feel free to share your opinion, good or bad, about it. Also, if you have any other social media networks to scrape.

Forem: Alaa Desouky

Movies addict?! Prove it.

Try it here: Movies Addict.

Tech used

Current issues

You can find the repository here: Movies Addict

I scraped social media platforms and built an api with it, cause why not 🤷‍♂️

So, what does it do exactly?

Tech used

Server configuration

Project structure

Module 1 - Router:

Module 2 - Counter:

Module 3 - Scraper:

Twitter

Reddit

Linkedin

Github

Behance

Youtube - you broke my heart

Quora

Instagram

Wraping up