Forem: Raicuparta

Ditching worthless friends with Facebook data and JavaScript

Raicuparta — Sat, 31 Aug 2019 12:06:41 +0000

Friendships are hard to maintain. So much energy is wasted maintaining friendships that might not actually provide any tangible returns. I find myself thinking "Sure I've known her since kindergarten, she introduced me to my wife, and let me crash at her place for 6 months when I was evicted, but is this really a worthwhile friendship?".

I need to decide which friends to ditch. But what's the criteria? Looks? Intelligence? Money?

Surely, the value of an individual is subjective. There's no way to benchmark it empirically, right? WRONG. There is one surefire way to way to measure the worth of a friend: the amount of emoji reactions received on Facebook Messenger.

More laughing reactions means that's the funny friend. The one with the most angry reactions is the controversial one. And so on. Simple!

Counting manually is out of the question; I need to automate this task.

Getting the data

Scraping the chats would be too slow. There's an API, but I don't know if it would work for this. It looks scary and the documentation has too many words! I eventually found a way to get the data I need:

Facebook lets me download all the deeply personal information they collected on me over the years in an easily readable JSON format. So kind of them! I make sure to select only the data I need (messages), and select the lowest image quality, to keep the archive as small as possible. It can take hours or even days to generate.

The next day, I get an email notifying me that the archive is ready to download (all 8.6 GB of it) under the "Available Copies" tab. The zip file has the following structure:

messages
├── archived_threads
│   └── [chats]
├── filtered_threads
│   └── [chats]
├── inbox
│   └── [chats]
├── message_requests
│   └── [chats]
└── stickers_used
    └── [bunch of PNGs]

The directory I am interested in is inbox. The [chats] directories have this structure:

[ChatTitle]_[uniqueid]
├── gifs
│   └── [shared gifs]
├── photos
│   └── [shared photos]
├── videos
│   └── [shared videos]
├── files
│   └── [other shared files]
└── message_1.json

The data I need is in message_1.json. No clue why the _1 suffix is needed. In my archive there was no message_2.json or any other variation.

For example, if the chat I want to use is called "Nude Volleyball Buddies", the full path would be something like messages/inbox/NudeVolleyballBuddies_5tujptrnrm/message_1.json.

These files can get pretty big, so don't be surprised if your fancy IDE faints at the sight of it. The chat I want to analyze is about 5 years old, which resulted in over a million lines of JSON.

The JSON file is structured like this:

{
  "participants": [
    { "name": "Ricardo L" },
    { "name": "etc..." }
  ],
  "messages": [
    " (list of messages...) " 
  ],
  "title": "Nude Volleyball Buddies",
  "is_still_participant": true,
  "thread_type": "RegularGroup",
  "thread_path": "inbox/NudeVolleyballBuddies_5tujptrnrm"
}

I want to focus on messages. Each message has this format:

{
  "sender_name": "Ricardo L",
  "timestamp_ms": 1565448249085,
  "content": "is it ok if i wear a sock",
  "reactions": [
    {
      "reaction": "\u00f0\u009f\u0098\u00a2",
      "actor": "Samuel L"
    },
    {
      "reaction": "\u00f0\u009f\u0098\u00a2",
      "actor": "Carmen Franco"
    }
  ],
  "type": "Generic"
}

And I found what I was looking for! All the reactions listed right there.

Reading the JSON from JavaScript

For this task, I use the FileReader API:

<input type="file" accept=".json" onChange="handleChange(this)">

function handleChange(target) {
  const reader = new FileReader();
  reader.onload = handleReaderLoad;
  reader.readAsText(target.files[0]);
}

function handleReaderLoad (event) {
  const parsedObject = JSON.parse(event.target.result);
  console.log('parsed object', parsedObject);
}

I see the file input field on my page, and the parsed JavaScript object is logged to the console when I select the JSON. It can take a few seconds due to the absurd length. Now I need to figure out how to read it.

Parsing the data

Let's start simple. My first goal is to take my messages_1.json as input, and something like this as the output:

output = [
  {
    name: 'Ricardo L',
    counts: {
      '😂': 10,
      '😍': 3,
      '😢': 4,
    },
  },
  {
    name: 'Samuel L',
    counts: {
      '😂': 4,
      '😍': 5,
      '😢': 12,
    },
  },
  // etc for every participant
]

The participants object from the original JSON already has a similar format. Just need to add that counts field:

const output = parsedObject.participants.map(({ name }) => ({
  name,
  counts: {},
}))

Now I need to iterate the whole message list, and accumulate the reaction counts:

parsedObject.messages.forEach(message => {
  // Find the correct participant in the output object
  const outputParticipant = output.find(({ name }) => name === message.sender_name)

  // Increment the reaction counts for that participant
  message.reactions.forEach(({ reaction }) => {
    if (!outputParticipant.counts[reaction]) {
      outputParticipant.counts[reaction] = 1
    } else {
      outputParticipant.counts[reaction] += 1
    }
  })
})

This is how the logged output looks like:

I'm getting four weird symbols instead of emojis. What gives?

Decoding the reaction emoji

I grab one message as an example, and it only has one reaction: the crying emoji (😢). Checking the JSON file, this is what I find:

"reaction": "\u00f0\u009f\u0098\u00a2"

How does this character train relate to the crying emoji?

It may not look like it, but this string is four characters long:

\u00f0
\u009f
\u0098
\u00a2

In JavaScript, \u is a prefix that denotes an escape sequence. This particular escape sequence starts with \u, followed by exactly four hexadecimal digits. It represents a Unicode character in UTF-16 format. Note: it's a bit more complicated than that, but for the purposes of this article we can consider everything as being UTF-16.

For instance, the Unicode hex code of the capital letter S is 0053. You can see how it works in JavaScript by typing "\u0053" in the console:

Looking at the Unicode table again, I see the hex code for the crying emoji is 1F622. This is longer than four digits, so simply using \u1F622 wouldn't work. There are two ways around this:

UFT-16 surrogate pairs. This splits the big hex number into two smaller 4-digit numbers. In this case, the crying emoji would be represented as \ud83d\ude22.
Use the Unicode code point directly, using a slightly different format: \u{1F622}. Notice the curly brackets wrapping the code.

In the JSON, each reaction uses four character codes without curly brackets, and none of them can be surrogate pairs because they're not in the right range.

So what are they?

Let's take a look at a bunch of possible encodings for this emoji. Do any of these seem familiar?

That's pretty close! Turns out this is a UTF-8 encoding, in hex format. But for some reason, each byte is written as a Unicode character in UTF-16 format.

Knowing this, how do I go from \u00f0\u009f\u0098\u00a2 to \uD83D\uDE22?

I extract each character as a byte, and then merge the bytes back together as a UTF-8 string:

function decodeFBEmoji (fbString) {
  // Convert String to Array of hex codes
  const codeArray = (
    fbString  // starts as '\u00f0\u009f\u0098\u00a2'
    .split('')
    .map(char => (
      char.charCodeAt(0)  // convert '\u00f0' to 0xf0
    )
  );  // result is [0xf0, 0x9f, 0x98, 0xa2]

  // Convert plain JavaScript array to Uint8Array
  const byteArray = Uint8Array.from(codeArray);

  // Decode byte array as a UTF-8 string
  return new TextDecoder('utf-8').decode(byteArray);  // '😢'
}

So now I have what I need to properly render the results:

Selecting a friend to ditch

I want to calculate a score based on the count of each type of reaction. I need some variables:

Total message count for participant (T)
Total reactions sent by participant (SR)
Global average message count per participant (AVG)

And for the received reactions, I made some categories:

👍: Approval (A)
👎: Disapproval (D)
😆 and 😍: Positive emotion (PE)
😢 and 😠: Negative emotion (NE)
😮: Neutral, I'll chuck it

The final formula is:

The higher the resulting score, the better the person. Here is an explanation of how I reached this equation.

In JavaScript it would go something like this:

participants.forEach((participant) => {
  const {
    reactions,
    sentReactionCount,
    messageCount,
  } = participant

  const approval = reactions['👍']
  const disapproval = reactions['👎']
  const positiveEmotion = reactions['😆'] + reactions['😍']
  const negativeEmotions = reactions['😢'] + reactions['😠']

  const positiveFactor = (2 * approval + 3 * positiveEmotion + sentReactionCount)
  const negativeFactor = (2 * disapproval + 3 * negativeEmotions)
  const totalMessageFactor = Math.abs(messageCount - messageCountAverage) / (messageCountAverage)

  participant.score = (positiveFactor - negativeFactor) / totalMessageFactor
})

Displaying the information in table form makes it easier to parse:

Note: Due to privacy concerns I replaced my friend's real names with their home addresses.

Goodbye

With a quick look at the table I can finally decide who I need to remove from my life.

Farewell, cousin Sam.

Compute values on component mount with React Hooks: State vs Ref

Raicuparta — Tue, 27 Aug 2019 02:07:25 +0000

I recently came across this question:

I have a functional React component that computes a value when the component is mounted. After mounting, this value is never updated. Which is the better approach:

const value = useMemo(() => computeValue(), [])
or
const [value] = useState(() => computeValue())
?

And the answer is that they both work, but neither is ideal. Let's see why.

`useMemo`

import computeValue from 'expensive/compute';

// ...

const value= useMemo(computeValue, []);

At a first glance, useMemo might seem perfect for this. It only recomputes the value if the list of dependencies (second argument) changes. With an empty array as the dependency list, the value will only be computed on the first render. And it works. But here is the problem:

You may rely on useMemo as a performance optimization, not as a semantic guarantee. In the future, React may choose to “forget” some previously memoized values and recalculate them on next render, e.g. to free memory for offscreen components. Write your code so that it still works without useMemo — and then add it to optimize performance.

So we can't rely on useMemo for making sure a value is only computed once. Even if it works fine now, we shouldn't assume the behavior will be the same moving forward.

So what can we rely on?

`useState`

import computeValue from 'expensive/compute';

// ...

const [value] = useState(computeValue)

This one is closer to the correct answer, and it actually kinda works. But it is semantically incorrect.

When we pass the computeValue function as an argument to useState, it is used for lazy initialization. The result is that the value will be computed, but only on the first render. Seems like what we're looking for.

The problem is that this will block the first render until our computeValue function is done. The value will then never be updated again. So is this really a good use for component state? Let's think, what is the purpose of state in a React component?

We need state when we want the component to be able to update itself. Is that the case here? There is only ever one possible value during the component's lifetime, so no. We are using state for something other than its purpose.

So if not in the state, where do we store our computed value?

`useRef`

Before Hooks, you might think of refs as something you use to access a child component and focus an input. But useRef is much simpler than that:

Essentially, useRef is like a “box” that can hold a mutable value in its .current property.

useRef() is useful for more than the ref attribute. It’s handy for keeping any mutable value around similar to how you’d use instance fields in classes.

Mutating the .current property doesn’t cause a re-render.

How is this useful here?

import computeValue from 'expensive/compute';

// ...

const value = useRef(null)

if (value.current === null) {
  value.current = computeValue()
}

We initialize our ref with null. Then, we immediately update value.current to the computed value, but only if it hasn't been defined already.

The resulting behavior is identical to the previous example with useState. The render is blocked while the value is being computed. After that, the value is immediately available to be used on the first render.

The difference is just in the implementation: we're not adding unnecessary state to the component. We are instead using a ref for its original purpose: keeping a value that persists between renders.

But what if we don't want to block rendering while the value is being computed?

`useState` and `useEffect`

This solution will be more familiar to anyone who has tried React Hooks, as it is the standard way to do anything on component mount:

import computeValue from 'expensive/compute';

// ...

const [value, setValue] = useState(null)

useEffect(() => {
  setValue(computeValue)
}, [])

useEffect will run after the first render, whenever the dependency list changes. In this case, we set the dependency list to [], an empty array. This will make the effect run only after the first render, and never again. It does not block any renders.

Our first render will run with value = null, while the value is being computed. As soon as the computation is done, setValue will be called, and a re-render is triggered with the new value in place. The effect won't run again unless the component is re-mounted.

And this time it makes sense to have state, because there are two states the component can be in: before and after computing the value. This also comes with a bonus: we can show a "Loading..." message while the value is cooking.

Conclusion: State vs Ref vs Memo

The main lesson here is the difference between these:

useState:
- for storing values that persist across renders;
- updates trigger a re-render;
- updates via setter function.
useRef:
- also for storing values that persist across renders;
- updates don't trigger a re-render;
- mutable directly via the .current property.
useMemo:
- for performance optimization only

Dealing with Platform Inconsistencies as a North Korean Front-End Developer

Raicuparta — Fri, 23 Aug 2019 12:39:10 +0000

You're a North Korean engineer who's been selected to develop a new government project. It's an HTML form, which North Korean political leaders will fill in for [REDACTED] purposes.

One of the fields requires the user to select the title by which they prefer to be addressed. Since the list can get pretty long, you decide to go for your good old <select> element. It looks like this:

Windows (Chrome)	macOS (Safari)

Nothing out of the ordinary, perfectly acceptable in most cases.

You know that <select> has that kind of "search" that jumps to the items as you type. But you're not sure if the Great Leader is aware of this. You feel like this is not too big of a deal, as long as the list is in alphabetical order.

What about mobile?

Android (Chrome)	iOS (Safari)

Android tries to use as much of the screen as possible, covering the address bar. On iOS, the small number of visible items makes for an awful experience with larger lists. Both of them lack a way to search or filter list items.

Will the Father of the Nation look the other way? Not wanting to take any chances, you take this matter into your own hands. You want something that can be filtered on mobile, and makes better use of screen real estate.

On desktop platforms this is not too hard to achieve: just a custom dropdown with a text input for filtering. For mobile, you'll need something different. Let's focus on the mobile version, and presume that you'll have some way to pick the correct implementation depending on the platform.

This is your plan for mobile:

A full-screen modal with a fixed text input at the top for filtering, and a scrollable list of items below it. Your first instinct tells you the implementation should go like this:

<button onclick="openModal()">Select a title</button>
<div class="modal" id="modal">
  <div class="modal-header">
    <input type="text" id="filter-input">
    <button onclick="closeModal()">X</button>
  </div>
  <div class="modal-body">
    <button>Item 1</button>
    <button>Item 2</button>
    <!-- remaining items... -->
  </div>
</div>

.modal {
  display: none;
  position: fixed;
  top: 0;
  left: 0;
  height: 100vh;
  flex-direction: column;
}

.modal.show {
  display: flex;
}

.modal-body {
  flex: 1;
  overflow-y: auto;
}

const modal = document.getElementById('modal')
const filterInput = document.getElementById('filter-input')

function openModal() {
  modal.classList.add('show')
  filterInput.focus()
}

function closeModal() {
  modal.classList.remove('show')
}

The important bits:

position: fixed to fix the modal to the screen;
height: 100vh to make the height 100% of viewport's;
Modal divided in two parts: header and body;
Header's height defined by its children, no need to set it explicitly;
Body fills the remaining height with flex: 1;
scrolly-y: auto in the body to make it scrollable when the list doesn't fit.

It looks like this:

Android (Chrome)	iOS (Safari)

Looks good on iOS, but on Android the last items are being cut off. Why?

Some mobile browsers hide the address bar when the user scrolls down. This changes the visible viewport height, but not the meaning of 100vh. So 100vh actually a bit taller than what is initially visible.

Your modal has position: fixed , so you don't need to use vh units. height: 100% will fill the available height correctly:

Neat! This is already an improvement from the native versions of <select> on mobile. Now you need to implement the filter behavior.

You're pretty sure that your Guiding Sun Ray wouldn't want to go through the trouble of having to touch the filter input every time after opening the modal. So you should focus() the filter input as soon as the modal opens. This way, the keyboard pops up and the user can start typing right away. Let's see how it looks:

Android (Chrome)	iOS (Safari)

This time everything looks fine on Android. On iOS, the modal header is scrolled out of bounds once you try to scroll the list. What's going on?

iOS without Keyboard	iOS with Keyboard

When you filter by "Leader", the list becomes small enough to fit the screen without scrolling, but only if the keyboard isn't visible. On Android, opening the keyboard shrinks the viewport down to the visible area. But on iOS, the viewport size remains unchanged; it is just being covered by the keyboard. iOS lets you scroll the page while the keyboard is open, revealing that missing portion of the page. This behavior can break position: fixed elements like yours.

To make matters worse, there's no way to know how tall the keyboard will be, or if it is there at all (the user can be using a hardware keyboard). No clever CSS trick can save you this time.

So you need to have a scrollable list, where all the items are accessible, without knowing if an arbitrary portion of the lower part of screen is visible or not. This is your workaround:

You add a spacer at the bottom of the list (highlighted in green for visibility). The height of this spacer is the height of the list area, minus one element. This way, it's always possible to scroll all the way to the bottom, bringing the last element to the very top of the list.

There are still ways to make the modal scroll outside the viewport, and you need to patch them.

One way is by swiping on any non-scrollable elements currently visible. In your case, that's the modal header. You can't just disable all pointer events through CSS, since you need the inner elements (filter input and close button) to still be usable. The solution is to disable scrolling on touchmove events:

const header = document.getElementById('modal-header')

header.addEventListener('touchmove', event => {
  event.preventDefault()
})

The default reaction to touchmove is scrolling, so blocking that with preventDefault() will make it unscrollable.

Now let's take a small detour. I've been writing these examples in HTML + JavaScript to make the article a bit more universal. But I came across this spiral of workarounds while developing in React. This is how I define my event handler in React:

function handleTouchMove(event) {
  event.preventDefault()
}

// …

<Element onTouchMove={handleTouchMove} />

The expectation might be that in plain JavaScript, this would translate to something like this:

const element = document.getElementById('element')

element.addEventListener('touchmove', event => {
  // call the callback for this element
})

But what happens is closer to this (not real code):

document.addEventListener('touchmove', event => {
  const element = React.getElementFromEvent(event)

  // call the callback for this element
})

React binds the events at the document level, instead of binding them at the level of each individual node. Here is what happens when I try to preventDefault() touch events in React:

The browser blocks it. This was introduced with a Chrome update that made events be "passive" by default, and those can't be blocked with preventDefault at the document level. The solution is to bind the event manually at the node level, instead of doing it through React's event system:

ref = React.createRef();

componentDidMount() {
  ref.addEventListener('touchmove', handleTouchMove)
}

function handleTouchMove (event) {
  event.preventDefault()
}

// …

<Element ref={ref} onTouchMove={handleTouchMove} />

So yes, particularly in React, this workaround requires a workaround.

As I write this, React's event system is being rewritten, so the problem may no longer exist by the time you read this article.

Now back to your problem.

There is one more way to scroll your hopes and dreams away. If the user insists on scrolling when there are no more items to show, the viewport can be scrolled up. None of this fazes you anymore, you just jam another workaround in there:

const modalBody = document.getElementById('modal-body')

menuScroll = () => {
  if (modalBody.scrollHeight - modalBody.scrollTop === modalBody.clientHeight) {
    modalBody.scrollTop -= 1
  }
}

modalBody.addEventListener('scroll', menuScroll)

You push the list's scroll position one pixel away from the edge when the scroll reaches the bottom. This way, the outer scroll is never triggered.

The solution is already pretty solid, but there is one more thing you'd like to improve. The modal suddenly covering the screen might be a bit jarring. What if His Excellency isn't paying attention and gets spooked? Who will take care of your kids?

A simple transition animation could make it easier to follow. Perhaps you could slide the modal from the bottom of the screen? Easy to achieve with CSS transitions:

.modal {
  /* ... */

  display: flex;
  top: 100vh;
  transition: top 500ms;
}

.modal.show {
  top: 0;
}

Now, instead of initializing your modal with display: none and top: 0, you start it already with display: flex, but pushed outside the viewport with top: 100vh. When the modal is set to visible, it will scroll smoothly to the top of the screen. Let's see the results:

Android (Chrome)	iOS (Safari)

So close! Android is behaving well again, while iOS blasts the modal to outer space as soon as it is visible. It seems like toggling the keyboard while the modal is being animated isn't a good idea. You feel pretty confident that showing the keyboard only after the animation is done should fix it:

function openModal() {
  modal.classList.add('show')

  // new
  setTimeout(() => {
    filterInput.focus()
  }, 500)
}

Simple enough. You wait for 500ms, the same as the transition duration, and only then you focus() the input to make the keyboard pop up. You tell yourself that you'll clean this up later, maybe using events or some fancy library, instead of relying on the values being consistent between JS and CSS. But you know it won't happen. The result:

Android	iOS

iOS doesn't seem to be focusing the input at all. Of course, it couldn't be that easy. iOS only allows focus events to happen as a direct result of a user interaction, and setTimeout isn't that. Your workaround is to turn the "Select a title" button into a text input:

<input onfocus="openModal()" readonly=true placeholder="Select a title">

The readonly hides the caret and makes sure the user can't type anything into this new input during the transition. This way, iOS will show the keyboard based on the first focus event, allowing you to change the focus to the second input after the transition is done.

And it works! You're finally done. You feel proud of your work, knowing your family will live at least another couple months.

Find the full code for the modal here