Forem: Osama Qarem

Parsing Custom Dates in JS

Osama Qarem — Sun, 05 Sep 2021 21:14:49 +0000

Recently at work we moved our iOS React Native app from JSC to Hermes which became available for iOS as of RN0.64. Previously we were using Hermes only on Android and it was working great. Now that Android and iOS run the same engine we have confidence that our JavaScript output for both platforms will be equal – they should work and fail in JS land in the same ways.

One thing which immediately broke on iOS was our Apple Health integration (which of course is iOS only) – specifically when parsing dates for user activities on the JS side. The date string returned from Objective-C to JavaScript looks like this:

2021-08-31T17:00:00.000+0300

We could parse this previously on JSC using new Date() but for some reason it results in an "Invalid Date" on Hermes. Turns out the problem was in the time zone offset. Hermes could only parse ISO dates containing a time zone offset in ±HH:mm format:

+03:00

but not in a format without a colon ±HHmm:

+0300

Reading about ISO 8601, it seems that both formats would be correct. I've submitted an issue about this on facebook/hermes to get more information and apparently JavaScript implements a simplified version of ISO 8601 which doesn't specify the time zone offset without a colon:

@neildhar: Hi @osamaqarem, thanks for reporting this. Note that the JavaScript uses a simplified version of ISO 8601, and does not specify the format without the colon for the timezone. However, as you mentioned, most other engines seem to support it, so we probably should too.

So until this gets fixed in Hermes we will need to implement a workaround. We used the custom date formatter from Day.js – a great library which we were already making use of in our codebase.

Parsing Custom Dates

Day.js depends on new Date() for parsing under the hood. Meaning it would still fail to parse our date when running Hermes. A small bundle size is one of the main features of Day.js so the package ships with only core functionality. To extend its capabilities, we use plugins. And the one we need is CustomParseFormat:

import dayjs from "dayjs"
import customParseFormat from "dayjs/plugin/customParseFormat"

dayjs.extend(customParseFormat)

That's it! Quite straightforward.

Now we just need to define our format. To give Dayjs the ability to parse our date, we need to tell it what our date looks like based on these defined formats, e.g.:

Format	Output	Description
`YY`	18	Two-digit year
`YYYY`	2018	Four-digit year
`M`	1-12	The month, beginning at 1
`MM`	01-12	The month, 2-digits
`D`	1-31	The day of the month
`DD`	01-31	The day of the month, 2-digits
`H`	0-23	The hour
`HH`	00-23	The hour, 2-digits
`m`	0-59	The minute
`mm`	00-59	The minute, 2-digits
`ss`	00-59	The second, 2-digits
`SSS`	000-999	The millisecond, 3-digits
`Z`	+05:00	The offset from UTC, ±HH:mm
`ZZ`	+0500	The offset from UTC, ±HHmm

You can find the full reference here.

Our date looks like 2021-08-31T17:00:00.000+0300, so the format we need would be:

'T' here is a constant which would be present in the expected date string
YYYY-MM-DDTHH:mm:ss.SSSZZ

Using our custom format:

dayjs("2021-08-31T17:00:00.000+0300", "YYYY-MM-DDTHH:mm:ss.SSSZZ").toISOString()
// 2021-08-31T14:00:00.000Z

That works! And once the fix for Hermes is in, we can replace it with a regular JS date constructor call.

Check out the source code for CustomParseFormat.

This article was cross-posted from my personal blog. Do subscribe to me there!

Uploading iOS apps with special characters to App Store Connect

Osama Qarem — Sun, 22 Aug 2021 12:10:53 +0000

Suppose you have an Xcode project where the product name includes special characters like an umlaut 'ü':

If you try to archive it and upload it to App Store Connect you might be faced with the following mysterious error:

ERROR ITMS-90035: "Invalid Signature. A sealed resource is missing or invalid. The file at path [Dümmy.app/Dümmy] is not properly signed. Make sure you have signed your application with a distribution certificate, not an ad hoc certificate or a development certificate. Verify that the code signing settings in Xcode are correct at the target level (which override any values at the project level).

This happens because the default Xcode configuration uses the same variable to set both the names of the .app package and the .ipa archive - both of which may not have special characters. And in this case it is set to "Dümmy".

The Info.plist file will have the following keys:

PRODUCT_NAME: the name of the .app bundle and executable file.
CFBundleName: the app name when CFBundleDisplayName isn't set.

The tricky part is CFBundleName which is set to the value of $(PRODUCT_NAME) by default. What Apple docs do not mention is that starting with Xcode 12 and beyond it also represents the .ipa file name.

The solution is to set both PRODUCT_NAME and CFBundleName to values with no special characters and to explicitly set the display name under Project Navigator → General → Display Name. This will generate a new key CFBundleDisplayName whose value will be preferred over CFBundleName for the app name.

Now you can keep the app name with special characters, but remove them from the file names for the app package and archive.

This article was cross-posted from my personal blog. Do subscribe to me there!

Bootstrapping a React Native App with Best Practices

Osama Qarem — Sun, 23 Aug 2020 05:06:29 +0000

Edit: While this was probably a decent template at the time – I no longer recommend using it today. For the latest and greatest check out create-t3-turbo or Solito.

If you're unaware, you can create your own react native template to be consumed by the react native cli.

Since I always end up referencing old projects for some code (e.g. navigators, automation scripts) I decided I to make one and save myself the headache.

Anyway, I've been maintaining it for some time now and I'm quite happy with where it's at. I only got around to writing some docs this weekend though, which is why I feel this is a good time to share. I should also mention that it is using TypeScript.

Check it out here!

osamaq/react-native-template

Thanks to the work by the authors of react-native-cli, using the template is as easy as:

$ npx react-native init MyApp --template @osamaq/react-native-template

If you are just starting to learn React Native or TypeScript, I don't actually recommend this template as it could be a bit overwhelming, unless you have a native iOS or Android background.

I've put a lot of work into it, from applying best practices to writing boilerplate components/logic that you can build upon. More than anything, I love the automation scripts. Please go through the docs or codebase and let me know what you think!

If you have questions or suggestions on how it could be improved or what you'd do differently, I'd really appreciate it. Thanks!

GPT-3: Programming in English

Osama Qarem — Sat, 25 Jul 2020 04:48:59 +0000

// Detect dark theme var iframe = document.getElementById('tweet-1284421499915403264-263'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1284421499915403264&theme=dark" }

I know GPT-3 and similarly fascinating tech have a long way to go, but I want you to envision a future where we use GPT-3 for real. This is my opinion after entertaining that thought:

COBOL was meant to be friendly to non-programmers using English words and such, but in the end, it was still programming.

At my workplace, people who are proficient at using Joget (low/no code platform) are called Joget Developers. Just like there are Joget Developers, there will be people trained at giving instructions to the AI and using its platform; GPT-3 Developers.

Using GPT-3 will still be programming. It will be coding in a language that uses English as its syntax. Let's consider that for a moment: Human spoken languages are less concise, less clear and more prone to misunderstanding. People will need to practice the right nouns, adjectives and verbs etc. to use when talking to GPT-3 to get the outcome they want. You would need to learn to "debug" your English sentences according to how GPT-3 understands it, and not according to what you really meant. Words and even sentences carry different meanings based on context.

Some might argue "they could create a simplified version of the English language that makes it easier to give instructions to GPT-3" - you just invented a programming language.

With programming languages, you say what you mean and mean what you say. Learning a logical language is easier than learning English.

React Native: Generating TypeScript Types for Environment Variables

Osama Qarem — Sun, 19 Apr 2020 10:02:01 +0000

As a React Native developer, I use react-native-config to manage different environments. I create .env, .env.staging, and .env.prod for development, staging and production at the root of my project.

Assuming my .env file looks like:

BASE_URL=https://localhost:8000

Then I'm able to do:

import BuildConfig from "react-native-config"

console.log(BuildConfig.BASE_URL)
// https://localhost:8000

Seems good. Works fine. But not for me. There is no autocomplete. It's not typesafe. It's prone to human error that is only noticable it at runtime.

Whenever I go back to native development with Android Studio I'd get jealous of that typesafe autocomplete. How can we get something like that for React Native?

Let's get some understanding of how it works for Android first. Gradle is the build tool used for android's build system. Whenever the android app is built, a class is generated describing environment variables allowing for typesafe environment variable access.

Here is an illustration:

To bring that experience to React Native, we need to make a type declaration file that describes our environment variables module. That will let typescript know how to autocomplete. With a single environment variable, it will look like this:

// .env
declare module "react-native-config" {
  interface Env {
    BASE_URL: "https://localhost:8000"
  }

  const BuildConfig: Env

  export default BuildConfig
}

Now once we import react-native-config module, we should get autocomplete.

But that's not as good. We don't want to have to update our type declaration file manually!

For that, I resorted to writing quite a lengthy Node.js script. In cough-cough plain javascript:

const fs = require("fs")

const contents = () => {
  const env = fs.readFileSync(".env", { encoding: "ASCII" })
  const envStaging = fs.readFileSync(".env.staging", { encoding: "ASCII" })
  const envProd = fs.readFileSync(".env.prod", { encoding: "ASCII" })

  const envLines = env.split("\n")
  const envStagingLines = envStaging.split("\n")
  const envProdLines = envProd.split("\n")

  let filteredEnv = []
  let filteredEnvStaging = []
  let filteredEnvProd = []

  // Assumption: all files have the same number of lines
  for (let index = 0; index < envLines.length; index++) {
    const envLine = envLines[index]
    const envStagingLine = envStagingLines[index]
    const envProdLine = envProdLines[index]

    if (envLine.includes("=")) {
      if (envLine.includes("#")) {
        filteredEnv.push(envLine.split("#")[1].trim())
      } else {
        filteredEnv.push(envLine.trim())
      }
    }

    if (envStagingLine.includes("=")) {
      if (envStagingLine.includes("#")) {
        filteredEnvStaging.push(envStagingLine.split("#")[1].trim())
      } else {
        filteredEnvStaging.push(envStagingLine.trim())
      }
    }

    if (envProdLine.includes("=")) {
      if (envProdLine.includes("#")) {
        filteredEnvProd.push(envProdLine.split("#")[1].trim())
      } else {
        filteredEnvProd.push(envProdLine.trim())
      }
    }
  }

  return [filteredEnv, filteredEnvProd, filteredEnvStaging]
}

const generate = () => {
  const [filteredEnv, filteredEnvProd, filteredEnvStaging] = contents()
  let envVariableNamesArray = []
  let envVariableValuesArray = []

  for (let i = 0; i < filteredEnv.length; i++) {
    // Assumption: the files we read are not just comments
    const envPair = filteredEnv[i].split("=")
    const envStagingValue = filteredEnvStaging[i].split("=")[1]
    const envProdValue = filteredEnvProd[i].split("=")[1]

    envVariableNamesArray.push(envPair[0])

    envVariableValuesArray.push(envPair[1], envStagingValue, envProdValue)
  }

  // Assumption: for every name/key there are 3 values (env, env.staging, env.prod)
  let table = []
  let valuesCursor = 0

  for (let i = 0; i < envVariableNamesArray.length; i++) {
    table[i] = [envVariableNamesArray[i], []]

    const totalPushCount = 3
    let current = 0
    while (current !== totalPushCount) {
      const valueToPush = envVariableValuesArray[valuesCursor]

      if (!table[i][1].includes(valueToPush)) {
        table[i][1].push(valueToPush)
      }
      valuesCursor++
      current++
    }
  }

  const stringArrayMap = table.map((nameValueArray) => {
    const name = nameValueArray[0]
    const valuesArray = nameValueArray[1]

    let string = `${name}: `

    valuesArray.forEach((value, index) => {
      if (index === 0) {
        string = string.concat(`"${value}"`)
      } else {
        string = string.concat(` | "${value}"`)
      }
    })

    return string
  })

  const string = `declare module "react-native-config" {
  interface Env {
    ${stringArrayMap.join("\n    ")}
  }

  const Config: Env

  export default Config
}`

  fs.writeFileSync("env.d.ts", string, "utf8")
}

generate()

In summary, this script will read all 3 environment files and generate a .env.d.ts describing the types. It will only work if all 3 .env files contain the same number of variables with the same names, which makes sense.

At the root directory of my react native project, I created a scripts folder and placed it there. It looks like this MyApp/scripts/generateEnvTypes.js. Next I added the following npm script to my package.json:

"generate-env-types": "node scripts/generateEnvTypes.js"

Now, whenever I update my environment variables, I simply run the npm script and a new type declarations file is automatically generated! 🎉

PS: I'm maintaining a React Native template with a lot of goodies like the one in the article.

Automatic Versioning for React Native Apps

Osama Qarem — Sat, 18 Apr 2020 12:27:42 +0000

Problem

You need to update your app's version to 1.0.0:

1. You open up android/app/build.gradle to update the version and bump the build number.

2. You do the same thing for iOS using Xcode because editing build configuration files directly is more error prone.

3. You need to keep it all consistent, so you open up package.json and update the version so the reference to the version shown to the user from the JS side is correct.



import { version } from "./package.json"

console.log(version)
// 1.0.0

I feel so productive and happy!

Said no developer ever after going through that.

Solution

The ideal experience is to update only a single version number. Here's what we're going to do:

1. Use npm version [patch|minor|major] to handle the JS package version (see semantic versioning).

The JS version is our source of truth. Therefore, the Android and iOS versions should match whatever the package.json version is set to.

2. Use fastlane to handle the Android and iOS sides.

fastlane is an amazing open source tool focused at automating Android and iOS tasks. It has a wide library of community developed plugins that can help us handle things like, versioning.

3. Combine the above 2 steps into a single npm script.

Steps

We will use a fresh React Native project as a starting point:



npx react-native init MyApp

Install fastlane if you do not already have it:



# Install the latest Xcode command line tools
xcode-select --install

# Install fastlane using RubyGems
sudo gem install fastlane -NV

# Alternatively using Homebrew
brew install fastlane

Set up a fastlane directory and create an empty fastfile:



cd MyApp
mkdir fastlane && cd fastlane 
touch Fastfile

We want to be able to run the fastlane command from the root of our React Native project. Therefore we will install our versioning plugins from the root directory:



cd ..

# Install plugins
fastlane add_plugin increment_version_name increment_version_code load_json

Say 'yes' if it asks about creating a gemfile.

The first two plugins are for handling the version, version code on android and the third one is for reading a JSON file (our package.json).

Next, we are going to add our fastlane scripts. Copy the following to the fastfile at fastlane/Fastfile.



  desc 'Android: Increment versionCode and set versionName to package.json version.'
  package = load_json(json_path: "./package.json")

  private_lane :inc_ver_and do
    increment_version_code(
      gradle_file_path: "./android/app/build.gradle",
    )

    increment_version_name(
      gradle_file_path: "./android/app/build.gradle",
      version_name: package['version']
    )
  end


  desc 'iOS: Increment build number and set the version to package.json version.'
  private_lane :inc_ver_ios do
  package = load_json(json_path: "./package.json")

    increment_build_number(
      xcodeproj: './ios/' + package['name'] + '.xcodeproj'
    )
    increment_version_number(
      xcodeproj: './ios/' + package['name'] + '.xcodeproj',
      version_number: package['version']
    )
  end


  desc 'Bump build numbers, and set the version to match the pacakage.json version.'
  lane :bump do
    inc_ver_ios
    inc_ver_and
  end

Next we are going to add the following scripts to our package.json for automatic patch, minor and major version bumps:



{ 
  "name": "MyApp",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "android": "react-native run-android",
    "ios": "react-native run-ios",
    "start": "react-native start",
    "test": "jest",
    "lint": "eslint .",
    "bump-patch": "npm version patch --no-git-tag-version && bundle exec fastlane bump",
    "bump-minor": "npm version minor --no-git-tag-version && bundle exec fastlane bump",
    "bump-major": "npm version major --no-git-tag-version && bundle exec fastlane bump",
},

The first part of the command will upate the JS package version without making a commit to the git repo. The second part will execute the fastlane bump command, which will automatically bump the android and iOS build numbers and update the version to match the JS side.



# npm
npm run bump-patch  

# yarn
yarn bump-patch

PS: I'm maintaining a React Native template with a lot of goodies like the one in the article.

Advanced Async Logic with Redux Observable

Osama Qarem — Sun, 13 Oct 2019 02:22:36 +0000

Imagine that you're building a social network for cats. To register, a cat has to be verified that they are who they say they are by uploading a selfie and a photo of its paw. Our backend runs both images through its FelineCatusML™ billion dollar machine learning model and helps us verify that the pawprint and face belong to the same furball by going through legal cat records. If we believe kitty is playing tricks, we will terminate the sign up process.

For example, if a cat says that its a brown Munchkin called Peanut, but the backend returned name: 'Peanut' for the selfie and name: 'Sir Williams' for the pawprint, registration will fail as names must match.

This is the flow of our imaginary app. Each step is a different page/screen:

Cat takes paw photo (API call #1).
Cat starts filling up form.
Cat takes selfie (API call #2).
Check the outcome of API calls #1 & #2 and determine if sign up is successful.

Here are some optimistic assumptions:

Because the first API call happens first, it completes first.

But in the real world, either one may complete first.

The user will wait patiently staring at loading indicators while any API calls are running :)

The user will want to be able to cancel whatever is happening on their screen whenever they want to. Especially cat users. Cats have very short attention spans and cannot stare at loading indicators for long.

Our apps need to be able to handle process cancellation. That means stopping all operations that are no longer needed. Not only are those considered memory leaks, but they can lead to very unexpected behavior. Imagine if a cat goes through step 1, then cancels (API call #1 still running) then comes back and does step 1 again. You can rarely ever predict what a cat is up to. It would not be out of the question to assume that we now have 2 API calls racing to update the same piece of state. And we cannot guarantee which one will finish first. So what is our next state? Unknown.

Of course, we could make the process not cancellable "Either you finish this right now, or you exit my app/close the browser". But that is not good user experience. We must aim for flexible applications. They are more pleasant to use.

Our API calls are probably happening in our redux actions. Besides process cancellation, how can we be notified when our app is in the following state:

Both API calls #1 & #2 have successfully returned results that are non-errors (remember, both calls are happening concurrently).

How can we handle cancellation and order of completion elegantly without writing too much code that is hard to follow?

We have options for Redux. There are several middleware for handling complex asynchronous logic well documented by the docs (best docs ever, by the way) [1] [2]. You can actually use whichever one you are comfortable with to deal with similar and more complex challenges. I was heavily considering Redux Saga as it is the most popular. But I am not as familiar with JavaScript generators as I am with RxJs, and so I went with Redux Observable.

Redux-Observable 🔁

As you can probably tell from the name, this library makes use of observables, particularly from RxJs. In fact, the API of redux-observable is about 3-4 functions. Most of the code we will write will actually be from RxJs.

🛑 If you are not familiar with Rx, then you will likely face difficulty following the code in this article.

Brief intro using promises

Official

With redux-observable we create epics. An epic is a function that takes in two arguments. The first one is a stream of actions running through your reducers.

  // Redux action payload shape.
  type Action = {type: string; payload?: any}

  // Actions stream. An observable that wraps a payload.
  actions$ : ActionsObservable<Action>

When you dispatch an action, it runs through your reducers, then your epics. Usually, an epic is set to execute when a specific action type is dispatched.
The second argument is the stream of our store's state. This means that an epic may access the store's current state. We won't need that here.

To keep this article short, I'll leave installing and setting up the library to you, as you can follow the instructions on the official docs.

Epics 🔥

Our API calls will happen in our epics. We want to execute the first API call as soon as we have the necessary data. Therefore, we will create separate epics for each API call.

1. getNameForPawEpic() will make the first API call.
2. getNameForFaceEpic() will make the second API call.

We will also have a third epic:

3. verificationEpic() will run when both epics above have dispatched success actions.

// Action creator
export const getNameForPaw = pawPhotoBase64 => ({
  type: GET_NAME_FOR_PAW,
  payload: pawPhotoBase64
});

This action creator is called from our component. The dispatched action runs through our reducers, but there is no match for its type therefore it does not change our store's state. Its purpose is to run our first epic which will be listening to its action type.

// Epic
export const getNameForPawEpic = actions$ =>
  actions$.pipe(
    ofType(GET_NAME_FOR_PAW),
    switchMap(({ payload }) => {
      const request = {
        url: "http://api.felinecatus.com/pawprint/verification",
        method: "POST",
        headers: {
          "content-type": "application/json"
        },
        body: {
          pawPhotoBase64: payload
        }
      };

      return ajax(request).pipe(
        map(res => {
          if (res.statusCode === 200) {
            return {
              type: GET_NAME_FOR_PAW_SUCCESS,
              payload: res.data
            };
          } else {
            return {
              type: GET_NAME_FOR_PAW_ERROR,
              payload: res.errorMessage
            };
          }
        }),
        takeUntil(actions$.pipe(ofType(CANCEL))),
        catchError(err => {
          return of({
            type: GET_NAME_FOR_PAW_ERROR,
            payload: res.errorMessage
          });
        })
      );
    })
  );

So what is our epic doing? It is:

1. Listening to a stream of actions using a redux-observable helper function ofType() which filters by the type we specify.

2. When GET_NAME_FOR_PAW action is dispatched, the epic will let that action through.

3. We switchMap() our action. This is simply the safe choice when mapping async actions. Since we don't want this action somehow executing more than once at a time, it will 'switch' to the most recent call, and map the result to what follows.

We destructure the payload from our action and create the API call using the ajax() operator. We map the result to either success or error types. What is to be done with the error is to be handled by our component.

Epics take in a stream of actions, and let out plain action objects. That is why we are returning action payloads.

   //  In
   action$: ActionsObservable<Action>

   // Out. Passes through our reducers.
   action:  Action

4. We have a an action type called CANCEL which when dispatched, will cancel all of our network requests. We might dispatch this when the user navigates away to cancel the process. This works because takeUntil() is listening to the stream of actions and completes our ajax observable if the cancellation action type comes through.

5. catchError() does what its called. In case something unexpected happens, we can handle it here. It must return an observable though, that is why we use of() on the action returned from within it.

This is our action creator for our second epic:

// Action creator for second epic
export const getNameForFace = facePhotoBase64 => ({
  type: GET_NAME_FOR_FACE,
  payload: facePhotoBase64
});

The code for getNameForFaceEpic() is very similar to the first epic, except it listens to GET_NAME_FOR_FACE and dispatches GET_NAME_FOR_FACE_SUCCESS on success and GET_NAME_FOR_FACE_ERROR on error. So we will stick to pretending that we have written it 🙂.

Our first 2 epics combined act as the action creator of our third epic:

// Third epic
export const verificationEpic = actions$ => {
  const paw$ = actions$.pipe(ofType(GET_NAME_FOR_PAW_SUCCESS));
  const face$ = actions$.pipe(ofType(GET_NAME_FOR_FACE_SUCCESS));
  const combined$ = zip(face$, paw$);

  const cancel$ = actions$.pipe(ofType(CANCEL));

  return combined$.pipe(
    map(([face, paw]) => {
      const verifiedKitty = face.payload === paw.payload;

      return { type: VERIFICATION_COMPLETE, payload: verifiedKitty };
    }),
    takeUntil(cancel$),
    repeat()
  );
};

1. We create paw$ & face$ which are streams of actions filtered by the success types of the first 2 epics.

2. We combine those two using the zip operator which creates a new stream that emits a value only when both streams emit once. This way, we can get a success notification only when both succeed.

3. When both API calls succeed, we process our payloads and map the output to VERIFICATION_COMPLETE which runs through our reducers and updates our store's state. Our component handles the outcome.

In case one epic sends off the success action, our combined$ stream will now wait for the other pair to emit its success action. In case the user cancelled the process or an error in the other request ocurred, no success actions are going to happen. Therefore, we need to be able to reset our combined$ stream to listen to both success actions again.

4. We do takeUntil(cancel$). This will complete our observable returned by our epic when the CANCEL type comes through. Since we completed the outermost observable in our epic, it is now dead and no longer working.

5. To remedy this, we pipe the repeat() operator, which restarts our combined$ observable as soon as it completes making it listen to both face$ and paw$ from square one again.

And that's how it's done! 🎉

Cancellable, flexible and concise async process management ✅ with redux. And we've only used a few RxJs operators. Remember, You can use more than one redux middleware at the same time. I am still using redux-thunk alongside redux-observable. Simple problems should still be solved with simple solutions.

Thank you for reading. I hope you found this useful.

Thinking in React for Native Android Apps

Osama Qarem — Fri, 20 Sep 2019 13:50:50 +0000

This article was originally published on my personal blog.

I like writing apps using React Native. In contrast, working on Java Android apps has always been less satisfying because:

Boilerplate.
ConstraintLayout? LinearLayout? RelativeLayout? TableLayout? GridLayout?
???

It's a simple fact that the more code you write, the more likely you are to write bugs. I knew using Kotlin would improve this aspect, but I wanted to get more experienced with Java, so that I'd better appreciate Kotlin when I made the move. I also didn't believe that just by using Kotlin, I'd suddenly really enjoy developing native apps.

Recently, I came across MvRx (Mavericks). An Android framework open-sourced by Airbnb. I learned that it is conceptually inspired by React which piqued my interest. It even brings over the familiar setState() syntax. Since MvRx is Kotlin-only, it tipped me over to start learning Kotlin.

The same todo app written using MvRx is about 1000 lines of code less than the same app written in Java.

Syntax-wise Kotlin has many similarties to TypeScript which I've always preferred using. I learned by going through the Kotlin official docs (which are awesome) and doing some of the the Kotlin Koans.

They say that no Java developer has tried Kotlin and wanted to go back to writing Java. I concur.

I previously used MVP architecture, whereas MvRx is MVVM. Presenters and ViewModels harbour the logic of your screens (similar to container components in React). The main difference between them is that a ViewModel never holds a reference to any view. It simply updates its state and the view observes the data changes and much like React, re-renders accordingly. Therefore, there is no fear of referencing a null view (similar to calling setState() on an unmounted React component). This greatly simplifies dealing with view lifecycles.

During my process of learning MvRx and Kotlin, I came across a few helper libraries that improve development experience. I decided to learn and use them.

I slowly came to the realization that sometimes, we might not enjoy working with a framework not because of the framework itself, but simply because of the way we apply it.

Guide

We're going to make a simple app utilizing the following libraries:

Navigation Component
Kotlin Android Extensions
MvRx
Flexbox Layout

The app flow will be as such:

Once the user presses the login button, we will mock a request using a simple delay. During the mock request, we will hide the view and show a loading indicator. Once the request is done, we will restore our view, hide the loading indicator and navigate to the landing screen.

Landing Screen:
- This screen will simply show the data entered in the previous text inputs and a logout button.

Simple enough.

Dependencies

Let's start by creating a blank Android Studio project with Kotlin and add our dependencies.

Add MvRx and the navigation component to your dependencies block, under app/build.gradle:

dependencies {
    def navVersion = "2.1.0"
    def mvrxVersion = '1.1.0'

    // Navigation component
    implementation "androidx.navigation:navigation-fragment-ktx:$navVersion"
    implementation "androidx.navigation:navigation-ui-ktx:$navVersion"

    // MvRx
    implementation "com.airbnb.android:mvrx:$mvrxVersion"

    // Flexbox
    implementation 'com.google.android:flexbox:1.1.0'
    ..

At the top of the same file:

// Kotlin Android Extensions
apply plugin: 'kotlin-android-extensions'

Layouts

We will use single activity - multiple fragments pattern. Fragments should be designed as resuable and modular components, just like presentational components in React.

Create our layout files: Right-click the res folder then select New > Android Resource File. Set the type as Layout.

activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <fragment
        android:id="@+id/host"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

</LinearLayout>

login_fragment.xml

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:background="#F1F1F1">

    <com.google.android.flexbox.FlexboxLayout

        android:layout_width="match_parent"
        android:layout_height="250dp"
        android:layout_marginTop="200dp"
        app:alignItems="center"
        app:flexWrap="wrap"
        app:justifyContent="center">

        <EditText
            android:id="@+id/loginNameText"
            android:layout_width="120dp"
            android:layout_height="60dp"
            android:hint="Name"
            android:importantForAutofill="no"
            android:inputType="text"
            app:layout_flexBasisPercent="80%"
            tools:text="Name" />

        <EditText
            android:id="@+id/loginCityText"
            android:layout_width="120dp"
            android:layout_height="60dp"
            android:hint="City"
            android:importantForAutofill="no"
            android:inputType="text"
            app:layout_flexBasisPercent="80%"
            tools:text="City" />

        <Button
            android:id="@+id/loginButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_marginTop="30dp"
            android:backgroundTint="#6200EE"
            android:text="LOGIN"
            android:textColor="#FFF"
            app:layout_flexBasisPercent="80%" />

        <ProgressBar
            android:id="@+id/loadingIndicator"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:visibility="gone"
            app:layout_flexBasisPercent="100%"
            />

    </com.google.android.flexbox.FlexboxLayout>


</FrameLayout>

The root is a <FrameLayout/>. The <fragment/> tag in activity_main.xml will be swapped for the contents (children) of <FrameLayout/> in each of our fragments. (a bit like React's children?)

The child of the root layout is <FlexboxLayout/>. Pretty cool. ConstraintLayout is nice if you prefer to customize the layout visually.

Feel free to use whatever you fancy.

landing_fragment.xml

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:background="#F1F1F1">

    <com.google.android.flexbox.FlexboxLayout
        android:layout_width="match_parent"
        android:layout_height="250dp"
        android:layout_marginTop="200dp"
        app:alignItems="center"
        app:flexWrap="wrap"
        app:justifyContent="center">

        <TextView
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:gravity="center"
            android:textColor="@android:color/black"
            android:textSize="24sp"
            app:layout_flexBasisPercent="50%"
            android:text="Name:" />

        <TextView
            android:id="@+id/landingNameText"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:gravity="center"
            android:textColor="@android:color/black"
            android:textSize="24sp"
            app:layout_flexBasisPercent="50%"
            tools:text="placeholder" />

        <TextView
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:gravity="center"
            android:textColor="@android:color/black"
            android:textSize="24sp"
            app:layout_flexBasisPercent="50%"
            android:text="City:" />

        <TextView
            android:id="@+id/landingCityText"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:gravity="center"
            android:textColor="@android:color/black"
            android:textSize="24sp"
            app:layout_flexBasisPercent="50%"
            tools:text="placeholder" />


        <Button
            android:id="@+id/logoutButton"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_marginTop="30dp"
            android:backgroundTint="#F05E54"
            android:text="LOGOUT"
            android:textColor="#FFF"
            app:layout_flexBasisPercent="80%" />

    </com.google.android.flexbox.FlexboxLayout>


</FrameLayout>

Classes

Create our Kotlin classes to associate with each layout.

To create an activity, we would usually extend the AppCompatActivity class directly. But since we want to use MvRx, we will extend BaseMvRxActivity instead (which inherits from AppCompatActivity) for MvRx support. We will also override onCreate() and inflate activity_main.xml here.

Tip: here is a super useful page for looking up Android lingo like 'inflate'.

MainActivity.kt

package com.example.mymvrxapp

import android.os.Bundle
import com.airbnb.mvrx.BaseMvRxActivity


class MainActivity() : BaseMvRxActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
    }
}

Every activity must be registered in the manifest. We will register MainActivity and set it as the starting activity.

AndroidManifest.xml

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.mymvrxapp">

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:roundIcon="@mipmap/ic_launcher_round"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <activity android:name="com.example.mymvrxapp.MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>
</manifest>

For our fragments, we will extend BaseMvRxFragment instead of Fragment. We must also implement invalidate(). We will leave it empty for and go over it later on.

LoginFragment.kt

package com.example.mymvrxapp

import android.os.Bundle
import android.view.LayoutInflater
import android.view.View
import android.view.ViewGroup
import com.airbnb.mvrx.BaseMvRxFragment

class LoginFragment : BaseMvRxFragment() {
    override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View? {
        return inflater.inflate(R.layout.login_fragment, container, false)
    }

    override fun invalidate() {
    }
}

LandingFragment.kt

package com.example.mymvrxapp

import android.os.Bundle
import android.view.LayoutInflater
import android.view.View
import android.view.ViewGroup
import com.airbnb.mvrx.BaseMvRxFragment

class LandingFragment : BaseMvRxFragment() {
    override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View? {
        return inflater.inflate(R.layout.landing_fragment, container, false);
    }

    override fun invalidate() {
    }
}

Navigation

If we run the app at this point, its going to crash. Our <fragment/> in activity_main.xml needs an ID and a name to associate it with a fragment. We've given it an ID but we haven't told it what fragment its going to host yet.

We're not going to do this. Instead, we are going to associate it with our navigation graph. using the Navigation Component.

Simply put, it's a library that simplifies how we handle navigation with a neat API and a friendly interface to visualize our routes.

The navigation component fulfills the same job as React Navigation and React Router.

Fun React Navigation Fact: We can configure it to use FragmentActivity
instead of plain View's under the hood for performance improvements.

Create our navigation graph. Right-click the res folder and then select New > Android Resource File. Set the type as Navigation.

nav_graph.xml

<?xml version="1.0" encoding="utf-8"?>
<navigation xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/nav_graph" />

Now that we've created the file for our navigation graph, we will add an ID to <fragment/> and designate it as our navigation host by adding the following attributes:

activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <fragment
        android:id="@+id/host"
        android:name="androidx.navigation.fragment.NavHostFragment"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:defaultNavHost="true"
        app:navGraph="@navigation/nav_graph" />

</LinearLayout>

Lets add our fragment classes to the navigation graph to mark them as possible destinations. I like to use the visual editor for this part.

Open nav_graph.xml in the visual editor, and add LoginFragment and LandingFragment.

Select the login fragment in the graph then click the home icon to mark it as the starting destination.

Next, drag from the edge of the login fragment to the landing fragment to create a navigation action.

Now your navigation graph and markup should look similar to this.

nav_graph.xml

<?xml version="1.0" encoding="utf-8"?>
<navigation xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:id="@+id/nav_graph"
    app:startDestination="@id/loginFragment">
    <fragment
        android:id="@+id/landingFragment"
        android:name="com.example.mymvrxapp.LandingFragment"
        android:label="LandingFragment"
        tools:layout="@layout/landing_fragment" />
    <fragment
        android:id="@+id/loginFragment"
        android:name="com.example.mymvrxapp.LoginFragment"
        android:label="LoginFragment"
        tools:layout="@layout/login_fragment">
        <action
            android:id="@+id/action_loginFragment_to_landingFragment2"
            app:destination="@id/landingFragment" />
    </fragment>
</navigation>

I've added tools:layout=... attribute for displaying your layouts in the navigation graph, otherwise you'd only see a plain and boring rectangle.

If we run the app now, we should see the Login screen.

Logic

Lets start by adding state to our text inputs. We need to do the following:

Create our data class which describes the shape of our state.
Create our view model class which will hold functions that trigger our state changes.

I created both the data class and the view model in the same file as MainActivity.kt for convenience but that is not a requirement.

FormState

data class FormState(
    val name: String = "",
    val city: String = "",
    val loggedIn: Async<Boolean> = Uninitialized
) : MvRxState

We must set the initial state by providing default arguments. Notice it implements MvRxState. That's required for any data class we wish to use as state.

In React, we might have a loading state and set it before and after the completion of asynchronous tasks. In MvRx, Async is a sealed class that comes with types like Loading and Success. We can simply refer to the current type of the async value to react to loading and success states. Super helpful.

FormViewModel

class FormViewModel(initialState: FormState) :
    BaseMvRxViewModel<FormState>(initialState, debugMode = BuildConfig.DEBUG) {

    init {
        logStateChanges()
    }

    fun setNameAndCity(name: String, city: String) {
        setState { copy(city = city, name = name) }
    }
    // We will go over this one in depth later on
    fun doLogIn() {
        Single.just(true)
            .delaySubscription(5, TimeUnit.SECONDS)
            .subscribeOn(Schedulers.io())
            .observeOn(AndroidSchedulers.mainThread())
            .execute { copy(loggedIn = it) }
    }

    fun doLogout() {
        setState {
            copy(loggedIn = Uninitialized)
        }
    }
}

debugMode = BuildConfig.DEBUG will make some safety checks when working with the debug build. The init block and logStateChanges() are also optional. logStateChanges() does exactly what it says. We will show its output when we finish our app.

Our setState reducers will be called from our views to update the state. Similar to React, the setState block is an async operation and a pure function that takes in the current state, and returns the new state.

Notice the copy() syntax within the setState block. Inside setState, this keyword would be our data class and copy() is a method that belongs to data classes in Kotlin. It allows you to modify select properties instead of all (we don't need to spread the current state, in React lingo).

Next, we want to be able to access state from our fragments. Our login and landing fragments must subscribe to the same view model instance we defined in our main activity.

LoginFragment.kt

class LoginFragment : BaseMvRxFragment() {
    // Fetch the ViewModel scoped to the current activity or create one if it doesn't exist
    private val formViewModel: FormViewModel by activityViewModel()

    ..

LandingFragment.kt

class LandingFragment : BaseMvRxFragment() {    
    // Fetch the existing ViewModel scoped to the current activity
    private val formViewModel: FormViewModel by existingViewModel()

    ..

Notice by activityViewModel(). It is a Kotlin delegate that lazily returns a view model scoped to the current activity. Since both of our fragments belong to the same activity, sharing state is very straightforward.

For LandingFragment.kt, we used existingViewModel() which returns the existing view model in scope of the current activity. The difference is that this function will throw an exception if no view model exists, instead of creating a new one.

MvRx lets you easily choose the scope of the view model you'd like to work with.

As soon as our view loads (React: mounts), we're going to:

Add a click listener to our login button.
When the user presses the button we will grab the user's input and update our name and city states, then call doLogIn to start the mock request/delay.
When the delay starts, we must hide our view and show the loading indicator.
When the delay ends, we must hide the loading indicator and show our view. Then, we navigate to the landing screen.

Override onViewCreated and implement the on-click listener as described:

LoginFragment.kt

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)

        loginButton.setOnClickListener {

            // Update the state
            formViewModel.setNameAndCity(
                loginNameText.text.toString(),
                loginCityText.text.toString()
            )

            formViewModel.doLogIn()
        }
    }

Because of Kotlin Android Extensions, we are able to directly reference the view without calling findViewById. This is called View Binding (similar to getting a ref to a node in React) .

Note: in the future, it will be recommended to use the View Binding component which has some advantages and none of the drawbacks. It will be available in newer versions of Android Studio.

    fun doLogIn() {
        Single.just(true)
            .delaySubscription(5, TimeUnit.SECONDS)
            .subscribeOn(Schedulers.io())
            .observeOn(AndroidSchedulers.mainThread())
            .execute { copy(loggedIn = it) }
    }

MvRx makes use of RxJava. Primer: RxJava is very useful in Android development as it offers a neat API for handling asynchronous logic (Like JavaScript promises, but with more capabilities).

It is important to execute long running tasks off the main thread (also called UI thread) to prevent the app from dropping frames. RxJava allows you to declaratively switch threads with ease.

To use any of RxJava's APIs, we must work with the observable type. RxJava has APIs to create observables from things.

doLogin() is called when the login button is pressed. Let's go through it in detail as Rx can be intimidating if you've never used it before:

Single is a type of observable that resolves to a single value, exactly like a JavaScript promise.
just() is used to denote that this observable resolves to just this item, in this case the item is true. So the first line is equivalent to Promise.resolve(true) in JavaScript.

..
.delaySubscription(5, TimeUnit.SECONDS)

We need to subscribe to an observable to receive a result from it. This line states that any subscription should be delayed by 5 seconds.

..
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())

Here, we state that we wish to subscribe to this observable using an io thread by the help of the io scheduler and ask for value notifications to be sent to the main thread.

..
.execute { copy(loggedIn = it) }

execute is a neat helper function by MvRx that maps the current state of this process to our loggedIn async type. When the observable is executed, loggedIn type is updated to Loading. When it is done, loggedIn type and value are updated to Success and true.

Now, the invalidate() function comes in handy. This function is called every time our state is updated (just like a React re-render). Here, we can make changes to our view according to the current state.

LoginFragment.kt

    ..

    override fun invalidate() {
        withState(formViewModel) { state ->
            loadingIndicator.isVisible = state.loggedIn is Loading
            loginNameText.isVisible = state.loggedIn !is Loading
            loginCityText.isVisible = state.loggedIn !is Loading
            loginButton.isVisible = state.loggedIn !is Loading

            if (state.loggedIn is Success) {
                findNavController().navigate(R.id.action_loginFragment_to_landingFragment2)
            }
        }
    }

withState allows us to access the current state of our view model. Inside, we map the loading state of loggedIn to the visibility of our loading indicator, inputs and button. If loggedIn is of type Success, then we navigate to the landing screen.

Navigation KTX helps us obtain a reference to the navigation controller here.

For the landing fragment, we need to implement invalidate() and update our texts using the current state. We will add a listener to our logout button that sets loggedIn to Uninitialized and then pops our fragment off of the navigation stack, going back to the login screen.

LandingFragment.kt

    ..

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)

        logoutButton.setOnClickListener {
            formViewModel.doLogout()
            findNavController().popBackStack()

        }
    }

    override fun invalidate() {
        withState(formViewModel) { state ->
            landingCityText.text = state.city
            landingNameText.text = state.name
        }
    }

Courtesy of logStateChanges():

D/FormViewModel: New State: FormState(name=Osama, city=Cyberjaya, loggedIn=com.airbnb.mvrx.Uninitialized@24591c4)
D/FormViewModel: New State: FormState(name=Osama, city=Cyberjaya, loggedIn=com.airbnb.mvrx.Loading@7749791c)
D/FormViewModel: New State: FormState(name=Osama, city=Cyberjaya, loggedIn=Success(value=true))

All done! Hope you've enjoyed this guide and found it useful.

If you'd like to learn more about MvRx, I suggest going through their wiki and code samples on their repo.

A Brief Intro to RxJs

Osama Qarem — Sun, 25 Aug 2019 11:41:51 +0000

This article was originally posted on my blog.

RxJs is the implementation of Reactive Extensions for Javascript. These extensions are a collection of tools allowing us to write declarative reactive code as opposed to imperative (e.g. do this when x instead of if x is this right now, then do the following...). Basically, it gives us operators (functions) that can enhance handling of event-based logic, especially when working with multiple correlated events.

Rx definitely has a learning curve. New concepts and and terms can be overwhelming at the start. But once you try enough times, eventually it clicks and you get much faster at learning the all the wizzabanga words and concepts. I'll try to keep this post as simple as possible, but you definitely need to be comfortable with the promises API.

Promises vs Observables

Imagine we have an array of strings describing our thoughts.

const thoughts = ["food", "sleep", "code"];

Promise

When we work with events in JavaScript we will typically uses the Promise API. For a promise, two things can happen:

1- It resolves with a single value.

2- It rejects with a single value, usually an error message.

Both 1 and 2 signal the completion of a promise.

const thoughtsPromise = () => {
  return new Promise((resolve, reject) => {
    resolve(thoughts);
  });
};

Observable

An observable is a stream of data from which we can get notifications with values. We can get those notifications in three different scenarios:

1- When there is a new value from the stream.

2- When an error occurs giving us the error value.

3- When the stream is complete.

One difference is that observables can resolve many times with new values. For example, imagine you want to show a video on your app to the user. Would it be better to let the user download the entire video all at once, or to stream it little by little? Observables can help you stream your data.

Let's create an observable.

The from operator can transform data such as arrays or promises into observables which are streams of data.

import { from } from "rxjs";

// The trailing $ sign is a naming convention for observables in JS
const thoughts$ = from(thoughts);

That's it!

Getting Data

Back to our promise. What do you suppose happens when thoughtsPromise resolves?

const getValue = async () => {
  try {
    const thoughtsArray = await thoughtsPromise();
    console.log(thoughtsArray);
    // Output: ["food", "sleep", "code"]
  } catch (err) {
    // handle error
  }
};

We got the array all at once. And what do you think happens when we start to listen for values, in other words subscribe to our thoughts$ observable?

// We subscribe to an observable to get values from it
thoughts$.subscribe(
  value => console.log(value),
  err => null, //handle error,
  () => null // handle completion,
);
// Output:
//    food
//    sleep
//    code

We get the values from the array one by one. A stream of data. Cool.

Play with this example (React)

Do I have to know Rx?

Nope. But here are some use cases to consider:

1- To keep code flat in JS if you can't use async await for some reason.

2- To handle complex event-based logic whether its network related or UI related (e.g. websockets / drag and drop).

3- If your teammates come from different languages but you all know Rx, it might be easier for them to just get going with RxJs than to use promises and async await.

4- In other languages, Rx is super useful for handling multi-threading.

5- If you enjoy reactive programming and want to apply it everywhere, go ahead 🙂.

Debouncing user input (React)

Useful links:

List of Rx operators

http://reactivex.io/documentation/operators.html

Awesome video on RxJs operators by Fireship

https://www.youtube.com/watch?v=ewcoEYS85Co

Reactive programming

https://en.wikipedia.org/wiki/Reactive_programming

Imperative programming

https://en.wikipedia.org/wiki/Imperative_programming

Declarative programming

https://en.wikipedia.org/wiki/Declarative_programming

The Observer Pattern

https://en.wikipedia.org/wiki/Observer_pattern

Chunking Arrays in Javascript

Osama Qarem — Thu, 22 Aug 2019 01:34:56 +0000

I recently ran into an issue inserting large data into an SQLite database on my react native app. I had purposely created a large mock data set to test what would happen should the user attempt something similar.

And I instantly ran into an SQLite limit, specifically number 9.

https://www.sqlite.org/limits.html

QueryFailedError:
too many SQL variables (code 1 SQLITE_ERROR):, while compiling:
INSERT INTO "table_name"("Id", "columnOne", "columnTwo") VALUES (...)

Apparently, this is to protect against excessive memory allocation. You can read more about it in the link above.

The problematic query:

// TypeORM
await connection
  .createQueryBuilder()
  .insert()
  .into("table_name")
  .values(largeArray.map(item => item))
  .execute();

What we can do here is chunk our data. We can then run multiple queries instead of just one to insert the same amount of data.

To illustrate the data that we currently have, I'll use an array of strings:

const beforeChunking = ["1", "2", "3", "4", "5"];

What we want to have is arrray of arrays with a specified chunk size. For example with a chunk size of 2:

const afterChunking = [["1", "2"], ["3", "4"], ["5"]];

We can go about achieving this by utilizing splice(). We create a function that takes in the data to chunk and the required chunk size.

const getArrayAsChunks = (array, chunkSize) => {
  let result = [];
  let data = array.slice(0);
  while (data[0]) {
    result.push(data.splice(0, chunkSize));
  }
  return result;
};

splice() will remove elements from data the size of chunkSize. The remaining elements will move backwards in the array so everything works out.

Now we can run our insert query(ies) like so:

const chunksArray = getArrayAsChunks(largeArray, 100);

chunksArray.map(async oneChunk => {
  await connection
    .createQueryBuilder()
    .insert()
    .into("table_name")
    .values(oneChunk.map(item => item))
    .execute();
});

This took less than two seconds to insert ~2000 rows on a low-end android phone, so I left it at that. But one performance improvement we can make here is to define the SQL query ourselves, rather than let our ORM handle that; generating the query many times over incurs a cost.

This article was originally posted on my blog.