Forem: Vishal Vijay

DataRepoArch - The path we traveled to reach react-query

Vishal Vijay — Sun, 27 Sep 2020 17:01:24 +0000

For the last few months, I was busy working with Kappitaan.com. And I was primarily responsible for building their website. And at the core, we used Next.js, Redux, a custom design system based on MaterialUI and Typescript.

In this article, I'm going to cover how and why we invented DataRepoArch and later found a better and mature opensource alternative for the same.

DataRepoArch is a client-side architecture designed to use with react for better server state handling. This architecture also guarantees a very neat, highly pluggable, and maintainable codebase.

Kappitaan.com help empower students and healthcare professionals by making it easier to train and get placement for work abroad.

So the story starts here

Though in the initial phase we didn't have many global state management requirements, we still integrated redux to our framework (without compromising code-splitting benefits). We were sure that we have a lot of server state handling coming for the dashboard part of the website.

Before we continue, please check the sample screenshots of the dashboard to get an understanding. Sample 1, Sample 2

With the help of screenshots, we can understand the below points and requirements:

Each page of the dashboard is consistent with multiple data cards.
Some of the cards require data from the same source.
Every page requires common dashboard related metadata to start loading.
Data can get modified from anywhere on the dashboard.
There are pages that require the same data which we fetched on other pages.
A single data card might require data from multiple sources.
If one of the API fails, users don't have to click retry on every card which uses the same data.

So to accomplish these requirements, we started the development plan with the traditional redux approach. This means, page component handles the fetching of the data and distributes it across the data cards based on its data requirements.

Listing the problems which we identified with this approach:

The page component needs to know what all data cards are there on the page and its data requirements.
Everything has to be routed through the page component as it handles the data fetching and caching using redux.
API error handling and retry also became challenging.
Lot of boilerplate code to handle data fetching.
On page navigation, it triggers multiple API calls if the previous API hasn't finished loading for the same data.
Data components are unable to abstract the business logic and it became really difficult to reuse the components.
So many props to be passed in each data card to handle different states.
Mutating the data also leads to a lot of boilerplates.

We understood that, if we take this approach, it is going to make our code difficult to maintain and adds a lot of boilerplates. Eventually, every feature additions are going to consume time in both developing and testing.

So, we decided to come with different approaches. Another suggestion was to move data loading inside a specific data card component. And that will help us solve many of the above problems mentioned such as data card can be reused, need not depend on parent or page component, etc.

But it still doesn't solve the problem of duplicate API calls, ease of development, and boilerplate code.

But we all agreed that the above is the way forward but we need a better framework in place. So we did our R&D and found a few libraries and architecture like react-query. When we initially looked at react-query we thought that this is another data fetching library similar to fetch. The fact is, we didn't do deep dive into the details of it.

So after the R&D, we didn't identify something which suits our requirements. So we decided to innovate ourselves.

DataRepoArch begins here

As we already have a redux system integrated, we decided to build something which is pluggable to it. And we started listing down the problems which we want to solve and the features we want to support.

It must allow using any data fetching logic. Can be from a remote source, local data, fake data, or even from already downloaded data.
All data should be by default in-memory cached and should get re-fetched if intentionally triggered (usually after mutation).
It should not allow duplicate API calls if different data cards make the same request when already one is being fetched.
It should easily allow handling data fetching states.
It should allow fetching and handle multiple data together for the same data card.
It should allow retrying API call if the previous one failed. And on retry, it should refresh all the data cards depended on the same data.

So we started architecting it

A working POC application can be found here. Randomly try navigating and refreshing the pages and experience the features like data fetching, caching, retry, parallel fetching, etc.

We are throwing errors randomly in the data fetching logic, to make it easier to test retry behavior.

Before we talk about the internal details of the architecture, let's see how a developer can use this architecture.

So let's look at Page 1

const Page1: React.FC = () => {
  return (
       ...
          <UserDetails />
       ...
          <UserDetails />
       ...
          <Product productId={200} />
       ...
          <Product productId={140} />
       ...
          <Product productId={200} />
       ...
          <UserAndProduct />
       ...
  );
};

Page 1 is consists of 3 unique data cards. In it, we are showing user details card 2 times, and 3 different product card. Also an example of fetching user and product details in a single card. Looking at the above sample you will understand, how pluggable are each card and all its business logic have been abstracted in itself.

Let's look at the code of all 3 unique data cards here

Before we continue, a quick note about DataLoadingHandler. It is a simple component that takes care of showing loader and API error based on the fetching state.

UserDetails

const UserDetails: React.FC = () => {
  const userDetailsRepo = useUserDetailsRepo();

  return (
    <Paper>
      <Box p={2}>
        <DataLoadingHandler
          successCode={userDetailsRepo.successCode}
          errorMessage="Failed to load user details"
          onRetry={userDetailsRepo.refresh}
          render={(): JSX.Element => (
            <>
              <Typography gutterBottom variant="h4">
                Name: {userDetailsRepo.data?.name}
              </Typography>
              <Typography gutterBottom>
                Email: {userDetailsRepo.data?.email}
              </Typography>
              <Typography gutterBottom>
                Total votes: {userDetailsRepo.data?.totalVotes}
              </Typography>
            </>
          )}
        />
      </Box>
    </Paper>
  );
};

The DataRepo API is as simple as this.

Use the custom data repo hook.
Pass successCode to DataLoadingHandler
Use the data the way you want in the render method.

Product

const Product: React.FC<Props> = ({ productId }) => {
  const productRepo = useProductRepo(productId);

  const product = productRepo.data;

  return (...);
};

Product card implementation is also similar to UserDetails the only difference is, we are passing productId to useProductRepo hook to fetch independent product details.

UserAndProduct

const UserAndProduct: React.FC = () => {
  const userDetailsRepo = useUserDetailsRepo();
  const productRepo = useProductRepo(23);

  const aggregatedRepo = useAggregatedRepo([userDetailsRepo, productRepo]);

  return (
    <Paper>
      <Box p={2}>
        <DataLoadingHandler
          successCode={aggregatedRepo.successCode});
    ...
};

The UserAndProduct example demonstrates the use case of useAggregatedRepo. This helps to aggregate multiple custom data repo hooks fetching state to a single state for better loading and API error handling.

Now let's look into custom data repo hooks

There are 2 data repo hooks we wrote for this POC

useUserDetailsRepo

const fetchUserDetails = synchronisedPromiseMaker(
  async (): Promise<UserDetails> => {
    // eslint-disable-next-line no-console
    console.log("Fetching user details...");

    await sleep(2000);

    if (Math.floor(Math.random() * 10) % 5 === 0) {
      throw Error("Failed to load user details");
    }

    return {
      email: "hi@example.com",
      name: "Sample name",
      totalVotes: 200,
    };
  }
);

const useUserDetailsRepo = (): Repo<UserDetails> =>
  useRepo<UserDetails>("userDetails", fetchUserDetails);

synchronisedPromiseMaker is a utility created to prevent duplicate API calls at the same time.

The key to DataRepoArch is useRepo, this is where all the magic happens. You simply have to provide a repo name, a synchronized promise (which takes care of data fetching the way the developer wants), and the rest of the arguments will be forwarded to the data fetching method.

That's it, the data repo is ready.

useProductRepo

const fetchProduct = synchronisedPromiseMaker(
  async (productId: number): Promise<Product> => {
    // eslint-disable-next-line no-console
    console.log("Fetching product...", productId);

    await sleep(2000);

    if (Math.floor(Math.random() * 10) % 5 === 0) {
      throw Error("Failed to load product");
    }

    return {
      id: productId,
      name: `Sample product ${productId}`,
      price: 450,
      quantityAvailable: 23,
      category: "Sample category",
    };
  }
);

const useProductRepo = (productId: number): Repo<Product> =>
  useRepo<Product>("product", fetchProduct, productId);

useProductRepo is also similar to userDetailsRepo but it accepts productId as an argument.

That's all... developer need not worry about anything else. We solved all the problems we discussed above.

Now let's look at the core components of DataRepoArch

Redux

The current POC implementation is an addon for Redux. We did it because it can be easily plugged into our current project. But this redux dependency for DataRepoArch was unnecessary. We had only a little time to implement this all architecture. So for this version, we didn't want to reinvent the wheel and we want to take the advantage of the already existing architecture.

useRepo

This is the core of DataRepoArch. It abstracts:

Redux communication
Promise execution
Loading state management
Data caching based on arguments
Reset data functionality
Refresh data functionality

Using the reset or refresh method you can ask to clear or refresh the data which is already cached in redux. This will come handy when you want to mutate any dependent data.

useAggregatedRepo

This hook will help to handle multiple data dependency loading. You can avoid a lot of duplicate code in your component.

Yes... that's all about DataRepoArch. It's as simple as that.

But we think react-query is much better

Our current Kappitaan.com website uses DataRepoArch in production and we are really happy with the current product deployed. But we know that there is a lot of opportunities for improvements.

When we initially looked at react-query, we were not sure about all the problems which we have to solve in the current project. Also, we didn't understand the full potential of react-query and it was quite new at that time (Major development of the library happened in early 2020). Our implementation of DataRepoArch started around March 2020.

Along with the project, DataRepoArch also started evolving and we enhanced the architecture to allow the developer to write better code. At the later stage of the project, we had a chance to read more about react-query and that is when we started comparing our feature with react-query. Listing some of the core features supported by it.

Transport/protocol/backend agnostic data fetching (REST, GraphQL, promises, whatever!)
Auto Caching + Refetching (stale-while-revalidate, Window Refocus, Polling/Realtime)
Parallel + Dependent Queries
Mutations + Reactive Query Refetching
Multi-layer Cache + Automatic Garbage Collection
Paginated + Cursor-based Queries
Load-More + Infinite Scroll Queries w/ Scroll Recovery
Request Cancellation
React Suspense + Fetch-As-You-Render Query Prefetching
Dedicated Devtools (React Query Devtools)

Though we were sad that we didn't realize the potential of react-query and reinvented similar features in DataRepoArch. But we were happy that the problems which we tried to solve are valid and the thought process and solutions which we put to it are aligned with a quality library like react-query. So we are proud of what we made 💪.

The future plans

We are not going to invest more in DataRepoArch. Instead, we are planning to migrate to react-query. And our team will look into potential contributions to it.

The story behind salesforce-orm Gem

Vishal Vijay — Thu, 24 Sep 2020 11:52:28 +0000

The story behind salesforce-orm Gem, a successful hack we used to speed up Nestaway's Salesforce migration.

This post was migrated from my old personal blog. This was originally published on Jan 2018

What is Salesforce ORM

An ActiveRecord like ORM for Salesforce which can be used in ruby projects.

The story

In early 2017, Nestaway decided to move many of its processes to the Salesforce platform. As part of this, across the teams, many core components from the backend services have to be migrated to Salesforce. Few services became just a proxy to Salesforce, and for many services, the source of truth for data became either Salesforce or our backend systems, with the proper syncing mechanisms in place.

So in short, our backend systems have to play with a lot of Salesforce data.

Our devs had very little experience working Salesforce. So the initial learning curve for the team was very high.

As this migration process was huge, so the entire process was divided across teams base on service ownership. My team (Tenant experience) was the initial team who started working on this process. We had 2 main responsibilities:

Completly reengineer and migrate service request market place (SRM) from Freshdesk to Salesforce service cloud.
Migrate Nestaway users (both tenants & owners) to Salesforce and build a 2-way sync mechanism.

So we started with the SRM implementation. As the entire SRM system was being reengineered, we had decided to keep both Freshdesk and Service cloud to be active until we archive the data from Freshdesk. So we architected the system with the bridge pattern and built drivers for Freshdesk, service cloud, and Google cloud data store (for archival).

At Nestaway, most of the backend service was written in ruby and ROR.

So now on let's focus on the driver we built for the service cloud.

As I told earlier, our team had only very little experience working with Salesforce APIs. So as expected our initial code was full of boilerplates. We also did a 2nd time refactor to reduce the boilerplate. But still as and when the service started growing the boilerplate started increasing which made unit testing and code maintenance horrible.

For this implementation, we used a great Gem called restforce to work with Salesforce REST APIs.

That is when we actually came to know that, Salesforce supports a query language called SOQL. When we looked at it, we found that it is very similar to SQL syntax, and they support all CRUD operation similar to SQL. Also, we understood that restforce has direct API to execute SOQL queries.

That was a Eureka moment, we immediately started working on a basic SQL to SOQL converter. And tested it. With that positive impression on that same night, we built a basic working MVP that has a similar API like ActiveRecord. The next morning we demoed it to the team and everyone loved it. That gave us a green sign for building a more concrete Gem. But we were already halfway through the project and now we had to build this new Gem and migrate the existing code to use it (available time was less).

But with 3 to 4 days effort, we built the first usable version of the gem and we started integrating it into the running project. Surprisingly, that migration was super easy.

About the salesforce-orm

This Gem was providing an ActiveRecord like API to work with Salesforce objects. Any rails developer can easily learn this with their ActiveRecord experience.

What is the real hack behind it, which helped us build this entire stuff within 4 days?

In simple words, its the power of ruby, Metaprogramming.

With a proxy, we actually used ActiveRecord itself to do the trick. The trick was, how we generated the SQL query. With the help of nulldb we created a fake ActiveRecord instance, which made it generate SQL query.

Later this SQL query is converted to SOQL with a simple converter and forwarded it to Salesforce using restforce gem.

Listing the core modules of the Gem

ObjectBase: Like ActiveRecord::Base, we have to create each Salesforce object (model) by inheriting this class. This class becomes, the primary proxy to bring in APIs similar to ActiveRecord.
Base: This is the brain of this Gem. It takes care of building the query, making API requests to Salesforce, and all required data conversions.
QueryBuilder: The real hack for generating SQL query.
SqlToSoql: A simple SQL to SOQL converter. We know that it is not a solid converter. With the available time, we made it support most of the regular scenarios. And in most cases, we didn't have any issues with this converter.

Conclusion

The decision to build salesforce-orm came out really valuable for Nestaway because it helped us to speed up the future releases and it got widely adopted across Nestaway teams. Developers loved it because it was providing ActiveRecord like API. And they didn't have to worry about, what is happening underneath.

Overall this project was a huge success, and we were really happy to contribute the project to the community. \m/