Forem: Navaneeth Pk

ToolJet x Hacktoberfest'22

Navaneeth Pk — Fri, 30 Sep 2022 10:43:09 +0000

It is October again!

In open source, we feel strongly that to really do something well, you have to get a lot of people involved - Linus Torvalds

Hacktoberfest is a one-month event that encourages ~~developers~~ everyone to contribute to the open-source projects that they love. We are excited to announce that ToolJet will be participating in Hacktoberfest 2022.

ToolJet is an open-source low-code platform for building internal tools. We open-sourced ToolJet in June 2021. We have more than 13,000 stars and 200 contributors for our GitHub repository. The open-source community consistently help us improve ToolJet by contributing new features, reporting & fixing bugs, improving the documentation, helping us debug the bugs, and more.

Before we dive in to the details of Hacktoberfest 2022, let's quickly recap of last year's Hacktoberfest.

We merged 159 pull requests from 72 new contributors during Hacktoberfest 2021.

Hacktoberfest 2022

Hacktoberfest is an opportunity for everyone to get involved in open-source projects.

With great opportunity comes great swag. Every valid PR is eligible for ToolJet swags!

1 valid PR: ToolJet stickers and t-shirt.
3 valid PRs - ToolJet stickers, t-shirt, water bottle.
4 valid PRs - ToolJet stickers, t-shirt, hoodie, water bottle.

Additionally, the top 5 contributors will get a surprise gift this time! 🥳

How to participate

Every contribution matters. It can be bug fixes, new features, new plugins, code refactoring, improving the documentation, creating tutorials, adding more unit tests etc.

Not familiar with our codebase? no problem, we have tagged issues on Github with the label good first issue . You can view them here: issues for newcomers. Additionally, we've added labels for the frontend (ReactJS) and backend (NestJS). Please make sure that you are creating PR only for issues that are assigned to you.

We do not tolerate spamming. If you're finding it difficult to contribute code, you can still help by reporting spam PRs, which are PRs created just for the sake of creating PRs. Spam PRs should be labeled as spam or invalid as per the guidelines of Hacktoberfest.

Resources

Where to get help during Hacktoberfest

We have a very active community on Slack. Feel free to join: https://tooljet.com/slack..
You can email us at hello@tooljet.com

Looking forward to see you on GitHub!🚀

From zero to 10,000 stars and 160 contributors on GitHub

Navaneeth Pk — Mon, 16 May 2022 12:10:29 +0000

We are excited to announce that our GitHub repository crossed 10,000 stargazers. It has been an amazing journey building ToolJet alongside our energetic and involved community. In this article, we will discuss about our learnings from this fast-paced journey of building an open-source product that the community loves.

For those who haven't heard about ToolJet, ToolJet is an open-source low-code framework for building custom internal tools.

The first commit to our GitHub repository was on March 31, 2021, and we made the repository public on June 7, 2021, and launched it on Hackernews.

Our journey since we made our repository public!

April 2021

The first commit to the repository was on March 31st, 2021. Most of the April was spent on pushing more and more commits to the repository. Major challenges were to get the drag and drop builder working.

ToolJet's POC looked like this:

and then improved to this at the end of April:

May 2021

I reached out to a few companies that I had relationships with to try out ToolJet and worked more on iterating ToolJet based their feedback. Some of my friends also stepped in to help build ToolJet by contributing to the repository. At this point, the repository was private.

At the beginning of May, I tried to reach out to angel investors/VC firms and applied for Y Combinator but everyone rejected since there was no traction to convince the idea. I've written about this in detail in this article: Building and launching ToolJet.

June 2021: The launch of public beta & fundraising.

Building the website, creating graphics for posting on ProductHunt and finding a hunter was done in the last week of May and first week of June.

ToolJet was launched on ProductHunt first and then a few hours later on Hacker News. ProductHunt post was among the top 5 of the day within a few minutes and later ended up being #1 product of the day.

Now we have some inbound interest from VC firms since we got the initial traction that we needed to convince that this is an idea worth pursuing. Now the challenge is to find investors who understand developer tools and open-source. Long story short ( because we have another article that explains why we raised funding and how did we chose the partners - link ), we were lucky that we quickly found Nexus interested in leading our seed round. They had great experience in developer tools and open-source as they were the early investors in companies like Hasura, Min.io and Postman.

We also managed to onboard two other VC firms and a few angel investors. We did not try to get more termsheets or reach out to more firms as we were optimising for quickly closing the round so that we can go back to building ToolJet. In the end, we raised a $1.55 million seed round.

Decision to go ahead with VC funding helped us in many ways, will write about it in in detail in the coming weeks.

July 2021: The importance of stack in open-source.

ToolJet server was initially built using Ruby ( Ruby on Rails ) and ToolJet client was built using ReactJS. Having two languages ( Ruby and JavaScript ) in the codebase was a barrier for many to contribute to ToolJet. The suggestion of choosing the right stack and a plugin-based architecture for an open-source project came up when I had a chance to talk to Tanmai Gopal, co-founder of Hasura.

Since ToolJet was still in it's early stages, we decided to port the server completely to JavaScript/TypeScript. The whole process took around 6 weeks but it helped us change the architecture to make the server modular. It was a difficult decision to stop working on new features and spend more than a month on migration but this decision paid off in the long run.

We've written in detail about this on our blog:
Part1: Why we are migrating to Node.js
Part2: How we migrated from Rails to NestJS

August & September 2021: Building the team.

We hired our first four engineers in July and onboarded them in August. The first 15-20 team members sets the culture of the company and thus we were focusing more on culture and less on experience in Node.js. We managed to hire senior engineers from our networks as well as from hiring platforms.

The rest of 2021

In the last quarter of 2021, we managed to ship a bunch of major features:

Team collaboration features: users can comment anywhere on canvas and tag other team members.
Ability to run JavaScript from within ToolJet.
Application template library for users to get started easily.
Connectors for cloud storages such as AWS S3, GCS & Min.io.
Ability to export applications and import them into different environments.
Permissions based on user groups.
Support for authentication using Google SSO.

During this time, our GitHub repository crossed 4,000 stargazers & 100 contributors. We also added the first developer advocate to the team to address the needs of our growing community.

January 2022: De-coupling connectors as plugins & preparing for v1.0 launch.

The initial version of ToolJet did not have the concept of plugins and every connector lived in the codebase. When we migrated to Node.js, the logic for connectors was abstracted into a **plugins** folder within the directory for ToolJet server in the codebase.

We soon realised that it will not be possible to scale connectors if they had to live within the codebase. Having hundreds of connectors can slow down the server and most of the users will not need more than 5 connectors.

Majority of the low-code frameworks have very less number of connectors and often the long tail of connectors are ignored. We wanted to chase this long tail of connectors and also give users the ability to quickly build connectors as per their requirements.

We decided to de-couple the connectors from the codebase as plugins and built a Plugin Development Kit for the community to easily build plugins.

Now the connectors can stay completely outside the codebase and these plugins can be easily bootstrapped using tooljet-cli (link).

We also decided to launch ToolJet v1.0 in February. ToolJet was being used by many large organisations in production since last few months but we did not want to call it a v1.0 unless we've de-coupled the plugins.

In January, we smashed numerous bugs and user experience issues that were causing trouble to the users. We also spent time on adding more templates to our template library, adding more ways to deploy ToolJet on premises, support for more SSO providers, etc.

February: Launching v1.0

On February 8th, we released ToolJet v1.0 and launched it on ProductHunt. Having the beta tag prevented many organisations from using ToolJet as beta is often read as "not stable, expect bugs". Now this issue is resolved!

We've written in detail about the changes that we've made in v1.0 here.

// Detect dark theme var iframe = document.getElementById('tweet-1490970429841838080-452'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1490970429841838080&theme=dark" }

We did not stop there, we shipped a lot of other features in February such as integrations with Snowflake, n8n, etc. Majority of our time was being spent on feature requests and bug reports from the community at this point.

We also crossed 5,000 stars in February!

March: Meeting the team in person for the first time

We've grown into a team of 12 at this point but most of the team members haven't met the others in person. Our plan was to organise team meetups every quarter but the covid situation did not let us do that until now. We spent a week in Goa, India collaborating and brainstorming on different features. It was exciting to meet the people with whom you talk on a daily basis ( mostly texts because we prefer asynchronous communication ) but never got a chance to meet in person.

During this month, we were focusing on three main things:

Support for multiplayer editing.
Ability to have multiple workspaces within ToolJet.
More connectors.

April: ToolJet now supports multiplayer editing & custom React components

At this point, large teams that were using ToolJet started facing issues because only one user will be able to edit an application at a given time. We started looking into making ToolJet a collaborative workspace. Figma has done this well and have also documented how they did it (link). This resource was very helpful for us. Here is our article explaining a POC built using CRDTs: link.

We've also added support for bringing your own React components to ToolJet. Low-code frameworks are not known for their flexibility but now with ToolJet, you can bring your own React components, create connectors using our plugin development kit and even create JavaScript snippets and run them from within ToolJet. That covers almost all areas where a user will need flexibility.

We also added support for multiple workspaces within ToolJet. Users can now be part of different workspaces and the workspaces can have their own SSO configurations.

During this period, we also crossed 150 contributors on GitHub!

// Detect dark theme var iframe = document.getElementById('tweet-1512473319022293000-536'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1512473319022293000&theme=dark" }

We also made it to #2 among the fastest growing open-source startups. Link.

What did we learn from this journey?

Getting product-community fit is crucial for any open-source startup. We have been working closely with our community to understand their needs and improve accordingly and it worked well for us. Moving forward, we are allocating more resources to help the community. This involves providing better support through our Slack group, faster turnaround times for issues & pull requests, creating more content for the community to get started easily, creating more short reference videos and articles on how to use ToolJet, organising more community events and more!

What's going on and what's next?

We believe we are still in the early days of ToolJet. A lot of things can be done that helps to get lot more things done using ToolJet. We are working on shipping more connectors and more complicated UI widgets. But that's not all, we are working on major features such as syncing applications with GitHub, support for even more SSO providers, improving query builders, making plugins installable, etc. We've a public roadmap here that captures what's going to be built over the next few months.

In short, we want to improve the productivity of developers by letting them build complicated business applications, internal tools and workflows with very minimal engineering effort. For this, we are expanding our engineering, product and community teams at the moment.

Stay tuned for more exciting updates in the coming weeks. We write this at the end of every article but we have also kept that promise by releasing a new and improved version of ToolJet every other week!

Announcing ToolJet 1.0 - separating the platform from the product

Navaneeth Pk — Tue, 08 Feb 2022 10:57:54 +0000

We launched the public beta of ToolJet in June 2021. Today, we are excited to release the ToolJet 1.0.

What were we waiting for?

A lot of users are using ToolJet on production environments since August and the platform did not show any stability or scalability issues. We were waiting for wrapping up one major feature before we call it 1.0.

We are today making public the ToolJet developer platform along with tooljet command-line tool. ToolJet developer platform allows any JavaScript developer to build and publish plugins for ToolJet. For this phase one, developers will be able to build connectors for ToolJet. For example, building a ToolJet connector for BigQuery takes only 30 minutes including integration tests.

We made this possible with the help of our command-line tool tooljet. Our command-line tool can bootstrap the file structure for plugins, add npm dependencies for the plugins and even run tests for specific plugins.

Since our launch, we have been getting a lot of contributions from developers around the world. Our developer platform will make it easy for any JavaScript developer to extend ToolJet using plugins. This was one of the reasons for us migrating the ToolJet server from Ruby to Node.js last year. We wanted to open up our platform for the growing JavaScript community.

Plugins are useful for our users as they do not have to wait for the core team to build the connectors that they need. Imagine using a closed source product where the users are at the mercy of the company to build features and fix bugs. Will they even care if the extension will not be used by the majority of their customers?

What else is new and what changed since the beta launch?

a) Ported ToolJet server from Ruby to Node.js

Yes, the server was built using Ruby when we launched ToolJet first. Right after the launch, we ported the server to Node.js. This has helped us leverage the contributions of the amazing and active JavaScript community. This has also helped us build the developer platform for a larger number of developers. We've written in detail about this in another blog post.

JavaScript is the most popular language of 2021.

-Source

b) Plugin based architecture

We altered the architecture to support extensibility. We built a plugin system so that any JavaScript developer will be able to extend ToolJet easily. For example, a simple plugin for connecting ToolJet with BigQuery can be built in less than 30 minutes.

We believe our open-source and plugin-based approach helps the engineering teams to customise our low-code framework as per their requirements.

c) Team collaboration features

Real-time collaboration between the team members is now possible with the conversations feature of ToolJet. Users can now tag their team members and comment on the canvas to collaborate.

d) Templates by ToolJet

Now you can choose from dozens of templates instead of creating applications from scratch. Templates even include full-fledged database viewer apps for PostgreSQL and MySQL.

e) More data sources

We launched first with 7 data sources. Now we have integrations with more than 20 data sources. We've even added support for cloud storage like AWS S3, Google Cloud Storage and Minio. But that's not all, our new architecture will enable us to build and test new data sources easily.

f) More UI widgets

We launched a public beta with 12 UI widgets. Now we have more than 35 widgets. We now have widgets for picking files, list layout, pagination, timer, tags and statistic.

g) Better application builder

We redesigned the application builder to make it super easy to use. We've also added a bunch of keyboard shortcuts for actions like undo, redo, delete widget and more. The focus of the redesign was the usability of the editor.

h) Support for running custom JavaScript snippets

You can now run JavaScript code from within ToolJet. This adds to the flexibility of the framework. Libraries like momentjs, papaparse, lodash, etc are supported within the snippets.

i) Debugger

Runtime errors that happen in your applications are now logged in the debugger. If a query failed or if a widget ran into an error, you can debug it easily using the debugger.

j) User groups and permissions

Permissions for every app and actions like user creation, folder creation, etc can now be controlled on a user group level as well as on an organization level.

And obviously a lot of many other features, bug fixes and improvements. But that's not all, exciting new features will be released in the coming weeks.

Checkout ToolJet v1 on GitHub — https://github.com/ToolJet/ToolJet

12 ways to get more GitHub stars for your open-source project

Navaneeth Pk — Mon, 17 Jan 2022 06:47:45 +0000

The steps that we took to grow from 0 to 4,500 stars on GitHub within a few months.

We launched ToolJet (https://github.com/ToolJet/ToolJet) in June 2021, since then we've got more than 4500 stars for our repository. Here is a list of things that worked for us. This is not an article about how to just get more stars for your repository. The article instead explains how to present your project well so that it is helpful for the open-source community. Some of these points have helped us get contributions from more developers, we have contributions from more than 100 developers now.

PS: The graph above was generated using an app built with ToolJet. You can use it here to generate a star history chart for your project - https://apps.tooljet.com/github-star-history

1) Readme matters

Readme is the first thing that a visitor to your repository sees. The readme should be able to convey what your project does, how to install the project, how to deploy the project ( if applicable ), how to contribute and how it works. Also, use badges that are helpful for the developers. We used https://shields.io/ for adding badges to our Readme.

Here is how our Readme looks like:

Examples of projects with great Readme:
a) https://github.com/nestjs/nest
b) https://github.com/typesense/typesense
c) https://github.com/airbytehq/airbyte
d) https://github.com/strapi/strapi

2) Documentation

We get more traffic to our documentation portal (https://docs.tooljet.com/) than our main website. A well-documented project is always loved by the community. Open-source projects like Docusaurus makes it super easy to build documentation portals that look great just out of the box. Adding links to the repository from the documentation can drive more visitors to your repository.

Here are some projects with great documentation:
a) https://docs.nestjs.com/
b) https://docs.n8n.io/
c) https://guides.rubyonrails.org/
d) https://plotly.com/python/
e) https://docs.mapbox.com/

3) Drive visitors from your website to GitHub

A lot of visitors checked out our repository after visiting our website first. Add banners, badges, etc to your website so that the website visitors will check out your repository. To drive more visitors to your website, writing blog posts about relevant topics helps.

4) Be active in developer communities

There are many discord/slack communities, forums, Reddit communities, etc where developers usually hang out. Be active in these communities without making it look like self-promotion ( which can get you banned for obvious reasons ). Try to add value to the communities by participating in relevant discussions. For example, if you are building a charting library and if someone is asking a question about plotting charts using React, you can pitch in to help.

Play nice. Do not try to link to your project if it does not add any value to the discussion.

5) Email campaigns

You might already have users signed up for your website. Add a link to your GitHub repository in the welcome email.

Do not spam people who haven't signed up for your updates.

6) Trending repositories on GitHub

If you make it to the list of trending GitHub repositories ( https://github.com/trending?since=daily ), it can get your repository a lot more visibility. Whenever we made it to the trending list, we always got more visitors to our repository and website. There are trending lists for specific languages too. Many Twitter bots and other tools notify developers whenever there is a new repository that has made it to the trending list.

7) Ask for feedback from relevant communities

Communities such as ProductHunt, Hackernews, Reddit communities, etc may find your project useful. This can bring in more visitors and stargazers to your repository.

Target only the relevant communities. If you think majority of the members won't find your project interesting, it is not a relevant community. Spamming can cause more harm than good. Also, it's just not nice.

8) Grow a community

Start a community on Discord or Slack for your users and contributors to hang out. Communities can be helpful when the members are stuck with something and if they want to propose something new. If there is an active community, your future posts and announcements might get more reach. We created the community on Slack since most of the developers have a Slack account. Do not use lesser-known platforms for building your community as it would take an additional step for the person to join the community.

Play nice. Appreciate the work of others.

9) Add a public roadmap

A public roadmap helps your users and contributors understand where your project is headed. There are many tools available for creating public roadmaps but in most cases, GitHub projects will be more than enough for creating a simple yet effective public roadmap. We have created one using GitHub projects - https://github.com/ToolJet/ToolJet/projects/2

10) Twitter

Being active on posts related to your projects can create awareness, increase the number of followers on Twitter and drive more visitors to your repository. Make sure to link your repository on the project's Twitter profile. Also, add a tweet button to your GitHub repository.

11) Respond to feedback

Open-source communities are usually very helpful and give a lot of feedback. Respond to all this feedbacks as the person has taken their valuable time to help you improve your project. Positive feedback helps you stay motivated while negative feedback helps you rethink.

Do not try to evade negative feedback, work on it if it aligns with your vision, otherwise politely explain.

12) Add relevant labels for contributors

Adding labels such as "good first issue" and "up for grabs" can attract more contributors to your repository. There are many platforms such as https://goodfirstissue.dev/ that scans for issues tagged with relevant labels to help contributors discover new repositories and issues to contribute to. Make sure you respond to contributors quickly. Contributors can be experienced developers as well as developers in the early stages of their careers or students. Try to help the first time contributors to help them onboard easily.

You landed on this article possibly because you have an interesting open-source project. I'd love to see your project. I'm available at navaneeth@tooljet.com and on Twitter.

Hope this article was helpful for you. We would really appreciate it if you can take a moment to give us feedback on ToolJet - https://github.com/ToolJet/ToolJet

Deploying a NestJS application with PostgreSQL database and react frontend on Heroku

Navaneeth Pk — Fri, 30 Jul 2021 03:43:15 +0000

Recently, we ported the ToolJet server from Ruby on Rails to NestJS, ToolJet can be deployed to Heroku using the one-click deployment feature of Heroku. ToolJet server is built using Nest.js with TypeORM as the ORM and PostgreSQL as the database. This article will explain how to deploy a NestJS API application on Heroku using the one-click deployment feature of Heroku.

Many of our users deploy the frontend and backend separately, the backend might be deployed on Heroku/K8S/EC2 while the frontend is served from Firebase/Netlify/etc. The first part of this guide explains how to deploy a NestJS API to Heroku and the last part explains how to deploy the frontend too.

1) Create app.json file in the root directory of your repository

{
  "name": "ToolJet",
  "description": "ToolJet is an open-source low-code framework to build and deploy internal tools.",
  "website": "https://tooljet.io/",
  "repository": "https://github.com/tooljet/tooljet",
  "logo": "https://app.tooljet.io/assets/images/logo.svg",
  "success_url": "/",
  "scripts":{
    "predeploy": "npm install && npm run build"
  },
  "env": {
    "NODE_ENV": {
      "description": "Environment [production/development]",
      "value": "production"
    }
  },
  "formation": {
    "web": {
      "quantity": 1
    }
  },
  "image": "heroku/nodejs",
  "addons": ["heroku-postgresql"],
  "buildpacks": [
    {
      "url": "heroku/nodejs"
    }
  ]
}

Environment variables, add-ons, buildpacks and other information about the app needs to be added to the app.json file. More details about the app.json manifest can be found here.

Using the predeploy hook, we are installing the npm packages required for the application and then build the application. npm run build runs the nest build command. More details about nest build can be found here.We have also added heroku-postgresql to the addons so that a Postgres database will be provisioned by Heroku.

2) Listen to the port assigned by Heroku

Heroku dynamically assigns a port for your app. We need to make sure that the application is listening to requests on the port assigned by Heroku. Modify the main.ts file to listen to the port assigned by Heroku and fallback to 3000. We also need to set 0.0.0.0 as the binding address.

app.listen(parseInt(process.env.PORT, '0.0.0.0') || 3000);

Note: you will come across the following error if the application is listening on a different port. Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch

3) Configuring TypeORM to use Postgres database provisioned by Heroku

Add the following options to your ormconfig(.json/.ts/.js) file.

url: process.env.DATABASE_URL,
ssl: { rejectUnauthorized: false }

DATABASE_URL config variable is added to your app's environment if a Postgres resource is provisioned for your app. Without setting the rejectUnauthorizedoption as false, Error: self signed certificate will be thrown by the application (the reason is explained here).

4) Procfile

Add web: npm run start:prod as a new line to Procfile. We are assuming that the start:prod script is defined in package.json as NODE_ENV=production node dist/src/main. TypeORM migrations can be run after every release.

Add release: npm run typeorm migration:run as a new line to your Procfile. The Procfile will now look like this:

web: npm run start:prod
release: npm run typeorm migration:run

5) Deploy!

You can visit https://heroku.com/deploy?template=https://github.com/your-organization/your-repository/tree/your-branch to deploy the application using the one-click deployment feature of Heroku.

If you want to deploy just NestJS API on Heroku, you can stop reading this guide. If you want to deploy the frontend too to Heroku, please continue.

In the following steps, we will explain how to make NestJS serve a React single page application. We are assuming that the React application lives under the frontend directory.

1) Install serve-static NestJS plugin

npm install --save @nestjs/serve-static

2) Modify AppModule

Add this to the imports.

ServeStaticModule.forRoot({
    rootPath: join(__dirname, '../../../', 'frontend/build'),
}),

3) Routing

Now the NestJS will serve index.html in the build directory of the frontend. This can be a problem when there are similar routes on the frontend and backend. For example, if the frontend application's path for users page is /users and the path to fetch users from the backend is also the same, NestJS will not serve the static files for that path. To solve this issue, let's add a prefix to the backend endpoints.

app.setGlobalPrefix('api');

This line needs to be added to main.ts to make sure the path for all API requests starts with api. For example: http://localhost/api/users.

4) Build the frontend while deploying to Heroku

We need to build the frontend for production to generate the build folder.

"scripts": {
    "build": "npm --prefix frontend install && NODE_ENV=production npm --prefix frontend run build && npm --prefix server install && NODE_ENV=production npm --prefix server run build",
    "deploy": "cp -a frontend/build/. public/",
    "heroku-postbuild": "npm run build && npm run deploy",
    "heroku-prebuild": "npm --prefix frontend install && npm --prefix server install "
}

Add this to the package.json on the root directory of the repository.

5) Deploy!

You can visit https://heroku.com/deploy?template=https://github.com/your-organization/your-repository/tree/your-branch to deploy the application using the one-click deployment feature of Heroku.
We would love you to check out ToolJet on GitHub: https://github.com/ToolJet/ToolJet/

Migrating ToolJet server from Ruby on Rails to Node.js

Navaneeth Pk — Tue, 13 Jul 2021 11:47:15 +0000

The background

We recently open-sourced (ToolJet)[https://github.com/ToolJet/ToolJet/]. ToolJet is an open-source no-code platform for building and deploying custom internal tools.

ToolJet has two main components, the client and the server. ToolJet client is a ReactJS application and ToolJet server is a Ruby on Rails API-only application. Whenever a new application is built using ToolJet, the frontend ( client ) generates the definition of the app in JSON and the server persists the versioned definitions on a PostgreSQL database. The server also acts as a proxy that run queries on top of the data sources ( data sources in ToolJet includes databases like MongoDB and API based services like Google Sheets ).

Ruby accounts for only 14.5% of our codebase. When we built ToolJet, we chose Ruby on Rails as the backend because of its ability to quickly prototype and iterate due to its strong adherence to ‘convention over configuration’.

We open-sourced ToolJet with the eventual goal of letting everyone build plugins ( database integrations, API integrations & UI widgets ) for ToolJet easily. But with the current architecture, if a developer wants to build a new data source integration for ToolJet, he/she should be familiar with both Ruby and JavaScript because running queries is handled by Rails and building queries is handled by query editors in the ReactJS frontend. Thus when we migrate the server to Node.js, the developers will be able to build ToolJet plugins using just one language.

TypeScript

The popularity of TypeScript has been exploding in recent years ( JavaScript Flavors ). TypeScript is a superset of JavaScript and provides an optional type system for JavaScript. The freedom of dynamic typing can cause bugs that are hard to detect. With TypeScript, you can avoid classic JS errors like undefined is not a function. We decided to use TypeScript as it helps us to avoid mundane errors and to improve the maintainability of our codebase.

JavaScript analysis paralysis

It's real!
In the Rails world, everything is already decided for you. But in the Node.js world, there is an endless possibility of file structures, naming conventions and tooling. For example, in Rails, we used ActiveRecord but in Node.js, we get to ( or have to) choose ORM ( if any ). When we started looking for a suitable Node.js backend framework. A huge range of frameworks was available to choose from. This can lead to Choice Paralysis but on the bright side, having too many options at our disposal helps us choose what's best for our use case.

Choosing the framework

SailsJS

Since we are migrating from Ruby on Rails, SailsJS was the obvious choice because of its similarity with Rails. We decided not to use SailsJs when we came across issues raised by developers related to the built-in ORM, waterline.

Express

Express is a very minimal yet powerful framework. Express, being an un-opinionated framework, doesn't come with an error handler, body-parser etc. This gives the developer a lot of freedom but with a lot of freedom comes a lot of responsibilities to choose the right way to do something. We did not want to spend a lot of time discussing what framework to use for every single requirement. Hence, we decided not to use Express.

Meteor

Meteor is a powerful full-stack Node.js framework. We did not go ahead with Meteor as it doesn't support PostgreSQL and migrating the database to MongoDB wasn't something we want to spend our time on. ( We came across meteor-postgres but as their documentation says, it is still a work in progress ).

NestJS

NestJS has everything that we were looking for in a backend framework. NestJS is a slightly opinionated framework but gives some level of flexibility by allowing the usage of other libraries. For example, NestJS uses Express under the hood but it can be configured to use Fastify as well.

The new architecture for the server

We decided to go ahead with NestJS because: a) It fully supports TypeScript b) Database agnostic: we can directly use any Node.js database integration libraries or ORM. NestJS documentation explains in detail integrating TypeORM and Sequelize. c) Excellent documentation: everything is explained well.

We started looking for an Object Relational Mapper (ORM) because we do not want to spend our time building and debugging SQL queries. TypeORM and Sequelize were the most mature choices. We chose TypeORM because it is a matured ORM available for TypeScript.

What's next

In the coming days, we will be working on migrating the Ruby on Rails endpoints and query services to Node.js in a way that the users will not have to change the existing data in their PostgreSQL database.

We would love you to check out ToolJet on GitHub: https://github.com/ToolJet/ToolJet/.