Forem: John Detlefs

A Tip on Reducing Complexity While Coding in React and Typescript

John Detlefs — Tue, 07 Apr 2020 23:10:34 +0000

A few years ago, I was talking to a very talented engineer about what I was working on, lamenting that the product was 'over-engineered', he stopped me and told me 'over-engineering' is a misnomer. When great engineering teams successfully collaborate, the result is a well built product that satisfies company objectives. What I was calling 'over-engineering' happened when workers sought out and attached themselves to complex issues without thoughtful attempts to reduce their complexity.

Some people call it 'speculative generality', I call it 'conflation'. Sometimes this happens because a developer wants to challenge themselves by finding a one-size-fits-all solution to their problem. Some of us do it because communication between product and technical management breaks down to the point where we don't effectively make features more manageable. Regardless, most developers are guilty of falling into this trap. I know I am.

At work we ran into this problem with our tables. We have a bunch of different tables throughout our CRUD admin app, some tables are server-side paginated, some load the data all at once. Some of them are data-rich tables, for those we use https://github.com/gregnb/mui-datatables, since we're using material-ui. We have some tables which are meant to act as form inputs for selecting items. There are a bunch of tables in our app!

This rich set of feature requirements can create a maintenance problem for your application, as we have found out. When building things from scratch, the desire to be clever and to adhere to Dont Repeat Yourself (DRY) can pull even best devs towards an inefficient approach.

Exclusive Tables

Tables with Overlap

Key

A: ServerSideDataTable
B: ClientSideDataTable
C: GeneralDataTable

Before building anything, we can't say with certainty that there will exist any code to share between the table that handles server-side paginated data and the table that handles data fetched on mount. Experience tells us that there will be some opportunity to share code, so it is easy to fall into the trap of a building one table to target the set of features encapsulated by the intersection, GeneralDataTable

For us, this approach became a maintenance burden. If our experience is any indication, the way that your codebase (ab)uses Typescript might be an indicator of conflation causing complexity. Naively, the props exclusively for the ServerSideTable, the non overlap A disjoint C, would likely be expressed via "maybe" types. Say we've done a bad job and our code is documented poorly. If we use maybe types, our lack of documentation is even worse! Without the benefit of a tight contract established by our type for C, we lose the ability to have the use of C define what props it requires. We could use merge-exclusive to have either all types for A or all types for B. This still leads to the complexity of managing the logic for what are things without complete logical overlap in the same component.

What we've done is break our GeneralDataTable into ServerSideDataTable and ClientSideDataTable. With this approach, the core logic for two fundamentally different tasks is kept distinct. We can define the type of props that are necessary for this logic in a way that is easy for all Typescript users to understand. This has already caught errors, and reduced the difficulty to juggle multiple concerns in our GeneralDataTable.

The core takeaway here is that DRY should be applied judiciously to code, but maybe not to so judiciously to your component architecture. Premature abstractions can slow you down and reduce how powerful your type-checking can be for you. Coupling your components to distinct features with their own requirements lets you build n components that are each focused on one job, rather than building one component that is handles n jobs.

CORS and SameSite Cookies Got You Down? An Effective Workaround for Browser Security Policies

John Detlefs — Wed, 01 Apr 2020 22:23:17 +0000

Of CORS My Request Failed

Like many of my colleagues I haven't ever felt like I really understood what Cross Origin Resource Sharing (CORS) policies accomplish. I feel like when I try to learn more about it, I understand it even less. CORS security policies are like the worlds worst child safety lock. Whenever I have looked at information behind it, the explanations are usually very good at explaining what the moving parts are, but rarely give a good explanation for why CORS policies exist. For more background look at these CORS explanations: MDN, Codecademy, Port Swigger, Auth0

There is so much content to look at, but really for most developers this stuff should be abstracted so you don't have to think about it. I doubt very many of us could give a comprehensive breakdown of what a browser request lifecycle looks like. Really though, you can get a long way without deeply understanding this.

As far as I understand it, CORS security defaults on the browser default to the most restrictive rules, any outbound request from content on a domain that isn't making a request to that same domain will be blocked. This is a good thing. Content in the DOM is inherently mutable, so its possible that bad things could happen after your browser renders your HTML/CSS/Javascript.

Your browser is essentially telling you "Hey, if you want to make this request you better put in the work of lying about where its coming from on your server. Get out of here with your funny business!" Servers responding to requests could do work to maintain whitelists for multiple domains, but whitelists are hard to secure and backend developers are justifiably hesitant to make changes to stuff like that.

At Meshify, we've put a fair amount of resources into using Dockerized NGINX servers in tandem with Create-React-App for most of our applications. As a result of this work we've done we are able to:

Share Cookies from authenticated requests between pods in our Kubernetes infrastructure
Develop against a local service that will work with our API with parity against production behavior
Develop against secure WebSockets locally
Develop against Service Workers locally
Maintain legacy apps to keep them working with stricter security standards

Before you get really started

You'll need a config that builds to a directory, in our case its called "build", and our build script is "yarn build"
Install Docker
Install Homebrew
After installing homebrew, brew install mkcert

Creating the certs and templates

mkdir templates in your project's root directory
mkdir templates/localcerts
mkdir templates/nginx

Add your Dockerfile in your root directory

FROM nginx:stable-alpine
COPY templates/nginx /usr/share/nginx/templates
COPY templates/localcerts /usr/share/nginx/certs
WORKDIR /usr/share/nginx/html
COPY build /usr/share/nginx/html

IMPORTANT: That last line build is whatever the name of your build directory is, copy paste errors can trip you up!

Adding some build scripts

scripts: {
    ...
    "build:docker": "yarn build && docker build -t <WHATEVER_YOU_WANT_TO_NAME_YOUR_CONTAINER> .",
    "mkcert": "mkcert -key-file ./templates/localcerts/key.pem -cert-file ./templates/localcerts/cert.pem admin-react.local *.admin-react.local",
    "start:docker": "cross-env REACT_APP_ENV=development PORT=3009 DANGEROUSLY_DISABLE_HOST_CHECK=true react-scripts start"
    ...
}

Adding your nginx config

# Added proxy_host and upstream_addr for better view of proxied requests
log_format extended '${D}remote_addr - ${D}remote_user [${D}time_local] ${D}proxy_host - ${D}upstream_addr '
                    '"${D}request" ${D}status ${D}body_bytes_sent '
                    '"${D}http_referer" "${D}http_user_agent"'
                    'rt=${D}request_time uct="${D}upstream_connect_time" uht="${D}upstream_header_time" urt="${D}upstream_response_time"';

access_log /var/log/nginx/access.log extended;

upstream api_server {
  server ${BACKEND_SERVER};
}

server {
  listen 443 ssl;
  listen [::]:443 ssl;
  ssl_certificate /usr/share/nginx/certs/cert.pem;
  ssl_certificate_key /usr/share/nginx/certs/key.pem;
  server_name admin-react.local;

  # redirect server error pages to the static page /50x.html
  #
  error_page   500 502 503 504  /50x.html;

  location = /50x.html {
    root   /usr/share/nginx/html;
  }
  # Create React App Specific
  location /sockjs-node/ {
    proxy_pass ${FRONTEND_URL}/sockjs-node/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade ${D}http_upgrade;
    proxy_set_header Connection "upgrade";
  }
  location /api/<YOUR_WEBSOCKET_ENDPOINT> {
    proxy_http_version 1.1;
    proxy_set_header Upgrade ${D}http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_pass ${BACKEND_URL}stream;
  }
  # It might need to change depending on what your api url looks like
  location /api/ {
    add_header 'Host' api_server always;
    add_header 'Access-Control-Allow-Origin' "${D}http_origin" always;
    add_header 'Access-Control-Allow-Credentials' 'true' always;
    add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
    add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;    
    # required to be able to read Authorization header in frontend
    if (${D}request_method = 'OPTIONS') {
      # Tell client that this pre-flight info is valid for 20 days
      add_header 'Access-Control-Allow-Origin' "${D}http_origin" always;
      add_header 'Access-Control-Max-Age' 1728000;
      add_header 'Content-Type' 'text/plain charset=UTF-8';
      add_header 'Content-Length' 0;
      return 204;
    }
    proxy_pass ${BACKEND_URL};
  }
  location / {
    proxy_pass ${FRONTEND_URL};
  }
}

Add your docker-compose file

version: "2.4"
services:
  <WHATEVER YOU WANT YOUR SERVICE NAME TO BE>:
    image: <YOUR DOCKER IMAGE NAME>:latest
    ports:
      - "3006:443"
    environment:
      - D=$$
      - FRONTEND_URL=http://host.docker.internal:3009/
      - BACKEND_SERVER=<YOUR_BACKEND_SERVER_URL_WITHOUT_API>
      - BACKEND_URL=<YOUR_BACKEND_SERVER_URL_WITH_API> 
    command: /bin/sh -c "envsubst < /usr/share/nginx/templates/localhost.conf > /etc/nginx/conf.d/localhost.conf && exec nginx -g 'daemon off;'"

Caveat!

Throughout these examples youll see text like <YOUR_BACKEND_SERVER_URL_WITH_API>. I'm expecting you to substitute these with your own endpoints and text. Trailing slashes can be important. If you're running your app and getting 404's, its possible your API slashes are mismatched with your NGINX config. Be careful!

Getting it all started

Run yarn mkcert in your root directory
Run yarn build:docker
Run yarn start:docker
In a separate window, run docker-compose up
Go to https://localhost:3006 click through the security warnings about self signed certs

Meshify's scheme for interacting with a third party service

In our case, we forward requests matching the string 'periscope' to a standalone service that we made to handle some business logic. The standalone service could also be lambdas or some other endpoint that we own. In this instance we get to use the cookie from an authenticated user in that standalone service to make another request to the API to ensure that user has permissions to read what they are accessing.

My brilliant coworker Danil did most of the heavy lifting getting this setup working smoothly with NGINX. Kubernetes works especially well with this setup, COMMAND and ENVIRONMENT exists in Kubernetes configuration the same way it does here, so there are few changes necessary.

I encourage you to comment here if you have any trouble getting this running and wish you luck in breaking out of the cage that your browser puts you in!

Memoization, Generators, Virtualization, Oh my! Building a High Performance Directory Component in React

John Detlefs — Fri, 10 Jan 2020 22:54:08 +0000

Developers often pretend to know what they're doing, especially when they're insecure newer developers like myself! Sometimes we happen to stumble upon interesting patterns, think they're elegant, and get attached to them rather than using the solution that performs better. In the course of building a file directory, I gleaned some interesting insights into recursion, search, memoization, virtualization, and generator functions. The path getting there exposed me to concepts that I haven't really dealt with since my algorithms course in college. Fortunately, my first slow-but-elegant solution, a recursive react component, was supplanted by the use of generator functions in react-vtree, an equally interesting technology. Dealing with folder-based file systems has been one of the more rewarding small features I've had the opportunity to work in my short career.

The idea of a folder-based file system is a ubiquitous abstraction in software. A folder-based filesystem exists as a tree structure. Each folder either contains files which can be thought of as leaf nodes in the tree structure or folders that have the aforementioned folder as a parent.

A Glossary for Terms in this Post:

Tree ← A set of elements where each element has only one parent, which may be itself (called a root node). All paths to a root node are unique → Directory
Node ← Any element in the tree → Folder or File
Leaf ← Any node in the tree without children → *File

Viewing a set of folders in a directory reveals a clear hierarchy in that we can conditionally render children based on some a folder's particular ‘hide/show’ icon that handles click and keypress events.

In the course of building a new product for my employer, Meshify, we worked on building a Directory that could:

Search by folder or filename and highlight matched text from the search
Highlight a selected folder based on a url folderId parameter
Show and hide folder contents from click events
Be able to handle ~10K+ folders without crashing or being overly laggy.

I wish I could say that I knew what I was doing when I started working on this problem. The first two insights I had regarded how to store and pass folder data and how to search recursively across folders.

Each folder in the list contains a parent folder id. Using this relationship, the list can be iterated over to return a set of children belonging to that folder. We should only have to do this once, invalidating data only on changes to the list of folders. This is the perfect case for a lookup table and memoization. In my case, I decided on a Map data structure and the useMemo hook. It’s worth noting that the use of object and memoization tools from another framework can work as well.

While being sure to write meaningful tests on different mocked folder lists, I built out the functionality for creating a memoized map that recomputes data associated with
The code that I ended up setting on looks like the folder provider in this example Folder Provider.

If you want to take anything away from the code above, the most useful part in my mind me was this code snippet.

const childrenMatch = annotatedRoot.children
        .map(childContainsMatch)
        .some(Boolean); // same as .some(item => item == true)

A child of a folder can contain a match to search text such that if any folder matches the search text somewhere deep in the tree, all of the folders in the path between the root folders and that folder have the information requisite to display their contents. The folder might need to be open even when a folder does not match the search text provided. In the case that a folder contains other folders, we need to use recursion to search those child folders for any elements that match independent of that folder's depth.

By knowing that we are guaranteed a return when we reach a folder without any children (you can think of this as a file if that helps), we should avoid potential stack overflow errors. The array method Array.prototype.some in this context will exit as soon as it finds one true return from childContainsMatch.

Given this map, we are able to build out a Directory component that handles most of the work we need to do (in theory, more to be revealed).
Initially, the component I built look something like this:

Control Flow for Folder Component

Get Folder Info from Map given folder id
If the folder has children:
- If search text is present:
  - If this folder name matches search:
    - Render Name with highlighted search text, show/hide icon with event handlers
  - Else:
    - If this folder contains children that match or this folder is set to open:
      - Map across this folders children, return new Folder component for each
- Else:
  - Render name & show/hide icon with event handlers
  - If the folder is set to open:
    - Map across children, return new Folder component for each
Else (is a leaf node):
- If search text is present:
  - If name matches search:
    - Render file name with search highlighting
- Else:
  - Render file name

As you can see, in the case that a folder has children, the Folder component renders itself recursively! Some of you may not think that’s cool, but it’s the first time I've had a compelling need to use recursion with a React component and I think it’s damn cool.

Unfortunately, this scheme doesn't perform amazingly with large lists of folders. After some investigation, it was pretty clear that there weren't unnecessary re-renders or obviously slow performance issues in the FolderProvider component. The unfortunate truth was that, in some cases, we were simply rendering too many things at once. Without changing any backend APIs, the best solution seemed to be virtualization. After using Twitter to ask what the current state of virtualization was, I was made aware of react-window. Scrolling through the readme of react-window led me to react-vtree. The npm package "provides a lightweight and flexible solution for rendering large tree structures", just what I was looking for.

Would it surprise you if I told you that this added even more complexity to the problem?

react-vtree is a quick and practical introduction to the utility of generator functions, as well as virtualization. The core functionality of react-vtree lies in a treeWalker generator function that is taken as a prop.

// In the component enclosing the scope of the tree walker funciton
const { annotatedFolderMap, searchText } = useContext(FolderContext)

function * treeWalker(refresh) { 
   const stack = []
   rootFolders.forEach(folder => { 
      const data = annotatedFolderMap.get(folder.id)
      if (searchText !== "" && isVisible) {
         stack.push(data);
      } else {
         stack.push(folder)
      }
  })
  while (stack.length !== 0) {
     const currentFolder = stack.pop()
     const isOpen = yield refresh ? { currentFolderData } : id
     if (currentFolder.children.length > 0 && isOpen) {
        children.map(child => {  
           const data = annotatedFolderMap.get(currentFolder.id)
           if (searchText !== "" && isVisible) {
              stack.push(data);
           } else {
             if (searchText === "") {
                stack.push(data);
             }
           }
        })
     } 
   }
}

The function treeWalker here is an example of lazily computed values. The tree that consumes the treeWalker function, looks up the default state for whether the particular folder is open, call this variable defaultIsOpen. The tree then sends that data back to the treeWalker function through the line const {value, done} = iter.next(defaultIsOpen). The const isOpen in the while loop is set through that call to iter.next. No data is collected unless we are sure it is a member of an open directory or it is a root folder. It is worth noting that the tree walker function is not as lazy as it could be, in that data that is not being rendered can still be collected as a result of calling this generator. This generator function is called any time a node's is open state is changed via the provided toggle function.

react-vtree is build on top of react-window. react-window is a virtualization tool, which means that it only renders items that are viewable within your window. The savings are two-fold, less unneccessary data is saved and no unnecessary nodes are rendered. Of course, there is no longer the interesting use of recursion; one can take solace in the fact that this solution uses some of the most modern features of Javascript and the react ecosystem to suitably render thousands of folders blazingly fast.

Heres a gif of the final product:

In retrospect, the process of building this component mirrored the adage "make it work, make it pretty, and then make it fast". I wish I could say I knew what I was doing, but I happened to luckily stumble on a convenient separation of concerns. By separating the data concerns from the actual rendered view, the process of refactoring this work to go from using a bespoke, recursive tree component to a virtualized tree with react-vtree was remarkably painless.

Cant use the blessed approach just yet? A quick way to solve browser redirects after auth using react-router

John Detlefs — Mon, 25 Nov 2019 01:02:30 +0000

Kent Dodds delineates a pretty good approach to auth in a react app here: https://kentcdodds.com/blog/authentication-in-react-applications

If you're using react-router and useContext in your app just auth that way. Its very convenient to have AuthenticatedApp and UnauthenticatedApp as the sides of your ternary in that authentication check.

If for some reason you cant get this in as a ticket and you need to solve redirects quickly in an additive fashion (very little refactoring necessary) use location.state! Location state lets us pass data on a router action like history.push.

Assuming you already have redirecting unauthenticated users to an unauthenticated view, the method of fixing bad redirects generalizes to two parts

Find the react router redirect to prop or history.push action that is causing that redirect and refactor it to this.

Before

<Redirect to='login' /> (could also be) /> or thishistory.push({pathname: 'login', state: { redirectPathName: pathname, redirectSearchParams: search})

After

const { pathname, search } = useLocation()
// in route declaration
<Redirect to={{pathname: 'login', state: { redirectPathName: pathname, redirectSearchParams: search}} />

Find the login method that pushes to the new authenticated route.

Before

handleLogin(username, password) {
   const { push } = useHistory();
   // handle login stuff
   // onSuccess do this
   push('/')
}

After

handleLogin(username, password) {
   const { push } = useHistory();
   const { state } = useLocation();
   // handle login stuff
   // onSuccess do this
   push(state.redirectPathname ?  { pathname: state.redirectPathName, search: state.redirectSearchParams ?? '' }  : '/')
}

Your app should be using query/search params

John Detlefs — Sun, 24 Nov 2019 19:46:21 +0000

If you're a long term web dev who is thinking "Of course I should be using query params! I've been using them since before you were born.", feel free to read something else.

Like many other of my contemporaries, there are concepts we can go years without learning. The use of query parameters or search parameters in single page apps was one such thing for me, and hopefully some of you as well!

Query parameters / search parameters are a part of the official url standard. Take this url for example:
https://www.youtube.com/watch?v=y8EQFhj8ca4, the bit after the question mark v=y8EQFhj8ca4 is a query parameter. If you go to that page and investigate in the console.

let params = new URLSearchParams(document.location.search.substring(1))
params.get('v')

You'll get access to the id of that video, y8EQFhj8ca4. I haven't checked but I'd bet serious money that id is used to fetch data in the client-side javascript without the necessity of defining a route for every video. This is one valuable use case for query parameters.

The core realization to be made is this: Query params make your urls shareable.

Don't see what I mean? Take an instance like a paginated table, each page of the table corresponds to a unique fetch or set of fetches for that page's data. Each fetch takes a "limit" argument, the number of rows we are looking to fetch, and an "offset" argument, the index at which we want to start fetching the rows. Assume they are always ordered the same way for the purpose of this discussion.

In that past I have made the mistake of managing the current page number and rows per page via React's useState hook, and building that into a larger usePagination hook. A clear limitation of this approach would be that whenever we took a url while from viewing some page that was not on the default state, that state was not saved anywhere and could not be reconstructed.

By using query params, we can react to the url to set determine current limit state and offset values to be sent to our fetch. When changing a page, we can use a react-router 5.1 and some url parsing to get what we need.

// import react, react-router...
const { push } = useHistory();
const { pathname, search }  = useLocation();
const params = new URLSearchParams(search.substring(1))
// convert to numbers
const limit =  +params.get('limit')
const pageNumber = +params.get('pageNumber')
// fetch using offset on your own
const offset = limit * pageNumber
// an event handler you might pass to your table
const handlePageChange = (event, newPage) => {
    push({ pathname, search: `?limit=${limit}&pageNumber=${newPage}`})

By using routes and our router to our advantage we this approach means that when viewing page number 10 of some table, an end user can take this link, share it with a coworker, and they should get the same view. (Assuming the data hasn't changed in that time), either way, it creates a much better experience to the end user.

Functions and utilities used while learning this stuff

URLSearchParams
react-router, specifically:
useHistory from react-router
useLocation from react-router
the use-query-params package on npm
mui-datatables

I'm trying not to punish myself by creating the perfect blog, so if anything is unclear or if you have anything you want to add, feel free to comment. I'm sure there is stuff I haven't brought up that is relevant!