Forem: Kevan Y

Working with Next.js + MUI (edit)

Kevan Y — Sat, 23 Apr 2022 03:21:36 +0000

Intro

This week I worked on the front-end part of telescope. I didn't touch that much on the front-end in telescope, but I had experience with Next.js before. The Issue I will be working on is to implement a UI for the dependency-discovery services.

Planning / Implementation

They were no designs pre-planned for this issue. I had to make a simple, fast, and functional design.
Before getting into the design we have to understand the dependency-discovery API, we need to know all the possible route and what kind of data each route return to make use of all our API data as much as possible.

`/projects`

This route returns an array of string of the list of the dependencies we use in telescope repo.

Samples of data

[
  "@algolia/client-common",
  "@babel/helpers",
  "react",
]

`/projects/:namespace/:name?`

This route returns the general information of the dependency. It's an object with id as string, license as string, and 'gitRepository' as object. gitRepository has a type as string, url as string, directory as string, and issuesUrl as string.

Samples of data

{
  "id": "@babel/core",
  "license": "MIT",
  "gitRepository": {
    "type": "git",
    "url": "https://github.com/babel/babel",
    "directory": "packages/babel-core",
    "issuesUrl": "https://github.com/babel/babel/issues?q=is%3Aopen+is%3Aissue+label%3A%22hacktoberfest%22%2C%22good+first+issue%22%2C%22help+wanted%22"
  }
}

`/github/:namespace/:name?`

This route returns an array of issue label hacktoberfest, Help wanted, and good first issue of the dependency. Each object has a htmlUrl as string, title as string, body as string, and createdAt as string.

Samples of data

[
  {
    "htmlUrl": "https://github.com/babel/babel/issues/7357",
    "title": "injecting external-helpers in a node app",
    "body": "<!---\r\nThanks for filing an issue 😄 ! Before you submit, please read the following:\r\n\r\nSearch open/closed issues before submitting since someone might have asked the same thing before!\r\n\r\nIf you have a support request or question please submit them to one of this resources:\r\n\r\n* Slack Community: https://slack.babeljs.io/\r\n* StackOverflow: http://stackoverflow.com/questions/tagged/babeljs using the tag `babeljs`\r\n* Also have a look at the readme for more information on how to get support:\r\n  https://github.com/babel/babel/blob/master/README.md\r\n\r\nIssues on GitHub are only related to problems of Babel itself and we cannot answer \r\nsupport questions here.\r\n-->\r\n\r\nChoose one: is this a bug report or feature request? (docs?) bug report\r\n\r\nI'm trying to use a package that assumes that the external-helpers are available as a global. From the [docs](https://babeljs.io/docs/plugins/external-helpers/#injecting-the-external-helpers), I should be able to inject them to `global` in my node app by using `require(\"babel-core\").buildExternalHelpers();`. However, use of that still results in the following error: `ReferenceError: babelHelpers is not defined`\r\n\r\n### Babel/Babylon Configuration (.babelrc, package.json, cli command)\r\nSince the `buildExternalHelpers()` function needs to run before the package is imported and my app uses es module imports, I'm using a bootstrap file as an entry point that is ignored from transpilation and just tries to inject the helpers before loading the actual app:\r\n\r\n```

js\r\nrequire(\"babel-core\").buildExternalHelpers();\r\nconst app = require('./app');\r\n

```\r\n\r\n### Expected Behavior\r\n\r\n`babelHelpers` should be added to `global` so that it is available for the package that assumes it is available there.\r\n\r\nfrom the docs:\r\n> This injects the external helpers into `global`.\r\n\r\n### Current Behavior\r\n\r\n`babelHelpers` is not made available on `global`, resulting in `ReferenceError: babelHelpers is not defined`\r\n\r\n### Possible Solution\r\n\r\nThe docs also [mention](https://babeljs.io/docs/plugins/external-helpers/#getting-the-external-helpers) generating a helpers file with `./node_modules/.bin/babel-external-helpers [options] > helpers.js`. It wasn't obvious to me that this file could be imported to accomplish the same goal as `buildExternalHelpers()` until I started reading the source of that file. Importing that file instead does work for my app. I'll need this file elsewhere, so I'll likely just continue importing that instead, even if there is a way to use `buildExternalHelpers()`.\r\n\r\nWith that approach, my bootstrap file has the following contents instead:\r\n\r\n```

js\r\nrequire('../../vendor/babel-helpers');\r\nconst app = require('./app');\r\n

```\r\n\r\n### Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\n\r\n| software         | version(s)\r\n| ---------------- | -------\r\n| Babel            | 6.26.0\r\n| node             | 8.9.4\r\n| npm              | 5.6.0\r\n| Operating System | macOS High Sierra \r\n\r\n### Forum\r\n\r\nWhile I was still trying to find a working solution, I was trying to find the best place to ask questions. The website still links to a discourse forum that no longer seems to exist. It'd be a good idea to either remove the link from the site or at least have it link to somewhere that describes the currently recommended approach for getting that type of help.\r\n",
    "createdAt": "2018-02-08T20:49:23Z"
  }
]

Now we know what the API return, let's makes a draft design. For simplicity, I'm gonna draw it by hand.

I started to look at what MUI component I could use for this.

Icons - link
Table - link
Collapse - link

After that, I need to plan how I will structure my code and what to add/modify.

1 - Create a route at /dependencies in src\web\app\src\pages in Next.js the name of the file is our route name.
The page should follow the other page in telescope which means it has an SEO, and Navbar component. Also, we need to add our DependenciesPage component which is our dependencies page.
I was thinking to use getStaticProps + revalidate features from Next.Js to make a static page. But since our API services need to be run at the time when Next.Js builds all static HTML, so we might need to modify our docker-compose to run our dependency-discovery services first then after our services it's up we can build our static HTML. For simplicity, I decided to just use useEffect to fetch the data.

import SEO from '../components/SEO';
import NavBar from '../components/NavBar';
import DependenciesPage from '../components/DependenciesPage';

const dependencies = () => {
  return (
    <div>
      <SEO pageTitle="Dependencies | Telescope" />
      <NavBar />
      <DependenciesPage />
    </div>
  );
};

export default dependencies;

2 - Add a new icon to redirect into our new route in the navbar src\web\app\src\components\NavBar\index.tsx

import { FiPackage } from 'react-icons/fi';

const iconProps: NavBarIconProps[] = [
 {
  ...
 },
 {
    href: '/dependencies',
    title: 'Dependencies',
    ariaLabel: 'Dependencies',
    Icon: FiPackage,
  },
]

3 - Set our dependencyDiscoveryUrl env.
In docker-compose docker\docker-compose.yml, we need forward DEPENDENCY_DISCOVERY_URL in the build args.

services:
  nginx:
    build:
      context: ../src/web
      dockerfile: Dockerfile
      cache_from:
        - docker.cdot.systems/nginx:buildcache
      # next.js needs build-time access to a number of API URL values, forward as ARGs
      args:
        ...
        - DEPENDENCY_DISCOVERY_URL

We also need to modify the Dockerfile in src\web\Dockerfile to add DEPENDENCY_DISCOVERY_URL as build args.

ARG DEPENDENCY_DISCOVERY_URL
ENV NEXT_PUBLIC_DEPENDENCY_DISCOVERY_URL ${DEPENDENCY_DISCOVERY_URL}

Now we need to forward that env to be accessible in Next.Js. We will need to modify src\web\app\next.config.js

const envVarsToForward = [
 ...,
 'DEPENDENCY_DISCOVERY_URL',
]

4 - Create our DependenciesPage components in src\web\app\src\components (Because it contains a lot of lines of code I'm just putting some parts. Read more)
Our DependenciesPage components should have a useEffect that runs once on onMount to fetch our dependencies at route /projects. As JSX, it will have our Page title, and a DependenciesTable component which is our table of dependencies, it has a dependencies(List of the dependencies) props.
We also need some style to make our page responsive and adjust color in light/dark mode.

import { dependencyDiscoveryUrl } from '../config';

import { makeStyles } from '@material-ui/core/styles';
import { useState } from 'react';

const useStyles = makeStyles((theme) => ({
  root: {
    backgroundColor: theme.palette.background.default,
    fontFamily: 'Spartan',
    padding: '1em 0 2em 0',
    paddingTop: 'env(safe-area-inset-top)',
    wordWrap: 'break-word',
    [theme.breakpoints.down(1024)]: {
      maxWidth: 'none',
    },
    '& h1': {
      color: theme.palette.text.secondary,
      fontSize: 24,
      transition: 'color 1s',
      marginTop: 0,
    },
    '& p, blockquote': {
      color: theme.palette.text.primary,
      fontSize: 16,
      margin: 0,
    },
  },
  container: {
    padding: '2vh 18vw',
    [theme.breakpoints.down(1024)]: {
      padding: '2vh 8vw',
      wordWrap: 'break-word',
    },
  },
}));
const DependenciesPage = () => {
   const [dependencies, setDependencies] = useState<string[]>();
   const classes = useStyles();

   useEffect(() => {
    (async () => {
      try {
        const fetchDependenciesData = await fetch(`${dependencyDiscoveryUrl}/projects`);
        setDependencies(await fetchDependenciesData.json());
      } catch (e) {
        console.error('Error Fetching Dependencies', { e });
      }
    })();
  });

  return (
    <div className={classes.root}>
      <div className={classes.container}>
        <h1>Dependencies</h1>
        <DependenciesTable dependencies={dependencies} />
      </div>
    </div>
  );
};

export default DependenciesPage;

5 - Create DependenciesTable component in src\web\app\src\components\DependenciesTable\index.tsx (Because it contains a lot of lines of code I'm just putting some parts. Read more). This would be the component that contains our table, search bar, and table navigation. We can get our dependency list from the props dependencies. Create a function to update the dependency list based on the search query.
Set the limit of the rows per page to 15.
We also need to add some style to match our drawing design and adjust color for light/dark mode.

type DependenciesTableProps = {
  dependencies: string[];
};

const DependenciesTable = ({ dependencies }: DependenciesTableProps) => {
  const classes = useStyles();
  const [page, setPage] = useState(0);
  const rowsPerPage = 15; // Set 15 element per page
  const [searchField, setSearchField] = useState('');

  // Compute dependencyList based on search query
  const dependencyList = useMemo(() => {
    setPage(0);
    if (!searchField) return dependencies;
    return dependencies.filter((dependency: string) => {
      return dependency.toLowerCase().includes(searchField.toLowerCase());
    });
  }, [dependencies, searchField]);

  return (
    <>
      <SearchInput text={searchField} setText={setSearchField} labelFor="Browse for a dependency" />

      <TableContainer>
        <Table sx={{ minWidth: 450 }} aria-label="custom pagination table">
          <TableBody>
            {dependencyList
              .slice(page * rowsPerPage, page * rowsPerPage + rowsPerPage)
              .map((dependency) => {
                return <Row key={dependency} dependency={dependency} />;
              })}
          </TableBody>
        </Table>
        <TablePagination
          className={classes.root}
          rowsPerPageOptions={[]}
          component="div"
          count={dependencyList.length}
          rowsPerPage={rowsPerPage}
          page={page}
          onPageChange={handleChangePage}
        />
      </TableContainer>
    </>
  );
};

export default DependenciesTable;

6 - Create Row component in src\web\app\src\components\DependenciesTable\Row.tsx (Read more). This content each of our row with the collapse. We have some useEffect waiting on a state called open (state for the collapsed component) to be changed to trigger the fetch for dependency information. For fetch GitHub issues, add more checks to see if API returns 403 which means API limit reached, if 403 we need to show a message saying Github API reached a limit, please use the link directly <Dependency name>.
We also need to add some style to match our drawing design and adjust color for light/dark mode.

7 - Testing part :

Making sure our design is responsive,
Show the right color in light/dark mode.
Working search bar.
Working pagination.
Working collapse and data fetch on collapse open.

Pull request reviews

Feedback from @humphd:

Use SWR instead of using useEffect for fetching
Add a paragraph to explain what is it after the title.
Add a spinner when content is loading.
Fix color, and font size issues.

Feedback from @joelazwar:

Reset item number to 1 when we search for something

Feedback from @DukeManh:

Use the Default SWR fetcher instead of creating one.
Rename some functions.

Final products

Once the code is merged the staging environment will ship this feature first.
On release 3.0.0, it will be shipped to production environment
Staging: https://dev.telescope.cdot.systems/dependencies/
Production: https://telescope.cdot.systems/dependencies/

Hard bug fixed by a simple fix

Kevan Y — Mon, 18 Apr 2022 03:16:07 +0000

Opened my Pull request on adding best practices for the SSO services. The e2e test was failing in the pipeline, despised testing locally.
On the CI, the test case was only failing on Chromium.

FAIL browser: chromium src/api/sso/test/e2e/auth-flows.test.js (28.142 s)
  Authentication Flows
    ✕ Login flow preserves state param (25222 ms)

First, some of my reviewers are using Mac M1 (ARM architecture), they were not able to build the image properly since the node-alpine version because it was missing some build-in package to build the image. I had to use a bigger node version.
After pushing those changes, @humphd helped me to debug, this issue wasn't able to be producible locally.

I started to run on different environment, and this is the result showed below:
Windows 11 pro 21H2 - test case pass. (by @Kevan-Y )
Windows 11 pro 21H2 on WSL2 Ubuntu-20.04 - test case pass (by @Kevan-Y )
Amazon Linux 2 AMI (HVM) - Kernel 5.10 (ami-0c02fb55956c7d316 (64-bit x86)) - t2.large - test case pass (by @Kevan-Y )
MacOS M1 - test case pass (by @humphd)

It wasn't looking good. This makes debugging hard since it's only happening on CI.

I started to increase the timeout of the test case to 30000ms

it('Login flow preserves state param', async () => {
    const { state } = await login(page, 'user1', 'user1pass');
    // Expect the state to match what we sent originally (see index.html <a>)
    expect(state).toEqual('abc123');
  }, 30000);

Despite increasing timeout it was failing.

Possibility due to some other container such as elastic search, it crashed, since that container had some issue with the memory not enough allocated. I increased the memory in the docker-compose file.

  elasticsearch:
    environment:
      # Limit the initial heap size. By default it will use 1/4 of available RAM
      - 'ES_JAVA_OPTS=-Xms512m -Xmx512m'

After pushing these changes, the CI pipeline was still failing.

I decided to skip all the e2e test cases and only run Login flow preserves state param of src/api/sso/test/e2e/auth-flows.test.js for the purpose of debugging and also turn on LOG_LEVEL to debug.
I pushed those changes, but it wasn't helping. They were 0 logs that helped to identify which place it when wrong. So I switch to the basic way of debugging. "Put console.log on each line"

In src\api\sso\test\e2e\auth-flows.test.js

 it('Login flow preserves state param', async () => {
    console.log('Start test case by await login');
    const { state } = await login(page, 'user1', 'user1pass');
    console.log('Done await login', state);

    // Expect the state to match what we sent originally (see index.html <a>)
    expect(state).toEqual('abc123');
  }, 30000);

In src\api\sso\test\e2e\browser-util.js

const login = async (page, username, password) => {
  console.log('Start login and wait for page');
  // Click login button and then wait for the login page to load
  await Promise.all([
    page.waitForNavigation({
      url: /simplesaml\/module\.php\/core\/loginuserpass\.php/,
      waitUtil: 'networkidle',
    }),
    page.click('#login'),
  ]);
  console.log('after click login');

  console.log('click input[name="username"]');
  // Fill the login form, star with username
  await page.click('input[name="username"]');
  console.log('after click input[name="username"]');

  console.log('fill input[name="username"]');
  await page.fill('input[name="username"]', username);
  console.log('after fill input[name="username"]');

  // Now enter the password
  console.log('click input[name="password"]');
  await page.click('input[name="password"]');
  console.log('after click input[name="password"]');

  console.log('fill input[name="password"]');
  await page.fill('input[name="password"]', password);
  console.log('after fill input[name="password"]');

  console.log('wait for the new page to load');
  // Click login button and then wait for the new page to load
  await Promise.all([
    page.waitForNavigation({
      url: /^http:\/\/localhost:\d+\/auth\.html\?access_token=[^&]+&state=/,
      waitUtil: 'load',
    }),
    page.click('text=/.*Login.*/'),
  ]);

  // The token and state will get returned on the query string
  return getTokenAndState(page);
};

After pushing this change, I finally got some good debug result. The test case was timing out in

await Promise.all([
    page.waitForNavigation({
      url: /simplesaml\/module\.php\/core\/loginuserpass\.php/,
      waitUtil: 'networkidle',
    }),
    page.click('#login'),
  ]);

One possible issue mentioned by @Humphd, the event was sitting on that page for some reason and never navigates.
Instead of waitUtil: 'networkidle', we change to waitUtil: 'load'. see reference
Even with that, it didn't work. But the ultimate fix was to try to update from version 1.19.2 to 1.20.2 of playwright which might potentially fix it.
It was the right solution, just an update of the dependencies, and everything works fine.

Spend a couple of days trying to debug this, and the fix was just a dependency update. It was a good experience, in the end. Sometimes we have to wait for the upstream to fix for us.

Discovered new tools.

Kevan Y — Thu, 07 Apr 2022 00:29:03 +0000

Oauth2-proxy

It's a reverse proxy and static files server that provides authentication in the middle of a user request.
In telescope, I had to use oauth2-proxy and nginx to secure the route for Supabase studio.
Add studio under our subdomain #3098

Process

Configure our docker-compose for staging/production

# production.yml
  nginx:
    depends_on:
      - oauth2-proxy

  oauth2-proxy:
    image: bitnami/oauth2-proxy:7.2.1
    container_name: 'oauth2-proxy'
    command:
      [
        '--provider=github',
        '--cookie-secure=true',
        '--cookie-secret=${OAUTH2_SUPABASE_COOKIE_SECRET}',
        '--upstream=http://studio:3000',
        '--http-address=0.0.0.0:8080',
        '--reverse-proxy=true',
        '--email-domain=*',
        '--github-org=Seneca-CDOT',
        '--github-team=telescope-admins',
        '--client-id=${OAUTH2_SUPABASE_CLIENT_ID}',
        '--client-secret=${OAUTH2_SUPABASE_CLIENT_SECRET}',
      ]
    depends_on:
      - studio

Set nginx services to depend on oauth2-proxy. So we let oauth2-proxy start first before starting nginx.
Declare our oauth2-proxy services to use image bitnami/oauth2-proxy:7.2.1, we give the following command.
- --provider=github, this tell to use GitHub Oauth services see.
- --cookie-secure=true, Set our cookie to be secure.
- --cookie-secret=${OAUTH2_SUPABASE_COOKIE_SECRET}, set a long seed string for secure cookies.
- --upstream=http://studio:3000, set the upstream to be redirected to after authorized, in our case it would be Supabase studio.
- --http-address=0.0.0.0:8080, listen on for HTTP clients.
- --reverse-proxy=true, set reverse-proxy to be true since we are putting our oauth2-proxy under nginx. It helps to automatically set a redirect route.
- --email-domain=*, this is for restricting certain email domains, but for us, we allow any domain.
- --github-org=Seneca-CDOT, this is for restricting by GitHub organization, for use we allow only member that is in Seneca-CDOT.
- --github-team=telescope-admins, this is for restricting within an organization to specific teams, for use we allow only member that is in Seneca-CDOT organization and telescope-admins teams.
- --client-id=${OAUTH2_SUPABASE_CLIENT_ID}, this is our GitHub Oauth client id see.
- --client-secret=${OAUTH2_SUPABASE_CLIENT_SECRET}, this is our GitHub Oauth client secret see.

Configure our nginx config for staging/production

# nginx.conf.template
  server {
    listen 443 ssl http2;
    server_name ${SUPABASE_HOST};

    location / {
      proxy_set_header Connection "";
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Scheme $scheme;
      proxy_pass http://oauth2-proxy:8080/;
    }
  }

In this PR #2964, we added the DNS and SSL.

In our nginx config, we declare a new server with a server name of our ${SUPABASE_HOST} and listen to port 443.
Set the location to be /, set the proxy to be our oauth2-proxy services.
Set the the proxy header proxy_set_header, proxy_set_header, proxy_set_header, this will give our oauth2-proxy the right information to set up the redirect route.

Hurl

Hurl is an open-source command-line tool that runs HTTP requests defined in a simple plain text format.
The sources code is written in Rust which makes this tool very fast to run.
It's made by a French Telecom company called Orange,

Linux installation

curl -LO https://github.com/Orange-OpenSource/hurl/releases/download/1.6.1/hurl_1.6.1_amd64.deb
sudo dpkg -i hurl_1.6.1_amd64.deb

To check if it's installed

hurl -h

Scenario to test

Note: any POST, PUT, DELETE, and GET request needs user to be authenticated. For that let's assume our API uses basic auth and the username is user1@email.com and the password is password1. For the authentication, we use a base64 which can be generated with the following code below.

console.log(btoa('user1@email.com:password1'))
# Output: dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

POST sending a file, it should return a status code of 201, header Locations of the URL to get the data, and JSON body containing an id based on nanoid, create date in string, update date in string
GET at URL of the location previously returned in POST, and compare if the data returned matches the content posted.
PUT at URL of the location previously returned in POST, and update the content, it should return a status code of 200, and JSON body containing an id based on nanoid, create date in string, newest update date in string (should be greater than create date).
GET at URL of the location previously returned in POST, and compare if the data returned matches the content updated.
DELETE at URL of the location previously returned in POST, it should return a status code of 200

Folder structure

.
├── .github
│   ├── workflows
│   │   └── ci.yml
├── tests
│   ├── fixtures
│   │   └── data.txt
│   ├── integration
│   │   ├── myTest.hurl
├── docker
│   ├── docker-compose.yml
├── Dockerfile

Test case file

# myTest.hurl

# 1. POST request 
# Make a POST and send our txt file located in the fixtures folder
# Set the content type to be a text/plain
POST http://localhost:8080/cat
Content-Type: text/plain
# https://hurl.dev/docs/request.html#file-body
file,./fixtures/data.txt;
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 201
HTTP/1.1 201

# https://hurl.dev/docs/asserting-response.html
[Asserts]
# Assert header Location to contain the URL for getting our data
header "Location" matches "^http:\/\/localhost:8080\/cat\/[A-Za-z0-9_-]+$"

# Assert id to matches id generated with nanoid https://www.npmjs.com/package/nanoid
jsonpath "$.id" matches "^[A-Za-z0-9_-]+$"

# Assert created to return a string
jsonpath "$.created" isString

# Assert created to return a string
jsonpath "$.updated" isString

# https://hurl.dev/docs/capturing-response.html#captures
[Captures]
# Capture the header location value and store to url
# This value can be call later in the code.
url: header "Location"
# Capture the id from the returned json and store cat_id
cat_id: jsonpath "$.id"
# Capture the created date from the returned json and store cat_created
cat_created: jsonpath "$.created"
# Capture the updated date from the returned json and store cat_updated
cat_updated: jsonpath "$.updated"

# 2. GET request 
# GET at the url previously captured
GET {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 200
HTTP/1.1 200
# Check the response content type to be text/plain
Content-Type: text/plain
# Check the response body to match the content of data.txt.
file,./fixtures/data.txt;

# 3. PUT request
# PUT at the url previously captured
PUT {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==
Content-Type: text/plain
# Set the request body with our new content 
# https://hurl.dev/docs/request.html#raw-string-body
```New content```

# Check the response to be 200
HTTP/1.1 200

[Asserts]
# Assert id to be the same as cat_id (id captured in the POST return)
jsonpath "$.id" == {{cat_id}}

# Assert created to be the same as cat_created (created captured in the POST return)
jsonpath "$.created" == {{cat_created}}

# Assert updated to be different as cat_updated (updated captured in the POST return)
jsonpath "$.updated" != {{cat_updated}}

# 4. GET request 
# GET at the url previously captured
GET {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 200
HTTP/1.1 200
# Check the response content type to be text/plain
Content-Type: text/plain
# Check the response body to match the new content updated from the PUT request
```New content```

# 5. DELETE request 
# DELETE at the url previously captured
DELETE {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 200
HTTP/1.1 200

To run the test case. See how to run and command option

Let's assume we are in the root of our folder.
Let's assume we have a docker-compose.yml that is already running our microservices and exposes at port 8080.

hurl --test --file-root "./tests" "./tests/integration/myTest.hurl"

--test, activate the test mode which enables --no-output, --progress, and --summary
--file-root set our file root, we need this because we want our fixture to be accessible.

Samples of response

❯ hurl --test --file-root "./tests" "./tests/integration/myTest.hurl"
./tests/integration/myTest.hurl: RUNNING [1/1]
./tests/integration/myTest.hurl: SUCCESS
--------------------------------------------------------------------------------
Executed:  1
Succeeded: 1 (100.0%)
Failed:    0 (0.0%)
Duration:  10ms

Run on CI pipeline

Let's assume we have a docker-compose.yml that spins up our microservices and exposes at port 8080.

# .github/workflows/ci.yml
name: ci
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

jobs:
  hurl-test:
  runs-on: ubuntu-latest
  steps:
    - name: 🏗 Setup repo
      uses: actions/checkout@v2
    - name: 📦 Install hurl
      run: |
        curl -LO https://github.com/Orange-OpenSource/hurl/releases/download/1.6.1/hurl_1.6.1_amd64.deb
        sudo dpkg -i hurl_1.6.1_amd64.deb
    - name: 🏗 Build Containers
      run: docker-compose -f docker/docker-compose.yml up -d
    - name: ✨ Run Hurl Tests
      run: hurl --test --file-root "./tests" "tests/integration/myTest.hurl"

Opinion about this new testing tool.

In overall, this is a pretty unique testing tool, it's super fast and powerful since the code base is written in rust. The learning curves are pretty simple (I managed to learn the basics in a couple of hours), less setup todo since it's just a plain file, the syntax is English friendly, besides the jsonPath that could take some times to adapt.

There is some limitation, like for the assert, sometimes it's not that easy to compare an object to an object or read from an object since when you read a capture of an object, it tries to construct into a string so you are not able to use jsonPath to get some property.

Thanks to @humphd who showed us this tool. And I'm happy because it's a French company that made this wonderful open-source tool.

Release 2.9

Kevan Y — Mon, 04 Apr 2022 02:23:51 +0000

My Work

Apply Dockerfile best practices from Snyk recommendations #1668
List of Dockerfile worked:

In overall it was pretty straightforward once one of the PR got reviewed, and I can use it as a template.

One of the process steps

References

For the image version we need to use an image to a specific version(Not using alpine version for package installation since M1 required more tools to work)

 - FROM node:lts as base
 + FROM node:16 as base

Setup multi staging

 + FROM node:16 as base
 + FROM base as dependencies
 + FROM node:16-alpine3.15 as deploy

Base stage is for installing the tool needed for package installation.
Dependencies stage is for installing node_modules.
Deploy stage is where we copy sources code from the build context and node_modules from Dependencies and run our services.

Run node app as node user

+ COPY --chown=node:node . .
+ USER node

This allows Node user to have permission to read our sources code.

Add a healthcheck

+ ENV PORT
+ HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
+   CMD wget --no-verbose --tries=1 --spider localhost:${DEPENDENCY_DISCOVERY_PORT}/healthcheck || exit 1

This is the change I made, one thing @humphd noticed building the application with a node-alpine image, on ARM architecture, it's required to use a bigger node image to be able to build it. See PR 3336

Release 2.9 rethinking about my goal

Kevan Y — Mon, 21 Mar 2022 15:58:35 +0000

Which part of Telescope 3.0 do I own?

In the past 2 weeks in telescope, I have been working in Docker, nginx, and oauth. So I guess I'm gonna own those parts.

Which existing Issues are part of this? Which new Issues do I need to file in order to properly describe the work? When will those issues be filed?

Add full OAuth2 capabilities to SSO service #3088, new issue, after adding oauth to Portainer, we found that it's not convenient to create 2 different oauth github app for staging and development, also in the future if we want to secure more route with oauth2-proxy, we will have to create more oauth github app. This issue will solve that, by enabling full OAuth2 capabilities in our SSO.
Apply Dockerfile best practices from Snyk recommendations #1668, undergoing, I started to refactoring some Dockerfile already and will keep refactoring the rest for 2.9 release.
Refactor all our dockerfile to use our base image. #3278, new issue, in #2803 we created a base to be used in all other of dockerfile.
Expose supabase studio under nginx #3033 - undergoing, waiting on #2979, once the secret thing is done. I can push my local change into it.
See if our nginx configs make any of these common mistakes #3029, new issue, this issue for check if we setup our nginx config correctly.

Who can I depend on for support in development, debugging, testing, and reviews? I can't do this alone.

@TDDR, is our Docker guy, along @manekenpix, and @humphd who knows most of the telescope part and provide a lot of help.

What are the risks that I see, which could prevent me from getting this shipped?

I think time is a big issue. With my course load specially PRJ class + job hunting + doing research and preparing the document to be able to work in Canada is starting, my schedule is getting filled quickly. I started 2 different side projects at the beginning of the semester utilizing web3, nextjs, and redux, but had to pause them because not enough time.

How will I mitigate these risks?

I think to mitigate those risks. It will be great to find more contributors to have an interest in oauth part and docker. Working as a group is much faster than working alone.

Telescope 2.8 release

Kevan Y — Tue, 15 Mar 2022 02:55:50 +0000

Release 2.8 was one of the biggest releases. Telescope set a new record with 99 tasks completed. So close to 100, but we will eventually make it one day.

What I did:

Use oauth2-proxy with GitHub Auth Provider to secure Admin apps
#3030 - This issue was to use oauth2-proxy to be our proxy with auth, to make it simple we want to secure some route, and only be accessible once authorize.
For this issue, we wanted to hide Portainer under oauth2-proxy.
I spend a couple of days doing research and trying oauth2-proxy in my local. I followed some sample of code https://github.com/oauth2-proxy/oauth2-proxy/blob/master/contrib/local-environment/docker-compose.yaml and run it locally. Instead of using oidc I switched to GitHub.

  oauth2-proxy:
    image: bitnami/oauth2-proxy:7.2.1
    container_name: 'oauth2-proxy'
    command:
      [
        '--provider=github',
        '--cookie-secure=false',
        '--cookie-secret=<Long_cookie>',
        '--upstream=<Redirect_to_after_login>',
        '--http-address=0.0.0.0:8080',
        '--skip-auth-regex=/forms/*',
        '--redirect-url=http://localhost:8085/oauth2/callback',
        '--email-domain=*',
        '--client-id=<Github_client_id>',
        '--client-secret=<Github_secret_id>',
      ]
    ports:
      - 8085:8080

I also need to setup GitHub oauth app. Followed this instruction.
https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app
Once I have oauth2-proxy working locally it was time to head in telescope repo and implement this.

One issue I noticed is that Portainer doesn't have anymore an option to enable no-auth after V.1. So we will have to login twice once in our oauth2-proxy and another one in Portainer which is what we don't want.
I had to find another option to implement oauth with Github provider in Portainer. After a bit of documentation reading. I found out that Portainer has a custom oauth and can be setup with Github. see
It was time to setup custom oauth for Portainer. First I needed @humphd to setup Github oauth app in Seneca-CDOT, after that, I just had to put the correct information following this image below.

How the custom oauth in Portainer works, only the default admin can login through Internal authentication. All other users will need to login through Github.
To give access to a GitHub User, you will need to add a new user and match the username to their Github username. Once that is created, user can login with their Github account.

Login flow

Go to the portainer URL
Login with Github
After authorize Conclusion of this, this issue what something simple but it required a bit of research. At the end I didn't implement oauth2-proxy but it's fine this research will help in securing route for supabase.

Set up Authorization tokens for CI #3
- This issue was to setup basic auth for our new Docker registry with docker_auth.
The documentation was just enough, I found it hard to setup since there is not a lot of sample code.

From reading from the docs we need a auth_config.yml.

server:
  addr: ':5001' # Address to listen to
  certificate: '/etc/letsencrypt/live/docker.cdot.systems/fullchain.pem' # TLS Certificate
  key: '/etc/letsencrypt/live/docker.cdot.systems/privkey.pem' # TLS Key

token:
  issuer: 'Docker auth issuer' # Issuer, must match issuer in the Registry config.
  expiration: 900

users:
  # Password is specified as a BCrypt hash. Use htpasswd to generate.
  'admin': # Setup an Admin account
    password: '' 
  'telescope-ci': # Setup an telescope-ci account for our CI
    password: '' 
  '': {} # Setup account that doesn't required password


acl:
  - match: { account: 'admin' } # Setup rule for admin to have full permission
    actions: ['*']
    comment: 'Admin has full access to everything.'
  - match: { account: 'telescope-ci' } # Setup rule for telescope-ci to have push and pull access.
    actions: ['push', 'pull']
    comment: 'Telescope CI has push and pull access'
  - match: { account: '' } # Setup rule for anyone else to have pull access.
    actions: ['pull']
    comment: 'Any user has pull access'

After this I have @TDDR, helped me out to setup the config.yml for the registry based on some of the comments I put in the Pull Request.

version: 0.1

#Log
log:
  level: info 
  fields:
    service: registry

# Storage 
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /mnt/docker0storage/registry

# Address to listen to
http:
  addr: ':5000'

# Our auth
auth:
  token:
    issuer: 'Docker auth issuer' # Issuer, must match issuer in the auth config.
    realm: 'https://docker.cdot.systems/auth' # Our auth path
    service: 'Docker registry'
    rootcertbundle: /etc/letsencrypt/live/docker.cdot.systems/fullchain.pem # TLS Certificate

After with the help of @humphd we added our docker_auth in our docker-compose.yml.

  docker_auth:
    image: cesanta/docker_auth
    volumes:
      - ./config/docker_auth:/config:ro
      - /var/log/docker_auth:/logs
      - /etc/letsencrypt:/etc/letsencrypt
    command: /config/auth_config.yml
    restart: always

I also had to look at the issue created in docker_auth repo to see if someone showed some steps to setup docker_auth under nginx. Thankfully to this https://github.com/cesanta/docker_auth/issues/184. I was able to setup nginx for docker_auth.
I just have to add this under our nginx.conf

    location /auth {
      proxy_pass                          https://docker_auth:5001/auth;
      proxy_ssl_trusted_certificate       /etc/letsencrypt/live/docker.cdot.systems/fullchain.pem;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
    }

To test if that works. I had a call with @humphd to test directly in the server at the same time fix stuff that wasn't working.

In issues, I wasn't the only one contributing to it. With the help of @humphd and @TDDR, we were able to implement this feature.

Telescope 2.8 planning

Kevan Y — Sun, 27 Feb 2022 01:35:33 +0000

For Telescope 2.8 releases. We decided to host our own docker, first decided to move our repo during the study week to another organization to use the tier for hosting our docker with GitHub, we also tried to option to email Docker to get the free tier for open sources projects. But @humphd hasn't heard back from Docker. Lucky thanks to Chris, we were able to have our own server for telescope.
There is several small tasks.

I assigned myself to the registry config, nginx config, outh2-proxy config, and GitHub Provider config with team, token access. I think it good enough issue to work on during the study week.
I'm also parallel working on Issue #3030, this issue is similar to setup docker with outh2-proxy but instead of docker I need to setup for portainer.
The only Pull request I did for this week was Pull Request #3041, to setup i18n for docusaurus.
This week I didn't do much but next week is where the fun start.

Biggest release on telescope

Kevan Y — Sun, 20 Feb 2022 16:18:24 +0000

This week, telescope has its biggest release more than 80 items. We also reached our 3000 created issues on telescope. I'm still a small contributor to telescope but I'm happy that I'm being part of this achievement.

What I have done for release 2.7

Issue #2791 - Fixed by Pull Request #2931. This issue looks simple with a bunch of information to guide me through but during the process, it was harder than I thought. I end up just landing kong to be rerouted with treafik to v1/supabase. Thanks for @humphd helping me through this issue otherwise I will be stuck for eternity. I guess this time I reached my limit of what I can do. But since this is resolved and learned a bunch of new things. Now I have a new limit yet to be discovered. There is a follow-up issue to use nginx to expose supabase studio. The issue I uncounted is that supabase studio is exposed but the static assets not fetching in the correct route.
Issue #2803 - Fixed by Pull Request #2821, this issue was about creating a base image that all our microservices can use to set up the base environment. This was a good issue to learn docker, with its basic CLI.
Issue #2732 - Fixed by Pull Request #2933, Pull Request #2954, and Pull Request #2986. This issue was more reach + implementation how to ship our mobile app. We decided at the moment to ship into Expo for financial purposes since to create a developer account in app store or play store you need to pay a certain amount of fees each year. Later on, we discovered that IOS users would not able to use the app once hosted to Expo due to some Apple policy. So our app end up scoped to android. I set up the CI for the release for mobile based on Expo GitHub Action. It was first simple but uncounted some issue with the authentication. The secret token stored in our repo cannot be used for fork repo. So we decided to merge and see if it would work. Unfortunately, there was some more thing to do. I first try in on my side project and one thing I was missing is expo-updates packages. This package is only used for deployment with any automation deployment. Manual deployment doesn't require this package. But this wasn't the end of our issue, there is more issue due to our metro.config.js file complaining about require('@rnx-kit/metro-resolver-symlinks'); and require('@rnx-kit/metro-config');. I wasn't sure why we needed this config, but after removing it, and doing some testing locally I was able to run the app without any issue. So I pushed those changes. I also noticed Expo was complaining about the wrong dependency version. This is because Expo required some specific version while using a certain version of expo SDK. To fix that I run an expo doctor to see what was the issue then run expo update to update to the correct version, after that I make sure the app is running properly then pushed my change. In the end, our mobile app got published at https://expo.dev/@humphd/Telescope?serviceType=classic&distribution=expo-go

Off topic

Back in last year October, I participated to the Hacktoberfest. And this week I received my T-shirt finally after several months of waiting.

My turn to be Sherriff

Kevan Y — Sun, 13 Feb 2022 17:43:04 +0000

Sheriff roles

Planning on Tuesday
Triage on Thursday
Assign Issue
Review PR
Make sure everyone is on track

Experience being a Sheriff

This is the first time I manage such a large group. On Tuesday, the meeting was more of an open-ended discussion about a current issue. I and Joel were leading the meeting. It was pretty hard, especially discussing a topic where I'm not familiar. I feel lost, and not worth enough to be the sheriff. This is also the first time that I'm able to speak that long. I usually spoke around 0-10 minutes in a 1-2 hours meeting. This time I spoke for the whole meeting. On Thursday, Joel was leading the triage meeting and I was taking note of all important stuff discussed during the meeting.
One thing I learned is that no one knows every part of telescope, if we don't know a part of telescope, we should know who in the call has expertise on that part. So we can directly ask for detail, such as what is this issue about, how long it will probably take...
Off meeting, we have to monitor slack and Github. Not gonna lie, it was pretty tough this week for me. I had one of my group members in PRJ that got Covid, and I was the only one in that week to implement most of the features for the frontend part. I feel this time I didn't do my job as a sheriff well enough compared to the others. But it's okay, we learn from mistakes and grow up.
Also, this was the best time to practice my English skill for speaking part.

Some of the issues I'm working
Issue #2732 - Still ongoing, discover some limitations with Apple restriction.
Issue #2791 - Almost done, just needed to figure out why Kong container crashed.
Issue #2801 - PR in draft mode

New knowledge

Kevan Y — Sat, 05 Feb 2022 04:41:09 +0000

This week I have been reading a lot of documentation. I got involved in Docker, Supabase, Renovate bot, Turborepo, Traefik, Nginx, and React-Native.

Docker

This week I have been learning a lot about Docker. I watched a quick intro into docker.

I started to do the small tutorial of docker by following the get-started of Docker.
Docker is a bit hard to learn, but it is just a matter of time. Once I get to use more and practice more, it will eventually become easier. The next step would be to learn Kubernetes.
Some of the issue I'm working that related to Docker

Issue - 2745 - Done
Issue - 2791 - Ready
Issue - 2799 - Ready
Issue - 2803 - Ready
Issue - 2830 - Ready

Issue - 2745 was my first issue with Docker stuff. I was looking at some documentation about how to install pnpm into the docker containers. I end up with this. I tried the solution followed the snippet code, but it wasn't working. Also, it is an experimental command which means it might have some breaking change in the non-major version. So I used the simple and basic solution RUN npm i -g pnpm. This would install pnpm into the global node_modules package.
To test if the container was built and working I ran pnpm services:start. Later on, working on Issue - 2803 I discovered I could start only 1 microservices instead of all of them by doing pnpm services:start image to start image microservice.

Supabase

Supabase is an open sources database. It's based on PostgreSQL, and adds more features to it.
We can say it can be an alternative to Firebase.
Read more
I watched a good quick intro about it.

Renovate Bot

Turborepo

Turborepo is a tool to manage repo. This is an amazing tool. It can cache things and only rerun stuff that changed in the code. Which makes the running process fast. It is a super useful and powerful tool for monorepo.
Read more
Also here is a good video of what tool on average used for monorepo.

Traefik / Nginx

Feb 04 meeting I learned more about Traefik / Nginx. Nginx is a tool to reroute/ load balance, but since it doesn't know about Docker exposed port, we need to use Traefik that communicates with Nginx and Docker.

Basically 
User > Nginx > Traefik > Docker

Why use Traefik / Nginx, is because we don't want to expose ports into the web, it would be bad for security, everyone in the world would try to hack into the system.
Issue - 2791 - This issue I think I'm gonna tackle since it would be a good learning touch Traefik / Nginx and also Docker.

React-Native

I recently read about a blog post of how to deploy a react native to IOS and Android. This would be useful for later on when telescope will be needed to be published into Play store and App store.
Based on the blog contains, it seems pretty simple to build and publish React-native app.
expo build is the command to build and expo upload:{andoid/ios} to publish.
But before that, we need to config App.json which defines our app name, icon, splash, and other configs for different systems.
Issue - 2760 - I referred to this documentation to complete this task.
Finale result what it would looks like:

Mid way before release 2.6

Kevan Y — Sat, 29 Jan 2022 15:49:31 +0000

A lot happened this week, on Tuesday meeting the sheriff, went through the task to-do for release 2.6, and on Thursday it was a triage of Pull requests and issues. During the triage, I go assigned to more issues, in the area of react native, docker, and a bit of front-end fix.
Also, did my first release on an open sources project in Satellite repo.

What I have done for release 2.6

Issue listed from my previous blog

Issue #2678 - Fixed by Pull Request #25. This wasn't too difficult to do since it was just removing @elastic/ecs-pino-format, elastic-apm-node, and its code. Then update jest for fixing deprecating dependencies.
Issue #2694 - Fixed by Pull Request #37. This one was a bit of research with pnpm engine and yml. Thanks to this Pull request #2545. It helped me to complete this issue quickly. I also had to learn about yml file and ci, how it's configured. Documentation I read, Github ci/cd, Actions setup-pnpm, Actions setup-node-js-environment, and Actions checkout.
Issue #2499 - Fixed by Pull Request #26. Thanks to my cloud computing class. I had used what I learned from that class and transferred to contribute to this issue. This makes me feel happy and glad to have taken this course, that I was able to use what I learned from a class and used it for other things besides class assignments.
Issue #2497 - Fixed by Pull Request #2700. This issue was similar to Pull Request #26, but for the date format, I had to read a bit of documentation pino-pretty, and node-dateformat. Why node-dateformat, because it's what pino-pretty uses for translateTime.
Issue #2306 - Issue still under progress, this one is a bit hard to reproduce since I get blocked by Medium. I had a hard time getting the feed data from telescope. I talked with Duc, the one who created that issue, he also had trouble reproducing it. Since most of our users use dev.to as blog posts. This issue is not my priority and will be put in the backlog.

What I have picked after my previous blog

Issue #39 - This was a hotfix fixed by Pull Request #40, telescope e2e was failing because in satellite repo we added preinstall to only use pnpm but docker was still using npm, so it was failing there during the installation. I had to revert and do a release. The process to do a release was pretty simple thanks to this Issue #38, documented by AmasiaNalbandian.
Issue #2707 - In progress will be fixed by Pull request #2751, this one was a bit tough to do, it was either grab the username from the regex and do a call to the API of GitHub to check if real username, but with that solution, we'll hit the API rate in a second. The second solution is less perfect but will work, it was to hardcode the list of reserved names, and in the filter only grab the ULR that doesn't match the reserved names. I had a nice talk with Andrew, the one who created that, and since we cannot filter all the GitHub names, it could be considered as The Ostrich algorithm.
Issue #2731 - This is still in progress and with some discussion with the group that does react native, it will be done by Hung.
Issue #2732 - Issue open for discussion about the deployment of the react native app.
Issue #2733 - Issue open for discussion, and mainly will be done by AmasiaNalbandian. I will assist in reviewing Pull request.
Issue #2734 - Issue open for discussion, and probably mainly will be done by AmasiaNalbandian and DukeManh. I will assist in reviewing Pull request.
Issue #2745 - I will be working on this one, I have a sort of working version in my local, and will create a Draft pull request for feedback since I'm not too familiar with docker. Used this documentation to complete this issue.

One of the best quote:

Never be afraid to fail. Be afraid of not learning from mistakes.
-- W. Brett Wilson

Open source is the place where you make mistakes and learn from it to become a better developer.

test post Kevan

Kevan Y — Thu, 27 Jan 2022 03:34:28 +0000

https://github.com/Seneca-CDOT/telescope/issues/2707
issue
https://github.com/Kevan-Y/Kryptfer/
repo
https://github.com/Seneca-CDOT/telescope/pull/2750
pull
https://github.com/Andrewnt219
manekenpix
https://github.com/datocms/

https://github.com/issues
https://github.com/marketplace
https://github.com/features/codespaces