Forem: Johnny Santamaria

Keeping Your Secrets in Sync: Environment Lifecycle Management That Actually Works

Johnny Santamaria — Sat, 14 Feb 2026 01:25:14 +0000

Introduction

Keyshade is an open-source real-time secret and configuration management platform that helps developers securely store, sync, and rotate sensitive data like API keys, passwords, and environment variables across multiple environments. It uses end-to-end encryption and live updates to eliminate insecure .env file sharing, manual dashboard updates, and secret sprawl. Designed for teams and solo developers alike, Keyshade improves collaboration, auditability, and security by providing version history, access controls, and seamless integrations with modern deployment platforms.

Try it out on https://keyshade.io/

The Problem: Silent Failures in Cloud Integrations (#1254)

The problem was that the integrations with Vercel or AWS Lambda would only respond to updates in secrets and variables. It would completely ignore situations where the user had deleted the environment or renamed the environment. This meant that when a user deleted an environment in Keyshade, the secrets and variables would still be available in Vercel or AWS Lambda but would be orphaned. This presented a problem from a security point of view. Also, when the environment was renamed, the integrations would be out of sync.

Why This Bug Mattered to Production Teams

This problem was significant because of several factors:

Security Risk: When a user deleted an environment using Keyshade (like an environment in staging mode), the secrets became active in Vercel and AWS Lambda. This meant the secrets were exposed on another platform. That is, credentials became potential attack vectors.
Data Integrity: The disconnect between Keyshade’s environment state and the external platforms led to a situation where users couldn’t be assured of data integrity. It defeats the purpose of having a reliable tool like Keyshade.
User Experience: It will be the responsibility of the consumers of Keyshade to manually log in to the Vercel or AWS Lambda interface to clean up after environment changes, thereby defeating the purpose of the integration in the first place. Such an interface might discourage the use of Keyshade.
Production Readiness: The integrations must be prepared to handle the full lifecycle of environments, not only the secrets within static environments, for Keyshade to be considered viable in production, especially by teams with high rates of environment changes (e.g., CI/CD workflows creating ephemeral environments). It was marked as "priority: high" and "difficulty: 4." This was my hardest PR that took around two months.

Under the Hood: Key Code Changes

keyshade/apps/api/src/integrations/plugins/vercel.integration.ts
Purposes: Integrates the Vercel platform - responsible for syncing Keyshade secrets/variables to Vercel projects as environment variables.
Key sections to edit:
getPermittedEvents(). Added ENVIRONMENT_UPDATED and ENVIRONMENT_DELETED to the set of events the integration subscribes to.
added case handlers to switch of emitEvent() for routing environment lifecycle events.
New functions added: handleEnvironmentDeleted(): This cleans up project-level environment variables scoped to the deleted environment and breaks the integration relationship handleEnvironmentUpdated(): Renames custom Vercel environments if available (pro plan users) & update integration metadata

apps/api/src/integration/plugins/aws-lambda.integration.ts
Purpose: Implements the AWS Lambda integration, syncing Keyshade secrets to Lambda function environment variables
Key sections modified:
getPermittedEvents(): Added the two environment events
emitEvent() switch statement: Added routing for environment events
New functions added:
handleEnvironmentDeleted(): Removes environment-scoped keys from Lambda function configuration and disconnects the environment relationship
handleEnvironmentUpdated(): No-op handler (Lambda has no environment rename concept) that logs an audit event

apps/api/src/common/util.ts
Purpose: Added retryWithBackoff() helper function to handle transient failures when calling external APIs (Vercel/AWS)
Why important: Cloud APIs can fail temporarily; this ensures cleanup operations are resilient and retry with exponential backoff

apps/api/src/integration/reconciler.ts
Purpose: Lightweight reconciliation system that processes failed cleanup operations stored in integration.metadata.pendingCleanup
Why important: When retries fail after multiple attempts, the reconciler can later replay these operations, ensuring eventual consistency even if external APIs are temporarily unavailable

apps/api/src/event/event.types.ts
Purpose: Defines TypeScript types for event metadata
Relevant types:
EnvironmentUpdatedEventMetadata: Contains name and description fields for environment updates
EnvironmentDeletedEventMetadata: Contains the environment name for deletion events

Test files:
apps/api/src/integration/plugins/vercel.integration.spec.ts: tests covering environment lifecycle handlers
apps/api/src/integration/plugins/aws-lambda.integration.spec.ts: tests covering Lambda-specific handlers
apps/api/src/integration/reconciler.spec.ts: testing the reconciler's pending cleanup processing

Supporting materials/artifacts
Before the fix:
When a user deleted an environment in Keyshade, the integration would:

Not receive or process the ENVIRONMENT_DELETED event
Leave all environment-scoped secrets active on Vercel/AWS Lambda
Not update the integration metadata to reflect the environment was gone
Create orphaned credentials that users would have to manually clean up

Error behavior:
While there wasn't a specific error message, the silent failure was the problem. Users would delete an environment expecting it to be cleaned up everywhere, but the secrets would persist on external platforms with no warning or indication that cleanup failed.
After the fix:
The integration now:
Subscribes to ENVIRONMENT_UPDATED and ENVIRONMENT_DELETED events
Automatically removes environment variables from Vercel projects when environments are deleted
Removes environment-scoped keys from AWS Lambda function configurations
Updates integration metadata to stay in sync with environment name changes
Uses retry logic with exponential backoff for resilience against transient API failures
Persists failed operations in pendingCleanup for later reconciliation
Logs audit events for compliance and debugging

Key Implementation Detail:
For Vercel's ENVIRONMENT_UPDATED handler, there's a special consideration: Vercel only allows renaming custom environments on Pro plans, not the standard "Preview", "Production", or "Development" environments. The code includes logic to attempt the rename and gracefully handle cases where it's not supported.

Challenges

While preparing my pull request, I ran into several real-world challenges that highlight what contributing to an active open-source project actually looks like behind the scenes. The first issue was merge conflicts between my branch (attempt-1254) and the target develop branch. In a fast-moving repository with many contributors, this is almost inevitable. I resolved the conflicts by pulling the latest upstream changes and manually reviewing each conflicting section to understand both sides. My goal was to preserve other contributors’ work while ensuring my environment lifecycle handler implementation remained intact. After merging, I tested everything locally to confirm nothing regressed, then committed with a clear explanation of the resolution. It was a reminder that merge conflicts aren’t just mechanical fixes — they’re moments where you have to understand the intent of multiple developers and reconcile them thoughtfully.
The next challenge surfaced during code review. A maintainer questioned my use of any type casts in TypeScript, particularly when accessing integration.id and working with the pendingCleanup metadata array. They pointed out that this could introduce long-term technical debt and suggested using @ts-expect-error instead. That feedback was valuable. While the any casts were originally a temporary measure to unblock a critical bug fix, the reviewer highlighted how @ts-expect-error is self-validating: TypeScript warns you if the suppressed error disappears later, which prevents stale workarounds from lingering in the codebase. I acknowledged the concern, added TODO comments recommending a future typed refactor, and committed to either replacing any with @ts-expect-error where appropriate or creating a follow-up issue to properly type the metadata structures. This exchange was a great example of how open-source review culture improves not just correctness, but long-term maintainability.
The biggest blocker came from the CI/CD pipeline. My PR was failing two checks: a Snyk security scan and an API validation build. The frustrating part was that I couldn’t reproduce the failures locally. All my local tests passed — dependencies installed cleanly, the build succeeded, and every unit test ran successfully. But Snyk repeatedly failed in CI with errors related to workspace handling and package manager compatibility. The CLI didn’t fully support the project’s pnpm workspace setup in the same way the CI environment did, and I couldn’t access the GitHub Actions logs directly from my development environment. I tried multiple workarounds, including running Snyk through pnpm and downloading the standalone CLI, but nothing mirrored the CI behavior. Even rerunning the workflows with empty commits didn’t resolve the issue.
At that point, I documented everything I had tried in a detailed PR comment and asked for help. A maintainer with admin access reviewed the CI logs and identified the root cause: a vulnerability in a transitive dependency that required updating the root package.json and regenerating the lockfile. They also explained that the project’s Snyk integration uses a specialized workspace configuration that isn’t trivial to replicate locally. This led to a broader discussion about improving CI transparency for external contributors, including the possibility of automatically surfacing relevant log snippets in PR comments. The experience reinforced an important lesson: as an external contributor, you won’t always have full visibility into infrastructure, and clear documentation of your debugging efforts is essential for maintainers to step in effectively.

Solution (PR #1255)

After working through these issues collaboratively, the pull request was successfully merged. The final solution included environment lifecycle event handlers for both Vercel and AWS Lambda, retry logic with exponential backoff, a reconciliation system for eventual consistency, and comprehensive test coverage. All CI checks passed, and review feedback was fully addressed. More importantly, the contribution closed a high-priority bug that had been blocking production adoption of cloud integrations. It was a strong reminder that persistence, careful communication, and respect for the review process are just as important as writing the code itself. Open-source success isn’t just technical, it’s collaborative.

What Did I Learn?

How to read official Typescript documentation to break down confusing code
How to ask beneficial questions to maintainers
How to navigate CI/CD failures as an external contributor
Understanding Event-Driven API Architecture
Snyk improves security in development

Keyshade Debugging: Mastering Workspace Role Tests and API Repair

Johnny Santamaria — Sat, 08 Nov 2025 01:06:38 +0000

"A good tester prevents problems; a great tester finds them." Keith Klain, Director of Quality Engineering at KPMG UK

Introduction

Four disabled tests. One cryptic comment: 'flaky.' When I dove into Keyshade's workspace role tests, I discovered the failures weren't random at all—they revealed a fundamental misunderstanding of how the API validates permissions.

Keyshade is a modern solution for secure, centralized credential and configuration management, designed for developers, DevOps engineers, and open-source communities who need to manage secrets without the headache.

The Issue (#1122)

I decided to continue my open source journey on Keyshade since my blog post going over how I solved issue 998, which was that the Keyshade command line interface’s “keyshade run” command fails to give a proper error that users can understand. Now I decided to explore Keyshade’s API since I have explored how the CLI functions in its codebase.

I found an issue where included a bug in workspace role tests, which meant the problem stemmed from how workspace roles were being set up and asserted in tests, some test fixtures and assertions depended on implicit ordering and shared state, which made the tests fragile and allowed timing or state-dependent flakes to slip into CI (Continuous Integration). By making the setup deterministic and isolating role state for each test (explicitly creating the roles and their relationships, avoiding shared mutable fixtures, and tightening assertions to check the actual role assignments rather than relying on incidental data), the tests now reliably exercise the API behavior they were intended to cover.

The result is a more robust test suite that accurately guards against regressions in role handling for workspaces, reduces CI noise from flaky failures, and gives us greater confidence in future changes to authorization and workspace logic. Here are some key locations in the codebase we found out to be useful in solving the issue:

The apps/api/src/workspace-role/ folder includes the workspace-role feature module — contains controller, service, DTOs, and possibly unit/integration tests. Focus on business logic, dependency wiring, and any in-module role constants.

Examples:

Inspect the service methods (assignRole, removeRole, listRoles) for edge-case logic (no-op on duplicate role, default role applied on creation).
Check module exports and DI config: ensure the controller gets the expected implementation of the role service (mock vs real) in tests.
Example check: verify if role constants are defined here (e.g., 'admin', 'member', 'viewer') and whether they match values used in DB/entities or frontend.

The apps/api/src/workspace-role/workspace-role-controller.ts file includes the controller / HTTP handlers for workspace-role endpoints — review request DTOs/validation, how the controller calls services, status codes, and permission checks performed at the controller layer.

Examples:

Example request mapping: POST /workspaces/:id/roles -> controller.assignRole(req.params.id, req.body). Check expected body shape (userId, role).
Example validation: if controller uses decorators or a schema, show the required fields (e.g., { userId: string, role: 'admin'|'member' }).
Example permission check: controller may call guard like ensureWorkspaceAdmin(currentUser, workspaceId) — confirm where that check happens and what error it throws (403 vs 401).
The apps/api/src/workspace-role/workspace-role/workspace-role.e2e.spec.ts file has end-to-end tests for workspace-role flows — tests that spin up the app (or a test server), seed DB data or fixtures, run real HTTP requests, and assert full-stack behavior and responses.

Examples:

Test setup/teardown: show DB seeding (create users, create workspace) and cleanup steps to avoid cross-test leakage.
Example test case: an e2e test that POSTs role assignment and then GETs membership list to assert the new role is present; include expected status codes and response shapes.
Flakiness checks: look for timeouts, async waits, or hard-coded delays that may make these tests flaky (and thus failing intermittently).

General Diagram Walkthrough of the Issue

Reproducing the Problem

Here are steps regarding how I reproduced the problem before assessing my plan, implementation, and solution

Scenario 1 — should not be able to update the workspace role with environments that do not belong to the project Steps to reproduce the behaviour:

Create (or use) a workspace W.
Create Project A in W and capture PROJECT_A_ID.
Create Project B in W and capture PROJECT_B_ID.
Create Environment A in Project A (ENV_A_ID) and Environment B in Project B (ENV_B_ID).
Create a workspace role R for Project A that includes only ENV_A_ID.
Configure your CLI to target workspace W and authenticate with a token that can update roles.
Use the CLI (or send an API request) to update role R to set environmentIds = [ENV_A_ID, ENV_B_ID].
Observe the response/error shown by the CLI.

Expected behaviour

The API should validate that ENV_B_ID does not belong to Project A and return a 4xx validation error (e.g., 400 or 422) with a clear, formatted message such as: "environment ENV_B_ID does not belong to project PROJECT_A_ID"
The CLI should print that formatted message to the user (no opaque stack trace or generic 500).
The update must be rejected and the role R must remain unchanged (still only ENV_A_ID).

Scenario 2 — should be able to add environment access for projects to the role with READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT authorities Steps to reproduce the behaviour:

Create (or use) a workspace W.
Create Project P in W and ensure it has at least one environment (ENV_P_1, …).
Create a workspace role R (it may initially have no project-level environment access).
Obtain an auth token that has the authorities: READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT.
Configure your CLI to use workspace W and that token.
Use the CLI (or send an API request) to add project-level environment access for Project P to role R (the endpoint that grants project environment access).
Observe the response/error shown by the CLI and then GET the role R to inspect its environment access.

Expected behaviour

The API should accept the request and return 200/201 with a clear success body describing the new project-level environment access (or list of environmentIds that were added).
The CLI should print a readable success message and show the role now has access to Project P environments.
If the token lacks any required authority, the API should return 403 Forbidden with a clear formatted error and the CLI should print that message.

Scenario 3 — should be able to add environment access for projects to the role (permission variation) Steps to reproduce the behaviour:

Create (or use) a workspace W, a Project X with environments, and a role R.
Prepare two auth tokens:

* AUTH\_ADMIN: token with UPDATE\_WORKSPACE\_ROLE (and related read authorities).

* AUTH\_RESTRICTED: token without UPDATE\_WORKSPACE\_ROLE.

Configure your CLI to use workspace W.
With AUTH_ADMIN, run the CLI command (or API request) to add Project X environment access to role R. Verify success.
With AUTH_RESTRICTED, attempt the same CLI command (or API request) to add Project X environment access to role R. Observe the response/error shown by the CLI.

Expected behaviour

With AUTH_ADMIN: API returns 200/201 and the role R gains access to Project X environments; CLI prints a readable success confirmation.
With AUTH_RESTRICTED: API returns 403 Forbidden with a clear formatted message like "missing authority: UPDATE_WORKSPACE_ROLE" and the CLI prints that message; role R must not be modified.

Scenario 4 — should not be able to create workspace role where environments do not belong to the project Steps to reproduce the behaviour:

Create (or use) a workspace W.
Create Project A and Project B in W.
Create Environment B in Project B (ENV_B_ID).
Configure your CLI to target workspace W with a token that can create roles.
Use the CLI (or POST API) to create a workspace role with:

* projectId = PROJECT\_A\_ID

* environmentIds = \[ENV\_B\_ID\]

Observe the response/error shown by the CLI.

Expected behaviour

The API must reject the create request with a 4xx validation error (e.g., 400 or 422) and a clear message such as: "environment ENV_B_ID does not belong to project PROJECT_A_ID"
The CLI should print that formatted error.
No role should be created.

Keyshade’s Codebase

Here is Keyshade's codebase and API related to this issue, and how would I use the code to diagnose and fix the failing workspace-role tests, displaying the importance of code organization.

High-level overview (relevant subsystems)

Schema / client shapes:
- packages/schema/src/workspace-role/index.ts defines the WorkspaceRole payload shape (projects + environments arrays, authorities, etc.). That tells you the expected request/response shape for workspace role create/update operations.
API service layer:
- apps/api/src/workspace-role/workspace-role.service.ts contains the logic that parses project -> environment inputs and applies DB operations. The key function is parseProjectEnvironmentsDBOperation which:
  - verifies that an environment slug exists and belongs to the provided project
  - throws NotFoundException if an environment is not part of the project
  - calls authorizationService.authorizeUserAccessToEnvironment to ensure the acting user has READ_ENVIRONMENT on that environment
Authorization checks:
- apps/api/src/auth/service/authorization.service.ts and apps/api/src/auth/service/authority-checker.service.ts implement the authority checks and higher-level authorization plumbing. authorizationService exposes methods such as authorizeUserAccessToEnvironment and authorizeUserAccessToSecret; these both call into the authority checker and also verify workspace membership.
Tests & client usage:
- packages/api-client/tests/workspace-role.spec.ts contains tests that exercise the workspace-role APIs from a client perspective (create workspace, create project, create environments, then create/update roles).
- apps/platform and controller-instance reference the WorkspaceRoleController from the generated API client; you can use the API client or the CLI to reproduce e2e scenarios.

Why the tests were failing (what the issue description lists)

The failing/disabled tests are focused on cases where environments provided for a workspace role do not belong to the project, and cases that require multiple authorities (READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT).
The service code enforces two separate checks:

1. Ownership check: the environment must belong to the provided project — otherwise parseProjectEnvironmentsDBOperation throws NotFoundException with a message like "Environment X is not part of project Y".

2. Authorization check: after confirming the environment belongs to the project, the service calls authorizationService.authorizeUserAccessToEnvironment which requires the user to have READ\_ENVIRONMENT on that environment (and also checks workspace access).

If tests tried to use an environment that belongs to a different project, the service will intentionally return 404/NotFound before attempting to grant access; tests must assert that behavior.
If tests expected that adding environment access requires specific authorities, the test setup must give the test-user those authorities (or use a workspace-admin) before calling the endpoint.

How to use the codebase to reproduce, diagnose, and fix the tests

Reproduce the failing case manually (fastest way to see precise error):

* Use packages/api-client/tests/workspace-role.spec.ts as a reference for how tests set up the workspace/project/environment/role chain.

* From an integration/e2e test or an API client, create:

    * workspace A

    * project P1 inside workspace A

    * project P2 inside workspace A

    * environment E1 in P1 and environment E2 in P2

* Then attempt to create/update a workspace role that includes project P1 but lists environment E2 (an environment that belongs to P2).

* Observe the response: workspace-role.service.ts is expected to detect the mismatch and return a 404 Not Found with message "Environment ... is not part of project ...".

Check the authority path:

* If you provide a correct project-&gt;environment mapping, but the requestor lacks READ\_ENVIRONMENT (or other expected authorities like UPDATE\_WORKSPACE\_ROLE), authorizationService.authorizeUserAccessToEnvironment will throw Unauthorized/Forbidden. Ensure test user has the required authorities when the test expects success.

Fix the tests (common issues & fixes):

* If a test was disabled because it used an environment from a different project but expected a different error (or success), restore the test and:

    * Either adjust the test to expect NotFound (because parseProjectEnvironmentsDBOperation intentionally rejects cross-project environments).

    * Or adjust test fixtures so the environment is created in the same project as the project entry in the request.

* For authority-related tests:

    * Make sure the test user has the necessary authorities before calling the API. Grant a role to the test user with READ\_WORKSPACE, UPDATE\_WORKSPACE\_ROLE, READ\_PROJECT, READ\_ENVIRONMENT when the test expects role creation/update to succeed.

    * If a test is verifying denial paths (should not be able to create/update), ensure the user has fewer/missing authorities and assert the API returns Unauthorized/Forbidden appropriately.

Where to change code if the behavior is incorrect:

* If you decide the server should validate authority first (instead of ownership first), modify parseProjectEnvironmentsDBOperation ordering; but be careful: the current behavior (check membership/ownership first, then authorization) is consistent with tests that expect NotFound for cross-project envs.

If messages or status codes are inconsistent with tests, adjust the thrown exceptions in parseProjectEnvironmentsDBOperation (NotFoundException) or in authorizationService (Unauthorized/Forbidden) to align with the test expectations — but prefer changing tests to reflect correct domain behavior unless an API bug is proven.

Useful files to inspect or instrument:

* packages/schema/src/workspace-role/index.ts (payload schema)

* apps/api/src/workspace-role/workspace-role.service.ts (parseProjectEnvironmentsDBOperation and create/update logic)

* apps/api/src/auth/service/authorization.service.ts (authorizeUserAccessToEnvironment)

* apps/api/src/auth/service/authority-checker.service.ts (lower-level authority checks)

* packages/api-client/tests/workspace-role.spec.ts (test setup and expected assertions)

Concrete actionable checklist to resolve the issue in the repository

Re-enable the disabled tests.
For each disabled test, verify which of the two server behaviors it is validating:
- Ownership validation (environment belongs to project) → expect 404 Not Found. Ensure test builds a request with a mis-matched environment and asserts a 404 and the message text if desired.
Authorization validation (user needs the listed authorities) → ensure test grants the required authorities to the requester when expecting success; when expecting failure, ensure the user lacks at least one required authority and assert Unauthorized/Forbidden.
Run the tests locally (or in CI):
- The workspace-role e2e test file: packages/api-client/tests/workspace-role.spec.ts is a good starting point.
- Add logging in workspace-role.service.ts parseProjectEnvironmentsDBOperation if you need to trace why an environment was rejected.
If the test failure shows unexpected status codes or messages, adjust the service's thrown exceptions to match the documented API or update the tests to assert the documented behavior.
If you change server code, add unit tests in apps/api for parseProjectEnvironmentsDBOperation to cover:
- environment not in project → NotFound
- environment in project but user lacks READ_ENVIRONMENT → Unauthorized/Forbidden
- environment in project and user has authority → success path produces correct DB ops

Short example of the expected behavior (mapping to the tests listed in the issue)

"should not be able to update the workspace role with environments that do not belong to the project":
- Call update role with project P1 + env from P2 → server should return 404 Not Found (environment not part of project).
"should be able to add environment access for projects to the role with READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT authorities":
- Ensure requestor has those authorities (assign a role in test fixture) → call create/update and assert success and that the role contains the project->environment mapping in response.
"should be able to add environment access for projects to the role":
- Same as above but likely using normal granted authorities; ensure fixtures grant appropriate permissions.
"should not be able to create workspace role where environments do not belong to the project":
- Create role payload with mismatched environment → expect 404 Not Found.

Where to look in code to make changes or confirm behavior

parseProjectEnvironmentsDBOperation in:
- apps/api/src/workspace-role/workspace-role.service.ts
authorization plumbing:
- apps/api/src/auth/service/authorization.service.ts
- apps/api/src/auth/service/authority-checker.service.ts
request/response schema:
- packages/schema/src/workspace-role/index.ts
tests:
- packages/api-client/tests/workspace-role.spec.ts

Use Case

Goal: Create/update a workspace role that grants access to specific projects and, within those projects, specific environments.
Client payload shape (what the API expects)

1. name, description, authorities (array), projectEnvironments: \[{ projectSlug, environmentSlugs: \[slug\] }\]

Server behavior (what the code enforces)

1. For each projectEnvironments entry, resolve projectSlug -&gt; projectId (getProjectSlugToIdMap).

2. For each environmentSlug in environmentSlugs:

    * Verify that environmentSlug exists AND belongs to that project (prisma.environment.findFirst where slug + projectId). If not, throw NotFoundException (404) with a body "Environment not found" / "Environment X is not part of project Y".

Authorize the acting user has READ_ENVIRONMENT for that environment (authorizationService.authorizeUserAccessToEnvironment). This also verifies workspace-level access and permitted authorities (via authorityCheckerService).

Finally, upsert projectWorkspaceRoleAssociation rows connecting roleId<>projectId and connect the environment slugs.

Why the workspace-role tests were failing

The failing/disabled tests covered two distinct types of validation:
- Ownership validation: environment must belong to the project in the request. If not, server should reject with 404 (Not Found).
- Authorization validation: even if environment belongs to project, the acting user must have proper authorities (READ_ENVIRONMENT, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_WORKSPACE_ROLE) to add environment access to a role.
Typical causes for failures:
- Tests that assert 404 were previously commented out or used incorrect fixtures (e.g., environment not created, or created but in same project), so the server behavior didn't match test expectations.
- Tests that assert successful updates weren't ensuring the test actor had the required authorities (so server returned Unauthorized/401/403 instead of 200).
My PR re-enabled those tests and fixed fixtures by:
- Creating environments in the correct projects (dev, stage).
- Explicitly setting the membership role authorities (e.g., update the workspace member role to include READ_ENVIRONMENT and UPDATE_WORKSPACE_ROLE) when the test expected updates to succeed.
- Using consistent test header 'x-e2e-user-email' (alice.email) for requests.

Walkthrough of the two failing test categories and what to change

"Environment doesn't belong to the project" (ownership)

Server side: parseProjectEnvironmentsDBOperation checks project membership:
- It runs prisma.environment.findFirst({ where: { slug: environmentSlug, projectId } })
- If not found, it throws: throw new NotFoundException(constructErrorBody('Environment not found', Environment ${environmentSlug} is not part of project ${pe.projectSlug}))
Tests expecting 404 must:
- Create a workspace and two projects in that workspace.
- Ensure the environment used in the request was created in THE OTHER project, or not created at all (but prefer creating in other project to avoid false positives).
- Make a POST/PUT with projectSlug set to projectA and environmentSlugs containing an environment slug that actually belongs to projectB.
- Assert statusCode === 404.

"Adding environment access requires specific authorities" (authorization)

Server side: after ownership check, parseProjectEnvironmentsDBOperation calls: await this.authorizationService.authorizeUserAccessToEnvironment({ user, slug: environmentSlug, authorities: [Authority.READ_ENVIRONMENT] })
That code path will call authorityCheckerService.checkAuthorityOverEnvironment which checks the acting user's permitted authorities for that environment (it can be from project-level or workspace-level roles).
Tests expecting success must ensure the acting user has:
- READ_WORKSPACE (if required by the flow), UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT — in practice, the tests can mutate an existing workspace role for the test user to include these authorities and also connect appropriate project/environment associations so the user has project-scoped authority.

Exact test changes — enable one disabled test and adjust fixture

Below is a concrete enabled test snippet for packages/api-client/tests/workspace-role.spec.ts that re-enables the “should not be able to create workspace role where environments do not belong to the project” scenario.
This patch enables the test and makes the fixture explicit: create two projects, create an environment in projectB, then attempt to create a workspace role that references projectA + the environment slug from projectB. The test asserts 404.

packages/api-client/tests/workspace-role.spec.ts

// (This is the test snippet to add/enable within the existing test suite)

it('should not be able to create workspace role where environments do not belong to the project', async () => {

// Precondition: two projects exist in the workspace

// projects[0] -> projectA, projects[1] -> projectB (test fixture established earlier in the file)

// Create an environment in projectB that should NOT be valid for projectA

await prisma.environment.create({

Notes about placement:
- Put this enabled test near the other workspace-role create tests after the workspace + project fixtures are created and available.
- Ensure projects variable references are the ones created during beforeAll/beforeEach steps in that test file.

A directly analogous enabled e2e test (what PR 1156 did)

PR 1156 enabled very similar tests in apps/api/src/workspace-role/workspace-role.e2e.spec.ts — it:
- Created dev/stage environments in the proper projects,
- Updated the Member workspace role to add project associations and required authorities for the actor,
- Re-enabled the previously commented tests and asserted status codes 200 or 404 as appropriate.

Small server-side unit test to lock in parseProjectEnvironmentsDBOperation behavior

I recommend adding a unit test that calls parseProjectEnvironmentsDBOperation (or exercises the public create/update API) with a project-environment mismatch and asserts a NotFoundException is thrown.
Example test file (unit) to confirm rejection for cross-project environment:

apps/api/src/workspace-role/workspace-role.service.spec.ts

import { Test, TestingModule } from '@nestjs/testing'

import { WorkspaceRoleService } from './workspace-role.service'

import { AuthorizationService } from '@/auth/service/authorization.service'

import { PrismaService } from '@/prisma/prisma.service'

import { NotFoundException } from '@nestjs/common'

Proposed server code changes — minimal / optional

The current server code in parseProjectEnvironmentsDBOperation is already correct about ownership-first validation (it checks slug + projectId before calling authorizeUserAccessToEnvironment). The PR you provided does not require a behavior change on the server — tests should reflect the server semantics.
If you did want to change server behavior, the main two options are:

1. Keep current semantics (ownership check first, then authorization). Pros: deterministic 404 for cross-project envs; avoids running authorization queries for environment slugs that don't belong to the provided project. Cons: reveals that the environment slug exists in a different project (server replies 404 with message referencing project mismatch).

2. Run authorization first (i.e., check the caller's permissions to read the environment before checking project membership). Pros: could collapse some error channels and avoid disclosing cross-project existence if the caller lacks permission. Cons: often you still need to fetch the environment by slug (so you still learn about its existence) — and the authorization check typically requires fetching the environment anyway to know project/workspace id. It also changes the status codes returned for the cross-project case (may return 401 instead of 404), which is a breaking API semantics change for clients that currently rely on 404 in that case.

Recommendation: do NOT change the ordering of checks unless there's a clear security/privacy reason. The current behavior (404 on ownership mismatch) matches the tests in PR 1156 and is explicit. Prefer to fix test fixtures and grant/revoke authorities in test setup rather than altering server semantics.

Why the PR's approach (what you already did in PR 1156) is correct

You re-enabled tests and fixed fixtures instead of changing server semantics. Specifically:
- Created missing environments in the appropriate projects before calling the API.
- Updated the test user/role authorities where tests expected success.
- Asserted the correct status codes: 404 for ownership mismatch and 200 for successful updates when authorities exist.
This is minimal and non-breaking.

Trade-offs if you change server behavior (ownership vs authorization ordering)

Changing the order to authorize first:
- May return 401/403 instead of 404 for the cross-project environment case — tests and clients must be updated.
- Could obscure whether an environment exists in another project (slight security benefit).
- Could introduce additional database lookups if you need to check membership as well; may not yield significant performance gain.
Keeping ownership-first (current):
- Clear, deterministic 404 response for mismatched environment ownership.
- Tests remain simple: assert 404 for the cross-project case.
- Slight information disclosure (you show that environment exists but in a different project), which may be acceptable in an internal system like Keyshade.

Summary of exact artifacts to change (concrete)

Add/enable the test in packages/api-client/tests/workspace-role.spec.ts as above (or in apps/api/src/workspace-role/workspace-role.e2e.spec.ts — PR 1156 already did the same).
Add a small unit test in apps/api/src/workspace-role/workspace-role.service.spec.ts to lock in behavior of parseProjectEnvironmentsDBOperation for cross-project envs (example above).
Ensure any tests that expect successful updates explicitly:
- create the environments in the projects they want to reference
- grant the test actor role(s) the required authorities (UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT) before calling create/update

Challenges

I reviewed PR #1156 and walked the tests and service code. Below I list concrete challenges we encountered while making this PR and practical mitigations. I narrate what I checked and what I'd recommend next.

What I did (quick recap)

Inspected the workspace-role service logic (parseProjectEnvironmentsDBOperation) and the authorization flow.
Reviewed the test files that were re-enabled/modified in the PR (workspace-role e2e/spec files).
Confirmed the PR fixes were basically test-fixture and authority-setup issues (creating environments in right projects, granting the test user required authorities), not major server-behavior changes.
Verified the PR merged successfully but noted a set of practical challenges that often show up in this work.

Challenges encountered (with examples and mitigations)

Test fixture brittleness and DB state coupling

Problem: E2E tests depend on precise workspace/project/environment setup and roles. Small differences in fixture creation order or leftover DB rows cause flakiness.
Example: A test assumed "production" env exists in projectA but CI used a different test run that already had production in projectB -> unexpected 200 instead of 404.
Mitigation:
- Always create and assert fixtures in test setup (do not assume names).
- Use randomized slugs (e.g., slug-<timestamp>-<random>) to avoid collisions across parallel CI jobs.
- Add teardown/cleanup that removes created rows or run tests inside isolated databases/transactions.

Authority/permission complexity in tests

Problem: Tests that check success paths require several authorities (READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT). If the test doesn't explicitly grant them, the server returns Unauthorized and the test fails.
Example: Test enabling environment access but not updating the Member role to include UPDATE_WORKSPACE_ROLE.
Mitigation:
- Explicitly set up the actor’s roles/authorities per test case.
- Add helper functions in tests for "grantMemberAuthoritiesForEnvironmentAccess" to make fixtures clear.
- Keep negative/positive authority tests separate and deterministic.

Merge conflicts (simple but annoying)

Problem: Enabling tests and modifying the same test files or fixtures on different branches led to conflicts (e.g., two branches each uncommented different blocks of tests).
Example: Two branches changed apps/api/src/workspace-role/workspace-role.e2e.spec.ts: one enabled a test, another refactored a beforeEach. Merge resulted in conflict markers around large commented blocks.
Mitigation:
- Rebase frequently on the develop branch to reduce long-lived divergences.
- Split large test-enabling changes into smaller PRs (enable + fix fixtures + authority wiring).
- When resolving conflicts, run the full test suite locally before pushing the merge commit.

Creating temporary test files or helper tests "to be removed later" causing issues

Problem: Developers often create throwaway tests / temporary files to debug behavior and forget to remove them. These can be picked up by CI and cause nondeterministic runs or duplicate test names.
Example: Added apps/api/src/workspace-role/tmp-debug.spec.ts to iterate quickly, later removed locally but remained in another branch → CI fails or duplicate DB setup occurs.
Mitigation:
- Avoid committing throwaway test files. If necessary, keep them in local-only branches or protect with a naming convention excluded from CI.
- Add CI test discovery rules to ignore test files matching tmp-*.spec.ts, if you must have temporary tests.
- Review diffs carefully before commit and run git status to ensure no debug/test files are included.

Test duplication across packages (api-client vs apps/api)

Problem: Similar workspace-role tests existed in both packages/api-client/tests and apps/api e2e test files; enabling one copy and forgetting to update the other caused inconsistent behavior.
Example: One test in packages/api-client used the API client and different headers than the e2e server-side test; runtime differences led to mismatched assertions.
Mitigation:
- Consolidate canonical e2e tests in one place (prefer apps/api e2e that hits the server directly), keep api-client tests focused on client behavior.
- If duplication is unavoidable, keep test expectations aligned and use shared helpers/mocks.

Race conditions in setup/cleanup (parallel tests)

Problem: Creating projects/environments concurrently across parallelized test jobs (or even within same job) created timing-based failures.
Mitigation:
- Make tests idempotent and isolate data per test.
- Use unique slugs per test run.
- Consider serializing tests that share DB resources or use test DB instances per worker.

Error/status-code semantics and client expectations

Problem: Tests rely on specific status codes (404 vs 401). If server-side code ordering changes (ownership check vs authorization check) the codes change and many tests break.
Example: If you move authorization first, a cross-project environment case might return 401 instead of 404.
Mitigation:
- Keep server semantics stable; if a change is required, update all tests and document the change.
- Add unit tests that assert the contract (e.g., cross-project environment -> NotFound).

CI flakiness from environment variables and headers

Problem: Some tests rely on headers like 'x-e2e-user-email'. If CI environment variable email differs or parallel runs reuse the same email, tests can mutate shared state.
Mitigation:
- Generate a test-specific email per run (e.g., alice+<run-id>@keyshade.xyz).
- Make test harness read and propagate headers from the same fixture source.

Commit noise and history (multiple tiny commits while fixing test)

Problem: WIP commits to enable tests, debug, revert, add files, remove files created a messy branch and bigger diffs, increasing review overhead and merge conflicts.
Mitigation:
- Squash local fixup commits before opening PR or use interactive rebase to tidy commits.
- Keep PRs small and focused: one PR for enabling tests and fixtures, another for refactors.

Accidental API contract changes vs test fixes

Problem: When tests fail, there’s a temptation to change server code to make tests pass instead of fixing tests. That can mask real bugs or break clients.
Mitigation:
- Prefer fixing tests/fixtures if the server behavior matches documented contract.
- If server needs change, write unit tests to cover new contract and get reviewer approval.

Concrete, practical steps I recommend next

Add a small checklist to PR templates for test-enabling PRs:
- Confirm unique slugs in fixtures
- Confirm actor authorities are set explicitly
- Run full test suite locally (or run the specific e2e group) before pushing
- Clean up any temporary test files
Add a helper in tests for deterministic unique slugs and user emails to avoid collisions in CI.
Consider adding a lint/CI guard that fails the build when "tmp-*.spec.ts" or "debug" test files are found in the commit.
If you expect future merges to touch the same big e2e files, coordinate branches or split PRs (e.g., create "enable tests" PR and a follow-up "refactor fixtures" PR).

Solution

Here is a summarized plan of my solution:

Lock the contract with unit tests by adding unit tests that assert parseProjectEnvironmentsDBOperation rejects cross-project environment slugs with NotFound. This prevents accidental server reordering from changing semantics. Make tests deterministic by ensuring every test explicitly creates its fixtures (workspace, project, environment) and set unique slugs per run, which will explicitly grant the required authorities to the test actor in success-path tests.

Prevent accidental temporary test files by adding the pre-commit hook above and a CI check to refuse pushes that include tmp/debug test files. Keep good merge strategies and PR hygiene by keeping PRs small and focused (enable tests separately from refactors). Make sure to rebase frequently and squash debug commits and add a PR/TODO checklist that requires: unique slugs, explicit authority setup, no temporary files, and local e2e run before requesting review. Consolidate test suites by deciding canonical location for workspace-role e2e tests (prefer apps/api e2e) and remove duplicates, or centralize test helpers. Here is my final PR solving the issue over a month:

PR #1156

Conclusion

In conclusion, Keyshade’s clean, well-structured codebase and deterministic tests make development faster, safer, and more collaborative. Clear intent, consistent conventions, and reliable CI build trust between maintainers and contributors—enabling focused changes, quicker reviews, and confident releases.

Fixing CLI Error Handling: A Deep Dive into Keyshade's WebSocket Communication Bug

Johnny Santamaria — Tue, 12 Aug 2025 15:00:00 +0000

“If you build software, every error message is marketing” - Jason Fried, Rework

Introduction

In today’s fast-paced software development landscape, managing credentials and sensitive configurations securely is more critical—and more challenging—than ever. Dispersed secrets, manual processes, and unplanned storage not only slow teams down but also increase the risk of breaches and operational errors. Keyshade addresses these challenges head-on with a centralized, automated approach to credential and configuration governance. Purpose-built for engineering teams, DevOps practitioners, startups, and open-source maintainers. Keyshade streamlines secret management while enhancing security, collaboration, and efficiency.

Keyshade is an advanced platform for centralized credential and configuration governance, designed to safeguard sensitive data, eliminate human error, and streamline operational complexity. By automating secret management, it fortifies security posture, amplifies cross-team collaboration, and liberates developers from the inefficiencies of fragmented secret storage. Its primary constituency—engineering teams, DevOps practitioners, early-stage startups, and open-source maintainers—adopts Keyshade to mitigate secret proliferation, minimize security liabilities, and embed robust secret orchestration seamlessly into their development pipelines. Users gravitate toward Keyshade for its ability to render secret management secure, intuitive, and collaborative, without imposing friction upon established workflows or toolchains. Below is a flow of control diagram discussing how users would typically use the product:

The user can send commands via keyshades terminal or on keyshades web app
On the web, the user can sign up, and initialize projects within workspaces
- Users can do the same in the CLI
Once the a project is made a http request is made to the API websocket service
The websocket service then retrieves that data to put in the Keyshade’s database to save the data for future use

Keyshade is currently in alpha and not live so there really are not any users as of posting this.

The issue

I partnered with a member of the Computing Talent Initiative with the task of solving issue 998 in Keyshade for a Summer Open Source Experience sponsored by . The main problem we noticed was that the Keyshade command line interface’s “keyshade run” command fails to give a proper error that users can understand. We would want to extract an error message and output it to the user. The issue is a bug that seems urgent to solve right now for users to understand errors they might make when working with Keyshade’s CLI, which is the main feature of Keyshade. Here are some key locations in the codebase we found out to be useful in solving the issue:

run.command.ts

Handles CLI logic for running commands, including WebSocket client interactions
Within a CLI folder

change-notifier.socket.ts

Contains the handleRegister method, which processes the register-client-app event
Handles authentication, authorization, and client registration logic
Emits the client-registered response to the client and logs errors
Within an API folder

run.types.d.ts

ClientRegisteredResponse describes the shape of the response sent by the server after registration (success, message)
Ensures type safety for client-side code handling server responses
Within a CLI folder

Here are steps regarding how we reproduced the problem before finding the important locations in the codebase

Steps to reproduce the behaviour:

Create a project on keyshade
Configure your CLI to tap into this project
Delete the project
Use keyshade run <> command
See the error

Expected behaviour

We would want the CLI to display what actually went wrong. The API should send the formatted message to the CLI. The CLI should print the error.

Keyshade’s Codebase

Before delving into the problem itself, and after pinpointing the critical components, we took a step back to develop a comprehensive understanding of the codebase’s architecture. Keyshade’s backend is architected with NestJS (a progressive Node.js framework) and TypeScript, leveraging Prisma as its database ORM, interfacing with a PostgreSQL relational database. It also integrates Redis for high-performance caching and publish/subscribe messaging, as well as Socket.IO for real-time, bidirectional communication between its command line interface and application programming interface.

On the frontend, the stack comprises React, TypeScript, JavaScript, and Next.js, presumably to power a performant and SEO-friendly web interface. Development workflows are streamlined through Nx (for monorepo orchestration), ESLint and Prettier (for code quality and style enforcement), alongside the TypeScript compiler for rigorous static type checking.

For security, Keyshade employs Elliptic Curve Cryptography for robust public-key encryption, JWT for token-based authentication, and API key authentication for service-to-service communication. The infrastructure is containerized via Docker and designed with cloud deployment in mind. A high-level system diagram encapsulates the overall architecture of the codebase as follows:

Here is a use case, using the components in the diagram to give you a better understanding of it.

Use Case: Developer Fetching Live Configuration Updates

Step 1: Developer Initiates CLI Command

keyshade run \--workspace my-company \--project web-app \--environment production

What happens: The developer runs the CLI command to start their application with live configuration updates from Keyshade.

The CLI documentation can be found [here](https://docs.keyshade.xyz/getting-started/installing-the-cli)

Step 2: CLI Processes Command (run.command.ts)

The run.command.ts file:

Parses the command arguments (workspace, project, environment)
Reads the local configuration file to get API credentials
Validates the required parameters using types from run.types.d.ts
Prepares to establish WebSocket connection for live updates

Step 3: WebSocket Connection Establishment

The CLI creates a websocket connection:

const socket = io('ws://api.keyshade.xyz/change-notifier')

What happens: The client connects to the NestJS backend's WebSocket gateway (change-notifier.socket.ts).

Step 4: Client Registration Request

The CLI emits the registration event:

socket.emit('register-client-app', { workspaceSlug: 'my-company', projectSlug: 'web-app', environmentSlug: 'production'});

What happens: This triggers the handleRegister method in change-notifier.socket.ts

Step 5a: Authentication

const userContext \= await this.extractAndValidateUser(client)  
Extracts API key from WebSocket headers (x-keyshade-token)

Queries PostgreSQL database via Prisma to validate the API key
Verifies user account is active

Step 5b: Authorization checks

await this.authorizationService.authorizeUserAccessToWorkspace({...})await this.authorizationService.authorizeUserAccessToProject({...})await this.authorizationService.authorizeUserAccessToEnvironment({...})

Checks if user has READ_WORKSPACE, READ_VARIABLE, READ_SECRET permissions

Validates access to the specific project

Confirms environment access rights

Step 5c: Client Registration

await this.addClientToEnvironment(client, environment.id)  
Creates entry in PostgreSQL mapping socket ID to environment ID

Adds client to Redis set for real-time notifications

Step 6: Success Response

What happens: Server sends confirmation back to CLI using the ClientRegisteredResponse interface from run.types.d.ts.

Step 7: Configuration Fetching

After successful registration:

CLI fetches initial configuration values (secrets, variables)
Application starts with these environment variables
CLI continues listening for real-time updates

Step 8: Real-Time Updates (Ongoing)

Another developer updates a secret value through the web interface.  

await this.redisSubscriber.subscribe(CHANGE\_NOTIFIER\_RSC, this.notifyConfigurationUpdate.bind(this))  
The web interface publishes change to Redis channel

All API instances receive the notification

this.server.to(clientId).emit('configuration-updated', data)

Server identifies all clients registered for that environment

Sends update notification to active CLI instances

CLI receives the configuration-updated event

Updates local environment variables

Application continues running with new values

Challenges

One specific technical challenge I encountered while attempting to solve issue 998 was figuring out why my changes in change-notifier.socket.ts were not being reflected when I tested them through the CLI. Despite updating the logic inside the Socket.IO handlers, the CLI behavior remained unchanged, and it was unclear whether the API changes were even being registered. Here is an attempt to solve this challenge and other challenges I encountered.

So I tried debugging the issue after reproducing the issue and found the debug information for keyshade issue 998

@johnny603 ➜ /workspaces/keyshade (teamAttempt/#998) $ DEBUG=* keyshade run "echo 'test'"INFO: Checking API key validity...INFO: API key is valid!INFO: Connecting to socket...socket.io-client:url parse wss://api.keyshade.xyz/change-notifier +0mssocket.io-client new io instance for wss://api.keyshade.xyz/change-notifier +0mssocket.io-client:manager readyState closed +0mssocket.io-client:manager opening wss://api.keyshade.xyz/change-notifier +0msengine.io-client:socket creating transport "websocket" +0msengine.io-client:socket options: {"path":"/socket.io/","agent":false,"withCredentials":false,"upgrade":true,"timestampParam":"t","rememberUpgrade":false,"addTrailingSlash":true,"rejectUnauthorized":true,"perMessageDeflate":{"threshold":1024},"transportOptions":{},"closeOnBeforeunload":false,"autoConnect":false,"extraHeaders":{"x-keyshade-token":"ks_a3adff2a9db8bf3c42fc817880ac4125983b83c171f0d04d"},"transports":[null],"hostname":"api.keyshade.xyz","secure":true,"port":"443","query":{"EIO":4,"transport":"websocket"},"socket":{"binaryType":"nodebuffer","writeBuffer":[],"_prevBufferLen":0,"_pingInterval":-1,"_pingTimeout":-1,"_maxPayload":-1,"_pingTimeoutTime":null,"secure":true,"hostname":"api.keyshade.xyz","port":"443","transports":["websocket"],"_transportsByName":{},"opts":{"path":"/socket.io/","agent":false,"withCredentials":false,"upgrade":true,"timestampParam":"t","rememberUpgrade":false,"addTrailingSlash":true,"rejectUnauthorized":true,"perMessageDeflate":{"threshold":1024},"transportOptions":{},"closeOnBeforeunload":false,"autoConnect":false,"extraHeaders":{"x-keyshade-token":"ks_a3adff2a9db8bf3c42fc817880ac4125983b83c171f0d04d"},"transports":[null],"hostname":"api.keyshade.xyz","secure":true,"port":"443"},"readyState":"opening"}} +0msengine.io-client:socket setting transport websocket +6mssocket.io-client:manager connect attempt will timeout after 20000 +8msengine.io-client:socket socket receive: type "open", data "{"sid":"qxJTaYpvOE4a6kIaAAB8","upgrades":[],"pingInterval":25000,"pingTimeout":20000,"maxPayload":1000000}" +756msengine.io-client:socket socket open +0mssocket.io-client:manager open +755mssocket.io-client:manager cleanup +0mssocket.io-client:socket transport is open - connecting +0mssocket.io-client:manager writing packet {"type":0,"nsp":"/change-notifier"} +1mssocket.io-parser encoding packet {"type":0,"nsp":"/change-notifier"} +0mssocket.io-parser encoded {"type":0,"nsp":"/change-notifier"} as 0/change-notifier, +0msengine.io-client:socket flushing 1 packets in socket +1msengine.io-client:socket starting upgrade probes +1msengine.io-client:socket socket receive: type "message", data "0/change-notifier,{"sid":"OpbhAQ0UYcfklhBoAAB9"}" +249mssocket.io-parser decoded 0/change-notifier,{"sid":"OpbhAQ0UYcfklhBoAAB9"} as {"type":0,"nsp":"/change-notifier","data":{"sid":"OpbhAQ0UYcfklhBoAAB9"}} +251mssocket.io-client:socket socket connected with id OpbhAQ0UYcfklhBoAAB9 +251mssocket.io-client:manager writing packet {"type":2,"data":["register-client-app",{}],"options":{"compress":true},"nsp":"/change-notifier"}+251mssocket.io-parser encoding packet {"type":2,"data":["register-client-app",{}],"options":{"compress":true},"nsp":"/change-notifier"} +0mssocket.io-parser encoded {"type":2,"data":["register-client-app",{}],"options":{"compress":true},"nsp":"/change-notifier"} as 2/change-notifier,["register-client-app",{}] +0msengine.io-client:socket flushing 1 packets in socket +1mssocket.io-client:socket draining queue +1msengine.io-client:socket socket receive: type "message", data "2/change-notifier,["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}]" +261mssocket.io-parser decoded 2/change-notifier,["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}] as {"type":2,"nsp":"/change-notifier","data":["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}]} +262mssocket.io-client:socket emitting event ["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}] +261msERROR: Error registering to API: [object Object]

I came to the conclusion that the client is sending an empty object {} in the registration message, but the server expects a ChangeNotifierRegistration object with workspace, project, and environment slugs. This results in undefined values being passed to the authorization check, causing the “Workspace undefined not found” error. The code correctly creates the registrationData object with workspaceSlug, projectSlug, and environmentSlug

But the debug output shows it’s sending an empty object {}

This suggests that the data parameter being passed to connectToSocket has undefined values for workspace, project, and environment. Our team reached out to the issue maintainer radjip-b with a question regarding our process of the issue so far.

This small technical challenge was solved.

Solution

The solution for Keyshade’s issue 998 took five and a half weeks to complete during the eight-week internship. In the remaining time, we addressed additional related issues, including issue 989 and 988, which involved adding Slack and Discord integration documentation, as well as contributing to PR 1093.

Changes Introduced by PR #1093:

Slack Integration Documentation: A brand-new “How It Works” modal was added to the Keyshade platform’s UI, providing a visual, step-by-step guide—including screenshots—on how to set up Slack integration. This includes selecting Slack in the Integrations tab, entering credentials (signing secret, Bot token, channel ID), and testing with sample secrets and variables.
Modal Improvements & Developer Feedback: The modal component underwent adjustments—such as replacing a bare <DialogClose asChild /> (which caused rendering issues) with a proper close button, and revising instructions for better clarity (“Bot Token from Slack”). Additionally, an unused function (_handleClose) was cleaned up and textual instructions were streamlined.
Discord Integration Documentation: The scope of the PR was expanded to include documentation for Discord integration as well—hence the updated title to “feat: added slack and discord integration docs.”
Internal Routing and Formatting Refactor: Responding to reviewer feedback, the team refactored the integration documentation routing logic—removing the DOCS_SLUG_MAP, simplifying routing, standardizing layout for project creation and event triggers, and cleaning up the formatting and positioning of images.
Lint Fixes: Minor linting errors (such as extraneous braces in index.tsx) were corrected to ensure code consistency and avoid build issues.

Summary Table

Area	What Changed
Docs & UI	“How It Works” modal with guide and screenshots
Component Cleanup	Dialog close functionality fixed, redundant code removed
Discord Docs	Documentation for Discord integration added
Slack Docs	Documentation for Slack integration added
Documentation Routing	Routing logic simplified, layout standardized, visuals aligned
Code Quality	Linting issues resolved

Solution Description For Issue #998

Our team resolved a bug where the socket notifier and run command error handling would display [object Object] instead of a meaningful error message. The root cause was improper formatting of the error object before logging it to the user. The fix ensures that error messages are now properly stringified or formatted, providing clearer and more helpful feedback during CLI failures. To verify the fix, you can run the keyshade run command with an invalid configuration or project setup that triggers a connection error. This update significantly improves the developer experience by making error outputs more understandable.

We determined that only three files required direct changes to address the issue:

NestJS WebSocket Gateway – The server-side component uses NestJS’s @WebSocketGateway decorator with Socket.IO to handle real-time communication between the CLI and API server. The handleRegister method in change-notifier.socket.ts processes client registration requests and can return different error message formats depending on the failure type (authentication, authorization, or database errors).
CLI Command Handler – The run.command.ts file contains the client-side logic that connects to the WebSocket server and handles the registration response. This component already included robust error parsing logic capable of handling both string and object formats, but the TypeScript interface was preventing proper type checking.
Type Checker – The run.types.d.ts file was updated to allow both string and object types, since the server can send either format.

After the maintainer approved the fix, we made additional updates to the codebase by modifying two more files: changelog.md to document the changes, and package.json to bump the semantic version to 3.2.3.

What is in the file changes

These were changes found using git’s diff command

What has been changed can also be found here

changelog.md:

🐛 Bug Fixes* **api, cli:** ambiguous error messages on keyshade run failure ([#1087](https://github.com/keyshade-xyz/keyshade/issues/1087)) ([206dda7](https://github.com/keyshade-xyz/keyshade/commit/206dda702b3f3f0cabeb73e4f74dd2ccfc962631))

apps/cli/package.json: "version": "3.2.2-stage.2",
apps/cli/src/types/command/run.types.d.ts:

message: string | object // Allow both string and object types since server can send either

apps/cli/src/commands/run.commands.ts:

Accurate WebSocket URL Parsing:
The CLI’s socket connection logic now uses the full host from the base URL (via the URL class), ensuring reliable and correct WebSocket connections especially for complex URLs.

Registration Timeout Added:A 30-second timeout was introduced for socket registration. If the server does not respond within this window, users receive a clear error message and the process exits gracefully. This prevents hanging connections and improves user feedback.

Enhanced Error Message Extraction:The registration response error handling is now more robust: It intelligently extracts meaningful error messages whether the server responds with a string, an object, or a nested structure. Attempts to parse JSON error details for more informative output. Falls back to stringifying unknown error types, ensuring users always get useful feedback. All error messages are now logged in a consistent, user-friendly format.

Minor Cleanup:Removed a redundant blank line after the socket connection initiation for improved code readability.

apps/api/src/socket/change-notifier.socker.ts:

1. Improved Error Handling & Feedback The registration process now provides much more detailed error messages to clients. Instead of a generic error or raw message, errors are parsed, logged, and emitted in user-friendly formats.

2. Custom Authentication Logic for Socket Connections The previous decorator-based guards (@UseGuards(AuthGuard, ApiKeyGuard) and @RequiredApiKeyAuthorities) have been replaced by custom methods. New methods (extractAndValidateUser and convertToAuthenticatedUser) manually handle authentication and authority checks, mimicking guard behavior but allowing for more granular error handling and feedback.

3. API Key Authority Checks The code now checks if the API key has the required authorities (READ_WORKSPACE, READ_PROJECT, and READ_ENVIRONMENT) directly within the socket handler. If the user has ADMIN authority, individual checks are bypassed.

4. Enhanced Logging Connection logs now include the presence or absence of authentication headers. Errors during registration are logged with more context, including error types and constructors.

5. Dependency Injection Updates The SlugGenerator service is now injected into the class, preparing for potential future use or slug-related operations.

6. Cleaned-Up Imports Removed unused decorators and guards, reflecting the shift to manual authentication and authority verification.

7. User Context Construction The code now constructs a detailed user context object for socket connections, including IP address, workspace, and API key authorities, improving downstream authorization and personalization.

Testing the solution

While testing Keyshade, I encountered an issue that was easy to reproduce: create a project in Keyshade, configure your CLI to connect to it, delete the project, and then run the keyshade run <> command. This sequence would trigger an unclear error message, leaving the cause ambiguous. The expected behavior was for the CLI to clearly display the actual problem, with the API sending a properly formatted error message that the CLI would then print for the user. After applying the solution, the CLI behaved as intended, providing a clear and informative error message, making the debugging process much smoother.

Another testing approach involved building custom WebSocket test clients designed to send invalid authentication data, deliberately triggering error conditions to assess system resilience. We closely monitored server-side logs to ensure errors were correctly detected and handled, while also verifying that the client could parse various error message formats. This end-to-end validation—from CLI input, through WebSocket communication and API processing, to the final CLI response—confirmed the integrity and reliability of the entire flow. This comprehensive process not only confirmed the integrity and reliability of the entire system but also successfully closed issue #998 with our PR #1087.

Conclusion

We resolved a complex WebSocket bug by fixing authentication handling, type safety, and error messaging across the backend and CLI. They used systematic debugging, targeted fixes to three core components, and custom test clients to replace vague errors with clear feedback, improve authentication timeouts, and update integration docs—showcasing technical excellence, thorough testing, and a user-focused, collaborative approach.

Product sense - Intro to Keyshade

Johnny Santamaria — Sat, 12 Jul 2025 02:42:42 +0000

Product sense is the intuitive understanding of what makes a product successful—the ability to identify user needs, spot market gaps, and build solutions that people actually want to use. It's the difference between building features and building the right features.

At its core, product sense combines three critical elements: deep empathy for users, keen awareness of market dynamics, and the judgment to prioritize what matters most. It's what separates products that solve real problems from those that collect digital dust.

The Anatomy of Strong Product Sense

Product sense isn't just about having good ideas—it's about having the right ideas at the right time. It manifests in several ways:
User-Centric Thinking: Understanding not just what users say they want, but what they actually need. This means digging beneath surface-level requests to uncover underlying pain points.

Market Awareness: Recognizing where opportunities exist in the competitive landscape and how user behavior is evolving.
Execution Wisdom: Knowing which features to build first, how to sequence development, and when to pivot based on feedback.

Keyshade: Product Sense in Action

Keyshade exemplifies strong product sense by addressing a fundamental yet often overlooked developer pain point: secure environment variable management. While many developers resort to scattered .env files or basic cloud solutions, Keyshade's creators recognized that teams needed something more robust yet accessible.

The product sense behind Keyshade reveals itself in several key decisions:
Solving a Real Problem: Rather than building another generic developer tool, Keyshade targets the specific friction developers face when managing secrets across environments. This isn't a nice-to-have—it's a daily necessity that most teams handle poorly.
Open Source Strategy: By making Keyshade open source, the team demonstrates understanding of developer culture and trust requirements. When it comes to handling sensitive data, transparency isn't just appreciated—it's essential.
Developer-First Design: The focus on seamless integration and straightforward workflows shows awareness of how developers actually work. Complex security tools often go unused because they create more friction than they solve.

Why This Matters for Product Builders

Keyshade's approach illustrates how product sense translates into tangible decisions. Instead of building a comprehensive DevOps platform (which would be feature-rich but unfocused), they identified one specific workflow that teams consistently struggle with and built a targeted solution.
This focused approach is classic product sense—recognizing that doing one thing exceptionally well often beats doing many things adequately. It's the difference between building what's technically possible and building what's strategically valuable.
The lesson for product builders is clear: strong product sense isn't about having all the answers upfront. It's about asking the right questions, staying close to user problems, and having the discipline to build solutions that matter rather than solutions that impress.

Product sense develops through experience, but it starts with genuine curiosity about user problems. Tools like Keyshade succeed because they're built by people who understand both the technical challenges and the human workflows they're trying to improve.

[Boost]

Johnny Santamaria — Mon, 23 Jun 2025 23:22:28 +0000

Meme Monday

Ben Halpern ・ Jun 23

#discuss #jokes #watercooler

Level Up Your Linux Skills

Johnny Santamaria — Mon, 23 Jun 2025 23:17:19 +0000

Deep Dive into OverTheWire Bandit

“So many books, so little time.” - Frank Zappa

How a gamified approach to cybersecurity education transforms learning from tedious to thrilling

Introduction

In today’s fast-paced digital world, cybersecurity skills are no longer optional — they’re essential. But learning them through static lessons, long-winded lectures, or dry textbooks? That often leads to burnout or boredom.

That’s where OverTheWire Bandit comes in — a brilliant learning platform that transforms traditional Linux security training into a hands-on, puzzle-based game. It’s not just engaging — it’s effective, because it teaches you by doing.

I started playing Bandit right after finishing a web development class, where I had learned how to navigate files and search through directories using Linux. Jumping into this wargame felt like leveling up — taking those foundational skills and applying them in a way that was both fun and deeply practical.

Unlike passive learning platforms, Bandit drops you straight into a live Linux environment. Every command has consequences. Every file might contain a hidden clue. Every level challenges you to solve real problems with real tools — just like in the field.

What Makes Bandit Special?

The Progressive Learning Model

Bandit follows a carefully crafted progression that mirrors real-world security scenarios. Each level builds upon the previous one, introducing new concepts while reinforcing established skills. This isn't just random command execution – it's a structured journey through the fundamentals of:

File system navigation and permissions
Text processing and pattern matching
Network protocols and services
Cryptography and encoding
Process management and privilege escalation
Version control systems

Real Environment, Real Skills

What sets Bandit apart from simulated environments is its authenticity. You're working with actual Linux systems, real SSH connections, and genuine security tools. The skills you develop aren't theoretical – they're immediately applicable to:

System administration tasks
Security auditing and penetration testing
Digital forensics investigations
DevOps and automation workflows

Technical Highlights: Key Learning Concepts

File System Mastery

Early levels focus on fundamental file operations, but with security-minded twists. You'll encounter:

# Working with special characters in filenames
cat ./-

# Handling spaces in filenames
cat "spaces in this filename"

# Finding hidden files and understanding permissions
ls -la
find . -type f -name ".*"

These aren't just command exercises – they simulate real scenarios where attackers hide malicious files or administrators need to locate specific system files.

Advanced Text Processing

As you progress, Bandit introduces sophisticated text manipulation techniques essential for log analysis and data extraction:

# Pattern matching for specific data
grep "pattern" large_file.txt

# Sorting and finding unique entries
sort data.txt | uniq -c

# Working with binary data
strings binary_file | grep "readable_text"

Cryptography in Practice

Rather than abstract mathematical concepts, Bandit presents cryptography through practical applications:

Base64 encoding/decoding for data transmission
ROT13 cipher for basic obfuscation
File compression and decompression chains
SSL/TLS communication with live services

Network Security Fundamentals

Advanced levels introduce network concepts that security professionals use daily:

# Port scanning and service identification
nmap -p port_range target

# Network communication
nc (netcat) for various protocols

# SSL/TLS client connections
openssl s_client -connect host:port

The Learning Psychology Behind Gamification

Immediate Feedback Loop

Each level provides instant validation. Success grants access to the next challenge, while failure requires problem-solving and research. This tight feedback loop maintains engagement and builds confidence progressively.

Problem-Based Learning

Instead of memorizing commands, you're solving authentic problems. Need to find a file with specific properties? You'll learn find command naturally. Need to analyze network traffic? netstat and nmap become tools, not just commands to memorize.

Community and Collaboration

The OverTheWire community fosters collaborative learning. Forums, IRC channels, and documentation encourage discussion without spoiling solutions, creating an environment where learners help each other grow.

Real-World Applications

For Aspiring Security Professionals

Bandit provides foundational skills for:

Penetration testing methodologies
Incident response procedures
Digital forensics investigations
Security operations center (SOC) work

For System Administrators

The challenges develop crucial administrative skills:

Log analysis and troubleshooting
Service configuration and management
Automation scripting capabilities
Security hardening techniques

For Developers

Modern development requires security awareness:

Secure coding practices
DevSecOps integration
Container security fundamentals
Infrastructure as code security

Advanced Concepts and Progression

Privilege Escalation

Later levels introduce sophisticated privilege escalation techniques, teaching:

SUID/SGID binary exploitation
Cron job manipulation
Service account abuse
Environment variable exploitation

Version Control Security

Git-based challenges demonstrate:

Repository forensics for sensitive data
Branch and tag analysis
Commit history investigation
Remote repository interaction

Building Your Learning Path

Getting Started Strategy

Set up your environment with proper SSH clients and tools
Document your progress – maintain notes on techniques learned
Join the community for support and additional resources
Practice consistently – regular engagement builds muscle memory

Advanced Techniques

As you progress, develop:

Automation scripts for repetitive tasks
Custom tools for specific challenges
Deep diving into manual pages and documentation
Cross-referencing with real-world security frameworks

The Technical Foundation for Cybersecurity Careers

Industry Relevance

The skills developed through Bandit directly align with industry certifications:

CompTIA Security+ foundational knowledge
CEH (Certified Ethical Hacker) practical skills
OSCP (Offensive Security Certified Professional) methodology
CISSP security management concepts

Portfolio Development

Completing Bandit demonstrates:

Self-directed learning capability
Problem-solving methodology
Technical documentation skills
Persistence and attention to detail

Conclusion: More Than Just a Game

OverTheWire Bandit represents a paradigm shift in technical education. By combining gamification with authentic security challenges, it creates an engaging learning environment that builds genuine, applicable skills.

The platform's genius lies not just in its technical content, but in its understanding of how people learn best – through doing, failing, researching, and ultimately succeeding. Each level completed isn't just a game achievement; it's a step toward genuine cybersecurity competence.

Whether you're starting your security journey or looking to sharpen existing skills, Bandit offers a structured, challenging, and ultimately rewarding path to technical mastery. The investment in time and effort pays dividends in real-world capability and career advancement.

Ready to begin your journey? The terminal is waiting, and the first challenge is just an SSH connection away.

OverTheWire Bandit

Note: Always approach learning platforms like OverTheWire with respect for their educational mission. Share knowledge and techniques, but avoid sharing specific solutions that might diminish the learning experience for others.

I suggest having a notepad ready! Whether it’s a physical notebook or a digital document, tracking your commands, mistakes, and lessons learned helps reinforce your understanding and gives you something to reference later.

[Boost]

Johnny Santamaria — Sat, 29 Mar 2025 05:28:39 +0000

Jess Lee for The DEV Team

Mar 5 '25

Join Our First-Ever WeCoded Challenge – Celebrating Underrepresented Voices in Tech Through Stories & Code!

#devchallenge #wecoded #dei #career

165

Comments 29

4 min read

Mastering Git Configuration: Essential Settings for an Efficient Workflow

Johnny Santamaria — Sat, 29 Mar 2025 05:22:57 +0000

Thorough Exploration of Git Configuration Options

“Every configuration of people is an entirely new universe unto itself.”

― Kristin Cashore

Introduction

Git is the backbone of modern software development, but its default configuration isn't always optimal for everyone. Customizing your Git configuration can significantly improve your workflow efficiency and help avoid common mistakes. In this guide, I'll walk you through essential Git configuration settings used by experienced developers, including Git core maintainers themselves.

Understanding Git Config Levels

Git configurations exist at three levels:

System-wide (/etc/gitconfig): Applied to every user on the system
Global (~/.gitconfig or ~/.config/git/config): Applied to all your repositories
Local (.git/config in each repository): Repository-specific settings

Each level overrides the previous one, giving you flexibility in how you apply configurations.

Essential User Identity Settings

Let's start with the basics. Your identity in Git is crucial for collaboration:

git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

Pro tip: You can set different identities for work and personal projects using conditional includes based on repository location:

# In ~/.gitconfig
[includeIf "gitdir:~/work/"]
    path = ~/.gitconfig-work
[includeIf "gitdir:~/personal/"]
    path = ~/.gitconfig-personal

Enhancing Usability with Aliases

Git aliases save time by shortening common commands:

git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.ci commit
git config --global alias.st status
git config --global alias.last 'log -1 HEAD'

Git core developers often use more complex aliases for powerful workflows:

git config --global alias.logline "log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit"

Preventing Common Mistakes

Some settings can save you from frustrating errors:

# Avoid pushing to wrong remote branch
git config --global push.default simple

# Require explicit branch name when creating branches
git config --global push.autoSetupRemote false

Many Git core developers recommend this safety setting:

# Warn before pushing to multiple remotes
git config --global push.recurseSubmodules check

Editor and Diff Tool Configuration

Configure your preferred editor for commit messages:

git config --global core.editor "code --wait"  # For VS Code
# or
git config --global core.editor "vim"  # For Vim

Improve diff readability:

git config --global diff.colorMoved zebra
git config --global diff.algorithm patience

Merge and Rebase Preferences

Customize your merge strategy:

# Create a backup file (.orig) when conflicts occur
git config --global merge.keepBackup true

# Auto-stash before rebasing
git config --global rebase.autoStash true

Git core developers often use this to maintain clean history:

# Enable fast-forward only merges by default
git config --global merge.ff only

Color and UI Enhancements

Make Git output more readable:

git config --global color.ui auto
git config --global color.status auto
git config --global color.branch auto

Advanced Productivity Settings

These settings are favorites among Git power users:

# Remember conflict resolutions for future merges
git config --global rerere.enabled true

# Set global gitignore file
git config --global core.excludesFile ~/.gitignore_global

An often-overlooked setting praised by Git core developers:

# Automatically prune remote-tracking branches during fetch/pull
git config --global fetch.prune true

Creating Your Personalized Config

The best configuration is one that matches your workflow. Start with these essentials and gradually add what you need:

Configure your identity first
Add safety measures to prevent mistakes
Set up time-saving aliases for commands you use often
Customize merge and rebase behavior based on your team's workflow

Remember that you can always check your current configuration with:

git config --list

And see where a specific setting is defined with:

git config --list --show-origin

Manual for the git config command could be found by:

git config --help

Conclusion

A well-configured Git setup is like a finely tuned instrument—it helps you work faster and with fewer errors. Take the time to customize your Git configuration, and you'll reap the benefits with every commit.

Whether you're a Git beginner or experienced developer, these settings provide a solid foundation for an efficient workflow. Start implementing them today, and watch your Git experience transform.

What's your favorite Git configuration trick? Share in the comments below!

Enhancing Asset Visualization in Dagster: A UI Filtering Implementation

Johnny Santamaria — Thu, 27 Feb 2025 19:28:00 +0000

"The ability to simplify means to eliminate the unnecessary so that the necessary may speak." — Hans Hofmann

Introduction

Over the past four weeks in February of 2025, I have contributed to Dagster with a team of other two CTI members. Dagster is a powerful data orchestration tool that organizes data from multiple sources. During the first week of this micro-internship, we were introduced to the project and learned about collaborating in open-source development. The tech stack we were given to examine in the issue we were given was Python and Docker. Since I had no experience with Docker, I looked at the official documentation to console on what it is and how it is used in Dagster.

CTI is Computing Talent Initiative which is a program designed to support and accelerate the careers of students from underrepresented backgrounds in computing.

Dagster is an important application for modern data orchestration because it provides a robust and scalable way to develop, test, and deploy data pipelines. Unlike traditional workflow schedulers, Dagster offers a declarative approach to defining data dependencies, ensuring reliability and maintainability. It integrates seamlessly with various data tools, supports modular pipeline development, and enhances observability through built-in monitoring and logging features. By enabling teams to track data lineage, enforce data quality checks, and automate workflows, Dagster helps organizations improve efficiency, reduce errors, and ensure data integrity in complex data engineering processes.

Dagster is typically used by data engineers, data scientists, and DevOps teams that need to build, manage, and monitor reliable data pipelines. These teams use it to automate workflows, ensure data quality, and track dependencies across complex data systems. For example, a financial services company could use Dagster to orchestrate daily data ingestion from multiple stock market APIs, validate the data for accuracy, and generate real-time reports for analysts, ensuring seamless and error-free data processing.

Here is how Dagster is used

Once the product is installed, run it on your machine and then click on assets -> Global asset lineage
Click on Materialize to run the pipeline of the assets
Click View and Run Details

This is where you can view asset logs and metadata
1. provide essential information for tracking, managing, and maintaining assets efficiently; like images or a piece of code
You can change how the assets are displayed while the program is running by toggling the view buttons on the top left corner

Dagster has around 3.4k users right now to manage data pipelines!

The issue (#26669)

The issue discusses the need for customizable asset ordering in Dagster's UI, particularly in graphs. Currently, the order of asset kinds (e.g., Python, BigQuery, GCS) is unclear and changes unpredictably when asset properties (such as location) are modified. The user wants to define a consistent order and have more flexibility in customization, including the ability to use custom icons or emojis beyond the existing compute_kind field.

Link to the issue: https://github.com/dagster-io/dagster/issues/26669

This issue is important to the Dagster open-source project because it enhances usability and clarity in visualizing data assets. Allowing users to customize the order and representation of asset kinds improves consistency, making data pipelines easier to interpret and manage. This flexibility is especially valuable for teams working with multiple asset types across different regions or services. Addressing this issue would enhance the user experience, making Dagster more adaptable to diverse workflows and increasing its adoption among data engineers.

I was with two other teammates to tackle the issue.

Files that may support the issue

The following folders are central to the filtering functionality:

js_modules/dagster-ui/packages/ui-core/src/ui/Filters: This folder contains the necessary code for implementing different filter types within the UI.
js_modules/dagster-ui/packages/ui-core/src/ui/BaseFilters: Here, the core base filter logic resides, acting as a foundation for all other filtering operations.
js_modules/dagster-ui/packages/ui-core/src: This broader directory encompasses the various filter components, including those related to emojis and icons, which are key for visualizing the data filtering process.

Files for Filtering and Asset Management

useKindFilter.tsx

This file manages a filter labeled kind, allowing the user to filter based on specific asset types. It is a vital component that ensures users can easily categorize assets based on predefined kinds, streamlining the process of narrowing down the dataset.

#### useStaticSetFilter.tsx

Found through React imports, this file is responsible for processing how filters are applied across the UI. It manages the logic that enables users to filter assets according to static sets, ensuring a flexible and efficient filtering system.

#### useStaticSetFilterSorte.oss.tsx

Although this file is invoked within the previous one, it primarily returns null, which suggests it either serves as a placeholder for further functionality or is conditionally used based on certain criteria.

#### AssetGraphFilterBar.oss.tsx

The AssetGraphFilterBar plays a critical role in the rendering of assets. It integrates the filters and determines how assets are displayed on the user interface, contributing to the visual aspect of the filtering system.

Dagsters Codebase

When you peek under the hood of Dagster, you'll find a fascinating ecosystem of technologies working in harmony to deliver robust data orchestration. Let's explore the key components that make this powerful platform tick.

The Foundation: Core Infrastructure

At its heart, Dagster relies on a Backend-as-a-Service (BaaS) infrastructure, eliminating the need for manual backend setup. Docker containers ensure consistent deployment across environments, while Vercel handles frontend and serverless applications with ease.

Data Processing Powerhouse

The data processing capabilities are particularly impressive. Spark and Dask work together to handle distributed computing and scale data science workflows. For analytical queries, DuckDB provides lightning-fast in-memory database operations. When it comes to streaming data, Kafka steps in as the backbone for real-time processing.

Modern Frontend Architecture

The user interface is built on React, enhanced with Next.js for server-side rendering. Style management is handled through a combination of Sass and Styled Components, providing a flexible and maintainable approach to CSS. For bundling and optimization, Webpack and Rollup ensure efficient delivery of frontend assets.

AI and Machine Learning Integration

What sets Dagster apart is its embrace of cutting-edge AI technologies. The platform integrates with:

OpenAI and Anthropic for natural language processing
PyTorch for deep learning capabilities
SciKit Learn for traditional machine learning tasks
Hugging Face Hub for access to a vast repository of models and datasets

Monitoring and Analytics

Reliability is crucial for data orchestration, and Dagster takes this seriously with:

Prometheus for metrics collection and analysis
Sentry for error tracking and performance monitoring
PostHog and Mixpanel for user behavior analytics
Apache Airflow integration for workflow management

Developer Experience

The development experience hasn't been overlooked. Tools like iPython provide an enhanced development shell, while Pytest and Jest ensure code quality through comprehensive testing. For documentation, Docusaurus and Sphinx generate clear, maintainable technical documentation.

Data Visualization and Reporting

Data visualization is powered by a robust stack including:

Matplotlib for core plotting capabilities
Seaborn for statistical visualizations
Graphviz for structured data representation
Rive for interactive motion graphics

API and Communication

The API layer leverages GraphQL for flexible data fetching, while FastAPI and Flask provide high-performance web frameworks. For handling concurrent requests, aiohttp enables asynchronous communication.

Scientific Computing Foundation

At its core, Dagster benefits from Python's scientific computing ecosystem:

NumPy for numerical operations
SciPy for scientific computations
Arrow for precise handling of dates and times

This comprehensive technology stack allows Dagster to handle complex data orchestration tasks while remaining flexible and maintainable. Whether you're dealing with big data processing, machine learning workflows, or real-time analytics, Dagster's thoughtfully chosen technology stack provides the tools needed for success.

The beauty of this architecture lies not just in the individual components, but in how they work together to create a seamless experience for data engineers and scientists. As data orchestration needs continue to evolve, Dagster's modern technology stack positions it well for future growth and adaptation.

Here is a System diagram of the codebase regarding the issue

The workflow of how the codebase functions regarding the issue

User Interface Layer

* The user starts by interacting with the Asset Graph UI (A)

* They access the filtering functionality through the Asset Graph Filter Bar (B)

* The Kind Filter Component (C) provides the interface for selecting asset types

Filter Logic Layer

* When a user selects filter criteria, the useKindFilter Hook (D) is triggered

* This hook coordinates with useStaticSetFilter (E) for managing filter states

* Filter Configuration (F) defines how different asset kinds should be displayed and sorted

Data Processing Layer

* Asset Kind Sorting (G) handles the prioritization of asset types (like python, bigquery, gcs)

* Asset Metadata Processing (H) processes the raw asset data

* Filter State Management (I) maintains the current state of selected filters

Storage Layer

* Asset Definitions (J) store the core information about each asset

* Asset Metadata Store (K) contains additional information like kinds and compute resources

Here's a specific use case workflow:

A user wants to filter assets to show only Python and BigQuery assets:

* They start at the Asset Graph UI (A)

* Click on the filter icon in the Asset Graph Filter Bar (B)

* The Kind Filter Component (C) displays available asset types

When the user selects "python" and "bigquery":

* useKindFilter (D) receives these selections

* The hook applies the prioritized sorting defined in the code

* useStaticSetFilter (E) updates the filter state

The Data Processing Layer then:

* Sorts the assets according to the prioritized order (G)

* Processes the metadata to match the filter criteria (H)

* Updates the filter state (I)

Finally:

* The UI updates to show only the selected asset types

* Custom icons are applied (python shows code\_block icon, bigquery shows graph icon)

* The filtered view maintains the prioritized ordering

This workflow implements the original issue's custom ordering and visualization requirements, ensuring consistent asset ordering and proper representation of different asset kinds in the UI.

Challenges Overcame

When I first started working with Dagster, I encountered an interesting challenge during the installation process. Despite following the documentation carefully on my Mac M2, including some specific commands for M2 compatibility, I ran into a puzzling error with the assets.py file.

After pushing my changes to a separate branch and returning to the project a few days later, I was greeted with a DagsterUserCodeLoadError. The error message was clear but cryptic at first glance:

dagster._core.errors.DagsterUserCodeLoadError: Error occurred during the loading of Dagster definitions in
executable_path=/workspaces/dagster/venv/bin/python, python_file=quickstart/assets.py, working_directory=/workspaces/dagster
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_grpc/server.py", line 417, in init
self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
^^^^^^^^^^^^^^^^^^^
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_grpc/server.py", line 239, in init
with user_code_error_boundary(
File "/usr/local/python/3.12.1/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_core/errors.py", line 299, in user_code_error_boundary
raise new_error from e
The above exception was caused by the following exception:
FileNotFoundError: [Errno 2] No such file or directory: 'quickstart/assets.py'
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_core/errors.py", line 289, in user_code_error_boundary
yield
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_grpc/server.py", line 250, in init
loadable_targets = get_loadable_targets(
^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_grpc/utils.py", line 41, in get_loadable_targets
else loadable_targets_from_python_file(python_file, working_directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_core/workspace/autodiscovery.py", line 24, in loadable_targets_from_python_file
loaded_module = load_python_file(python_file, working_directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/dagster/venv/lib/python3.12/site-packages/dagster/_core/code_pointer.py", line 73, in load_python_file
os.stat(python_file)

What I did to solve this challenge

1. Version Control

My first instinct was to check if this was a version mismatch issue. I systematically:

Uninstalled the current Dagster version
Installed specifically version 1.8.4
Updated the requirements.txt file
Verified the version using Python's import system

2. File Location Detective Work

When the version fix didn't solve the issue, I realized it might be a path problem. The error message suggested that Dagster couldn't find assets.py, so I:

Used the file finder to check my current directory
Discovered I was in the wrong folder
Attempted to run the command with the correct path: dagster dev -f quickstart/assets.py

3. Documentation Deep Dive

Finally, I traced the issue back to a potential documentation unclear point. The docs stated "navigate to your project's root directory," but this needed more specificity. I:

Reviewed the environment setup steps
Followed the "getting started" section
Realized I needed to be in the "dagster-quickstart" directory specifically

This is the Dagster documentation I consulted for the entirety of this issue, as well as tackling this challenge

Solution

The breakthrough came when I used Git's find command to locate the exact path of the assets.py file:

find /quickstart -name "assets.py"

Lessons learned

This experience taught me several valuable lessons about working with Dagster:

Always verify file paths when working with Dagster's file-based configuration
Use Git commands to help locate files in complex project structures
Pay close attention to the working directory when running Dagster commands
Keep track of Dagster versions to ensure compatibility

For anyone else setting up Dagster, I recommend double-checking your working directory and using the find command if you encounter similar path-related errors. Sometimes the simplest solutions are the most effective!

Solution of issue #26669 - A Breakdown

The solution has two main components:

Prioritized Asset Ordering

Implemented a priority system for asset kinds (python, bigquery, gcs)
Created a consistent sorting mechanism that maintains order even when asset properties change
Used React's useMemo hook for performance optimization

Custom Icon Mapping

Added a configurable icon mapping system for different asset types
Implemented fallback to default icons for undefined asset kinds
Enhanced visual distinction between different asset types

Technical Implementation

The solution leverages several key technologies from our stack:

React Components and Hooks

Used React's useMemo hook for efficient asset kind sorting
Implemented custom hooks for state management
Integrated with Dagster's existing UI component library

Styled Components

Utilized Styled Components for consistent visual styling
Implemented flexible layout using Box component
Maintained visual hierarchy through consistent spacing

The useKindFilter.tsx file is central to the repositories issue since it handles two critical aspects of the Dagster UI enhancement:

Asset Type Filtering

It's a React custom hook that manages how different types of assets (Python, BigQuery, GCS, etc.) are filtered in the UI
Controls what users see when they want to view specific types of assets in their data pipelines

Asset Ordering

This file contains the logic for how assets are sorted and displayed
Implements the prioritization system that determines which asset types appear first
Solves the original issue where asset ordering was inconsistent and unpredictable

The main issue was that users had no control over how their assets were ordered in the UI, and the ordering would change unexpectedly. By modifying useKindFilter.tsx, we could add:

Consistent ordering rules
Custom icons for different asset types
A more predictable filtering experience

Think of it as the "traffic controller" for how assets appear and are organized in Dagster's interface - it's where we define the rules for what shows up where and how it looks.

Pathway to Pull request #26669: Asset Filtering and Metadata System

The main parts of the tech stack we needed:

Python, hosts the UI of the assets
React, displays the assets

Our team tackled the challenge of enhancing asset filtering and visualization in our data pipeline management system. The solution involved several key components:

Customization of Icons: We expanded the icon options in the useKindFilter.tsx file, allowing users to associate custom icons with different asset types. This improvement leverages the @dagster-io/ui-components library, a core part of our tech stack for building intuitive user interfaces.
Asset Metadata Enhancement: My teammate Brent made significant contributions by introducing a new location field to the asset metadata. This update required modifications to several core Dagster files:

* python\_modules/dagster/dagster/\_core/assets.py  

* python\_modules/dagster/dagster/\_core/definitions/asset\_out.py  

* python\_modules/dagster/dagster/\_core/definitions/decorators/asset\_decorator.py

These changes allow us to specify where an asset is hosted, enhancing our system's ability to track and manage assets across different locations.
Filter Optimization: We implemented a prioritized sorting system for asset kinds in the useKindFilter hook. This improvement ensures that commonly used asset types like 'python', 'bigquery', and 'gcs' appear at the top of the filter list, improving user experience and efficiency.
Integration with Existing Components: Our solution integrates seamlessly with the Asset Catalog component of our system diagram. By enhancing the filtering and metadata capabilities, we've improved how users interact with and visualize assets within the catalog.

To verify the functionality of our solution, we ran a series of tests:

We created a test pipeline to ensure the new icon customizations rendered correctly in the UI.
We manually tested the asset kind filter to confirm that prioritized kinds appeared at the top of the list.
While Brent's changes to the asset metadata weren't fully tested, we plan to create comprehensive unit tests to verify the correct handling of the new location field.
We ran the local development version of the app in the Dagster documentation in the section titled “Developing the Dagster webserver/UI”

Here's a code snippet showcasing the icon customization in useKindFilter.tsx:

const customIcons: {[key: string]: IconName | string} = {
python: 'code_block',
bigquery: 'graph',
gcs: 'cloud',
bug: 'bug',
calendar: 'calendar'
// More icons can be added here
};

// Use custom icon or emoji if available, otherwise default to 'compute_kind'
const icon = customIcons[value.value] || 'compute_kind';

return (
<Box flex={{direction: 'row', gap: 4, alignItems: 'center'}}>
{typeof icon === 'string' ? <span>{icon}</span> : <Icon name={icon} />}
<TruncatedTextWithFullTextOnHover tooltipText={value.value} text={value.value} />
</Box>
);

This code allows for easy expansion of custom icons for different asset types, improving the visual representation of assets in our system.

While we haven't submitted a pull request yet, we plan to do so once we've completed thorough testing of all components, especially the new asset metadata features introduced by Brent.

—--

To-Do after meeting with our CTI Codeday assigned mentor: Dhiraj Patil

Add customization of icons, be able to customize what icons users can use in icon.tsx
More icons in useKindFilter.tsx
Run a pipeline to see how the issue functions

Create a pull request with the work so far with tests

File I changed:

useKindFilter.tsx

The file with what code I changed:

import {Box, Icon, IconName} from '@dagster-io/ui-components';
import {useMemo} from 'react';

import {COMMON_COLLATOR} from '../../app/Util';
import {TruncatedTextWithFullTextOnHover} from '../../nav/getLeftNavItemsForOption';
import {StaticBaseConfig, useStaticSetFilter} from '../BaseFilters/useStaticSetFilter';

const emptyArray: any[] = [];

export const useKindFilter = ({
allAssetKinds,
kinds,
setKinds,
}: {
allAssetKinds: string[];
kinds?: null | string[];
setKinds?: null | ((s: string[]) => void);
}) => {
// Sort asset kinds with prioritized kinds first
const sortedAssetKinds = useMemo(() => {
// Define prioritized kinds
const prioritizedKinds = ['python', 'bigquery', 'gcs'];
return [
...prioritizedKinds,
...allAssetKinds.filter((kind) => !prioritizedKinds.includes(kind)).sort((a, b) => COMMON_COLLATOR.compare(a, b)),
];
}, [allAssetKinds]);

return useStaticSetFilter<string>({
...BaseConfig,
allValues: useMemo(
() =>
sortedAssetKinds.map((value) => ({
value,
match: [value],
})),
[sortedAssetKinds],
),
menuWidth: '300px',
state: kinds ?? emptyArray,
onStateChanged: (values) => {
setKinds?.(Array.from(values));
},
canSelectAll: true,
});
};

export const getStringValue = (value: string) => value;

export const BaseConfig: StaticBaseConfig<string> = {
name: 'Kind',
icon: 'compute_kind',
renderLabel: (value) => {
// Define custom icons or emojis for specific kinds
const customIcons: {[key: string]: IconName | string} = {

// icons pulled from icon.tsx
python: 'code_block', // Emoji for python
bigquery: 'graph', // Emoji for bigquery
gcs: 'cloud', // Emoji for gcs
bug: 'bug',
calendar: 'calendar'

// Add more custom icons or emojis here
};

// Use custom icon or emoji if available, otherwise default to 'compute_kind'
const icon = customIcons[value.value] || 'compute_kind';

return (
<Box flex={{direction: 'row', gap: 4, alignItems: 'center'}}>
{typeof icon === 'string' ? <span>{icon}</span> : <Icon name={icon} />}
<TruncatedTextWithFullTextOnHover tooltipText={value.value} text={value.value} />
</Box>
);
},
getStringValue,
getKey: getStringValue,
matchType: 'all-of',
};

export function useAssetKindsForAssets(
assets: {definition?: {kinds?: string[] | null} | null}[],
): string[] {
return useMemo(
() =>
Array.from(new Set(assets.map((a) => a?.definition?.kinds || []).flat())).sort((a, b) =>
COMMON_COLLATOR.compare(a, b),
),
[assets],
);
}

One of my teammates made a change toward the issue solution

“I introduced a new location field to the asset metadata to specify where an asset is hosted, I then updated relevant classes and decorators to support the location, I modified three files:

python_modules/dagster/dagster/_core/assets.py

python_modules/dagster/dagster/_core/definitions/asset_out.py

-I included a location parameter to the AssetOut class

-Modified the constructor to handle the location field and ensure it is correctly passed to the asset metadata.

python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py

-Modified the asset decorator to hand the location parameter

-Made sure that the location is correctly propagated in the asset definition

Unfortunately, I wasn't able to test these changes for now, I might do so in the following days, I will submit a new pull request if I succeed”

Conclusion

Throughout this four-week micro-internship, my team and I achieved a lot on the topic given to us in Dagster, even though we could not finish it fully. We learned and applied Python and Docker in our tasks, and even though I had no experience with Docker at first, I soon knew about it through the official documentation that I went through. Collaboration with an open-source community provided invaluable hands-on experience in integration of a wider group of code, debugging, and problem-solving. Our mentor, Dhiraj Patil, acknowledged our development, reaffirming the value of our work. Through this process, my data orchestration expertise further gained strength while both my technical ability and teamwork potential were established well for potential projects in the open-source community.

Following is the link to our solution:

https://github.com/dagster-io/dagster/pull/28007

Mastering Bug Identification Using Git Bisect Techniques

Johnny Santamaria — Fri, 14 Feb 2025 05:36:04 +0000

A deep dive into dissecting a line of code

“Beware of bugs in the above code; I have only proved it correct, not tried it.”

― Donald Knuth

Introduction

Through my experience debugging complex codebases and helping teams track down elusive bugs, I've found git bisect to be an invaluable tool. This guide explores how this powerful command can save developers hours of manual debugging time by using binary search to pinpoint exactly when a bug was introduced.

What is Git Bisect?

Git bisect is a debugging command that uses binary search to find the commit that introduced a bug. Instead of manually checking every commit, git bisect helps you identify the problematic commit in a logarithmic number of steps.

How Git Bisect Works Under the hood

Git bisect works by traversing Git's internal commit graph structure. Here's what happens:

Git maintains a Directed Acyclic Graph (DAG) of commits
Each commit node contains:
- SHA-1 hash (commit identifier)
- Parent commit reference(s)
- Timestamp
- Author information
- Complete snapshot of the codebase

When you start bisect, Git:

Creates a .git/BISECT_LOG file to track the bisect session
Stores the following information:

* List of "good" commits

* List of "bad" commits

* Current bisect state

* Reference to the original HEAD

Git uses this binary search algorithm to find the first bad commit

function findFirstBadCommit(commits):
    start = oldest_good_commit
    end = newest_bad_commit

    while (end - start) > 1:
        middle = start + (end - start) / 2
        checkout(middle)

        if user_marks_as_bad():
            end = middle
        else:
            start = middle

    return end  # First bad commit

But what makes a bad commit?

A "bad" commit in git bisect is entirely determined by YOU, the developer. There's no automatic detection - a commit is "bad" if it exhibits the problem you're trying to track down. Examples include:

A feature stops working
A test starts failing
Performance degrades significantly
UI elements render incorrectly
The security vulnerability is present
Any undesired behavior appears

How Git Bisect Works Over the hood

Start the bisect process:
```
git bisect start
```
Mark the current (broken) state:
```
git bisect bad
```

Mark a known good commit:

git bisect good <commit-hash> # you can find commit hashes using "git log

G”it will automatically check commits between these points, and you mark each as 'good' or 'bad' until the problematic commit is found:
```
git bisect good  # If the current commit works
git bisect bad   # If the current commit has the bug
```

Real-World Example

Let's say you have a web application where the login button suddenly stopped working. You know it was working a month ago, but with hundreds of commits since then, finding the exact change that broke it would be like finding a needle in a haystack.

Here's how git bisect helps:

git bisect start
git bisect bad                          # Current version is broken
git bisect good release-2024-01-15      # Last known working version

# Git checks out a commit halfway between these points
# You test the login button

git bisect good                         # If it works
# OR
git bisect bad                          # If it's broken

# Repeat until Git identifies the problem commit

Best Practices

Create a test script
- Automate the process with git bisect run ./test-script.sh
- Ensures consistent testing across commits
Keep commits atomic
- Small, focused commits make bisect more effective
- Easier to identify exactly what change caused the issue
Document the findings
- Record the problematic commit and root cause
- Share learnings with the team to prevent similar issues

Benefits

Time Efficiency
- Binary search is dramatically faster than linear searching
- Find bugs in O(log n) steps instead of O(n)
Precision
- Pinpoints the exact commit that introduced the bug
- Shows who made the change and why
Learning Opportunity
- Understanding what caused bugs helps prevent future ones
- Improves team's code quality awareness

Conclusion

Git bisect transforms bug hunting from a tedious manual process into an efficient, systematic approach. Whether you're dealing with regression bugs, performance issues, or unexpected behavior, git bisect is your detective tool for tracking down when and why things went wrong.

Useful Bisect Commands to Remember

git bisect start              # Begin the bisect process
git bisect bad               # Mark current version as broken
git bisect good <commit>     # Mark a known good commit
git bisect reset            # End the bisect session
git bisect run <script>     # Automate the process

Want to learn how to create good commits?

Git in ~5 minutes

Learn to Make a Location Tracking App file Using React and Mapbox

Johnny Santamaria — Sun, 09 Feb 2025 02:04:49 +0000

Easy guide to developing the map component in a React location tracking app

If you want to see yesterday's challenge: Building a leaderboard component

If you're going to try the challenge yourself: DailyUI

Overview

Today we will create a location tracking app using React and Mapbox. Our application will allow users to share their location with friends and track each other's movements in real time. We will build a user interface using React and utilize Mapbox, a mapping and location cloud platform, for the mapping and geolocation features.

What is Mapbox?

Mapbox is a powerful mapping and location cloud platform that offers a comprehensive suite of tools and services for building custom, interactive maps, location-based applications, and data visualizations. It is an open-source platform designed to be highly customizable and scalable, making it suitable for a wide range of industries, including transportation, logistics, real estate, and social networking.

now let’s get to building!

Prerequisites

Before getting started, make sure you have the following:

Node.js installed
- Used to run Javascript everywhere
A Mapbox account and access token
Terminal to write commands

Setup

Begin by creating a new React project using Create React App:

npx create-react-app location-tracker
cd location-tracker

This command uses npx, a package runner tool that comes with Node.js, to execute the create-react-app command. This popular CLI tool generates a new React project with a default configuration and no build configuration required. In this case, location-tracker is the project name. The next line changes the directory to the location-tracker folder where the project will be.

Install the required dependencies:

yarn add react-mapbox-gl mapbox-gl geolocation react-map-gl-geocoder

This command installs the required dependencies for building a React application that uses Mapbox and geolocation features. Here's a brief explanation of each package:

react-mapbox-gl: A React binding library for Mapbox GL JS, which allows you to create interactive, customizable maps within your React application using WebGL
mapbox-gl: The core Mapbox GL JS library that provides the mapping functionality, including map rendering, interactive map controls, and support for various map sources and styles
geolocation: A package that enables you to retrieve the user's current geolocation coordinates (latitude and longitude) in the browser using the HTML5 Geolocation API
react-map-gl-geocoder: A React component that provides geocoding functionality for Mapbox GL JS maps. It allows you to convert addresses and place names into map coordinates and vice versa, enabling search and location input in your application

Implementing the Mapbox app

Create a Map component in your src directory:

// components/Map.js

import React, { useRef, useEffect } from "react";
import MapGL from "react-mapbox-gl";
import { geolocate } from "geolocation";
import MapGlGeocoder from "react-map-gl-geocoder";

const Map = (
) => {
  const mapRef = useRef(null);
  const geocoderRef = useRef(null);

  useEffect(() => {
    const map = new MapGL(mapRef.current, {
      accessToken: "[your_mapbox_access_token]",
      style: "mapbox://styles/mapbox/streets-v11",
    });

    const geocoder = new MapGlGeocoder({
      map,
      accessToken: "[your_mapbox_access_token]",
    });

    geocoderRef.current = geocoder;

    // Geolocate user
geolocate().then(({ coords }) => {
      const { latitude, longitude } = coords;
      map.jumpTo({ center: [longitude, latitude], zoom: 15.5 });
    });

    // Cleanup
return () => {
      map.remove();
    };
  }, []);

  return (
    <>
      <MapGL
        ref={mapRef}
        width="100vw"
        height="100vh"
        onViewportChange={() => geocoderRef.current.update()}
      />
      <MapGlGeocoder ref={geocoderRef} />
    </>
  );
};

export default Map;

Update your App.js to include the Map component:

// App.js

import React from "react";
import Map from "./Map";

const App = (
) => {
  return (
    <div className="App">
      <Map />
    </div>
  );
};

export default App;

Adding User Location Sharing and Tracking

Implement a backend API for user authentication and sharing locations. This will require knowledge of backend development using a technology such as Node.js, Express.js, and MongoDB.
Add user authentication to the React application using libraries like React Hooks for authentication.
Implement features to share and track users' locations on the map by fetching user coordinates from the backend API.

Here is what the entire structure would look like in a repository

- public
  - index.html
- src
  - assets
    - logo.svg
    - images
  - components
    - Map.js
    - MapGeocoder.js
    - UserLocationMarker.js
    - LoginForm.js
    - RegistrationForm.js
    - UserProfile.js
  - pages
    - Home.js
    - Login.js
    - Register.js
    - UserProfile.js
  - styles
    - App.css
    - Map.css
  - index.js
  - App.js
  - setUpTests.js
- .gitignore
- package.json
- README.md

public: This folder contains the index.html file, which serves as the entry point of the application.
src: The source code of your application resides here. It contains the following subfolders:

assets: Houses static files like images, logos, and icons.
components: Contains reusable UI components, including Map.js, MapGeocoder.js, UserLocationMarker.js, LoginForm.js, RegistrationForm.js, and UserProfile.js.
pages: Stores top-level React components, such as Home.js, Login.js, Register.js, and UserProfile.js.
styles: Contains CSS stylesheets for your components, like App.css and Map.css.

Root Files: Inside the src folder, you'll find the following important files:

index.js: The entry point of your application, responsible for rendering the top-level App component.
App.js: The main component that serves as the container for your entire application.
setUpTests.js: A file for setting up testing environments, if needed.

Root Files in Project Directory: The following files are located in the root directory:

.gitignore: A file specifying which files and directories should be ignored by Git.
package.json: The main project configuration file for Node.js and React applications, containing dependencies, scripts, and metadata.
README.md: A file providing general information, installation instructions, and usage details for your project.

Conclusion

In this tutorial, we learned how to create a basic location-tracking application using React and Mapbox. We implemented a map with geolocation capabilities, which is the foundation for user location sharing and tracking. To further enhance the app, consider adding features like search, user markers, and detailed user information. Happy coding!

Here you just saw only a few files made, here are some notes

Creating small files in software development, particularly in a React project, offers several benefits that can improve code maintainability, collaboration, and overall project organization. Here are some reasons why developers may choose to focus on small files:

Modularity: Smaller files help promote modularity in code, making it easier to separate concerns, test, and reuse code. This approach also helps with the Single Responsibility Principle, which suggests that a module or function should have only one responsibility.
Code Readability: Small files with a specific focus make the codebase more readable and easier to navigate, allowing developers to quickly understand the purpose of each file and its contents.
Collaboration: In a team setting, smaller files simplify collaboration by enabling multiple developers to work on different components, services, or features simultaneously, with fewer merge conflicts.
Performance: While modern development tools like Webpack and rollup.js can efficiently handle large files, splitting the code into smaller chunks can sometimes help with tree-shaking, lazy-loading, and improving initial load times, especially in browser-based applications.
Debugging and Testing: It's easier to debug and test small, focused files compared to large monolithic files. Developers can pinpoint issues faster and ensure that each component or function works as expected, leading to better code quality and fewer bugs.
Code Reusability: Small files with specific functionality can be easily reused across a project or even in other projects. This leads to a more maintainable and DRY (Don't Repeat Yourself) codebase.

In summary, focusing on creating small, modular files in a React project can enhance code organization, readability, and maintainability, and improve collaboration among team members.

Building a Dynamic Game Leaderboard Modal with React and Tailwind CSS

Johnny Santamaria — Sat, 08 Feb 2025 23:37:42 +0000

Explore the steps to create a sleek and adaptive leaderboard modal using React and Tailwind CSS.

“In a world that’s changing really quickly, the only strategy that is guaranteed to fail is not taking risks.” – Mark Zuckerberg

Mark Zuckerberg is the co-founder and CEO of Facebook, who has seen firsthand how competitive elements in social platforms, including gaming, drive user engagement and innovation. The quote is from an October 2011 interview at Y Combinator’s Startup School in Palo Alto, California.

If you want to see yesterday's challenge: Analytics in React

If you're going to try the challenge yourself: DailyUI

So I missed a total of two challenges, life happens…

So I came up with a prompt combining the idea of creating a leaderboard with the idea of creating a popup overlay screen

Prompt: Create a popup of a leaderboard

Is it a for a game? politics? movies? How would the overlay be closed?

So today, we're designing a game leaderboard popup, combining engaging competitive elements with smooth user interaction. I've created an interactive popup that displays player rankings while maintaining performance and accessibility.

The interface should include:

Animated popup overlay with a backdrop
Player rankings with scores
Visual indicators for top positions
Player avatars and status
Smooth transitions
Close button functionality
Responsive design

The solution 💡

Here's the React component that brings together these requirements using Tailwind CSS and Lucide icons:

{/* LeaderBoardPopup.js */}

import React, { useState } from 'react';
import { X, Trophy, Medal } from 'lucide-react';

const LeaderboardPopup = () => {
  const [isOpen, setIsOpen] = useState(true);

  const leaderboardData = [
    { rank: 1, username: "ProGamer123", score: 25000, wins: 48 },
    { rank: 2, username: "NinjaWarrior", score: 23450, wins: 45 },
    { rank: 3, username: "PixelMaster", score: 22100, wins: 42 },
    { rank: 4, username: "GameKing99", score: 21000, wins: 39 },
    { rank: 5, username: "StarPlayer", score: 20500, wins: 37 }
  ];

  const getMedalColor = (rank) => { {/* Use CSS to edit the color */}
    switch(rank) {
      case 1: return "text-yellow-500";
      case 2: return "text-gray-400";
      case 3: return "text-amber-600";
      default: return "text-gray-600";
    }
  };

  if (!isOpen) return null;

  return (
    <div className="fixed inset-0 bg-black bg-opacity-50 flex items-center justify-center">
      <div className="bg-white rounded-lg shadow-xl w-full max-w-md mx-4 relative overflow-hidden">
        {/* Header */}
        <div className="bg-indigo-600 p-4 flex justify-between items-center">
          <div className="flex items-center space-x-2">
            <Trophy className="text-white" size={24} />
            <h2 className="text-xl font-bold text-white">Top Players</h2>
          </div>
          <button 
            onClick={() => setIsOpen(false)}
            className="text-white hover:text-gray-200 transition-colors"
          >
            <X size={24} />
          </button>
        </div>

        {/* Leaderboard Content */}
        <div className="p-6">
          {leaderboardData.map((player) => (
            <div 
              key={player.rank}
              className="flex items-center justify-between mb-4 p-3 bg-gray-50 rounded-lg hover:bg-gray-100 transition-colors"
            >
              <div className="flex items-center space-x-4">
                <div className="flex items-center justify-center w-8">
                  {player.rank <= 3 ? (
                    <Medal className={getMedalColor(player.rank)} size={24} />
                  ) : (
                    <span className="text-gray-600 font-medium">{player.rank}</span>
                  )}
                </div>
                <div>
                  <p className="font-semibold text-gray-800">{player.username}</p>
                  <p className="text-sm text-gray-500">{player.wins} wins</p>
                </div>
              </div>
              <div className="text-right">
                <p className="font-bold text-indigo-600">{player.score.toLocaleString()}</p>
                <p className="text-xs text-gray-500">points</p>
              </div>
            </div>
          ))}
        </div>

        {/* Footer */}
        <div className="bg-gray-50 p-4 border-t">
          <p className="text-center text-sm text-gray-600">
            Rankings update every hour
          </p>
        </div>
      </div>
    </div>
  );
};

export default LeaderboardPopup;

Note: This code looked like HTML didn’t it?

React uses a syntax called JSX (JavaScript XML), which allows you to write HTML-like code within JavaScript. Although it looks like HTML, it's actually a syntactic sugar over JavaScript and isn't pure HTML. JSX is compiled into JavaScript at runtime, where React elements are created.

To use this component in your application:

{/* app.js */}

import LeaderboardPopup from './LeaderboardPopup';

function App() {
  return (
    <div className="min-h-screen bg-gray-100">
      <LeaderboardPopup />
    </div>
  );
}

Run the code using Github codespaces:

1. Create a Repository on GitHub

Go to GitHub and create a new repository (e.g., leaderboard-app).
Initialize it with a README, or leave it empty.

2. Open GitHub Codespaces

Once your repository is created, go to the repository’s page on GitHub.
Click on the "Code" button and then select "Open with Codespaces"
Click on "New codespace" to open a fresh Codespace instance

3. Set up Your React App in Codespaces

In the terminal within GitHub Codespaces, run the following commands to set up a new React app:
```
npx create-react-app leaderboard-app
cd leaderboard-app
```
This will create a new React application and navigate into the leaderboard-app directory

4. Install Dependencies

Install any required dependencies, like lucide-react for the icons:
```
npm install lucide-react
```

5. Add Your Code

In the Codespaces editor, go to the src folder and create a new file called LeaderboardPopup.js
Paste your LeaderboardPopup component code into this file

Open src/App.js and import the LeaderboardPopup component:

import LeaderboardPopup from './LeaderboardPopup';

Then, add <LeaderboardPopup /> within the App component’s JSX.

6. Run Your App

In the Codespaces terminal, start your React development server:
```
npm start
```
GitHub Codespaces will automatically open a preview window for your app, where you can see the LeaderboardPopup in action.

Some Notes:

Dependencies: Make sure all necessary dependencies (like lucide-react) are installed in the Codespace.
React app initialization: By running npx create-react-app leaderboard-app, you're initializing a new React app, so ensure the folder structure is correct (the app should be created in the leaderboard-app directory).
Preview: GitHub Codespaces automatically sets up a preview for your React app, so you'll be able to see your changes live.

Sample

The component leverages several key features:

Tailwind CSS for responsive and modern styling
Lucide icons for consistent visual elements
React hooks for state management
Modal overlay for focused user attention
Responsive design that works on all devices

Key features implemented:

Clean, modern interface with a clear, visual hierarchy
Medal indicators for the top 3 players
Score and win statistics display
Interactive close functionality
Hover states for enhanced user feedback
Consistent spacing and typography
Responsive layout that adapts to screen size

To enhance this component further, consider:

Adding entrance/exit animations
Implementing sorting options
Including player avatars
Adding filtering capabilities
Supporting real-time updates
Including player progression tracking

Conclusion

Building an effective game leaderboard requires balancing visual appeal with functionality. This implementation demonstrates how modern React features and Tailwind CSS can be combined to create an engaging and responsive leaderboard interface. The modal approach ensures focused attention on the rankings while maintaining accessibility and user experience.

For future development, this approach offers several advantages. The component structure allows for easy feature expansion, while Tailwind CSS ensures consistent styling and responsiveness. The clear visual hierarchy helps players quickly understand their standing and encourages healthy competition.

Remember that a great leaderboard is more than just a list of scores – it's a tool for driving engagement and creating community within your game. This implementation provides the foundation for building an engaging competitive feature that can enhance player retention and satisfaction.