Forem: Nikola Balic

Browser Automation Built for Agents

Nikola Balic — Thu, 05 Mar 2026 16:08:17 +0000

Today we're launching the new Steel CLI and steel-browser skill, which brings browser automation redesigned from the ground up for agents.

Agents handle code, reasoning, and tool use well enough. Then they hit a real website and everything falls apart. Login walls, dynamic content, anti-bot systems, session state that doesn't persist. A five-minute human task becomes a twenty-minute debugging session.

This release targets that gap. It's also our first serious implementation of agent experience (AX): building tools where agents get clear inputs, predictable outputs, and failures they can recover from.

TL;DR

New agent skill and CLI
agent-browser integration
Stealth: captcha solving & proxies for agents
Run background browser sessions in parallel

What this actually does

The skill and CLI work together to make web tasks agents can finish reliably.

Run multi-step flows through login and dynamic UI
Pull clean markdown from cluttered pages
Capture screenshots and PDFs as evidence
Handle anti-bot measures and CAPTCHAs
Maintain session state across longer runs
Return structured outcomes you can verify

A working agent web run should be boring. Predictable, reviewable, debuggable. Agent starts a browser session via CLI, follows the skill contract, executes commands (open, snapshot, click, fill, type, wait), collects artifacts, returns status.

That's it. No heroics required.

SKILL.md: A contract, not a prompt

SKILL.md is a specification that tells an agent how to use a capability.

For web tasks, it tells the agent:

When to invoke the skill
How to execute the workflow
What shape the output takes
How to handle blockers

The goal is simple: less prompt glue, more repeatable behavior. Instead of rebuilding web handling for every project, your team adopts one stable path that carries across agents and harnesses.

What the steel-browser skill handles

This steel-browser skill is designed for autonomous web tasks where basic fetch tools fall short, often failing across many sites due to blocking, complexity, and other limitations.

Anti-bot and CAPTCHA flow

When automation is blocked, agent can use these patterns:

steel browser start --session checkout-bot-check --session-solve-captcha
steel browser open https://example.com
steel browser captcha solve --session checkout-bot-check

For automation-first sessions:

steel browser start --session checkout-bot-check --stealth

This is the difference between an agent that reports what it can't do and one that finishes the job.

The new Steel Browser CLI workflow

The CLI is the operator layer. It handles session lifecycle and gives agents and humans a clean interface for browsing work.

Session lifecycle, cloud or local

The CLI manages sessions and gives agents and humans a clean interface.

steel browser start creates or attaches a named session
steel browser live prints the live view URL for the active session
steel browser sessions lists sessions as JSON for scripting and agent loops
steel browser stop stops the active session
steel browser stop --all stops every session
steel browser start --api-url connects through a self-hosted endpoint

Passthrough commands for agent browser actions

The steel browser command forwards inherited agent-browser actions with a command prefix swap, including open, snapshot, fill, click, and wait.

Local runtime now lives under `steel dev`

Local runtime orchestration is now explicit and separate:

steel dev install — install dependencies
steel dev start — start local runtime
steel dev stop — stop it

This makes it easier to develop and debug workflows locally, then run the same shape of workflow in the cloud.

Get started

# Install and auth
npm i -g @steel-dev/cli
steel login

Full command reference: Steel CLI docs.

Install the skill

Available in the CLI repo's skills package:

npx skills add steel-dev/cli --skill steel-browser

Or via skills.sh.

Where this helps

This is not about browsing for its own sake. It is about unlocking agents and workflows where the web is the source of truth.

Competitive research with verifiable screenshots
Lead enrichment from JavaScript-heavy sites
Bug reproduction with session recordings
QA testing on live interfaces
Compliance documentation captured as PDFs
Data extraction from gated portals

Give your agent a real browser

Run one real workflow. Tell us what worked and what didn't.

Join our Discord and share your experience with the new skill and CLI.

[Boost]

Nikola Balic — Wed, 25 Feb 2026 12:43:30 +0000

Making Browser Agents More Observable with Raindrop and Steel

Niksa Kuzmanic ・ Feb 24

#ai #webdev #observability

How Websites Decide You're Human

Nikola Balic — Wed, 18 Feb 2026 15:12:02 +0000

Automated bots account for nearly half of all web traffic. According to Akamai's 2024 State of the Internet report, bots make up about 42% of traffic—and nearly two-thirds of those are malicious. They scrape protected data, stuff credentials, hijack accounts, and exploit competitive advantages.

This article explains how anti-bot protection works under the hood: the signals collected, the layers involved, and the logic that separates humans from sophisticated automation. Understanding the machinery helps you evaluate solutions, debug issues, and appreciate what it takes to defend modern applications.

Understanding the Bot Threat Landscape

Three Categories of Bot Traffic

Modern websites see three types of automated traffic.

Friendly Bots

Search engine crawlers, uptime monitors, feed aggregators, and copyright verification tools. They follow robots.txt, identify themselves, and behave predictably. Let them through.

Jekyll and Hyde Bots

Price comparison scrapers, social media automation tools, ticketing bots. Sometimes useful, sometimes abusive—depends on frequency, context, and business impact.

Malicious Bots

Credential stuffing, DDoS attacks, payment fraud, account takeover, inventory hoarding, spam. This is what anti-bot systems exist to stop.

Why Modern Anti-Bot Systems Need Multiple Layers

Bots range from simple scripts to full-blown headless browsers that mimic human behavior. No single detection method catches everything. Network-level checks stop basic automation but fail against residential proxies. Fingerprinting catches device anomalies but can be spoofed. Behavioral signals work well, but high-risk flows still need challenges.

Modern anti-bot systems layer these techniques so weaknesses in one area get covered by strengths in another. Websites evaluate risk dynamically and escalate friction only when something looks truly suspicious.

Layer 1: Network-Level Detection

IP Reputation Analysis

Anti-bot systems check incoming IPs against global threat intelligence sources to identify datacenter IPs, proxy services, VPN endpoints, Tor exit nodes, and known malicious addresses. IPs with poor reputation get flagged for additional verification like CAPTCHA challenges.

Rate Limiting

Rate controls detect abnormal request patterns and prevent abuse. Common techniques:

Fixed window — counters reset at fixed intervals, such as 100 requests per hour
Sliding window — distributes limits over rolling time intervals for smoother traffic management
Token bucket — allows short bursts while maintaining sustained rate constraints

When clients exceed limits, servers return HTTP 429 Too Many Requests. Advanced systems adapt limits based on authentication status, endpoint sensitivity, and past behavior.

Layer 2: Browser and Device Fingerprinting

Browser and device fingerprinting generates stable identifiers by analyzing browser environment, operating system, and hardware characteristics. These identifiers don't rely on cookies, making them effective against bots that rotate identities, clear storage, or run in private mode.

Canvas Fingerprinting

Canvas fingerprinting uses the HTML5 canvas element to detect differences in GPU hardware, graphics drivers, and rendering engines. The system draws text or graphics on a hidden canvas and captures the pixel data. Small variations in hardware or driver configuration produce different pixel patterns, creating a unique fingerprint.

Conceptual example

// Canvas fingerprint generation (conceptual example)
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');

ctx.textBaseline = 'top';
ctx.font = '14px Arial';
ctx.fillText('Browser fingerprint test', 2, 2);

const dataURL = canvas.toDataURL();
const fingerprint = hash(dataURL);

WebGL Fingerprinting

WebGL fingerprinting uses the browser's 3D rendering pipeline. It collects GPU vendor, renderer strings, supported shader extensions, and rendering behavior. These parameters differ across devices and help identify automation tools using virtual GPUs.

Audio Context Fingerprinting

The Web Audio API generates audio signals and measures how the device processes them. The resulting waveform varies across devices due to hardware audio pipelines, drivers, and browser implementations. These differences form a stable signature that's difficult for bots to spoof.

Font Enumeration

Font fingerprinting checks the list of installed system fonts. Each operating system, enterprise device, or custom environment tends to have a unique font combination. These lists help differentiate between real users, automated browsers, and headless environments.

Hardware and Environment Profiling

Anti-bot systems gather device metadata to build a broader fingerprint. Common signals:

Screen size and pixel density
Available memory
CPU core count
Color depth
Battery status (on mobile)
Installed plugins, MIME types, or extensions
System locale and timezone

Bots often fail to replicate the full set of these attributes, and small inconsistencies can reveal automated behavior.

Layer 3: Behavioral Analysis

Behavioral analysis identifies patterns that deviate from natural human interaction.

Mouse Movement Analysis

Systems track cursor paths, acceleration, hesitations, and micro-adjustments. Humans move their mouse with natural curves and irregular pauses. Bots generate straight, overly precise, or uniform movements.

Keyboard Dynamics

Systems evaluate typing cadence, key-press duration, rhythm variability, and error behavior. Human typing includes inconsistencies and occasional mistakes. Automated input shows perfectly uniform timing.

Navigation Patterns

Systems observe page sequence, dwell time, scroll depth, speed, and interaction timing. Warning signs include instant form submissions, zero page dwell time, or clicking elements immediately after load.

Session Consistency

Systems check whether user attributes stay stable during a session. Abrupt changes in fingerprints, IP, geolocation, or device traits indicate account takeover or bot-driven manipulation.

Modern anti-bot systems apply machine learning models trained on massive datasets of real user interactions to identify statistical anomalies in real time.

Layer 4: Challenge Mechanisms

Challenge mechanisms act as the final barrier when a request appears risky. These controls require the client to prove a human or genuine browser environment operates it.

CAPTCHA Systems

CAPTCHAs remain the most well-known challenge method. Modern variants:

Image-based CAPTCHAs where users read distorted text or pick the correct characters
Object-recognition challenges that ask users to identify items like vehicles or signs
Invisible CAPTCHAs that silently assess risk and only show a challenge when necessary

CAPTCHAs add friction and impact accessibility. Use them sparingly, only when risk is high.

Adaptive Interactive Challenges

Some systems present lightweight interactive puzzles—dragging sliders, matching shapes, completing simple tasks. These adapt to the perceived risk level, making them harder for automated tools but easy for humans.

JavaScript Execution Challenges

These challenges verify the environment supports full JavaScript execution and browser APIs. Examples include:

Checking DOM manipulation behavior
Requiring accurate execution of cryptographic or timing-based operations
Creating dynamic elements that must be interpreted in a real browser

Bots running in incomplete or emulated environments fail these checks.

The Constant Evolution of Bots and Defenses

Anti-bot engineering is a continuous arms race. As detection methods improve, attackers adapt with better browser automation tools, real-device spoofing, residential proxy networks, human-assisted solving, and AI-driven interaction patterns.

Anti-bot systems must evolve continuously—updating signals, tuning thresholds, combining heuristics with learned models. Static defenses don't last long. Layered and adaptive ones do.

Conclusion

Modern anti-bot systems operate as a layered security pipeline—analyzing network signals, device characteristics, behavior patterns, and environmental clues to distinguish real users from sophisticated automation. No single technique stands alone. Effective defenses combine reputation data, fingerprinting, behavioral modeling, and adaptive challenges.

Understanding what happens behind the scenes demystifies why requests get flagged, why CAPTCHAs appear, and why bot mitigation is more complex than blocking IP ranges or checking user agents. Bots evolve quickly. Defending against them requires systems that evolve just as fast.

Balancing Speed and UX: Making Reasoning AI Work for Users

Nikola Balic — Sun, 09 Feb 2025 00:00:00 +0000

Large Language Models (LLMs) are advancing beyond simple parameter scaling, incorporating complex architectures like mixtures of experts and enhanced reasoning capabilities.

Model makers such as OpenAI, Anthropic, Mistral, Google, and even DeepSeek are progressively releasing these reasoning models, which are now considered state-of-the-art (SOTA). These models excel at tackling complex, multi-step tasks, enabling solutions to challenges that were previously out of reach.

However, this advancement comes with a significant user experience (UX) challenge: increased response times.

The Power and Limitations of Reasoning Models

Reasoning models offer substantial benefits by providing detailed, accurate answers to intricate problems.

For example, while a standard LLM might succinctly answer, "The capital of France is Paris," a reasoning model would elaborate on the reasoning process: "Let me think. I remember learning in school that Paris is a major city in France. It's known for things like the Eiffel Tower, the Louvre museum, and the Seine River..."

This enhanced capability is invaluable for tasks such as debugging complex code, solving advanced mathematical problems, and navigating intricate dependencies.

Recently, I encountered persistent errors related to Pydantic validations imposed by dependencies. While multiple models struggled with these issues, the o1-mini model resolved them on the first attempt.

In such scenarios, the quality and reliability of the response outweighed the inconvenience of a longer wait time.

The UX Challenge: Speed vs. Quality

Despite their superior performance, reasoning models introduce latency due to the additional computational steps required for complex reasoning.

Users today expect immediate feedback, and any noticeable delay can be perceived as a weakness or inefficiency in the system. This perception poses a significant UX challenge: how to deliver high-quality, accurate responses without compromising on speed.

Instantaneous interactions have become the norm in digital communications, and users are likely to abandon or lose trust in tools that introduce delays, even if those delays result in better outcomes.

Balancing the need for quick responses with the demand for detailed, accurate information is crucial for the widespread adoption of reasoning models.

Leveraging Smart Model Routing

One effective strategy to address this challenge is smart model routing. This approach dynamically selects the most appropriate model based on the complexity of the user's query:

Simple Queries : For straightforward tasks such as summarization, translation, or basic question-answering, faster, less resource-intensive models can be employed. These models provide instant responses, ensuring a seamless user experience.
Complex Tasks : When faced with intricate problems that require deep reasoning, the system can route the query to more powerful reasoning models. Although these models may take slightly longer to respond, the accuracy and depth of their outputs justify the additional wait time.

Smart model routing ensures users receive a tailored balance of speed and intelligence, enhancing overall satisfaction while maintaining problem-solving quality.

Integrating Advanced Tools for Enhanced Workflows

In addition to model routing, integrating specialized tools can further mitigate UX concerns associated with reasoning models. Daytona SDKexemplifies how tool integration can enhance AI workflows:

Ephemeral Environments : Daytona SDK allows AI workflows or agents to create temporary, full-fledged environments—similar to throwaway laptops—for executing any piece of code. This capability enables models to handle tasks requiring real-time computation without significant delays.
Tool Adaptability : Agents can develop their own tools or install and invoke existing ones, providing flexibility in addressing a wide range of challenges. This adaptability ensures that the system remains responsive and efficient even when leveraging powerful reasoning models.

By incorporating such tools, developers can create a more responsive user experience in their AI products and service that leverages the strengths of reasoning models without compromising on the perceived speed.

Future Directions: Combining Strategies for Optimal Performance

The path forward involves integrating smart model routing with advanced toolsets like Daytona SDK. This combination can create a robust AI framework capable of:

Adaptive Responsiveness : Seamlessly switching between models based on task complexity ensures optimal performance and user satisfaction.
Scalable Problem-Solving : Leveraging specialized tools within ephemeral environments allows the system to tackle diverse challenges efficiently.
Enhanced User Trust : Delivering accurate, high-quality responses without unacceptable delays builds user confidence in AI systems.

Tools to the Rescue

The development of reasoning LLMs marks a significant advancement in AI, offering remarkable capabilities to solve complex problems. However, the associated increase in response times presents a UX challenge that cannot be overlooked.

By implementing smart model routing and integrating powerful tools like Daytona SDK, it is possible to harness the full potential of reasoning models while maintaining the immediacy that users expect. Striking this balance is essential for advanced AI systems' continued adoption and success in real-world applications.

How to Set Up Dev Environment on a Remote Machine

Nikola Balic — Tue, 16 Apr 2024 00:00:00 +0000

In this step-by-step guide, we'll walk you through setting up a dev environment with Daytona on a remote machine. Daytona is a powerful tool for creating and managing development environments on local or remote machines.

In this example, we'll use an Azure ARM machine with 8 vCPUs, 16 GB of RAM, and 300 GB of storage. You can play around with different server specifications or utilize whatever is accessible to you, maybe a homelab machine or that server you have tucked away in a corner. This will affect the number of workspaces you can have concurrently and the overall user experience.

TLDR:

Prepare Remote Machine: Install Docker and set permissions.
Configure Daytona Target: Add a remote machine as a target in Daytona.
Integrate Git Providers (Optional): Add GitHub or other providers to Daytona.
Create and Access Workspace: Use Daytona to create and open a workspace in VS Code.

Prerequisites:

Daytona daemon running on a local machine
A remote machine with Docker installed
SSH access to the remote machine
A GitHub account with a personal access token

Step 0: Set Up Daytona on Your Local Machine

Before you can start using Daytona with a remote machine, you need to ensure that Daytona is properly set up on your local machine. If you haven't installed Daytona yet, don't worry! We've got you covered with a simple one-liner that will get you up and running in no time.

Installing Daytona

To install Daytona on your local machine, open a terminal and run the following command:

(curl -sf -L https://download.daytona.io/daytona/install.sh | sudo bash) && daytona server -d

This command will download the Daytona installation script from the official Daytona website and execute it using sudo privileges. The script will guide you through the installation process, ensuring that all the necessary dependencies are installed and configured correctly.

Starting the Daytona Server

Once the installation is complete, the command will automatically start the Daytona server in the background using the -d flag. This means that you can continue using your terminal for other tasks while Daytona runs in the background.

If you ever need to stop the Daytona server, you can do so by running:

daytona server stop

And to start it again, simply run:

daytona server start

Verifying the Installation

To verify that Daytona is installed and running correctly on your local machine, you can run the following command:

daytona version

This command will display the currently installed version of Daytona, confirming that the installation was successful.

With Daytona now set up on your local machine, you're ready to move on to the next step and prepare your remote machine for use with Daytona. In the following steps, we'll guide you through the process of installing Docker on the remote machine, configuring the necessary permissions, and adding the remote machine as a target in Daytona.

Step 1: Prepare the Remote Machine

Install Docker on the remote machine if it's not already installed.
Add your user to the Docker user group by running the following command:

sudo usermod -aG docker $USER

Set the appropriate permissions for the Docker socket by running:

sudo chmod 666 /var/run/docker.sock

Step 2: Set Up the Target

Daytona utilizes the concept of targets, allowing you to add remote machines for deploying and managing development environments. By default, Daytona includes a Docker provider, enabling you to spin up environments on your local or remote machine inside Docker container.

Open the Daytona CLI and run daytona server target list to view the available targets.
Set a new target by running daytona server target set and selecting the desired provider (e.g., Azure).
Provide the necessary information, such as the IP address, username, and SSH key path.
Verify that the target was successfully added by running daytona server target list again.

Step 3: Add Git Providers (Optional)

This step is optional since you can simply paste the public Git URL to create a workspace from it.

If you want to use your own Git repositories directly with Daytona, you need to add a Git provider. We will use GitHub, but you can use GitLab, Bitbucket, Gitea, or Codeberg. You can also check our guide on how to select Git provider.
Go to your GitHub settings and navigate to the Developer settings.
Create a personal access token with the necessary permissions (repositories access).
Add the GitHub provider to Daytona using the generated access token and command daytona git-providers add.

Step 4: Create a Workspace

Run daytona create to create a new workspace.
Choose the desired Git repository, by pasting the repo URL or selecting it from the list of your repositories, and select the target on which you wish to run your workspace.
Wait for Daytona to pull the necessary images and set up the workspace.
Once the workspace is created, you can view it by running daytona list.

Step 5: Open the Workspace in VS Code

Set your preferred IDE by running daytona ide and selecting VS Code. You can also use web IDE powered by OpenVSCode or any of the JetBrains IDEs.
Run daytona code to open the workspace in VS Code.
VS Code will establish an SSH connection to the remote machine and set up the remote server.
Once the connection is established, you can start working on your files within the workspace.

Step 6: Open the Workspace from a Development Container Specification (Optional)

If your workspace includes a dev container specification, you can rebuild the container from it.
VS Code will download the necessary images and start the container.
Once the container is running, you can access the workspace and make changes on your code.

Conclusion

Congratulations! You have successfully set up Daytona with a remote machine and created a workspace. You can now work on your projects seamlessly, whether locally or on a remote machine.

If you encounter any issues or have further questions, feel free to contact the Daytona team in our Slack or add an issue to the Daytona GitHub repository.

How We Missed #1 on Product Hunt but Still Won Big

Nikola Balic — Fri, 29 Mar 2024 00:00:00 +0000

When we decided to launch our open-source dev environment management tool, Daytona, on Product Hunt, we had no idea what we were getting ourselves into. With just a week to prepare, we began an unusual trip that normally requires weeks of strategic planning.

TL;DR

Launching on Product Hunt with just a week's notice, we faced challenges and learned valuable lessons about authenticity, engagement, and measuring success beyond vanity metrics. Despite missing the #1 spot, we still reaped significant benefits and discovered the true value of a Product Hunt launch.

Following our project's successful launch on GitHub as open-source under the Apache license, we were riding a wave of success. With stars in our eyes and a week to prepare, we dove headfirst into the world of Product Hunt launches. But here's the twist - we gave ourselves just one week to prepare. Crazy? Maybe. Educational? Definitely.

Little did we know that our quest for Product Hunt glory would lead us down a path of epic struggle, valuable lessons, and ultimately, a newfound appreciation for the true value of launching on the platform.

But, reaching 5,000 GitHub stars in just three weeks secured us an engaged community and momentum, which we capitalized on to our advantage.

It's not about the numbers or the vanity metrics – it's about the connections you make, the lessons you learn, and the impact you have on your community.
Ivan Burazin, CEO and founder of Daytona

Debunking the Product Hunt Myths

Let's begin by debunking a few widespread myths regarding Product Hunt.

"You need a month to prepare for a Product Hunt launch." - False. We did it in less than a week, and you can too. It's all about focus, determination, and a healthy dose of caffeine.
"You need a fancy video and polished graphics." - Nope. While great visuals help, what matters most is showcasing your product's value and engaging with the community. We prepared our materials by repurposing the assets already available in our team's Figma and reusing an existing YouTube video.
"You need to know someone to succeed." - Absolutely not. We didn't have any insider connections, or hunters, just a genuine desire to share our project with the world.

Obstacles of Our Time-Constrained Product Hunt Launch

Our accelerated approach came with its own set of challenges:

Limited Time for Preparation: With only a week to prepare, we had to be laser-focused and prioritize our efforts.
Leveraging Existing Assets: Instead of creating entirely new assets for the launch, we had to be resourceful and repurpose what we already had. This meant adapting our existing content, graphics, and messaging to fit the Product Hunt format and audience.
Building Buzz on Short Notice: Typically, companies spend weeks or even months building anticipation for their Product Hunt launch. We had to compress this process into just a few days, which required a lot of creative thinking.

How We Epically Failed to Reach #1 on Product Hunt

The Epic Failure:

When our highly anticipated product launch on Product Hunt fell flat, it was a humbling and eye-opening experience. Here's three main reasons:

Underestimating the Competition: We thought we had a killer product, a solid strategy, and enough caffeine to power a small nation. But we quickly learned that the competition on Product Hunt is no joke. We were up against some seriously impressive products and teams with deep pockets and even deeper connections. To make matters worse, we found ourselves competing with trendy AI startups popping up left and right, promising to revolutionize every industry.
Overestimating Our Influence: We thought our existing community and open-source success would give us an advantage. Boy, were we wrong. Everything requires more effort than you might anticipate. Remember, communities are made up of individuals who can be forgetful or preoccupied with urgent matters of their own.
Ignoring the Subtle Art of Upvote Hunting: We naively believed that if we built it, they would come (and upvote). Little did we know that there's an entire art form dedicated to the subtle (and not-so-subtle) art of the deal on Product Hunt. We were like a bunch of amateurs trying to play chess against Hans Niemann.

The Brutal Honesty:

Despite our best efforts, we ended up at #2 on Product Hunt. And you know what? It was one of the best "failures" we've ever experienced. Here's why:

Exposure, Exposure, Exposure: We reached thousands of potential users, investors, and partners that we might never have reached otherwise.
Valuable Connections: Through our launch, we connected with countless developers, entrepreneurs, and industry leaders who shared our passion for making development environments more accessible and efficient.
Lessons Learned: Our "failure" taught us more about marketing, community building, and perseverance. We learned what works, what doesn't, and how to approach our next launch.

5 Key Benefits of Launching Your Dev Tool on Product Hunt

The Value of Being #2 (or #3, or #4...): While the allure of being #1 on Product Hunt is undeniable, the true value lies in the exposure, connections, and lessons learned along the way. Here are five reasons why being #2 (or any other spot on the top 10 leaderboard) is still a massive win:

Widespread Exposure: Despite being the second-ranking product, we were highlighted in Product Hunt's daily newsletter (top 10 gets in), which boasts an estimated reach of over 750,000 subscribers.
Social Proof: Being at the top on Product Hunt serves as a powerful social proof, validating your product's value and attracting potential users and investors.
Social Media Boost: Due to our massive social media presence on the launch day, we saw a 50% increase in Twitter impressions and a 10% increase in followers.
Multiplication Opportunities: Top products on Product Hunt often catch the attention of other creators, journalists, and influencers, leading to potential additional coverage.
Community Engagement: Topping the charts on Product Hunt can spark conversations and engagement within the tech community, providing valuable feedback and insights.

And one bonus point: Bragging Rights - Let's be honest – being able to say your product was ranked top on Product Hunt is a pretty cool feat.

The Product Hunt Launch Day Frenzy

On the day of the launch, the energy was palpable, like a scene straight out of a heist movie. Our team, was ready to jump on any opportunity to engage with the Product Hunt community. I, fueled by a steady drip of caffeine, was answering questions and comments with the speed of a speeding bullet. We even had a secret weapon: a stash of memes, ready to be deployed at a moment's notice, adding an element of playfulness to our interactions.

Throughout the day, I dedicated 20 hours to responding to messages, comments, and publishing updates across our social media channels. Although exhausting, this experience taught me the invaluable lesson of being present and responsive during a product launch.

Worthy mention — One of the great tools that helped us track our launch progress throughout the day was Hunted Space. It gave us a clearer understanding of how to understand the leaderboard mechanics and the fluctuations in upvotes.

As the day drew to a close, we were left in the #3 spot, which left us perplexed for a few hours. Unbeknownst to us, one of the other products had gained an astounding number of upvotes in the final minutes, propelling them to the coveted #1 position. This sudden shift caught the attention of the watchful team at Product Hunt, who promptly investigated and corrected the anomaly in the upvote count.

After the final tally, we were delighted to discover that we had safely landed in the second-place spot, earning the silver medal. This outcome was a testament to the hard work and dedication our team had poured into the launch, as well as the recognition from the Product Hunt community.

Step-by-Step Launch Preparation Guide

To those considering launching on Product Hunt, we encourage you to define success holistically, focus on authenticity, engage with the community, leverage your existing audience, and learn from the experience.

Set your launch date and commit to it, even if it is in two days. No backing out now!
Gather your assets. Screenshots, GIFs, and a concise description of your product are essential.
Craft your pitch. Tell your story, highlight your product's unique value, and don't be afraid to show some personality.
Rally your troops. Notify your existing community about the upcoming launch and ask for their support.
Get some rest. Trust us, you'll need it.
Launch day: Buckle up and enjoy the ride! Block off some time in the team's calendar to ensure everyone participates in engaging with your social media posts, giving them an initial boost.

One End Game Tip for Winning Product Hunt

If you pushed me into a corner and forced me to share just one genuine tip, it would be this:

Cross-promote like crazy: Share your Product Hunt link on every relevant platform, from social media to industry forums. You never know where your next upvote will come from.

Our Product Hunt Launch Adventure with Daytona Ends (for now)

Launching on Product Hunt in just one week was a wild ride, but we wouldn't have it any other way. We learned valuable lessons that success isn't just about vanity metrics, but about the connections we make, the lessons we learn, and the impact we have on our community.

So, if you're considering a Product Hunt launch, don't let the myths hold you back. Embrace the unconventional, stay true to yourself, and most importantly, have fun!

Ready to Try Daytona? If you're a developer looking to streamline your development environment setup, head over to our GitHub repository and give Daytona a try. We're excited to continue building this open-source tool alongside our growing community.

Where to Look for More Inspiration and Tips

For more insights and inspiration on launching your product on Product Hunt, be sure to check out some of the fantastic resources we've read through while prepping our launch.

We highly recommend the comprehensive guide by Product Hunt themselves, which covers everything from crafting your launch strategy to engaging with the community.

For a fresh perspective, don't miss the recent post by Steven Tey, detailing how his startup Dub.co achieved #1 Product of the Day and #1 Product of the Week. Steven shares his step-by-step playbook, emphasizing the importance of building a strong community, creating stellar launch assets, and staying engaged throughout the launch process.

Also, for another firsthand account of achieving 'Product of the Day' with a dev tool, check out Daniel Bass's insights on growth tips and community engagement during Permit.io's launch.

Other noteworthy articles include Plausible's honest account of their successful launch, Softr's tips for reaching the top, and the insightful behind-the-scenes look at Klever Suite's Product Hunt journey.

Daytona Goes Open Source: Embarking on a Bold New Journey

Nikola Balic — Tue, 05 Mar 2024 00:00:00 +0000

Daytona is leaping into the open source. It's a decision that feels akin to sending our beloved brainchild into the world, allowing it to grow beyond the confines of our aspirations.

We've nurtured Daytona with love and dedication. But the time has come for us to share our creation with the community that has been the backbone of our industry. Today marks not just another product release, but a heartfelt invitation to developers everywhere to join us in shaping Daytona's future.

Why Open Source?

The decision to go open source stems from a simple truth: innovation thrives in collaboration. By opening up Daytona's codebase, we're inviting developers to co-create the future of development environments with us. The tools that shape our digital world will no longer be confined behind proprietary walls. Instead, they will be shaped by the people who use them, in full view of the community that needs them.

Why Now?

Since Daytona's inception, our primary goal has been to address the challenges that enterprises encounter with their development environments. We have concentrated our efforts on enhancing code security, scaling and orchestrating environments efficiently, and streamlining the experience for developers when initiating development environments. We take pride in the significant progress we have made in these areas.

We recently began to wonder why this experience isn't shared by every developer across the globe. It's already 2024, yet setting up development environments still requires an average of at least 30 minutes and involves numerous steps. More often than not, developers encounter issues that impede their progress. We've discussed this in detail in a full article, which you can read here.

Our team quickly got to work. They skillfully extracted the core of Daytona, the development environment manager, and transformed it into a standalone binary that is now freely accessible for anyone to use.

But, What Does It Do?

Daytona is a radically simple open-source development environment manager. With Daytona, you need only to execute a single command daytona create.

Daytona automates the entire process: provisioning the instance, interpreting and applying the configuration, setting up prebuilds, establishing a secure VPN connection, securely connecting your local or a Web IDE, and assigning a fully qualified domain name to the development environment for easy sharing and collaboration.

While this is the alpha release, it represents the first step towards realizing our vision of a completely automated development environment.

As a developer, you can immediately start focusing on what matters most—your code.

What Can I Do With It?

Daytona can be downloaded and executed as a binary and you can just use it in your everyday life. Daytona is extendable by design, allowing you to create plugins that meet your specific needs. Moreover, it is genuinely open source under the Apache 2.0 license, providing you with unlimited freedom to modify and use Daytona as you see fit.

Promise of Open Source Project Sustainability

Daytona, the company, already generates revenue with an expanded product lineup that addresses orchestration, scalability, and security needs, catering to large enterprises while also focusing on enhancing the developers' experience.

Additionally, as previously noted, since Daytona is licensed under Apache 2.0, the community has the safeguard to create a fork should the project veer off course.

Join the Open Source Movement

We believe in a world where every developer can access the best tools, regardless of background or resources. By going open source, we're committing to that world. Whether you're an individual coder, part of a small startup, or leading a large enterprise team, Daytona is now your tool to shape as you see fit.

We invite you to join us on this journey. Contribute to the codebase, share your ideas, and help us build a tool that reflects the diverse needs of the global developer community.

Ready to Dive In?

Visit our GitHub repository (https://github.com/daytonaio/daytona) to start contributing today, sign up for our newsletter updates, and follow our voyage on social media.

The open source waters are warm, and the winds of change are favorable. Here's to the open source. Here's to Daytona—by developers, for developers.

DIY Guide to Transform Any Machine into a Codespace

Nikola Balic — Mon, 26 Feb 2024 00:00:00 +0000

Setting up any Dev Environment is usually painful, but the pain instantly doubles when you have to set it up on a remote machine. But why should you work on a remote machine?

There are many reasons: the size of the machine (for complex builds, dependencies, data locality, or running large ML/AI models), security, your employer needs the code in their data center, you're on the go with your iPad or Chromebook, or you're just tinkering with a Raspberry Pi.

No matter your reason, we wanted to make an easy step-by-step guide to let you set up a standardized, collaborative, and secure dev environment – essentially giving you a Codespaces-like experience on any machine.

Setting up a remote development environment can streamline your workflow and enhance collaboration. Instead of relying on a proprietary service like Codespaces, this guide will walk you through the process of creating your own custom remote dev environment on a Linux server.

There are many ways to go about it, but for this guide, we will assume you will use Visual Studio Code and leverage Docker containers to isolate dependencies and services needed for development.

Prerequisites

To follow this guide and set up your own remote dev environment, you will need:

A Linux server running Ubuntu 20.04+ with at least 2 GB of RAM and 2 CPU cores. This can be a virtual private server (VPS) from a cloud provider or maybe even a Raspberry Pi.
Admin/root access to the server to install software.
We will assume your local machine runs a MacOS or Linux variant (maybe even a WSL), if you are using Windows be sure to adapt some commands but everything should work as advertised.
A Visual Studio Code on your local machine.
Basic knowledge of command line interfaces and experience with Docker is helpful.
Some courage.

Step 1a - Provision Linux Server

First, you'll need to provision a Linux server that will host your remote development environment.

While you can repurpose an existing server, starting fresh helps avoid conflicts. Provision an Ubuntu 20.04 server with at least 2 GB RAM and 2 CPU cores from your preferred hosting provider.

Many cloud providers like AWS, GCP, Azure, and DigitalOcean allow instant provisioning of Ubuntu VPS instances at reasonable prices. Some of them even provide a complimentary initial credit; for instance, Digital Ocean offers credit for the first 60 days.

For example, Oracle offers a free-tier cloud account that allows you to utilize 4 cores ARM machine and up to 24GB of memory. Just prepare to be patient, as registering and setting up your account will require some effort.

A Raspberry Pi also works well if you want to save costs and don't need significant compute resources. Install the latest Raspberry Pi OS (formerly Raspbian) on compatible Pi hardware. To be honest, we haven't attempted it, but if you decide to try it, please let us know.

Once your Linux server is up and running, SSH into it to begin setup.

Step 1b - Generate SSH Keys (optional)

If you haven't already configured your SSH keys, we'll need to create them to establish a secure connection to the remote server.

Based on the remote server you're using, you'll need to prepare your SSH keys for the provisioning process so that you can input them into the dashboard of your preferred cloud provider. Alternatively, you can initially configure your server using a password and then add the SSH keys by following the instructions in the second section below.

### On your local machine:

# Open a terminal and generate a keypair:
ssh-keygen -t rsa

# Save the private and public keys in the default location.

# Copy the public key:
cat ~/.ssh/id_rsa.pub

### On the remote server:

# Log in and create an .ssh folder:
mkdir .ssh

# Create an authorized_keys file and paste your public key:
nano .ssh/authorized_keys

# Set permissions:
chmod 700 .ssh
chmod 600 .ssh/authorized_keys

Now, you can SSH into the server without a password using the private key.

ssh user@your-server-ip

Step 2 - Install Docker

Since we will leverage Docker to isolate and run services needed for development, Docker Engine is essential. Follow the official Docker install directions for Ubuntu to get up and running.

With SSH set up, let's install Docker on the remote server:

# Update package lists
sudo apt update 

# Install Docker prerequisites
sudo apt install apt-transport-https ca-certificates curl software-properties-common

# Add Docker's GPG key 
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Set up the Docker apt repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io

Check it worked:

docker --version
docker run hello-world

This should successfully download a test container image and print a confirmation message. If the last command encounters a permission error, attempt to run it again using sudo before the command.

While optional, Docker Compose makes working with multi-container apps much easier. We'll use it later to orchestrate our dev environment services.

First things first, ensure Docker Compose is properly installed by confirming the version number.

docker compose version

If that is not the case, install Docker Compose it with the following commands.

sudo apt-get update
sudo apt-get install docker-compose-plugin

Add your user to the docker group by using a terminal to run:

sudo usermod -aG docker $USER

Please log out and then log back in to ensure that the changes to the settings are applied.

Adding your username to the Docker group in a Linux system is a common practice to allow non-root users to run Docker commands without needing to use sudo each time. By default, Docker requires elevated privileges, and running Docker commands as a regular user without proper permissions would result in permission-denied errors.

Step 3 - Connect Visual Studio Code to Remote Environment

To allow connecting to your remote dev environment from Visual Studio Code, we need to install VS Code Server. The Visual Studio Code Server is a service designed to operate on a remote development environment, such as your cloud virtual machine (VM). This will run on the remote host and communicate with the VS Code client on our local machine.

Back on our local machine, launch VS Code and install the Remote - SSH extension.

Use the Command Palette (Ctrl/Cmd+Shift+P or F1) in VS Code to run Remote-SSH: Connect to Host... and enter your server info user@your-server-ip from the last step.

Once connected, VS Code will automatically install the VS Code Server on your remote machine.

You now have full VS Code functionality on the remote server, you can open folders and work directly on the remote server!

Step 4 - Clone Sample Project

With our base environment setup, let's clone the repository of a sample web project to use for development.

Feel free to use your own project repository here if you already have code to work on. For our example, we will use the Astro web framework git repository.

You can use the file explorer button and option to "Clone Repository" :

You will be prompted: "Cloning a repository in a Dev Container may execute arbitrary code." to confirm.

Or you could use the VS Code terminal and execute the commands directly.

# Git clone the project 
git clone https://github.com/my-user/my-project.git  
# Change directory 
cd my-project

Open the project folder by clicking on the pop-up alert or using File > Open Folder and select the folder you cloned previously.

For the Astro repository, you will be offered to install a set of helpful extensions for your Visual Studio Code, such as language support for Astro, the Prettier code formatter, EditorConfig, and ESLint.

Step 5 - Standardizing Your Development Environment with Dev Container

The Astro repository contains multiple devcontainer.json files, enabling you to set up and run a standardized development environment tailored to your needs. Whether you're contributing to the Astro project, contributing to documentation, or working within a particular framework like React, these configurations facilitate a seamless development experience.

Understanding and Harnessing Dev Containers

Development Containers is an open standard designed by Microsoft to enhance containers with content and settings tailored specifically for development purposes.

Dev Containers are defined using devcontainer.json files, which specify the environment's settings, extensions, and Docker configurations necessary for development. This approach ensures that all developers working on a project have a consistent set of tools and runtime environments, minimizing the "works on my machine" problem.

Dev Containers will ensure a seamless development experience across different machines, similar to what one might expect from using a platform like Codespaces, but with the flexibility and control of a custom setup. This setup is particularly beneficial for collaborative projects or when working in diverse environments, as it abstracts the underlying system configurations and focuses on providing a unified development experience.

Dev Container Basic Example from Astro

The following JSON snippet is an example of an Astro Basics devcontainer.json configuration.

{
  "name": "Basics",
  "build": {
    "dockerfile": "../examples.Dockerfile"
  },

  "workspaceFolder": "/workspaces/astro/examples/basics",

  "portsAttributes": {
    "4321": {
      "label": "Application",
      "onAutoForward": "openPreview"
    }
  },

  "forwardPorts": [4321],

  "postCreateCommand": "pnpm install && cd /workspaces/astro && pnpm run build",

  "waitFor": "postCreateCommand",

  "postAttachCommand": {
    "Server": "pnpm start --host"
  },

  "customizations": {
    "codespaces": {
      "openFiles": ["src/pages/index.astro"]
    },
    "vscode": {
      "extensions": ["astro-build.astro-vscode", "esbenp.prettier-vscode"]
    }
  }
}

Let's break down its components:

build: Here, "dockerfile": "../examples.Dockerfile" points to a Dockerfile relative to the location of the devcontainer.json file. This Dockerfile contains instructions for building the Docker image that the development environment will use.
portsAttributes: Defines attributes for ports used by the container. The configuration for port 4321 includes the label "Application" and specifies that when this port is auto-forwarded, a preview should be opened automatically ("onAutoForward": "openPreview").
forwardPorts: Lists the ports that should be forwarded from the container to the host machine, allowing access to services running inside the container.
postCreateCommand: Specifies a command to run after the container is created but before a user connects. Here, it installs project dependencies using pnpm install, then builds the project with pnpm run build, ensuring the environment is ready for development.
postAttachCommand: Defines commands to run after a user attaches to the container. This example specifies that the Server should start using pnpm start --host, preparing the Astro development server for use.
customizations: Contains environment-specific customizations. Under codespaces, it automatically opens the file src/pages/index.astro upon starting, which is useful for getting straight to editing the main page. The vscode section lists recommended extensions (astro-build.astro-vscode for Astro support and esbenp.prettier-vscode for code formatting with Prettier) to enhance the development experience within VS Code.

Each part of this configuration ensures that the development environment is tailored to the project's needs, from the Docker build process to the setup and ready state of the development server, including the developer's workspace setup in Visual Studio Code.

Open Project in Dev Container

To open your project folder in a dev container:

Open the command palette again.
Select "Remote-Containers: Open Folder in Container..."
Choose your project folder.

VS Code will start the dev container, install extensions, and connect automatically.

The setup time for your development environment will vary based on the configuration specified in your devcontainer.json file, so anticipate a short wait. While prebuilds can often mitigate this delay, that is a subject better suited for a separate discussion.

You now have a fully featured, standardized remote development environment accessible through VS Code on your local machine!

Step 6: Run Preview Dev Server

In VS Code, once the container is ready, and if the dev container of your choice didn't run the server, open a new terminal within VS Code and start your application.

You can execute a command to start the preview if your project is web-based. In the case of our Astro project, the command is as follows. However, your specific command may vary.

pnpm run dev

Step 7 - Securely Share and Expose Your Projects on the Internet Behind a Domain

To share your environment or collaborate with others, we can expose services securely over the internet. There are reverse proxy services that allow this to function without a public IP and even behind a firewall. For instance, in our example, we are using ngrok.

First, retrieve your ngrok authentication token from the ngrok dashboard.

Install Ngrok from their site, and authenticate it with your account auth token:

# Download and extract Ngrok  
curl -s https://ngrok-agent.s3.amazonaws.com/ngrok.asc | \
  sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null && \
  echo "deb https://ngrok-agent.s3.amazonaws.com buster main" | \
  sudo tee /etc/apt/sources.list.d/ngrok.list && \
  sudo apt update && sudo apt install ngrok

# Add authtoken 
ngrok config add-authtoken <TOKEN>

To give internet access to our app container from earlier at port 4321, run:

ngrok http 4321

Copy the ngrok tunnel URL provided in the output.

Your application should now be running and accessible via the Ngrok tunnel URL.

Paste the Ngrok tunnel URL into your browser to access your application. If you encounter a connection refused error, ensure that the necessary port is open on your server's firewall.

To maintain a consistent URL for each ngrok session, set up a static domain via your dashboard. Once you've created it, you can instruct the ngrok agent to utilize this domain using the --domain flag. Begin by halting the current ngrok process with Ctrl+C, and then restart ngrok to apply the changes.

ngrok http 4321 --domain jumpy-red-mollusk.ngrok-free.app

If you prefer to limit access to your application, you can easily implement authentication with ngrok without modifying your app. For instance, if you use the Google account alan@example.com, you can restrict access to yourself alone by executing ngrok with the following command:

ngrok http 4321 --oauth=google --oauth-allow-email=alan@example.com

When someone attempts to use your app, they must sign in using Google, and access will be exclusively permitted to your account. Please be aware that each time you restart ngrok without using the --domain flag, your app's URL will change.

If you wish to go a step further, ngrok also enables the use of your own custom domain. By leveraging CNAME records, ngrok hosts an endpoint on your custom domain and offers comprehensive management of the TLS certificate lifecycle on your behalf.

Step 8 - Code on the Go with a Web IDE on Any Device (optional)

After setting up your application and ensuring it's accessible via Ngrok, you might want to take advantage of the OpenVSCode Server, an open-source project licensed under MIT and maintained by Gitpod, for a full web IDE experience, which allows you to run a web version of VS Code on your server and access it through your web browser.

Here's how to set it up and run it:

Start the OpenVSCode Server : Within your remote server, install the OpenVSCode Server by running the following Docker command.

docker run -it --init -p 3000:3000 -v "$(pwd):/home/workspace:cached" gitpod/openvscode-server

Configure Inbound Port Rules : To access the OpenVSCode Server, you need to configure the network settings to allow traffic on the port it's running on. If you're using a service like Azure, navigate to the network settings and set an inbound port rule to allow TCP traffic on port 3000. Ensure that you review the firewall configurations on your server, utilizing tools such as ufw or iptables.
Test the Connection : Once the port is open, try accessing http://remote_server_ip:3000 in your web browser. Replace remote_server_ip with your server's actual IP address. If you encounter a security warning about the connection not being private (which may happen in some browsers like Chrome), try using a different browser like Safari that may not present this issue. Please note that configuring HTTPS falls outside the scope of this guide.
Access web IDE VS Code : If everything is configured correctly, you should see the OpenVSCode Server interface in your browser. You can now access your project straight from the Docker image without additional configuration.

Now, you'll have a powerful, browser-based web IDE to access from anywhere, making your development process even more flexible and accessible.

Conclusion

Congratulations! You've successfully set up your own fully-featured remote development environment without being locked into a proprietary platform!

You can now develop directly inside a container, ensuring a consistent development stack isolated from your local environment. With Ngrok, you have also secured your development environment with a tunnel to be shared with your team for collaboration.

Key aspects included:

Provisioning a Linux server to host your infrastructure
Installing Docker Engine and Docker Compose to define environments
Getting Visual Studio Code Server running to enable remote coding
Using Docker containers to encapsulate infrastructure dependencies
Adding security through SSH, HTTPS encryption, authentication
Scripting setup steps to allow automation and portability
Exposing services securely over the internet with Ngrok

You can now develop from anywhere using this robust, self-managed environment tailored to your needs!

Some additional pointers:

Commit the .devcontainer folder to source control to share with the team.
Use Docker Compose for multi-service applications.
Install the Remote - Containers extension to use dev containers locally.
Read the docs of each tool we have used for more advanced configuration and troubleshooting.
Remember to delete any temporary accounts or services you no longer need to maintain security and avoid unnecessary charges.

Let me know if you have any other questions.

Step 9 - Can we automate this?

This is the process you'll undergo for setting up each remote development environment you create, and it's no simple task.

As an engineer, you might often find yourself thinking, "Could I automate this task with some scripts?" While the answer is yes to some extent, it's important to remember that any modifications you make will necessitate script updates. Eventually, you may find that you have to perform the task manually once more.

For consistency and portability across projects, we can automate the process using provisioning scripts.

First, create a setup script setup.sh that executes our past commands:

#!/bin/bash

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update and upgrade the system packages
sudo apt update && sudo apt upgrade -y

# Install necessary packages for Docker installation
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y

# Install Docker Engine, CLI, and containerd
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

# Add current user to the Docker group to run Docker commands without sudo
sudo usermod -aG docker ${USER}

# Print Docker version to verify installation
docker --version

# Install and configure Ngrok
echo "Installing Ngrok..."
curl -s https://ngrok-agent.s3.amazonaws.com/ngrok.asc | \
  sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null && \
  echo "deb https://ngrok-agent.s3.amazonaws.com buster main" | \
  sudo tee /etc/apt/sources.list.d/ngrok.list && \
  sudo apt update && sudo apt install ngrok

echo "Please enter your Ngrok auth token (You can find it on your Ngrok dashboard):"
read NGROK_AUTH_TOKEN
ngrok config add-authtoken $NGROK_AUTH_TOKEN
echo "Ngrok installed and configured."

# Install OpenVSCode Server using Docker
echo "Installing OpenVSCode Server..."
docker run -d --init -p 3000:3000 -v "/workspace:/home/workspace:cached" gitpod/openvscode-server
echo "OpenVSCode Server is running on port 3000. Access it at http://your_server_ip:3000"

# VS Code Server installation instructions
echo "To connect Visual Studio Code to this remote environment:"
echo "1. Install the Remote - SSH extension in VS Code on your local machine."
echo "2. Use the Command Palette in VS Code to run 'Remote-SSH: Connect to Host...' and enter this server's SSH credentials."

echo "Setup complete. Please log out and back in for group changes to take effect."

Run the script:

Copy the script above into a file named setup.sh on the server.
Make the script executable with chmod +x setup.sh.
Execute the script with ./setup.sh.

Consider wrapping them into a simple CLI tool to simplify further.

Frankly, the situation is quite chaotic. The script is horrible and unlikely to function effectively because it requires customization to fit the particular system in question. Moreover, it doesn't address the routine tasks and interactions with repositories, workspaces, and the public preview, among other elements.

Perhaps there's a better way.

Follow Daytona on GitHub

Stay tuned for the imminent launch of Daytona on GitHub, our open-source project designed to automate development environment management. The aim to save developers and their teams substantial time by seamlessly standardizing dev environments across any machine, solving this challenge once and for all.

Join us in our mission to transform and accelerate the development process.

Redefining Success in the Engineering Organization

Nikola Balic — Fri, 16 Feb 2024 00:00:00 +0000

Over the past decade, the image of the lone “rockstar” coder has become pervasive in engineering culture. The rockstar represents the pinnacle of achievement – a virtuoso developer capable of single-handedly architecting complex systems and cranking out flawless code at superhuman speeds. This archetype holds powerful sway in the minds of many engineers, shaping identities and perceptions of success.

But as tech leaders, we must look honestly at the implications of idolizing this myth:

What behaviors and mindsets does it promote or suppress?
How might it impact diversity, collaboration, and sustainability in our organizations?
Are we setting realistic expectations or inadvertently fostering burnout and imposter syndrome?

At Daytona, we’ve worked to redefine engineering success by creating an environment optimized for developer empowerment. Through this experience, I believe sustainable high performance requires abandoning the rockstar model for a more nuanced definition – one rooted in collaboration, continuous learning, and resilience.

Collaboration, rather than solitary effort, is frequently the source of genuine innovation.

The Allure and Risks of the Rockstar Engineer

The trope of the rockstar coder has been glamorized in media, perpetuated through industry hype, and stubbornly persists in engineering culture. Qualities typically associated with this image include raw technical brilliance, an anti-social streak, extremely long hours in intense solitary focus, and often shunning documentation or “boring” tasks.

This archetype holds undeniable allure – the software magnate crafting ingenious code in a feverish haze of Red Bull and brilliance. Their heroic efforts seem to singlehandedly save the day. It’s a narrative that fuels many engineers' aspirations and egos.

But this myth also carries significant downsides at both individual and organizational levels. Internalizing it can fuel anxiety and burnout as developers stretch themselves thin trying to live up to unrealistic expectations. It encourages siloed work and discourages collaboration or mentoring. It often dismisses the less glamorous but crucial work of documenting systems, reviewing code, or interviewing users. The rockstar attitude also tends to disdain entry-level tasks beneath their abilities – depriving newcomers of mentoring and growth opportunities.

This mindset rewards raw individual performance over sustainability, diversity, or team success. It inhibits the cross-pollination of skills and institutional knowledge sharing vital for organizational resiliency. Ultimately, the lone rockstar represents a high-risk, low-scalability model. Their abilities become single points of failure and bottlenecks as systems grow more complex.

So, how can leaders reshape engineering culture to move past this limiting myth?

Redefining Success as a Team Sport

Transitioning from the rockstar model requires reorienting from individual heroics toward collaborative team success.

This starts with the reconceptualization of projects as complex, multifaceted endeavors that draw on diverse skill sets. A healthy engineering culture values specialists ranging from user researchers to security auditors and generalist coders. It recognizes documentation, QA processes, and DevOps as integral to delivering robust, scalable systems.

With this breadth of contributors, stellar outcomes become team achievements earned through coordination. We must assess performance based on aligned cross-functional metrics versus raw individual output. Shared ownership and collective responsibility should be instilled at all levels.

Incentive structures also require rethinking. Rewarding individual metrics like lines of code can incentivize destructive practices like copy-pasted code or cruft accumulation. Consider incentives for cross-training, mentorship, documenting systems, or volunteering for unglamorous tasks that advance the project. Make collaboration and knowledge sharing part of performance reviews.

We tend to underestimate the value of simplicity in solutions, mistakenly equating complexity with superiority.

Platforms like Daytona enable the transition toward team success by removing tedious configuration and maintenance tasks. Automating these distractions immerses developers in collaboration and core engineering work. Streamlined coordination frees attention toward higher-order team dynamics that enable breakthrough innovations.

Embracing Continuous Learning and Growth

Sustainable excellence requires continuous skills development, not just innate genius. The myth of the 10x rockstar coder rests on the idea of innate brilliance – that mastery stems from raw talent rather than practice. But engineering evolves rapidly – today’s hot skill fades into obsolescence within a few years. The lone coding cowboys of yore stranded themselves on soon-outdated islands of expertise.

Organizations seeking sustained excellence foster a culture of continuous learning and growth. With support for conferences, online courses, and skill-stretching assignments, veteran engineers avoid becoming dinosaurs. Regular retention-focused training and mentorship opportunities are crucial for upskilling staff and transferring institutional knowledge.

Rotational programs allowing developers to shift across focus areas prevent siloed knowledge and cultivate empathy between teams. Hackathons, open-source Fridays, and casual skill shares encourage enriching side projects and serendipitous discoveries.

Daytona’s automated dev environment provisioning helps enable continuous learning. Spinning up sandboxed environments for experimentation allows risk-free tinkering. Quick reproducibility makes replicating and debugging learning scenarios faster. Built-in collaboration capabilities support interactive training across locations.

Rewarding Resilience and Grit

Sustainable excellence requires resilience and grit. The rockstars’ apparent one-shot success hides their dependence on fickle inspiration. But true engineering prowess reveals itself when persevering through protracted debug cycles, integrating complex legacy systems, or grinding through regulations.

Resilience emerges as a theme across many high-performing teams. MIT research found the highest-achieving software teams were often internally mediocre at cooperation or technical skills. Their advantage lies in sheer grit to push through challenges. Members leaned on and learned from each other during demoralizing setbacks.

Leaders should assess performance under adversity – how developers respond to tangled dependencies or tight deadlines reveals character. Reward signs of resilience include carefully considering alternative approaches, seeking help appropriately, and implementing post-mortems.

Part of resilience involves minimizing needless adversity in the first place. Daytona’s development environment automation reduces frustrations from manual configuration and repetitive tasks. Its integrated tools remove painful context-switching. The platform aims to amplify human potential by eliminating unnecessary self-imposed obstacles.

Final Thoughts

Redefining engineering success requires examining assumptions and incentives permeating tech culture today. The mythic rockstar represents an antiquated archetype mismatched to the complexity of modern systems—sustainable excellence springs from collaborative teamwork, continuous learning, and resilience when facing inevitable challenges.

As leaders, we shape culture through the metrics, processes, and narratives we endorse. Daytona offers a development environments management platform designed to bring out the best in engineering teams beyond individual heroics. Our automated environments empower developers with the space and support structure to reach new levels of productivity and innovation.

The future belongs not to the lone coding cowboys but to the empowered, cross-trained engineering teams equipped to tackle exponentially growing complexity. By redefining success beyond outdated myths, leaders can foster cultures where every contributor can creatively thrive, and organizations can sustainably excel.

You Are Overthinking Startup Hiring

Nikola Balic — Wed, 14 Feb 2024 10:05:35 +0000

I. The Problem: Seeking Diamonds But Finding Cubic Zirconia

The thrusters are rumbling. The countdown has begun.

Your startup rocket ship is ready to blast off into the stratosphere, prepared to soar past the competition. But who will navigate the controls, monitor the engines, and chart the course to those distant stars? Certainly not the seasoned astronauts from NASA or hotshot pilots with fancy degrees.

No, for this wild ride, you need a scrappy, daring crew with sharp instincts, quick wits, and brave hearts beating for adventure.

Yet finding these gems is harder than spotting diamonds glinting in the rough.

Despite your best mining efforts, your hiring screens often turn up cubic zirconia — candidates with credentials sparkling but inner fire lacking. They may boast the polish and pedigree you sought on paper, but fail to thrive in your scrappy startup environment.

So, how do you uncover the real diamonds?

II. Flipping the Script: Assess Attitude Over Aptitude

Rather than judging polish or prior experience, seek out the self-starters and diligent doers, those who carved their own trails because the beaten path looked boring.

Find thoughtful minds not constrained by shoulds and woulds but led by their own internal compass. While lacking the CV shine, they have knowledge hard won from wrestling real problems, emerging smarter and tougher from the tussle.

You have your way. I have my way. As for the right way, the correct way, and the only way, it does not exist.—Nietzsche

So, how to identify these diamonds glinting just under the surface?

First, watch how they tell their tales, highlighting their grit, creativity, and ingenuity. Do they describe meeting challenges with scrappy solutions, mustering the motivation to push projects over the finish line when others wavered? Listen for how they connected the dots in fresh ways, with more visionary vista than narrow focus.

Second, listen for their vision, that spark of originality that drives them. Is it coherent and considered or a mere fantasy? Can they articulate how past choices and interests have informed their outlook?

Third, check their chassis for dents and cracks. They should bear some battle scars but not be battered beyond repair. A healthy chip on the shoulder can provide powerful fuel.

Score attitude over aptitude. Skills can be learned but mindset is harder to instill.

III. The Rare Rebels: Why They're Worth the Work

These diamonds in the rough are not as rare as the Hope Diamond, but they glimmer with special brilliance. Once aboard your rocket ship, they will take pressure and polish beautifully, forming the heart of your crew. Their daring and dedication makes them invaluable in startups.

Yet being so singular, they are accustomed to charting their own course. After all, they carved their own unconventional path rather than following the herd. So attracting them takes insight into what makes them tick.

The only way to deal with an unfree world is to become so absolutely free that your very existence is an act of rebellion.—Camus

First, recognize you are competing against their own inner drive and ambition. They are used to wrestling their own demons and dragons. Make sure your mission excites them as much as their personal goals. Appeal to their appetite for adventure and audacity.

Second, acknowledge you cannot motivate them. They propel themselves. Offer fuel and direction but let their engines run hot on their own spark. Stoke their fire with challenges to conquer and autonomy to chart their own trajectories.

Third, actively affirm their attitudes and approaches. These diamonds are used to being misunderstood, dismissed as too raw. Continually reinforce that you value their originality and fresh thinking as assets.

In short, do not try to polish or improve these rare gems. Let their sharp edges glint. Help them shine brighter by removing roadblocks and clearing the path for them to charge ahead. Feed their fire and follow their fearless lead to the stars.

IV. The Final Countdown: Making It Work With These Rare Rebels

Once you have these audacious diamonds aboard, how do you retain them for the long haul? With care and feeding of their independent spirits.

Man is condemned to be free; because once thrown into the world, he is responsible for everything he does.—Sartre

First, give them ownership and autonomy suited to their appetite. They left the safe but boring beaten paths to find freedom on your rocket ship. Let them take the controls at times and choose creative ways to contribute.

Second, help them help themselves. Guide their growth but resist over-managing. Lend support dismantling obstacles and provide resources for them to build their skills. Mentor lightly to supplement their self-motivation.

Third, keep your ear to the ground. Listen closely for signs of frustration and fading engagement. Nip restlessness in the bud by revisiting their goals and remixing their roles. Head off departure by showing there is still space to soar within your startup.

Finally, align values and validate their vision. Beyond thrills, these diamonds seek meaning. Link their quest to your mission. Show how their daring and dedication helps achieve shared goals. Let them spark big ideas and blaze new trails to the target.

V. Final Boarding Call

The startup launch clock is ticking. Soon that rocket ship will sail skyward with or without you. So stop overthinking and start seeking the fire within your candidates. Never mind the lack of credentials. Someone with the right stuff will shine through. Once aboard, these audacious diamonds will take your startup into the stratosphere and beyond.

Everything we hear is an opinion, not a fact. Everything we see is a perspective, not the truth.—Marcus Aurelius

Who needs seasoned astronauts when you have raw rebels ready to chart the course to the stars? All aboard!

Score attitude over aptitude. Skills can be learned but mindset is harder to instill.

Streamlining DevOps with Earthly and Daytona

Nikola Balic — Fri, 22 Dec 2023 00:00:00 +0000

This guide will walk you through creating a Docker environment using Earthly and DevContainer. We'll begin by setting up a Docker file to utilize Ubuntu's snapshot repositories, then explore how to build images for different architectures. Finally, we'll integrate Earthly for streamlined builds and testing, and set up a development container for a complete development environment.

Utilizing Ubuntu Snapshot Repositories

Open your Daytona workspace.
Create a Docker file to access Ubuntu snapshot repositories, which allow you to use a specific snapshot of Ubuntu packages from a certain date.
Define the snapshot date in the config.toml file. For example, set it to November 5, 2023.
In your Docker file, instruct it to fetch packages from the Ubuntu snapshot as of the specified date and update the sources list accordingly.

Building for Multiple Architectures

To support both x86 and ARM architectures, use Docker's buildx feature.
Execute the Docker buildx command, specifying platforms for AMD and ARM.
Incorporate variables into the build process to manage different configurations.

Simplifying with Earthly

Instead of separate Docker and make files, combine your configuration into one Earthly file.
Set global variables for the build date and snapshot in the Earthly file.
Use the Earthly file to build your Docker target and add a simple smoke test to verify package installation.

Setting Up DevContainer

Start with an Ubuntu Jammy base for your DevContainer.
Configure environment variables to pass GitHub credentials and set up Docker and Earthly.
Ensure the DevContainer includes necessary extensions for make, Docker, and Earthly.
Automate login to GitHub registry upon starting the DevContainer.

Testing and Building with Earthly

Verify Earthly installation by running a test command.
Observe Earthly's build process, which includes pulling the base image and updating package lists.
Push the completed build to your repositories using the Earthly push command.

Advantages of Using Earthly

Earthly caches your builds, making subsequent builds faster.
A single Earthly file simplifies the build and test process.
Integration with DevContainer provides a ready-to-use development environment with all necessary tools installed.

Code and Full Tutorial on YouTube

For those interested in exploring the code and configurations discussed in this guide, the Git repository is available at https://github.com/metcalfc/snapshot-base. Additionally, to see these steps in action, you can view the full video tutorial on YouTube, which provides a comprehensive walkthrough of the entire process.

Conclusion

By following these steps, you can set up a Docker environment that leverages the power of Earthly and DevContainer. This setup allows for efficient management of Docker builds, streamlined testing, and a fully equipped development container, ensuring a smooth and productive workflow.

Harnessing AI through Standardization and Isolation

Nikola Balic — Thu, 30 Nov 2023 00:00:00 +0000

Rapid AI development promises enormous opportunities, yet fragmented tools and environments often hinder capturing their full potential. What if developers didn’t need to waste precious time battling configurations and compatibility issues? How can teams accelerate innovation given skills gaps and disjointed systems? Enter Standardized Development Environments (SDEs)—the key to transforming workspaces into unmatched AI sandboxes.

In this guide, we'll explore how SDEs, by encouraging standardization and streamlining setup, can foster innovation in AI application development. We’ll also demonstrate how integrating popular tools like Chroma vector database and OpenAI API within SDEs can further enhance productivity and creativity. The result is the ultimate AI playground, ready for developers to unleash their skills and shape the future.

TL;DR

Sitemap Fetching and Parsing: Automates retrieval and analysis of sitemap XML from websites to extract URLs.
Content Extraction and Storage: Downloads web page content and stores it in Chroma, a flexible database system.
AI-Powered Search: Leverages OpenAI's embedding functions for smart content retrieval within Chroma.
Response Generation: Creates response based on prompts and search results, using latest OpenAI's GPT-4 model.
GitHub Repo link: https://github.com/nkkko/ai-sandbox-demo

If privacy is your priority, taking advantage of the security and isolation offered by Standardized Development Environments (SDEs) is advisable. In such cases, you can use local models and sentence transformers. The workspace I've set up is equipped with 12 GB of memory and 4 virtual CPUs, which should be sufficient to run a 7-billion-parameter model at reading speed. However, my example will utilize OpenAI's services for simplicity and improved performance.

Before diving into the integration of AI with SDEs, it's crucial to grasp the need for AI sandbox and to understand foundational concepts of SDEs and Dev Containers.

The Importance of a Sandbox for AI Projects

In the rapidly evolving landscape of artificial intelligence, the ability to experiment safely and efficiently is paramount. A sandbox environment serves as an essential incubator for innovation, providing AI developers with a controlled and flexible space to build and test their projects. Here's why a sandbox is a crucial tool for any AI endeavor:

Risk-Free Experimentation : AI development often involves trial and error, and a sandbox offers a risk-free zone where you can explore new ideas without affecting production systems. This freedom to experiment encourages creativity and can lead to AI functionality and performance breakthroughs.

Realistic Testing Conditions : Sandboxes can simulate real-world conditions, allowing you to observe how your AI behaves under various scenarios. This realistic testing ensures your AI solutions are robust, scalable, and ready for deployment.

Rapid Prototyping : Speed is a competitive advantage in AI development. Sandboxes facilitate rapid prototyping, enabling you to iterate on concepts and refine your models quickly. This agility accelerates the development cycle and helps bring AI applications to market faster.

Learning and Development : Sandboxes are educational playgrounds for newcomers and seasoned professionals. They provide an opportunity to learn new technologies, frameworks, and languages in a practical, hands-on manner, essential for staying current in AI.

Resource Optimization : AI projects can be resource-intensive. Sandboxes allow you to allocate resources dynamically, optimizing usage and reducing costs. This efficient resource management is critical, especially when working with complex models and large datasets.

In summary, a sandbox is more than just a development tool—it's an essential part of the AI ecosystem that supports innovation, learning, and growth. Whether you're a solo developer or part of a large team, a sandbox is the foundation upon which you can build the future of AI.

The Problem of Fragmented Environments

Developing AI applications is challenging enough without managing half-broken tools and dependencies. Unfortunately, this is the reality for many developers struggling with:

Inconsistencies: Different machines, OS versions, and ad-hoc configurations lead to errors and delays. Collaborating with teams also becomes tricky.

Error: Module X version mismatch 
(expected 1.1.0, got 1.0.2)

Onboarding Issues: Getting others up to speed on a project's tools and setup is time-consuming and frustrating.

New Dev: How do I get this project running locally?
Senior Dev: Oh boy, just follow this 12 step guide...

Security & Compliance Risks: Following best practices around vulnerabilities and regulations becomes difficult across fragmented environments.

Lack of Portability: Projects that depend on specific machines or OS configurations aren't easily portable or sharable across teams.

Wasted Time: Hours spent debugging environment issues is time lost building innovative applications.

SDEs address these problems through standardization, saving developers hours of configuration headaches so they can focus on creating.

Introducing the Power of Standardized Dev Envs and modern DEM Platforms

SDEs are more than just tools - they represent a philosophy that champions consistency and efficiency. By providing standardized configurations, SDEs allow developers to dive right into coding without worrying about environment setups. This uniformity becomes especially important in regulated industries like finance, healthcare, etc. where strict governance policies need adherence alongside security.

SDEs provide predefined configurations encompassing everything from dependencies and versions to security policies and tooling rules. This standardization brings immense advantages over traditional virtualized environments by traveling with code across devices. Unlike ephemeral containers, SDEs persist tools, credentials, and settings critical for long-term governance. By codifying best practices instead of reinventing the wheel, SDEs give developers instant access to compliant systems where they can simply code.

SDEs may be manually created or defined through a declarative specification. In the case of Development Containers, the environment specifications are encapsulated in a devcontainer.json file. This enables consistency across different machines while retaining flexibility to update configurations.

The Role of Development Environment Management

DEM platforms like Daytona takes the SDE approach further through automating creation, management and optimization of these standardized environments. It balances developer productivity with compliance requirements by:

Managing configurations, access controls & workflows from a central system
Automating provisioning and deployment of tools/languages
Enforcing security standards through predefined policies

By using Daytona, teams can boost collaboration and innovation within a secure, consistent sandbox.

A Standardized Development Environment (SDE) provides consistent tools and configurations tailored to a project’s specific needs. The environment encompasses everything from software dependencies and versions to security policies and tooling rules.

Key Benefits of SDEs

Portability - The environment travels with code across devices.
Collaboration - Teams use the same dependencies and tools.
Onboarding - New developers spin up instantly with no setup.
Compliance - Standardization facilitates best practices.
Consistency - Eliminates half-broken tooling across machines.
Efficiency - Less downtime from environment issues.

By providing predefined workspaces, SDEs allow developers to go from zero to coding in minutes. For AI innovation to flourish, minimizing tooling distractions is essential.

Unleashing Innovation with RAG and Vector Databases

Retrieval Augmented Generation (RAG) unites the generative power of language models with the precision of information retrieval, enabling the creation of nuanced, contextually accurate content.

Generative models like GPT excel in creating coherent text for tasks such as text completion and question answering. However, they may struggle with vague prompts or limited data, leading to less reliable outputs.

Conversely, retrieval models are adept at sourcing exact answers from large databases, essential for chatbots and search functionalities. Yet, they lack the creative dynamism of generative models, being restricted to predefined responses.

Vector databases complement this duo by streamlining the retrieval process, quickly aligning queries with pertinent information. The fusion of RAG and vector databases thus forges systems that are both creative and precise, enhancing content generation capabilities.

Integrating OpenAI for embeddings creation and generative functionalities

As AI capabilities continue rapidly advancing, developers are eager to test the limits of generative models like GPT-4 and GPT-4 Turbo (gpt-4-1106-preview). Yet high costs, unpredictable outputs, and difficult tooling often restrain the exploration process.

By providing easy-to-use APIs instead of complex tooling, OpenAI lowers the barrier to leveraging innovations like completions and embeddings. Integrating OpenAI via user-friendly calls unlocks capabilities making applications smarter and more creative.

By leveraging OpenAI in combination with a vector database, developers gain access to state-of-the-art AI with minimized overhead. Specifically, the OpenAI API enables:

Content Generation : Automated generation of text matching specified tones, styles, and topics with the state of the art models such as GPT-4 turbo.

Code Completion : Suggestions of relevant code snippets and examples using various tools such as Copilot, Tabnine, Phind or Continue.

Search & Filtering : Relevant document retrieval via semantic similarity rankings.

Data Labeling : Automated classification, entity extraction, and sentiment analysis.

Image Generation : Creative images matching textual descriptions using DALL-E 3.

AI Demo: Setting Up Your AI Playground

Up to this point, we've explored the conceptual advantages of integrating Standardized Development Environments (SDEs), Chroma, and OpenAI. What does this look like in a real-world scenario? Let's introduce our demo project—an AI application that showcases the rapid prototyping enabled by this combination.

The project utilizes:

Dev Container Specification - Dev Containers are configured via a devcontainer.json file, which automates the setup of your development environment.
Chroma - For storage, indexing, and embedding-based semantic search with the help of embedding functions (all-MiniLM-L6-v2 or OpenAI text-embedding-ada-002 with enormous 1536 dimension vectors).
OpenAI - To generate new writings matching specified topics and content fetched from the vector database.

Understanding the AI Demo Project

Our demo project showcasing AI integration with popular developer tools. Here's an overview of what it does:

Web Content Extraction : Automatically extracts text and metadata from web page sitemap.
Storage and Search : Stores the extracted content in a vector database (Chroma) and enables intelligent search.
AI Generation : Uses Large Language Model (LLM) to generate articles new writing in relation to the stored content context.

It utilizes standardized development environments (SDEs) to allow collaborators to instantly replicate the setup and start using it our contributing to the project.

Preparing for Installation

Before beginning, ensure you have:

An SDE that supports Dev Container Specification, such as Daytona.io.
Python version 3.10 or later.
An OpenAI API key (optional).

Installation Steps

Set up your environment using one of the following methods:

Using an SDE:

Navigate to your preferred SDE, such as Daytona.io or a cloud IDE.
Point the SDE to the project's Git repository URL: https://github.com/nkkko/ai-sandbox-demo. Or use a shorthand https://yourdaytonainstance.com/#https://github.com/nkkko/ai-sandbox-demo

That's it! The IDE will automatically build a container with all dependencies based on the .devcontainer config.

Manual Setup:

Clone the repository to your machine.
Create virtual Python environment using venv or conda.
Install the required packages by running:pip install -r requirements.txtAlternatively, execute:pip install openai chromadb python-dotenv bs4 argparse lxml
Resolve the issues with your environment.
Create an .env file and insert your OPENAI_API_KEY.

Choice of Embedding Model

It is important to note that after you clone and run the repository you need to set up the .env file with your OPENAI_API_KEY in case you would prefer to use their embeddings.

Notably, as others have shown model's dimension size does not strongly predict its performance. Several models with fewer dimensions than text-embedding-ada-002's 1536 show similar levels of performance. For example, Supabase noted that when maintaining a constant accuracy@10 of 0.99, pgvector with all-MiniLM-L6-v2 outperformed text-embedding-ada-002 by 78%.

Example Usage

Content Harvesting:

python populate.py https://examples.com/sitemap.xml --n 100 --ef openai

This extracts 100 pages from the site's sitemap into Chroma using OpenAI embeddings.

Semantic Search:

python search.py "SDE best practices" --n 3

Finds the 3 most relevant pages on SDE best practices using vector similarity.

AI Generation:

python write.py "How can SDE improve productivity" --s "Software Development" --n 1

Generates an entire article on the prompt while referring to the top search result from the database for context.

Main Components of the Demo Project

populate.py: Extracts content from a website's sitemap and saves to the database.
search.py: Intelligently searches the database content.
write.py: Generates articles using the database content.
SynthSpyder.py: Core module with main logic.
db/: Chroma storage and utilities.
.devcontainer/: Configuration for standalone environments.
.env: Stores your OpenAI API key.

Why We Chose Chroma for Our Vector Database

Chroma's simplicity and user-friendly approach made it the standout choice for our AI Demo Project. Its straightforward APIs facilitate quick integration, allowing our team to prioritize feature development. Moreover, Chroma's innovative design, optimized for high-dimensional vector data, ensures efficient and accurate searches, which are vital for our project's NLP capabilities.

Additionally, Chroma offers the flexibility of local hosting, with options for both persistent and in-memory databases, catering to our project's scalability and performance needs. Its compatibility with advanced AI tools, such as LangChain and OpenAI, enables us to harness the full spectrum of AI technology effectively.

Designed specifically for high-dimensional vector data, Chroma simplifies storing, managing, and searching knowledge for cutting-edge AI applications. Its intuitive SDKs, robust production-ready deployment options, and specialized focus on embedding-powered features make Chroma a standout for innovation.

Out-of-the-box, Chroma handles critical functions like:

Vector embedding of text
Metadata storage
Efficient ANN search
Document storage
Query embedding
Relevance ranking

Chroma also shines through usability and scalability. Its intuitive SDKs and integrations facilitate rapid prototyping, while the production-ready server application easily handles growth.

Putting It All Together

By encapsulating the runtime toolchain into portable Dev Containers, adding a vector database like Chroma into the mix, and harnessing generative algorithms from OpenAI, developers can focus purely on building intelligently. Platforms like Daytona further accelerate this by managing provisioning and security of these SDEs at scale across teams.

Step 1 - Structuring the Dev Container

A dev container is structured using a devcontainer.json file that defines its Docker container and customize it for a particular project. Here is an example config:

{
  "name": "AI Sandbox Demo",
  "build": {
    "dockerfile": "Dockerfile"
  },
  "features": {
    "ghcr.io/devcontainers/features/github-cli:1": {
      "installDirectlyFromGitHubRelease": true,
      "version": "latest"
    },
    "ghcr.io/devcontainers/features/sshd:1": {
      "version": "latest"
    },
    "ghcr.io/devcontainers-contrib/features/mypy:2": {
      "version": "latest"
    }
  },
  "postCreateCommand": "pip install -r requirements.txt",
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python",
      ]
    }
  }
}

It allows configuring Docker build instructions, tools/languages to install, and IDE customizations for the project.

Step 2 - How to use Chroma as your embeddings vector database

Here is sample code to index and search documents with Chroma:

import chromadb

# Initialize ChromaDB Client
chroma_client = chromadb.Client()

# Create a collection 
collection = chroma_client.create_collection(
  name="articles",
  embedding_function="all-mpnet-base-v2" 
)

# Index documents
collection.upsert(
  documents=["Text content..."],
  metadatas=[{"url": "http://example.com/article"}]  
)

# Search documents
results = collection.query(
  query_texts=["search keywords"],
  n_results=5  
)

print(results['documents']) 
print(results['metadatas'])

This simplicity enables rapid development of AI prototypes on top of Chroma.

Step 3 - Populating the Database

With our dev environment ready, let's start using the demo scripts to extract and store web content.

The populate.py script handles content ingestion. To start, we need:

A website sitemap URL
(Optional) OpenAI API key for enhanced ML search (free trial and somestarting credits are available)

Let's walk through the script:

sitemap_url = "https://example.com/sitemap.xml" 

import SynthSpyder

# Fetch, parse and process sitemap asynchronously 
await SynthSpyder.process_sitemap(sitemap_url)  

# Saves content to ChromaDB collection

This illustrates the simplicity of the content pipeline. Under the hood, it:

Fetches sitemap XML.
Extracts listed URLs.
Downloads each page.
Scrapes main text content.
Stores in the database including metadata like the page URL.

Our database is now populated with structured web content ready for search and analysis!

Step 4 - Searching Content with AI

With a collection of content ingested, we can leverage AI search capabilities.

The search.py script allows queries against the database:

search_query = "Self driving cars" 

import SynthSpyder

# Search the database collection
results = SynthSpyder.search(search_query)  

# Results contain text snippets and metadata 
print(results)

By default, this uses approximate nearest-neighbor search provided by the Chroma vector database.

For semantic search, we can enable OpenAI embeddings:

results = SynthSpyder.search(query, ef_name="openai")

This showcases how SDEs allow us to easily swap out components like ML models.

Now let's generate some articles!

Step 5 - Querying GPT-4 within the set context from vector database

The API can be easily installed and imported into any Python environment:

pip install openai 

import openai

openai.api_key = "sk-..."

response = openai.Completion.create(
  engine="text-davinci-003",
  prompt="Hello world in Python",
  max_tokens=5
)

print(response["choices"][0]["text"])

This simplicity of integration with SDEs allows focusing efforts on creating intelligent applications rather than hassling with dependencies.

The write.py script ties together our content pipeline:

Query Chroma to fetch context around a topic
Feed context into GPT-4 to generate a unique response

For example:

query = "Self driving cars"
prompt = "Write an article about self driving cars" 

import SynthSpyder

# Fetch related content from the database
context = SynthSpyder.search_context(query)

prompt += f"\n\nContext:\n{context}"

# Generate the article
article = SynthSpyder.write(prompt)  

print(article)

And we have an AI-generated article personalized to our database content!

The standardized environment enabled us to easily:

Spin up a reproducible dev container
Ingest and store web content
Build an AI search pipeline
Integrate GPT-4 to generate articles

This demonstrates the power of SDEs as sandboxes for innovating with modern data tools and AI systems.

Opportunities for Improvements

To optimize and improve our project, we could:

Deploy a local Large Language Model (LLM) to enhance data privacy.
Introduce a configurable option in the .env file to select different OpenAI models, allowing for flexible model switching.
Enable the selection of the embedding function within the .env file instead of passing it as an argument every time.
Support the use of multiple collections for diverse data management and cross-collection search.
Develop a user-friendly web interface with Flask or a comparable framework to simplify interaction with the system.

The Future of AI lies in Standardization

As our demo illustrates, SDEs uniquely remove friction from the development process, saving developers hours upon hours. This compounds over the course of a project’s lifespan, enabling teams to achieve exponentially more through unlocked innovation capabilities. These environments turn "What if?" questions into "Why not!" breakthroughs. The only limit is your ambition.

Certainly, the rate at which AI development is evolving demands environments focused on flexibility and experimentation support. Only through standardization can developers hope to keep pace and push boundaries further.

In embracing the SDE approach, organizations also invest in their own future competitiveness within the AI landscape. Those still bogged down by fragmented tools will struggle to attract top talent and innovate quickly enough to compete. To encourage ingenuity, establishing a culture rooted in productivity and consistency is essential.

The next epoch of AI promises to stretch our imaginations beyond what is currently possible. But to reach this full potential, developers need environments tailor-built to support their ambitions. SDEs standardized using templates, containerization, and automation fill this need perfectly.

So whether you're an AI enthusiast eager to experiment, a seasoned veteran pursuing the next big innovation, or a team manager focused on supercharging productivity SDEs are the ultimate sandbox. Offering uniformity but not rigidity, they pave the way for scaling creativity quickly.

The future of AI has arrived. Are your environments ready to handle it?

Forem: Nikola Balic

Browser Automation Built for Agents

TL;DR

What this actually does

SKILL.md: A contract, not a prompt

What the steel-browser skill handles

Anti-bot and CAPTCHA flow

The new Steel Browser CLI workflow

Session lifecycle, cloud or local

Passthrough commands for agent browser actions

Local runtime now lives under steel dev

Get started

Install the skill

Where this helps

Give your agent a real browser

[Boost]

Making Browser Agents More Observable with Raindrop and Steel

Niksa Kuzmanic ・ Feb 24

How Websites Decide You're Human

Understanding the Bot Threat Landscape

Three Categories of Bot Traffic

Why Modern Anti-Bot Systems Need Multiple Layers

Layer 1: Network-Level Detection

IP Reputation Analysis

Rate Limiting

Layer 2: Browser and Device Fingerprinting

Canvas Fingerprinting

Conceptual example

WebGL Fingerprinting

Audio Context Fingerprinting

Font Enumeration

Hardware and Environment Profiling

Layer 3: Behavioral Analysis

Mouse Movement Analysis

Keyboard Dynamics

Navigation Patterns

Session Consistency

Layer 4: Challenge Mechanisms

CAPTCHA Systems

Adaptive Interactive Challenges

JavaScript Execution Challenges

The Constant Evolution of Bots and Defenses

Conclusion

Balancing Speed and UX: Making Reasoning AI Work for Users

The Power and Limitations of Reasoning Models

The UX Challenge: Speed vs. Quality

Leveraging Smart Model Routing

Integrating Advanced Tools for Enhanced Workflows

Future Directions: Combining Strategies for Optimal Performance

Tools to the Rescue

How to Set Up Dev Environment on a Remote Machine

Prerequisites:

Step 0: Set Up Daytona on Your Local Machine

Installing Daytona

Starting the Daytona Server

Verifying the Installation

Step 1: Prepare the Remote Machine

Step 2: Set Up the Target

Step 3: Add Git Providers (Optional)

Step 4: Create a Workspace

Step 5: Open the Workspace in VS Code

Step 6: Open the Workspace from a Development Container Specification (Optional)

Conclusion

How We Missed #1 on Product Hunt but Still Won Big

Debunking the Product Hunt Myths

Obstacles of Our Time-Constrained Product Hunt Launch

How We Epically Failed to Reach #1 on Product Hunt

The Epic Failure:

The Brutal Honesty:

5 Key Benefits of Launching Your Dev Tool on Product Hunt

The Product Hunt Launch Day Frenzy

Step-by-Step Launch Preparation Guide

One End Game Tip for Winning Product Hunt

Our Product Hunt Launch Adventure with Daytona Ends (for now)

Where to Look for More Inspiration and Tips

Daytona Goes Open Source: Embarking on a Bold New Journey

Why Open Source?

Why Now?

But, What Does It Do?

What Can I Do With It?

Local runtime now lives under `steel dev`