Forem: DigitalOcean

Build an End-to-End RAG Pipeline for LLM Applications

DigitalOcean — Wed, 01 Apr 2026 01:06:34 +0000

This article was originally written by Shaoni Mukherjee (Technical Writer)

Large language models have transformed the way we build intelligent applications. Generative AI Models can summarize documents, generate code, and answer complex questions. However, they still face a major limitation: they cannot access private or continuously changing knowledge unless that information is incorporated into their training data.

Retrieval-Augmented Generation (RAG) addresses this limitation by combining information retrieval systems with generative AI models. Instead of relying entirely on the knowledge embedded in model weights, a RAG system retrieves relevant information from external sources and provides it to the language model during inference. The model then generates a response grounded in this retrieved context.

An end-to-end RAG pipeline refers to the full system that manages this process from beginning to end. It includes ingesting documents, transforming them into embeddings, storing them in a vector database, retrieving relevant information for a user query, and generating an answer using a large language model.

This architecture is increasingly used in modern AI systems such as enterprise knowledge assistants, internal documentation search engines, developer copilots, and AI customer support tools. Organizations adopt RAG because it allows models to remain lightweight while still accessing large knowledge bases that change frequently.

In this tutorial, we will walk through how to design and build a complete RAG pipeline. Along the way, we will explore architectural considerations, optimization strategies, and production challenges developers encounter when deploying retrieval-based AI systems.

Key Takeaways

RAG combines retrieval and generation for more accurate AI systems: Retrieval-Augmented Generation (RAG) bridges the gap between static language models and dynamic, real-world data. Instead of relying only on pre-trained knowledge, it fetches relevant information at runtime and uses it to generate answers. This makes responses more accurate, up-to-date, and context-aware. It is especially useful for applications like chatbots, internal knowledge assistants, and search systems. Overall, RAG helps reduce hallucinations and improves trust in AI-generated outputs.
Vector embeddings are the foundation of semantic search in RAG: Embeddings convert text into numerical vectors that capture meaning rather than exact wording. This allows the system to understand similarity between queries and documents even if they use different phrasing. As a result, retrieval becomes more intelligent and context-driven instead of keyword-based. High-quality embedding models like text-embedding-3-large or bge-large-en can significantly improve retrieval performance. Choosing the right embedding model directly impacts the overall quality of your RAG system.
Each component of the pipeline plays a critical role: A RAG system is made up of multiple steps, including ingestion, chunking, embedding, storage, retrieval, and generation. If any one component is poorly optimized, it can affect the entire pipeline’s performance. For example, bad chunking can lead to irrelevant retrieval, even if your embedding model is strong. Similarly, weak retrieval will result in poor answers, no matter how powerful the language model is. This is why building an end-to-end RAG system requires careful design and tuning at every stage.
Evaluation is essential for building reliable RAG applications: It is not enough to just build a RAG pipeline, but you must also evaluate how well it performs. This includes checking whether the system retrieves the correct documents and whether the generated answers are accurate and grounded. Metrics like precision and recall help measure retrieval quality, while human evaluation helps assess answer correctness. Creating benchmark datasets with known questions and answers makes it easier to track improvements over time. Continuous evaluation ensures your system remains reliable in production.

Understanding the RAG System Architecture

Before implementing the pipeline, it is important to understand how the different components interact. A typical RAG system architecture can be divided into two major workflows: the indexing pipeline and the retrieval pipeline.

The indexing pipeline prepares the knowledge base so that it can be searched efficiently. During this stage, documents are ingested, cleaned, split into chunks, converted into embeddings, and stored in a vector database. This process is usually executed offline or periodically when new data becomes available.

The retrieval pipeline operates during inference. When a user asks a question, the system converts that query into an embedding, searches the vector database for semantically similar chunks, and provides those retrieved passages to the language model. The model then generates a response using both the query and the contextual information.

A simplified representation of the pipeline looks like this:

Document Sources
       (PDFs, Docs, APIs, Knowledge Base)
                        |
                        v
               Document Processing
                        |
                        v
                  Text Chunking
                        |
                        v
               Embedding Generation
                        |
                        v
               Vector Database Index
                        |
                        v
User Query → Query Embedding → Similarity Search
                        |
                        v
             Retrieved Context Chunks
                        |
                        v
                  LLM Generation
                        |
                        v
                  Final Response

This architecture enables the system to retrieve information dynamically rather than relying solely on model training.

Data Ingestion in a RAG Pipeline

The first stage of the pipeline involves gathering the data that the AI system will use as its knowledge source. In many real-world applications, this information is distributed across multiple systems. Organizations may store documentation in internal knowledge bases, PDFs, wikis, product manuals, or database records.

The ingestion stage extracts textual information from these sources and prepares it for processing. Depending on the data format, ingestion may involve parsing HTML pages, converting PDFs to text, or querying APIs to retrieve structured records.

At this stage, developers often implement preprocessing steps such as removing redundant formatting, normalizing whitespace, and filtering irrelevant sections. These steps are important because retrieval performance strongly depends on the quality of the text data stored in the system.

For enterprise knowledge retrieval systems, ingestion pipelines are usually automated and scheduled. For example, an internal documentation chatbot might update its knowledge base daily by ingesting the latest documentation changes from a repository.

Text Chunking: Preparing Documents for Retrieval

After ingestion, documents must be divided into smaller pieces before they can be embedded. This step, known as text chunking, plays a critical role in the overall performance of the RAG pipeline.

Large documents cannot be embedded effectively because embedding models have token limits and because large chunks reduce retrieval precision. Instead, documents are broken into manageable segments that capture a coherent piece of information.

Chunk size is typically chosen between 200 and 500 tokens. Smaller chunks provide more precise retrieval results, while larger chunks preserve more contextual information. Many production pipelines use overlapping chunks to prevent important sentences from being split across boundaries.

The following diagram illustrates how a long document is transformed into multiple overlapping chunks:

Original Document
-------------------------------------------------------
| Paragraph 1 | Paragraph 2 | Paragraph 3 | Paragraph 4 |
-------------------------------------------------------

After Chunking
-------------------------------------------------------
| Chunk 1 | Chunk 2 | Chunk 3 | Chunk 4 | Chunk 5 |
-------------------------------------------------------

Chunk Example
Chunk 1: Paragraph 1 + part of Paragraph 2
Chunk 2: Paragraph 2 + part of Paragraph 3

Choosing an effective chunking strategy significantly improves retrieval accuracy because each chunk represents a focused semantic concept.

Embedding Generation

Once documents are divided into chunks, each chunk must be converted into a numerical representation called an embedding. Embeddings transform text into high-dimensional vectors that capture semantic meaning.

For example, two sentences that express similar ideas will produce vectors that are close to each other in vector space. This property allows vector databases to retrieve semantically related text even when the wording differs.

Embedding models are trained using large datasets and transformer architectures. When a chunk is processed, the model generates a vector with hundreds or thousands of dimensions. These vectors serve as the foundation for similarity search.

Embedding generation occurs during both indexing and retrieval. During indexing, embeddings are generated for each document chunk. During retrieval, the user’s query is also converted into an embedding so that it can be compared against stored vectors.

This mechanism allows the RAG system to perform semantic search, which is far more powerful than traditional keyword matching.

Vector Embedding

Vector embeddings are dense numerical representations of data, which can be text, images, or audio. Vector embeddings are used to capture the semantic meaning of the data in a high-dimensional vector space. In an end-to-end RAG pipeline, embeddings are used to convert both documents and user queries into vectors so that similarity between them can be measured using metrics like cosine similarity. This allows the system to retrieve context based on meaning rather than exact keyword matches, making responses more accurate and relevant.

For example, even if a query doesn’t contain the same words as a document, embeddings can still identify it as relevant if the underlying intent is similar. Popular embedding models used in RAG systems include text-embedding-3-large, all-MiniLM-L6-v2, bge-large-en, and e5-large-v2, each offering different trade-offs in performance, cost, and deployment flexibility.

Storing Vectors in a Database

After embeddings are created, they must be stored in a specialized database capable of performing fast similarity searches. These systems are known as vector databases and form the core of the RAG retrieval infrastructure.

Unlike traditional databases that index numeric or textual fields, vector databases are optimized to search across high-dimensional vectors. They use approximate nearest neighbor algorithms to identify vectors that are closest to a query embedding.

The structure of a stored vector typically includes the embedding itself, the original text chunk, and metadata describing the source of the information. Metadata can include document identifiers, timestamps, or categories that allow filtering during retrieval.

A simplified representation of vector storage looks like this:

Vector Database

ID     Vector Embedding        Text Chunk
---------------------------------------------------------
1   [0.12, -0.44, 0.92...]   "RAG combines retrieval..."
2   [0.55, 0.33, -0.14...]   "Vector databases enable..."
3   [-0.77, 0.08, 0.62...]   "Embeddings represent..."

Popular vector database technologies include managed services and open-source platforms designed specifically for AI workloads. The choice often depends on scale, infrastructure preferences, and latency requirements.

Retrieval in a RAG Pipeline

When a user submits a question, the system begins the retrieval stage. The query is first converted into an embedding using the same embedding model used during indexing. Maintaining the same embedding model is important because similarity comparisons rely on consistent vector representations.

The query embedding is then sent to the vector database. The database performs a similarity search to find document chunks whose embeddings are closest to the query vector. These chunks represent the pieces of information most relevant to the user’s question.

The retrieved chunks are then combined and passed to the language model as contextual input. The model uses this context to generate a response grounded in actual documents rather than relying solely on its training data.

This process ensures that answers are based on real knowledge sources and can be updated whenever the underlying documents change.

Generation with a Large Language Model

The final stage of the pipeline involves generating a response using a language model. At this point, the system already has two pieces of information: the user’s question and the retrieved context.

These elements are combined into a prompt that instructs the model to answer the question using the provided information. Because the context is derived from authoritative documents, the model’s output becomes significantly more reliable and factual.

This stage also allows developers to control how responses are generated. Prompts may instruct the model to summarize information, provide citations, or answer in a specific format. Some systems also include guardrails that prevent hallucinations or restrict responses to retrieved information.

For example, if a user asks a question, the system first pulls the most relevant text from your knowledge base, then the LLM rewrites that content into a helpful answer, making it more conversational, structured, and easy to understand. This step is what makes RAG powerful, because it combines accurate, up-to-date information with fluent natural language generation, reducing hallucinations and improving answer quality.

Code Demo: Building a Simple End-to-End RAG Pipeline

The following example demonstrates how a basic RAG pipeline for LLM applications can be implemented in Python. The example uses document loading, chunking, embeddings, and a vector database to create a minimal working pipeline.

Install dependencies

pip install langchain chromadb sentence-transformers openai

Load documents

from langchain.document_loaders import TextLoader

loader = TextLoader("knowledge_base.txt")
documents = loader.load()

Split documents into chunks

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
   chunk_size=500,
   chunk_overlap=100
)

chunks = splitter.split_documents(documents)

Generate embeddings

from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
   model_name="sentence-transformers/all-MiniLM-L6-v2"
)

Store vectors

from langchain.vectorstores import Chroma

vector_db = Chroma.from_documents(
   documents=chunks,
   embedding=embeddings
)

Retrieval and generation

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI()

qa_chain = RetrievalQA.from_chain_type(
   llm=llm,
   retriever=vector_db.as_retriever()
)

response = qa_chain.run(
   "What is retrieval augmented generation?"
)

print(response)

This simple implementation demonstrates how document retrieval and language models can be combined into a working RAG system.

Evaluating RAG System Performance

Evaluating a RAG system is important because you need to be sure that it is not only retrieving the right information but also generating correct and useful answers from it. In simple terms, a good RAG pipeline should find the right content and then explain it correctly.

First, let’s look at retrieval evaluation. This checks whether the system is pulling the right documents from your database. Imagine you have a knowledge base about cloud services, and a user asks, “How can I run AI models on GPUs?”. If your system retrieves documents about GPU Droplets or AI infrastructure, that’s a good sign. But if it returns unrelated content like pricing pages or networking docs, retrieval quality is poor. Metrics like recall (did we find all relevant documents?) and precision (were the retrieved documents actually relevant?) help measure this. For example, if 5 documents are relevant but your system only retrieves 2, recall is low.

Next is generation evaluation, which focuses on the answer produced by the language model. Even if retrieval is correct, the model (like GPT-4 or Llama 3) might still generate incomplete or incorrect responses. For instance, if the retrieved document clearly says “GPU droplets support CUDA workloads”, but the model responds with “GPU support is limited”, that’s a problem. This is why human evaluation is often needed to check if the answer is factually correct, complete, and grounded in the provided context. Automated metrics struggle to detect things like s or subtle inaccuracies.

To make evaluation consistent, teams usually create an evaluation dataset. This is a collection of sample questions along with their correct answers and sometimes the expected source documents. For example:

Question: “What are GPU droplets used for?”
Expected answer: “They are used for AI/ML workloads, training models, and high-performance computing.”

You can then run your RAG system on this dataset and compare its answers against the expected ones. Over time, this helps you track improvements, catch errors, and tune your system (for example, by improving chunking, choosing a better embedding model, or adjusting prompts).

In practice, strong RAG evaluation combines:

Retrieval checks: Did we fetch the right information?
Answer checks: Did we explain it correctly?
Continuous testing: Are we improving over time?

This ensures your RAG pipeline is reliable, accurate, and ready for real-world use.

Scaling and Production Considerations

Prototype RAG pipelines often work well with small datasets, but production deployments introduce additional challenges. Large organizations may store millions of document chunks, requiring scalable infrastructure for indexing and retrieval.

Latency also becomes an important concern. Vector searches, embedding generation, and LLM inference all contribute to response time. Developers must carefully optimize these components to ensure interactive performance.

Production systems frequently incorporate caching layers, query batching, and efficient indexing strategies. Monitoring tools are also used to track retrieval accuracy, system latency, and cost per query.

Cost and Latency Optimization

Operating a RAG pipeline at scale can become expensive if not carefully optimized. Each query may require embedding generation, vector search, and language model inference.

Several strategies help reduce these costs. Caching responses for frequently asked questions prevents repeated model inference. Limiting the number of retrieved chunks also reduces token usage and speeds up generation.

Another important technique is re-ranking. Instead of sending many retrieved documents to the language model, a re-ranking model selects the most relevant passages before generation. This improves response quality while reducing computational overhead.

RAG vs Fine-Tuning

A common question among developers is whether to use retrieval-augmented generation or fine-tuning.

Fine-tuning changes a model’s internal weights by training it on additional datasets. This approach works well for teaching models specific styles or behaviors. However, it is less effective for continuously changing knowledge because retraining the model is expensive and time-consuming.

RAG systems take a different approach by keeping the model unchanged while retrieving knowledge dynamically. This makes them ideal for applications where information changes frequently, such as product documentation or customer support knowledge bases.

For most knowledge-intensive applications, RAG provides a more flexible and maintainable solution.

Conclusion

Building an end-to-end RAG pipeline is about combining the strengths of retrieval systems and large language models to create applications that are both accurate and context-aware. Instead of relying only on pre-trained knowledge, a RAG system can fetch relevant information in real time and use models like GPT-4 or Llama 3 to generate clear, human-like responses grounded in that data. In this article, we understood each of the steps used to create the RAG pipeline from data ingestion and chunking to vector embeddings, retrieval, and response generation. Each component plays a critical role, and even small improvements (like better chunking strategies or choosing the right embedding model) can significantly impact overall performance. As organizations continue to build AI-powered applications, RAG stands out as a practical and scalable approach for use cases like chatbots, knowledge assistants, and document search. By continuously evaluating and refining your pipeline, you can create systems that are not only intelligent but also reliable and production-ready.

Resources

Tutorial: Deploy NVIDIA's NemoClaw in One Click

DigitalOcean — Mon, 23 Mar 2026 18:28:14 +0000

This article was originally written by Amit Jotwani (Staff Developer Advocate at DigitalOcean)

Key Takeaways

NemoClaw is an open-source stack from NVIDIA designed to help developers run OpenClaw securely.
DigitalOcean offers NemoClaw 1-Click Droplets that enable you to set up this stack on a CPU-optimized virtual machine and run NemoClaw.
This tutorial illustrates how to SSH into your Droplet, configure inference settings and policies, connect to NemoClaw, and effectively reconnect after the initial setup.

At GTC 2026, NVIDIA announced NemoClaw, an open-source stack that makes it easy to run OpenClaw autonomous agents securely. OpenClaw is an open-source agent platform that Jensen Huang called “the operating system for personal AI.” We covered how to run OpenClaw on a Droplet in an earlier tutorial. NemoClaw takes a different approach — it wraps OpenClaw with sandboxing, security policies, and inference routing through NVIDIA’s cloud.

NemoClaw is still in alpha, so expect rough edges. Interfaces may change, features might be incomplete, and things could break. But if you’re curious to try it out or just want to see what NVIDIA’s vision for agents looks like, this tutorial will get you up and running on a DigitalOcean Droplet in under 10 minutes.

Prerequisites

Before you begin, you’ll need:

A DigitalOcean account (sign up here if you don’t have one)
An NVIDIA account to generate an API key at build.nvidia.com

Step 1 - Create a Droplet from the Marketplace

Head to the NemoClaw 1-Click Droplet on the DigitalOcean Marketplace. Click Create NemoClaw Droplet.

When configuring the Droplet, select the CPU-Optimized plan with Premium Intel. You’ll want the option with 32 GB of RAM and 16 CPUs. NemoClaw runs Docker containers, a Kubernetes cluster (k3s), and the OpenShell gateway, so it needs the headroom.

Pick a data center region near you, add your SSH key, and hit Create Droplet.

Heads up: This Droplet costs $336/mo, so make sure to destroy it when you’re done experimenting. It adds up fast if you forget about it.

Step 2 - SSH into the Droplet

Once your Droplet is ready, SSH in:

ssh root@your_server_ip

You’ll see the usual Ubuntu login banner, and then the NemoClaw onboarding wizard will kick off automatically. It runs through a series of preflight checks, making sure Docker is running, installing the OpenShell CLI, and spinning up the gateway. You’ll see checkmarks fly by as each step completes.

Step 3 - Walk Through the Onboard Wizard

The onboarding wizard will ask you a few things. Here’s what to do at each prompt:

Sandbox Name

The first prompt asks for a sandbox name. Just press Enter to accept the default (my-assistant). The wizard will then create the sandbox, build the container image, and push it to the gateway. This takes a couple of minutes, and you’ll see it run through about 20 steps as it builds and uploads everything.

NVIDIA API Key

Once the sandbox is ready, the wizard asks for your NVIDIA API key. In this setup, inference is routed through NVIDIA’s cloud using the nvidia/nemotron-3-super-120b-a12b model, so it needs a key to authenticate.

To get your key, head to build.nvidia.com/settings/api-keys, sign in, and click Generate API Key. Give it a name, pick an expiration, and hit Generate Key.

Copy the key (it starts with nvapi-), paste it into the terminal prompt, and press Enter.

The wizard saves the key to ~/.nemoclaw/credentials.json and sets up the inference provider. You’ll see it confirm the model and create an inference route.

Policy Presets

After the inference setup, NemoClaw sets up OpenClaw inside the sandbox and then asks about policy presets. You’ll see a list of available presets including Discord, Docker Hub, Hugging Face, Jira, npm, PyPI, Slack, and more. These control what external services the agent is allowed to reach.

At the bottom, the wizard asks:

Apply suggested presets (pypi, npm)? [Y/n/list]:

Type n and press Enter. These presets grant the sandbox network access to package registries, which you don’t need for a basic setup. You can always add them later if your agent needs to install packages.

Once onboarding finishes, you’ll see a clean summary with your sandbox details and the commands you’ll need going forward:

Sandbox    my-assistant (Landlock + seccomp + netns)
Model      nvidia/nemotron-3-super-120b-a12b (NVIDIA Cloud API)
NIM        not running

Run:       nemoclaw my-assistant connect
Status:    nemoclaw my-assistant status
Logs:      nemoclaw my-assistant logs --follow

Step 4 - Connect to NemoClaw

Now for the fun part. Connect to your sandbox.

nemoclaw my-assistant connect

This drops you into a shell inside the sandboxed environment. From here, launch the OpenClaw TUI (terminal user interface):

openclaw tui

That’s it. You should see the OpenClaw chat interface come up. The agent will greet you and introduce itself, ready to chat.

Type a message and hit Enter. You’re now talking to an AI agent running inside a secure, sandboxed environment on your own Droplet.

Reconnecting After a New SSH Session

If you close your terminal and SSH back into the Droplet later, you’ll find that nemoclaw and related commands aren’t available. That’s because the onboarding script installed everything through nvm in a separate shell, and that doesn’t carry over to new sessions.

Run this once to fix it permanently. It adds nvm to your .bashrc so it loads automatically on every login:

echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc && echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> ~/.bashrc && echo '[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"' >> ~/.bashrc && source ~/.bashrc

Then reconnect to your sandbox and launch the TUI the same way as before:

nemoclaw my-assistant connect

openclaw tui

Everything picks up right where you left off. Your sandbox and agent are still running.

What's Next

By default, the sandbox has limited network access, so the agent can’t reach external services out of the box. To unlock more capabilities - like connecting to Slack, GitHub, or pulling packages from PyPI - you’ll want to configure policy presets. Check the NemoClaw documentation for the full list of available integrations and how to set them up.

NemoClaw is still very early, so expect things to be rough around the edges. But if you want to get a feel for where always-on agents are headed, this is a good way to start poking around.

Resources

GPT 5.3 Codex is the Next Level for Agentic Coding

DigitalOcean — Thu, 19 Mar 2026 20:00:00 +0000

Agentic Coding models are one of the obvious and most impressive applications of LLM technologies, and their development has gone hand in hand with massive impacts to markets and job growth. There are numerous players vying to create the best new LLM for all sorts of applications, and many would argue no company and their products in this space have more of a significant impact than OpenAI.

GPT‑5.3‑Codex is a truly impressive installment in this quest to create the best model. OpenAI promises that GPT-5.3-Codex is their most capable Codex model yet, advancing both coding performance and professional reasoning beyond GPT-5.2-Codex. Benchmark results show state-of-the-art performance on coding and agentic benchmarks like SWE-Bench Pro and Terminal-Bench, reflecting stronger multi-language and real-world task ability. Furthermore, the model is ~25% faster than GPT-5.2-Codex for Codex users thanks to infrastructure and inference improvements. Overall, GPT‑5.3‑Codex might be the most powerful agentic coding model ever released (Source).

So let’s see what it can do. Now available on the DigitalOcean GradientTM AI Platform and all OpenAI ChatGPT and Codex resources, we can test the model to see how it performs. In this tutorial, we will show how to use Codex to write a completely new project from scratch. We are going to make a Z-Image-Turbo Real-Time image-to-image application using GPT‑5.3‑Codex, without any user coding! Follow along to learn what GPT‑5.3‑Codex has to offer, how to use GPT‑5.3‑Codex for yourself, and a guide to vibe coding new web applications from scratch!

Key Takeaways

State-of-the-Art Agentic Performance: GPT-5.3-Codex delivers impressive results across software engineering and agentic tasks, outperforming GPT-5.2-Codex in reasoning, multi-language capability, and real-world coding evaluations like SWE-Bench Pro and Terminal-Bench 2.0.
Getting Started with GPT-5.3-Codex on GradientTM AI Platform is easy: All you need is access to the DigitalOcean Platform to begin integrating your LLM’s calls seamlessly into your workflows at scale.
From Prototype to Production in Record Time: With roughly 25% improved speed and real-time interactive steering, GPT-5.3-Codex feels less like a static generator and more like a responsive engineering partner capable of iterating, debugging, and refining projects alongside you. By handling scaffolding, architecture decisions, edge cases, and deployment-ready details, GPT-5.3-Codex can dramatically compress development timelines, making it possible to ship fully functional applications from scratch more quickly than ever (Source).

GPT‑5.3‑Codex Overview

GPT-5.3-Codex is a major agentic coding model upgrade that combines stronger reasoning and professional knowledge with enhanced coding performance, runs about 25 % faster than GPT-5.2-Codex, and excels on real-world and multi-language benchmarks like SWE-Bench Pro and Terminal-Bench. It’s designed to go beyond simple code generation to support full software lifecycle tasks (e.g., debugging, deployment, documentation) and lets you interact and steer it in real time while it’s working, making it feel more like a collaborative partner than a generator. It also has expanded capabilities for long-running work and improved responsiveness, with broader availability across IDEs, CLI, and apps for paid plans. (Source)

As we can see from the table above, GPT‑5.3‑Codex is a major step forward over GPT‑5.2‑Codex across software engineering, agentic, and computer use benchmarks. This, paired with the marked improvement in efficiency, make for an incredible indicator of how great this model is. We think this is a significant upgrade to previous GPT Codex model users, as well as new users looking for a powerful agentic coding tool to aid their process.

Getting Started with GPT-5.3-Codex

There are two ways to get started with GPT-5.3-Codex that we recommend to developers. First, is accessing the model with Serverless Inference through the GradientTM AI Platform. With Serverless Inference, we can Pythonically integrate the LLM generations into any pipeline. All you need to do is create a model access key, and begin generating! For more information on getting started, check out the official documentation.

The other way to get started quickly is the official OpenAI Codex application. It’s easy to get started with Codex on your local machine. Simply download the application onto your computer, and launch it. You will then be prompted to log in to your account. From there, simply choose which project you wish to work in, and you’re ready to get started!

Vibe Coding a Z-Image-Turbo Web Application with GPT‑5.3‑Codex

So now that we have heard about how GPT‑5.3‑Codex performs, let’s see it in action. For this experiment, we sought to see how the model performed on a relatively novel assignment that has a basis in past applications. In this case, we asked it to create a real-time image-to-image pipeline for Z-Image-Turbo that uses webcam footage as image input.

To do this, we created a blank new directory/project space to work in. We then asked the model to create a skeleton of the project to begin, and then iteratively added in the missing features on subsequent queries. Overall, we were able to create a full working version of the application with just 5 prompts and 30 minutes of testing. This extreme speed made it possible to ship the project in less than a day, from inspiration to completion. Now let’s take a closer look at the application project itself.

This project, which can be found here, is a real-time webcam-driven image-to-image generation application built in Python around a Gradio interface and a dedicated Z-Image-Turbo inference engine, where the UI in app.py presents side-by-side live input and generated output panes, parameter controls, and explicit Start/Stop gating so inference only runs when requested, while the backend in inference.py loads Tongyi-MAI/Z-Image-Turbo via ZImageImg2ImgPipeline, introspects the pipeline signature to bind the correct image-conditioning argument, enforces true img2img semantics instead of prompt-only generation, and executes inference in torch.inference_mode() with dynamic argument wiring so behavior adapts to the installed diffusers API. Critically, it can compute per-frame target resolution from webcam aspect ratio, snapping dimensions to a model-friendly multiple (default 16), and caps both sides below 1024, then applies post-generation safeguards that made the app stable in practice: dtype strategy (auto preferring bf16 then fp32, avoiding fp16 black-frame failure modes), degenerate-output detection with automatic float32 recovery, robust PIL/NumPy/Tensor output decoding and normalization, effective-strength clamping to preserve source structure, frame-hash seed mixing so scene changes influence results, and configurable structure-preserving input blending, all parameterized in config.py and documented in the README.md, with runtime status reporting latency plus internal diagnostics (pipe, dtype, size, effective strength, blend, seed, warnings) so you can observe exactly how each frame is being processed.

Closing Thoughts

GPT-5.3-Codex feels less like an incremental update and more like a meaningful shift in how developers interact with code. The combination of stronger reasoning, benchmark gains seen in testing, and a noticeable speed improvement makes it clear that agentic coding is maturing into something even more production-ready. What once required hours of boilerplate, debugging, and manual wiring can now be orchestrated through iterative prompts and high-level direction. As we demonstrated with the Z-Image-Turbo real-time application, a fully functional project can move from blank directory to working prototype in much less time traditionally required. While the actual results and performance benefits you experience will vary based on specific project requirements, complexity, and individual developer workflows, we are confident that GPT-5.3-Codex provides a substantial upgrade and a meaningful step forward in agentic coding capability, as evidenced by its stronger reasoning and measurable benchmark gains.

We recommend trying out GPT-5.3-Codex in all contexts, especially with DigitalOcean’s GradientTM AI Platform!

Getting Started with Qwen3.5 Vision-Language Models

DigitalOcean — Tue, 17 Mar 2026 16:00:00 +0000

This article was originally written by James Skelton (Senior AI/ML Technical Content Strategist II)

Vision Language models are one of the most powerful and highest potential applications of deep learning technologies. The reasoning behind such a strong assertion lies in the versatility of VL modeling: from document understanding to object tracking to image captioning, vision language models are likely going to be the building blocks of the incipient, physical AI future. This is because everything that we can interact with that will be powered by AI - from robots to driverless vehicles to medical assistants - will likely have a VL model in its pipeline.

This is why the power of open-source development is so important to all of these disciplines and applications of AI, and why we are so excited about the release of Qwen3.5 from Qwen Team. This suite of completely open source VL models, ranging in size from .8B to 397B (with activated 17B) parameters, is the clear next step forward for VL modeling. They excel at bench marks for everything from agentic coding to computer use to document understanding, and nearly match closed source rivals in terms of capabilities.

In this tutorial, we will examine and show how to make the best use of Qwen3.5 using a Gradienttm GPU Droplet. Follow along for explicit instructions on how to setup and run your GPU Droplet to power Qwen3.5 to power applications like Claude Code and Codex using your own resources.

Key Takeaways

Qwen3.5 VL demonstrates the growing power of open multimodal AI. The fully open-source model suite spans from 0.8B to 397B parameters and achieves strong benchmark performance across tasks like coding, document understanding, and computer interaction, approaching the capabilities of leading proprietary models.
Its architecture enables efficient large-scale multimodal training. By decoupling vision and language parallelism strategies, using sparse activations, and employing an FP8 training pipeline, Qwen3.5 improves hardware utilization, reduces memory usage, and maintains high throughput even when training on mixed text, image, and video data.
Developers can deploy Qwen3.5 on their own infrastructure. With tools like Ollama and GPU Droplets, it is possible to run large Qwen3.5 models locally or in the cloud to power applications such as coding assistants, computer-use agents, and custom AI tools without relying on proprietary APIs.

Qwen3.5: Overview

Qwen3.5 is a fascinating model suite with a unique architecture. It “enables efficient native multimodal training via a heterogeneous infrastructure that decouples parallelism strategies across vision and language components” (Source). This helps to make it avoid uniform approaches’ inefficiencies, such as over-allocating compute to lighter modalities, synchronization bottlenecks between vision and language towers, memory imbalance across devices, and reduced scaling efficiency when both modalities are forced into the same parallelism strategy.

By leveraging sparse activations to enable overlapping computation across model components, the system reaches nearly the same training throughput as pure text-only baselines even when trained on mixed text, image, and video datasets. Alongside this, a native FP8 training pipeline applies low-precision computation to activations, Mixture-of-Experts (MoE) routing, and GEMM operations. Runtime monitoring dynamically preserves BF16 precision in numerically sensitive layers, reducing activation memory usage by roughly 50% and delivering more than a 10% training speed improvement while maintaining stable scaling to tens of trillions of tokens.

To further leverage reinforcement learning at scale, the team developed an asynchronous RL framework capable of training Qwen3.5 models across all sizes, supporting text-only, multimodal, and multi-turn interaction settings. The system uses a fully disaggregated training–inference architecture, allowing training and rollout generation to run independently while improving hardware utilization, enabling dynamic load balancing, and supporting fine-grained fault recovery. Through techniques such as end-to-end FP8 training, rollout router replay, speculative decoding, and multi-turn rollout locking, the framework increases throughput while maintaining strong consistency between training and inference behavior.

This system–algorithm co-design also constrains gradient staleness and reduces data skew during asynchronous updates, preserving both training stability and model performance. In addition, the framework is built to support agentic workflows natively, enabling uninterrupted multi-turn interactions within complex environments. Its decoupled architecture can scale to millions of concurrent agent scaffolds and environments, which helps improve generalization during training. Together, these optimizations produce a 3×–5× improvement in end-to-end training speed while maintaining strong stability, efficiency, and scalability (Source).

Qwen3.5 Demo

Getting started with Qwen3.5 is very simple. Thanks to the foresight of Qwen Team & their collaborators, their are numerous ways to access and run the Qwen3.5 model suite’s models from your own machine. Of course, running the larger models will require significantly more computational resources. We recommend at least an 8x NVIDIA H200 setup for the larger models in particular, though a single H200 is sufficient for this tutorial. We are going to use Ollama to power Qwen3.5-122B-A10B.

To get started, simply start up a GPU Droplet with an NVIDIA H200 with your SSH key attached, and SSH in using the terminal on your local machine. From there, navigate to the base directory of your choice. Create a new directory with mkdir to represent your new workspace, and change into the directory.

Creating a custom game with Qwen3.5 running on Ollama and Claude Code

For this demo, we are going to do something simple: create a Python based video game for one of the most popular Winter Olympics sports: curling. To get started, paste the following code into the remote terminal:

curl -fsSL https://ollama.com/install.sh | sh
ollama launch claude --model qwen3.5:122b

This will launch Claude Code. If everything worked, it should look like above. From here, we can begin giving instructions to our model to begin generating code!

For this demo, provide it with a base set of instructions. Try customizing the following input:

“I want to create a simple game of curling in python code. i want it to be playable on my computer. Please create a sample Python program.

Packages: pygame”

This will give you, if your model ran predictably, a python file named something like “curling_game.py” with a full game’s code inside. Simply download this file onto your local computer, open the terminal and run it with python3.11 curling_game.py. Our game looks like this:

But looks are deceiving: this game is far from playable in the one-shot state. It requires serious work to amend the code to make the game playable, especially for two players. We can either use Claude Code with Qwen3.5 to make those adjustments, switch to an Anthropic Model like Sonnet 4.6 or Opus 4.6, or make the changes manually. From this base state, it took Qwen3.5 over an hour and at least 10 requests to make the game playable. Time was notably constrained by the single H200 GPU deployment we used for this demo, but the code output leaves significant room for improvement nonetheless. We expect that Opus 4.6 could accomplish the same task in a much quicker time frame, given its optimization for Claude Code, relatively superior benchmark scores, and more optimized infrastructure for inference.

If you want to try it out, this file can be found on Github Gist.

Closing Thoughts

Qwen3.5 VL represents an important step forward for open-source multimodal AI, demonstrating that publicly available models can increasingly rival proprietary systems in capability while offering far greater flexibility for developers. With its scalable architecture, efficient training infrastructure, and strong performance across tasks like coding, document understanding, and computer use, the Qwen3.5 suite highlights the growing maturity of the open AI ecosystem. As tools like GPU Droplets and frameworks such as Ollama make deploying large models easier than ever, vision-language systems like Qwen3.5 are poised to become foundational components in the next generation of AI-powered applications and physical AI systems.

7 OpenClaw Security Challenges to Watch for in 2026

DigitalOcean — Thu, 12 Mar 2026 16:00:00 +0000

This article was originally written by Fadeke Adegbuyi (Manager, Content Marketing)

OpenClaw isn’t just another chatbot wrapper. It executes shell commands, controls your browser, manages your calendar, reads and writes files, and remembers everything across sessions. The project runs locally on your machine and connects to WhatsApp, Telegram, iMessage, Discord, Slack, and over a dozen other platforms via pre-built integrations. It functions as a truly connected personal assistant. As a result, the use cases people have dreamed up for OpenClaw are wild.

One user showed an OpenClaw agent making money on Polymarket by monitoring news feeds and executing trades automatically. Another gave their bot access to home surveillance cameras. Someone else unleashed subagents to apply for UpWork freelancing jobs on their behalf.

// Detect dark theme var iframe = document.getElementById('tweet-2019044317621567811-81'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2019044317621567811&theme=dark" }

But this kind of access to your digital life comes with real consequences when things go wrong. And things have gone wrong. Security researchers found that the agent shipped with serious flaws that made it possible for attackers to hijack machines with a single malicious link. Meanwhile, Moltbook, a Reddit-style platform with over 2.8 million AI agents, had its database completely exposed, so anyone could take control of any AI agent on the platform.

None of this means you should avoid OpenClaw entirely. It means you should understand OpenClaw security challenges and take precautions before spinning up an agent with root access to your laptop. Running OpenClaw in an isolated cloud environment can help neutralize some of these risks—DigitalOcean's 1-Click Deploy for OpenClaw, for example, handles authentication, firewall rules, and container isolation out of the box so your personal machine stays out of the equation.

What are OpenClaw security challenges?

OpenClaw security challenges boil down to a design tension: the tool needs broad system permissions to be useful, but those permissions create a massive attack surface when something goes wrong. The agent runs with whatever privileges your user account has—full disk, terminal, and network access—by design.

It's also agentic and self-improving, meaning it can modify its own behavior, update its memory, and install new skills autonomously. This is impressive from a capability standpoint, but another vector that can cause things to spiral when guardrails are missing. Pair that with defaults that skip authentication, an unvetted skill marketplace, and persistent memory storing weeks of context, and trouble follows. The takeaway: approach with caution, isolate from production systems, and carefully scrutinize the defaults.

To his credit, OpenClaw creator Peter Steinberger has been openly vocal about these risks and actively encourages running OpenClaw in a sandboxed environment, which isolates tool execution inside Docker containers to limit filesystem and process access when the model misbehaves. DigitalOcean's one-click deployment does exactly this out of the box, giving you that isolation without the manual setup.

7 OpenClaw security challenges to watch out for

We've already seen a security audit uncover 512 vulnerabilities (eight critical) and malicious ClawHub skills stealing cryptocurrency wallets. None of these challenges are theoretical. They're all based on incidents that have already played out within weeks of OpenClaw’s launch.

These are the challenges you need to have on your radar if you're experimenting with OpenClaw:

1. One-click remote code execution through WebSocket hijacking

One of the most alarming OpenClaw vulnerabilities discovered so far is CVE-2026-25253, a one-click remote code execution flaw that Mav Levin, a founding researcher at DepthFirst, disclosed in late January 2026. The attack worked because OpenClaw's local server didn’t validate the WebSocket origin header—so any website you visited could silently connect to your running agent. An attacker just needed you to click one link. From there, they chained a cross-site WebSocket hijack into full code execution on your machine. The compromise happened in milliseconds. This is the core danger of running an agent locally on the same machine you're browsing the web with—one careless click and an attacker is already inside.

Levin's proof-of-concept showed that visiting a single malicious webpage was enough to steal authentication tokens and gain operator-level access to the gateway API—giving an attacker access to change your config, read your files, and run commands.

Security checks: In this instance, the fix landed in version 2026.1.29, so update immediately if you’re a version behind. Beyond that, best practices include avoiding running OpenClaw while browsing untrusted sites and considering putting the agent behind a reverse proxy with proper origin validation for an additional layer of protection.

2. Tens of thousands of unprotected OpenClaw instances sitting open on the internet

Here's the thing about OpenClaw's early defaults: the agent trusted any connection from localhost without asking for a password. That sounded fine until the gateway sits behind a misconfigured reverse proxy—at which point every external request got forwarded to 127.0.0.1, and your agent thought the whole internet was a trusted local user. SecurityScorecard's STRIKE team found over 30,000 internet-exposed OpenClaw instances.

Security researcher Jamieson O'Reilly showed just how bad this gets. He accessed Anthropic API keys, Telegram bot tokens, Slack accounts, and complete chat histories from exposed instances, even sending messages on behalf of users and running commands with full admin privileges. No authentication required.

This has since been addressed—gateway auth is now required by default, and the onboarding wizard auto-generates a token even for localhost.

// Detect dark theme var iframe = document.getElementById('tweet-2015401219746128322-801'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2015401219746128322&theme=dark" }

Security checks: At a minimum, check whether your instance is reachable from the public internet. Use a firewall to restrict access, enable gateway token authentication, and never expose the control plane without a VPN or SSH tunnel in front of it. This is a case where a managed cloud deployment can solve the problem outright—because your personal API keys, chat histories, and credentials aren’t sitting on an exposed local machine in the first place.

3. Malicious skills on ClawHub are poisoning the supply chain

ClawHub, OpenClaw's public skill marketplace, lets anyone publish an extension—the only requirement is a GitHub account older than one week. That low bar has unfortunately turned the marketplace into a target. Koi Security audited all 2,857 skills on ClawHub and found 341 that were outright malicious. Bitdefender's independent scan put the number closer to 900 malicious skills, roughly 20% of all packages. A single account—"hightower6eu"—uploaded 354 malicious packages by itself.

The attack is clever. You install what looks like a useful skill and the documentation looks professional. But buried in a "Prerequisites" section, it asks you to install something first—and that something is Atomic Stealer (AMOS), a macOS credential-stealing malware.

Security checks: OpenClaw has since partnered with VirusTotal to scan new skill uploads, but Steinberger himself admitted this isn't a silver bullet. At a minimum, before installing any skill, read its source code. Check the publisher's account age and history. Put simply, treat every skill as untrusted code running with your agent's full permissions. Unlike some exposure risks, malicious skills are a threat regardless of where OpenClaw runs—a poisoned skill executes the same way on a cloud server as it does on your laptop.

4. Credential storage in plaintext and API key leakage

One of the less glamorous but more dangerous issues is how OpenClaw handles secrets. The platform stores credentials in plaintext—including API keys for your LLM provider and tokens for every messaging platform your agent connects to—and those become targets the moment your instance is accessible to anyone other than you. Prompt injection attacks can also trick the agent into exfiltrating credentials by embedding hidden instructions in content the agent processes.

Cisco's team tested a skill called "What Would Elon Do?" and surfaced nine security findings, two of them critical. The skill instructed the bot to execute a curl command sending data to an external server controlled by the skill's author. Functionally, it was malware hiding behind a joke name.

Security check: At a minimum, rotate your API keys regularly and store secrets using environment variables or a dedicated secrets manager rather than config files. It's also worth setting spending limits on your LLM provider accounts. That way, even if a key is compromised, it can't rack up thousands in charges.

5. Prompt injection attacks amplified by persistent memory

What makes prompt injection in OpenClaw worse than in a typical chatbot is the persistent memory. The agent retains long-term context, preferences, and conversation history across sessions—which is one of its best features. But it also means a malicious instruction embedded in a website, email, or document doesn't have to execute immediately. Palo Alto Networks warned that these become "stateful, delayed-execution attacks". A hidden prompt in a PDF you opened last Tuesday could sit dormant in the agent's memory until a future task triggers it days later.

Security check: There's no perfect fix for prompt injection right now; it's an unresolved problem in agentic AI. But you can reduce the blast radius by limiting what tools and permissions your agent has access to, segmenting its access to sensitive systems, and reviewing its memory and context periodically for anything unexpected.

6. Shadow AI spreading through enterprise networks

This one's for anyone working at a company where developers tinker on their work machines. Token Security found that 22% of their enterprise customers have employees running OpenClaw as shadow AI without IT approval. Bitdefender confirmed the same, showing employees deploying agents on corporate machines connected to internal networks. An OpenClaw agent on a developer's laptop with VPN access to production means every vulnerability above is now a business problem.

Security check: If you're on a security team, you should scan your network for OpenClaw instances now. Set up detection for its WebSocket traffic patterns, and mandate that any approved use runs in an isolated environment—a VM or cloud server—rather than on laptops with internal access. Giving teams an approved, isolated deployment path is the fastest way to get ahead of shadow AI—it's much easier to enforce guardrails when the alternative isn't 'don't use it at all.'

7. The Moltbook database breach exposing millions of agent credentials

The security mess isn't limited to OpenClaw itself. Moltbook, the social network for AI agents built by Matt Schlicht, suffered a database exposure that cybersecurity firm Wiz discovered in early February. The database had zero access controls. Anyone who found it could view 1.5 million API tokens, 35,000 email addresses, and private messages between agents—enough to take control of any agent on the platform. China's Ministry of Industry and Information Technology issued a formal warning about OpenClaw security risks, citing incidents like this breach.

Security check: If you've used Moltbook, rotate every API key and token associated with your agent. Treat third-party platforms in the OpenClaw ecosystem with the same skepticism you'd apply to any new service asking for your credentials and consider additional security checks.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.

Pricing and product information accurate as of February 2026.

[Boost]

DigitalOcean — Tue, 10 Mar 2026 18:07:19 +0000

DigitalOcean for DigitalOcean

Mar 10

GPU Programming for Beginners: ROCm + AMD Setup to Edge Detection

#gpu #amd #programming #ai

Comments

2 min read

GPU Programming for Beginners: ROCm + AMD Setup to Edge Detection

DigitalOcean — Tue, 10 Mar 2026 16:00:00 +0000

In this hands-on tutorial, we demystify GPU computation and show you how to write your own GPU programs from scratch. Understanding GPU programming is essential for anyone looking to grasp why AI models depend on this specialized hardware.

We'll use ROCm and HIP (AMD's version of CUDA) to take you from zero to running real GPU code, culminating in a computer vision edge detector that processes images in parallel.

You can find the code in the project repository: https://github.com/oconnoob/intro_to_rocm_hip/blob/main/README.md

👇 WHAT YOU'LL LEARN IN THIS VIDEO 👇

🔧 Getting Set Up with ROCm Two ways to get started: spin up a GPU Droplet on DigitalOcean with ROCm pre-installed, or install ROCm yourself on an Ubuntu system with an AMD GPU. We cover both methods step-by-step.

➕ Example 1: Vector Addition (The Basics) Learn the fundamental structure of GPU programs—kernels, threads, blocks, and memory management. We'll add one million elements in parallel and verify our results.

⚡ Example 2: Matrix Multiplication (Why Libraries Matter) Discover why optimized libraries like rocBLAS dramatically outperform naive implementations. This is the operation powering most AI models you use daily.

👁️ Example 3: Edge Detection with Sobel Filter (The Cool Stuff) Apply your GPU programming skills to a real computer vision problem—detecting edges in images using a classic Sobel filter, all running massively parallel on the GPU.

Whether you're an AI enthusiast wanting to understand the hardware layer or a developer looking to harness GPU compute power, this tutorial gives you the foundation to start writing efficient parallel programs.

In case you haven't heard, we're back! Follow the DigitalOcean organization for updates, tutorials, and hands-on AI learning.

DigitalOcean — Fri, 06 Mar 2026 22:25:26 +0000

Jess Lulka for DigitalOcean

Feb 2

DigitalOcean on Dev.to: Practical AI Insights for Builders

#ai #digitalocean #machinelearning #learning

Comments 1

2 min read

We're DigitalOcean and we're excited to be here with you!

DigitalOcean — Thu, 23 Jul 2020 11:47:06 +0000

Hey everyone! We're so excited to be here at CodeLand:Distributed.

DigitalOcean offers the most easy-to-use and developer-friendly cloud platform. We help you manage and scale apps with an intuitive API, multiple storage options, integrated firewalls load balancers, and more. We're on a mission to simplify cloud computing so developers and businesses can spend more time creating software that changes the world!

Got Questions?! Let's chat!

Sammy the shark and our team members are here to connect with any questions you might have. We'll be at our DEV Connect channel all day, so stop by and say hello!

We're also happy to respond to any comments down below. 👇

Digital Swag

Today, we're offering all CodeLand attendees a $100 USD free trial. Sign up below and we'll follow up with all the details:

Sign up for a $100 DigitalOcean Promo!

You must first sign in to DEV Community.

We also have some fun wallpapers for anyone looking to spruce up their desktop or virtual backgrounds. ✨

Job Opportunities at DigitalOcean

DigitalOcean is a values-driven organization. Here is what we believe in:

Our community is bigger than just us
Simplicity in all we DO
We speak up when we have something to say and listen when others DO
e are accountable to deliver on our commitments
Love is at our core

Come swim with us: https://do.co/careers

How to Code in Go eBook

DigitalOcean — Mon, 22 Jun 2020 15:46:31 +0000

Introduction to the eBook

This book is designed to introduce you to writing programs with the Go programming language. You’ll learn how to write useful tools and applications that can run on remote servers, or local Windows, macOS, and Linux systems for development.

This book is based on the How To Code in Go tutorial series found on DigitalOcean Community. The topics that it covers include how to:

Install and set up a local Go development environment on Windows, macOS, and Linux systems
Design your programs with conditional logic, including switch statements to control program flow
Define your own data structures and create interfaces to them for reusable code
Write custom error handling functions
Building and installing your Go programs so that they can run on different operating systems and different CPU architectures
Using flags to pass arguments to your programs, to override default options

Each chapter can be read on its own or used as a reference, or you can follow the chapters from beginning to end. Feel free to jump to the chapter or chapters that best suits your purpose as you are learning Go with this book.

Download the eBook

You can download the eBook in either the EPUB or PDF format by following the links below.

After you’re finished this book, if you’d like to learn more about how to build tools and applications with Go, visit the DigitalOcean Community’s Go section.

How to Code in Go (Tutorial Series)

DigitalOcean — Mon, 30 Dec 2019 17:36:30 +0000

Go (or GoLang) is a modern programming language originally developed by Google that uses high-level syntax similar to scripting languages. It is popular for its minimal syntax and innovative handling of concurrency, as well as for the tools it provides for building native binaries on foreign platforms.

How To Install Go and Set Up a Local Programming Environment on Ubuntu 18.04

Go is a programming language that was designed for fast compilation, ease of programming, and efficient execution in production. This tutorial will guide you through installing and configuring a programming workspace with Go via the command line on Ubuntu 18.04.

How To Install Go and Set Up a Local Programming Environment on macOS

How To Install Go and Set Up a Local Programming Environment on Windows 10

How To Write Your First Program in Go

The “Hello, World!” program is a classic and time-honored tradition in computer programming. It's a simple and complete first program for beginners, and it's a good way to make sure your environment is properly configured. This tutorial will walk you through creating this program in Go.

Understanding the GOPATH

This article will walk you through understanding what the GOPATH is, how it works, and how to set it up. This is a crucial step for setting up a Go development environment, as well as understanding how Go finds, installs, and builds source files.

How To Write Comments in Go

Comments are lines that exist in computer programs that are ignored by compilers and interpreters. Including comments in programs makes code more readable for humans as it provides some information or explanation about what each part of a program is doing. In this article, you'll learn how to work with comments in Go.

Understanding Data Types in Go

Data types specify the kinds of values that particular variables will store when you are writing a program. The data type also determines what operations can be performed on the data. In this article, we will go over the important data types native to the Go programming language. Understanding some basic data types will enable you to write clearer code that performs efficiently.

An Introduction to Working with Strings in Go

This Go tutorial will go over the basics of working with strings, including how to create and print strings, concatenate and replicate strings, and store strings in variables.

How To Format Strings in Go

In this tutorial, we’ll go over some of the ways we can work with Go strings to make sure that all output text is formatted correctly. Topics we will cover include: quotes, apostrophes, multiple lines, escape characters, and raw strings.

An Introduction to the Strings Package in Go

Go's string package has several functions available to work with the string data type. These functions let us easily modify and manipulate...

How To Use Variables and Constants in Go

Variables are an important programming concept to master. They are symbols that stand in for a value you’re using in a program. This tutorial will cover some variable basics and best practices for using them within the Go programs you create.

How To Convert Data Types in Go

In Go, data types are used to classify one particular type of data, determining the values that you can assign to the type and the operations you can perform on it. When programming, there are times you will need to convert values between types in order to manipulate values in a different way. This tutorial will guide you through converting numbers and strings, as well as provide examples to help familiarize yourself with different use cases.

How To Do Math in Go with Operators

Effectively performing mathematical operations in programming is an important skill to develop because of how frequently you’ll work with numbers. This tutorial will review operators that we can use with the integer and float data types in Go.

Understanding Boolean Logic in Go

The Boolean data type can be one of two values, either True or False. We use Booleans in programming to make comparisons and to control the flow of the program. In this tutorial, we’ll go over the basics you’ll need to understand how Booleans work in Go, including Boolean comparison and logical operators, and truth tables.

Understanding Maps in Go

Most modern programming languages have the concept of a dictionary or a hash type. These types are commonly used to store data in pairs with a key that maps to a value. In Go, the map is what most programmers would think of as the dictionary type. It maps keys to values, making key-value pairs that are a useful way to store data in Go. Understand how Go maps work in this article.

Understanding Arrays and Slices in Go

This article will cover the array and slice data structures in the Go Programming language, which will provide you with the necessary information to make the appropriate choice when choosing between them. You'll also review the most common ways to declare and work with both arrays and slices. The tutorial will first provide a description of arrays and how to manipulate them, followed by an explanation of slices and how they differ.

Handling Errors in Go

Robust code needs to react correctly to unexpected circumstances like bad user input, faulty network connections, and failing disks. Error handling is the process of identifying when your program is in an unexpected state, and taking steps to record diagnostic information for later debugging.

Creating Custom Errors in Go

When communicating more complicated error information to your users, or to your future self when debugging, sometimes these two mechanisms are not enough to adequately capture and report what has happened. To convey this more complex error information we can implement the standard library interface type, error, to get more functionality.

Handling Panics in Go

Panics are unforeseeable errors that will spontaneously terminate and exit a running Go program. Common mistakes are often responsible for creating panics. In this tutorial, we'll examine a few ways that common operations can produce panics in Go, and we'll also see ways to avoid those panics. We'll also use defer statements along with the recover function to capture panics before they have a chance to unexpectedly terminate our running Go programs.

Importing Packages in Go

Making use of packages allows us to make our programs more robust and powerful. This tutorial will walk you through installing, importing, and aliasing packages.

How To Write Packages in Go

Go packages are directories that consist of Go code. This tutorial will guide you through writing Go packages for use within other programming files.

Understanding Package Visibility in Go

Visibility in the Go programming language means the file space from which a package or other construct can be referenced. In this article, you will learn how to control package visibility, as well as how to protect parts of your code that should only be used inside your package. To do this, we will create a basic logger to log and debug messages, using packages with varying degrees of item visibility.

How To Write Conditional Statements in Go

Conditional statements are part of every programming language. With conditional statements, we can have code that sometimes runs and at other times does not run, depending on the conditions of the program at that time. This tutorial will take you through writing conditional statements in the Go programming language.

How To Write Switch Statements in Go

switch is an alternative conditional statement useful for communicating actions taken by your Go programs when presented with different options. Everything we can write with the switch statement can also be written with if statements. We'll look at a few examples of what the switch statement can do, the if statements it replaces, and where it's most appropriately applied.

How To Construct For Loops in Go

In the Go programming language, a for loop implements the repeated execution of code based on a loop counter or loop variable. In this tutorial, you will learn how Go’s for loop works, including the three major variations of its use: ForClause, Condition, and RangeClause. We'll start by showing how to create different types of for loops, followed by how to loop through sequential data types in Go. We'll end by explaining how to use nested loops.

Using Break and Continue Statements When Working with Loops in Go

Using for loops in Go allow you to automate and repeat tasks in an efficient manner. Learning how to control the operation and flow of loops will allow for customized logic in your program. You can control your loops with the break and continue statements.

How To Define and Call Functions in Go

A function is a section of code that, once defined, can be reused. Functions are used to make your code easier to understand by breaking it into small, understandable tasks that can be used more than once throughout your program. In this tutorial, we’ll go over how to define your own functions to use in your coding projects.

How To Use Variadic Functions in Go

A variadic function is a function that accepts zero, one, or more values as a single argument. While variadic functions are not the common case, they can be used to make your code cleaner and more readable.

Understanding defer in Go

Go has many of the common control flow keywords found in other programming languages such as if, switch, for, etc. One keyword that isn't found in most other programming languages is defer, and though it's less common you'll quickly see how useful it can be in your programs. In this article we will learn how to properly use the defer statement for cleaning up resources as well as several common mistakes that are made when using defer.

Understanding init in Go

In Go, the predefined init() function sets off a piece of code to run before any other part of your package. This code will execute as soon as the package is imported, and can be used when you need your application to initialize in a specific state. In this tutorial, you'll learn how init() is used for the setup and initialization of specific package variables, one time computations, and the registration of a package for use with another package.

Customizing Go Binaries with Build Tags

In Go, a build tag, or a build constraint, is an identifier added to a piece of code that determines when the file should be included in a package during the build process. This allows you to build different versions of your Go application from the same source code and to toggle between them in a fast and organized manner. In this article, you will use build tags in Go to generate different executable binaries that offer Free, Pro, and Enterprise feature sets of a sample application.

Understanding Pointers in Go

When writing software in Go you'll be writing functions and methods. You pass data to these functions as arguments. Sometimes, the function needs a local copy of the data, and you want the original to remain unchanged. In this article, you will learn how to create and use pointers to share access to the memory space for a variable.

Defining Structs in Go

Structs allow storing data from several variables in a single entity with one name. They allow Go developers to describe the world in which a Go program operates. Instead of reasoning about strings describing a Street, City, or a PostalCode, structs allow us to instead talk about an Address. They also serve as a natural nexus for documentation. Structs can be defined and used in a few different ways, which are discussed in this tutorial.

Defining Methods in Go

Methods are Go functions that operate on instances of a specific type. Methods allow you to communicate not only what the data is, but also how that data should be used. Methods are the core concept that makes Go interfaces possible.

How To Build and Install Go Programs

In Go, distributing or deploying your application requires you to build your code into a shareable binary executable. To do this, you can use the Go toolchain to build and install your program. In this tutorial, you will use the Go toolchain to run, build, and install a sample Hello, World! program, allowing you to use, distribute, and deploy future applications effectively.

How To Use Struct Tags in Go

Struct tags are small pieces of metadata attached to fields of a struct that provide instructions to other Go code that works with the struct. When you read information from systems such as databases, or APIs, you can use struct tags to control how this information is assigned to the fields of a struct.

How To Use Interfaces in Go

In this article, we will learn how to compose custom types that have common behaviors, which will allow us to reuse our code. You'll also learn how to implement interfaces for your own custom types that will satisfy interfaces defined from another package.

Building Go Applications for Different Operating Systems and Architectures

Go supports cross-platform compiling by building support for multiple platforms directly into the go build tool. By using the GOOS and GOARCH environment variables and build tags, you can control which OS and architecture your final binary is built for. In this tutorial, you will build binaries for multiple operating systems and system architectures on your own system.

Using ldflags to Set Version Information for Go Applications

In this tutorial, you will use the Go flag -ldflags to change the value of variables at build time and introduce your own dynamic information into a binary, using a sample application that prints version information to the screen. This passes a flag to the underlying Go toolchain linker, cmd/link, that allows you to change the values of imported packages at build time from the command line.

How To Use the Flag Package in Go

In this tutorial you'll explore various ways to use the flag package to build different kinds of command-line utilities. You'll use a flag to control program output, introduce positional arguments where you mix flags and other data, and then implement sub-commands.

How To Configure Nginx as a Web Server and Reverse Proxy for Apache on One Ubuntu 18.04 Server

DigitalOcean — Fri, 13 Jul 2018 21:42:00 +0000

By Jesin A

The author selected the Electronic Frontier Foundation to receive a donation as part of the Write for DOnations program.

Introduction

Apache and Nginx are two popular open-source web servers often used with PHP. It can be useful to run both of them on the same virtual machine when hosting multiple websites which have varied requirements. The general solution for running two web servers on a single system is to either use multiple IP addresses or different port numbers.

Servers which have both IPv4 and IPv6 addresses can be configured to serve Apache sites on one protocol and Nginx sites on the other, but this isn't currently practical, as IPv6 adoption by ISPs is still not widespread. Having a different port number like 81 or 8080 for the second web server is another solution, but sharing URLs with port numbers (such as http://example.com:81) isn't always reasonable or ideal.

In this tutorial you'll configure Nginx as both a web server and as a reverse proxy for Apache – all on a single server.

Depending on the web application, code changes might be required to keep Apache reverse-proxy-aware, especially when SSL sites are configured. To avoid this, you will install an Apache module called mod_rpaf which rewrites certain environment variables so it appears that Apache is directly handling requests from web clients.

We will host four domain names on one server. Two will be served by Nginx: example.com (the default virtual host) and sample.org. The remaining two, foobar.net and test.io, will be served by Apache. We'll also configure Apache to serve PHP applications using PHP-FPM, which offers better performance over mod_php.

Prerequisites

To complete this tutorial, you'll need the following:

A new Ubuntu 18.04 server configured by following the Initial Server Setup with Ubuntu 18.04, with a sudo non-root user and a firewall.
Four fully-qualified domain names configured to point to your server's IP address. See Step 3 of How To Set Up a Host Name with DigitalOcean for an example of how to do this. If you host your domains' DNS elsewhere, you should create appropriate A records there instead.

Step 1 — Installing Apache and PHP-FPM

Let's start by installing Apache and PHP-FPM.

In addition to Apache and PHP-FPM, we will also install the PHP FastCGI Apache module, libapache2-mod-fastcgi, to support FastCGI web applications.

First, update your package list to ensure you have the latest packages.

sudo apt update

Next, install the Apache and PHP-FPM packages:

sudo apt install apache2 php-fpm

The FastCGI Apache module isn't available in Ubuntu's repository so download it from kernel.org and install it using the dpkg command.


wget https://mirrors.edge.kernel.org/ubuntu/pool/multiverse/liba/libapache-mod-fastcgi/libapache2-mod-fastcgi_2.4.7~0910052141-1.2_amd64.deb

sudo dpkg -i libapache2-mod-fastcgi_2.4.7~0910052141-1.2_amd64.deb

Next, let's change Apache's default configuration to use PHP-FPM.

Step 2 — Configuring Apache and PHP-FPM

In this step we will change Apache's port number to 8080 and configure it to work with PHP-FPM using the mod_fastcgi module. Rename Apache's ports.conf configuration file:

sudo mv /etc/apache2/ports.conf /etc/apache2/ports.conf.default

Create a new ports.conf file with the port set to 8080:

echo "Listen 8080" | sudo tee /etc/apache2/ports.conf

Note: Web servers are generally set to listen on 127.0.0.1:8080 when configuring a reverse proxy but doing so would set the value of PHP's environment variable SERVER_ADDR to the loopback IP address instead of the server's public IP. Our aim is to set up Apache in such a way that its websites do not see a reverse proxy in front of it. So, we will configure it to listen on 8080 on all IP addresses.

Next we'll create a virtual host file for Apache. The <VirtualHost> directive in this file will be set to serve sites only on port 8080.

Disable the default virtual host:

sudo a2dissite 000-default

Then create a new virtual host file, using the existing default site:

sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/001-default.conf

Now open the new configuration file:

sudo nano /etc/apache2/sites-available/001-default.conf

Change the listening port to 8080:

/etc/apache2/sites-available/000-default.conf

<VirtualHost *:8080>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Save the file and activate the new configuration file:

sudo a2ensite 001-default

Then reload Apache:

sudo systemctl reload apache2

Verify that Apache is now listening on 8080:

sudo netstat -tlpn

The output should look like the following example, with apache2 listening on 8080:

OutputActive Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1086/sshd
tcp6 0 0 :::8080 :::* LISTEN 4678/apache2
tcp6 0 0 :::22 :::* LISTEN 1086/sshd

Once you verify that Apache is listening on the correct port, you can configure support for PHP and FastCGI.

Step 3 — Configuring Apache to Use mod_fastcgi

Apache serves PHP pages using mod_php by default, but it requires additional configuration to work with PHP-FPM.

Note: If you are trying this tutorial on an existing installation of LAMP with mod_php, disable it first with sudo a2dismod php7.2.

We will be adding a configuration block for mod_fastcgi which depends on mod_action. mod_action is disabled by default, so we first need to enable it:

sudo a2enmod actions

Rename the existing FastCGI configuration file:

sudo mv /etc/apache2/mods-enabled/fastcgi.conf /etc/apache2/mods-enabled/fastcgi.conf.default

Create a new configuration file:

sudo nano /etc/apache2/mods-enabled/fastcgi.conf

Add the following directives to the file to pass requests for .php files to the PHP-FPM UNIX socket:

/etc/apache2/mods-enabled/fastcgi.conf

<IfModule mod_fastcgi.c>
  AddHandler fastcgi-script .fcgi
  FastCgiIpcDir /var/lib/apache2/fastcgi
  AddType application/x-httpd-fastphp .php
  Action application/x-httpd-fastphp /php-fcgi
  Alias /php-fcgi /usr/lib/cgi-bin/php-fcgi
  FastCgiExternalServer /usr/lib/cgi-bin/php-fcgi -socket /run/php/php7.2-fpm.sock -pass-header Authorization
  <Directory /usr/lib/cgi-bin>
    Require all granted
  </Directory>
</IfModule>

Save the changes and do a configuration test:

sudo apachectl -t

Reload Apache if Syntax OK is displayed:

sudo systemctl reload apache2

If you see the warning Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message., you can safely ignore it for now. We'll configure server names later.

Now let's make sure we can serve PHP from Apache.

Step 4 — Verifying PHP Functionality

Let's make sure that PHP works by creating a phpinfo() file and accessing it from a web browser.

Create the file /var/www/html/info.php which contains a call to the phpinfo function:

echo "<?php phpinfo(); ?>" | sudo tee /var/www/html/info.php

To see the file in a browser, go to http://your_server_ip:8080/info.php. This will give you a list of the configuration settings PHP is using. You'll see output similar to this:

At the top of the page, check that Server API says FPM/FastCGI. About two-thirds of the way down the page, the PHP Variables section will tell you the SERVER_SOFTWARE is Apache on Ubuntu. These confirm that mod_fastcgi is active and Apache is using PHP-FPM to process PHP files.

Step 5 — Creating Virtual Hosts for Apache

Let's create Apache virtual host files for the domains foobar.net and test.io. To do that, we'll first create document root directories for both sites and place some default files in those directories so we can easily test our configuration.

First, create the document root directories:

sudo mkdir -v /var/www/foobar.net /var/www/test.io

Then create an index file for each site:


echo "<h1 style='color: green;'>Foo Bar</h1>" | sudo tee /var/www/foobar.net/index.html

echo "<h1 style='color: red;'>Test IO</h1>" | sudo tee /var/www/test.io/index.html

Then create a phpinfo() file for each site so we can test that PHP is configured properly.


echo "<?php phpinfo(); ?>" | sudo tee /var/www/foobar.net/info.php

echo "<?php phpinfo(); ?>" | sudo tee /var/www/test.io/info.php

Now create the virtual host file for the foobar.net domain:

sudo nano /etc/apache2/sites-available/foobar.net.conf

Add the following code to the file to define the host:

/etc/apache2/sites-available/foobar.net.conf

    <VirtualHost *:8080>
        ServerName foobar.net
        ServerAlias www.foobar.net
        DocumentRoot /var/www/foobar.net
        <Directory /var/www/foobar.net>
            AllowOverride All
        </Directory>
    </VirtualHost>

The line AllowOverride All enables .htaccess support.

These are only the most basic directives. For a complete guide on setting up virtual hosts in Apache, see How To Set Up Apache Virtual Hosts on Ubuntu 16.04.

Save and close the file. Then create a similar configuration for test.io. First create the file:

sudo nano /etc/apache2/sites-available/test.io.conf

Then add the configuration to the file:

/etc/apache2/sites-available/test.io.conf

    <VirtualHost *:8080>
        ServerName test.io
        ServerAlias www.test.io
        DocumentRoot /var/www/test.io
        <Directory /var/www/test.io>
            AllowOverride All
        </Directory>
    </VirtualHost>

Save the file and exit the editor.

Now that both Apache virtual hosts are set up, enable the sites using the a2ensite command. This creates a symbolic link to the virtual host file in the sites-enabled directory:


sudo a2ensite foobar.net

sudo a2ensite test.io

Check Apache for configuration errors again:

sudo apachectl -t

You'll see Syntax OK displayed if there are no errors. If you see anything else, review the configuration and try again.

Reload Apache to apply the changes once your configuration is error-free:

sudo systemctl reload apache2

To confirm the sites are working, open http://foobar.net:8080 and http://test.io:8080 in your browser and verify that each site displays its index.html file.

You'll see the following results:

Also, ensure that PHP is working by accessing the info.php files for each site. Visit http://foobar.net:8080/info.php and http://test.io:8080/info.php in your browser.

You'll see the same PHP configuration spec list on each site as you saw in Step 4.

We now have two websites hosted on Apache at port 8080. Let's configure Nginx next.

Step 6 — Installing and Configuring Nginx

In this step we'll install Nginx and configure the domains example.com and sample.org as Nginx's virtual hosts. For a complete guide on setting up virtual hosts in Nginx, see How To Set Up Nginx Server Blocks (Virtual Hosts) on Ubuntu 18.04.

Install Nginx using the package manager:

sudo apt install nginx

Then remove the default virtual host's symlink since we won't be using it any more:

sudo rm /etc/nginx/sites-enabled/default

We'll create our own default site later (example.com).

Now we'll create virtual hosts for Nginx using the same procedure we used for Apache. First create document root directories for both the websites:

sudo mkdir -v /usr/share/nginx/example.com /usr/share/nginx/sample.org

We'll keep the Nginx web sites in /usr/share/nginx, which is where Nginx wants them by default. You could put them under /var/www/html with the Apache sites, but this separation may help you associate sites with Nginx.

As you did with Apache's virtual hosts, create index and phpinfo() files for testing after setup is complete:


echo "<h1 style='color: green;'>Example.com</h1>" | sudo tee /usr/share/nginx/example.com/index.html

echo "<h1 style='color: red;'>Sample.org</h1>" | sudo tee /usr/share/nginx/sample.org/index.html

echo "<?php phpinfo(); ?>" | sudo tee /usr/share/nginx/example.com/info.php

echo "<?php phpinfo(); ?>" | sudo tee /usr/share/nginx/sample.org/info.php

Now create a virtual host file for the domain example.com:

sudo nano /etc/nginx/sites-available/example.com

Nginx calls server {. . .} areas of a configuration file server blocks. Create a server block for the primary virtual host, example.com. The default_server configuration directive makes this the default virtual host which processes HTTP requests which do not match any other virtual host.

/etc/nginx/sites-available/example.com

server {
    listen 80 default_server;

    root /usr/share/nginx/example.com;
    index index.php index.html index.htm;

    server_name example.com www.example.com;
    location / {
        try_files $uri $uri/ /index.php;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        include snippets/fastcgi-php.conf;
    }
}

Save and close the file. Now create a virtual host file for Nginx's second domain, sample.org:

sudo nano etc/nginx/sites-available/sample.org

Add the following to the file:

/etc/nginx/sites-available/sample.org

server {
    root /usr/share/nginx/sample.org;
    index index.php index.html index.htm;

    server_name sample.org www.sample.org;
    location / {
        try_files $uri $uri/ /index.php;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        include snippets/fastcgi-php.conf;
    }
}

Save and close the file.

Then enable both sites by creating symbolic links to the sites-enabled directory:


sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com

sudo ln -s /etc/nginx/sites-available/sample.org /etc/nginx/sites-enabled/sample.org

Then test the Nginx configuration to ensure there are no configuration issues:

sudo nginx -t

Then reload Nginx if there are no errors:

sudo systemctl reload nginx

Now access the phpinfo() file of your Nginx virtual hosts in a web browser by visiting http://example.com/info.php and http://sample.org/info.php. Look under the PHP Variables sections again.

["SERVER_SOFTWARE"] should say nginx, indicating that the files were directly served by Nginx. ["DOCUMENT_ROOT"] should point to the directory you created earlier in this step for each Nginx site.

At this point, we have installed Nginx and created two virtual hosts. Next we will configure Nginx to proxy requests meant for domains hosted on Apache.

Step 7 — Configuring Nginx for Apache's Virtual Hosts

Let's create an additional Nginx virtual host with multiple domain names in the server_name directives. Requests for these domain names will be proxied to Apache.

Create a new Nginx virtual host file to forward requests to Apache:

sudo nano /etc/nginx/sites-available/apache

Add the following code block which specifies the names of both Apache virtual host domains and proxies their requests to Apache. Remember to use the public IP address in proxy_pass:

/etc/nginx/sites-available/apache

server {
    listen 80;
    server_name foobar.net www.foobar.net test.io www.test.io;

    location / {
        proxy_pass http://your_server_ip:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Save the file and enable this new virtual host by creating a symbolic link:

sudo ln -s /etc/nginx/sites-available/apache /etc/nginx/sites-enabled/apache

Test the configuration to ensure there are no errors:

sudo nginx -t

If there are no errors, reload Nginx:

sudo systemctl reload nginx

Open the browser and access the URL http://foobar.net/info.php in your browser. Scroll down to the PHP Variables section and check the values displayed.

The variables SERVER_SOFTWARE and DOCUMENT_ROOT confirm that this request was handled by Apache. The variables HTTP_X_REAL_IP and HTTP_X_FORWARDED_FOR were added by Nginx and should show the public IP address of the computer you're using to access the URL.

We have successfully set up Nginx to proxy requests for specific domains to Apache. Next, let's configure Apache to set the REMOTE_ADDR variable as if it were handling these requests directly.

Step 8 — Installing and Configuring mod_rpaf

In this step you'll install an Apache module called mod\_rpaf which rewrites the values of REMOTE_ADDR , HTTPS and HTTP_PORT based on the values provided by a reverse proxy. Without this module, some PHP applications would require code changes to work seamlessly from behind a proxy. This module is present in Ubuntu's repository as libapache2-mod-rpaf but is outdated and doesn't support certain configuration directives. Instead, we will install it from source.

Install the packages needed to build the module:

sudo apt install unzip build-essential apache2-dev

Download the latest stable release from GitHub:

wget https://github.com/gnif/mod_rpaf/archive/stable.zip

Extract the downloaded file:

unzip stable.zip

Change into the new directory containing the files:

cd mod_rpaf-stable

Compile and install the module:


make

sudo make install

Next, create a file in the mods-available directory which will load the rpaf module:

sudo nano /etc/apache2/mods-available/rpaf.load

Add the following code to the file to load the module:

/etc/apache2/mods-available/rpaf.load

LoadModule rpaf_module /usr/lib/apache2/modules/mod_rpaf.so

Save the file and exit the editor.

Create another file in this directory called rpaf.conf which will contain the configuration directives for mod_rpaf:

sudo nano /etc/apache2/mods-available/rpaf.conf

Add the following code block to configure mod_rpaf, making sure to specify the IP address of your server:

/etc/apache2/mods-available/rpaf.conf

    <IfModule mod_rpaf.c>
        RPAF_Enable On
        RPAF_Header X-Real-Ip
        RPAF_ProxyIPs your_server_ip 
        RPAF_SetHostName On
        RPAF_SetHTTPS On
        RPAF_SetPort On
    </IfModule>

Here's a brief description of each directive. See the mod_rpaf README file for more information.

RPAF_Header - The header to use for the client's real IP address.
RPAF_ProxyIPs - The proxy IP to adjust HTTP requests for.
RPAF_SetHostName - Updates the vhost name so ServerName and ServerAlias work.
RPAF_SetHTTPS - Sets the HTTPS environment variable based on the value contained in X-Forwarded-Proto.
RPAF_SetPort - Sets the SERVER_PORT environment variable. Useful for when Apache is behind a SSL proxy.

Save rpaf.conf and enable the module:

sudo a2enmod rpaf

This creates symbolic links of the files rpaf.load and rpaf.conf in the mods-enabled directory. Now do a configuration test:

sudo apachectl -t

Reload Apache if there are no errors:

sudo systemctl reload apache2

Access the phpinfo() pages http://foobar.net/info.php and http://test.io/info.php in your browser and check the PHP Variables section. The REMOTE_ADDR variable will now also be that of your local computer's public IP address.

Now let's set up TLS/SSL encryption for each site.

Step 9 — Setting Up HTTPS Websites with Let's Encrypt (Optional)

In this step we will configure TLS/SSL certificates for both the domains hosted on Apache. We'll obtain the certificates through [Let's Encrypt](https://letsencrypt.org]. Nginx supports SSL termination so we can set up SSL without modifying Apache's configuration files. The mod_rpaf module ensures the required environment variables are set on Apache to make applications work seamlessly behind a SSL reverse proxy.

First we will separate the server {...} blocks of both the domains so that each of them can have their own SSL certificates. Open the file /etc/nginx/sites-available/apache in your editor:

sudo nano /etc/nginx/sites-available/apache

Modify the file so that it looks like this, with foobar.net and test.io in their own server blocks:

/etc/nginx/sites-available/apache

    server {
        listen 80;
        server_name foobar.net www.foobar.net;

        location / {
            proxy_pass http://your_server_ip:8080;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
    server {
        listen 80;
        server_name test.io www.test.io;

        location / {
            proxy_pass http://your_server_ip:8080;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }

We'll use Certbot to generate our TLS/SSL certificates. Its Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary.

First, add the official Certbot repository:

sudo add-apt-repository ppa:certbot/certbot

Press ENTER when prompted to confirm you want to add the new repository. Then update the package list to pick up the new repository's package information:

sudo apt update

Then install Certbot's Nginx package with apt:

sudo apt install python-certbot-nginx

Once it's installed, use the certbot command to generate the certificates for foobar.net and www.foobar.net:

sudo certbot --nginx -d foobar.net -d www.foobar.net

This command tells Certbot to use the nginx plugin, using -d to specify the names we'd like the certificate to be valid for.

If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the Let's Encrypt server, then run a challenge to verify that you control the domain you're requesting a certificate for.

Next, Certbot will ask how you'd like to configure your HTTPS settings:

OutputPlease choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

Select your choice, then press ENTER. The configuration will be updated, and Nginx will reload to pick up the new settings.

Now execute the command for the second domain:

sudo certbot --nginx -d test.io -d www.test.io

Access one of Apache's domains in your browser using the https:// prefix; visit https://foobar.net/info.php and you'll see this:

Look in the PHP Variables section. The variable SERVER_PORT has been set to 443 and HTTPS set to on , as though Apache was directly accessed over HTTPS. With these variables set, PHP applications do not have to be specially configured to work behind a reverse proxy.

Now let's disable direct access to Apache.

Step 10 — Blocking Direct Access to Apache (Optional)

Since Apache is listening on port 8080 on the public IP address, it is accessible by everyone. It can be blocked by working the following IPtables command into your firewall rule set.

sudo iptables -I INPUT -p tcp --dport 8080 ! -s your_server_ip -j REJECT --reject-with tcp-reset

Be sure to use your server's IP address in place of the example in red. Once port 8080 is blocked in your firewall, test that Apache is unreachable on it. Open your web browser and try accessing one of Apache's domain names on port 8080. For example: http://example.com:8080

The browser should display an "Unable to connect" or "Webpage is not available" error message. With the IPtables tcp-reset option in place, an outsider would see no difference between port 8080 and a port that doesn't have any service on it.

Note: IPtables rules do not survive a system reboot by default. There are multiple ways to preserve IPtables rules, but the easiest is to use iptables-persistent in Ubuntu's repository. Explore this article to learn more about how to configure IPTables.

Now let's configure Nginx to serve static files for the Apache sites.

Step 11 — Serving Static Files Using Nginx (Optional)

When Nginx proxies requests for Apache's domains, it sends every file request for that domain to Apache. Nginx is faster than Apache in serving static files like images, JavaScript and style sheets. So let's configure Nginx's apache virtual host file to directly serve static files but send PHP requests on to Apache.

Open the file /etc/nginx/sites-available/apache in your editor:

sudo nano /etc/nginx/sites-available/apache

You'll need to add two additional location blocks to each server block, as well as modify the existing location sections. In addition, you'll need to tell Nginx where to find the static files for each site.

If you've decided not to use SSL and TLS certificates, modify your file so it looks like this:

/etc/nginx/sites-available/apache

server {
    listen 80;
    server_name test.io www.test.io;
    root /var/www/test.io;
    index index.php index.htm index.html;

    location / {
        try_files $uri $uri/ /index.php;
    }

    location ~ \.php$ {
        proxy_pass http://your_server_ip:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location ~ /\.ht {
        deny all;
    }
}

server {
    listen 80;
    server_name foobar.net www.foobar.net;
    root /var/www/foobar.net;
    index index.php index.htm index.html;

    location / {
        try_files $uri $uri/ /index.php;
    }

    location ~ \.php$ {
        proxy_pass http://your_ip_address:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location ~ /\.ht {
        deny all;
    }
}

If you also want HTTPS to be available, use the following configuration instead:

/etc/nginx/sites-available/apache

server {
    listen 80;
    server_name test.io www.test.io;
    root /var/www/test.io;
    index index.php index.htm index.html;

    location / {
        try_files $uri $uri/ /index.php;
    }

    location ~ \.php$ {
        proxy_pass http://your_server_ip:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location ~ /\.ht {
        deny all;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/test.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/test.io/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

server {
    listen 80;
    server_name foobar.net www.foobar.net;
    root /var/www/foobar.net;
    index index.php index.htm index.html;

    location / {
        try_files $uri $uri/ /index.php;
    }

    location ~ \.php$ {
        proxy_pass http://your_ip_address:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location ~ /\.ht {
        deny all;
    }

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/foobar.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/foobar.net/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

The try_files directive makes Nginx look for files in the document root and directly serve them. If the file has a .php extension, the request is passed to Apache. Even if the file is not found in the document root, the request is passed on to Apache so that application features like permalinks work without problems.

Warning: The location ~ /\.ht directive is very important; this prevents Nginx from serving the contents of Apache configuration files like .htaccess and .htpasswd which contain sensitive information.

Save the file and perform a configuration test:

sudo nginx -t

Reload Nginx if the test succeeds:

sudo service nginx reload

To verify things are working, you can examine Apache's log files in /var/log/apache2 and see the GET requests for the info.php files of test.io and foobar.net. Use the tail command to see the last few lines of the file, and use the -f switch to watch the file for changes:

sudo tail -f /var/log/apache2/other_vhosts_access.log

Now visit http://test.io/info.php in your browser and then look at the output from the log. You'll see that Apache is indeed replying:

Output test.io:80 your_server_ip - - [01/Jul/2016:18:18:34 -0400] "GET /info.php HTTP/1.0" 200 20414 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"

Then visit the index.html page for each site and you won't see any log entries from Apache. Nginx is serving them.

When you're done observing the log file, press CTRL+C to stop tailing it.

With this setup, Apache will not be able to restrict access to static files. Access control for static files would need to be configured in Nginx's apache virtual host file, but that's beyond the scope of this tutorial.

Conclusion

You now have one Ubuntu server with Nginx serving example.com and sample.org, along with Apache serving foobar.net and test.io. Though Nginx is acting as a reverse-proxy for Apache, Nginx's proxy service is transparent and connections to Apache's domains appear be served directly from Apache itself. You can use this method to serve secure and static sites.

Build, test, and deploy something new on DigitalOcean - the all-in-one cloud platform developers and their teams love. Get started with a free $100 account credit for new users: do.co/devto

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.