How ChatGPT Uses Server-Sent Events to Stream Real-Time Conversation

Rohit Dhas ⚡ — Wed, 28 Jun 2023 07:21:49 +0000

Whether you're a developer or not, you're probably using ChatGPT in your daily work. But have you ever wondered how ChatGPT manages to deliver real-time conversations that feel so incredibly human?

In this article, I'll show you how the ChatGPT API uses SSE (Server-Sent Events) to stream responses in real time as they're being generated. Additionally, I'll guide you through the process of creating your own basic SSE endpoint using Node.js and Express.js.

So what is SSE?

In simple words, It's one request and a very long response.

SSE is one of my favorite backend communication patterns. It allows a server to push data to a client as soon as it becomes available, without the client having to continuously ask or poll the server for new data. This allows for real-time updates to be delivered to the client without the need for constant requests.

In the case of ChatGPT, when you send a message, the server may immediately start generating a response using machine learning algorithms. As soon as the server generates a new piece of text, it can send it to the client using SSE, which allows the client to render the response as soon as it arrives.

It's worth noting that SSE is a unidirectional protocol, meaning that the client cannot send data back to the server using the same connection.

Implementation

Now that you're familiar with SSE, Let's try to implement a basic SSE endpoint using Node. JS & Express. JS.

To get started, Set up your Node.js project and install the necessary dependenciess



$ mkdir sse-example
$ cd sse-example
$ npm init -y
$ npm install express

Now, create a index.js file and initialize an express app. Then define a route to handle SSE connection.



const express = require('express');
const app = express();

app.get('/', (req, res) => {  
    res.send(`  
    <!DOCTYPE html>  
    <html>  
        <body>  
            <h1>SSE Example:</h1>  
            <div id="result"></div>  
            <script>  
                var source = new EventSource("/stream");  
                source.onmessage = function(event) {  
                document.getElementById("result").innerHTML += event.data + "<br>";  
                };  
            </script>  
        </body>  
    </html>  
`)  
})

app.get('/stream', (req, res) => {
  // Required header: It tells client that the response will be an event stream
  res.setHeader('Content-Type', 'text/event-stream');
  sendCount(res);
});

let counter = 0;
function sendCount(res) {
  setInterval(() => {
    res.write(`data: Count: [${counter++}]\n\n`);
  }, 1000);
}

Let me break down the code for you:

First there's a GET endpoint ("/"), that serves an HTML page.
- The HTML page includes some markup and a script tag.
- Within the script tag, an EventSource object is created with the URL /stream to establish a connection between the client and the SSE endpoint.
- The onmessage event listener is set on the source object, for capturing events received from server.
- Inside the callback, we insert the data to the div to display received message in HTML page.
Then there's GET endpoint ("/stream"), which will be used to stream real-time data.
- I have created a function called sendData that takes the response object and sends data every 1 second using res.write. This mimics a real-world scenario where GPT generates responses through its machine learning algorithms.
- It is worth mentioning that you have to send the word data: in every event because it determines the boundary of an event. By starting event data with word data: and ending with \n\n, allows clients to capture and parse the events correctly.

After setting up these two endpoints, start the server and listen on port 8080.



const server = app.listen(8080, () => {
  console.log('Server is listening on port 8080!');
});

Now, Open your browser, and head over to http://localhost:8080.

You should see some text being rendered on the page every 1 second, representing the events received from the server.

This is just a basic implementation to get you started. You can build upon this foundation to create more complex SSE applications tailored to your specific needs.

Thanks for reading!

Forem: Rohit Dhas ⚡

How ChatGPT Uses Server-Sent Events to Stream Real-Time Conversation

So what is SSE?

Implementation