Forem: Roman Glushko

What's OAuth2, anyway?

Roman Glushko — Sun, 26 Jan 2025 13:30:51 +0000

I have finally gotten a chance to dive into the weeds of the OAuth2 protocol and ask all why questions around its components and design 🙌

I have posted my notes and thoughts as the article that covers:

🤔Why do we need OAuth2 and what were the alternatives before it came?
🤝The OAuth2 roles, the general workflow and TOFU
🤖OAuth2 Client Applications, Static Registration and Credentials
🔒Authorization Servers and their typical API
🎟️Access tokens. Why do we need them?
🔄 What’s the point of having access tokens and what they represent?
📚OAuth2 Scopes. What do they really mean?
💃 OAuth2 Authorization Code Flow. Why is it designed this way? The PKCE extension.
💃 OAuth2 Implicit Flow. What’s so implicit about it? Why it was created in OAuth2.0 and deprecated in the OAuth2.1 Draft
🤖 OAuth2 Client Credentials Flow or how to access the Resource Server on Client Application behalf?
🔑 OAuth2 ROC Flow and why was it “deprecated” from day one?
📟 OAuth2 Device Flow or how to do OAuth2 when there is no browser on your target device?
🗺️ Guide how to pick the right flow for your use case

The full article ⬅️

Hope someone find this helpful 🙌

The Era of LLM Infrastructure

Roman Glushko — Thu, 15 Feb 2024 21:18:36 +0000

API access to large language models has opened up a world of opportunities. We have seen many simple proof-of-concept applications show promise in being effective. However, as the complexity of these applications grows, several crucial issues arise when putting these systems into production. These issues include unreliable API endpoints, slow token generation, LLM lock-in, and cost management. Clearly, the LLM era will require solutions to manage LLM API endpoints.

Glide is a cloud-native LLM gateway that provides a lightweight interface to manage the complexity of working with multiple LLM providers.

Unified API

Glide offers a comprehensive API that facilitates interaction with multiple LLM providers. Instead of dedicating considerable time and resources to developing custom integrations for individual LLM providers, Glide provides a single API interface that allows users to interact with any LLM provider. Adopting this approach can significantly enhance application development efficiency. By working off a standardized API, engineers can minimize complexity and development time, leading to faster and more efficient application development. Additionally, there is zero LLM model lock-in, as underlying models can be switched without knowledge from the client application.

Glide Routers

A fundamental principle in Glide is the concept of routers. Routers enable you to group models together for shared logic. An excellent example of this is illustrated by a RAG power chatbot, which allows users to search over a documentation set. It is directly built on GPT-3.5 Turbo and entirely depends on OpenAI to keep its API operable. This dependency poses a significant risk to the application and user experience. Therefore, it is recommended to set up a Glide router in resilience mode by adding a single backup model to a router. If the OpenAI API fails, Glide will automatically send the API call to the next model specified in the configuration. In addition, model failure knowledge is shared across all routers, reducing wasteful retries when an LLM provider has a known issue.

Another essential router type is the least-latency router. This router selects the model with the lowest average latency per generated token. Since we don’t know the actual distribution of model latencies, we attempt to estimate it and keep it updated over time. Over time, old latency data is weighted lower and eventually dropped from the calculation. This ensures latencies are constantly updated. As with all routers, if a model becomes unhealthy, it will pick the second-best, etc.

Other routing modes are available, such as round-robin, which is excellent for A/B testing, and weighted round-robin, which helps specify the percentage of traffic that should be sent to a set of models.

One Glide deployment can support multiple applications with diverse requirements since it can support numerous routers. There are also exciting routers on the roadmap, such as intelligent routing, which ensures your request is sent to the model best suited for that request.

Declarative Configuration

Glide simplifies the setup process through declarative configuration, which defines the state of the Glide gateway in one place. This also means that secret management is centralized, enabling the rotation of API keys from a single location.

Furthermore, this approach enables the separation of responsibilities between teams. One team can manage the infrastructure, deploy Glide, and make it available to other teams (such as AI/DS teams) while also being responsible for rotating keys. Meanwhile, other teams can solely focus on working with models and not worry about these configurations.

Here is a bare-bones configuration example:

routers:
  language:
    - id: my-chat-app
      strategy: priority
      models:
        - id: primary
          openai:
            model: "gpt-3.5-turbo"
            api_key: ${env:OPENAI_API_KEY}
        - id: secondary
          azureopenai:
            api_key: ${env:AZUREOAI_API_KEY}
            model: "glide-GPT-35" # the Azure OpenAI deployment name
            base_url: "https://mydeployment.openai.azure.com/"

With this simple configuration a priority/fallback router has been created. All requests will be sent to OpenAI, should the OpenAI API fail the request will be sent to an Azure OpenAI deployment.

What's Next?

The future of LLM applications will be multi-modal, with text, speech, and vision models employed together to create rich user experiences. Glide will be the go-to gateway for these applications. Glide plans to support various features over the next several months, including exact and semantic caching, embedding endpoints, speech endpoints, safety policies, and monitoring features.

If you are interested in using Glide, here is a list of links for you to check out:

🛠️ Github: https://github.com/EinStack/glide
📚 Docs: https://glide.einstack.ai/
💬 Discord: https://discord.gg/upVttzRfpn
🗺️ Roadmap: https://github.com/EinStack/glide/blob/develop/ROADMAP.md

360-degree Review is Dead

Roman Glushko — Mon, 26 Jul 2021 06:59:11 +0000

No reasons to be sad about it, because there is a better ways to organize your feedback flow.

I have shared my experience of 360 review applications in the real world and how I replaced it with a much more effective approaches.

https://www.romaglushko.com/thoughts/360-review-is-dead/

Hope this will be useful for someone 🙌

Awesome List of Distributed System Projects

Roman Glushko — Wed, 07 Jul 2021 07:38:29 +0000

I have recently dug into distributed system design. I have found lots of open source projects to inspire from 💫

So I decided to start composing an awesome list on GitHub:

https://github.com/roma-glushko/awesome-distributed-system-projects

The list currently contains projects like:

Databases (SQL, NoSQL, KV)
Locking services
Streaming Services
Message Queues
Task Schedulers
File Systems
OS, Service Discovery
frameworks and components

Most of the projects are written in Golang, some in Java, Erlang, C++ and a few in Python, Rust, Kotlin.

Hope this is going to be helpful 🙌
Feel free to give it a star if you like it and share your fav projects which is not yet on the list 🚀

How to design LRU cache?

Roman Glushko — Wed, 16 Jun 2021 12:57:28 +0000

This is, by the way, not a random question, but a popular coding interview challenge and an interesting problem by itself which made me write a whole blog post:

https://www.romaglushko.com/blog/design-lru-cache/ ⬅️

TLDR: you need to design a kind of custom data structure in order to make the cache storage to work in constant time which actually makes it practically useful. So be sure to check out the whole walkthrough above.

Learn Heaps via Visualizations

Roman Glushko — Sun, 16 May 2021 08:31:46 +0000

Heaps (or priority queues) are tree-based data structure that makes it fast to track order of retrieving priority items 🛠

I have reviewed how heaps work in depth in the following blog post ⬇️

https://www.romaglushko.com/blog/heapify/

The article contains a lot of visualizations and could be a good way to understand how heaps work, especially for beginners. Also, heap is a popular topic on coding interviews.

Hope it'll be helpful 🙌

How I built my deep learning workstation?

Roman Glushko — Wed, 21 Apr 2021 15:17:58 +0000

Recently I have managed to build my little deep learning workstation 🎉
And I wanted to share my experience with you. So I wrote a blog post where tried to cover all aspects of building such a project including:

reasons why I went for this
PC parts selection
hardware installation
software and CUDA installation
my current workflow

I feel like this may be helpful for enthusiasts like myself who think about own GPU-accelerated machine:

https://www.romaglushko.com/blog/how-i-built-my-ml-workstation/ ⬅️

I hope this will be helpful. Let me know what do you think in the comments!

Tango is on the scene 💃

Roman Glushko — Sun, 23 Feb 2020 09:37:36 +0000

Everybody, Tango is on the scene 💃

I have released a simple CLI tool to streamline access log analyzing workflow 🎉

Wondering what's inside?

https://github.com/roma-glushko/tango

Tango is designed to generate different reports to be helpful during log analyzing routine:

analyzing how incoming traffic affects website performance
investigating security incidents and their impact on the website
filtering/aggregating logs to get other kinds of insights from them

Currently, the following reports are available for you 🛠:

Custom Report: converts access logs to CSV format with the ability to filter records
Geo Report: collects GEO information for all IPs that accessing the website with the ability to sort by countries and continents.
Browser Report: groups requests by browsers/crawlers
Request Report: see which URLs were requested and how many times on the website
Pace Report: check how many requests were sent during a certain time frame by each IP

It is possible to install Tango on macOS/Linux/Windows 📦

# macOS (via Homebrew)
brew tap roma-glushko/tango
brew install roma-glushko/tango/tango

# Linux (via Snap)
sudo snap install tango

# Windows (via Scoop)
scoop bucket add tango https://github.com/roma-glushko/scoop-tango.git
scoop install tango

Finally, here are some examples of usage:

# generate the pace report
tango pace -l access-log.log -r custom.csv

# get all requests from 128.2.3.4 IP
tango --ip-filter "128.2.3.4" custom -l access-log.log -r custom.csv

# get all requests to /newsletter/subscriber/new URL
tango --keep-uri-filter "/newsletter/subscriber/new/" custom -l access-log.log -r custom.csv

# get all IPs that requested /admin URL with GEO data
tango --keep-uri-filter "/admin/" geo -l access-log.log -r custom.csv

Give it a try today and you won't be disappointed:

https://github.com/roma-glushko/tango 💃💃💃