Forem: Tiioluwani

5 Best Python Vector Database Libraries

Tiioluwani — Mon, 18 May 2026 12:42:35 +0000

Most comparisons of Python vector database libraries focus on retrieval speed, indexing algorithms, or benchmark results. These metrics matter, but production failures stem from various factors: installation inconsistencies, client packaging differences, version churn, and unexpected API changes. In reality, a different class of problems appears once the application leaves the notebook environment and runs inside a production service.

A typical example occurs with embedded ChromaDB setups. A project may work perfectly during development, only to fail in production with an error such as:

RuntimeError: Chroma running in http-only client mode.

A structural conflict between the chromadb and chromadb-client packages triggers this error because the client-only package lacks the default embedding functions the application depends on. Diagnosing this can take hours.

Client packaging choices and library design decisions, not retrieval quality or indexing performance, produce this type of failure.

This article compares the leading Python vector database libraries from that perspective, examining client architecture, installation stability, API design, and long-term maintainability, rather than benchmark numbers alone.

TL;DR

ChromaDB: Fastest setup for prototyping and notebook environments with minimal configuration.
Pinecone: Fully managed cloud solution with zero infrastructure management overhead.
Qdrant: Zero code changes from local development to production; the strongest open-source option for API stability.
Weaviate: Hybrid search combining vector similarity and keyword filtering at scale.
Actian VectorAI DB: On-premises deployment with same architecture from laptop to production; Actian designed it for edge and air-gapped environments.

The Python Landscape: Understanding the Options

The relationship between a Python vector database library and its storage backend determines how you will develop, test, and eventually scale your application. The wrong library choice often triggers the environment-specific failures described above, since each architecture handles local and production environments differently.

These differences generally fall into four distinct categories, each with its own approach to the interaction between infrastructure and code.

Four client architectures

Cloud-only (e.g., Pinecone): These clients act as a full API abstraction for serverless environments. The primary advantage is zero infrastructure management, but this requires an active internet connection and an API key for all local development and testing.
OSS with managed option (e.g., Qdrant, Weaviate, Milvus): This set of tools uses the same API for both self-hosted Docker instances and managed cloud services. This provides excellent development-production parity, though it often requires managing a local server or a Docker container during development.
Embedded libraries (e.g., ChromaDB, FAISS): These tools run in-process and embed the database logic in your Python application. While they are ideal for notebooks and rapid prototyping, their developers never designed them for distributed production environments, and they do not offer a well-defined migration path as the application scales.
Extension approach (e.g., pgvector via Timescale vector): This model adds vector search capabilities to traditional relational databases. It allows existing PostgreSQL infrastructure to support vector similarity search. However, query performance varies with index configuration, dataset size, and workload characteristics; some scenarios benefit from the relational foundation, while others favor purpose-built vector architectures.

These four models describe how a client connects to storage, but they also show a practical separation between standalone search libraries and managed database systems. Choosing the wrong model generates some of the most persistent production friction in vector search applications.

A vector database provides the infrastructure required for production readiness, going beyond what developers' standalone libraries offer. Libraries such as FAISS or Annoy are static, in-memory tools focused on approximate nearest neighbor search across large datasets. They are highly efficient for similarity search within a fixed vector space, but they cannot manage data over time.

Specialized databases like Pinecone, Qdrant, or Milvus go further, providing full CRUD support, metadata-based filtering, and distributed persistence for large datasets.

The table below summarizes where each architecture fits across common use cases.

Category	Primary trade-off	Production migration path
Cloud-only	No infrastructure management; requires network connectivity and API authentication for all environments	Same client code across development and production
OSS + managed	Identical API for local and cloud deployments; requires Docker or server setup for local development	Zero code changes between local Docker instance and managed cloud service
Embedded	In-process execution with minimal setup; limited to single-machine architecture	Client class replacement required; distributed deployment needs architecture redesign
Extension	Integrates with existing PostgreSQL infrastructure; performance depends on index configuration and dataset characteristics	Current PostgreSQL setup and scale requirements determine the migration path

Client Comparison: Developer Experience Deep Dive

Architecture does narrow your options, but the day-to-day experience of working with a Python vector database library comes down to how each client handles connection setup, version stability, and the friction points encountered during active development.

We're comparing the four clients below based on what developers deal with in real-world use.

1. Pinecone Python client

Pinecone offers one of the more polished connection experiences among cloud-only vector database clients, with extensive type hints and a straightforward initialization pattern.

from pinecone import Pinecone

    pc = Pinecone(api_key="your-api-key")
    index = pc.Index("your-index-name")

Strengths:

Extensive type hints and IDE autocompletion support.
Pinecone introduced AsyncIO support in v6 via Pinecone Asyncio.
gRPC mode offers higher throughput for demanding workloads.
Well-maintained official documentation.

Pain points:

Pinecone shipped three major versions in 18 months (v5, v6, and v7), introducing breaking changes to the connection logic and renaming the package from pinecone-client to pinecone.
Historical confusion between the pinecone and pinecone-client packages.
query_namespaces async operations under load require thread pool tuning.

When to choose Pinecone:

Fully managed infrastructure is a hard requirement.
The team has no appetite for self-hosted database management.
The Budget allows for cloud-only pricing at the target scale.

When to avoid Pinecone:

Local development requires a live API key.
API stability across versions is a priority.
Cost at scale above 100M vectors is a constraint.

2. Weaviate Python client

Weaviate's v4 client is a meaningful step forward from v3, adding typed classes and gRPC support that noticeably improve query performance.

import weaviate

    client = weaviate.connect_to_local()
    collection = client.collections.get("your-collection-name")

Strengths:

gRPC mode delivers 40-70% faster query performance than v3.
Typed property and DataType classes replace untyped v3 dictionaries.
Built-in hybrid search combining vector and keyword search.
Strong support for multi-tenancy workloads.

Pain points:

Weaviate has fully deprecated the v3 API, and teams report that the migration takes weeks of work.
gRPC requires port 50051 to be open, which creates friction in restricted network environments.
Batch API redesign caused significant confusion (Issue #433).
LangChain did not ship v4 support until several months after Weaviate's release (Issue #14531).

When to choose Weaviate:

Hybrid search combining vector similarity and keyword filtering is a core requirement.
gRPC performance gains justify the network configuration overhead.
The team has the capacity to manage the v3-to-v4 migration.

When to avoid Weaviate:

Migration resources are limited, and API stability is a priority.
Network environments restrict non-standard port access.

3. ChromaDB Python client

ChromaDB offers one of the easiest onboarding experiences among Python vector database libraries, making it a natural starting point for notebooks and early-stage prototyping.

 import chromadb

    # In-memory mode
    client = chromadb.Client()

    # Persistent mode
    client = chromadb.PersistentClient(path="/your/path")

    # HTTP client mode
    client = chromadb.HttpClient(host="localhost", port=8000)

Strengths:

Simplest API surface of any client in this comparison.
Mature LangChain integration with well-documented examples.
In-memory mode requires zero configuration for notebook environments.
Large and active open-source community.

Pain points:

Python 3.13 incompatibility (Issue #3651).
Windows instability above 99 records (Issue #3058).
Confusing chromadb with chromadb-client breaks production deployments.
hnswlib compilation errors on ARM Mac processors.
Requires SQLite 3.35 or higher, creating an environment-specific setup overhead.

When to choose ChromaDB:

Rapid prototyping in a notebook environment is the primary use case.
The dataset fits comfortably within a single process.
LangChain integration with minimal configuration is a priority.

When to avoid ChromaDB:

Production deployment on Windows or Python 3.13 is required.
The application needs to scale beyond a single machine.
A clear and well-defined migration path to a distributed infrastructure is important.

4. Qdrant Python client

Developers value Qdrant for its local-to-production parity. The same client code runs against an in-memory instance during development and a fully managed cloud deployment in production, without requiring any modifications.

from qdrant_client import QdrantClient

    # In-memory mode for local development
    client = QdrantClient(":memory:")

    # Production mode - zero code changes required
    client = QdrantClient(url="https://your-cluster-url", api_key="your-api-key")

Strengths:

:memory: mode enables a zero-code-change local-to-production workflow.
Qdrant introduced a native AsyncQdrantClient for high-concurrency workloads.
Pydantic model type safety throughout the client interface.
Rust-backed implementation with lower memory overhead compared to JVM-based alternatives.

Pain points:

Developers must explicitly set prefer_grpc=True to enable gRPC, a step they often overlook.
Port split between REST (6333) and gRPC (6334) requires careful network configuration.
Pydantic version constraints: v1.10.x or v2.21 and above only.
Cloud connection issues (Issue #112).

When to choose Qdrant:

Local-to-production parity is a priority, and zero code changes between environments matter.
High-concurrency async workloads require native AsyncQdrantClient support.
You prefer a self-hosted, open-source vector database deployment over a managed cloud service.
Hybrid search that combines dense and sparse vectors is a core requirement.

When to avoid Qdrant:

The team has no Docker experience and needs a simpler local setup.
The target environment cannot support gRPC network configuration.
Pydantic version constraints conflict with existing project dependencies.

Each client brings distinct trade-offs to the connection layer. These differences also extend further into how each client handles installation and platform compatibility.

Installation and Environment Management

In ideal environments, installing a Python vector database library is straightforward. In practice, the target platform, Python version, and existing package dependencies each introduce variables that can turn a simple pip install into a multi-hour debugging session. A quick compatibility check before committing to a client is worth the effort, since most of these issues only surface after the setup is complete.

The compatibility matrix

The following matrix shows client behavior across Python 3.8–3.13 on macOS ARM, Windows, and Linux.

Client	macOS ARM (M1/M2)	Windows	Linux (Debian)	Python 3.13
Pinecone	✓ Full support	✓ Full support	✓ Full support	✓ Supported
Weaviate	✓ Full support	✓ Full support	Requires Docker for gRPC	✓ Supported
ChromaDB	hnswlib compilation errors	Instability above 99 records (#3058)	Requires Debian Bookworm+	✗ Broken (#3651)
Qdrant	✓ Full support	✓ Full support	✓ Full support	✓ Supported

ChromaDB carries the heaviest compatibility burden of any client in this comparison. On macOS with ARM processors, hnswlib produces compilation errors during installation, forcing developers to manually pin Python to 3.11 or 3.12.

On Windows, ChromaDB destabilizes once a collection exceeds 99 records, making the embedded client unsuitable for anything beyond early prototyping. On Linux, Debian-based distributions require Bookworm or later to install and run ChromaDB cleanly.

Virtual environment best practices

Setting up a virtual environment before installing any vector database client saves a lot of debugging time, especially with ChromaDB, where developers know package conflicts occur frequently.

python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install chromadb

Pinning the client version in a requirements.txt file matters equally, since several of these clients have a history of introducing breaking changes between minor releases.

chromadb==0.4.x
    qdrant-client==1.7.x
    pinecone==3.x
    weaviate-client==4.x

ChromaDB's two-package architecture confuses many developers. When someone installs chromadb-client instead of chromadb, the application throws this error on its first attempt to call the default embedding function.

ValueError: You must provide an embedding function

chromadb-client is a lightweight HTTP-only package that does not include DefaultEmbeddingFunction. Applications that rely on the default embedding behavior need the full chromadb package instead.

Extras and optional dependencies

Beyond the base installation, each client supports additional dependencies that can improve performance for production workloads.

 # Pinecone with gRPC support
    pip install pinecone[grpc]

    # Qdrant with FastEmbed for local embedding generation
    pip install qdrant-client[fastembed]

    # ChromaDB with sentence-transformers for local embedding support
    pip install chromadb sentence-transformers

gRPC has the highest impact on optional installation for query performance. Weaviate sees 40-70% faster queries over gRPC than over REST, while Qdrant gains roughly 15% in query speed. The trade-off is that gRPC requires additional network configuration, which may not be feasible in restricted environments.

FastEmbed and sentence-transformers both enable local embedding generation without an external API dependency, keeping latency and embedding costs down for semantic search and similarity search workloads.

Qdrant's native AsyncQdrantClient and Pinecone's PineconeAsyncio deliver 3–5x throughput improvements under high-concurrency workloads.

Local Development Workflows

Developers make most vector database decisions in the local development environment. The critical question is: Which client requires the least code change when moving to production?

The migration path
Here is how each client handles the move from local deployment to production.

 # Qdrant - zero code changes required
    client = QdrantClient(":memory:")              # Development
    client = QdrantClient(                         # Production
        url="https://your-cluster-url",
        api_key="your-api-key"
    )

    # ChromaDB - client class change required
    client = chromadb.Client()                     # Development
    client = chromadb.HttpClient(                  # Production
        host="your-host",
        port=8000
    )

    # Pinecone - same code in both environments
    pc = Pinecone(api_key="your-api-key")          # Development and production
    index = pc.Index("your-index-name")

Qdrant :memory: mode carries identical client code from local development all the way to production. The vector store configuration, cosine similarity settings, and hnsw index parameters remain the same across environments.

ChromaDB requires a change to the client class when moving to production. The more widely the codebase uses the client, the more of the application this change touches.

Pinecone uses the same code in both development and production since everything runs in the cloud regardless of the stage.

These migration differences stem from three distinct local development approaches: embedded mode, Docker, and cloud-only.

Embedded mode

ChromaDB’s default embedded client stores data only in memory. When the application stops running, the data is lost. For developments involving persistent collections, PersistentClient writes data to disk instead.

 # In-memory only: data lost when process ends
    client = chromadb.Client()
    collection = client.create_collection("my_collection")
    collection.add(documents=["doc1", "doc2"], ids=["1", "2"])

    # Persistent local storage
    client = chromadb.PersistentClient(path="/local/path")

Qdrant’s :memory: mode uses the same client interface as a production deployment. Whatever code works locally also works in production without any changes.

client = QdrantClient(":memory:")
    client.create_collection(
        collection_name="my_collection",
        vectors_config=VectorParams(size=384, distance=Distance.COSINE)
    )

Both clients work well for early prototyping and notebook environments, and the differences surface only at the production boundary.

Docker for local development

Docker runs the vector database in an isolated local container using the same configuration as in a production deployment. Qdrant and Weaviate are both open-source vector databases that support this approach.

# Qdrant
    docker run -p 6333:6333 qdrant/qdrant

    # Weaviate
    docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:latest

Once the container is running, the client connects to localhost the same way it would to a self-hosted vector database in production.

 # Qdrant
    client = QdrantClient(url="http://localhost:6333")

    # Weaviate
    client = weaviate.connect_to_local()

The main advantage is that vector index configuration behaves the same way locally and in production, and issues that surface locally are genuine issues rather than environment-specific artifacts.

The trade-off is the overhead of Docker installation and port configuration, particularly Weaviate's requirement for both ports 8080 and 50051.

Cloud-only development

Pinecone operates entirely in the cloud. Every operation, from index creation to vector upsert to real-time search, requires an active API key and a live network connection.

The setup overhead is minimal since there is no local infrastructure to configure or maintain. The same code runs across all environments, with API key management and network connectivity as the only constant requirements.

Moving beyond local development, how each client integrates with the broader Python ecosystem, particularly LangChain and LlamaIndex, adds another layer to the comparison.

Integration Ecosystem: LangChain and LlamaIndex

LangChain and LlamaIndex sit at the center of most Python-based retrieval-augmented generation workflows. All four clients integrate with both frameworks, though the quality of these integrations varies.

LangChain maturity

 from langchain_pinecone import PineconeVectorStore      # Dedicated package
    from langchain_chroma import Chroma                      # Mature, widely used
    from langchain_qdrant import QdrantVectorStore           # Actively maintained
    from langchain_weaviate import WeaviateVectorStore       # Lagged during v3 to v4

Pinecone's LangChain integration benefits from thorough documentation and a stable release history across major version changes. Teams widely rely on ChromaDB's integration to prototype retrieval-augmented generation pipelines.

Qdrant’s dedicated langchain-qdrant package keeps pace closely with client releases. Weaviate is the exception. LangChain took several months to catch up with the v3-to-v4 migration, and the subsequent breaking changes forced many teams to pin their dependency versions until the LangChain integration caught up.

LlamaIndex support

All four clients have LlamaIndex connectors through the llama-index-vector-stores namespace.

 from llama_index.vector_stores.qdrant import QdrantVectorStore

Here, the support pattern is consistent across all four clients. When a client releases a new version, LlamaIndex integration typically follows within four to eight weeks. Any breaking changes in the client affect pipelines built on top of it in the meantime.

The integration tax

Every time a vector database client releases a major update, a predictable sequence follows.

The client releases a new version with breaking changes.
LangChain and LlamaIndex updated their integrations weeks later.
Production pipelines built on top of those integrations break in the meantime.

This pattern has played out with Weaviate’s v3-to-v4 migration, Pinecone’s three major releases in 18 months, and ChromaDB’s ongoing compatibility issues.

API stability matters more than feature richness when selecting a Python vector database library for production retrieval-augmented pipelines. A client with fewer features but a stable API causes considerably less disruption than one with a rich feature set and frequent breaking changes.

Performance Considerations: Beyond Raw Speed

Most client comparisons overlook three factors that significantly affect vector database performance: protocol choice, the quality of async support, and connection pooling.

Protocol choice: REST vs. gRPC

gRPC and REST are the two transport protocols available across these clients. As mentioned earlier, Weaviate sees 40 to 80% faster queries over gRPC, and Qdrant gains roughly 15% in query speed with gRPC enabled. In restricted network environments where port 50051 is not accessible, REST is the more practical option.

Async support quality

Most teams build production LLM applications on FastAPI or similar async frameworks, which makes async client support a meaningful performance consideration. Using a synchronous client within an async application results in blocking calls, which sharply reduces throughput.

Qdrant’s native AsyncQdrantClient, available since v1.61, provides a well-established async implementation. Pinecone introduced PineconeAsyncio in v6, bringing proper async support to cloud-only vector search workloads. Weaviate added async support in v4.7, making it the most recent of the four to reach production-ready async capabilities. ChromaDB’s async support remains limited across all four.

The throughput difference is substantial. For I/O-bound workloads where network latency is the bottleneck, async clients typically deliver 3–5x higher throughput than their synchronous equivalents.

Connection pooling and resource management

This is one of the configuration areas where default settings tend to fall short in production. Qdrant and Pinecone both expose parameters that give more control over connection management under sustained production traffic.

 # Qdrant connection pool configuration
    client = QdrantClient(
        url="https://your-cluster-url",
        api_key="your-api-key",
        timeout=30,
        pool_size=10
    )

    # Pinecone connection pool configuration
    index = pc.Index(
        "your-index-name",
        pool_threads=30,
        connection_pool_maxsize=30
    )

For Pinecone, query_namespaces requires tuning pool_threads and connection_pool_maxsize for production workloads. For Qdrant, increasing pool_size above the default reduces connection contention for applications that handle large volumes of document embeddings in parallel.

Teams that tune these settings before deployment avoid considerable debugging time when the application runs under load.

Error Handling and Debugging

Vector database libraries handle a lot of complexity internally. When something fails, how clearly the client communicates that failure determines how quickly teams can fix it.

Error message quality

The quality of error messages varies considerably across the four clients.

Pinecone produces clear, actionable error messages that typically suggest a solution alongside the failure description, reducing the time teams spend searching for the root cause.

Qdrant error messages are helpful and point directly to the source of the problem. The UnexpectedResponse exception includes a specific reason field that identifies exactly which parameter failed validation.

    qdrant_client.http.exceptions.UnexpectedResponse: Status 400, reason: "Wrong input: Vector dimension error: expected dim: 384, got 768"

ChromaDB error messages are frequently vague and require a GitHub search to diagnose. When the two-package mix-up occurs, ChromaDB raises a ValueError about missing embedding functions instead of reporting the actual root cause. The SQLite version requirement produces a similarly unhelpful error:

 RuntimeError: Your system has an unsupported version of sqlite3. Chroma requires sqlite3 >= 3.35.0.

This error is a common roadblock for Python developers deploying on older Amazon Linux 2 or Streamlit environments.

Weaviate v3 silently failed, returning null objects or dictionaries with an errors key that developers had to check manually. The v4 rewrite addressed this with typed exceptions, such as WeaviateQueryError and WeaviateGRPCUnavailableError.

Logging and observability

Observability capabilities differ across the four clients.

Qdrant supports structured logging, distributed tracing, and metrics without additional configuration, which makes it a strong fit for production machine learning applications that require visibility into vector search engine performance.
Pinecone provides basic logging through its managed infrastructure.
ChromaDB has limited logging with no structured output, which makes diagnosing issues in production AI applications considerably harder.

Common pitfalls and solutions

Three error patterns recur across all four clients in production environments.

Client version mismatches cause frequent, unexpected failures, particularly amid Pinecone's three releases over 18 months and Weaviate's v3-to-v4 migration. Teams can control this by pinning client versions in a requirements.txt file.
Embedding dimension mismatch occurs when the query embedding dimensions do not match the collection's expectations. Verifying that the embedding model's output size matches the collection configuration before deployment prevents this.
Rate limiting affects cloud-only deployments on Pinecone and Weaviate Cloud. Implementing exponential backoff on API calls is the standard solution for production workloads that approach rate limits under sustained traffic.

How often version mismatches surface, how broadly platform incompatibilities spread, and how clearly error messages communicate failures together determine a client's real maintenance cost in production.

The version churn across Pinecone, Weaviate, and ChromaDB has left many production teams looking for a client that prioritizes operational stability over feature velocity. Actian VectorAI DB addresses this directly.

Actian VectorAI DB

Actian designed Actian VectorAI DB for large and small-scale deployment of vector search in edge and on-premises environments, addressing operational stability through the following characteristics:

Architecture and design

Same architecture and APIs across all environments.
Docker-based deployment from local laptops to production infrastructure.
HNSW-based indexing for approximate nearest neighbor search.
Real-time indexing architecture with immediate update availability.
Python and JavaScript SDKs with REST and SQL APIs.
Native LangChain and LlamaIndex integration support.

Deployment options

Docker containers for local development and production.
Data center, private cloud, and public cloud infrastructure support.
Edge, remote, and air-gapped environment capability with offline operation.

Compliance design

On-premises deployment architecture for data sovereignty.
Supports GDPR, HIPAA, and data residency requirements.
Aligns with SOC 2 and ISO 27001 compliance frameworks.

Performance targets

Sub-15ms query latency goal for local deployment.
Uses HNSW to optimize for high recall accuracy.

These characteristics represent important considerations beyond features and benchmarks alone.

Decision Framework: Choosing Your Python Client

Selecting the right Python vector database library comes down to six criteria.

Deployment model: Cloud-only deployments point towards Pinecone. On-premises or self-hosted vector database requirements point towards Qdrant, Weaviate, or Actian. Teams already running PostgreSQL should test pgvector against their workload before adding a new dependency.
Team experience: Junior teams or teams new to vector databases benefit from ChromaDB or Actian, where API stability and clear error messages reduce the learning curve. Senior teams comfortable with Docker and gRPC configuration get more out of Qdrant and Weaviate.
Scale:

Under 10 million vectors: ChromaDB or FAISS
10 million to 100 million vectors: Qdrant, Pinecone, or Actian
Over 100 million vectors: Milvus

API stability: Production environments with zero tolerance for breaking changes point towards Actian or Pinecone v7+. Teams that can absorb migrations can work with Weaviate or Qdrant. POC and experimental workloads can use ChromaDB or FAISS.
Integration: LangChain-dependent pipelines should verify version support before committing to a client. Weaviate's v3-to-v4 lag is the clearest example of what happens when this check is skipped.
Cost: Pinecone's managed pricing runs from approximately $50 to $500 or more per month, depending on scale. Cost-conscious teams running large datasets on self-hosted infrastructure should evaluate Qdrant or pgvector. ChromaDB and FAISS are both free for local, open-source vector database workloads.

Decision matrix

Criteria	Pinecone	Qdrant	Weaviate	ChromaDB	Actian VectorAI DB
API stability	Medium	Good	Improving	Low	High
Local development	✗ No local mode	✓ :memory: mode	Docker required	✓ Embedded	✓ :memory: + SQLite
Platform compatibility	✓ Cloud only	✓ All platforms	✓ All platforms	✗ Issues on ARM, Win	✓ All platforms
Async support	✓ v6+	✓ Native	✓ v4.7+	✗ Limited	✓ Native
Cost	$50-500+/mo	Free / Self-hosted	Free / Managed	Free	Enterprise pricing

Final Thoughts

The ChromaDB production failure from the opening example stems from client packaging issues that developers only encounter after deployment. This comparison helps avoid similar failures: platform incompatibilities, breaking changes from version migrations, and client class redesigns that propagate across codebases.

ChromaDB gets projects started quickly but tends to show its limitations once the application moves to production. Pinecone is polished and well-managed, but version churn and permanent cloud dependencies are real costs. Qdrant is the strongest open-source option for teams that want local-to-production parity without code changes. Weaviate's v4 client significantly improves on v3 and is well-suited for teams that need hybrid search at scale.

For teams where API stability and platform compatibility are critical, enterprise-grade clients like Actian VectorAI DB provide production-ready stability with verified cross-platform support.

Explore Actian VectorAI DB for guaranteed production stability.

Can AI in Manufacturing Work Without the Cloud? A Guide

Tiioluwani — Mon, 18 May 2026 12:40:42 +0000

Keeping external traffic out of operational networks is a best practice that most manufacturing facilities build into their architecture from the ground up.

Manufacturing networks use the Purdue Model, a five-level system that has shaped industrial network design for decades. At the lowest level are the physical machines: sensors, motors, and actuators at Level 0; real-time controllers and SCADA systems at Level 1; and supervisory servers and HMI systems at Level 2. Level 3 manages operations. Levels 4 and 5 connect to the enterprise network and to the internet.

IEC 62443 enforces strict boundaries between these levels. Traffic from Level 2 does not reach the internet. For defense contractors, ITAR compounds the problem. Technical data must stay on U.S. soil and remain accessible only to U.S. persons. Cloud-hosted vector databases like Pinecone, Weaviate Cloud, and Qdrant Cloud fail both requirements. Level 2 has no way to send that request, and other industries learned this lesson the hard way.

Latency compounds the problem. Cloud round-trips average 50 to 500 milliseconds. PLC-level control loops require responses in under 10 milliseconds. Teams that need AI during outages use edge deployment patterns designed for disconnected environments.

Cost adds another layer. AWS standard egress starts at $0.09 per GB. At any serious production scale, sensor and vision data add up quickly, and the bill arrives faster than most teams expect.

Architecture, latency, and cost all point in the same direction. AI on the factory floor needs to run where the data lives.

This tutorial shows you how to build a local RAG pipeline that runs entirely on factory-floor hardware, where a technician can ask a question about any piece of equipment and get a cited answer from decades of maintenance records, with no internet connection required.

What You Are Building

You’ll build a three-layer RAG pipeline that runs fully inside your factory network. The ingestion layer processes PDF maintenance documents and stores them in Actian VectorAI DB. The query layer takes a technician's question and returns a cited answer fast enough for interactive use on factory-floor hardware.

Ingestion: Reads the PDF maintenance documents, splits them into 256-token chunks with a 25-token overlap, generates embeddings using sentence-transformers on a CPU, and stores everything in VectorAI DB with metadata for equipment line, document date, and source file.
Query: Takes a technician's question, embeds it with the same model, runs a hybrid search in VectorAI DB filtered by equipment line and date range, and sends the top results to a local LLM running with Ollama, which generates a cited answer in plain English.
Audit: Logs every ingestion and query event as a structured JSON entry to ./data/audit.log, timestamped in UTC, and stored inside your security boundary to satisfy IEC 62443 traceability requirements.

VectorAI DB sits at the center of all three layers. It stores the embeddings that the ingestion layer produces, and serves the search results that the query layer runs. Running it on-premises instead of in the cloud keeps the whole pipeline inside your security boundary.

The pipeline runs on standard factory edge server hardware, with Ubuntu 22.04 LTS, 16 GB of RAM, and a 4-core CPU.

Building a Local RAG Pipeline with VectorAI DB

Set up VectorAI DB, build the ingestion pipeline, run your first query, add hybrid filters, and connect a local LLM.

Prerequisites

Docker and Docker Compose
Python 3.10 or higher
uv package manager. Install with curl -LsSf https://astral.sh/uv/install.sh | sh
Ollama. Install from Ollama.com and pull the model with ollama pull llama3.2:3b

Your machine needs at least 8 GB RAM (16 GB or more recommended) and 10 GB of disk space (100 GB or more recommended) to run VectorAI DB. If you're on Windows, the uv install command needs 'sh', which PowerShell doesn't have. Run all commands in WSL2 (Windows Subsystem for Linux). To set up WSL2, run 'wsl --install' in PowerShell, then use the Ubuntu terminal for this tutorial.

Project structure

Set up your project folder like this:

 factory-rag/
├── docker-compose.yml
├── data/
│   └── audit.log
├── config/
└── src/
    ├── healthcheck.py
    ├── ingest.py
    ├── query.py
    ├── llm.py
    ├── audit.py
    └── test_e2e.py

Create the directories:

mkdir -p factory-rag/{data,config,src}
cd factory-rag

Step 1: Deploy VectorAI DB

Create docker-compose.yml in your project root:

services:
  vectorai-db:
    image: williamimoh/actian-vectorai-db:latest
    platform: linux/amd64
    container_name: vectorai-db
    ports:
      - "50051:50051"
    volumes:
      - ./data:/data
      - ./config:/config
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "nc -z localhost 50051 || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 15s

Start the container with:

docker compose up -d

Install the SDK with:

uv add actian_vectorai-0.1.0b2-py3-none-any.whl

Install these required libraries:

uv add sentence-transformers pypdf

Check that the server is running. Make a file called src/healthcheck.py:

from actian_vectorai import VectorAIClient

with VectorAIClient("localhost:50051") as client:
    info = client.health_check()
    print(f"✓ VectorAI DB is running")
    print(f"  Title:   {info['title']}")
    print(f"  Version: {info['version']}")

Run the script:

uv run python src/healthcheck.py

Terminal output:

Step 2: Build the ingestion pipeline

Put your PDF maintenance documents in the data/ folder before running this step. Add any equipment maintenance records, inspection reports, or failure logs there.
The pipeline uses sentence-transformers/all-MiniLM-L6-v2, which needs less than 200 MB of RAM on CPU. We split text into 256-token chunks with a 25-token overlap to keep enough context for good retrieval.

Create src/ingest.py:

from __future__ import annotations

import argparse
import uuid
from pathlib import Path

from actian_vectorai import Distance, PointStruct, VectorAIClient, VectorParams
from pypdf import PdfReader
from sentence_transformers import SentenceTransformer

from audit import log_ingestion

COLLECTION = "maintenance_records"
HOST = "localhost:50051"
MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
VECTOR_DIM = 384
CHUNK_TOKENS = 256
OVERLAP_TOKENS = 25

def chunk_text(text, tokenizer, chunk_size=CHUNK_TOKENS, overlap=OVERLAP_TOKENS):
    token_ids = tokenizer.encode(text, add_special_tokens=False)
    chunks = []
    start = 0
    while start < len(token_ids):
        end = min(start + chunk_size, len(token_ids))
        window = token_ids[start:end]
        decoded = tokenizer.decode(window, skip_special_tokens=True).strip()
        if decoded:
            chunks.append(decoded)
        if end >= len(token_ids):
            break
        start += chunk_size - overlap
    return chunks

def ingest_pdf(pdf_path, equipment_line, doc_date, model, client):
    reader = PdfReader(str(pdf_path))
    full_text = "\n".join(page.extract_text() or "" for page in reader.pages)
    if not full_text.strip():
        print(f"  [warn] No extractable text in {pdf_path.name}, skipping.")
        return 0
    tokenizer = model.tokenizer
    chunks = chunk_text(full_text, tokenizer)
    points = []
    for idx, chunk in enumerate(chunks):
        embedding = model.encode(chunk, show_progress_bar=False).tolist()
        points.append(
            PointStruct(
                id=str(uuid.uuid5(uuid.NAMESPACE_DNS, f"{pdf_path.name}:{idx}")),
                vector=embedding,
                payload={
                    "equipment_line": equipment_line,
                    "doc_date": doc_date,
                    "source_file": pdf_path.name,
                    "text": chunk,
                    "chunk_index": idx,
                },
            )
        )
    if points:
        client.points.upsert(COLLECTION, points)
    return len(points)

def main(data_dir, equipment_line, doc_date):
    data_path = Path(data_dir)
    pdfs = sorted(data_path.glob("*.pdf"))
    if not pdfs:
        print(f"No PDF files found in '{data_dir}'. Add PDFs to ./data/ and retry.")
        return
    print(f"Loading embedding model '{MODEL_NAME}'...")
    model = SentenceTransformer(MODEL_NAME)
    with VectorAIClient(HOST) as client:
        if not client.collections.exists(COLLECTION):
            client.collections.create(
                COLLECTION,
                vectors_config=VectorParams(size=VECTOR_DIM, distance=Distance.Cosine),
            )
            print(f"Created collection '{COLLECTION}' ({VECTOR_DIM}-dim, Cosine)")
        else:
            print(f"Collection '{COLLECTION}' already exists, appending chunks.")
        total = 0
        for pdf_path in pdfs:
            print(f"Ingesting {pdf_path.name} ...")
            count = ingest_pdf(pdf_path, equipment_line, doc_date, model, client)
            print(f"  → {count} chunks stored")
            log_ingestion(pdf_path.name, equipment_line, count)
            total += count
        print(f"\nDone. {total} total chunks stored in '{COLLECTION}'.")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Ingest PDFs into VectorAI DB")
    parser.add_argument("--data-dir", default="./data")
    parser.add_argument("--equipment-line", required=True)
    parser.add_argument("--doc-date", required=True)
    args = parser.parse_args()
    main(args.data_dir, args.equipment_line, args.doc_date)

Run the ingestion step:

uv run python src/ingest.py --equipment-line turbine-A --doc-date 2024-03-15

Expected output:

The metadata schema saves the equipment line, document date, and source file with each chunk. This lets you filter searches by equipment line or date range without searching the whole collection.

Step 3: Run your first query

Your ingestion pipeline has stored the maintenance records in VectorAI DB. The pipeline can answer questions. When a technician asks something in plain English, the pipeline embeds the question, searches the maintenance_records collection, and returns the top five most relevant chunks with similarity scores.

Create src/query.py:

from __future__ import annotations

import argparse
import time

from actian_vectorai import Field, FilterBuilder, VectorAIClient
from sentence_transformers import SentenceTransformer

from audit import log_query

COLLECTION = "maintenance_records"
HOST = "localhost:50051"
MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
TOP_K = 5

def build_filter(equipment_line=None, doc_date=None, doc_date_to=None):
    fb = FilterBuilder()
    if equipment_line:
        fb.must(Field("equipment_line").eq(equipment_line))
    if doc_date and doc_date_to:
        fb.must(Field("doc_date").range(gte=doc_date, lte=doc_date_to))
    elif doc_date:
        fb.must(Field("doc_date").eq(doc_date))
    return fb.build() if (equipment_line or doc_date) else None

def search(question, equipment_line=None, doc_date=None, doc_date_to=None):
    model = SentenceTransformer(MODEL_NAME)
    embedding = model.encode(question, show_progress_bar=False).tolist()
    query_filter = build_filter(equipment_line, doc_date, doc_date_to)
    with VectorAIClient(HOST) as client:
        hits = client.points.search(
            COLLECTION,
            vector=embedding,
            limit=TOP_K,
            filter=query_filter,
        )
    return [
        {
            "score": round(r.score, 4),
            "source_file": r.payload.get("source_file", ""),
            "equipment_line": r.payload.get("equipment_line", ""),
            "doc_date": r.payload.get("doc_date", ""),
            "chunk_index": r.payload.get("chunk_index", -1),
            "text": r.payload.get("text", ""),
        }
        for r in hits
    ]

def main():
    parser = argparse.ArgumentParser(description="Search maintenance records")
    parser.add_argument("question", help="Natural language question")
    parser.add_argument("--equipment-line", default=None)
    parser.add_argument("--doc-date", default=None)
    parser.add_argument("--doc-date-to", default=None)
    args = parser.parse_args()

    start = time.monotonic()
    results = search(args.question, equipment_line=args.equipment_line,
        doc_date=args.doc_date, doc_date_to=args.doc_date_to)
    latency_ms = (time.monotonic() - start) * 1000
    log_query(args.question, args.equipment_line or "", results, latency_ms)

    if not results:
        print("No results found.")
        return

    print(f"Top {len(results)} results for: \"{args.question}\"\n")
    for i, r in enumerate(results, 1):
        print(f"[{i}] score={r['score']:.4f}  {r['source_file']} "
              f"(chunk {r['chunk_index']})  {r['doc_date']}  {r['equipment_line']}")
        print(f"     {r['text'][:200].strip()}...")
        print()

if __name__ == "__main__":
    main()

Try your first query:

uv run python src/query.py "What caused the bearing failure?"

The search uses the same model as ingestion to embed the query, keeping both the query and stored vectors in the same semantic space. For maintenance records with this model, similarity scores between 0.4 and 0.6 indicate relevant matches.

Step 4: Add hybrid filters

Filtering by equipment line and date helps keep search results relevant to the technician's current work. Run the same query from Step 3, but add these filters:

uv run python src/query.py "What caused the bearing failure?" --equipment-line turbine-A

Add a date filter to narrow the results even more:

uv run python src/query.py "What caused the bearing failure?" --equipment-line turbine-A --doc-date 2024-03-15

Expected output:

The build_filter function constructs a FilterBuilder query that combines vector similarity with exact metadata matching. A technician working on turbine-A only sees results from that equipment line, not from the entire maintenance history.

Step 5: Connect the local LLM

The search results feed into a local LLM running via Ollama, which generates a cited answer in plain English. The entire round trip runs on factory-floor hardware.

Create src/llm.py:

from __future__ import annotations

import json
import os
import sys
import urllib.request
from typing import Any

OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434")
OLLAMA_MODEL = "llama3.2:3b"
MAX_NEW_TOKENS = 256
TEMPERATURE = 0.1
TIMEOUT_SECONDS = 300

def build_prompt(question: str, results: list[dict[str, Any]]) -> str:
    if not results:
        return f"Question: {question}\n\nAnswer: I have no relevant context to answer this question."
    context_blocks = []
    for i, r in enumerate(results, 1):
        source = r.get("source_file", "unknown")
        date = r.get("doc_date", "unknown")
        equip = r.get("equipment_line", "unknown")
        text = r.get("text", "").strip()
        context_blocks.append(
            f"[{i}] Source: {source} | Equipment: {equip} | Date: {date}\n{text}"
        )
    context = "\n\n".join(context_blocks)
    return (
        "You are a maintenance records assistant. "
        "Answer the question using ONLY the provided context. "
        "Cite sources inline using [1], [2], etc. "
        "If the context does not contain enough information, say so.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n\n"
        "Answer:"
    )

def generate(question: str, results: list[dict[str, Any]]) -> str:
    prompt = build_prompt(question, results)
    payload = json.dumps({
        "model": OLLAMA_MODEL,
        "prompt": prompt,
        "stream": False,
        "options": {
            "num_predict": MAX_NEW_TOKENS,
            "temperature": TEMPERATURE,
        },
    }).encode()
    req = urllib.request.Request(
        f"{OLLAMA_HOST}/api/generate",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT_SECONDS) as resp:
        body = json.loads(resp.read().decode())
    return body["response"].strip()

def answer(question: str, results: list[dict[str, Any]]) -> str:
    reply = generate(question, results)
    print(reply)
    print()
    print("Sources")
    for i, r in enumerate(results, 1):
        print(
            f"  [{i}] {r.get('source_file', '?')} "
            f"(chunk {r.get('chunk_index', '?')}, score {r.get('score', 0):.4f}) "
            f"{r.get('doc_date', '?')} / {r.get('equipment_line', '?')}"
        )
    return reply

if __name__ == "__main__":
    question = sys.argv[1] if len(sys.argv) > 1 else "What maintenance was performed?"
    dummy_results = [
        {
            "source_file": "example.pdf",
            "doc_date": "2024-03-15",
            "equipment_line": "turbine-A",
            "chunk_index": 0,
            "score": 0.95,
            "text": (
                "Performed scheduled bearing inspection on turbine-A. "
                "Replaced worn bearing race on shaft 2. "
                "Torque settings verified per spec TRB-004."
            ),
        }
    ]
    answer(question, dummy_results)

Wire everything together by creating src/test_e2e.py:

from query import search
from llm import answer

question = "What maintenance was performed on the gearbox?"
results = search(question, equipment_line="turbine-A")
answer(question, results)

Run the full pipeline:

uv run python src/test_e2e.py

llama3.2:3b fits in the memory of a standard factory edge server. The LLM receives only the retrieved chunks as context, not the full document collection, which keeps responses fast and grounded in cited sources.

Expected output:

The pipeline is fully up and running. A technician can ask a question, get a cited answer from local maintenance records, and never need to use the internet.

Step 6: Add audit logging

IEC 62443 requires full traceability for every operation within the OT network. Without a local audit trail, your pipeline has no record of what was queried, when, or what it returned.

Create src/audit.py:

from __future__ import annotations

import json
import logging
from datetime import datetime, timezone
from pathlib import Path

LOG_PATH = Path("./data/audit.log")
LOG_PATH.parent.mkdir(parents=True, exist_ok=True)

handler = logging.FileHandler(str(LOG_PATH))
handler.setLevel(logging.INFO)

logger = logging.getLogger("actian_vectorai.audit")
logger.setLevel(logging.INFO)
logger.addHandler(handler)

def log_query(question: str, equipment_line: str, results: list, latency_ms: float) -> None:
    entry = {
        "event": "query",
        "timestamp": datetime.now(tz=timezone.utc).isoformat(),
        "question": question,
        "equipment_line": equipment_line,
        "results_returned": len(results),
        "latency_ms": round(latency_ms, 2),
    }
    logger.info(json.dumps(entry))

def log_ingestion(source_file: str, equipment_line: str, chunks_stored: int) -> None:
    entry = {
        "event": "ingestion",
        "timestamp": datetime.now(tz=timezone.utc).isoformat(),
        "source_file": source_file,
        "equipment_line": equipment_line,
        "chunks_stored": chunks_stored,
    }
    logger.info(json.dumps(entry))

Run the audit script with this command:

cat data/audit.log

Expected output:

The pipeline now keeps a structured record of every ingestion and query event in ./data/audit.log, timestamped in UTC and stored inside your security boundary.

Wrapping Up

You just built a local RAG pipeline that runs entirely on factory-floor hardware, serves queries during network outages, and returns cited answers from decades of maintenance records.

AI in manufacturing can operate without a cloud connection. VectorAI DB enables this by running entirely within the IEC 62443 security boundary, without relying on the cloud. Cut the internet connection entirely, and the pipeline keeps working.

Your pipeline ingests PDF maintenance documents, stores embeddings in the VectorAI DB at Level 2 of your OT network, and answers natural-language questions using a local LLM with no cloud dependency at any step. From here, you can extend the pipeline by adding more document types, tuning the embedding model for your specific equipment vocabulary, adding role-based query filtering by technician, or scaling ingestion across multiple equipment lines.

Find the full VectorAI DB documentation and the GitHub repository to explore further.

Join the community and learn more about Actian.

Navigating Search Solutions: A Comprehensive Comparison Guide to Meilisearch, Algolia, and ElasticSearch

Tiioluwani — Sat, 21 Dec 2024 12:37:49 +0000

Introduction

As more information is available at our fingertips, one faces the challenge of filtering through large datasets to obtain relevant information within the shortest time possible. This necessity has resulted in the widespread use of search solutions. They are essential for up-to-date applications because they allow you to find the required information in a large dataset.

Well-developed and well-functioning search solution features help improve user satisfaction, increase activity, and ultimately increase business and organizational efficiency by finding the ‘hidden’ correlation in large datasets.

This article will compare Meilisearch, Algolia, and ElasticSearch for developers, tech enthusiasts, and companies seeking to explore various search solutions. We'll discuss each of them indicating its distinctive features, benefits, and applications, which would be helpful for decision-making when choosing the most suitable search solution for certain project needs and limitations of the software environment

Key Considerations When Choosing a Search Solution

When choosing the type of search solution for an application there are several factors that one must put in mind. Some of these factors include:

Scalability and Performance: An ideal search solution should meet the requirement of serving a large volume of data and large numbers of traffic. It should have quick query response time and should be scalable, that is, it should be able to respond to queries out of large traffic and should be able to handle search queries with various structural queries.
Ease of Integration: It should be easy to fit into your application environment and offer a basic and user-friendly API that makes it easy to add the search solution to any application. A suitable solution should also have good documentation and support materials to assist these developers in getting started.
Indexing and Querying: The search solution has to contribute to the development of effective indexing and query solutions and needs to be capable of handling various types of data that can be textual, numerical, or geographical data types. The solution should also contain extensions to basic search and sort functions with features such as fuzzy search, geo-search and filters, etc.
Relevance and Customization: There should be a relevance algorithm delivered with a search solution that can be adapted to your precise requirements. The solution should also contain diverse options that define the look and functionality of the search results interface to offer you a well-incorporated and easy-to-use environment.
Cost and Licensing: The search solution must be affordable and it must incorporate a licensing model that is going to suit your needs. Hardware, software, and maintenance costs must also be considered, and the licensing model should be manageable as needs change.

Unveiling the Contenders

Here we take a look at the peculiar features and capabilities of these search solutions for your application. In this section, we will delve into the details of three prominent search solutions: Meilisearch, Algolia, and ElasticSearch.

Meilisearch

Image Source

Meilisearch is a lightweight, developer-first, API-first search engine that is designed to be easily integrated into an existing application. Its principal components are high speed, easy configuration, and good documentation.

Strengths
Below are some of the strengths of Milliesearch:

Fast Performance: Meilisearch is characterized by incredibly fast search speed, so it can be appropriate for applications with high response rates.
Simple Setup: Meilisearch does not require much configuration which implies that developers can easily implement it.
Excellent Documentation: Meilisearch has a comprehensive knowledge base which is relatively easy to navigate and which contains a wealth of information for developers.
Open-Source: Meilisearch is an open-source tool, which makes it free to use and can also be self-hosted, which gives users more control over the underlying architecture.

Limitations
Despite its many benefits, Meilisearch also has some limitations. Some limitations of Millesearch include:

Lacks Advanced Features: Despite the fact that Meilisearch has many features that it can implement, it may not be the best suited for data sets of large size. There could be other additional features useful in some cases of massive data that can be missing in Meilisearch.
Limited Scalability: Compared to ElasticSearch, Meilisearch might not be as highly scalable, and it might become an issue with an extremely large amount of users or data.
Limited Community Support: Meillisearch has a smaller community of developers when compared to ElasticSearch. This could be a disadvantage as there could be fewer resources available in the solution's community.
Limited Geo-Search Capabilities: While it is also possible to implement certain basic forms of geo-search capabilities (the ability of a search solution to perform a search based on the location of the user), it is not as specifically developed as Algolia’s geo-search capabilities.

Algolia

Image Source

Algolia is a blazing-fast search engine that has rich API support and geo-search features. Its fundamental capabilities include high performance, rich extensibility, and robust tools for developers.

Strengths
Below are some of the strengths of Algolia:

Superior Speed: It has a nearly instant search result which can be a plus when you are dealing with applications that are sensitive to delay.
Extensive Customizations: Algolia also has many features that allows for flexibility to configure the search based on specific requirements.
Powerful Developer Tools: Among the tools that Algolia provides to developers, there are analytics and monitoring tools that help developers improve the search process and solve the problems they may face.
Geo-Search Capabilities: Geo-search functionality is available in Algolia, and this can be very useful for developers as they can search for results depending on location.

Limitations
As valuable as it may be, Algolia has its weaknesses that potential users should take into account. Below are some limitations of Aloglia:

Primarily a Hosted Solution: Algolia is predominantly a hosted platform which can be a potential limitation where users need a higher level of control over its architecture.
Higher Costs: Since Algolia is primarily hosted, there are additional costs that clients are required to meet. This may be a crucial point for some developers and businesses.
Steeper Learning Curve: Compared to Meilisearch Algolia has a steeper learning curve that makes the application of the solution challenging for the developers.
Limited Open-Source Options: Due to it being a hosted service, there are limited open-source options. This may reduce the possibilities for customization compared to other open-source platforms similar to ElasticSearch.

ElasticSearch

Image Source

ElasticSearch is a popular scalable open-source search solution used today due to its ability to provide strong, powerful queries. Some of its key capabilities involve high scalability, a very large and, indeed, highly active base of users, and numerous features.

Strengths
Below are key strengths ElasticSearch possesses:

Flexibility: ElasticSearch is indeed very flexible in that, developers can customize the solution to fit their needs.
Vast Community Support: With ElasticSearch having a large developer community, there is a lot of support and information available for the solution.
Extensive Features: ElasticSearch has many features such as query time complexities, real-time indexing, and distributed indexing.
Highly Scalable: ElasticSearch is highly scalable and thus can be used in an application requiring thousands of users or massive data.

Limitations
Like any other solution, ElasticSearch has its drawbacks as well. Below are some limitations of ElasticSearch

Steeper Learning Curve: ElasticSearch has a complicated learning curve compared to Meilisearch and Algolia, which may discourage developers from using the solution.
Requires More Operational Overhead: ElasticSearch requires a higher level of configuration and management compared to setting up and running Meilisearch or Algolia, which could present challenges in maintaining the solution.
Limited Geo-Search Capabilities: While ElasticSearch offers basic geo-search capabilities, it may not be as powerful as the geo-search capabilities offered by Algolia.
Higher Resources Requirement: Compared to Meilisearch and Algolia, ElasticSearch requires more resources, which can become a problem when it comes to running the solution on small or old hardware.

Feature Comparison Table

Feature	Meilisearch	Algolia	ElasticSearch
Deployment	Self-Hosted (Binary)	Hosted (SaaS)	Self-hosted (Java)
Pricing	Free, Open-source	Paid	Free, Open-source
Scalability	Good	Excellent	Excellent
Performance	Fast	Very Fast	Fast
Ease of Use	Easy	Easy	Moderate
Features	Basic	Extensive	Extensive
Customization	Limited	Extensive	Extensive
Geo-Search	Yes	Yes	Yes
Full-Text Search	Yes	Yes	Yes
Real-Time Indexing	Yes	Yes	Yes
Community Support	Growing	Strong	Strong
Best Use Case	Small to medium-sized applications	Large-scale applications, high traffic	Large-scale applications, complex data
Data Ingestion	Simple	Simple	Complex
Security	Good	Strong	Strong

For a deeper view, let’s examine the key factors to consider when choosing a search solution: We’ll also explain how those factors connect to Meilisearch, Algolia, and ElasticSearch.

Scalability

Meilisearch: Meilisearch is built with horizontal scalability in mind, meaning that you can add more nodes to your cluster if necessary. This makes it suitable for use in applications that are expected to register high growth in the future.
Algolia: Horizontal scalability is another feature of the Algolia solution, which is optimal for large traffic loads. Features such as automatic sharding and replication ensure that the database offers its best performance and is always reliable.
ElasticSearch: ElasticSearch is highly scalable and thus can work with very large sets of data and big loads of work. It has distributional indexing and search capabilities, thus making it appropriate for use in large applications.

Performance

Meilisearch: As a fast indexing and query search system, Meilisearch is famous for its speed in handling searches. There is evidence that it fares well in benchmarks, particularly when working with smaller to midsize datasets. Its performance is attributed to its ability to use efficient data structures and indexing algorithms.
Algolia: Algolia is also praised for the high speed of search and the platform is built for large-scale applications with extensive traffic. It is designed for high-speed performance and has wide usage for applications where searches should take almost no time.
ElasticSearch: ElasticSearch has been made fairly efficient for most applications as well as offering multiple potent functionalities. However, it is not as fast as Algolia in every case, although it can perform outstandingly in certain situations if it is finely tuned to suit specific environments. It depends on different factors, including the characteristics of the data and the queries, as well as the available computing power.

Ease of Integration

Meilisearch: Meilisearch is easy to install and integrates well into applications due to the API and well-documented documentation. It provides client libraries in several programming languages, so novice programmers can also work with it.
Algolia: Algolia has a welcoming API that comes with relevant documentation for developers to work with. It supports many languages and it can also integrate with other software frameworks.
ElasticSearch: ElasticSearch can be harder to implement compared to other systems, particularly for those new to distributed systems. However, it has a strong API and a vast number of users who can turn to it for solutions in case of any issue.

Indexing

Meilisearch: It allows for efficient indexing and applies to different types of data. It also allows real-time indexing, which means it can be used where data is frequently changing.
Algolia: It allows for indexing and can also update the index in real-time. It can handle complex volumes of data and data structure that varies in type.
ElasticSearch: It is primarily acknowledged for its indexing capabilities and supports many data types in its framework. One benefit is that it indexes the content of files in real-time and there are some features included as well like dynamic mapping.

Querying

Meilisearch: It has basic search query syntax and multiple forms of filtering. They also include additional characteristics like the option of the fuzzy search (a sort of search that allows for searching with makes use of spelling differences) and geo-search.
Algolia: It offers great opportunities regarding filtering functionality, and the query language is rather rich. It also provides more advanced features for the search of information including the options of fuzzy search, geo search, and tolerance of typos.
ElasticSearch: It provides certain levels of query string elasticity and supports the most possible filtering. It offers advanced features like fuzzy search, grouping, geo search, and much more.

Relevance
Meilisearch: Among the features provided within Meilisearch is the relevance algorithm which you can fine-tune to obtain the desired outcome for your specific circumstances.
Algolia: Algolia offers a relevance function and some options like synonyms or stop words for the further improvement of searching.
ElasticSearch: ElasticSearch has a strong relevance algorithm and many customization options. It also supports query time boosting and relevance scoring.

Customization and User Experience
Meilisearch: Meilisearch is quite limited in how it allows users to customize the appearance of the search results interface and the interactions that occur within that space. However, it can be integrated with the user interface of your existing application.
Algolia: Algolia offers more flexibility regarding the templates and display of search results and their usage. It can also be implemented alongside your existing application user interface.
ElasticSearch: The users can also interact with ElasticSearch in multiple ways; there are many possibilities as to how the search results are presented to the users. It can also be associated with the user interface of the existing application that you have.

Choosing the Right Solution

It is advisable to consider certain factors that may be vital when choosing a search solution for your project. Depending on the data size, the budget, the technical skills needed, and preferences in terms of features, users can select different search solutions. Here are guidelines based on different project situations.

1. Data Size

Small Projects: Meilisearch is a perfect option for applications with less data as it provides clear and fast solutions. They are easy to implement and are not very demanding in terms of costs and resources and as such are recommendable for use in small businesses or during the initial stages of business.
Medium to Large Projects: If you are dealing with a medium to a large set of data, Algolia is faster and has more features to work with if the project has some level of complexity. As it provides strong API support and extensive choices of customization, it is good for e-commerce sites or sites with large content.
Enterprise-Level Applications: When it comes to big data processing with high availability and extensibility and when complex queries are likely to be needed, it is better to use ElasticSearch. It can handle large quantities of data and offers a rich set of analytical tools, which will be useful for large organizations requiring processing large amounts of information.

2. Budget
Cost-Conscious Solution: In cases where one needs to work within a certain budget, then this is where Meilisearch comes in handy. The platform is highly performant but lacks the requisites of self-hosting that other services require. This enables developers to incorporate Search as an additional feature with minimal extra expense.
Mid-Range Budget: Algolia can be considered a paid solution that is pricier, though it has more features compared to the free solution. It is ideal for businesses that are ready to spend resources, which will lead to a more advanced search option that improves customer experience.
Unlimited Budget: For organizations that have large budgets, ElasticSearch can be implemented with extra resources for maintenance as well as fine-tuning. This is because flexibility may enable several extensions that can justify the investment.

3. Technical Expertise
Limited Technical Skills: If the team does not have deep technical experience, Meilisearch is easy to manage and implement, which is suitable for devs with different levels of experience. Its comprehensive documentation makes it even easier to implement.
Moderate Technical Skills: Algolia offers powerful functionality while remaining easy to use. Though it is not very intuitive, getting into it requires a little technical know-how but its great tools accompanied by comprehensive documentation make it quite usable by mid-level developers.
High Technical Skills: ElasticSearch is suitable for teams with advanced technical knowledge since it requires additional technical guidance for installation and maintenance. Its use is best suited for organizations that have the technical expertise to understand and operate search technologies because of its complexity and abundance of features.

4. Desired Features

Basic Search Functionality: For applications that only need basic search functionalities on projects, Meilisearch provides built-in features like indexing and real-time search, making it a practical choice.
Advanced Features: For specific features such as geo-search, faceted search, and extensive customizations, Algolia is a strong contender because of the abundance of features it offers developers.
Complex Querying and Analytics: ElasticSearch is the most powerful choice for applications that need to provide complex search skills, work in real-time, and process unstructured data. It has a comprehensive array of features that allow it to meet extended search needs and assist with data processing.

Conclusion

The choice between Meilisearch, Algolia, and Elasticsearch boils down to your specific project needs. Meilisearch is most suitable for smaller projects with limited resources, offering simplicity and performance. Algolia's speed and customization options are perfect for complex search requirements. Elasticsearch is recommended for large-scale applications needing advanced querying and data processing capabilities.

Consider factors like the amount of data involved, budget constraints, and desired features when making your decision. An effective search solution can greatly improve user experience and have a direct impact on broad business goals. Selecting the appropriate tool increases the chances of innovation, and distinguishes your application from others in the market.

How to Build Low-latency AI Video Avatars for Real-time Interactions

Tiioluwani — Wed, 27 Nov 2024 09:48:45 +0000

AI avatars are changing the face of content creation and how appealing videos can be created for various purposes, whether it’s training video content, internal team communication, or customer support. Advanced AI tools and video generators now make it possible to create studio-quality, customized videos featuring avatars that can speak multiple languages. An AI video generator enables you to create videos with human-like digital avatars that can connect with a global audience.

In this guide, you'll explore how to build a low-latency AI video avatar using Simli's API and WebRTC. By the end, you’ll be able to create and upload AI-generated videos that sync perfectly with real-time speech, ideal for training, interactive simulations, and presentations. You’ll also be able to generate stunning AI avatars and animated videos in minutes, reaching audiences at scale with human-like, multilingual content.

Prerequisites

To follow through this tutorial, you'll need:

A basic understanding of Javascript and Next.js
A Simli account. To get started, create a free Simli account.
OpenAI account (paid version).

The complete source code is available on GitHub.

With these essentials in place, let's now explore the core elements driving low-latency AI avatars and why low latency is required to provide a fluid, realistic user experience.

Understanding the Key Components for Low-Latency AI Avatars

To demonstrate the importance of low latency, imagine a virtual assistant that takes several seconds to respond to a user’s question. Frustrating, right? Another illustration is an AI instructor in a training video whose voice and lip movements lag behind the audio in AI videos. These irregularities affect the effectiveness of AI interactions, making low latency an absolute necessity.

Latency determines the amount of interaction and realistic experience with AI-powered avatars. In scenarios like training sessions, live presentations, or customer support, an avatar needs to respond as naturally as a human would. A latency of more than 200 milliseconds makes the responses appear delayed and unnatural for any human interaction. Low latency, therefore, gives way to flawless and real-time engagement, thereby leading to a lot more satisfaction and a much better user experience.

Let's look closer at the core technologies and tools behind low-latency AI video avatars and see how each adds to creating an ideal, real-time interaction.

Core Technologies for Building Low-Latency AI Video Avatars

To accomplish low-latency setup, you’ll use the following technologies and format:

Simli's API: Simli's API allows you to create lip-synced AI avatars that convert audio into video in real time. This API provides the foundation for transforming audio into synchronized visual avatars quickly.
WebRTC: This handles the media stream capture and adds to the peer connection.
PCM and Audio Downsampling: To minimize latency in audio transmission, pulse code modulation (PCM) is used for audio data, which WebRTC can handle efficiently. PCM digitizes audio signals by sampling analog waveforms and converting them into uncompressed binary data. Downsampling to 16kHz PCM ensures seamless compatibility with WebRTC and the Simli client, supporting real-time, synchronized audio playback.
OpenAI API: Powers real-time language understanding and response generation, enabling the avatar to engage in dynamic, intelligent interactions.

Having covered the technologies and tools, let's dive into how you can set up your API environment. For the following steps, you will be configuring access to both Simli and OpenAI so you can have smooth integration and interaction within your application.

Set Up Your API Environment

You'll need API keys for both Simli and OpenAI to set up the API environment. To get a Simli API key, go to Simli's official website, click "Get API Access" and create an account.

After logging in, you should see your developer dashboard with your API keys as shown below.

Copy and keep the API key somewhere safe because you will need it to authenticate requests within your application.

With your API key ready, let’s move on to customizing the appearance of your AI avatar. Navigate to the ‘Available Faces’ section in Simli’s API documentation to find listed Avatar face IDs as seen below:

You can also create your own avatar by clicking on the “Create Avatar” Icon on your dashboard. Once you follow the steps highlighted on the page, you’ll be able to create your own Avatar.

To get the ID for each face, copy the random text after the name. For example, the ID for Jenna will be tmp9i8bbq7c. If you created your own Avatar, an ID would also be given alongside. Now that you have the face ID and the Simli API key, let’s create a Next.js app.

Create the Project

Create a Next.js app by running this command in your terminal:

npx create-next-app@latest simli-demo

Answer the following prompts:

What is your project named? my-app
Would you like to use TypeScript? No / Yes
Would you like to use ESLint? No / Yes
Would you like to use Tailwind CSS? No / Yes
Would you like your code inside a `src/` directory? No / Yes
Would you like to use App Router? (recommended) No / Yes
Would you like to use Turbopack for `next dev`?  No / Yes
Would you like to customize the import alias (`@/*` by default)? No / Yes
What import alias would you like configured? @/*

Typescript and Tailwind are essential for this tutorial, as they simplify styling and provide type safety, making it easier to create a responsive and maintainable avatar interface. Once that is done, install the dependencies or packages for this project. They include:

simli-client: This will be used for handling facial animations or avatar interactions.
openai: This is the main OpenAI SDK and will be used for accessing the AI service.
openai/realtime-api-beta: This is OpenAI's real-time language API that will be used for voice conversations.
Ws: WebSocket client and server implementation for real-time communication.

Open up your terminal once more and run the command below, it will install all the dependencies needed for the project.

npm install simli-client openai github:openai/openai-realtime-api-beta ws

Add Environment Variables
Create a .env file and copy your Simli API key and OpenAI key. Paste it in as shown below:

NEXT_PUBLIC_SIMLI_API_KEY="SIMLI-API-KEY"
NEXT_PUBLIC_OPENAI_API_KEY="OPENAI-API-KEY"

Create the VideoBox Components

You'll need to create a component to handle the video feed display and the audio playing. Inside the app folder, create a folder called components, and inside it, create a file called VideoBox.tsx and add the following code:

// app/components/VideoBox.tsx

    // Render a styled video and audio container with auto-play functionality.

    export default function VideoBox(props: any) {
       return (
           <div className="aspect-video flex rounded-sm overflow-hidden items-center h-[350px] w-[350px] justify-center bg-simligray">
               <video ref={props.video} autoPlay playsInline></video>
               <audio ref={props.audio} autoPlay ></audio>
           </div>
       );
    }

The code defines a VideoBox component that displays a video and audio player within a styled container. This component expects two props: video and audio, which are references passed to the <video> and <audio> elements. Both elements have autoPlay enabled, starting playback immediately, and playsInline is set on the video to prevent it from going fullscreen on mobile devices. These ref attributes (props.video and props.audio) allow the primary component to control playback, pause, and other media interactions.

Build the SimliOpenAIPushToTalk Component

Inside the app folder, create a file called SimliOpenAIPushToTalk.tsx. This file will contain much of the core functionality. Start by adding the following code:

// app/SimliOpenAIPushToTalk.tsx

    // import the necessary packages
    ...

    import React, { useCallback, useEffect, useRef, useState } from "react";
    import { RealtimeClient } from "@openai/realtime-api-beta";
    import { SimliClient } from "simli-client";
    import VideoBox from "./Components/VideoBox";
    import IconExit from "@/media/IconExit";
    interface SimliOpenAIPushToTalkProps {
      simli_faceid: string;
      openai_voice: "echo" | "alloy" | "shimmer";
      initialPrompt: string;
      onStart: () => void;
      onClose: () => void;
    }
    const simliClient = new SimliClient();
    const SimliOpenAIPushToTalk: React.FC<SimliOpenAIPushToTalkProps> = ({
      simli_faceid,
      openai_voice,
      initialPrompt,
      onStart,
      onClose,
    }) => {
      const [isLoading, setIsLoading] = useState(false);
      const [isAvatarVisible, setIsAvatarVisible] = useState(false);
      const [error, setError] = useState("");
      const [isRecording, setIsRecording] = useState(false);
      const [userMessage, setUserMessage] = useState("...");
      const [isButtonDisabled, setIsButtonDisabled] = useState(false);
      const videoRef = useRef<HTMLVideoElement>(null);
      const audioRef = useRef<HTMLAudioElement>(null);
      const openAIClientRef = useRef<RealtimeClient | null>(null);
      const audioContextRef = useRef<AudioContext | null>(null);
      const streamRef = useRef<MediaStream | null>(null);
      const processorRef = useRef<ScriptProcessorNode | null>(null);
      const audioChunkQueueRef = useRef<Int16Array[]>([]);
      const isProcessingChunkRef = useRef(false);
    ...

In the above codes, the necessary packages and hooks are imported, a prop interface SimliOpenAIPushToTalkProps is defined for component configuration, and an instance of SimliClient is created to manage multimedia interactions. The SimliOpenAIPushToTalk component requires a specific set of props, which are defined in the SimliOpenAIPushToTalkProps interface.

References are also set up for video and audio elements, the OpenAI client, audio context, media stream, and audio processor, allowing direct control over multimedia resources within the component.

Next, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // initializing the SimliClient
    ...

    const initializeSimliClient = useCallback(() => {
        if (videoRef.current && audioRef.current) {
          const SimliConfig = {
            apiKey: process.env.NEXT_PUBLIC_SIMLI_API_KEY,
            faceID: simli_faceid,
            handleSilence: true,
            videoRef: videoRef,
            audioRef: audioRef,
          };
          simliClient.Initialize(SimliConfig as any);
          console.log("Simli Client initialized");
        }
      }, [simli_faceid]);
      const initializeOpenAIClient = useCallback(async () => {
        try {
          console.log("Initializing OpenAI client...");
          openAIClientRef.current = new RealtimeClient({
            apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
            dangerouslyAllowAPIKeyInBrowser: true,
          });
          await openAIClientRef.current.updateSession({
            instructions: initialPrompt,
            voice: openai_voice,
            turn_detection: { type: "server_vad" },
            input_audio_transcription: { model: "whisper-1" },
          });
          openAIClientRef.current.on(
            "conversation.updated",
            handleConversationUpdate
          );
          openAIClientRef.current.on(
            "input_audio_buffer.speech_stopped",
            handleSpeechStopped
          );
          await openAIClientRef.current.connect();
          console.log("OpenAI Client connected successfully");
          setIsAvatarVisible(true);
        } catch (error: any) {
          console.error("Error initializing OpenAI client:", error);
          setError(`Failed to initialize OpenAI client: ${error.message}`);
        }
      }, [initialPrompt]);
      const handleConversationUpdate = useCallback((event: any) => {
        console.log("Conversation updated:", event);
        const { item, delta } = event;
        if (item.type === "message" && item.role === "assistant") {
          console.log("Assistant message detected");
          if (delta && delta.audio) {
            const downsampledAudio = downsampleAudio(delta.audio, 24000, 16000);
            audioChunkQueueRef.current.push(downsampledAudio);
            if (!isProcessingChunkRef.current) {
              processNextAudioChunk();
            }
          }
        } else if (item.type === "message" && item.role === "user") {
          setUserMessage(item.content[0].transcript);
        }
      }, []);
    ...

The code above does the following:

Initializes the SimliClient with configuration parameters, including simli_faceid for Simli's face recognition and video references, and manages audio-visual synchronization with the AI avatar using the API key and face ID. With this, you'll have a real-time lip-sync between the avatar's visuals and audio input.
Sets up and connects the OpenAI client—registers event listeners for updates in conversation and speech detection. The initializeOpenAIClient function sets up the connection to the OpenAI client, using the provided API key and settings, and registers event listeners to handle conversation updates and detect when speech stops.
Handles real-time updates from the conversation, processing both assistant messages and user inputs. For audio responses from the assistant, the code downsamples and adds chunks to a queue.

Directly after handleConversationUpdate, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // process audio chunks and send to Simli
    ...

    const processNextAudioChunk = useCallback(() => {
        if (
          audioChunkQueueRef.current.length > 0 &&
          !isProcessingChunkRef.current
        ) {
          isProcessingChunkRef.current = true;
          const audioChunk = audioChunkQueueRef.current.shift();
          if (audioChunk) {
            const chunkDurationMs = (audioChunk.length / 16000) * 1000; 

            simliClient?.sendAudioData(audioChunk as any);
            console.log(
              "Sent audio chunk to Simli:",
              chunkDurationMs,
              "Duration:",
              chunkDurationMs.toFixed(2),
              "ms"
            );
            isProcessingChunkRef.current = false;
            processNextAudioChunk();
          }
        }
      }, []);

      const handleSpeechStopped = useCallback((event: any) => {
        console.log("Speech stopped event received", event);
      }, []);

      const downsampleAudio = (
        audioData: Int16Array,
        inputSampleRate: number,
        outputSampleRate: number
      ): Int16Array => {
        if (inputSampleRate === outputSampleRate) {
          return audioData;
        }
        const ratio = inputSampleRate / outputSampleRate;
        const newLength = Math.round(audioData.length / ratio);
        const result = new Int16Array(newLength);
        for (let i = 0; i < newLength; i++) {
          const index = Math.round(i * ratio);
          result[i] = audioData[index];
        }
        return result;
      };
    ...

Here:

The processNextAudioChunk manages the processing of audio chunks from the assistant, sending them immediately to Simli.
The downsampleAudio converts audio data to a 16kHz PCM format. PCM captures audio in an uncompressed, raw format, ideal for maintaining low latency. The PCM audio is then sent to SimliClient without encoding delays.

After the downsampleAudio function, add the following code:

// app/SimliOpenAIPushToTalk.tsx
    ...

    // Manages audio recording setup and cleanup for real-time processing.
    ...

    const startRecording = useCallback(async () => {
        if (!audioContextRef.current) {
          audioContextRef.current = new AudioContext({ sampleRate: 24000 });
        }
        try {
          console.log("Starting audio recording...");
          streamRef.current = await navigator.mediaDevices.getUserMedia({
            audio: true,
          });
          const source = audioContextRef.current.createMediaStreamSource(
            streamRef.current
          );
          processorRef.current = audioContextRef.current.createScriptProcessor(
            2048,
            1,
            1
          );
          processorRef.current.onaudioprocess = (e) => {
            const inputData = e.inputBuffer.getChannelData(0);
            const audioData = new Int16Array(inputData.length);
            for (let i = 0; i < inputData.length; i++) {
              audioData[i] = Math.max(
                -32768,
                Math.min(32767, Math.floor(inputData[i] * 32768))
              );
            }
            openAIClientRef.current?.appendInputAudio(audioData);
          };
          source.connect(processorRef.current);
          processorRef.current.connect(audioContextRef.current.destination);
          setIsRecording(true);
          console.log("Audio recording started");
        } catch (err) {
          console.error("Error accessing microphone:", err);
          setError("Error accessing microphone. Please check your permissions.");
        }
      }, []);

      const stopRecording = useCallback(() => {
        if (processorRef.current) {
          processorRef.current.disconnect();
          processorRef.current = null;
        }
        if (streamRef.current) {
          streamRef.current.getTracks().forEach((track) => track.stop());
          streamRef.current = null;
        }
        setIsRecording(false);
        console.log("Audio recording stopped");
      }, []);
    ...

The startRecording function accesses the microphone in real time using getUserMedia, captures audio with AudioContext, processes it as 16-bit PCM, and sends it to the OpenAI client. The stopRecording function stops the stream and releases resources. This setup uses AudioContext for real-time audio, while WebRTC configuration is managed by the Simli client.

Next to add is the stopRecording function, paste this code below:


// app/SimliOpenAIPushToTalk.tsx
...
// Control interaction, push-to-talk, and audio visualization.

...

const handleStart = useCallback(async () => {
    setIsLoading(true);
    setError("");
    onStart();
    try {
      await simliClient?.start();
      await initializeOpenAIClient();
    } catch (error: any) {
      console.error("Error starting interaction:", error);
      setError(`Error starting interaction: ${error.message}`);
    } finally {
      setIsAvatarVisible(true);
      setIsLoading(false);
    }
  }, [initializeOpenAIClient, onStart]);
  const handleStop = useCallback(() => {
    console.log("Stopping interaction...");
    setIsLoading(false);
    setError("");
    setIsAvatarVisible(false);
    simliClient?.close();
    openAIClientRef.current?.disconnect();
    if (audioContextRef.current) {
      audioContextRef.current.close();
    }
    stopRecording();
    onClose();
    console.log("Interaction stopped");

    window.location.reload();
  }, [stopRecording]);

  const handlePushToTalkStart = useCallback(() => {
    if (!isButtonDisabled) {
      startRecording();

      simliClient?.ClearBuffer();
      openAIClientRef.current?.cancelResponse("");
    }
  }, [startRecording, isButtonDisabled]);
  const handlePushToTalkEnd = useCallback(() => {
    setTimeout(() => {
      stopRecording();
    }, 500);
  }, [stopRecording]);

  const AudioVisualizer = () => {
    const [volume, setVolume] = useState(0);
    useEffect(() => {
      const interval = setInterval(() => {
        setVolume(Math.random() * 100);
      }, 100);
      return () => clearInterval(interval);
    }, []);
    return (
      <div className="flex items-end justify-center space-x-1 h-5">
        {[...Array(5)].map((_, i) => (
          <div
            key={i}
            className="w-2 bg-black transition-all duration-300 ease-in-out"
            style={{
              height: `${Math.min(100, volume + Math.random() * 20)}%`,
            }}
          />
        ))}
      </div>
    );
  };

The code above does the following:

Initializes the Simli and OpenAI clients through the handleStart function, and resets the error state. It also sets isLoading to true while the clients start.
Stops recording, disconnects clients through the handleStop function and resets the component state.
Manages push-to-talk functionality through the Push-to-Talk functions starting and stopping recording when the user interacts with the button.
Displays an animated audio visualizer that gives feedback on microphone volume through the audio visualizer.

Following the audioVisualizer function, add the code:

// app/SimliOpenAIPushToTalk.tsx
    ...
    // Initialize Simli client and render video and push-to-talk UI.

    useEffect(() => {
        initializeSimliClient();
        if (simliClient) {
          simliClient?.on("connected", () => {
            console.log("SimliClient connected");
            const audioData = new Uint8Array(6000).fill(0);
            simliClient?.sendAudioData(audioData);
            console.log("Sent initial audio data");
          });
        }
        return () => {
          try {
            simliClient?.close();
            openAIClientRef.current?.disconnect();
            if (audioContextRef.current) {
              audioContextRef.current.close();
            }
          } catch {}
        };
      }, [initializeSimliClient]);
      return (
        <>
          <div className="transition-all duration-300 ">
            <VideoBox video={videoRef} audio={audioRef} />
          </div>
          <div className="flex flex-col items-center">
            {!isAvatarVisible ? (
              <button
                onClick={handleStart}
                disabled={isLoading}
                className="w-full h-[52px] mt-4 disabled:bg-gray-600 disabled:text-white disabled:hover:rounded-full bg-green-500 text-white py-3 px-6 rounded-full transition-all duration-300 hover:text-black hover:bg-white hover:rounded flex justify-center items-center"
              >
                {isLoading ? (
                  <span>Loading...</span>
                ) : (
                  <span className="font-abc-repro-mono font-bold w-[164px]">
                    Test Interaction
                  </span>
                )}
              </button>
            ) : (
              <>
                <div className="flex items-center gap-4 w-full">
                  <button
                    onMouseDown={handlePushToTalkStart}
                    onTouchStart={handlePushToTalkStart}
                    onMouseUp={handlePushToTalkEnd}
                    onTouchEnd={handlePushToTalkEnd}
                    onMouseLeave={handlePushToTalkEnd}
                    disabled={isButtonDisabled}
                    className={`mt-4 text-white flex-grow bg-green-500 hover:rounded hover:bg-opacity-70 h-[52px] px-6 rounded-full transition-all duration-300 ${
                      isRecording ? "bg-gray-900 rounded hover:bg-opacity-100" : ""
                    }`}
                  >
                    <span className="font-abc-repro-mono font-bold w-[164px]">
                      {isRecording ? "Release to Stop" : "Push & hold to talk"}
                    </span>
                  </button>
                  <button
                    onClick={handleStop}
                    className=" group w-[52px] h-[52px] flex items-center mt-4 bg-red text-white justify-center rounded-[100px] backdrop-blur transition-all duration-300 hover:bg-white hover:text-black hover:rounded-sm"
                  >
                    <IconExit className="group-hover:invert-0 group-hover:brightness-0 transition-all duration-300" />
                  </button>
                </div>
              </>
            )}
          </div>
        </>
      );
    };
    export default SimliOpenAIPushToTalk;

In the code above:

The useEffect hook runs when the page mounts to initialize simliClient and sets up cleanup operations to close simliClient and audioContextRef when the component unmounts.
Render the UI to:
- Display the VideoBox component, which receives video and audio props for controlling media playback, along with controls to start interaction or initiate push-to-talk.
- A button triggers handleStart or handleStop, while the push-to-talk button manages recording on mouse down and up.

Now that the core components and UI elements are set up, let’s move on to integrating these into the main demo component. Head over to page.tsx and replace its contents with the following code to complete the setup.

Set Up Demo Component
Navigate to page.tsx and replace it with the following code:

    // app/page.tsx

    // render the AI avatar with push-to-talk functionality
    "use client";
    import React, { useEffect, useState } from "react";
    import SimliOpenAIPushToTalk from "./SimliOpenAIPushToTalk";
    interface avatarSettings {
      name: string;
      openai_voice: "echo" | "alloy" | "shimmer";
      simli_faceid: string;
      initialPrompt: string;
    }
    const avatar: avatarSettings = {
      name: "Frank",
      openai_voice: "echo",
      simli_faceid: "5514e24d-6086-46a3-ace4-6a7264e5cb7c",
      initialPrompt:
        "You are a helpful AI assistant named Frank. You are friendly and concise in your responses. Your task is to help users with any questions they might have. Your answers are short and to the point, don't give long answers be brief and straightforward.",
    };
    const Demo: React.FC = () => {
      const [interactionMode, setInteractionMode] = useState<
        "pushToTalk" | undefined
      >(undefined);
      useEffect(() => {
        const storedInteractionMode = localStorage.getItem("interactionMode");
        if (storedInteractionMode === "pushToTalk") {
          setInteractionMode("pushToTalk");
        }
      }, []);
      const onStart = () => {
        console.log("Setting setshowDottedface to false...");
      };
      const onClose = () => {
        console.log("Setting setshowDottedface to true...");
      };
      return (
        <div className="bg-black min-h-screen flex flex-col items-center font-abc-repro font-normal text-sm text-white p-8">
          <div className="flex flex-col items-center gap-6 bg-effect15White p-6 pb-[40px] rounded-xl w-full">
            <div>
              <SimliOpenAIPushToTalk
                openai_voice={avatar.openai_voice}
                simli_faceid={avatar.simli_faceid}
                initialPrompt={avatar.initialPrompt}
                onStart={onStart}
                onClose={onClose}
              />
            </div>
          </div>
        </div>
      );
    };
    export default Demo;

In the above code, you created a component that renders the AI avatar with push-to-talk functionality. It also initializes settings for the avatar. You can also customize your avatar here by changing it to any face you want for interaction. Simli supports a range of custom avatar faces that you can apply to various use cases.

Set Up Layout
The last thing you’ll need to do is edit the layout, in the layout.tsx, paste this code below:

 // app/layout.tsx

    // edit the layout
    import type { Metadata } from "next";
    import { Inter } from "next/font/google";
    import { abcRepro, abcReproMono } from './fonts/fonts';
    import "./globals.css";

    const inter = Inter({ subsets: ["latin"] });
    export const metadata: Metadata = {
     title: "Simli App",
     description: "create-simli-app (OpenAI)",
    };
    export default function RootLayout({
     children,
    }: Readonly<{
     children: React.ReactNode;
    }>) {
     return (
       <html lang="en" className={`${abcReproMono.variable} ${abcRepro.variable}`}>
         <body className={inter.className}>{children}</body>
       </html>
     );
    }

Test the application by running the following command on your terminal:

npm run dev

Check the video demonstration of the project to see it in action. You can also explore the Simli repository to build avatars and work on more projects using Simli API.

Conclusion

In this project, you have implemented a low-latency AI video avatar for realistic and dynamic interactions. Simli's API with WebRTC allows developers to easily deploy and manage latency-free, perfectly synchronized audio-visual avatars that are very well-suited for a wide range of applications, such as training simulations, interactive presentations, customer support, and real-time virtual assistants.

This setup opens the door to engaging, human-like interactions across various settings.

Explore the documentation to learn more about the features and tools Simli offers.

Creating A Virtual Assistant Using Python

Tiioluwani — Sat, 03 Dec 2022 05:23:19 +0000

Are you attempting to obtain a Python project that will familiarize you with particular Python libraries? A project that could undoubtedly help you with that is a virtual assistant.

The Python libraries used in this project can help you improve your Python programming skills. Working on this project is also enjoyable because having a virtual assistant is incredible.read more

Understanding Python Functions

Tiioluwani — Tue, 02 Aug 2022 20:37:41 +0000

As you progress in Python programming, you will undoubtedly want to use some functions to perform specific programming tasks, so understanding what they are is critical.

In this article, we'll look at these functions, learning what they are and when and how to use them.To read more

How To Enable "running Scripts" On Your System

Tiioluwani — Sat, 16 Jul 2022 21:31:48 +0000

Have you ever been in a situation whereby you're unable to perform specific tasks on your source code editor? For instance, in trying to activate a virtual environment on your editor, you come across an error that says "the command cannot be loaded on your system because running scripts is disabled on your system". We'll take a look at how to solve this in this article.To know more

Basic Built-in Python Functions To Know As A Beginner

Tiioluwani — Tue, 14 Jun 2022 19:53:32 +0000

In python, getting familiar with built-in functions helps a lot while programming. They help in breaking big tasks into smaller chunks and increase code’s clarity. Also, big programs are broken down into smaller units thus making it easier to read, understand and debug if the need arises. In this tutorial, we’ll be taking a look at some built-in functions in python programming.To continue reading

SETTING UP A VIRTUAL ENVIRONMENT

Tiioluwani — Mon, 13 Jun 2022 11:41:37 +0000

It is a fact that as we engage in various kinds of projects in python, we'll come across dependencies. These dependencies are a general phrase in software engineering that refers to when one piece of software depends on another or, simply put, a set of other people's code that we can install and utilize instead of having to rewrite it ourselves. When we come across these dependencies, having a virtual environment then comes in handy.To continue reading

Forem: Tiioluwani

5 Best Python Vector Database Libraries

The Python Landscape: Understanding the Options

Four client architectures

Client Comparison: Developer Experience Deep Dive

1. Pinecone Python client

2. Weaviate Python client

3. ChromaDB Python client

4. Qdrant Python client

Installation and Environment Management

The compatibility matrix

Virtual environment best practices

Extras and optional dependencies

Local Development Workflows

Embedded mode

Docker for local development

Cloud-only development

Integration Ecosystem: LangChain and LlamaIndex

LangChain maturity

LlamaIndex support

The integration tax

Performance Considerations: Beyond Raw Speed

Protocol choice: REST vs. gRPC

Async support quality

Connection pooling and resource management

Error Handling and Debugging

Error message quality

Logging and observability

Common pitfalls and solutions

Actian VectorAI DB

Architecture and design

Deployment options

Compliance design

Performance targets

Decision Framework: Choosing Your Python Client

Final Thoughts

Can AI in Manufacturing Work Without the Cloud? A Guide

What You Are Building

Building a Local RAG Pipeline with VectorAI DB

Prerequisites

Project structure

Step 1: Deploy VectorAI DB

Step 2: Build the ingestion pipeline

Step 3: Run your first query

Step 4: Add hybrid filters

Step 5: Connect the local LLM

Step 6: Add audit logging

Wrapping Up

Navigating Search Solutions: A Comprehensive Comparison Guide to Meilisearch, Algolia, and ElasticSearch

Introduction

Key Considerations When Choosing a Search Solution

Unveiling the Contenders

Meilisearch

Algolia

ElasticSearch

Feature Comparison Table

Scalability

Performance

Ease of Integration

Indexing

Querying

Relevance

Customization and User Experience

Choosing the Right Solution

1. Data Size

2. Budget

3. Technical Expertise

4. Desired Features

Conclusion

How to Build Low-latency AI Video Avatars for Real-time Interactions

Prerequisites

Understanding the Key Components for Low-Latency AI Avatars

Core Technologies for Building Low-Latency AI Video Avatars

Set Up Your API Environment

Create the Project

Create the VideoBox Components

Build the SimliOpenAIPushToTalk Component

Conclusion

Creating A Virtual Assistant Using Python

Understanding Python Functions