Forem: Adheeban Manoharan

🗃️ Lessons Learned from Migrating Huge BigQuery Datasets Across Regions

Adheeban Manoharan — Sat, 14 Sep 2024 16:11:00 +0000

TL;DR We were tasked with migrating a large dataset from the US to the EU due to the client's EU-based operations and GDPR compliance requirements. In this post, I’ll walk you through the challenges I faced while transferring a BigQuery dataset across regions. Since Google's documentation on this subject is quite limited, I believe this guide will be invaluable for anyone attempting a similar migration.

Prelude

When tasked with migrating datasets across the globe (for example, from the US to the EU region), we had to address three key questions before starting the process:

What limitations does Google impose on data exports?

Understanding the constraints of Google's infrastructure was crucial to avoid unforeseen issues.

What volume of data are we dealing with?

The size and complexity of the dataset would significantly impact the migration strategy.

How can we ensure zero downtime during the migration?

Minimizing disruption to services while the data was being transferred was a top priority.

Mind you, our task is creating the same dataset with the same name and same number of tables in a different region. Now, Let's go over the questions one after the other.

Google's Limitations

Google's BigQuery by nature has some acceptable limitations (at the time of writing this blog) due to its architecture . Let me list them down. Some of them might sound pretty obvious but I'm still listing them down as we would have the full context when we go through how we solved this migration problem.

You cannot create a second dataset with the same name.
You cannot create a multi-regional dataset (This limitation applies only for the Asia region, US and EU multi regional buckets can be created)
You cannot export the contents of a BigQuery table that resides in the US region to a GCS bucket that resides in the EU region. (Cross regional export)
When you create a dataset, it usually takes about 20-120 seconds for it to be available for the APIs that consume this dataset, until then you'd get 404s.
Google doesn't give support for migrating these datasets, so we have to do it ourselves.

Data Size

Our application stores audit data in BigQuery, serving as our cold storage. Given the nature of audit logs, the datasets have grown to a colossal size. For example, some tables, even in their compressed form (Parquet format with Snappy compression), contained up to 4 terabytes of data. Since this is critical audit data, ensuring zero data loss was absolutely essential during the migration.

Zero Downtime

As mentioned earlier, our audit system writes data continuously to these BigQuery datasets. This means that migrating a customer’s data without downtime or prior notification could result in data loss during the migration window. In our case, it was crucial that the migration proceeded seamlessly, with zero downtime, so as not to disrupt the customer's operations.

How it actually went down

After getting all the information we needed, Below is the approach we finalised. In this illustration let's go with the assumption that we are migrating a US multi-regional dataset to a EU multi-regional location.

Yes, I hear you. I know how confusing this image is. I suck at these engineering illustrations. So let me explain.

1) Export Table Data as Parquet:

We initiated the migration by exporting the table data from the US dataset to a Google Cloud Storage (GCS) bucket in the US region. The data was saved in Parquet format, which is ideal for both storage and transfer due to its efficient compression.

2) Extract Table Schema and Partition Details:

Along with the data, we extracted the table schema, partition and clustering details, saving them as JSON files. This ensured that the tables would be accurately recreated during the migration.

3) Transfer Data to the EU bucket:

Using Google Cloud Storage Transfer service, we moved the exported data from the US bucket to a bucket in the EU region. This step circumvented BigQuery’s restriction on direct cross-regional exports. We chose storage transfer as this was the most efficient way google had for transferring data from one bucket to the other.

4) Create a Temporary Dataset:

A temporary dataset was created in the EU region to serve as an intermediary for data migration. This dataset mirrored the structure of the original tables from the US dataset. The application will write it's data during the migration window to this dataset.

5) Enable the Migration Flag:

At this stage, the migration flag was enabled in the application, ensuring that all new data writes occurred in the temporary EU dataset. This step allowed us to maintain zero downtime during the migration, as the application seamlessly redirected its writes without any interruption to the customer’s operations.

6) Delete the US Dataset:

Once we verified that the application was writing to the EU region without any issues, the old US dataset was deleted to comply with GDPR and avoid any confusion or conflict with the migrated data.

7) Create a Permanent EU Dataset:

A new permanent dataset was created in the EU region. This dataset would become the final destination for the migrated data using the schema that we saved in step 2, which ensured that the original dataset its exactly replicated. After creating this dataset, the data in the EU bucket is loaded into the newly created EU dataset's tables.

8) Turn off the Migration Flag:

Since a new EU dataset with the same name is created now, the migration flag that was enabled in step 5 can now be safely disabled, making the application redirect it's writes to the new EU dataset as it rightfully should.

9) Export Data from the Temporary Dataset:

Now during this migration window, the interim application data would've been written to the temp dataset. So, this data also has to be imported into the EU dataset. The data in the temporary dataset was exported to the EU Bucket.

10) Import Data into the Permanent EU Dataset:

The exported temporary data in the EU bucket is imported into the EU dataset in the same flow of operations. This marked the end of the migration process for the customer’s data.

11) Cleanup:

Now after ensuring that the migration is successful. The contents in the two buckets (US and EU) along with the temp dataset was deleted. Until then, this data served as a fallback.

Conclusion

Thanks for sticking with me till the very end! I know this might not be the most efficient way out there, but this ensured zero data-loss with zero-downtime during this migration activity for us. There are very few documentations on this and browsing endlessly for the documentation made me write this blog. I'd love hear your opinions on this.

Cheers, see you in the next one! 😉

The Battle of PR-Agents⚡

Adheeban Manoharan — Mon, 11 Dec 2023 13:25:45 +0000

In the ever-evolving landscape of software development, the effective management of pull requests (PRs) stands as a linchpin for collaboration, code quality, and project success. This blog is entirely dedicated to my personal quest for a great AI driven PR-Agent, along with my findings. Among the myriad of tools available, CodiumAI's PR-Agent and GitHub Copilot emerge as contenders that promise to streamline the PR review process and elevate the developer experience. In this extensive exploration, we will delve into the depths of these tools, dissecting each feature to understand their nuances, strengths, and how they contribute to reducing time exceptionally – a crucial aspect in the fast-paced world of modern software development.

1. Command Arsenal: A Multifaceted Symphony

CodiumAI PR-Agent:

Auto Description (/describe): This feature stands out by automatically generating a comprehensive PR description that includes the title, type, summary, code walkthrough, and labels. It ensures that the reviewer gets a holistic view of the changes introduced.
Auto Review (/review): Offering flexibility, PR-Agent's auto review provides adjustable feedback on various aspects such as the PR's main theme, type, relevant tests, security issues, score, and diverse content suggestions.
Question Answering (/ask ...): Facilitating communication, this feature enables free-text interaction by answering questions related to the PR, promoting collaboration and clear understanding.
Code Suggestions (/improve): A powerful addition to the arsenal, this feature provides commit-worthy code suggestions, aiding developers in enhancing the quality of their code.
Update Changelog (/update_changelog): Streamlining documentation, PR-Agent automatically updates the CHANGELOG.md file with the PR changes, ensuring that the documentation stays up-to-date.
Find Similar Issue (/similar_issue): Enhancing issue management, this feature retrieves and presents similar issues associated with the PR, aiding in thorough issue resolution.
Add Documentation (/add_docs): Addressing completeness, PR-Agent automatically adds documentation to undocumented functions/classes in the PR, promoting better code documentation practices.
Generate Custom Labels (/generate_labels): Streamlining organization, this feature suggests custom labels based on PR code changes, ensuring a standardized labeling process.

GitHub Copilot:

GitHub Copilot primarily focuses on a single command, centering around code suggestions derived from natural language prompts. While proficient in generating code snippets, it lacks the multifaceted approach seen in PR-Agent's symphony of commands.

Analysis:
PR-Agent's multifaceted command set orchestrates a harmonious blend, catering to the diverse needs of the PR review process. In contrast, Copilot's spotlight on code suggestions is a solo performance, lacking the richness found in PR-Agent's symphony of commands.

2. Platform Universality: Bridging Git Divides

CodiumAI PR-Agent:

Compatible with all Git platforms, PR-Agent ensures flexibility for teams using diverse Git repositories. This universality promotes collaboration across various platforms, breaking down silos.

GitHub Copilot:

Tailored for GitHub, Copilot's integration capabilities are potentially limited when it comes to non-GitHub Git platforms. This may pose challenges for teams working in mixed Git environments.

Analysis:
PR-Agent's universal compatibility allows for seamless collaboration across various Git repositories, surpassing the constraints observed in GitHub Copilot, which caters primarily to GitHub users.

3. Open-Source Dynamics: A Community-Driven Ethos

CodiumAI PR-Agent:

Flourishing as an open-source solution, PR-Agent embraces transparency and encourages community contributions. The collaborative ethos allows the community to contribute to the tool's continuous improvement.

GitHub Copilot:

Enveloped in closed code, Copilot restricts visibility into underlying algorithms. While a robust tool, its closed-code nature limits the community-driven enhancements seen in open-source solutions.

Analysis:
PR-Agent's open-source nature invites community-driven improvements, fostering transparency. In contrast, Copilot's closed-code approach may hinder collaborative contributions.

4. Integration Landscape: A Seamless Experience

CodiumAI PR-Agent:

PR-Agent embeds itself seamlessly into both Git and popular Integrated Development Environments (IDEs). This ensures a harmonious experience for developers, regardless of their preferred environment.

GitHub Copilot:

Primarily designed for Git, Copilot may have limited or no native integration into popular IDEs. This could pose challenges for developers who rely heavily on integrated development environments.

Analysis:
PR-Agent's dual integration into both Git and IDE environments ensures a consistent developer experience, offering flexibility not necessarily found in Copilot's primarily Git-centric design.

5. Time Efficiency: The Quintessence of PR Management

In the fast-paced world of software development, time is a precious commodity. Both CodiumAI's PR-Agent and GitHub Copilot recognize the urgency and need for efficiency in the PR management process. The multifaceted command set of PR-Agent, encompassing auto-description, review, question answering, and more, is crafted to expedite the review process while maintaining a high standard of quality. The ability to automate documentation updates, find similar issues, and suggest custom labels significantly reduces the manual overhead, allowing developers to allocate their time to more critical tasks.

Similarly, GitHub Copilot's prowess in generating code snippets from natural language prompts streamlines the coding process, reducing the time required to write and review code. While Copilot's focus is more centered on code suggestions, its efficiency in this domain is undeniable, contributing to accelerated development cycles.

In essence, both tools present a paradigm shift in how developers approach PR management, emphasizing time efficiency without compromising on quality. By automating repetitive tasks, offering intelligent suggestions, and promoting collaboration, these tools empower development teams to navigate the complex PR landscape with agility.

In Conclusion: CodiumAI's PR-Agent Takes the Lead

While both CodiumAI's PR-Agent and GitHub Copilot bring innovation to the PR management landscape, the versatility, open-source nature, and holistic approach of PR-Agent make it stand out. CodiumAI's commitment to reducing the manual overhead, fostering collaboration, and accommodating diverse Git platforms positions PR-Agent as a go-to tool for development teams striving for efficiency and excellence in their PR workflows.

As you navigate the intricacies of PR solutions, the key consideration remains:

If it aligns with your multifaceted needs, accelerates your development workflow, and provides a comprehensive solution, CodiumAI's PR-Agent is a tool worth exploring!

The Problem

Now let me explain why I did this. The reason I went into this PR-Agent exploration is because I had a problem at hand. I'm an independent developer and for most of the projects I work for, mostly I would review my own PR, and there is a lot of room for oversight. I wanted an intelligent system that serves as my co-developer that reviews my PRs for me and provide valuable review comments. CodiumAI's PR-Agent did exactly what I needed. Attaching here the review done by CodiumAI for one of my PRs.

Now that's about it. Share your insights and thoughts on this detailed comparison in the comments below. Hope this helps. Until the next one - cheers!

Merge Sort

Adheeban Manoharan — Tue, 24 Oct 2023 06:56:08 +0000

Today, we're going to dive into the world of Merge Sort, a widely used sorting algorithm known for its efficiency, reliability, and stability.

What is Merge Sort?

Merge Sort is a sorting algorithm that falls under the "divide and conquer" category. It was developed in 1945 by John von Neumann and is often used for sorting arrays. The basic idea behind Merge Sort is to divide a large array into smaller, more manageable sub-arrays, sort them, and then merge these sub-arrays back together in a sorted manner.

Merge Sort operates recursively and is known for its stable sorting properties, making it an excellent choice for sorting large data sets.

How Merge Sort Works

The key to understanding Merge Sort lies in its name. It "merges" and "sorts" to achieve its goal. Here's a high-level overview of the process:

Divide: The unsorted array is divided into two equal halves until individual elements are reached.
Conquer: Each of these smaller arrays is sorted recursively using Merge Sort.
Merge: Finally, the sorted sub-arrays are merged back together, maintaining their order.

This process continues until the entire array is sorted.

Time and Space Complexity

Time Complexity: Merge Sort has a time complexity of O(n log n), which makes it efficient for sorting large data sets. The "divide and conquer" approach ensures that each element is compared only once.
Space Complexity: The space complexity of Merge Sort is O(n), which means it requires additional memory for creating and merging sub-arrays. This can be a drawback for large arrays with limited memory.

Implementation with Python

Let's take a look at a Python implementation of Merge Sort. This code demonstrates the divide-and-conquer approach to sorting an array.

def merge_sort(arr):
    if len(arr) <= 1:
        return arr

    # Divide the array into two halves
    middle = len(arr) // 2
    left_half = arr[:middle]
    right_half = arr[middle:]

    # Recursively sort both halves
    left_half = merge_sort(left_half)
    right_half = merge_sort(right_half)

    # Merge the sorted halves
    return merge(left_half, right_half)

def merge(left, right):
    result = []
    left_index, right_index = 0, 0

    while left_index < len(left) and right_index < len(right):
        if left[left_index] < right[right_index]:
            result.append(left[left_index])
            left_index += 1
        else:
            result.append(right[right_index])
            right_index += 1

    result.extend(left[left_index:])
    result.extend(right[right_index:])
    return result

Now that's about merge sort. See you in the next one. Until then, Sayonara!

Doubly Linked Lists

Adheeban Manoharan — Mon, 23 Oct 2023 14:12:25 +0000

In the last blog post we saw the implementation of Singly Linked List. Now in this post, we'll have detailed a look at his brother the Doubly Linked List.

Understanding Doubly Linked Lists

Just like singly linked lists, a doubly linked list is a data structure that consists of a collection of nodes. Each node, in this case, is a separate object, but what makes it unique is that each node is linked not only to the next but also to the previous node. This two-way linkage allows for more efficient traversal in both directions.

In contrast to the singly linked list, which only points to the next element, a doubly linked list has two pointers in each node, one pointing to the next element and one pointing to the previous element.

Adding a Node

Adding a node in a doubly linked list is a bit more intricate than in a singly linked list. Suppose we have a node cur that we want to insert between the prev node and the next node. To achieve this, we must perform the following steps:

Set the reference field of the prev node to point to the cur node.
Set the reference field of the cur node to point to the next node.
Adjust the reference field of the next node to also point back to the cur node, completing the two-way linkage.

The insertion process in a doubly linked list still maintains its efficiency, and you can insert a new node in O(1) time complexity when you have references to the prev and next nodes.

Deleting a Node

Deleting a node in a doubly linked list, much like insertion, requires a bit more work due to the two-way linkage. Let's consider a node cur that we want to delete:

Find the cur node's previous node prev and next node next first.
Adjust the reference field of the prev node to point to the next node, breaking the linkage from the cur node.
Adjust the reference field of the next node to point back to the prev node.

In a doubly linked list, finding the next node using the reference field is simple. However, finding the previous node prev is equally efficient since we can traverse the list in both directions. The time complexity for deleting a node remains O(1).

Implementation with Python

Now, let's dive into the implementation of a doubly linked list in Python:



# ListNode class represents a node in the doubly linked list.
class ListNode:
    def __init__(self, val):
        self.val = val  # Value of the current node
        self.prev = None  # Reference to the previous node
        self.next = None  # Reference to the next node

# MyLinkedList class implements the doubly linked list with the specified methods.
class MyLinkedList:
    def __init__(self):
        self.head = None  # Reference to the head node (the first node)
        self.tail = None  # Reference to the tail node (the last node)
        self.size = 0     # Number of nodes in the linked list

    # Get the value of the node at the specified index in the linked list.
    def get(self, index):
        if index < 0 or index >= self.size:
            return -1  # Return -1 for an invalid index

        current = self.head
        for _ in range(index):
            current = current.next

        return current.val

    # Add a new node with the given value to the head of the linked list.
    def addAtHead(self, val):
        new_node = ListNode(val)
        if self.size == 0:
            self.head = self.tail = new_node
        else:
            new_node.next = self.head
            self.head.prev = new_node
            self.head = new_node
        self.size += 1

    # Append a new node with the given value to the tail of the linked list.
    def addAtTail(self, val):
        new_node = ListNode(val)
        if self.size == 0:
            self.head = self.tail = new_node
        else:
            new_node.prev = self.tail
            self.tail.next = new_node
            self.tail = new_node
        self.size += 1

    # Add a new node with the given value before the node at the specified index.
    def addAtIndex(self, index, val):
        if index < 0 or index > self.size:
            return  # Do nothing for an invalid index

        if index == 0:
            self.addAtHead(val)
        elif index == self.size:
            self.addAtTail(val)
        else:
            new_node = ListNode(val)
            current = self.head
            for _ in range(index - 1):
                current = current.next

            new_node.prev = current
            new_node.next = current.next
            current.next.prev = new_node
            current.next = new_node
            self.size += 1

    # Delete the node at the specified index in the linked list.
    def deleteAtIndex(self, index):
        if index < 0 or index >= self.size:
            return  # Do nothing for an invalid index

        if self.size == 1:
            self.head = self.tail = None
        elif index == 0:
            self.head = self.head.next
            self.head.prev = None
        elif index == self.size - 1:
            self.tail = self.tail.prev
            self.tail.next = None
        else:
            current = self.head
            for _ in range(index):
                current = current.next
            current.prev.next = current.next
            current.next.prev = current.prev

        self.size -= 1

Now that's about Doubly Linked Lists. See you in the next one. Until then, Sayonara!

Tortoise and Hare Algorithm

Adheeban Manoharan — Sun, 22 Oct 2023 14:24:17 +0000

We all would have heard about the tortoise and hare story right? Well, there's an algorithm inspired by this story.

What is this Algo?

Floyd's Cycle Detection Algorithm, often referred to as the "tortoise and hare" algorithm, is used to detect the presence of a cycle in a linked list. This algorithm is named after American computer scientist Robert W. Floyd, who described it in 1967. It's a simple yet efficient technique that works by using two pointers to traverse the linked list at different speeds.

Yes, it works like the story. There will be two pointers running simultaneously within the same linked list. If the linked list has a cycle (i.e if the tail is connected to another node) the two pointers will meet together at some point. If not, the pointers will never meet.

First of all, what do I mean by a cycle in a linked list? Refer the below images: image 1 is a linked list without cycle (i.e tail node is not connected to any other node) and image 2 is a linked list with cycle because the tail node (-4) is connected to another node (2) in the list. Floyd's algo is efficient in detecting cycles in the linked lists.

Implementation with python

Below is the implementation of Floyd's algo with python. The detect_cycle method takes in head (LinkNode object) and traverses through the list and checks if the Linked List has a cycle.



class ListNode:
    def __init__(self, val):
        self.val = val
        self.next = None

def detectCycle(head):
    # Initialize two pointers, slow and fast, both initially at the head of the linked list.
    slow = head
    fast = head

    # Step 1: Detect the cycle (Floyd's algorithm)
    while fast and fast.next:
        # Move the slow pointer one step at a time.
        slow = slow.next
        # Move the fast pointer two steps at a time.
        fast = fast.next.next

        # If the slow and fast pointers meet, it indicates the presence of a cycle.
        if slow == fast:
            return True  # Cycle detected

    # If the fast pointer reaches the end of the list (or becomes None), there is no cycle.
    return False  # No cycle found

See you in the next one. Until then, Sayonara!

Singly Linked Lists

Adheeban Manoharan — Sat, 21 Oct 2023 13:55:54 +0000

In this new blog series of mine, we'll be adventuring into the realm of DSA. The intent of this blog series is to explore the fundamentals of DSA while trying to maintain the KISS (keep it simple, stupid) principle. Hence, all the implementations throughout this series will be with python.

Now that you have the context, without further ado, let's dive in.
To start with one of the most basic data structures we'll be implementing the linked lists in this post.

What are Singly Linked Lists?

A linked list is a type of data structure in which each element is actually a separate object while all the objects are linked together by a reference field. If you try to envision it, it might look something like the below representation:

There are two types of linked lists:
1) Singly Linked List
2) Doubly Linked list
In this post, we'll be exploring the singly linked list specifically.

Each object is a node in a linked list and is connected to only one other node in a Singly linked list.

Adding a node

To add a node to a singly linked list, lets consider the node that we want to add as cur and initialize it. We want to insert the cur node between prev node and next node. To do this we have to:

Connect the reference field of prev node to the cur node
Connect the reference field of cur node to the next node

Unlike an array, we don’t have to move all elements past the inserted element. Therefore, you can insert a new node into a linked list in O(1) time complexity if you have a reference to prev, which is very efficient.

Deleting a node

To delete a node, lets consider the same example with cur, prev and next. cur is the node you want to delete.

Find the cur node's previous node prev and next node next first
Link prev node's reference field to the next node, eventually unlinking cur node in the process

Finding the next node using the reference field is easy. But to find the previous node prev, we have to traverse the linked list from the head node, hence the time complexity of deleting a node in a linked list is O(n) where n is the length of the list.

Implementation with Python

Here's the implementation of the singly linked list with python:

class LinkedNode:
    def __init__(self, val):
        self.val = val
        self.next = None


class SinglyLinkedList:

    def __init__(self):
        self.head = None
        self.length = 0

    def get_with_index(self, index):
        if index < 0 or index >= self.length:
            # Raise an error if the index is out of bounds
            raise IndexError("Index out of bounds")

        cur = self.head
        for _ in range(index):
            # Traverse the list to the specified index
            cur = cur.next

        return cur.val

    def add_to_head(self, val):
        new = LinkedNode(val)

        if not self.head:
            # If the list is empty, the new node becomes the head
            self.head = new
        else:
            # Otherwise, make the new node the new head and link it to the previous head
            new.next = self.head
            self.head = new
        self.length += 1

    def add_to_tail(self, val):
        new = LinkedNode(val)

        if self.length == 0:
            # If the list is empty, the new node becomes the head
            self.head = new
        else:
            cur = self.head
            while cur.next:
                # Traverse to the end of the list
                cur = cur.next
            cur.next = new
        self.length += 1

    def add_at_index(self, index, val):
        new = LinkedNode(val)

        if index < 0 or index > self.length:
            # Raise an error if the index is out of bounds
            raise IndexError("Index out of bounds")

        if index == 0:
            # If index is 0, the new node becomes the head
            self.head = new
            self.length+=1
            return

        cur = self.head
        for _ in range(index - 1):
            # Traverse to the node just before the specified index
            cur = cur.next

        new.next = cur.next
        cur.next = new
        self.length += 1

    def delete_at_index(self, index):
        if index < 0 or index >= self.length:
            # Raise an error if the index is out of bounds
            raise IndexError("Index out of bounds")

        if index == 0:
            # If index is 0, remove the head node
            self.head = self.head.next
            self.length -= 1

        cur = self.head
        for _ in range(index - 1):
            # Traverse to the node just before the specified index
            cur = cur.next

        cur.next = cur.next.next
        self.length -= 1

Now that's about singly linked lists, in the next blog post we'll have a look at the doubly linked lists. Until then, Sayonara!

LlamaHub: The one stop data solution for your LLMs🦙🏠

Adheeban Manoharan — Mon, 26 Jun 2023 10:21:00 +0000

In the 4th part of this blog series, we'll have a look at the most popular adapters available at llamahub.ai. LLaMahub is a library of data connectors that translate raw data from various sources into vector(s) aiding in ease of access and transformation by LLMs like ChatGPT and Google Bard.

This is an open-source project where the adapters/connectors that are created would slowly get integrated into projects like Langchain and Llama-index. If you are building a custom document bot for your firm or even for your personal use I think you would find this super useful.

Now, lets get started !

For the sake of simplicity in this walkthrough, We will access the adapters of llamahub through Llama-Index

Adapters Assemble 🔨⚡

First up here's how you could load any loader with Llama-Index:

from llama_index import download_loader

Loader = download_loader("<NAME_OF_THE_LOADER>")

the download_loader helper method will make sure to load the mentioned loader along with all the needed dependencies. To check the name of the loader that you'd want to use, visit this documentation. With that out of the way, lets have a look at some of the important data loaders in Llama-Index.

`SimpleDirectoryReader`:

This is a universal data connector available in the llama-index that should be able to handle pretty much all the file types you throw at it. Under the hood it uses various third party libraries like pypdf2 for PDFs, python-pptx for PPTs etc... The general idea behind this data connector is to read a directory of different file types in entirety or to read single files, doesn't matter what kind of file it is.

SimpleDirectoryReader = download_loader("SimpleDirectoryReader")
documents = SimpleDirectoryReader(input_dir='foo/').load_data()

`SimpleWebPageReader`:

This is the web equivalent of the SimpleDirectoryReader. This can read any webpage by using BeautifulSoup4.

SimpleWebPageReader = download_loader("SimpleWebPageReader")
documents = SimpleWebPageReader().load_data(urls=['https://foo.com/index'])

`YoutubeTranscriptReader`:

This loader will attempt to transcript any youtube video that you provide. It will first try to search for any available captions using the youtube-transcript-api and if the captions are not available, it will use OpenAI's whisper API to transcript the video.

YoutubeTranscriptReader = download_loader("YoutubeTranscriptReader")
documents = YoutubeTranscriptReader().load_data(ytlinks=['https://youtu.be/5MuIMqhT8DM'])

The above are some of the loaders that I think have a lot of use cases. But there are more and you can find the complete list here. If you want to load the documents into langchain's Document format, you can use <Loader>.load_langchain_documents() instead of load_data, it will output the documents in the langchain supported format.

Once you load the documents, it is a cake walk from there. You can visit my old blogs in this series on how you could use this as context for the ChatGPT API.

Now, that's about LlamaHub, See ya in the next one 😉

I built a tool that creates and posts AI content to social media automatically 🧌

Adheeban Manoharan — Mon, 19 Jun 2023 04:55:34 +0000

I have always been intrigued by the idea of automating a social media handle. With the advent of LLMs, generating content has never been easier.

I wanted to experiment a little in this space and thus created LLM Influencer, A bot powered by OpenAI's GPT-3 and DALL-E 2 generative AIs.

💡 How does it work?

LLM Influencer utilizes the power of AI to create thought-provoking daily content accompanied by visually appealing images and posts it to twitter on a daily basis. The type of content that gets generated are available as separate plug-and-play modules.

🚀 Features:

✅ Generates daily motivational content for Twitter and other social media platforms.
✅ Offers easy plug-and-play modules for different types of content generation.
✅ Leverages ChatGPT and DALL-E 2 to provide creative and unique outputs.
✅ Includes built-in scheduling for automatic posting at your desired time every day.
✅ Provides email notifications to keep you informed in case of any job failures.

🧩 Currently Available Modules:

1️⃣ Quoter: Delivers motivational quotes paired with captivating images, designed to inspire and uplift you each day.
2️⃣ Tweet Storm: Explores various aspects of life through engaging tweet storms, offering valuable insights and perspectives.

I have been posting the generated content to my own twitter handle for while now, I have added some of the posts here.

I'll add the links for the repository and the twitter handle where I post the generated content.

Life is a journey between the bright marble of our dreams and the muddy realities of our days, but both are equally important. ✨🧱💭 pic.twitter.com/yjPHfahtAr
— Adheeban Manoharan (@iamadhee_) June 16, 2023

iamadhee / llm-influencer

🎙 AI-driven social media content generation

LLM Influencer

Your AI-powered content generator

About

This project is an experimentation with AI, that aims on AI-Driven content generation. The aim is to be able to generate various types of content, such as podcasts, articles, and more using AI.

Installation

Clone the repository
Install the required dependencies using poetry install
- This project uses poetry for dependency management, so make sure you have it installed. If not, you can install it using pip install poetry

Usage

Create a .env file in the root directory of the project, there is an example .env.example file provided. Fill in the required environment variables.
If you want, you can play around with the config.toml file to change the configurations of the content. Each type of content has its own section in the config file.
To create content of your choice, run the following command
```
python generate.py <type> <description>
```
- <type> is…

View on GitHub

Kindly have a look and share your valuable thoughts.

Combine LangChain 🦜🔗 & Llama-Index 🦙

Adheeban Manoharan — Mon, 22 May 2023 03:56:55 +0000

In the last post of this blog series, we saw how we could use LangChain's memory classes for building a chatbot-like interface with OpenAI's GPT-3 API. In this blog, we are basically gonna use these memory classes to keep track of our previous conversations with the LLM all the while making it answer our queries regarding the contents of the document we feed to it.

We'll try to keep the dependencies on LlamaIndex to a bare minimum, because Langchain almost covers all the aspects of utilities that we would need to build this. We'll only be using the adapters available in the LlamaIndex library, because they perform better than the Langchain ones.

Now before we dive in, let me set the context for you as to what we are trying to achieve. We are trying to build an intelligent system that reads the contents of the document and should be able to answer any queries from that content. This intelligent should also be able to keep track of our past conversations and if at all any question pointing one of the previous queries, it should be able to answer that.

Remember when I told you in the last blog about completion tokens and context tokens? The maximum context window for GPT-3 is 4096 tokens. We should be able to squeeze all our context, memory, prompt and response all within the same window.

Something like this ->

With that out of the way, let's hustle !

Memory meets Custom Knowledge 🧠📔

Let's start right away with the code:



from langchain.chat_models import ChatOpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferWindowMemory
from langchain.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from llama_index import download_loader
import os

os.environ["OPENAI_API_KEY"] = 'Your API Key Here'

file_path = input('Enter the youtube link: ')

YoutubeTranscriptReader = download_loader("YoutubeTranscriptReader")
documents = YoutubeTranscriptReader().load_data(ytlinks=[file_path])

documents = [doc.to_langchain_format() for doc in documents]
text_splitter = RecursiveCharacterTextSplitter()
documents = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()
retriever = FAISS.from_documents(documents, embeddings).as_retriever(k=4)

llm=ChatOpenAI(temperature=1, model_name="gpt-3.5-turbo", max_tokens=2048)

memory = ConversationBufferWindowMemory(
                                    memory_key='chat_history',
                                    return_messages=True,
                                    k=6
                                  )

conversation = ConversationalRetrievalChain.from_llm(
    llm=llm, 
    retriever=retriever,
    # verbose=True, 
    memory=memory,
    max_tokens_limit=1536  
)

def chatbot(pt):
    res = conversation({'question': pt})['answer']
    return res

if __name__=='__main__':
    while True:
        print('########################################\n')
        pt = input('ASK: ')
        if pt.lower()=='end':
            break
        response = chatbot(pt)
        print('\n----------------------------------------\n')
        print('ChatGPT says: \n')
        print(response, '\n')

You know the drill, just copy the code and paste it in a file and while running the file, install the dependencies if anything is prompted.

These are some important points, that I thought I should highlight in the above code:

FAISS is a library built my MetaAI for semantic search, that helps us with the similarity search within the context.

ConversationBufferWindowMemory is the memory that we've used, this type of memory keeps track of the last k conversations between the user and the LLM. I have given k=6.

RecursiveCharacterTextSplitter is an utility from LangChain that splits the context into chunks.

ConversationalRetrievalChain is a type of chain that aids in a conversational chatbot-like interface while also keeping the document context and memory intact.

That should give you an idea. If you have any more questions about the code, feel free to comment below.

Now, for a change, I have used the YoutubeTranscriptReader from the llama-index, this reads the transcript of the youtube link that you provide and builds the context out of it for the LLM. In the last blog we had used SimpleDirectoryReader.

Here's what I tried with Matt Walker's talk on Sleep is your superpower:



Enter the youtube link: https://youtu.be/5MuIMqhT8DM


########################################

ASK: list down the key points in the speech

----------------------------------------

ChatGPT says: 

1. Regularity in sleep timing is important to anchor sleep and improve its quantity and quality.
2. Keeping the bedroom cool aids in initiating and maintaining sleep.
3. Sleep is a non-negotiable biological necessity for human beings' health and well-being.
4. Sleep loss is a silent epidemic in industrialized nations and a significant public health challenge.
5. Sleep debt cannot be accumulated and repaid at a later point in time.
6. Getting out of bed when unable to sleep for too long establishes the association between bed and sleep.
7. The importance of prioritizing sleep for overall health and well-being. 

########################################

ASK: now convert that into a song for a 5 year old

----------------------------------------

ChatGPT says: 

Sure, here's a simple song about the importance of sleep:

Verse 1:
We all need to rest our heads
And go to sleep in our cozy beds
Regular sleep is the key
To feeling good and healthy

Chorus:
Sleep, sleep, the magic elixir
It helps us grow and gives us vigor
Don't stay up too late at night
Or you won't feel so bright

Verse 2:
Keep your room nice and cool
And your sleep will be extra smooth
And if you can't fall asleep
Get out of bed and don't make a peep

Chorus:
Sleep, sleep, the magic elixir
It helps us grow and gives us vigor
Don't stay up too late at night
Or you won't feel so bright

Verse 3:
Sleep is important, it's true
For our brains and our bodies too
We won't feel so good without it
So let's all just go to bed

Chorus:
Sleep, sleep, the magic elixir
It helps us grow and gives us vigor
Don't stay up too late at night
Or you won't feel so bright

Outro:
So let's all get a good night's sleep
And tomorrow, we'll feel so sweet
Sleep tight and have nice dreams
And wake up feeling like a new team! 

########################################

Notice how the LLM is able to sense that I'm referring the previous conversation in the second question and converts that into a song? Exactly what we were trying to achieve. Do try it out with your own Youtube video.

Now, that's about LangChain and Llama-Index. In the next blog of this series, we'll have a look at all the awesome adapters that are available in the Llama-Hub. Cheers, see ya in the next one 😉

Make ChatGPT keep track of your past conversations 💬

Adheeban Manoharan — Thu, 11 May 2023 02:43:51 +0000

In our previous blog, we fed ChatGPT with the custom knowledge from documents and made it answer our queries using Llama-Index. One problem with the previous implementation is that the chatbot wouldn't be able to answer any of our questions addressing the previous questions/prompts. Here's what I'm talking about:

ASK: You are an AI assistant, specialized in programming.                                    

----------------------------------------

ChatGPT says: 

Hello! I'm here to assist you with anything related to
programming. What can I help you with today? 

########################################

ASK: what did I tell you earlier?                                  

----------------------------------------

ChatGPT says: 

I'm sorry, but as an AI language model, I don't have access to
any context or information about our previous interactions
unless you provide it to me. Could you please clarify what you
asked me earlier? I'll do my best to assist you.

So, lets set the custom knowledge base aside for a short-while and focus on this memory problem in this blog. We will be using the Langchain framework that offers various kinds of memory classes that will help us to build a chatbot system that keeps track of the past conversations we had with it.

If you ever visited chat.openai.com, you would know that OpenAI maintains histories of chats, ChatGPT had with the user in seperate containers called sessions. Basically, we'll be trying to mimic these sessions using langchain.

Now, that should give you an idea. Before we start, here are some more things you should know.

OpenAI's ChatGPT has a token limit of 4096 (1 token =~ 4 characters)
This limit includes both your prompt and the response (completion tokens) that is returned from ChatGPT
Anything that has to be generated post this token limit will be ignored abruptly without even a single warning
So, ideally we should be keeping our prompt token limit with 2048 (half the limit) and leave the rest for ChatGPT's completion. This buffer changes based on your use case.

Well, why am I blabbering all this now? I'll explain in a while. Now, without further ado, let's get started with langchain.

Enter Langchain 🦜🔗

So, what's our goal? We don't have to worry about feeding document context into the LLM for now. We rather have to build a simple chatbot that remembers our past conversations. With langchain, it is as easy as 1, 2, 3 !

from langchain.chat_models import ChatOpenAI
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory
import os

os.environ["OPENAI_API_KEY"] = 'Your API Key Here'

llm=ChatOpenAI(temperature=0.7, model_name="gpt-3.5-turbo", max_tokens=512)

conversation = ConversationChain(
    llm=llm, 
    # verbose=True, 
    memory=ConversationBufferMemory()
)

def chatbot(pt):
    res = conversation.predict(input=pt)
    return res

if __name__=='__main__':
    while True:
        print('########################################\n')
        pt = input('ASK: ')
        if pt.lower()=='end':
            break
        response = chatbot(pt)
        print('\n----------------------------------------\n')
        print('ChatGPT says: \n')
        print(response, '\n')

That's all you gotta do. Now you know the drill, copy the above snippet and name the file whatever you want and run it with python3 <filename>.py>

Here's the conversation I had:

ASK: Hello, who are you?

----------------------------------------

ChatGPT says: 

Hello! I am an artificial intelligence program designed to
interact with humans and assist with various tasks. How can I
help you today? 

########################################

ASK: Who was martin luther king?

----------------------------------------

ChatGPT says: 

Martin Luther King Jr. was an American Baptist minister and
activist who became the most visible spokesperson and leader
in the civil rights movement from 1954 until his assassination
in 1968.

########################################

ASK: what was his profession?

----------------------------------------

ChatGPT says: 

Martin Luther King Jr. was a Baptist minister and activist. He
became a prominent leader in the civil rights movement and
advocated for nonviolent civil disobedience to advance civil
rights for African Americans. 

########################################

ASK: Okay, what was my first question to you?

----------------------------------------

ChatGPT says: 

Your first question to me was "Hello, who are you?"

As you could clearly see, now the LLM is able to remember my past conversations. Even the first question I asked. In the code setting the verbose argument to True during the initialization of ConversationChain class is upto you. Setting the option to true will give you a glimpse of what's happening in the backend as we sail through the conversation with the LLM.

Our conversations will be stored in the Memory class of langchain, if you want to store the conversation for later retrieval, you can do so by pickling the chain.messages class and save it in a file. Later when you need to load your history, you just have to overwrite your chain's messages class with the pickled string, like how I have done below:

qa = ConversationChain(
            llm=llm, memory=memory)

with open(f'sessions/{self.history_key}','rb') as f:
     user_ses = pickle.loads(f.read())
     qa.memory = user_ses

Enough with the code! Now, What's happening underneath?

In the example code snippet above, I have used the ConversationBufferMemory memory class, this is the most simplest implementation of memory in langchain. There are other memory classes too in langchain, each with its own perk. Before I get into that, Here's what happens when you have a conversation with a chain with memory in langchain:

When you send in the first question or prompt to the LLM, it returns a response. Cool.
Now, when you send the second prompt, langchain also sends the previous conversation the user had with the LLM as context for the LLM, so that the LLM would be able to answer any questions addressing the previous questions.
This happens with every conversation. Previous conversations will be linked as context to the LLM with every hit to the LLM's API.

A very simple engineering problem handled effortlessly with langchain. Now, Remember when I told you about the token limit in ChatGPT? It plays a crucial role now. Whatever conversation you had previously with the LLM will be linked with your original prompt and sent to the API. As the depth of the conversation increases, the percentage of tokens the history uses in the whole 4096 token limit increases rapidly. Which will eventually leave no room for the LLM's response to the current question.

Langchain knows about this problem very well, that's why they have different memory classes in their memory module that suits your unique use case, I'll give short note on a impressive few.

ConversationSummaryMemory - This type of memory creates a summary of the conversation over time. This can be useful for condensing information from the conversation over time resulting in a lesser usage of tokens within the 4096 limit.

ConversationKGMemory - This type of memory uses a knowledge graph to recreate memory. It identifies the entities and maps out the relation between each of those entities.

These memories use the LLM itself to identify entities, to summarize the previous conversations etc. This should be taken into account as this is an extra hit to the API before every conversation. There are a lot more memory classes in langchain, Be sure you give langchain's documentation a read here

Now, that's about Langchain's memory classes. In the next blog of this series, we'll try to combine the vectore store from the first blog with the memory classes in this blog. LlamaIndex + Langchain 🔥. See ya in the next one 😉

Chat with your documents using ChatGPT 🦾

Adheeban Manoharan — Sat, 06 May 2023 09:04:23 +0000

Ever since OpenAI announced their language model ChatGPT, it has been making headlines in the AI world on a daily basis. ChatGPT is being used as the foundation for countless new tools and applications, ranging from customer service chatbots to creative writing assistants. With its ability to generate high-quality, human-like responses to complex prompts, ChatGPT has quickly become a game-changing technology.

The applications of LLMs like ChatGPT are virtually limitless. Our imagination would be the only barrier. In this blog series though, We'll be focusing on how we can make ChatGPT, or any LLM for that matter to answer our queries with the context of the custom knowledge from the documents we feed to it. We'll start off with a simpler implementation using Llama-index that reads almost all types of basic document formats and returns the response to your queries based upon it. As we progress, we'll be using Langchain to build a full fledged chatbot framework that reads the content from almost any link or document that you give to it and answers your queries accordingly. Langchain is a great framework for developing applications powered by language models. We'll be building a web app using these frameworks. Exciting times ahead !

Let me lay the prerequisites first:

You will need a working OpenAI key, because we'll be using the GPT-3 model underneath. If you don't have one, here's how to get an OpenAI API Key
You need to have python>=3.6 installed in your machine

That's about everything you'd need. Rest of the things we'll take care of, as we sail through. Now without further ado, let's dive right in.

Behold, the power of Llama-Index 🦙

As I said earlier, We are gonna use llama_index for this tutorial. We are not building anything fancy as of now. We'll not be building any UI. The sole purpose of this is to give an understanding of how llama_index works underneath. Below is the implementation using llama_index and langchain. llama_index uses langchain models under the hood.

from gpt_index import download_loader, SimpleDirectoryReader, GPTSimpleVectorIndex, LLMPredictor, PromptHelper
from langchain.chat_models import ChatOpenAI
import os

os.environ["OPENAI_API_KEY"] = 'Your API Key Here'

file_path = input('Enter the path of the file/doc: ')

def build_index(file_path):
    max_input_size = 4096
    num_outputs = 512
    max_chunk_overlap = 20
    chunk_size_limit = 256

    prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

    llm_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0.7, model_name="gpt-3.5-turbo", max_tokens=num_outputs))

    download_loader('SimpleDirectoryReader')
    documents = SimpleDirectoryReader(input_files=[file_path]).load_data()
    index = GPTSimpleVectorIndex(documents, llm_predictor=llm_predictor, prompt_helper=prompt_helper)
    return index


index = build_index(file_path=file_path)

def chatbot(prompt):
    return index.query(prompt, response_mode="compact")

while True:
    print('########################################')
    pt = input('ASK: ')
    if pt.lower()=='end':
        break
    response = chatbot(pt)
    print('----------------------------------------')
    print('ChatGPT says: ')
    print(response)

Copy the above code in entirety and paste it in a file and name it whatever you want. I'm naming it main.py. Replace the API key placeholder in the code, with your own OpenAI API key and you are done.

First up, run the below command to install the libraries that we'd need:

pip3 install gpt_index==0.4.24 && pip3 install langchain==0.0.142

You can run the code by running this command python3 main.py at the location of this file, when prompted give the path of your context file to the program. I'm using Martin Luther King's "I have a dream" speech transcript PDF file.

➜  python3 main.py
Enter the path of the file/doc: blogs/docbot/martin.pdf

You can give any type of file format. Depending on the file
size and your machine's power, it'll take some time to read and convert your file contents into vectors. In my case, it takes around 10 seconds to process a 30MB document.

Once it is done, you can start asking questions to your document using ChatGPT. Below are some of the example prompts and responses.

ASK: 

what is this document about?

----------------------------------------

ChatGPT says: 

The document is about a speech or written piece that discusses
the promise of equal rights for all Americans, particularly
black Americans, as outlined in the Constitution and Declaration of Independence.
It addresses the fact that this promise has not been fully
realized and that there is a shameful condition that needs to be dramatized.

ASK: 

summarize the speech

----------------------------------------

ChatGPT says: 

The speech urges America to remember the importance of freedom and justice, and emphasizes the urgent
need for action in the face of racial injustice. The speaker encourages the audience to rise above 
segregation and work towards brotherhood. The overall message is that now is the time to make real the
promises of democracy.

That's it. As simple as that. 35 lines of code. Llama-Index and other tools like it have made it so easy and user friendly to leverage the full power of LLMs.

Okay that's cool, but what really happens underneath?

Llama-index and langchain have made the whole process very seamless, which otherwise would've been a really cumbersome task. So, here's what happens:

When you give llama_index the document, it uses one of the adapters that actually suits for the file type from the collection of pre-built adapters from the Llama-Hub and parses the contents of the file.
Once the parsing is done, llama_index converts the whole content into chunks of vectors.
When you put up a question to ChatGPT, llama_index takes in your question retrieves the chunks of vectors from the parsed file that are relevant to your prompt using similarity search
Once it retrieves the relevant chunks, llama_index overrides your original prompt by adding what it retrieved as the context to the model.
With the original question and the context it just has been provided with, ChatGPT should be able to understand your question.

And voila! you'll get a relevant response from ChatGPT. This is how llama_index makes LLMs understand custom knowledge. There's more to this and new features are getting added to llama_index everyday. Make sure you explore more of what it could do.

Now, that's about llama_index. In the next blog of this series. We'll see how to use langchain to build a more robust chatbot framework that keeps track of the previous conversations it had with the user. See ya in the next one 😉

Know when to automate things (and when not to) 👾

Adheeban Manoharan — Sat, 24 Dec 2022 11:52:21 +0000

According to a 2020 study on "shift in number working hours from 1870 to 2020", It was found that an average worker in a rich country today really does work many fewer hours than the average worker 50 years ago. Although, this steady decline in the number of working hours could have been the direct cause of multiple events like the end of the great depression, major global wars etc.., one major and inevitable reason is the rapid technological advancement that we've been experiencing in almost all the fields in the last few decades.

Every task in today's world is either already automated or people are trying to automate it. A McKinsey Report estimates that automating tasks can increase annual productivity by about 1.4% in industries. Yes, Automation plays out really well in several of today's fields.

Now, how about in the tech industry? Well, this is where modern day automation was born. As techies, we all have this constant urge to automate things. But don't you think that some of us overdo this a bit? Well, atleast I do. I try to automate almost everything that I do in a day, if I see the opportunity to do so. That too with a versatile and powerful scripting language like python, not even a second's thought. Adding to that, the thrill that gives us as developers while facing the challenges the task poses while trying to automate.

As cool as that sounds, while trying to automate a particular task, I sometimes end up spending more time on the automation in itself than it would've required to perform that original task, which technically defeats the whole purpose of automating in the first place. In short, I ended up wasting time trying to find the easier route.

So, that task clearly was not a suitable candidate for automating. Right, so when is a task automation worth your time?

Repetitive, manual tasks 🔁

I heard this from a colleague whose work I admire a lot, ' If you are doing something more than twice, automate it ! ', He said. Tasks that need to be done frequently or on a set schedule are perfect for automation because the benefits are immediate. You'll spend less time each day on repetitive tasks that can quickly become annoying or irritating. Even it is a task you ought to perform once in a year, go for it. Time intervals don't matter, as long as it is repetitive and consumes a lot of your precious time and energy.

Boring and complex tasks 🥱

If a task doesn't excite you and if it is quite complex to perform, automate it. Doesn't matter if it takes quite some time to automate, because doing something that you don't like brings your whole spirit down, which in turn dials your excitement down in the other tasks resulting in the decline of your overall productivity. Also, if that task is complex, but we'll have to do it manually, it is by nature prone to many human errors. In this case, an automation framework to perform this particular task goes a long way in preventing these kind of human errors, provided that you get the framework right.

Distractive tasks 😶‍🌫️

When you start your day, you might have a lot of tasks on your plate and it would be very tempting to get started with the most easiest one, rather than the important ones. These kind of tasks will easily sidetrack you from the very important and impactful tasks. But, these are the kind of tasks that cannot be skipped, no matter how insignificant they are. If time allows you, you should automate these kind of tasks, to stay focused on your main tasks.

If a task checks all the boxes I have listed above, It is already begging to be automated, go for it. Don't spend your precious time on the insignificant ones //

Here's a thumb rule:

If you hate it, automate it !

Good luck on your automation journey, I would love to hear your thoughts in this, Please do comment. Cheers! See you in the next post.

References:

Charlie Giattino, Esteban Ortiz-Ospina and Max Roser (2020) - "Working Hours". Published online at OurWorldInData.org. Retrieved from: 'https://ourworldindata.org/working-hours'
Mckinsey Global Institute (2017) - A FUTURE THAT WORKS: AUTOMATION, EMPLOYMENT, AND PRODUCTIVITY

Forem: Adheeban Manoharan

🗃️ Lessons Learned from Migrating Huge BigQuery Datasets Across Regions

Prelude

Google's Limitations

Data Size

Zero Downtime

How it actually went down

1) Export Table Data as Parquet:

2) Extract Table Schema and Partition Details:

3) Transfer Data to the EU bucket:

4) Create a Temporary Dataset:

5) Enable the Migration Flag:

6) Delete the US Dataset:

7) Create a Permanent EU Dataset:

8) Turn off the Migration Flag:

9) Export Data from the Temporary Dataset:

10) Import Data into the Permanent EU Dataset:

11) Cleanup:

Conclusion

The Battle of PR-Agents⚡

1. Command Arsenal: A Multifaceted Symphony

CodiumAI PR-Agent:

GitHub Copilot:

2. Platform Universality: Bridging Git Divides

CodiumAI PR-Agent:

GitHub Copilot:

3. Open-Source Dynamics: A Community-Driven Ethos

CodiumAI PR-Agent:

GitHub Copilot:

4. Integration Landscape: A Seamless Experience

CodiumAI PR-Agent:

GitHub Copilot:

5. Time Efficiency: The Quintessence of PR Management

In Conclusion: CodiumAI's PR-Agent Takes the Lead

The Problem

Merge Sort

What is Merge Sort?

How Merge Sort Works

Time and Space Complexity

Implementation with Python

Doubly Linked Lists

Understanding Doubly Linked Lists

Adding a Node

Deleting a Node

Implementation with Python

Tortoise and Hare Algorithm

What is this Algo?

Implementation with python

Singly Linked Lists

What are Singly Linked Lists?

Adding a node

Deleting a node

Implementation with Python

LlamaHub: The one stop data solution for your LLMs🦙🏠

Adapters Assemble 🔨⚡

SimpleDirectoryReader:

SimpleWebPageReader:

YoutubeTranscriptReader:

I built a tool that creates and posts AI content to social media automatically 🧌

💡 How does it work?

🚀 Features:

🧩 Currently Available Modules:

iamadhee / llm-influencer

🎙 AI-driven social media content generation

LLM Influencer

About

Installation

Usage

Combine LangChain 🦜🔗 & Llama-Index 🦙

Memory meets Custom Knowledge 🧠📔

Make ChatGPT keep track of your past conversations 💬

Enter Langchain 🦜🔗

Enough with the code! Now, What's happening underneath?

Chat with your documents using ChatGPT 🦾

Let me lay the prerequisites first:

Behold, the power of Llama-Index 🦙

Okay that's cool, but what really happens underneath?

Know when to automate things (and when not to) 👾

Repetitive, manual tasks 🔁

Boring and complex tasks 🥱

`SimpleDirectoryReader`:

`SimpleWebPageReader`:

`YoutubeTranscriptReader`: