Forem: Muhammad Zeeshan

Apache AGE:Code Style Guide

Muhammad Zeeshan — Sun, 24 Sep 2023 11:24:15 +0000

In this blog we will look at the coding style guide to make a pull request in Apache AGE repository. In our previous blog we explored that how we can create a PR in apache AGE repository. If you want to know about doing a Pull request to Apache AGE you can checkout this blog first.

Indentation

In AGE's official coding style guide they mentioned about using 4 spaces per indentation level instead of using a tab for indentation. Use the same method to indent code even working with switch statement.

switch (suffix) {
    case 'G':
    case 'g':
        mem <<= 30;
        break;
    case 'M':
    case 'm':
        mem <<= 20;
        break;
    case 'K':
    case 'k':
        mem <<= 10;
        // fall through
    default:
        break;
}

In the code above we have used 4 spaces instead of a single tab.

Breaking long lines and Strings

Line length is 79 columns. So, if the length of code line exceeds 79 columns, move the string to next line. And if there is a case in which the String is exceeding 79 columns we don't need to break the string.

Placing Braces and Spaces

All opening or closing braces are to be placed on a single line solely. Example

void function(int x)
{
    body of function
}

if(x is true)
{
do task a
}
else if(y is true)
{
do task b
}
else
{
do task c
}

The main thing to keep in mind in the above code is braces, each opening or closing brace is placed on a single line and no code on that line.
Note
If all the bodies of if/else statements contains a single line, omit braces. Example

if(x is true)
    do task a
if(y is true)
    do task b
else
    do task c

Naming

Use the underscore name convention for all variables, functions, structs, and enums.

Commenting

For multi-line comments, use C style multi-line commments. And for single line comments use the style format as shown below.

/*
 * This function
 * does x
 */
void f(void)
{
    // This is to check...
    if (y is true)
        we do b
/*
 * We need to do this here
 * because of ...
 */
for (;;)

}

NewLines

For newlines, only \n is allowed, not \r\n and \r.

Note

For a list of all the coding style guidelines, visit the clang-format.5 in the official AGE git repository.

Reference

https://age.apache.org/contribution/guide/#code-style-guilde

Creating a pull request to any open source project like Apache AGE

Muhammad Zeeshan — Fri, 15 Sep 2023 16:12:33 +0000

I was working on an open source project Apache AGE. I have done so many PR's in my local repo project. But doing a Pull request to an open source project is a bit different from doing it in your local repository.
For this i assumed that you have github setup on system if no you can follow this blog. So, let's get started.

Forking the Repository

It's the very first step you need to do before creating a pull request. Forking a github repo will create a copy of the project that you forked under your account. So, to fork AGE repository you need to go to AGE's source code. Then fork the repo and it will create the copy under your account.

Clone

Next step is clonning the repository. This step will clone the AGE's source code into your working environment. And to clone the code you can use the command.

git clone path_to_the_repo

replace the path_to_the_repo with the path of your copy of AGE's source code. This will make sure that the code is now available on your computer.

Branch Creation

Create a new branch. In this branch you will complete your task and will add your task as a branch in AGE repository. To create a branch you can run the command

git checkout -b branch_name

Replace the branch_name with the name of the branch that you want to set. This command will create a new branch and checkout to that branch as well. If the branch already exists it simply checkout to that and if not it will create a branch with that name and then checkout to that. If you are using vs code you can also create a branch by using git/github extension of vscode.

Add and commit your task

Now everything is setup. No you need to do your changes that you want to add or remove code in the AGE's repo. Now it's time to complete your task. After completing your task you need to push it to your repository. For that you need to run the following commands.

git add. 
git commit -m "Your commit message"

These two commands will add your changes to stages and then add the commit message. You can replace the your commit message with the message you want to add. Let's say you have changed a data types of some variables you can replace it with changed/updated the data types and so.

Now after this push the changes to github.
For this use the commad.

git push

If it's not working or giving some upstream warning. You can also use this command.

git push --set-upstream origin branch_name

This will push all the changes to your copy of code on the github. Now the next and final step is creating a Pull Request to the AGE's source code.

Pull request

For this you need to go to your local copy of AGE's code. Click on the New pull request button. This will redirect to a page where you have to set the base repository and then its base branch. Make sure these both are the one that you need to create a PR to. After that there will be an option of compare branch. Set that to the branch that you just created.
Then add title and description of the PR. These could be something that explains what you have done in your code and what problem you solved. After adding these click on create pull request.

Note

To get you work merged you need to follow the community guidlines to create a pull request. For that you can check this https://github.com/apache/age.

References

https://age.apache.org/
https://dev.to/rrrokhtar/guide-for-creating-pull-requests-on-apache-age-1nkm

Query Planning: How citus plans query

Muhammad Zeeshan — Thu, 14 Sep 2023 13:04:10 +0000

Distributed query planner

When a SQL query references a Citus table during execution, the distributed query planner produces a PostgreSQL query plan that contains a Custom Scan node, which contains the distributed query plan.

A distributed query plan constists of the following tasks.

Queries on shards to run on the worker nodes.
A set of subplans whose result need to be broadcast or re-partitioned.
It has four main query planning strategies to scale and to distribute the queries.

A: Fast Path Planner

It handles and work for simple queries like create read, update, and delete i.e, CRUD queries on a single table with a single distributed column value. It works by extracting the distributed column value directly from a filter in the query and determines the value of shard that matched with that value.

Planner then rewrites the name of the table to the shard name that matches the value in the column and construct the query to run on the worker node, this task is done with the minimal usage of the CPU overhead.

B:Router Planner

It handles arbitrary complex queries, that can be scoped to one set of co-located shards. It works by checking and infering whether all distributed tables have the same distributed column filters or not. If so, it then rewrites the table name in the query to the names of the co-located shards that matches the distributed column value. It supports all SQL features that PostgreSQL supports since it will simply delegate the full query to another PostgreSQL server.

C: Logical Push Planner

It works by detecting whether the join tree( queries that have some joins statements) can be fully pushed down. This requires that all the distributed tables have co-located joins between them and subqueries do not require a global merger step. In this a GROUP BY must include the distribution column. If so, the distribution planner can be largely agnostic to the SQL constructs being used woth the join tree, because they can be fully delegated to the worker nodes, and so our distribution query planner can work in planer.

D: Logical Join Order Planner

It determines the optimal execution order for join tree which involves joins that are not co-located. Firsly, it evaluates all possible join order between distributed tables and subqueries using co-located joins, broadcast joins and re-partition joins. And after evaluation it chooses the order that minimizes the network traffic. From the mentioned joins broadcst and re-partition joins results in subplans that includes the filter and projections pushed into the subplan.

These above are the main four plans that Citus cinsiders while query execution. For each query, Citues iterates over these four planners, from lowest to highest overhead. Then it uses the planner that can plan and executes that query with minimum overhead.

Conclusion

So, to sum up the query planner, Citus Distributed query planner takes a SQL query and then plans it for distributed execution. Then after performing some optimizations planner breaks the query into two parts. The coordinatoe query which runs on the coordinator and worker query fragments which runs on individual shards on the worker. Planner then assign these fragments to the workers keeping in view the best utilization of its resources. After this query has been passed to the Distributed query executor for execution.

Step by Step Guide to Install Citus: Linux

Muhammad Zeeshan — Tue, 05 Sep 2023 16:26:00 +0000

This blog is the continuation of last blog. Where we explored about citus. In this blog we will look at the two different ways to install citus.

Method 1

Packaging repo

If you already have PostgreSQL installed its better to install the citus using this method.Run the following commands.
Install packages on ubuntu.

curl https://install.citusdata.com/community/deb.sh > add-citus-repo.sh
sudo bash add-citus-repo.sh
sudo apt-get -y install postgresql-15-citus-12.0

This will install the citus and the compatible postgres version. Citus version 12 and Postgres version 15.

add Citus to your local PostgreSQL database, add the following to postgresql.conf:

shared_preload_libraries = 'citus'

Method 2: Install using source code.

Step 1

create the required directories. Open the terminal and run the commands.

mkdir citus
cd citus
mkdir pg
cd pg

Step 2

Install required dependencies.

sudo apt-get install build-essential libreadline-dev zlib1g-dev flex bison

Step 3

Download and install postgres. In method one it automatically installed the postgres but here we need to download and install it manually.

Download

wget https://ftp.postgresql.org/pub/source/v15.3/postgresql-15.3.tar.gz && tar -xvf postgresql-15.3.tar.gz && rm -f postgresql-15.3.tar.gz

Install

cd postgresql-15.3

# configure by setting flags
./configure --enable-debug --enable-cassert --prefix=$(pwd) CFLAGS="-ggdb -Og -fno-omit-frame-pointer"

# now install
make install

# go back to main citus directory
cd ../../

Step 4

Now after installing postgres we need to install citus. Before that we need to install the required dependencies.

sudo apt-get install -y postgresql-server-dev-15 postgresql-15 \
                        autoconf flex git libcurl4-gnutls-dev 
libicu-dev \
                        libkrb5-dev liblz4-dev libpam0g-dev libreadline-dev \
                        libselinux1-dev libssl-dev libxslt1-dev libzstd-dev \
                        make uuid-dev

Step 5

Clone citus using the github repo.

git clone https://github.com/citusdata/citus.git
cd citus

Step 6

Install and configure citus

PG_CONFIG=/home/username/citus/pg/postgresql-15.3/bin/pg_config ./configure
make
sudo make install

After running this config comamnd Here you can have an error that some dependencies are missing. You can run the command with --without flag or can download those dependencies.

You have successfully installed citus and postgres from the source code or using package. To load you again need to add the above prelaod command in postgresql.conf

Citus Introduction

Muhammad Zeeshan — Thu, 31 Aug 2023 10:06:23 +0000

In the evolving landscape of data management, the challenges of horizontal scaling are becoming crucial. Citus is an open-source extension for PostgreSQL that gives it the power of scalability with distrubuted tables. It takes the strength of PostgreSQL's data integrity and combines it with the scalability and performance of a distributed database management. In this blog we will explore about citus, about how it handles large-scale data.

Introduction

Traditionally relational databases like SQL and PostgreSQL maintains data integrity and along with that also provides powerful querying capability. PostgreSQL is one of the mist popular open source database management system. One of the main issue or the limitation we can say is dealing with extremely large data sets which requires high concurrency demand. Citus addresses these challenges by providing a solution of architecture that spread all the data across different or multiple nodes. Which enables better hardware resource management and improvement in performance.

Citus Architecture

Citus is designed in such a way that a in a citus cluster, all servers run PostgreSQL having citus extension along with other number of extensions installed. Citus mainly uses the PostgreSQL's extension API's to change the behaviour of database in two ways.

It replicates database objects, which mainly includes custom types and function, to all servers.
It add two new table types that are used to take the advantages of the additional servers.

It basically uses the sharding mechanism to deal with scalability. Which means it divide the large datasets into different managable parts called chunks or shards and then divides these chunks on different nodes. It manages the query very intelligently by sending those to the relevant node and then collection result from those.

Key Features

Horizontal Scalability:

It allows us to divide or distribute our data across multiple nodes, allowing us to scale out by adding more computer or machines to our cluster instead of vertically scaling by increasing the power of already machines we have to manage our workload.

Parallel Processing

As our data is distributed across multiple nodes, so citus can execute queries in parallel which in turns combines the query processing power from all nodes and to speed up performance. It can speed up query porcessing by 20x to 300x or more times with the help of this parallelism.

High Throughput

It is well suited and optimized for large-scale data application. As it uses different nodes to process high volume of data and queries which helps it to prevent different bottlenecks and ensure highly efficient utilzation of resources.

Multi-Tenancy

Citus can be used to build multi-tenant applications, in which data is stored across separate distrubuted tables.

Ease of use

Because of its compatibility with POstgreSQL, you can use familiar PostgreSQL tools, extensions and techniques. So you can also easily use and learn Citus if you are familar with PostgreSQL.

Conclusion

It provides a strength to relational database having an option to scale it horizontally. Making it a good choice for all the user applications that are in need of both powerful query capability and handle large-data along with concurrent users. It's main use case is in real-time analytics, large-scale applications where we need high power system so instead of vertical scaling we can do the same job using horizontal scaling of Citus.

Running Apache AGE using docker

Muhammad Zeeshan — Thu, 24 Aug 2023 16:27:03 +0000

This blog is the continuation of last blog. In this blog we will run Apache AGE using docker. In our last blog we worked on setup of docker. So, if you are new to docker you can first follow this blog. And then start working on this one.

Apache AGE

Apache AGE, which is an extension of PostgreSQL. It's a graphical representation of relational database. You can setup and install AGE on docker.

Run the command

docker pull apache/age

If your docker is setup properly it will pull Apache AGE from the docker hub. In our last blog we also setup docker hub.
If everything is successful it will show an output of this type.

You have successfully pulled the apache AGE from the doecker hub. Now its time to configure and run the docker image.
And for that we need to setup a few things. like port, User, password and database. To setup these all you can run the commands.

docker run \
    --name age  \
    -p 5455:5432 \
    -e POSTGRES_USER=postgresUser \
    -e POSTGRES_PASSWORD=postgresPW \
    -e POSTGRES_DB=postgresDB \
    -d \
    apache/age

If it successful, it will return the container id. This will setup an and run the apache AGE container. To check the running container you can run the commad.

docker ps

This will show all the running containers.
To see the list of all containers irrespective of those are running or not you can add -a flag to the above command.

docker ps -a

Now to start or stop a specific container you can run the command.

docker stop container-name
docker start container-name

To check all the images that you have on your machine locally you can run the command.

docker images

That's all for this time. Happy learning!

References

https://hub.docker.com/r/apache/age

Setting Up Docker on Linux Apache AGE: A step-by-Step Guide

Muhammad Zeeshan — Thu, 17 Aug 2023 20:14:29 +0000

In this blog we will look and explore about docker. How we can install it on linux Operating system and in next blogs we will look at how we can setup our developer environment to run Apache AGE image using docker.

Introduction

Docker is an open-source plateform for building, scaling, shipping and management of applications inside portable containers. Containers are basically isolated working environemnts that allows user to package their working application along with its required dependencies. And allow them to run it consistently across different devices having different working environemnt. It can be used for various purposes like application isolation, portability, version control,microservices and resource efficiency.

Installation

There are different ways to install dokcer on linux. You can check all those from docker official webiste. In this we will look at one of the easiet way to install.

Before installing docker for the first time on our machine. We need to setup docker repository. And after that we can install and update docker from the repository.

Step 1

Update the apt packages and install required dependencies.

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg

Step 2

Add docker's official GPG key.

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Step 3

Setup docker repository on your machine.

echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Step 4

Update the apt package index.

sudo apt-get update

Everything is setup now. We only need to install the latest version of docker.

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Command given above may take some time depending upon the network speed. After installation you can us docker.

Install Check

To check if it is successfully installed or not you can run this command.

sudo docker run hello-world

Note
If you have any permission error or issues like this one

you need to create an account on docker hub.
After creating an account you can login using terminal.

docker login -u your_user_name

It will ask for user password that you entered on the time of creation of account on docker hub. Provide that password. It will show login succeeded.

In the next blog we will look at how we can setup and use apache AGE using docker.

References

(https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
https://blog.idrisolubisi.com/how-to-fix-error-response-from-daemon-pull-access-denied-for-yourusernameyourrepository
https://www.australtech.net/top-tips-to-install-docker-on-ubuntu-18-04/

Graph Database: Apache AGE.

Muhammad Zeeshan — Thu, 10 Aug 2023 19:54:22 +0000

Hi, In this blog we will be looking and exploring what is a graph database. Its applications, advantages and how Apache AGE can help us in this regards.

Graph Database

A graph database is a special type of database which is used to manage our data. From the very begining data has been an important part of the computer. And we have different ways to store, process and show that data. Traditional ways are relational databases. In relational databases we store data in tables with some predefined schemas for that table.

And we store data in the table based on that specific schema. Infact we can't store anything that does not match with that schema.

On the other hand, Graph databases organize data in the form of nodes and edges instead of table and documents. Infact, it shows the data as node and relationalship between the data as an edge between the data.
A graph database is more easy to understand relations between the data as compared to traditional database.

Understanding relation between data is much easier through the image above as compared with the simple tables of these data.

Key Features

1. It uses graph structure to explain relation between data and connect them using edges between data. In this way we can easily visualize complex relations.
2. As it stores data on nodes. So, nodes can also store some metadata which can provide us some additional information.
3. It's easy to traverse a graphed structure database to navigate from one node to the other one through the defined relation. This traversing capability enables very fast and efficient querying of difficult to understand relationships.
4. As tables database required predefined schema. Graph database are flexible towards these predefined schema. They allow nodes and relationship to have different properties and structures.
5. These graph databases are so designed to manage the relationship between nodes elegantly. As compared to complex joins in traditional database, graph database can get the data from databse more quickly.

Apache AGE:

Apache AGE is a PostgreSQL extension that provides the graph databse functionality over the top of PostgreSQL. AGE stands for A Graph Extension. You can learn more about Apache AGE.

It provides many functionalities like hybrid queries, fast graph query processing graph visualization and analytics and much more.

References

https://age.apache.org/download
https://neo4j.com/developer/graph-database/

Debugging AGE source code using VSCode.

Muhammad Zeeshan — Fri, 21 Jul 2023 15:50:37 +0000

Hi, In today's blog we will be exploring how we can setup our working environment to attach a debugger and debug AGE source code.

We will be adding some breakpoints at random files in the code just to make sure that debugger has been attached. In further blogs we will see how we can debug custom functions.

Prerequisites

AGE and PostgreSQL Source code/installed
VSCode
GDB

You can checkout this blog to install AGE and PostgreSQL on you machine.

STEP 1

First of all go to the directory where postgres is installed. And start postgres. In my case its the folder named, postgresql-16beta1 So,

cd postgresql-16beta1

Then initialize a database cluster.

bin/initdb demo

Start the server using the command.

bin/pg_ctl -D demo -l logfile start

Now create a database named demodb.

bin/createdb demodb

Start the PostgreSQL server.

bin/psql demodb

After running above commands you will see a terminal like this.

Now run the command to check the backendId where postgres is running.

SELECT pg_backend_pid();

For now we are done with PostgreSQL directory.

STEP 2

Open AGE directory in the VS-Code.
Install these two VS-Code extensions if you don't have.

After installing these extensions press F5. You will be directed to Configure Debugger for the code. It will open

launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(gdb) Attach",
            "type": "cppdbg",
            "request": "attach",
            "program": "/home/zeeshan/age16/pg/postgresql-16beta1/bin/postgres",
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "description": "Set Disassembly Flavor to Intel",
                    "text": "-gdb-set disassembly-flavor intel",
                    "ignoreFailures": true
                }
            ]
        }


    ]
}

Edit this line in the Launch.json file.

"program": "/home/zeeshan/age16/pg/postgresql-16beta1/bin/postgres",

In my case postgres16-beta is installed in age16 and then in pg directory. Change it with path where you have installed PostgreSQL.
Save changes,

STEP 3

Again press F5, this time you will be asked to add process Id. Paste the backend-id that you copied from PostgreSQL after running command SELECT pg_backend_pid();.
If it successful you will be asked to enter your device password. Enter the password.

Now add some random breakpoints in cypher_clause.c and cypher_analyze.c file(These files are in AGE's folder under src->backend->parser).

Step 4

Head back to PostgreSQL terminal where you are in demodb database.

Load Age extension.

CREATE EXTENSION age;
LOAD 'age';
SET search_path = ag_catalog, "$user", public;

Try some cypher queries.

SELECT create_graph('demo_graph');
SELECT * FROM cypher('demo_graph', $$ CREATE (n:Person {name : "james", bornIn : "US"}) $$) AS (a agtype);
SELECT * FROM cypher('demo_graph', $$ MATCH (v) RETURN v $$) as (v agtype);

During each command debugger will start and ask you to move forward you can move into specific function and move forward using the debugger.

You have successfully attached the debugger. Happy debugging :)

REFERENCES

https://dev.to/rrrokhtar/how-to-debug-age-source-code-on-vscode-3op7

https://imranzaheer.hashnode.dev/install-age-psql-from-source

Getting Started With Git and Github: Beginner's Guide

Muhammad Zeeshan — Fri, 30 Jun 2023 09:41:59 +0000

Hi, If you are looking for a beginner guide to git and github, you are at right place. In this we will be exploring about what is git and github. And how we can push and share our code with others.

Introduction

In today's software development system we need to share and maintain the different versions of our code. So, that developer working in a collaborative system can work on others code and can view the code of different collabotors. Git and github serve the purpose.

Git

Git is a free and open source distributive and collaborative version controlling system. It's main purpose it to help developers in management of source code effectively. Git has changed the way developers work. Using git developers can keep track of their work, collaborate seamlessly and can manage all of their work very easily. One of it's imporatant and powerful feature is it's ability is to allow developers to create lines and version of code using branching and merging feature.

Github

At high level github is code hosting platform mainly used for collaboration and version control. It helps developer in store and management of code as well as keep track of changes in the code.

Let's explore how we can use both.
For this i will be using Windows Operating system.

Setup

First of all install git on your machine. For this you can download and install git from here. you can read more about git and all its features here.

After installing git restart you machine and check installation by running git --verison in terminal.

Now create an account on github if you don't have.

After creating an account create a github repo to push your code.
To create repo click new and you will be shown a page like this.

Add repository name. you can make this private or public. Fill required details and click create new.

open terminal in the directory of your code that you want to upload to github.

First of all setup your username and email.

git config --global user.name "Your Name"

git config --global user.email "your.email@example.com"

replace "Your Name" and "your.email@example.com" with the name and email you want to setup.

check name and password.

# Check
git config user.name
git config user.email

Now it's time to upload code to github.

run git init

Initialize the repository to current directory.

Now setup the remote origin.

git remote add origin https://github.com/username/repository.git

Or you can copy this from github.

Now add all the files to staging area.

git add <filename>

To add file changes in the current directory and its subdirectories to staging area.

git commit -m "Commit Message"

Add your commit message to keep track of code.

git branch -M main

Setup head to main branch.

git push -u origin main

This will push you code to github now open github to check your code.

That's all for this part 💻 . In the later we will explore some advance features of github.

References

https://github.com/

https://git-scm.com/

Install PostgreSQL 16beta1 and AGE, along with possible error Solution's

Muhammad Zeeshan — Mon, 26 Jun 2023 19:14:25 +0000

Introduction

Apache AGE, which is an extension of PostgreSQL. It's a graphical representation of relational database. Currently it(AGE) supports PostgreSQL version 12 and 13. We are also working on its capability to support PG15 and 16. In this, we will be installing it for PostgreSQL 16beta1 and on linux operating system.

Create Directories

mkdir age16
cd age16
mkdir pg
cd pg

Install required Dependencies

sudo apt-get install build-essential libreadline-dev zlib1g-dev flex bison

After installing required dependencies. Now we will first install PostgreSQL 16beta1 and then AGE.

PostgreSQL

wget https://ftp.postgresql.org/pub/source/v16beta1/postgresql-16beta1.tar.gz && tar -xvf postgresql-16beta1.tar.gz && rm -f postgresql-16beta1.tar.gz

This command will download and extract postgresql 16beta1.

If it successful it will be shown like this.

We have downloaded and extracted it successfully. Now move inside beta16 directory and install postgresql.

cd postgresql-16beta1
# configure by setting flags
./configure --enable-debug --enable-cassert --prefix=$(pwd) CFLAGS="-ggdb -Og -fno-omit-frame-pointer"

#install 
make install

There could be an error of this type.

ERROR: `bison' is missing on your system. It is needed to create the
file `src/backend/parser/cypher_gram.c'. You can either get bison from a GNU mirror site

To resolve this error you can visit this link.

Installing it will take a bit of time. After successful installation it will show a windows like this.

Now go back to main directory,

cd ../
cd ../

AGE

Clone and install AGE from sources latest repository.

git clone https://github.com/panosfol/age

This will clone and create a repo age. Go to age directory.

cd age

Now checkout to latest branch and install age from that.

git checkout PG16

install

make PG_CONFIG=/home/username/age16/pg/postgresql-16beta1/bin/pg_config install

To check username, run whoami and replace it in the above command.

Now you can check if its successfully installed or not.

make PG_CONFIG=/home/username/age16/pg/postgresql-16beta1/bin/pg_config installcheck

Currently its showing 19/24 test cases failed.

You have successfully installed it now can work on it and contribute to this.

References

1. https://github.com/panosfol/age/tree/PG16

2. https://ftp.postgresql.org/pub/source/v16beta1/

Feasibility of Converting RTE to PNSI

Muhammad Zeeshan — Tue, 13 Jun 2023 15:33:46 +0000

Introduction

In this post we will be exploring about a change we are looking to implement at AGE.

Before going further into details of what RTE and PNSI is, let me first explain what is the problem statement and what we are looking for.

Problem Statement

PostgreSQL version 13(pg 13) uses PNSI instead of RTE in many functions. PNSI also exists in some of functions in PG11 and PG12 along with RTE as well.

What we are looking for?

Find a way or research about converting RTE to PNSI in PG11 and PG12 version. If we are successful, multiple version of PostgreSQL can be closer to each other. Which can help us in development of the further versions.

Now lets see what actually is RTE and PNSI.

RTE

RTE stands for Range Table Entry, in the source code it is a structure RangeTblEntry. This plays an important role in the query processing of the PostgreSQL. RTE basically represents tables, subqueries in a postgresql query. It refers not only to tables or subqueries but also some results of joins and records that SQL statement operates on. It keeps data of these tables, subqueries etc.

This RTE structure has many fields. Here I'm explaining some important fields from the RTE struct.

One of them is subquery pointer of type Query which is used to hold subquery. other is jointype of type JoinTypewhich is used to hold the information about join and some other boolean flags to keep the record of different boolean values. Which is used is further processing of query.

PNSI

PNSI stand for "ParseNameSpaceItem", which is a struct in parse_node.h file. Which represents an element of a namespace list. It has various fields, various boolean flags and important thing is that it has a field for RTE as well, to hold the data that we used to store in RTE. And other fields include p_nscolumns of type ParseNameSpaceColumn which is an array containing information about how to construct vars referencing corresponding element of the RTE's colnames list.

And there are various flags like p_rel_visible and p_cols_visible that define which RTEs are accessible by
qualified and unqualified names respectively. While processing the FROM clause various flags like
p_lateral_only and p_lateral_ok they serve the purpose of visibility to LATERAL subexpressions and errors respectively.

Relationship between RTE and PNSI

RTE is used in older versions of PostgreSQL like pg11 and pg12. PNSI is used in PostgreSQL version 13. It
also has a field for RTE to refer to “RangeTblEntry”. So, we can say that PNSI is an updated version of RTE
as it provides some extra values along with RTE field. Whenever a subquery is created within a query, an
RTE is formed to represents the information of that subquery. And in version pg13 this RTE is then
included in the PNSI structure along with other information.

Conclusion/feasibility

According to our current research status we are unable to find a way to convert RTE to PNSI. The reason is absence of some information that is required to convert but is not available. So, for now its not feaible to convert RTE to PNSI for some versions of PG.

References

https://github.com/apache/age
https://github.com/postgres/postgres/