Forem: Mahmoud Ahmed

Comparative guide for SQL Subqueries vs CTEs vs Temp Tables vs Views vs Materialized Views in AWS Aurora

Mahmoud Ahmed — Wed, 24 Sep 2025 17:52:38 +0000

In modern data-driven applications, the efficiency and readability of SQL queries can affect performance, maintainability, and developer productivity. AWS Aurora, a fully managed relational database service compatible with MySQL and PostgreSQL, offers several techniques to manage query complexity and optimize performance through: Subqueries, Common table expressions, Temporary Tables, Views, and Materialized views.

Each of these approaches serves different use cases, from simplifying nested logic to organizing reusable queries until persisting precomputed results.

In this article, we will dive into definitions, comparisons, real-world use cases, and examples in AWS Aurora, helping you to distinguish between them and answer when and why to use them.

1.Subquery

It is simply a query nested inside another query, which runs from the inner query to the outer one during execution and produces temporary results.

Used heavily in simple filtering or transformations, especially when the intermediate result is disposable.

The example below clearly explains how to write it in Aurora:


-- Find departments where the average salary exceeds 80,000

SELECT department_id, avg_salary

FROM (

    SELECT department_id, AVG(salary) AS avg_salary

    FROM employees

    GROUP BY department_id

) dept_avg

WHERE avg_salary > 80000;

As is known, for any nested logic, it can lead to performance issues if it is not optimized well, and by optimization, here we talk about complex joins or using it repeatedly. However, Aurora’s query optimizer can handle many cases efficiently, but excessive nesting can hurt performance.

2. Common Table Expression (CTE)

Always, when redundant code the initial solution comes to mind is to “define it once, use it many”, and it’s exactly what the common table expressions do, it is a named temporary result set defined with the “WITH” clause, and as same as subqueries, both live during query execution.

If we take the same example in a subquery and rewrite it again with CTEs, it will be like this:


-- Find departments where the average salary exceeds 80,000 

WITH dept_avg AS (

    SELECT department_id, AVG(salary) AS avg_salary

    FROM employees

    GROUP BY department_id

)

SELECT d.department_id, d.avg_salary

FROM dept_avg d

WHERE d.avg_salary > 80000;

As we can observe, the CTE approach is more readable and organized and its value appears more with complex or recursive queries and while combining multiple result sets, making it highly recommended against subqueries that fit with simple queries.

3.Temporary Table

For jumping from execution level approaches to session level duration, the temporary tables are the ones to choose as it is an explicitly created table that persists only during session duration serving when the result set needs to be reused multiple times in the same session. It’s clearly improved performance when working with larger datasets repeatedly, but it requires more permissions than subqueries/CTEs, meaning it requires extra privilege to create tables or alter database/schema objects, unlike others that require only select permission.

Here is an example to clarify how we can use this approach to calculate the same result as previous queries:


-- Temp table example (persisting for the whole session)

CREATE TEMP TABLE dept_avg_temp AS

SELECT department_id, AVG(salary) AS avg_salary

FROM employees

GROUP BY department_id;




-- You can run multiple queries against the temp table:

SELECT department_id, avg_salary

FROM dept_avg_temp

WHERE avg_salary > 80000;




-- (When done) optionally drop it

DROP TABLE IF EXISTS dept_avg_temp;

4. View

A “View” is a virtual table defined by a query. It simplifies the query but does not store data, can be considered as a blueprint to provide a layer of abstraction and encapsulation for reuse, besides simplifying maintenance and consistency, even if the performance is not improved.

Below is an example of how to use “views” in Aurora:


-- Create a view of department's average salary

CREATE VIEW dept_avg_view AS

SELECT department_id, AVG(salary) AS avg_salary

FROM employees

GROUP BY department_id;




-- Usage of them

SELECT *

FROM dept_avg_view

WHERE avg_salary > 80000;

5. Materialized View

The “Materialized View” stores the results of query physically unlike standard view, and effectively used for precomputing expensive aggregations (e.g., reporting dashboards) which speeds up repeated queries on large datasets, this concept is supported in AWS Aurora PostgreSQL, but not Aurora MySQL, but it can be simulated by a table and scheduled refresh with permission in both for creation part. On the persistence side views mainly are permanent until dropped as definition (views/ materialized views), and refresh stored result data for materialized views, the refresh MV frequency is a controversial topic and heavily based on the use case, because high refresh rate can hurt performance with massive dataset and low rate can break the consistency of old snapshot data and being behind the recent result.

Below is an example of how to write a “Materialized view” for our result


-- Create a materialized view 

CREATE MATERIALIZED VIEW dept_avg_mv AS

SELECT department_id, AVG(salary) AS avg_salary

FROM employees

GROUP BY department_id;




-- Query it like a normal table

SELECT *

FROM dept_avg_mv

WHERE avg_salary > 80000;




-- Refresh when data changes

REFRESH MATERIALIZED VIEW dept_avg_mv;

The table below summarizes the differences between these approaches:

Technique	Persistence	Permissions	Best For	Aurora Notes
Subqueries	Execution only	Fewer	Simple one-off queries	Can degrade performance if deeply nested
CTEs	Execution only	Fewer	Readability, recursive queries	Inline execution not materialized
Temporary Tables	Session duration	More	Reuse of intermediate results	Useful for large intermediate sets
Views	Permanent until dropped	More	Abstraction maintainability and query reuse	No inherent performance gains
Materialized Views	Permanent until dropped, stored results	More	Performance optimization via caching	Aurora PostgreSQL only

Note: Here, we are using basic comparative examples to focus on demonstrating the differences between approaches on the same idea and results. I encourage you to go a step further and experiment with these methods using more complex queries in order to put this information to use and broaden the true significance of these elements.

Conclusion:

In AWS Aurora, selecting between Subqueries, CTEs, Temporary tables, Views, and Materialized Views is all about balancing readability, reuse, and performance, based on business needs.

The flow of selecting the best approach begins with Subqueries or CTEs for simplicity, like infrequent queries, but if the intermediate results are reused multiple times through the session, we can switch to Temporary tables. However, views are superior for maintainability and abstraction, but not speed. Finally, the materialized view caching solution is performance-critical for high-volume and computation reporting queries, which can be utilized effectively in Aurora PostgreSQL.

ETL on AWS: Unlocking Data's Potential

Mahmoud Ahmed — Mon, 21 Jul 2025 18:13:31 +0000

Extracting data from multiple sources, transforming them into a unified analyzable format, then loading them to a specified place is termed ETL, or Extract, Transform, Load, which is a very vital step in data management. To facilitate proper analysis and rightly decision-making, this activity plays a vital role in assuring data quality, consistency, and relevance.

ETL is critical in the handling of disparity of data sources, formats, and structures. Through handling disparate data formats, null values, and inconsistencies, it effectively prepares the data for business intelligence, analytics, and reporting. ETL operations play a major role in the tuning of varied datasets against the backdrop of constantly changing business needs and in facilitating free flow of information from source to target.

What are the sources of data available in AWS, and where each is best used?

There are numerous data related services on AWS, they are running with high reliable and market demand, below is the major sources which are being used by millions of users on daily basis:

AWS Service	Type	Description	Use Case
Amazon RDS	Relational Database Service	Offers managed relational databases supporting engines such as MySQL, PostgreSQL, Oracle, and Microsoft SQL Server.	Defined for storing structured data and keeping traditional relational databases.
Amazon DynamoDB	NoSQL Database	Fully managed NoSQL database service with high availability and optimized for high-performance and scalable applications.	Suitable for dynamic and unstructured data applications with fast and transparent access to large amounts of data.
Amazon S3	Simple Storage Service	Scalable object storage designed to store and serve any amount of data, anywhere on the globe, at any time.	Usually employed as a data lake, with structured and unstructured data in high volumes, available to use for analytics and other processing.
Amazon Kinesis	Data Streaming	Service for real-time data streaming and analytics to process streaming data at scale.	Mainly used for real-time analytics use cases, including clickstream analysis, IoT data processing, and logging.
Amazon Redshift	Data Warehouse	Fully managed data warehouse offering, designed for high-performance SQL query analysis.	Suitable for data warehousing, allowing for fast, efficient analysis of large data sets.
AWS Lambda	Serverless Computing	Serverless compute service to allow code execution as a result of events.	Generally used within data pipeline ETL to process and transform data as a result of events that trigger based on changes in data sources.

This allows customers to design end-to-end data solutions using varied storage and processing capacity as a function of particular requirements.

How do AWS ETL processes integrate data across these diverse sources?

AWS ETL processes would integrate data across diverse sources with a mix of services and capabilities. The below is an illustration of how ETL processes can be used to bring data from multiple sources together, and other than AWS services, ETL processes can also efficiently bring data from divergent sources, such as relational databases, NoSQL storage, streaming data, or outside systems. Modularity and scalability of AWS services allow companies to create dynamic ETL pipelines that cope with the complications of their multi-dimensional data landscapes.

How AWS services integrate well with each other to serve the purpose of an end-to-end ETL pipeline?

AWS services integrate well with one another and have a solid platform to create end-to-end ETL (Extract, Transform, Load) pipelines. The arrangement of these services makes it a cost-effective and well-integrated process to manage various data sources.

Below is a sample of unified usage of these AWS services, organizations can build responsive and scalable ETL pipelines that are capable of handling the complexity of today's secured data environments, providing an integrated and networked environment for data processing.

My AWS Journey Just Leveled Up!

Mahmoud Ahmed — Tue, 22 Apr 2025 21:48:00 +0000

Hello, this is Mahmoud Certified AWS Machine Learning Specialist and I'm so happy to announce that I've been selected as an AWS Community Builder in the Networking and Content Delivery category! This is an amazing opportunity, and I'm really looking forward to diving deeper into the world of AWS, learning from the best, and contributing back to this incredible community.

For those unfamiliar with the AWS Community Builder Program and its offerings, you can find more details here. In a line, This community is designed for developers and enthusiasts to learn, connect, and collaborate on all things AWS.

A big thank you to Jason Dunn for this opportunity. I'm also eager to connect with Corey Strausman, Thembile Ndlovu, and all the other brilliant minds in the AWS Community Builders program. Let's build it together!

🎁 And the excitement doesn't stop there! My AWS Community Builder Swag Kit has arrived!

A huge thank you to Jason Dunn, and everyone who makes this community so awesome. I'm so ready for this journey ahead and can't wait to learn and grow.

Let's build it together!

Step-by-Step Guide to Upgrading an Existing AWS EKS Cluster

Mahmoud Ahmed — Tue, 10 Sep 2024 17:50:01 +0000

Upgrading your Amazon Elastic Kubernetes Service (EKS) cluster is crucial, but why is it necessary? Why not leave it as it is if it’s already functioning?”
To answer these questions you need to know some points about Kubernetes and AWS EKS in general:

First, Kubernetes is a rapidly evolving open-source project with periodic releases and adopt concept of regular upgrades. They have created a versioning and releasing process that follows a quarterly release cycle, with a cadence ranging between 2 to 5 months.

Second, The Pricing, any running software prefer to use minimal resources that costs an affordable amount of money, but what if I told you this: "your running software will cost you x6 times the current cost starting from next month", I'm sure you won't be happy with that especially if you have high scale solutions.

AWS introduced two types of support for EKS :

1- Standard support: begins when a version becomes available in Amazon EKS and continues for 14 months — the same as the upstream Kubernetes support window for minor versions, which will cost you $0.10 per cluster per hour.

2- Extended support: in Amazon, EKS begins immediately at the end of standard support and continues for 12 months but with $0.60 per cluster per hour, and if you don't upgrade the EKS cluster until end of extended support AWS will force upgrade the cluster to next version of extended support, which is not recommended because it's can produce incompatibility or downtime for your cluster.

So it's always recommended that you update it yourself and stay in the standard support plan to ensure you are taking advantage of the latest features, security fixes, and low pricing plan. However, it is a process that must be handled carefully to avoid any downtime.

In this article, we will walk you through safely upgrading your EKS cluster using the command line.

Before You Begin

Before diving into the upgrade process, ensure you have:

AWS CLI installed and configured with appropriate permissions, you can check this guide.
kubectl configured to interact with your EKS cluster, here's is how to configure it?

Step 1: Assess Your Current Environment

Begin by assessing your current cluster version:

aws eks describe-cluster --name your-cluster-name --query "cluster.version"

Then, list the available upgrade versions:

aws eks list-updates --name your-cluster-name

Step 2: Plan Your Upgrade

Choose an appropriate version to upgrade to, It's recommended to review the release notes for the target version to understand any changes or necessary actions before and after the upgrade.

Be aware of the upgrading plan:
The Upgrading comes to many points in the cluster (Control plane, Data plane, Add-ons, policies, and other related components like a load balancer, ... etc)
And can be concluded into some points:

Understand Deprecation Policies.
Review Kubernetes upgrade insights provided by AWS EKS, to understand any possible impact on your cluster, such as breaking changes that may affect your workloads.
Check Cluster Add-Ons Compatibility, understand the compatibility of any existing cluster add-ons with the cluster version you intend to upgrade to, as EKS doesn’t automatically upgrade add-ons when new versions are released or after a cluster update.
Validate that your services still work, nodes are in active status and pulled images correctly, endpoints are routed to the right resources, and any other components that don't upgrade automatically with the cluster are compatible and running.

Step 3: Upgrade the AWS EKS Cluster

Now, initiate the upgrade:

aws eks update-cluster-version --name your-cluster-name --kubernetes-version new-version

Monitor the update status:

aws eks describe-cluster --name the-cluster-name --query "cluster.status"

Step 4: Update Your Node Groups

If your cluster uses managed node groups, they will also require updating:

List your node groups:

aws eks list-nodegroups --cluster-name the-cluster-name

Update the node group:

aws eks update-nodegroup-version --cluster-name the-cluster-name --nodegroup-name the-nodegroup-name --kubernetes-version new-version

Step 5: Upgrade Add-Ons

For those using AWS EKS add-ons, ensure they are also updated:

aws eks update-addon --cluster-name the-cluster-name --addon-name the-addon-name --addon-version the-new-addon-version

Step 6: Verify system functionality

After the upgrade, it’s essential to check the cluster’s health and functionality:

Use kubectl to check the pods and deployments status.
Verify that all services are operating smoothly and test essential functionalities.

Step 7: Update Your Kubeconfig

Lastly, update your kubeconfig file:

aws eks update-kubeconfig --name the-cluster-name

Conclusion

Now you have an initial insight about upgrading your AWS EKS cluster which is a direct process when done carefully. Always follow best practices, such as backing up your data and carefully reading version release notes. And don't forget to re-deploy a service that may surface a failure and require you to update your objects/charts to support the new version.
Regular upgrades are essential for security, performance, accessing new features, and saving your budget, making this knowledge an invaluable part of managing Kubernetes infrastructure.

When to use (SELCET * FROM CTE) or just LEFT JOIN CTE?

Mahmoud Ahmed — Fri, 14 Apr 2023 11:54:13 +0000

Whether to use

SELECT * FROM CTE or LEFT JOIN CTE

directly depends on the specific requirements of your query and the structure of your Common Table Expression (CTE).

If your CTE returns a single result set that can be easily joined to other tables in your query, then using LEFT JOIN CTE directly can be more efficient and easier to read.

However, if your CTE includes multiple queries or subqueries that need to be combined before being joined to other tables, or if you need to apply further filtering or sorting to the CTE results before joining to other tables, then using SELECT * FROM CTE can provide more flexibility.

Additionally, keep in mind that when using LEFT JOIN CTE directly, the CTE will be executed for each row of the left-hand table in the join. This can cause performance issues if the CTE is complex or returns a large number of rows.

Let's take an example

Suppose we have a table called orders that contains information about customer orders, and we want to use a CTE to calculate the total revenue generated by each customer. The CTE might look something like this:

WITH customer_revenue AS (
  SELECT customer_id, SUM(price * quantity) AS total_revenue
  FROM orders
  GROUP BY customer_id
)

Now, suppose we want to join the results of this CTE with another table called customers that contains information about each customer. We could do this using either SELECT * FROM customer_revenue or LEFT JOIN customer_revenue:

-- Option 1: SELECT * FROM CTE
SELECT *
FROM customers
JOIN (SELECT * FROM customer_revenue) AS revenue
  ON customers.customer_id = revenue.customer_id;

-- Option 2: LEFT JOIN CTE
SELECT *
FROM customers
LEFT JOIN customer_revenue
  ON customers.customer_id = customer_revenue.customer_id;

In this case, both options will produce the same result. However, there are some important differences to consider:

Option 1 (SELECT * FROM CTE) allows us to apply further filtering or sorting to the CTE results before joining to other tables. For example, we could add a WHERE clause to the inner query to only include customers who have generated more than $100 in revenue:

SELECT *
FROM customers
JOIN (SELECT * FROM customer_revenue WHERE total_revenue > 100) AS revenue
  ON customers.customer_id = revenue.customer_id;

Option 2 (LEFT JOIN CTE) can be more efficient if the CTE is complex or returns a large number of rows. This is because the CTE is only executed once and then joined to the other tables, rather than being executed once for each row in the join.

In summary, both options are valid and can be used depending on the specific requirements of your query and the structure of your CTE. It is recommended to test both approaches and choose the one that provides the best performance and readability for your use case.

How I passed AWS Machine Learning Specialty Exam?

Mahmoud Ahmed — Mon, 13 Sep 2021 12:20:45 +0000

Motivation

My experience grew in the area of designing ML-based systems from experimenting with ideas to publishing working systems, but I wanted to become the kind of engineer who works on powerful scalable platforms like AWS, and from this point I focused to deep dive into AWS services especially those cooperating with machine learning system lifecycle.

While I was doing my research I encountered a scholarship program offered by MCIT in Egypt, that aims to train 500 software engineers on AWS Certified Machine Learning Specialty to be certified specialists. I applied for it and after some selection phases I got the opportunity to be a Certified AWS ML Specialist and the challenge spark was ignited.

What is the exam about?

AWS offers many types of certificates for all career levels as families and one of this them is the "specialty" family which is my machine learning specialty certificate belongs to, and its goal is below according to the AWS certification website:

"The AWS Certified Machine Learning - Specialty certification is intended for individuals who perform a development or data science role. It validates a candidate's ability to design, implement, deploy, and maintain machine learning (ML) solutions for given business problems."

The exam certificate is valid for 3 years and to pass it must pass 75% of the exam 65 questions, which cover many areas in cloud architecture patterns, development, and services distributed across these domains:

Why this exam is different than other AWS exams?

This type of exam is different because it depends on many aspects represented in the chart below, and without one of them you will have to put in more effort to fill this gap.

ML Concepts is the science behind models or basic statistical approaches which are required to be aware of before doing the exam.
ML Experience is a hand-held experience in building systems and solving problems related to the machine learning life cycle.
AWS Experience is the experience that comes from working with AWS services whether there are ML-related services or general services that work in various domains, this type of experience will help a lot in moving the mindset towards cloud and AWS especially.

What are the resources I used?

Exam Readiness AWS Certified Machine Learning - Specialty
"AWS Certified Machine Learning Specialty 2021 - Hands On!" by Frank Kane on Udemy (they promote a lot of discounts btw).
AWS ML services mind map by Julien Simon.
Exam topics dump questions.
AWS documentations.
"Amazon SageMaker Technical Deep Dive Series" by Emily Webber on YouTube.
"AWS Certified Machine Learning Specialty: MLS-C01 Certification Guide" Book By Somanath Nanda , Weslley Moura.

What about my studying strategies?

Using the resources above requires a good plan to maximize the final outcome and this strategy works well for me:

Step #1: Study the AWS Exam Readiness by watching every domain video and doing the related quiz, then solve the "Additional Study Questions" module which is around 40 questions covers all domains, and gets an initial view on weak spots, I recommend keeping this score as a baseline score.
Step #2: Watching the Udemy course, taking notes, and doing practice labs will give you experience with concepts and become more familiar with AWS services.
Step #3: Reviewing the mind map helped a lot in connecting points together, organizing your memory, and answering some questions I had from the previous steps.
Step #4 & #5 : Solving the dump questions was a very important step (a lot of exam questions are very similar to these types of questions), but the power of exam topics questions not in question itself but in the discussions under it, they have a lot of references to AWS documentation and answer points of view, reading these links will open your mind towards areas not viewed before, and knowing how others answer the questions will help you in defining keywords in the question that makes the difference in selecting the right answer. You can answer these questions multiple times but avoid memorizing them, but instead justifying the answer you selected.
Step #6: Watching Sage Maker Deep Dive Series helps me to focus on the sage maker as a platform and what, when, and how to use its services, which is an important part of my exam preparation. But I'm not dedicated to it but just watching one or two videos besides going into Step #7.
Step #7: Reading the AWS MLS Guide Book, has been a great experience because it's designed to be a summary with GitHub code snippets and questions after every chapter, which is to revise your concepts and make sure you are now familiar with every topic in it, without going into too many details. I highly recommend taking notes that would be great for revision on the day before an exam.
Step #8 (optional): resolve the "Additional Study Questions" module in the exam readiness again, and compare it with the baseline score you get in Step #1. My score jumps from 60% to 94% and was an amazing feeling gives me more confidence in my skills.

How was the exam experience?

I registered for my exam after preparing the studying plan and estimated time I need to be ready for the exam, or I thought I will be ready but actually being ready 100% is just a dream and if I tried to follow the ready state will reschedule the exam date one or two times, so my tip is preparing plan, estimating time, adapt with current deadline.
The day before the exam I used a summary for the points and modules I would need to review on the fly, like important formulas, and mind maps, because it would take a lot of time to revise every single point in the content.
I registered for my exam at the Global Knowledge testing center which is a certified center for Pearson VUE, that I preferred over an online one, and to avoid taking the risk of a dropped connection or other unexpected issues plus the pre-exam setup headache. The experience was great and exactly as I expected in the "What to expect when testing with Pearson VUE?" video. But for me the hardest thing I faced was sitting for 3 hours continuously, while I wasn't used to it.

What are the benefits I got after passing the exam?

Besides the value of the certificate, promoting career, and opening new job positions, AWS offers other great benefits such as:
Supporting for next challenge by having free practice exam voucher as well as a 50% discount on the next exam valid for next 3 years.
Engaging with the AWS Certified Community on LinkedIn, which helps in making new connections, interacting with peers, and learning from others who have validated their technical skills with AWS certifications.
Becoming a Subject Matter Expert by applying to the AWS SME program to help in deciding exam topics, developing questions, and determining passing scores.

So being certified with AWS, not just an exam and a certificate but it's a package and the relationship that started and will continue for a long time.

Conclusion

This journey was amazing and the experience and the value I got opened my horizons even more. I hope you've enjoyed reading this post and maybe even feel inspired to get AWS certified yourself (let me know if you are). It's not too late to make this one of your goals of the year!

Python Access Modifiers

Mahmoud Ahmed — Sun, 01 Nov 2020 17:05:44 +0000

We can impose access restrictions on different data members and member functions. The restrictions are specified via access modifiers. Access modifiers are levels of hidden information hiding specified for each data member or method defined in the class.

There are 3 types of access modifiers in python:

Public
Protected
Private

Public

Public attributes are those that be can be accessed inside the class and outside the class.
In python, all the attributes are public by default, if you want to change the access level you have to set it to this attribute.

Below is an example of implementing a public access modifier:

class Vehicle:

    def __init__(self, name, speed):  
         self.name = name #public data member
         self.speed=speed #public data member

    def display_name(self):#public method
      print("name:", self.name)


my_vechile = Vehicle("USS Nautilus", 3000)
my_vechile.display_name()

#Output
name: USS Nautilus

In the code above, the properties name and speed, and the method display_name are public as they can be accessed in the class as well as outside the class.

Protected

Protected attributes can be accessed through the class they declared in, or its subclasses (derived classes) only, and can be defined by adding a single underscore _ before the class attribute.

Below is a simple implementation for inheritance with protected attributes:

class Vehicle:  #Parent class

    def __init__(self, name, speed):  
         self._name = name #protected data member
         self._speed=speed #protected data member

    def _display_name(self): #protected data member
      print("name:", self._name)

class Car(Vehicle):  #Child class

      def __init__(self, name, speed, year):
          super().__init__(name, speed) #used super to point to parent class
          self.year=year #public data member

      def display_details(self): #protected method
          self._display_name()
          print("speed: ", self._speed)
          print("year:", self.year)


my_car = Car("GMC Hummer EV", 300, 2020)
my_car.display_details()

#Output
name: GMC Hummer EV
speed:  300
year: 2020

In the above code we have a Car class which inherits from the Vehicle class, and as we see the vehicle protected members can be accessed easily from the derived class Car like its own member year.

However, python doesn't fully perform the functionality of the protected modifier. The attribute defined in the above program is accessible outside the class scope. It can also be modified as well.

Private

Private attributes cannot be accessed directly from outside the class but can be accessed from inside the class.

The aim at a private level is to keep data hidden from the users and other classes and can be done by adding double underscore __ before attribute name.

class Vehicle:

   def __init__(self, name, speed):
       self.__name = name  #private data member
       self.__speed = speed  #private data member

   def __display_name(self): #private method
       print("name:", self.__name)

   def __display_speed(self):  #private method
       print("speed:", self.__speed)

   def display_details(self): #public method
       self.__display_name()
       self.__display_speed()


my_vehicle = Vehicle("USS Nautilus", 3000)
my_vehicle.display_details()

#Output
name: USS Nautilus
speed: 3000

In the above code we declared the same Vehicle class but with private data members then, tried to access them from inside and outside the class, and found it succeeded when we accessed the name in display_name method, but it failed when we accessed the speed from outside the class.

Types of Machine Learning Systems From Other Perspectives

Mahmoud Ahmed — Tue, 01 Oct 2019 14:29:17 +0000

Many people classified the machine learning system into supervised learning and unsupervised learning, but the truth, the machine learning systems can be categorized into many categories and the criteria of categorizing them not exclusive in one way.
Some of the factors which the systems classified based on it are:

Human Supervision (Supervised, Unsupervised, Semisupervised, and Reinforcement Learning).
Incrementally Learning (Online and Batch learning).
Making Prediction (Instance-Based and Model-Based Learning).

Let's look deeper into each of these criteria …

Human Supervision

Machine learning systems can be classified based on the type of supervision into four major categories:

Supervised Learning

In supervised learning, the training data introduced to the system have to include the label of each sample. For example in a spam classifier, the algorithm needs each message with its label, to know if this message is spam or not to learn how to classify new messages, and this task called classification.
And in another example with house prices prediction, the algorithm needs every house specifications, the label of the house and this case will be the price, to make the system able to predict the new house price and this task called Regression.

Unsupervised Learning

In unsupervised learning, the training data fed to the system haven't any label describes the sample belongs to which class, and for that it's must rely on itself to determine patterns of samples, which reduces the accuracy of the algorithm, However, it is the most important of these categories because most the data unlabeled and need to find patterns between them.
Some of the most used tasks in unsupervised learning are Clustering (split the dataset into groups, based on similar patterns between data points), Anomaly detection (discover unusual data points in the dataset and used it, is useful for finding fraudulent transactions, and finding outliers in data preprocessing phase) and Dimensionality reduction (also used in data preprocessing phase to reduce the number of features in the dataset).

Semi-supervised Learning

In supervised learning, the mix of labeled and unlabeled data make the model combines between the supervised and unsupervised techniques, for example by using labeled data in model training, and then use the trained model to classify the unlabeled data, and then feed all the high probability predicted data to re-train the model, with a larger amount of labeled data, this technique called pseudo-labeling.

Reinforcement Learning

In reinforcement learning, the system is very different the learning system called agent can observe the environment and learn from it, by performing actions then get a reward for a good action or penalty for a bad action, this strategy schema called policy which defines what action the agent should choose when it is in a given situation.

Incrementally Learning

Another criterion to classify machine learning systems is based on the ability to learn incrementally from a stream of coming data.

Batch Learning

This type of systems need to train on all of the available data and take more time and computation resources and calling offline learning because it learns offline and then launched into production and runs without learning, after that when we need to train new version of the system, we have to retrain the model from scratch away from production version on all data (old & new) then replace the trained model with the new one.
The simple example on this type of systems, when you try to build cat vs not cat image classifier trained on cats images on the day and after some time we need to introduce the night images, then we need to take all night and day cat images and retrain our model then deploy the new model in new release.

Online Learning

The idea of online learning is to train the system incrementally by feeding it data points sequentially, or by a small group called mini-batches, online learning is great for systems that receive data as continues flow and need to adapt quickly with new changes, and don't care too much with a long history, and the most known example of this systems is "Stocks prices prediction model" which learns on the fly to adapt with recent changes in stocks prices with less importance to too old reading.
But one of the big challenges in this type of systems is the bad quality data fed to the system, the system's performance will gradually decline, some examples of the bad data are wrong readings in stock prices or someone spamming a search engine in a try to rank high in search results, and to reduce this risk we need to add monitoring layer like anomaly detection algorithm able to detect any abnormal data can affect model performance, and switch learning off until the quality of data improving.

Making Prediction

The last factor that can separate machine learning systems into two types is how the model generalizes to making predictions for new data has never seen before.

Instance-based Learning

To be able to classify a new data point the algorithm calculates the distance between the new data point, and all the points in your data set and predict the class in which the new data point belongs.

Model-based Learning

Another way to generalize from a set of examples is to build a model of these examples, then use that model to make predictions. This is called model-based learning, and build the model here means some equation has parameters to tune alternate the instance-based learning which depends on similarity as the main choice.

If we look for the house prices prediction problem and try to fit it in these types we can find two scenarios for learning:

1st scenario: take the unknown house that we need to find its price, and measure the similarity between house features, and all labeled data points, then set the new house price based on the most similar house in our data, or average of the most similar k houses, that called model-based learning.
2nd scenario: by building equation from features and use an algorithm to tune these parameters the can fit, and generalize the house prices data, then get the feature values, and substitute with it in the equation to get the new house price.

Finally, all of these most common categories is a try to look for machine learning systems from multiple perspectives.

Python Programming Styles

Mahmoud Ahmed — Fri, 08 Mar 2019 21:39:05 +0000

While reading an article about a new version of PyTorch, my eyes stop on sentence

Intrinsically, there are two main characteristics of PyTorch that distinguish it from other deep learning frameworks:

Imperative Programming

Dynamic Computation Graphing

and want to explore more about Imperative programming and find it so pretty and clear but what about other programming styles? and can code them in python?
from here I started my search journey about python programming styles then share it here.

Initially, Python is a great language not for coding simplicity only but because it differs about other languages in limitations of coding styles, most of the other languages use just one coding style like which reduce flexibility for the programmer but in python, the programmer can many coding styles to achieve different effects.

Here's an overview of Python coding styles:

1.Procedural
tasks proceed a step at a time and you can imagine the program sequential flow, it mostly used for sequencing, selection, and modularization. more reading about them

def get_total(my_list):
    total=0
    for x in my_list:
        total+=x
    return total

print(get_total([1,2,3,4,5]))

2.Functional
it's like a math equation and any forms of state or mutable data are avoided, some developers say it as part of the procedural style, it supports solution-focused thinking (not what to do but what to accomplish). Therefore the academics teach functional languages as first programming languages and data scientists prefer it for recursion and lambda.

from functools import reduce

def sum_nums(a, b):
    return a + b

print(reduce(lambda a, b: sum_nums(a,b), [1, 2, 3, 4, 5]))

3.Imperative
computations are performed as a direct to change in the program state, it produces simple code and very useful in data structures manipulation.

total = 0
for x in my_list:
    total += x
print(total)

4.Object-oriented
it simplifies the code by using objects to model the real world and gives the programmer ability to reuse code and encapsulation allows to treat code as a black box, and inheritance make it easier to expand the functionality.

class list_opertaions(object):
    def __init__(self, my_list):
        self.my_list = my_list
        self.total = 0

    def get_total(self):
      self.total = sum(self.my_list)

obj = list_opertaions(my_list)
obj.get_total()
print(obj.total)