Forem: Mateo Ramirez Rubio

Configuring AWS VPC, Load Balancers, and DNS for WordPress and Moodle Integration

Mateo Ramirez Rubio — Sat, 07 Sep 2024 02:15:02 +0000

Implementing High-Availability VPC Architectures, DNS Routing, and Load Balancing for WordPress and Moodle in AWS

Introduction.
System Architecture.
Implementation and Deployment.

* Configuration of the **VPC** and Subnets.

* Configuration of the **DNS Server.**

* Configuration of the **Load Balancer (HAProxy).**

Installation and Configuration of WordPress.
Installation and Configuration of Moodle.
Verification and Results.
Issues Encountered During Development, Troubleshooting, and/or Recommendations.
Conclusions.
References.

List of Figures

AWS Resource Map.
Screenshot of EC2 Instances in AWS.
BIND Configuration File named.conf.local.
BIND Configuration File db.learxyz.online.
BIND Configuration File db.10.
HAProxy Configuration File haproxy.cfg.
HAProxy Statistics Page.
WordPress Main Page.
Moodle Learning Portal.
AWS Security Group Configuration.
Route 53 Configuration.
DNS Propagation Verification on WhatsMyDNS.

Introduction

The project involves the deployment of a cloud services environment for XYZ S.A. Manufacturas (Fictitious not real company). The goal is to create a scalable and highly available infrastructure using AWS, which includes a CMS (WordPress), an LMS (Moodle), and a load balancer (HAProxy). The setup includes creating a VPC with public and private subnets, installing services on EC2 instances, and configuring a domain in Route 53.

System Architecture

AWS Resource Map

Figure 1: AWS Resource Map

The resource map shows the VPC structure implemented for XYZ S.A. Manufacturas. It includes:

VPC (Virtual Private Cloud): XYZSA-vpc-01, which houses all network resources.
Subnets:
- Public-subnet-XYZSA-01: Public subnet containing the public instance.
- Private-subnet-XYZSA-01: Private subnet containing application and database instances.
Route Tables:
- Public-Route-Table-XYZSA-VPC: Manages traffic for the public subnet.
- Private-Route-Table-XYZSA-VPC: Manages traffic for the private subnet.
Network Connections:
- Internet Gateway (ig-XYZSA-vpc): Enables Internet connectivity for the VPC.
- NAT Gateway (Nat-gateway-XYZSA-VPC): Enables instances in the private subnet to securely access the Internet.

Implementation and Deployment

VPC and Subnet Configuration

Figure 2: Screenshot of EC2 Instances in AWS

This screenshot shows the EC2 instances deployed in AWS:

private-instance-CMS-01-XYZSA and private-instance-CMS-02-XYZSA: These instances host the WordPress CMS. Having two instances allows for high availability and load balancing.
private-instance-DB-01-XYZSA: This instance hosts the MariaDB database, accessible only from the application instances.
private-instance-LMS-01-XYZSA: This instance runs the Moodle LMS.
public-instance-01-XYZSA: This public instance acts as a load balancer and DNS server, directing traffic to the appropriate instances in the private subnet.

DNS Server Configuration

Figure 3: BIND Configuration File named.conf.local

This file configures the BIND DNS server to resolve domain names to IP addresses:

zone "learxyz.online": Defines the primary zone for the domain learxyz.online, indicating that DNS records for this domain are in db.learxyz.online.
zone "1.0.10.in-addr.arpa": Defines the zone for reverse IP resolution in the 10.0.1.0 subnet, with records in db.10.

Figure 4: BIND Configuration File db.learxyz.online

This file contains the specific DNS records for learxyz.online:

SOA Record: Defines the origin server and zone update parameters.
NS Records: ns1.learxyz.online and ns2.learxyz.online are the authoritative name servers.
A Records: ns1 and ns2 point to 100.27.144.145 (elastic IP of the public instance). The @ (root domain) also points to 100.27.144.145.
CNAME Record: www.learxyz.online is an alias pointing to @, resolving to 100.27.144.145.
A Record: lms.learxyz.online points to 100.27.144.145.

These records ensure that all DNS requests for learxyz.online and its subdomains are directed to the elastic IP of the public instance, where HAProxy will load balance the traffic.

Figure 5: BIND Configuration File db.10

This file contains the reverse resolution records:

PTR Records: Maps IP addresses to corresponding domain names. 10.0.1.10 resolves to www.learxyz.online and 10.0.1.12 to lms.learxyz.online.

These records allow for reverse DNS resolution, useful for diagnostics and certain applications requiring domain name verification from IPs.

Load Balancer Configuration (HAProxy)

Figure 6: HAProxy Configuration File haproxy.cfg

This file configures HAProxy, the load balancer:

Listen stats: Enables the statistics interface on port 8080, accessible with basic authentication.
Frontend http_front: Defines a frontend listening on port 80. Uses ACLs to redirect traffic based on the host name:
- learxyz.online and www.learxyz.online are directed to the backend wordpress_servers.
- lms.learxyz.online is directed to the backend moodle_servers.
Backend wordpress_servers: Balances traffic between two WordPress instances (10.0.1.119 and 10.0.1.129).
Backend moodle_servers: Balances traffic to the Moodle instance (10.0.1.174).

HAProxy distributes incoming requests to the appropriate instances, ensuring high availability and redundancy.

Figure 7: HAProxy Statistics Page

This page provides real-time statistics on HAProxy's performance:

Connection Status: Shows the number of active and queued connections.
Processed Traffic: Incoming and outgoing bytes, denied requests, and errors.
Backend Instance Status: Indicates whether instances are active and available for load balancing.

WordPress Installation and Configuration

Figure 8: WordPress Homepage

These images show the homepage of the WordPress site at www.learxyz.online. The page changes language when reloaded in the browser to demonstrate load balancing, distributing requests between the two WordPress instances. This ensures the site remains available even if one instance fails.

Moodle Installation and Configuration

Figure 9: Moodle Learning Portal

This image shows the Moodle learning portal at lms.learxyz.online. The course "Introduction to XYZ Manufacturing" is available, designed to train employees on the company's manufacturing processes. Moodle is configured to provide a robust and scalable platform for online education and training.

Verification and Results

Figure 10: AWS Security Group Configuration

Security Group rules control access to EC2 instances:

ICMP: Allows all ICMP packets for diagnostics (ping).
SSH (22): Allows SSH access for remote administration.
MySQL/Aurora (3306): Allows MySQL traffic from any IP, enabling connection to the database.
DNS (UDP 53) and DNS (TCP 53): Allow DNS traffic.
Custom TCP (8080): Allows access to the HAProxy statistics page.
HTTP (80) and HTTPS (443): Allow HTTP and HTTPS traffic for web access.

These rules ensure that instances are accessible only through the necessary ports, improving security.

Figure 11: Route 53 Configuration

Route 53 manages traffic routing for learxyz.online:

A Record: learxyz.online points to elastic IP 100.27.144.145, directing all traffic to the load balancer.
NS Records: Defines the authoritative name servers.
SOA Record: Contains origin server information and zone parameters.
A Record: lms.learxyz.online points to 100.27.144.145.
CNAME Record: www.learxyz.online is an alias pointing to learxyz.online.

These records ensure that requests for learxyz.online and its subdomains are properly directed to the public instance.

Figure 12: DNS Propagation Verification in WhatsMyDNS

This image shows DNS record propagation using WhatsMyDNS. Successful global propagation ensures that learxyz.online resolves correctly to 100.27.144.145 from multiple locations worldwide, guaranteeing accessibility and redundancy.

Issues Encountered, Troubleshooting, and Recommendations

Several issues were encountered during project development that required attention and resolution:

DNS Server Configuration:

* **Issue:** DNS records did not propagate correctly initially.

* **Solution:** The BIND records were reviewed and corrected, ensuring the name servers pointed to the correct elastic IP. WhatsMyDNS was used to confirm global propagation of the records.

HAProxy Load Balancer Configuration:

* **Issue:** HTTP traffic was not evenly distributed between the WordPress instances.

* **Solution:** ACLs and backend settings in haproxy.cfg were adjusted to ensure all requests were properly load balanced.

WordPress and Moodle Installation:

* **Issue:** WordPress instances displayed the default Apache page instead of WordPress content.

* **Solution:** WordPress files were correctly moved to the web server's root directory, and the proper permissions were set.

Security Group Configuration:

* **Issue:** Access to necessary service ports was restricted.

* **Solution:** Security Group rules were updated to allow traffic on specific ports (80, 443, 22, 3306, 53, 8080).

Moodle Configuration:

* **Issue:** Missing required PHP extensions during Moodle installation.

* **Solution:** Required PHP extensions (xml, intl, zip, etc.) were installed and enabled, and PHP configurations (max\_input\_vars, memory\_limit) were adjusted to meet Moodle’s requirements.

Conclusions

The project demonstrated XYZ S.A. Manufacturas’ ability to implement a scalable and highly available cloud infrastructure. AWS and HAProxy provided the necessary flexibility and redundancy to support the company’s critical applications. The following points highlight the project’s conclusions:

Scalability: The infrastructure can easily scale to handle increased traffic and workload.
High Availability: Redundancy and load balancing ensure that services remain available even if an instance fails.
Security: Proper configuration of Security Groups and access policies ensures resources are accessible only by authorized users.

Future Work

Future recommendations for XYZ S.A. Manufacturas include:

Automation: Implement AWS Lambda functions to automate system monitoring and scaling.
Monitoring: Use AWS CloudWatch for real-time monitoring of infrastructure performance and resource utilization.
Disaster Recovery: Set up automatic backups and a disaster recovery plan to ensure business continuity in case of failures.

References

"Amazon Web Services (AWS) Documentation," AWS, [Online]. Available: https://docs.aws.amazon.com/.
"HAProxy Documentation," HAProxy Technologies, [Online]. Available: https://www.haproxy.org/.
"Moodle Documentation," Moodle, [Online]. Available: https://docs.moodle.org/.
"WordPress Documentation," WordPress, [Online]. Available: https://wordpress.org/documentation/.

Quick Introduction to Parallel Computing in Go: A Practical Approach with Goroutines, WaitGroups, Mutexes and Atomic Operations

Mateo Ramirez Rubio — Mon, 19 Aug 2024 01:44:56 +0000

Parallelism Fundamentals in Go: Learn how to Implement Goroutines and WaitGroups to Run Tasks Concurrently

Introduction

A fundamental technique for improving performance is parallel computing, which allows multiple tasks to run simultaneously to make the best use of system resources. In this blog, we will explore how to implement parallel programming techniques in Go, using practical examples and key concepts such as goroutines, WaitGroups, Mutexes and atomic operations.

Introduction to Parallel Computing

Parallel computing is a form of processing in which tasks are divided into sub-tasks that can be executed at the same time. This contrasts with sequential computing, where tasks are executed one after the other. The main idea is that by executing multiple operations simultaneously, the overall execution time can be reduced and the overall efficiency of the system can be improved.

Parallelism vs. Concurrency

It is important to differentiate between parallelism and concurrency:

Parallelism: This refers to the simultaneous execution of multiple tasks. In the context of computing, this generally involves the use of multiple CPU cores to execute tasks at the same time. In Go, this is achieved through the use of goroutines and the thread-based execution model.
Concurrency: The ability to handle multiple tasks in such a way that they appear to be running simultaneously, even if they are not necessarily running in parallel. Concurrency can involve switching between tasks on a single CPU core. In Go, concurrency is handled by goroutines and channels, facilitating the creation of programs that can do multiple things at once.

Parallel Computing in Go

Go, known for its simplicity and efficiency, offers powerful tools for parallel programming. Among them, goroutines are fundamental. Goroutines are functions that run concurrently with other functions. They are lightweight and managed by the Go runtime, allowing highly parallelised applications to be created without additional complications.

Goroutines: These are the basic unit of execution in Go. They are created using the go keyword, and the Go runtime handles the concurrent and parallel execution of these goroutines.
WaitGroups: These are used to wait for a set of goroutines to finish execution. They provide a form of synchronisation between goroutines, ensuring that all parallel tasks complete before continuing.
Mutexes: These are mechanisms to control concurrent access to shared resources. They prevent race condition problems, where two or more goroutines attempt to access and modify the same resource simultaneously, which can lead to unexpected results or errors.

In this blog, we will use a practical example of parallel processing in Go to illustrate these concepts. Through the analysis of code that performs parallel mapping, filtering and reduction operations, we will see how to apply these concepts to improve performance and efficiency in Go applications. We will explore how goroutines and other tools provided by the language can be used to achieve faster and more efficient data processing.

Key Components in the Code

In this section, we will discuss in detail the essential components of our code, explaining how goroutines, sync.WaitGroup, sync.Mutex and atomic operations are used to implement parallel programming in Go.

You can find the complete code here:

MateoRamirezRubio1 / learning_Go_Golang

Goroutines

Goroutines are a fundamental feature of Go that allows concurrent execution of functions. They are created by using the go keyword before a function or function expression. Goroutines are lightweight and managed by the Go runtime, which means that they can be created and managed in large numbers without a high resource cost.

In the code, goroutines are used to perform parallel operations on different parts of a slice. This is done in the parallelMap, parallelFilter, and parallelReduce functions:

// Each worker applies the mapping function to its chunk of the slice.
        go func(start, end int) {
            defer wg.Done()
            for j := start; j < end; j++ {
                result[j] = mapFunc(nums[j])
            }
        }(start, end)

Each goroutine takes a segment of the nums slice and applies the mapFunc function to each element of that segment. The use of goroutines allows multiple segments of the slice to be processed at the same time, reducing the overall execution time.

WaitGroups

The sync.WaitGroup is a synchronisation tool used to wait for a group of goroutines to complete their execution. It is used to coordinate the completion of multiple goroutines and ensure that all goroutines have finished before continuing the execution of the program.

In the code, sync.WaitGroup is used to ensure that the main program waits for all goroutines to finish before continuing:

// Create a WaitGroup to synchronize the workers.
    var wg sync.WaitGroup

    // Start the workers.
    for i := 0; i < numWorkers; i++ {
        start := i * chunkSize
        end := start + chunkSize
        if i == numWorkers-1 {
            end = n
        }
        wg.Add(1)
        // Each worker applies the mapping function to its chunk of the slice.
        go func(start, end int) {
            defer wg.Done()
            for j := start; j < end; j++ {
                result[j] = mapFunc(nums[j])
            }
        }(start, end)
    }

    // Wait for all workers to finish.
    wg.Wait()

Here, wg.Add(1) increments the WaitGroup counter each time a new goroutine is started. Each goroutine calls wg.Done() when it finishes, decrementing the counter. wg.Wait() blocks the main thread until the counter reaches zero, i.e. until all goroutines have finished their work.

Mutexes

The sync.Mutex is a synchronisation mechanism used to prevent race conditions, which occur when multiple goroutines attempt to access and modify shared data at the same time. A mutex ensures that only one goroutine can access a critical section of code at a time.

In the code, sync.mutex is used to protect access to the result slice during filtering and reduction operations:

// Create a Mutex to protect the result slice.
    var mu sync.Mutex

// Each worker filters its chunk of the slice and appends the results to the local result slice.
        go func(start, end int) {
            defer wg.Done()
            localResult := make([]int, 0, end-start)
            for j := start; j < end; j++ {
                if filterFunc(nums[j]) {
                    localResult = append(localResult, nums[j])
                }
            }
            // Lock the Mutex and append the local result slice to the global result slice.
            mu.Lock()
            result = append(result, localResult...)
            mu.Unlock()
        }(start, end)

The mutex mu is used to ensure that only one goroutine can modify result at any given time. mu.Lock() acquires the mutex before modifying result, and mu.Unlock() releases the mutex after the modification.

Atomic Operations

Atomic operations allow operations on shared variables to be performed safely without the need for mutexes. The sync/atomic package provides operations such as addition, subtraction, and compare and shift, which are atomic and ensure that operations are safe in a concurrent environment.

In the code, atomic.AddInt32 is used to perform the reduction safely:

var result int32

// Each worker reduces its chunk of the slice and atomically adds the result to the global result variable.
        go func(start, end int) {
            defer wg.Done()
            localResult := nums[start]
            for j := start + 1; j < end; j++ {
                localResult = reduceFunc(localResult, nums[j])
            }
            // Lock the Mutex and atomically add the local result to the global result variable.
            mu.Lock()
            atomic.AddInt32(&result, int32(localResult))
            mu.Unlock()
        }(start, end)

Here, atomic.AddInt32(&result, int32(localResult)) adds the localResult value to the result variable atomically, avoiding synchronisation problems without the need to use a mutex.

Parallel Execution in Action: Mapping, Filtering and Reduction

Now, we will examine how mapping, filtering, and reduction operations are implemented in parallel using goroutines, sync.WaitGroup, sync.Mutex, and atomic operations. These operations are common in data processing and benefit greatly from parallel programming to improve performance.

Parallel Mapping

Mapping is the process of applying a function to each element of a data set and transforming those elements into new values. In the code provided, parallelMap performs this operation in parallel.

Implementation

// parallelMap applies a mapping function to each element of a slice in parallel.
func parallelMap(nums []int, mapFunc func(int) int, numWorkers int) []int {
    n := len(nums)
    // Create a result slice of the same length as the input slice.
    result := make([]int, n)
    // Calculate the chunk size for each worker.
    chunkSize := n / numWorkers
    // Create a WaitGroup to synchronize the workers.
    var wg sync.WaitGroup

    // Start the workers.
    for i := 0; i < numWorkers; i++ {
        start := i * chunkSize
        end := start + chunkSize
        if i == numWorkers-1 {
            end = n
        }
        wg.Add(1)
        // Each worker applies the mapping function to its chunk of the slice.
        go func(start, end int) {
            defer wg.Done()
            for j := start; j < end; j++ {
                result[j] = mapFunc(nums[j])
            }
        }(start, end)
    }

    // Wait for all workers to finish.
    wg.Wait()
    return result
}

Explanation

Creating the Result Slice: A result slice is created with the same length as the nums input slice. This slice will store the mapping results.
Division of Labour: The chunk size for each goroutine is calculated as n / numWorkers. Each goroutine will process a section of the nums slice, from index start to end.
Launching Goroutines: A goroutine is started for each slice segment. Each goroutine applies the mapFunc function to its segment of nums and stores the result in result.
Synchronisation: sync.WaitGroup is used to wait for all goroutines to finish. wg.Add(1) is called before starting each goroutine, and wg.Done() is called at the end.
Wait for Completion: wg.Wait() blocks execution until all goroutines have finished their work, ensuring that result is completely filled with mapped values.

Parallel Filtering

Filtering is the process of selecting elements from a dataset that meet a specific condition. In parallelFilter, this process is done in parallel to improve performance.

Implementation

// parallelFilter filters a slice in parallel using a filtering function.
func parallelFilter(nums []int, filterFunc func(int) bool, numWorkers int) []int {
    n := len(nums)
    chunkSize := n / numWorkers
    var wg sync.WaitGroup
    // Create a Mutex to protect the result slice.
    var mu sync.Mutex
    result := make([]int, 0, n)

    for i := 0; i < numWorkers; i++ {
        start := i * chunkSize
        end := start + chunkSize
        if i == numWorkers-1 {
            end = n
        }
        wg.Add(1)
        // Each worker filters its chunk of the slice and appends the results to the local result slice.
        go func(start, end int) {
            defer wg.Done()
            localResult := make([]int, 0, end-start)
            for j := start; j < end; j++ {
                if filterFunc(nums[j]) {
                    localResult = append(localResult, nums[j])
                }
            }
            // Lock the Mutex and append the local result slice to the global result slice.
            mu.Lock()
            result = append(result, localResult...)
            mu.Unlock()
        }(start, end)
    }

    // Wait for all workers to finish.
    wg.Wait()
    return result
}

Explanation

Creating the Result Slice: A result slice is created which is initially empty. This slice will store the elements that pass the filter.
Division of Labour: Each goroutine processes a section of the nums slice, applying the filterFunc function to select the elements that meet the condition. Accumulation of
Local Results: Each goroutine maintains a local slice localResult to temporarily store the elements that pass the filter. This avoids data contention between goroutines.
Mutex Access Protection: After a goroutine completes its filtering, it uses mu.Lock() to ensure exclusive access to the result slice and aggregate the filtered items. mu.Unlock() releases the mutex after the operation.
Synchronisation: sync.WaitGroup is used to wait for all goroutines to finish. wg.Add(1) is called before starting each goroutine, and wg.Done() is called at the end.
Wait for Completion: wg.Wait() ensures that result is completely filled before returning it.

Parallel Reduction

Reduction is the process of combining all elements of a dataset into a single value using a reduce function. In parallelReduce, this process is performed in parallel.

Implementation

// parallelReduce reduces a slice in parallel using a reduction function.
func parallelReduce(nums []int, reduceFunc func(int, int) int, numWorkers int) int {
    n := len(nums)
    chunkSize := n / numWorkers
    var wg sync.WaitGroup
    var mu sync.Mutex
    var result int32

    for i := 0; i < numWorkers; i++ {
        start := i * chunkSize
        end := start + chunkSize
        if i == numWorkers-1 {
            end = n
        }
        wg.Add(1)
        // Each worker reduces its chunk of the slice and atomically adds the result to the global result variable.
        go func(start, end int) {
            defer wg.Done()
            localResult := nums[start]
            for j := start + 1; j < end; j++ {
                localResult = reduceFunc(localResult, nums[j])
            }
            // Lock the Mutex and atomically add the local result to the global result variable.
            mu.Lock()
            atomic.AddInt32(&result, int32(localResult))
            mu.Unlock()
        }(start, end)
    }

    // Wait for all workers to finish.
    wg.Wait()
    return int(result)
}

Explanation

Result initialisation: result is a variable of type int32 that is used to store the final accumulated value. It is initialised with a default value.
Division of Labour: Each goroutine processes a segment of the nums slice, applying the reduceFunc function to combine the values into a partial result.
Local Result Calculation: Each goroutine maintains a localResult variable to accumulate the partial result of its segment. The reduceFunc function is applied iteratively to the elements of the segment.
Atomic Update: After calculating the partial result, atomic.AddInt32 is used to add localResult to the result variable atomically. This ensures that the update is safe in a concurrent environment without using a mutex for synchronisation.
Synchronisation: sync.WaitGroup is used to wait for all goroutines to finish. wg.Add(1) is called before starting each goroutine, and wg.Done() is called on completion.
Wait for Completion: wg.Wait() blocks until all goroutines have finished, ensuring that result contains the final accumulated value.

Detailed Explanation of main

main() is the main function in a Go program, and in this case, it coordinates the execution of parallel processing in the program. In the following, we will break down each part of main to understand how it works and what its purpose is.

### Data Generation

// Generate a random slice of 1000000 integers between 1 and 9.
nums := generateRandomSlice(1000000, 1, 9)
fmt.Println("First numbers:", nums[:10])

Data Generation: The generateRandomSlice function is used to create a 1,000,000 slice of random integers in the range 1 to 9. This slice simulates a large data set to process.
Print First Numbers: A portion of the slice (nums[:10]) is printed to show the first 10 numbers generated. This is useful to verify that the data generation was successful.

### Measuring Execution Time

// Start measuring execution time.
start := time.Now()

Measurement Start: time.Now() is used to capture the current time, marking the start of the measurement of parallel processing execution time. This will allow to calculate how long it takes to complete all parallel operations.

### Configuring the Number of Workers

// Get the number of CPUs available.
numWorkers := runtime.NumCPU()

Number of Workers: runtime.NumCPU() returns the number of CPU cores available on the machine. This value is used to define how many workers (goroutines) will be used in parallel processing. Using a number equal to the number of CPU cores helps to optimise performance. In the next section when we run the code we will go more into this.

### Application of Parallel Functions

// Apply the mapping function in parallel.
mapped := parallelMap(nums, func(x int) int { return x * 2 }, numWorkers)

Application of parallelMap: The parallelMap function is called to apply a mapping function to each element of the slice nums. In this case, the mapping function multiplies each number by 2.

// Apply the filtering function in parallel.
filtered := parallelFilter(mapped, func(x int) bool { return x > 10 }, numWorkers)

Application of parallelFilter: The parallelFilter function is called to filter the mapped slice, keeping only elements greater than 10.

// Apply the reduction function in parallel.
result := parallelReduce(filtered, func(acc, x int) int { return acc + x }, numWorkers)

Application of parallelReduce: The parallelReduce function is called to reduce the filtered slice by summing all elements. The reduce function takes an accumulator and an element, and returns their sum.

### Printout of Results

fmt.Println("Result:", result)

Print Result: The final result of the reduction is printed. This is the aggregated value of all elements that passed the filter.

### Measuring Execution Time

// Stop measuring execution time and print the duration.
duration := time.Since(start)
fmt.Println("Execution time:", duration)

End of Measurement: time.Since(start) calculates the duration from the start of parallel processing to the current time. This duration is printed to show how long it took to perform all parallel operations.

Examples of results and numWorkers

numWorkers := runtime.NumCPU()

The line numWorkers := runtime.NumCPU() is a key part of the code that allocates the number of workers to be used to process tasks in parallel. This line uses the runtime.NumCPU() function, which returns the number of CPU cores available on the system where the program is running. This number is then assigned to numWorkers, which determines how many goroutines will be created to split the work.

Use Cases and Considerations

Parallelism Optimisation:

* **CPU Leverage:** By using `runtime.NumCPU()`, the code attempts to take full advantage of the available hardware by allocating one goroutine per CPU core. This can improve performance by distributing the workload across all cores, allowing multiple tasks to run in parallel without saturating a single core.

Avoid Goroutine Overload:

* **Worker/Task Balance:** If `numWorkers` is equal to or close to `runtime.NumCPU()`, the program is likely to maintain an appropriate balance between the number of workers and system capabilities. However, if this number is significantly exceeded, performance may suffer due to system overhead from creating and managing too many goroutines, which can lead to unnecessary competition for system resources.

Problems with Small Data Sizes:

* **More Workers than Tasks:** If the number of workers (`numWorkers`) is greater than the number of elements in the array, a problem can occur. For example, if you have an array with 4 elements and `numWorkers` is 8 (assuming the CPU has 8 cores), some goroutines will not have enough elements to process. This can cause incorrect or inefficient behaviour, such as:

    * **Resource Waste:** Some goroutines may end up not doing any useful work, which wastes system resources.

    * **Incorrect results:** Depending on how the code is structured, if the work is divided into more goroutines than elements in the array, some goroutines might not process any data, which could lead to incorrect results.

Manual Tuning of numWorkers:

* **Optimisation for Specific Data:** In some cases, it may be beneficial to manually set `numWorkers` instead of using `runtime.NumCPU()`. For example, if you know the array size is small, you can set `numWorkers` to a lower number to avoid over-allocating goroutines. This can result in more efficient code that is easier to debug.

Behaviour with Very Large Arrays:

* **Equal Division of Labor:** When working with very large arrays, `runtime.NumCPU()` can help ensure that each CPU core gets a reasonable share of the work. This is especially useful in intensive operations where parallelism can significantly reduce execution time.

Practical Example

Suppose you are running code on a system with 8 CPU cores (runtime.NumCPU() returns 8). If you have an array with 100 elements, and you use numWorkers := 8, each goroutine will process 12 or 13 elements (100/8 = 12.5). This is usually efficient.

However, if the array has only 4 elements and numWorkers is still 8, then each goroutine will handle half an element, which is impractical and may cause some goroutines to do nothing, or even throw errors if the code does not handle this scenario correctly.

Execution and result verification

To:

nums := generateRandomSlice(10, 1, 9)
fmt.Println("First numbers:", nums[:10])

// Start measuring execution time.
start := time.Now()
// Get the number of CPUs available.
numWorkers := 3

When running the code with the command go run parallel_map_filter_reduce.go, with the above parameters set in the main(), we get in console:

First numbers: [3 6 1 2 8 1 7 2 8 6]
Result: 70
Execution time: 514.3µs

Execution Explanation

Random Slice Generation:

* `generateRandomSlice(10, 1, 9)` generates a slice of 10 random integers, each within the range of 1 to 9.

* In this particular run, the numbers generated were: `[3, 6, 1, 2, 2, 8, 1, 7, 2, 8, 6]`.

* This line also prints the numbers generated: First numbers: `[3 6 1 2 8 1 7 2 8 6]`.

Number of Workers:

* We define `numWorkers := 3`, indicating that 3 goroutines (workers) will be used to process the slice in parallel.

Execution Result:

* The code processed the generated slice using the parallel mapping, filtering and reduction functions.

* The final result was `70`.

Checking the Result

Let's check the result step by step:

Mapping (Multiplication by 2):

* The numbers `[3, 6, 1, 2, 2, 8, 1, 7, 2, 8, 6]` are mapped to `[6, 12, 2, 4, 16, 2, 14, 4, 16, 12]`.

Filtering (Greater than 10):

* Numbers greater than 10 are filtered out, leaving `[12, 16, 14, 16, 16, 12]`.

Reduction (Sum):

* The sum of the filtered numbers is `12 + 16 + 14 + 16 + 16 + 12 = 70`.

The result obtained (70) is correct, as it corresponds to the sum of the values that passed the filter after being mapped.

Warning about `numWorkers`

When using numWorkers, it is important to choose an appropriate value for the number of workers in relation to the size of the slice. Here are some warnings:

Too many Workers: if numWorkers is greater than the number of elements in the slice, some workers will have no work, which could cause parallelism to be inefficient. This can also lead to problems if the code does not handle these cases correctly, as happened in the example above with an incorrect result.
Too Few Workers: If the number of workers is too low, parallelism will not be properly exploited, which could cause the program to run slower than if more workers were used.

In this case, with numWorkers := 3 for a 10-element slice, each worker processes approximately 3-4 elements, which is reasonable and results in correct processing.

Conclusion

We have explored the fascinating world of parallel computing in Go, focusing on how to take advantage of the language's features to improve the efficiency and performance of our applications. Through a practical example that performs parallel mapping, filtering and reduction operations, we have seen how goroutines, WaitGroups and Mutexes can be used to handle concurrent tasks and synchronise access to shared resources.

How would you improve this code to make it even more efficient or robust?

Your perspective can help to further optimise the code and adapt it to new situations.

I look forward to hearing from you in the comments.

You may be interested:

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Mateo Ramirez Rubio ・ Jul 22

#testing #webdev #beginners #programming

Using Environment Variables with flutter_dotenv and DotEnv in Flutter: Step by Step Guide

Mateo Ramirez Rubio — Tue, 06 Aug 2024 05:10:08 +0000

Load and access your environment variables securely and efficiently in your Flutter applications.

Introduction

When building Flutter applications, it's essential to manage sensitive data like API keys, database credentials, or environment-specific configurations. Hardcoding these values directly in your code is a security risk and can lead to issues when deploying your app to different environments.

That's where .env variables come in – a simple and effective way to store and manage environment-specific configurations. In this blog, we’ll explore how to use .env files to manage environment variables in your Flutter app, with a step-by-step guide and a practical example.

Some concepts

What Are `.env` Files?

.env files are simple text files used to store environment-specific variables. They typically contain key-value pairs, like API keys or base URLs, which can be accessed throughout your application.

Why Use `.env` Files?

Separation of Concerns: Keep configuration settings separate from your code.
Security: Store sensitive information like API keys securely.
Flexibility: Easily switch between different environments (development, staging, production).

Step-by-Step Guide

Prerequisites

Before we begin, ensure you have the following:

A Flutter project set up.

Step 1: Add the `dotenv` package

First, add the dotenv package to your pubspec.yaml file:

dependencies:
  flutter_dotenv: ^5.1.0

Step 2: Create a `.env` File

Create a .env file in the root directory of your Flutter project. This file will store your environment variables.

API_KEY=your_api_key_here
BASE_URL=https://example.com

Step 3: Update `pubspec.yaml`

Ensure that the .env file is included as an asset in your project. Modify your pubspec.yaml file to include the .env file:

flutter:
  assets:
    - .env

Important: Run flutter pub get to apply the changes.

Step 4: Load the `.env` File

In your main.dart file, import flutter_dotenv and load the .env file before running the app:

import 'package:flutter_dotenv/flutter_dotenv.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized(); // Ensure Flutter is initialized
  try {
    await dotenv.load(fileName: ".env"); // Load environment variables
  } catch (e) {
    throw Exception('Error loading .env file: $e'); // Print error if any
  }
  runApp(const MainApp()); // Runs the app
}

Ensure Flutter is initialized: WidgetsFlutterBinding.ensureInitialized() ensures that Flutter's framework is initialized before loading the .env file. This is important because dotenv.load() relies on Flutter's file system to read the .env file.
Load environment variables: await dotenv.load(fileName: ".env") loads the environment variables from the .env file. The await keyword ensures that the loading process is completed before proceeding.
Error handling: The try-catch block catches any errors that might occur while loading the .env file. If an error occurs, it throws an exception with a meaningful error message.
Run the app: Finally, runApp(const MainApp()) runs the app.

Step 5: Access Environment Variables

You can access the environment variables using dotenv.env:

import 'package:flutter_dotenv/flutter_dotenv.dart';

class ApiService {
  final String baseUrl = dotenv.env['BASE_URL'] ?? 'default_url';
  final String apiKey = dotenv.env['API_KEY'] ?? 'default_key';

  // Use baseUrl and apiKey in your API calls
}

Troubleshooting

Common Issues:

File Not Found: Ensure the .env file is in the root directory and correctly specified in pubspec.yaml.
File Not Loading: Confirm that await dotenv.load() is called before running the app.

Note: Remember to add your .env file to your .gitignore file to avoid committing sensitive data to your version control system.

By following these steps and using flutter_dotenv in your Flutter app, you'll be able to manage your environment-specific configurations securely and efficiently.

I hope this helps! Let me know if you have any questions.

Aditional Resources:

Flutter Dotenv documentation.

Topics you may be interested in:

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Mateo Ramirez Rubio ・ Jul 22

#testing #webdev #beginners #programming

How to Name Endpoints in a REST API: Complete Guide with Best Practices and Practical Example

Mateo Ramirez Rubio ・ Jul 7

#api #rest #backend

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Mateo Ramirez Rubio — Mon, 22 Jul 2024 11:05:21 +0000

Introduction

In this blog we'll see a basic guide on how to test a Django project using the Django REST Framework! Here, we'll learn the basics of testing with pytest, including essential concepts such as fixtures and mocks, to test execution and test coverage with coverage.

This is the second part of a project we have already done in the blog, in this blog we will do the testing of this project.

You can see how the code base of the Django project was developed step by step with the Django REST framework here:

Complete Guide to the Django Services and Repositories Design Pattern with the Django REST Framework

Mateo Ramirez Rubio ・ Jul 8

#webdev #django #backend #api

In this guide we will cover:

Initial Setup: How to install and configure pytest and other packages needed to test your Django project.
Fixtures: How to use fixtures to set up test data and prepare the test environment efficiently.
Mocks: How to use pytest-mock to simulate behaviours and conditions in your tests, ensuring that you can test different scenarios without relying on real implementations.
Model Testing: We will verify the creation of model instances and their string representations to ensure they work as expected.
Repository Testing: We will test the operation of repository methods, including database manipulation and error handling.
API View Testing: We will validate that API views respond correctly to HTTP requests and handle data properly.
Web View Testing: We will ensure that web views load correctly and display the expected content.
Code Coverage: We will run tests and generate coverage reports to identify areas of the code that need further testing.
Recommendations, Test Types and Best Practices.

We will look at practical examples of configuring files such as pytest.ini and demonstrate how to run the commands to get detailed coverage reports with coverage.

Introduction to Testing with Pytest

Testing is a crucial part of software development that ensures the quality, functionality and reliability of code. In the Python ecosystem, pytest is one of the most popular and powerful tools for testing.

What is Pytest?

pytest is a Python testing framework that makes it easy to write simple, scalable tests. It is known for its simplicity and its ability to adapt to projects of any size. With pytest, you can write unit tests, integration tests, and more, using a clear and concise syntax.

Advantages of Using Pytest:

Simplicity and Flexibility:pytest allows you to write tests using simple functions. You don't need to learn a complex syntax or follow a rigid structure.
Powerful Fixtures:pytest provides a fixture system that allows you to configure the state before and after testing, making it easy to create repeatable and isolated test environments (more on fixtures later in the blog).
Extensible Plugins:pytest has a lot of plugins that extend its functionality, such as pytest-django for Django projects, pytest-mock for mocks and many more.
Clear Error Reporting: Error messages in pytest are detailed and easy to understand, which helps to identify and fix problems quickly.
Automatic Test Detection:pytest automatically detects tests in your project without the need to explicitly record them, which simplifies test organisation.

Integration with Django

pytest integrates easily with Django through the pytest-django plugin. This allows you to efficiently test views, models and other components of a Django project. With pytest-django, you can use Django-specific fixtures, such as the test database, users and more.

Initial setup for Testing with Pytest in a Django project with Django REST Framework

Before starting to write and run tests with pytest in a Django project, it is essential to properly configure the test environment.

Installing the requirements

To install the pytest, pytest-django, beautifulsoup4, coverage and pytest-mock dependencies, follow these steps:

Create a virtual environment (optional but recommended):
```
python -m venv env
source env/bin/activate
```

Install the dependencies using pip:

pip install pytest pytest-django beautifulsoup4 coverage pytest-mock

Little explanation:

pytest: The core testing framework.
pytest-django: Plugin that makes it easy to integrate pytest with Django.
beautifulsoup4: Used to parse and check HTML in view tests.
coverage: Tool to measure test coverage, ensuring that all parts of the code are tested.
pytest-mock: Plugin to facilitate the creation and use of mocks in tests.

Pytest configuration

To configure pytest in a Django project, we need to create a pytest.ini file in the root directory of the project. This file will define the specific pytest settings.

Create a file called pytest.ini in the root directory of your Django project and add the following configuration:

[pytest]
DJANGO_SETTINGS_MODULE = my_project_blog.settings
python_files = tests.py test_*.py *_tests.py

DJANGO_SETTINGS_MODULE: Specifies the Django configuration module that pytest should use.
python_files: Defines the filename patterns that pytest will recognise as test files. In this case, pytest will look for files named tests.py, files starting with test_, and files ending in _tests.py.

Structure

For this project, all the tests of the project will be located in the tests directory which is located in the apps directory. We will follow the following file structure of the tests for better separation of functions and readability:

In this blog, we will cover some of the main tests (Note: We will only look at the main tests as the others have a similar structure), but you can find the full test code here:

MateoRamirezRubio1 / mini-blog-rest-api

A mini blog Django project demonstrating the implementation of the Services and Repositories design pattern for a blog application.

This project is a Django-based mini blog application designed to illustrate the implementation of the Services and Repositories design pattern. It features a clear separation of concerns, making the codebase easier to maintain and extend.

You can follow the step by step of the making of the app on my blog: My blog.
You can see the application test here: Testing App.

Features

Posts and Comments: Users can create, update, and delete blog posts and comments.
RESTful API: Provides endpoints for interacting with the application programmatically.
Web Interface: Offers a basic user-friendly interface for managing posts and comments.
Service Layer: Contains business logic and orchestrates interactions between views and repositories.
Repository Layer: Encapsulates data access logic, providing a clean API for the service layer.
Django Admin: Allows administrative management of posts and comments.

Project Structure

The project follows a modular structure, with each app having its own models…

View on GitHub

Definitions before starting (Fixtures)

In order to get started with the testing, we will create some Fixtures which we will use in several of the tests later on. To give you some context, let's quickly look at what a Fixture is in Pytest.

What is a Fixture?

In the context of testing, a Fixture is a set of data, objects, or configurations that are prepared before running a test.

Fixtures commonly define the steps and data that constitute the Arrange phase of a test (see the anatomy of a test), as well as the Cleanup phase using Python's yield statement (see more on this).

Basically, test functions request the fixtures they need by declaring them as arguments to the test function where they want to use them. So, when pytest is going to run a test, the first thing it does is look at the parameters that this test function has, and then it looks for fixtures that have the same names as those parameters. Once pytest finds them, it executes them, captures what they return (if anything), and passes those objects to the test function as arguments.

In order for Pytest to know that a function is a Fixture, we must decorate it with @pytest.fixture.

What are fixtures used for?

Fixtures are widely used for different functions, some of these are:

Prepare the Test Environment: Configure the necessary state for testing to be performed under specific conditions. For example, creating objects in the database or initialising configurations.
Code Reuse: Allows the same set of data or configurations to be used in multiple tests, avoiding code duplication.
Test Isolation: Ensures that each test runs in a clean and consistent environment, avoiding side effects between tests.
Configuration Automation: Simplifies the automatic configuration of the test environment, ensuring that tests are set up correctly before execution.

Coding Fixtures

Already having some knowledge about what fixtures are, for this case, we will create 4 fixtures which we will set in the conftest.py file:

Fixture post:

Purpose: To create an instance of Post that can be used for testing.
Reason: To test any functionality that involves the Post model, you need an existing Post object in the database. By using a fixture to create this object, you can ensure that each test has a consistent, well-defined Post to work with.

Benefit: Using fixtures avoids duplication of code in every test that needs a Post, ensuring that all tests that depend on a Post use the same data set.

@pytest.fixture
def post(db):
    """
    Create a Post instance with title and content for testing.

    Args:
        db: The pytest fixture that sets up a test database.

    Returns:
        Post: A Post object with predefined title and content.
    """
    return Post.objects.create(title="Test Post", content="This is a test post")

Fixture comment:

Purpose: Create an instance of Comment that is associated to a previously created Post.
Rationale: Similar to the Post case, if your tests involve the Comment model, you need a Comment object in the database. By associating the Comment with a Post, you can test the comment-related functionality of a specific post.

Benefit: By using this fixture, you can test how comments interact with posts without having to manually create a Post in each test. It also ensures that the Comment is correctly linked to a Post.

@pytest.fixture
def comment(db, post):
    """
    Create a Comment instance associated with the provided Post.

    Args:
        db: The pytest fixture that sets up a test database.
        post: The Post fixture providing a Post object.

    Returns:
        Comment: A Comment object linked to the provided Post.
    """
    return Comment.objects.create(content="This is a test comment", post=post)

api_client fixture:

Purpose: Provide an instance of APIClient to make HTTP requests to API Endpoints.
Rationale: When testing API views and Endpoints, you need a tool to simulate HTTP requests and receive responses. APIClient is ideal for this because it is designed to interact with API Endpoints and verify responses.

Benefit: Allows you to test API logic, validations, and responses in a controlled test environment. Ensures that the test client is configured consistently for all API testing.

@pytest.fixture
def api_client():
    """
    Provide an instance of APIClient for making API requests.

    Returns:
        APIClient: An instance of the APIClient class.
    """
    return APIClient()

Fixture client:

Purpose: Provide an instance of the standard Django test client for making HTTP requests.
Rationale: In addition to testing the API, you may need to test HTML-based views. Client allows you to perform these tests by simulating HTTP requests and verifying the responses of template-based views.

Benefit: Allows you to test user interaction with the web application, such as page display and form handling, ensuring that the UI part of the application works correctly.

@pytest.fixture
def client():
    """
    Provide a Django test client instance for making HTTP requests.

    Returns:
        Client: An instance of Django's test Client class.
    """
    return Client()

Coding the tests

Now that we have everything configured, we'll start testing.

We'll do it in blocks, first we'll test the Django models, then the repositories, the services, the API REST views and finally the web views.

Testing the models

File:test_models.py

The testing of the models will verify the correct creation and representation of the Post and Comment models, ensuring that the data is handled and presented correctly in different situations.

test_post_creation(post):

Purpose: Verifies that a Post object is created correctly with the expected title and content.
- Rationale:
- Input: Uses the post fixture, which creates an instance of Post with predefined data.
- Verification: Ensures that the title and content attributes of the Post object match the expected values (‘Test Post’ and ‘This is a test post’, respectively).
- Why: Ensures that the creation of Post objects works as expected by ensuring that the attributes are set correctly at creation time.

    def test_post_creation(post):
        """
        Verify that a Post object is created with the correct title and content.

        Args:
            post: The Post fixture providing a Post object.
        """
        assert post.title == "Test Post"
        assert post.content == "This is a test post"

test_comment_creation(comment):

Purpose: Verifies that a Comment object is correctly created with the expected content and that it is associated with the correct Post.
- Rationale:
  - Input: Uses the comment fixture, which creates an instance of Comment linked to a specific Post.
  - Verification: Ensures that the content attribute of the Comment object is ‘This is a test comment’ and that the associated Post has the expected title ‘Test Post’.
  - Why: Confirms that the comment is correctly associated with the Post and that the data is stored as expected.

    def test_comment_creation(comment):
        """
        Verify that a Comment object is created with the correct content and associated Post title.

        Args:
            comment: The Comment fixture providing a Comment object.
        """
        assert comment.content == "This is a test comment"
        assert comment.post.title == "Test Post"

test_comment_string_representation(comment):

Purpose: Verifies that the string representation of a Comment object is correctly truncated to 20 characters.
- Rationale:
- Input: Uses the comment fixture, which provides a Comment with specific content.
- Verification: Checks that the output of str(comment) matches the first 20 characters of the comment content (‘This is a test comment’).
- Why: Ensures that the ___str___ method of the Comment model behaves correctly, truncating the content to 20 characters as expected, which is useful for displaying a preview of the comment in user interfaces.
```
def test_comment_string_representation(comment):
    """
    Verify that the string representation of a Comment is correctly truncated to 20 characters.

    Args:
        comment: The Comment fixture providing a Comment object.
    """
    strComment = "This is a test comment"
    assert str(comment) == strComment[:20]
```

Testing repositories

Files:test_repositories_comment.py, test_repositories_post.py

These tests will be in charge of ensuring that the repository methods work correctly both in normal conditions (when data exists) and in error situations (when data does not exist or there are database errors), thus ensuring the reliability of the repository code.

test_repository_get_comment_by_id_exists(comment):

Purpose: Verify that a Comment object can be retrieved by its ID when it exists in the database.

Rationale:

Input: Uses the comment fixture to provide an existing Comment object.
Verification: Uses the get_comment_by_post_and_id method of the CommentRepository to retrieve the comment by post ID and comment ID.
Asserts: Checks that the retrieved comment is equal to the Comment object provided by the fixture.

Why: Ensures that the repository method works correctly to retrieve existing comments, thus validating the basic functionality of the repository.

def test_repository_get_comment_by_id_exists(comment):
    """
    Verify that a Comment can be retrieved by its ID when it exists in the database.

    Args:
        comment: The Comment fixture providing a Comment object.

    Asserts:
        The result should be equal to the provided Comment object.
    """
    result = CommentRepository.get_comment_by_post_and_id(comment.post.id, comment.id)
    assert result == comment

test_repository_get_comment_by_id_is_does_not_exist():

Purpose: Verify that trying to retrieve a Comment by its ID when it does not exist returns None.

Rationale:

Input: Provides a post ID and comment ID that do not exist in the database.
Verification: Uses the get_comment_by_post_and_id method of the CommentRepository to try to retrieve the non-existent comment.
Asserts: Checks that the result is None.
Why: Ensures that the repository method correctly handles cases where the comment does not exist, returning None as expected.
```
@pytest.mark.django_db
def test_repository_get_comment_by_id_is_does_not_exist():
    """
    Verify that attempting to retrieve a Comment by ID when it does not exist returns None.

    Asserts:
        The result should be None.
    """
    result = CommentRepository.get_comment_by_post_and_id(232, 9999)  # Non-existent ID
    assert result is None
```
(

Little interlude so we know a bit more about what @pytest.mark.django_db is and why we use it:

@pytest.mark.django_db
- Purpose: The @pytest.mark.django_db decorator is used to indicate that the test interacts with the Django database.
  
  Rationale:
  - Input: Some tests need to access or modify the database to verify correct code behaviour.
  - Verification: Without this decorator, tests that attempt to interact with the database would throw an error, as pytest isolates tests from accessing the database by default to improve performance.
  - Why: This decorator is crucial for tests that verify the creation, update, delete or query of objects in the database, ensuring that database operations are performed in a controlled and isolated test environment.
)

test_database_error_repository_create_comment(mocker):

Purpose: To test whether the create_comment repository method handles database errors (DatabaseError) correctly.

Rationale:
- Input: Uses mocker to patch the Comment.objects.create method and simulate a DatabaseError.
- Verification: Attempts to create a comment using the create_comment method of the CommentRepository.
- Asserts: Checks that a DatabaseError is thrown as a result of the simulation.
- Why: Ensures that the repository handles database errors correctly during comment creation, thus validating the robustness and error handling capability of the repository.
```
def test_database_error_repository_create_comment(mocker):
    """
    Test if the repository method create_comment handles DatabaseError properly.

    Args:
        mocker: The pytest-mock fixture used to patch objects.

    Asserts:
        The method should raise a DatabaseError when a DatabaseError is simulated.
    """
    mocker.patch('apps.comments.repositories.comment_repository.Comment.objects.create', side_effect=DatabaseError)

    with pytest.raises(DatabaseError):
        CommentRepository.create_comment({}, 1)
```
(

Little interlude for us to learn a bit about what mocker is and how to use it:

Using mocker
- Purpose:mocker is a tool provided by pytest-mock that is used to patch methods or functions during testing. This allows to simulate different behaviours and situations of the components under test.
  
  Rationale:
  - Input: In tests that verify database error handling, the Comment.objects.create method is patched to simulate a DatabaseError.
  - Verification: By patching this method, the desired behaviour (in this case, throwing a DatabaseError) is forced without the need for the database to actually fail.
  - Asserts: Verifies that the code handles the simulated errors correctly.
  - Why: The use of mocker allows testing how the code handles exceptions and other specific cases without relying on unpredictable external conditions. It is a fundamental technique in unit testing to ensure that the code responds appropriately to different scenarios, including errors.
)

Testing REST API views

Files:test_api_comment.py, test_api_post.py

These tests will ensure that the REST API views correctly handle the creation and updating of comments and posts, including input validation and error handling.

This approach will help ensure that the REST API views are robust and handle different usage scenarios correctly, improving the quality and reliability of the code.

test_create_comment(api_client, post):

Purpose: Verify that a new Comment can be created for a Post via the API.

Rationale:

Input: Uses the api_client fixture to make API requests and the post fixture to provide a Post object.
Action: Makes a POST request to the post-comment-create URL with the data needed to create a new comment.
Asserts:
- Verifies that the response status is HTTP 201 Created.
- Verifies that the response data includes the content of the new comment.
Why: Ensures that the API handles the creation of new comments correctly by validating the resource creation functionality.

def test_create_comment(api_client, post):
    """
    Verify that a new Comment can be created for a Post via the API.

    Args:
        api_client: The APIClient fixture for making API requests.
        post: The Post fixture providing a Post object.

    Asserts:
        The response status should be HTTP 201 Created.
        The response data should include the content of the new Comment.
    """
    response = api_client.post(
        reverse("post-comment-create", args=[post.id]),
        {"post": post.id, "content": "New comment"},
        format="json",
    )

    assert response.status_code == status.HTTP_201_CREATED
    assert response.data["content"] == "New comment"

test_perform_update_raises_kwargs_does_not_exists():

Purpose: Verify that perform_update throws an APIException if comment_pk is not present in kwargs.

Rationale:

Input: Creates an instance of CommentRetrieveUpdateDestroyAPIView and sets kwargs to an empty dictionary.
Action: Attempts to execute perform_update without providing comment_pk.
Asserts:
- Verifies that an APIException is thrown.
- Verifies that the exception message is ‘Comment ID is required to update the comment.’

Why: Ensures that the view correctly handles cases where the required arguments are not present, providing proper validation and error handling.

def test_perform_update_raises_kwargs_does_not_exists():
    """
    Verify that perform_update raises an APIException if 'comment_pk' is not provided in kwargs.

    Asserts:
        The exception message should be "Comment ID is required to update the comment."
    """
    view = CommentRetrieveUpdateDestroyAPIView()
    view.kwargs = {}

    try:
        view.perform_update({})
    except APIException as exc:
        assert str(exc) == "Comment ID is required to update the comment."

test_perform_update_invalid_comment_id():

Purpose: Verify that perform_update throws an APIException if comment_pk has an invalid format.

Rationale:

Input: Creates an instance of CommentRetrieveUpdateDestroyAPIView and sets kwargs with comment_pk as an invalid value.
Action: Uses a MockSerializer to simulate a validated serializer and then attempts to execute perform_update.
Asserts:
- Verifies that an APIException is thrown.
- Verifies that the exception message is ‘Invalid Comment ID format.’
Why: Ensures that the view correctly handles cases where arguments have an invalid format, thus validating the robustness of error handling and data validation.

class MockSerializer(Serializer):
    """
    Mock serializer class to simulate serializer behavior in tests.
    """
    validated_data = {"content": "Test comment"}

def test_perform_update_invalid_comment_id():
    """
    Verify that perform_update raises an APIException if 'comment_pk' has an invalid format.

    Asserts:
        The exception message should be "Invalid Comment ID format."
    """
    view = CommentRetrieveUpdateDestroyAPIView()
    view.kwargs = {"comment_pk": "invalid"}

    serializer = MockSerializer(data={"content": "Test comment"})
    serializer.is_valid()  # Simulate successful serializer validation

    with pytest.raises(APIException) as excinfo:
        view.perform_update(serializer)
    assert str(excinfo.value) == "Invalid Comment ID format."

Testing web views

Files:test_web_views_comment.py, test_web_views_post.py

These tests will verify that the ‘Post’ and ‘Comment’ web views are working correctly. Helping to ensure that users can see the details and lists of ‘Posts’ and ‘Comments’ as expected in the web interface, improving the quality and reliability of the application.

test_post_detail_view(client, post):

Purpose: Verify that the detail view of a Post returns a successful response and that it includes the title of the specific Post.

Rationale:
- Input:
  - client: Uses Django's client fixture to make HTTP requests.
  - post: Use the post fixture to provide a Post object.
- Action: Make a GET request to the Post's detail URL using reverse to resolve to the correct URL and pass the Post ID.
- Asserts:
  - Verifies that the status of the response is HTTP 200 OK.
  - Verifies that the content of the response includes the Post title in binary format (b ‘Test Post’).
- Why: This test ensures that the Post detail view works correctly and that the Post title is displayed as expected, validating both functionality and data representation.
```
def test_post_detail_view(client, post):
    """
    Verify that the Post detail view returns a successful response and includes the title of the specific Post.

    Args:
        client: The Django test client fixture for making HTTP requests.
        post: The Post fixture providing a Post object.

    Asserts:
        The response status should be HTTP 200 OK.
        The response content should include the title of the Post.
    """
    response = client.get(reverse("post-detail", args=[post.id]))
    assert response.status_code == 200
    assert b"Test Post" in response.content
```

test_comment_detail_view(client, comment):

* **Purpose:** Verify that the detail view of a `Comment` returns a successful response and that it includes the content of the specific `Comment`.

    **Rationale:**

    * **Input:**

        * `client`**:** Uses Django's `client` fixture to make HTTP requests.

        * `comment`**:** Uses the `comment` fixture to provide a `Comment` object.

    * **Action:** Make a GET request to the Comment's detail URL using `reverse` to resolve to the correct URL and pass the `Post` and `Comment` IDs.

    * **Asserts:**

        * Verify that the status of the response is `HTTP 200 OK`.

        * Uses `BeautifulSoup` to parse the HTML content of the response and extract all elements.

        * Verifies that the content of the elements includes the text of the specific comment.

    * **Why:** This test ensures that the `Comment` detail view works correctly and that the `Comment` content is displayed as expected, validating both functionality and data representation.

```python
def test_comment_detail_view(client, comment):
    """
    Verify that the Comment detail view returns a successful response and includes the content of a specific Comment.

    Args:
        client: The Django test client fixture for making HTTP requests.
        comment: The Comment fixture providing a Comment object.

    Asserts:
        The response status should be HTTP 200 OK.
        The response content should include the text of the Comment.
    """
    response = client.get(
        reverse("post-comment-detail", args=[comment.post.id, comment.id])
    )
    assert response.status_code == 200

    soup = BeautifulSoup(response.content, "html.parser")
    comments = soup.find_all("p")
    comment_texts = [comment.get_text() for comment in comments]
    assert "This is a test comment" in comment_texts
```

Using Coverage with Pytest in our Django project

Coverage is an essential tool for measuring test coverage in a project. It allows us to see which parts of the code are being tested and which are not, helping to identify areas that need more testing. In this section, we will look at two key commands for using coverage in conjunction with pytest.

Command 1:`python -m coverage run -m pytest`:

python -m coverage run -m pytest

python -m coverage: This command runs the coverage module as a script. Using -m instead of just coverage ensures that the version of coverage installed in the current virtual environment is used.
run: This subcommand of coverage indicates that we want to run a script (in this case, pytest) and measure the coverage while it is running.
-m pytest: This part of the command tells Python to execute the pytest module. This way, coverage will run pytest and record code coverage during test execution.

This command runs all tests using pytest and, at the same time, records the code coverage, generating a .coverage file in the root directory of the project containing the coverage data.

Command 2:`python -m coverage report --include=‘apps/’ --show-missing`:

python -m coverage report --include="apps/*" --show-missing

python -m coverage: As in the previous command, executes the module coverage.
report: This coverage subcommand generates a coverage report on the terminal.
--include=‘apps/*’: This argument indicates that we only want to include files in the apps folder in the coverage report. You can adjust this path according to the structure of your projects and the scope you want to give to your test coverage. This is useful to exclude files you are not interested in, such as configuration files or third-party libraries.
--show-missing: This argument adds a column to the report that shows which lines of each file were not covered by the tests. This makes it easier to identify specific areas of code that need further testing.

Running the tests with Coverage and Pytest

First command:

The command python -m coverage run -m pytest runs the tests using pytest and also measures the coverage of the code.

If you run the command, you should get something like this:

Collection analysis and execution of tests:

Total tests collected: 44.
Results:
- All tests (44) passed successfully.
- The total execution time was 0.83s.

Explanation:

This result shows that all the tests that we defined so far in the project were executed correctly and passed. There were no failures, bugs or skipped tests, which indicates that the code under test behaves as expected in all cases covered by the tests.

Second command:

Code Coverage Report Explanation:

After running the command to generate the code coverage report, we will get a detailed breakdown of how the tests cover the source code as we can see in the image above.

Let's take a closer look at what all this data means:

Sections of the Report

Name:

* This column shows the path to each file in the project that contains executable code.

* **For example**, `apps\comments\models.py` refers to the models.py file in the comments app.

Stmts:

* This column indicates the total number of statements (lines of executable code) in each file.

* **For example**, `apps\comments\models.py` has 9 statements.

Miss:

* This column shows how many statements in the file were not executed during testing.

* **For example**, if `apps\posts\views\web_views.py` has 4 lines not covered, it means that 4 lines of code in that file were not executed by the tests.

Cover:

* This column shows the percentage of coverage for each file. It represents the percentage of lines of code executed during testing.

* **For example**, `apps\posts\views\api_views.py` has a coverage of 71%, which means that 71% of its lines of code were executed during the tests.

Missing:

* This column lists the specific lines that were not covered by the tests.

* **For example**, `apps\comments\views\web_views.py` shows `21, 24-25, 54, 58, 60-63`, indicating that these lines were not executed.

Meaning and Measurements

Lines Not Covered (Miss):

* **Meaning:** The lines of code listed in this column were not executed during testing. This could be because certain paths in the code are not being tested.

* **Action:** Add tests to cover these paths. For example, if there is error handling on these lines, make sure your tests include cases that trigger those errors.

Low Cover (Cover):

* **Meaning:** A low percentage indicates that a large part of the code in that file is not being tested.

* **Action:** Review files with low coverage and add tests to cover more use cases. This may include additional unit tests or functional tests that exercise more code paths.

Full Coverage (100% Cover):

* **Meaning:** All statements in these files were executed during testing.

* **Measurement:** Although the coverage is 100%, make sure that the tests also correctly verify the logic of the code, not just that it executes.

Recommendations and Best Practices

Finally, we will look at some recommendations, types of tests and best practices for effective application testing, especially in the context of applications such as this one.

Types of Tests

To achieve complete and effective coverage, consider implementing the following types of tests:

* **Unit Tests:** Verify the operation of individual pieces of code, such as functions, methods or classes. Make sure that each unit of code does what is expected without relying on other parts of the system.

* **Integration Tests:** Validate how different components of the system interact. In the Django REST Framework, this may include testing that API endpoints work correctly and that data is handled properly between views and models.

* **Functionality Testing:** This focuses on the full functionality of the system from the user's point of view. For example, make sure that users can create, read, update and delete resources correctly through the API.

* **Regression Testing:** Ensures that code changes do not break existing functionality. They should be run regularly to detect problems introduced by new features or bug fixes.

* **Performance Testing:** Verify the behaviour of the application under load and stress conditions. It is useful to identify bottlenecks and ensure that the application can handle the expected traffic.

Best Practices

To ensure that your testing is effective and reliable, consider the following practices:

* **Code Coverage:** Although not a guarantee of quality, high code coverage (ideally over 80%) helps identify untested areas. Use tools such as `coverage.py` to measure and improve coverage.

* **Test Independence:** Each test should be independent and run in any order. Use fixtures to set the initial state and clean up after each test if necessary.

* **Use of Mocks and Stubs:** For unit tests, use mocks to simulate external behaviours and dependencies. This ensures that tests are fast and do not depend on external services or shared state.

* **Error and Exception Testing:** Be sure to test cases where the application must handle errors and exceptions. Verify that errors are handled properly and that useful messages are returned to the user.

* **Security Testing:** Include tests that verify the security of your application, such as permission validation and protection against common vulnerabilities (e.g., SQL injection, Cross-Site Scripting).

* **Testing in Real Environments:** Run your tests in environments that mimic the production environment as closely as possible to detect problems that only occur in specific configurations.

Additional Recommendations

* **Keep Tests Readable and Maintainable:** Write tests that are clear and easy to understand. Use descriptive test names and keep test code clean and well organised.

* **Automate Test Execution:** Integrate your tests into the continuous integration (CI) process so that they run automatically on every code change. This helps identify problems quickly and ensures that new code does not break existing functionality.

* **Review and Refactor:** Regularly review and refactor your tests to improve their coverage and efficiency. Test code can also become obsolete over time, so keep it up to date.

And that's it for our basic guide to testing in a Django REST Framework project!

Remember that testing is a crucial part of software development, and maintaining good coverage and well-defined tests will help you prevent bugs and improve the quality of your code over time.

If you have additional questions, feel free to leave them in the comments.

Thanks for reading and happy code testing!

Other topics you may be interested in:

How to Name Endpoints in a REST API: Complete Guide with Best Practices and Practical Example

Mateo Ramirez Rubio ・ Jul 7

#api #rest #backend

Complete Guide to the Django Services and Repositories Design Pattern with the Django REST Framework

Mateo Ramirez Rubio — Mon, 08 Jul 2024 16:59:18 +0000

Introduction to the Django Services and Repositories Design Pattern with Django REST Framework

In the world of software development, code organisation and maintainability are crucial to the long-term success of any project. In particular, when working with frameworks like Django and the Django REST Framework to build robust web applications and APIs, it's essential to follow design patterns that help us keep our code clean, modular and easy to scale.

In this blog, we will explore one of these widely used design patterns: Services and Repositories. This pattern allows us to separate the concerns of data access and business logic, improving the structure and clarity of our code. Through this approach, we not only make our applications easier to maintain and test, but also more flexible and future-proof.

Join us as we break down this pattern step-by-step, from initial project setup to service and repository deployment, and discover how it can transform the way you develop your Django apps.

The realisation of the project is divided into three main sections: the structuring and design of the project; the coding of the structured project; testing.

You can find the complete code and structure of the project in the following GitHub link:

MateoRamirezRubio1 / mini-blog-rest-api

A mini blog Django project demonstrating the implementation of the Services and Repositories design pattern for a blog application.

You can follow the step by step of the making of the app on my blog: My blog.
You can see the application test here: Testing App.

Features

Posts and Comments: Users can create, update, and delete blog posts and comments.
RESTful API: Provides endpoints for interacting with the application programmatically.
Web Interface: Offers a basic user-friendly interface for managing posts and comments.
Service Layer: Contains business logic and orchestrates interactions between views and repositories.
Repository Layer: Encapsulates data access logic, providing a clean API for the service layer.
Django Admin: Allows administrative management of posts and comments.

Project Structure

The project follows a modular structure, with each app having its own models…

View on GitHub

Also, now you can find the second part of this project where we do the testing of the complete app:

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Mateo Ramirez Rubio ・ Jul 22

#testing #webdev #beginners #programming

Creation and design of the project

In this first section, we will see how to create a new Django project with DRF (Django REST Framework) and we will analyse how the main parts of the project and REST APIs will be structured.

Start and creation of the Django project

Before starting the project, you must install Django and Django REST Framework if you don't already have it:
```
pip install django djangorestframework
```
Now that we've installed the main tools we're going to use for the project, we'll create a new Django project using the django-admin startproject command:
```
django-admin startproject my_project_blog
```
This command generates the basic structure of a Django project, including configuration files and a project directory.
Creation of Django applications

For this small blog project, we will have two main functionalities which we will divide into two Django apps: Posts and Comments.

For this, in the terminal, inside the main project directory (my_project_blog), create two applications: Posts and Comments using the python command manage.py startapp:
```
python manage.py startapp posts
python manage.py startapp comments
```
Apps are modular Django components that group related functionalities. In this case, Posts will handle blog posts and Comments will handle comments.
File settings:

Add the created apps and the DRF to the INSTALLED_APPS section in the project settings file: /my_project_blog/my_project_blog/settings.py :

INSTALLED_APPS = [
    "rest_framework",
    "apps.posts",
    "apps.comments",
...

### Project structure:

Next I will show you how the project will be structured, you can create all the directories/folders and files (ignore README.md, .env, .gitignore), and then we will fill them with code as we learn.

Main structure:

Structure of the Comments app:

Structure of the Posts app:

### System design:

For the realisation of the project we opted for an architecture based on layers and classic design patterns such as Repository and Services.

The Services and Repositories pattern divides business logic (services) and data access logic (repositories) into separate layers:

Layer	Description
Web Layer	It presents the data to users. In this case, the views are for both REST APIs (using DRF) and web views (using Django Views). *Note:* REST APIs views handle all the API logic (CRUD, etc), while web views only handle sending data to the user via templates.
Service layer	Contains the business logic. It communicates with the `Repository Layer` to retrieve data and performs additional operations (without interacting directly with the database) before returning data to the views or controllers.
Repository Layer	It is responsible for interacting directly with the database. This layer is responsible for basic CRUD operations. *Note:* This layer is the only one in charge of interacting directly with the database.
Model Layer	Data models representing the database tables.

We will delve a little deeper into each layer later as we code.

### Design of REST APIs

REST APIs allow communication between the frontend and the backend. We use the Django REST Framework to create RESTful APIs. This is the structure we have for their respective naming and functions:

HTTP Method	Endpoint	Description
GET	`/posts/`	Gets a list of all posts.
POST	`/posts/`	Create a new post in the database.
GET	`/posts/{post_id}/`	Gets the details of a specific post.
PUT	`/posts/{post_id}/`	Update a specific post.
DELETE	`/posts/{post_id}/`	Delete a specific post.
GET	`/posts/{post_id}/comments/`	Gets a list of comments for a post.
POST	`/posts/{post_id}/comments/`	Create a new comment for a post.
GET	`/posts/{post_id}/comments/{comment_id}/`	Gets the details of a comment on a post.
PUT	`/posts/{post_id}/comments/{comment_id}/`	Update a comment on a post.
DELETE	`/posts/{post_id}/comments/{comment_id}/`	Remove a comment from a post.

Coding and deepening

Now that we are clear about the layout of the project, for the second section, we will code our blogging project by delving into several of the layers we talked about earlier.

Model:

The Model represents the data. It defines the database tables. In Django, models are classes that inherit from models.Model.

For this project we will create two models, one for each Django app:

Modelo	Fields	Description
Post	`'title', 'content'`	It represents a blog post.
Comment	`'post', 'content'`	Represents a comment associated with a post.

In the archive:/my_project_blog/apps/posts/models.py :

class Post(models.Model):
    title = models.CharField(max_length=200)
    content = models.TextField()
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.title

In the archive:/my_project_blog/apps/comments/models.py :

from apps.posts.models import Post


class Comment(models.Model):
    post = models.ForeignKey(Post, related_name="comments", on_delete=models.CASCADE)
    content = models.TextField()
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.content[:20]

### Views

The View handles user requests and returns responses. Django offers both class-based views (CBV) and function-based views (FBV).

CommentListView and CommentDetailView are examples of Class-Based Views (CBV).

Separation of concerns is a design principle that promotes the separation of a program into distinct sections, with each section addressing a separate concern.

As we saw earlier for this project, we separated the views into two:

API REST Views: Handle API-specific logic, such as serialisation, validation and returning JSON responses.
Traditional Django Views (Web Views): They handle rendering templates, session management and other web-specific logic.

For the Posts app:

In the archive:/my_project_blog/apps/posts/views/api_views.py :

from rest_framework import generics
from ..serializers import PostSerializer
from ..services.post_service import PostService

# API view for listing all posts and creating a new post.
# Utilizes Django REST Framework's ListCreateAPIView for listing and creating resources.
class PostListCreateAPIView(generics.ListCreateAPIView):
    serializer_class = PostSerializer  # Defines the serializer class used for converting model instances to JSON and vice versa.

    # Fetches all posts from the database.
    # `get_queryset` method specifies the queryset for listing posts.
    def get_queryset(self):
        return (
            PostService.get_all_posts()
        )  # Delegates the database query to the PostService layer.

    # Handles the creation of a new post.
    # `perform_create` is called after validation of the serializer.
    def perform_create(self, serializer):
        # Creates a new post using validated data from the serializer.
        PostService.create_post(
            serializer.validated_data
        )  # Uses PostService to handle creation logic.

# API view for retrieving, updating, and deleting a specific post.
# Extends RetrieveUpdateDestroyAPIView for detailed operations on a single resource.
class PostRetrieveUpdateDestroyAPIView(generics.RetrieveUpdateDestroyAPIView):
    serializer_class = PostSerializer  # Specifies the serializer class for retrieving, updating, and deleting resources.

    # Retrieves a post object based on the provided post_id.
    # `get_object` method returns the post instance for the specified post_id.
    def get_object(self):
        post_id = self.kwargs.get("post_id")  # Extracts post_id from the URL kwargs.
        post = PostService.get_post_by_id(
            post_id
        )  # Fetches the post using PostService.
        if post is None:
            raise NotFound(
                "Comment not fund"
            )  # Raises a 404 error if the post does not exist.
        return post  # Returns the post instance.

    # Updates an existing post instance.
    # `perform_update` is called after the serializer's data is validated.
    def perform_update(self, serializer):
        post_id = self.kwargs["post_id"]  # Extracts post_id from the URL kwargs.
        # Updates the post with new data.
        PostService.update_post(
            serializer.validated_data, post_id
        )  # Delegates the update logic to PostService.

    # Deletes a post instance.
    # `perform_destroy` is called to delete the specified post.
    def perform_destroy(self, instance):
        post_id = self.kwargs["post_id"]  # Extracts post_id from the URL kwargs.
        # Deletes the post using PostService.
        PostService.delete_post(post_id)  # Delegates the deletion logic to PostService.

In the archive:/my_project_blog/apps/posts/views/web_views.py :

from django.views.generic import ListView, DetailView
from ..models import Post
from ..services.post_service import PostService

# Class-based view for listing all posts in the web interface.
# Utilizes Django's ListView to handle displaying a list of posts.
class PostListView(ListView):
    model = Post  # Specifies the model to be used in the view.
    template_name = (
        "posts/post_list.html"  # Path to the template for rendering the list of posts.
    )
    context_object_name = "posts"  # Context variable name to be used in the template.

    # Overrides the default get_queryset method to fetch all posts from the service layer.
    def get_queryset(self):
        return (
            PostService.get_all_posts()
        )  # Delegates the database query to the PostService.

# Class-based view for displaying the details of a single post.
# Utilizes Django's DetailView to handle displaying detailed information of a single post.
class PostDetailView(DetailView):
    model = Post
    template_name = "posts/post_detail.html"
    context_object_name = "post"

    # Overrides the default get_object method to fetch a specific post based on post_id.
    def get_object(self, queryset=None):
        post_id = self.kwargs.get("post_id")  # Extracts post_id from the URL kwargs.
        return PostService.get_post_by_id(
            post_id
        )  # Fetches the post using the PostService.

For the Comments app:

In the archive:/my_project_blog/apps/comments/views/api_views.py :

from rest_framework import generics
from ..serializers import CommentSerializer
from ..services.comment_service import CommentService
from rest_framework.exceptions import NotFound

class CommentListCreateAPIView(generics.ListCreateAPIView):
    serializer_class = CommentSerializer

    def get_queryset(self):
        # Retrieve the 'post_id' from the URL kwargs. This ID is used to filter comments related to a specific post.
        post_id = self.kwargs.get("post_id")
        # Fetch comments related to the given post ID using the CommentService. The repository layer handles actual data fetching.
        return CommentService.get_comments_by_post_id(post_id)

    def perform_create(self, serializer):
        # Retrieve the 'post_id' from the URL kwargs. This ID is used to associate the new comment with a specific post.
        post_id = self.kwargs.get("post_id")
        # Create a new comment for the specified post using the CommentService. The service layer handles data manipulation.
        CommentService.create_comment(serializer.validated_data, post_id)

class CommentRetrieveUpdateDestroyAPIView(generics.RetrieveUpdateDestroyAPIView):
    serializer_class = CommentSerializer

    def get_object(self):
        # Retrieve the 'post_id' and 'comment_pk' from the URL kwargs.
        # 'post_id' is used to ensure the comment belongs to the post, while 'comment_pk' identifies the specific comment.
        post_id = self.kwargs.get("post_id")
        comment_id = self.kwargs.get("comment_pk")
        # Fetch the specific comment for the given post ID and comment ID using the CommentService.
        # Raise a 404 error if the comment is not found.
        comment = CommentService.get_comment_by_post_and_id(post_id, comment_id)
        if comment is None:
            raise NotFound("Comment not fund")
        return comment

    def perform_update(self, serializer):
        # Retrieve the 'comment_pk' from the URL kwargs for updating the specific comment.
        comment_id = self.kwargs["comment_pk"]
        # Update the specified comment using the CommentService. The service layer handles data manipulation.
        CommentService.update_comment(serializer.validated_data, comment_id)

    def perform_destroy(self, instance):
        # Retrieve the 'comment_pk' from the URL kwargs for deleting the specific comment.
        comment_id = self.kwargs["comment_pk"]
        # Delete the specified comment using the CommentService. The service layer handles data manipulation.
        CommentService.delete_comment(comment_id)

In the archive:/my_project_blog/apps/comments/views/api_views.py :

from django.views.generic import ListView, DetailView
from ..models import Comment
from ..services.comment_service import CommentService

class CommentListView(ListView):
    model = Comment
    template_name = "comments/comment_list.html"
    context_object_name = "comments"

    def get_queryset(self):
        # Extract 'post_id' from URL parameters to fetch comments associated with a specific post.
        post_id = self.kwargs.get("post_id")
        # Use the CommentService to retrieve comments for the specified post.
        # The service layer handles data access logic, keeping the view simple.
        return CommentService.get_comments_by_post_id(post_id)

    def get_context_data(self, **kwargs):
        # Get the default context from the parent class and add additional context for 'post_id'.
        context = super().get_context_data(**kwargs)
        # Include 'post_id' in the context so it can be used in the template.
        # This is necessary for rendering links or forms related to the post.
        context["post_id"] = self.kwargs.get("post_id")
        return context

class CommentDetailView(DetailView):
    model = Comment
    template_name = "comments/comment_detail.html"
    context_object_name = "comment"

    def get_object(self, queryset=None):
        # Extract 'post_id' and 'comment_id' from URL parameters to retrieve a specific comment for a post.
        post_id = self.kwargs.get("post_id")
        comment_id = self.kwargs.get("comment_id")
        # Use the CommentService to retrieve the specific comment based on 'post_id' and 'comment_id'.
        # If the comment is not found, `CommentService.get_comment_by_post_and_id` will return None.
        # In this case, a 404 error will be raised automatically by the `DetailView` if the object is not found.
        return CommentService.get_comment_by_post_and_id(post_id, comment_id)

### Repositories

Repositories handle data persistence and encapsulate data access logic. They are defined in separate files (repositories.py) within each application.

For the Post app:

In the archive:/my_project_blog/apps/posts/repositories/post_repository.py :

from ..models import Post

class PostRepository:
    # Fetch all posts from the database
    @staticmethod
    def get_all_posts():
        return Post.objects.all()

    # Fetch a specific post by its primary key (ID)
    @staticmethod
    def get_post_by_id(post_id):
        return Post.objects.get(pk=post_id)

    # Create a new post with the provided data
    @staticmethod
    def create_post(data):
        return Post.objects.create(**data)

    # Update an existing post with the provided data
    @staticmethod
    def update_post(data, post_id):
        post = Post.objects.get(pk=post_id)
        for attr, value in data.items():
            setattr(post, attr, value)  # Dynamically update each attribute
        post.save()
        return post

    # Delete a post by its primary key (ID)
    @staticmethod
    def delete_post(post_id):
        post = Post.objects.get(pk=post_id)
        post.delete()

For the Comments app:

In the archive:/my_project_blog/apps/comments/repositories/comment_repository.py :

from ..models import Comment

class CommentRepository:
    @staticmethod
    def get_comments_by_post_id(post_id):
        # Retrieve all comments associated with a specific post_id
        return Comment.objects.filter(post_id=post_id)

    @staticmethod
    def get_comment_by_post_and_id(post_id, comment_id):
        # Retrieve a specific comment by post_id and comment_id
        try:
            return Comment.objects.get(post_id=post_id, id=comment_id)
        except Comment.DoesNotExist:
            # Return None if no comment is found
            return None

    @staticmethod
    def create_comment(data, post_id):
        # Create a new comment associated with a specific post_id
        return Comment.objects.create(post_id=post_id, **data)

    @staticmethod
    def update_comment(data, comment_id):
        # Update an existing comment identified by comment_id
        comment = Comment.objects.get(pk=comment_id)
        for attr, value in data.items():
            setattr(comment, attr, value)
        comment.save()
        return comment

    @staticmethod
    def delete_comment(comment_id):
        # Delete an existing comment identified by comment_id
        comment = Comment.objects.get(pk=comment_id)
        comment.delete()

### Services

Services contain the business logic and act as intermediaries between views and repositories. They are defined in separate files (services.py) within each application.

For the Post app:

In the archive:/my_project_blog/apps/posts/services/post_service.py :

from ..repositories.post_repository import PostRepository

class PostService:
    @staticmethod
    def get_all_posts():
        # Retrieve all posts from the repository
        return PostRepository.get_all_posts()

    @staticmethod
    def get_post_by_id(post_id):
        # Retrieve a post by its ID from the repository
        return PostRepository.get_post_by_id(post_id)

    @staticmethod
    def create_post(data):
        # Create a new post with the given data
        return PostRepository.create_post(data)

    @staticmethod
    def update_post(data, post_id):
        # Update an existing post with the given data
        return PostRepository.update_post(data, post_id)

    @staticmethod
    def delete_post(post_id):
        # Delete a post by its ID from the repository
        return PostRepository.delete_post(post_id)

For the Comments app:

In the archive:/my_project_blog/apps/comments/services/comment_service.py :

from ..repositories.comment_repository import CommentRepository

class CommentService:
    @staticmethod
    def get_comments_by_post_id(post_id):
        # Delegate the task of retrieving comments by post_id to the repository layer
        return CommentRepository.get_comments_by_post_id(post_id)

    @staticmethod
    def get_comment_by_post_and_id(post_id, comment_id):
        # Delegate the task of retrieving a specific comment by post_id and comment_id to the repository layer
        return CommentRepository.get_comment_by_post_and_id(post_id, comment_id)

    @staticmethod
    def create_comment(data, post_id):
        # Delegate the task of creating a new comment to the repository layer
        return CommentRepository.create_comment(data, post_id)

    @staticmethod
    def update_comment(data, comment_id):
        # Delegate the task of updating an existing comment to the repository layer
        return CommentRepository.update_comment(data, comment_id)

    @staticmethod
    def delete_comment(comment_id):
        # Delegate the task of deleting a comment to the repository layer
        return CommentRepository.delete_comment(comment_id)

### Serializers

Serializers convert complex data into native data formats (JSON) and vice versa. Serialisers are an important part of creating REST APIs with DRF which are not usually used in a normal Django project.

For the Post app:

In the archive:/my_project_blog/apps/posts/serializers.py

from rest_framework import serializers
from .models import Post

# Serializer for the Post model
# This class is responsible for converting Post instances into JSON data and validating incoming data for creating or updating posts.
class PostSerializer(serializers.ModelSerializer):

    # Meta class specifies the model and fields to be used by the serializer.
    class Meta:
        model = Post  # The model associated with this serializer. It tells DRF which model the serializer will be handling.
        fields = "__all__"  # Specifies that all fields from the model should be included in the serialization and deserialization process.

For the Comments app:

In the archive:/my_project_blog/apps/comments/serializers.py :

from rest_framework import serializers
from .models import Comment

# CommentSerializer is a ModelSerializer that automatically creates fields and methods for the Comment model.
class CommentSerializer(serializers.ModelSerializer):
    class Meta:
        model = Comment  # The model that this serializer will be based on.
        fields = "__all__"  # Automatically include all fields from the Comment model.

### Configuration of URLs

URLs are organised into two separate files for the API and web views, allowing a clear separation between accessing data via the API and presenting data via web views. This organisation facilitates code management and maintenance by distinguishing between paths that serve JSON data and paths that present HTML views.

API routes are defined only in the Posts application to centralise the management of resources related to posts and their comments. By not having its own urls.py file in the Comments application, redundancy is reduced and the project structure is simplified by grouping all related API paths in one place. This makes it easier to understand the data flow and the interrelationship between posts and comments, promoting a more coherent and maintainable architecture, and the correct use of REST best practices.

URLs API REST:

In the archive:/my_project_blog/apps/posts/urls/api_urls.py :

from django.urls import path
from ..views.api_views import PostListCreateAPIView, PostRetrieveUpdateDestroyAPIView
from ...comments.views.api_views import (
    CommentListCreateAPIView,
    CommentRetrieveUpdateDestroyAPIView,
)

urlpatterns = [
    # Route for listing all posts or creating a new post
    path("", PostListCreateAPIView.as_view(), name="post-list-create"),
    # Route for retrieving, updating, or deleting a specific post by post_id
    path(
        "<int:post_id>/",
        PostRetrieveUpdateDestroyAPIView.as_view(),
        name="post-retrieve-update-destroy",
    ),
    # Route for listing all comments for a specific post or creating a new comment for that post
    path(
        "<int:post_id>/comments/",
        CommentListCreateAPIView.as_view(),
        name="post-comment-create",
    ),
    # Route for retrieving, updating, or deleting a specific comment by comment_pk for a specific post
    path(
        "<int:post_id>/comments/<int:comment_pk>/",
        CommentRetrieveUpdateDestroyAPIView.as_view(),
        name="post-comment-retrieve-update-destroy",
    ),
]

URLs web:

In the archive:/my_project_blog/apps/posts/urls/web_urls.py :

from django.urls import path
from ..views.web_views import PostListView, PostDetailView
from ...comments.views.web_views import CommentListView, CommentDetailView

urlpatterns = [
    # Route for listing all posts.
    # The URL is "web/posts/", and it maps to PostListView to display a list of all posts.
    path("", PostListView.as_view(), name="post-list"),
    # Route for displaying details of a specific post identified by post_id.
    # The URL is "web/posts/<post_id>/", and it maps to PostDetailView to display details of a single post.
    path("<int:post_id>/", PostDetailView.as_view(), name="post-detail"),
    # Route for listing all comments for a specific post identified by post_id.
    # The URL is "web/posts/<post_id>/comments", and it maps to CommentListView to display a list of comments for the given post.
    path(
        "<int:post_id>/comments",
        CommentListView.as_view(),
        name="post-comments",
    ),
    # Route for displaying details of a specific comment identified by comment_id for a specific post.
    # The URL is "web/posts/<post_id>/comments/<comment_id>/", and it maps to CommentDetailView to display details of a single comment.
    path(
        "<int:post_id>/comments/<int:comment_id>/",
        CommentDetailView.as_view(),
        name="post-comment-detail",
    ),
]

General project URLs:

In the archive:/my_project_blog/my_project_blog/urls.py :

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("admin/", admin.site.urls),
    path("posts/", include("apps.posts.urls.web_urls")),
    path("api/posts/", include("apps.posts.urls.api_urls")),
]

Web application and API testing

Now, in the third section of this blog, we will create some posts and add several comments to each of the posts to verify the correct functioning of the web application and the API and web URLs.

Execute the project

- Creates Migrations:
```
python manage.py makemigrations
```
- Apply migrations:
```
python manage.py migrate
```
- Run the Django development server:
```
python manage.py runserver
```
Note: Open your browser and navigate to http://127.0.0.1:8000/, from now on the urls that I will give you will be after the domain (in this case our localhost), example: If I give you the url /api/posts/, in the browser you will have to put http://127.0.0.1:8000/api/posts/.

Creation of posts

Access the Django REST API interface:

* **URL:**`/api/posts/`

* **Action:** Fill in the form with the details of the post to be created and click on the "POST" button.

Add comments to posts:

Access the Django REST API interface:

* **URL:**`/api/posts/{post_id}/comments/`

* **Action:** Fill in the form with the details of the comment to be created for a specific post and click on the "POST" button.

You can repeat these steps to create as many posts and comments as you want.

Also, in this same Django REST API interface you can edit posts or comments or delete them.

### Verify the creation of posts and comments:

Having already created several posts and comments for some of these via the REST API, we can see these in the web URLs that interact with the HTML templates.

So that when you enter the given URLs you can see the same design, you can see the HTML code of each template in the HTML files of the project.

Full project code on GitHub: Click.

1. See the list of posts:
```
* **URL:**`/posts/`
```

2. **See the details of a specific post:**

    * **URL:**`/posts/{post_id}/`

3. **See comments on a specific post:**

    * **URL:**`/posts/{post_id}/comments/`

4. **View the details of a comment on a specific post:**

    * **URL:**`/posts/{post_id}/comments/{comment_id}/`

Conclusions

Benefits of the Services and Repositories Employer

Separation of Responsibilities:

Separation of responsibilities is one of the fundamental principles in software design. In the Services and Repositories pattern, we clearly separate data access concerns (repositories) from business logic (services).

Benefits:

* **Clarity and Organisation:** The code is organised so that each class and function has a single, clearly defined responsibility. This makes the code easier to understand and maintain.

* **Maintainability:** When the business logic changes, only the services need to be modified, while the repositories remain unchanged. This reduces the number of changes and the risk of introducing errors.

* **Team Collaboration:** In a development team, different members can work at different layers (e.g. one on services and one on repositories) without interfering with each other, facilitating collaboration and integration.

Re-use of the Code:

The pattern encourages code reuse by centralising business logic and data access in specific classes. This avoids code duplication and facilitates code reuse in different parts of the application.

Benefits:

* **Duplication Reduction:** By having business logic in services and data access in repositories, we avoid repeating code in multiple places, making the code more DRY (Don't Repeat Yourself).

* **Consistency:** Reusing the same code in different parts of the application ensures that the same logic and rules are followed everywhere, which increases consistency and reduces errors.

* **Ease of Update:** If you need to change the way data is accessed or business logic is implemented, you only need to update the corresponding repository or service. The other parts of the application that depend on them will automatically benefit from the changes.

Ease of Testing:

The separation of business logic and data access logic facilitates the creation of unit and integration tests. Services and repositories can be tested in isolation, which simplifies the testing process.

Benefits:

* **Unit testing:** By having business logic in services and data access in repositories, it is easier to create unit tests for each component independently. This makes it possible to quickly detect errors and ensure that each part works correctly.

* **Mocks and Stubs:** During testing, it is easy to use mocks and stubs to simulate the behaviour of repositories or services. This allows testing business logic without relying on the database or other external services.

* **Reduced Testing Complexity:** By having clear and well-defined responsibilities, testing becomes less complex and more specific. This improves test coverage and software reliability.

Maintainability and Extensibility:

The Services and Repositories pattern makes code more maintainable and extensible. This is especially important in long-term projects, where requirements may change over time.

Benefits:

* **Code Evolution:** When you need to add new functionality, it is easy to extend existing services and repositories without affecting the rest of the application. This allows the code to evolve in a controlled and safe way.

* **Easy Refactoring:** If you identify an improvement in the code structure or business logic implementation, it is easy to refactor services and repositories without great risk. The separation of responsibilities makes it easy to identify which parts of the code need to be changed.

* **Adaptability to New Technologies:** If at some point you decide to change the persistence technology (for example, from a SQL database to a NoSQL one), you only need to modify the repositories without affecting the business logic implemented in the services.

Finally, in closing, I thank you for your time and attention in following this guide to the Services and Repositories design pattern in a Django project with the Django REST Framework. I hope this detailed explanation has given you a solid understanding of how to structure your applications in an efficient and maintainable way.

Implementing this pattern will not only improve the organization of your code, but also facilitate its evolution and scalability as your project grows. If you have any additional questions or opinions on the topic, feel free to post in the comments, they are left open for discussion.

Good luck with your development and best of luck with your future projects!

See you next time!

How to Name Endpoints in a REST API: Complete Guide with Best Practices and Practical Example

Mateo Ramirez Rubio — Sun, 07 Jul 2024 18:57:46 +0000

Introduction:

Naming the endpoints of a REST API may seem like a simple task, but in reality, it is one of the keys to ensuring that your API is intuitive, easy to use and maintain. Imagine trying to navigate a system where the paths don't make sense or don't follow a logical pattern. It would be like trying to find your way in a city with no road signs. In this blog, we'll look at some best practices for naming your endpoints effectively, ensuring that your users and developers can interact with your API clearly and efficiently. At the end, you'll see a complete example that you can use as a practical exercise to apply what you've learned - read on to master the art of naming endpoints in REST APIs!

Introduction.
Recommendations for naming your endpoints.
1. Represent resources and not actions.
2. Nested hierarchical structure.
3. Use of clear and descriptive names.
4. Consistency in the use of upper and lower case letters.
5. Avoid file extensions.
6. Endpoint naming formula.
7. Complete REST API endpoint naming design example.
Acknowledgments.

Recommendations for naming your endpoints

When we design a REST API, we want it to be easy to understand and use. To achieve this we must follow some rules to name our URLs in a consistent way.

Represent resources and not actions (resources should be named in plural):

You can fall into the trap of using actions or verbs when naming your endpoints such as:

‎

~~How not to do it:~~

GET /deleteBook/321

GET /deleteBook?id=321

POST /books/321/delete

‎

Following the REST parameters, this is not well done, the correct way to do it for this example is that there is a resource called Books and through the HTTP methods we say the action we want to do as for example delete it (DELETE) like this:

‎

How to do it right:

DELETE /books/321

‎
Nested hierarchical structure:

To maintain good consistency and coherence in our URLs, we must use a hierarchical structure that shows relationships between resources. Example:

‎

~~How not to do it:~~

/books?libraryId=123

/comments?bookId=456

‎

How to do it right:

/libraries/123/books

/books/456/comments

‎
Use of clear and descriptive names:

Continuing with consistency in the naming of endpoints and knowing that verbs should not be used to name resources, remember to use names that clearly describe the resources.

‎

~~How not to do it:~~

/data

/info

‎

How to do it right:

/users

/products

‎
Consistency in the use of upper and lower case letters:

~~How not to do it:~~ Mixing upper and lower case inconsistently:

/UserProfiles

/Order_Details

‎

How to do it right: Use lowercase letters and hyphens consistently:

/user-profiles

/order-details

‎
Avoid file extensions:

It may seem very obvious, but let's not overlook the fact that we should not use file extensions in the resource name in our URL, we should let the content type be determined by the HTTP headers.

‎

~~How not to do it:~~

/users.json

/orders.xml

‎

How to do it right:

/users

/orders

As a more visual aid, following the conventions, we can use the following formula for naming our URLs:

‎

/{collection} /{store} /{document}/{controller}

‎

Collection (plural): A set of resources:
- /books
Store (plural): Contains resources managed by the client (stores are not used as much):
- /users/12/favorites
Document (singular): An instance in a collection:
- /books/2
- /books/maths
Controller (verb): Executable functions (usually when we use controllers the HTTP method we use is POST):
- /users/12/reset-password

Complete example of how to name our endpoints:

In this section we will see a complete example of an api design, first we will see how a bad REST API design would look like and then its correction.

Note: Before seeing how a well designed REST API would look like, correct yourself the errors of the badly designed API and compare at the end. This way you will learn more!

‎

~~Poorly Designed API:~~

Collection of books:
- Obtain all books:
  - GET /getAllBooks
- Create a new book:
  - POST /createBook
A user's favorite books:
- Get a specific user's favorite books:
  - GET /getUserFavorites?userId=12
- Add a book to the favorites of a specific user:
  - POST /addFavorite?userId=12&bookId=123
Specific Book
- Obtain a specific book:
  - GET /getBookById?bookId=123
- Update a specific book:
  - POST /updateBook?bookId=123
- Delete a specific book:
  - POST /deleteBook?bookId=123
Action/Controller in Specific Book
- Mark a book as read:
  - POST /markBookAsRead?bookId=123
- Request a loan extension:
  - POST /requestExtension?bookId=123

Problems with Bad Design

Verbs in URLs: URLs should represent resources, not actions.
Inappropriate use of HTTP methods: POST is used for update and delete, instead of PUT and DELETE.
Use of query parameters: Query parameters (?bookId=123) are being used instead of resource identifiers in the path.

‎

Now we will see the API as it would be well designed.

‎

API well designed:

Book Collection:
- Obtain all books:
  - GET /books
- Create a new book:
  - POST /books
User Favorite Books
- Get a specific user's favorite books:
  - GET /users/12/favorites
- Add a book to the favorites of a specific user:
  - POST /users/12/favorites
Specific Book
- Obtain a specific book:
  - GET /books/123
- Update a specific book:
  - PUT /books/123
- Delete a specific book:
  - DELETE /books/123
Action/Controller in Specific Book
- Mark a book as read:
  - POST /books/123/mark-as-read
- Request a loan extension:
  - POST /books/123/request-extension

Benefits of Good Design

Clarity: URLs are clear and represent resources.
Correct use of HTTP methods: GET, POST, PUT, and DELETE are used properly.
Identifiers in the path: Resource identifiers are included in the path, not as query parameters.
Logical structure: The hierarchical and logical structure facilitates the understanding and use of the API.

‎

Thus, by following the right structure and proper use of HTTP methods, the well-designed API is more intuitive, scalable and easy to maintain.

I hope this guide has provided you with the tools you need to design clear and consistent endpoints for your REST API. Good endpoint naming not only improves the user experience, but also facilitates the maintenance and scalability of your system. Remember that consistency and clarity are essential in API development. Don't forget to practice with the full example. Until next time, and happy coding!

Forem: Mateo Ramirez Rubio

Configuring AWS VPC, Load Balancers, and DNS for WordPress and Moodle Integration

Table of Contents

List of Figures

Introduction

System Architecture

AWS Resource Map

Implementation and Deployment

VPC and Subnet Configuration

DNS Server Configuration

Load Balancer Configuration (HAProxy)

WordPress Installation and Configuration

Moodle Installation and Configuration

Verification and Results

Issues Encountered, Troubleshooting, and Recommendations

Conclusions

Future Work

References

Quick Introduction to Parallel Computing in Go: A Practical Approach with Goroutines, WaitGroups, Mutexes and Atomic Operations

Introduction

Introduction to Parallel Computing

Parallelism vs. Concurrency

Parallel Computing in Go

Key Components in the Code

MateoRamirezRubio1 / learning_Go_Golang

Goroutines

WaitGroups

Mutexes

Atomic Operations

Parallel Execution in Action: Mapping, Filtering and Reduction

Parallel Mapping

Parallel Filtering

Parallel Reduction

Detailed Explanation of main

Examples of results and numWorkers

Use Cases and Considerations

Practical Example

Execution and result verification

Execution Explanation

Checking the Result

Warning about numWorkers

Conclusion

How would you improve this code to make it even more efficient or robust?

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Mateo Ramirez Rubio ・ Jul 22

Using Environment Variables with flutter_dotenv and DotEnv in Flutter: Step by Step Guide

Introduction

Some concepts

What Are .env Files?

Why Use .env Files?

Step-by-Step Guide

Prerequisites

Step 1: Add the dotenv package

Step 2: Create a .env File

Step 3: Update pubspec.yaml

Step 4: Load the .env File

Step 5: Access Environment Variables

Troubleshooting

Common Issues:

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Mateo Ramirez Rubio ・ Jul 22

How to Name Endpoints in a REST API: Complete Guide with Best Practices and Practical Example

Mateo Ramirez Rubio ・ Jul 7

Mastering Testing in Django REST Framework: From Pytest and Fixtures to Mocks and Coverage

Introduction

Complete Guide to the Django Services and Repositories Design Pattern with the Django REST Framework

Mateo Ramirez Rubio ・ Jul 8

Contents

Introduction to Testing with Pytest

What is Pytest?

Advantages of Using Pytest:

Integration with Django

Initial setup for Testing with Pytest in a Django project with Django REST Framework

Installing the requirements

Pytest configuration

Structure

MateoRamirezRubio1 / mini-blog-rest-api

A mini blog Django project demonstrating the implementation of the Services and Repositories design pattern for a blog application.

Features

Project Structure

Warning about `numWorkers`

What Are `.env` Files?

Why Use `.env` Files?

Step 1: Add the `dotenv` package

Step 2: Create a `.env` File

Step 3: Update `pubspec.yaml`

Step 4: Load the `.env` File

`@pytest.mark.django_db`

Using `mocker`

Command 1:`python -m coverage run -m pytest`:

Command 2:`python -m coverage report --include=‘apps/’ --show-missing`: