Forem: vamsikrishna71

Neural Network Model Using PyTorch

vamsikrishna71 — Thu, 23 Nov 2023 05:41:10 +0000

Overview:

Tensor Basics
- Create, Operations, NumPy, GPU Support
Autograd
- Linear regression example
Training Loop with: Model, Loss & Optimizer
- A typical PyTorch training pipeline
Neural Network
- Also: GPU, Datasets, DataLoader, Transforms & Evaluation
Convolutional Neural Network
- Also: Save/Load model

1. Tensors

Everything in PyTorch is based on Tensor operations. A Tensor is a multi-dimensional matrix containing elements of a single data type:

import torch

# torch.empty(size): uninitiallized
x = torch.empty(1) # scalar
print("empty(1):", x)
x = torch.empty(3) # vector
print("empty(3):",x)
x = torch.empty(2, 3) # matrix
print("empty(2,3):",x)
x = torch.empty(2, 2, 3) # tensor, 3 dimensions
#x = torch.empty(2,2,2,3) # tensor, 4 dimensions
print("empty(2, 2, 3):",x)

# torch.rand(size): random numbers [0, 1]
x = torch.rand(5, 3)
print("rand(5,3):", x)

# torch.zeros(size), fill with 0
# torch.ones(size), fill with 1
x = torch.zeros(5, 3)
print("zeros(5,3):", x)

# check size
print("size", x.size())  # x.size(0)
print("shape", x.shape)  # x.shape[0]

# check data type
print(x.dtype)

# specify types, float32 default
x = torch.zeros(5, 3, dtype=torch.float16)
print(x)

# check type
print(x.dtype)

# construct from data
x = torch.tensor([5.5, 3])
print(x, x.dtype)

# requires_grad argument
# This will tell pytorch that it will need to calculate the gradients for this tensor
# later in your optimization steps
# i.e. this is a variable in your model that you want to optimize
x = torch.tensor([5.5, 3], requires_grad=True)
print(x)

Operations with Tensors

# Operations
x = torch.ones(2, 2)
y = torch.rand(2, 2)

# elementwise addition
z = x + y
# torch.add(x,y)

# in place addition, everythin with a trailing underscore is an inplace operation
# i.e. it will modify the variable
# y.add_(x)

print(x)
print(y)
print(z)

# subtraction
z = x - y
z = torch.sub(x, y)

# multiplication
z = x * y
z = torch.mul(x,y)

# division
z = x / y
z = torch.div(x,y)

# Slicing
x = torch.rand(5,3)
print(x)
print("x[:, 0]", x[:, 0]) # all rows, column 0
print("x[1, :]", x[1, :]) # row 1, all columns
print("x[1, 1]", x[1,1]) # element at 1, 1

# Get the actual value if only 1 element in your tensor
print("x[1,1].item()", x[1,1].item())

# Reshape with torch.view()
x = torch.randn(4, 4)
y = x.view(16)
z = x.view(-1, 8)  # the size -1 is inferred from other dimensions
# if -1 it pytorch will automatically determine the necessary size
print(x.size(), y.size(), z.size())

NumPy

Converting a Torch Tensor to a NumPy array and vice versa is very easy

a = torch.ones(5)
print(a)

# torch to numpy with .numpy()
b = a.numpy()
print(b)
print(type(b))

# Careful: If the Tensor is on the CPU (not the GPU),
# both objects will share the same memory location, so changing one
# will also change the other
a.add_(1)
print(a)
print(b)

# numpy to torch with .from_numpy(x), or torch.tensor() to copy it
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
c = torch.tensor(a)
print(a)
print(b)
print(c)

# again be careful when modifying
a += 1
print(a)
print(b)
print(c)

GPU Support

By default all tensors are created on the CPU. But we can also move them to the GPU (if it's available ), or create them directly on the GPU.

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

x = torch.rand(2,2).to(device)  # move tensors to GPU device
#x = x.to("cpu")
#x = x.to("cuda")

x = torch.rand(2,2, device=device)  # or directy create them on GPU

2. Autograd

The autograd package provides automatic differentiation for all operations on Tensors. Generally speaking, torch.autograd is an engine for computing the vector-Jacobian product. It computes partial derivates while applying the chain rule.

Set requires_grad = True:

import torch

# requires_grad = True -> tracks all operations on the tensor.
x = torch.randn(3, requires_grad=True)
y = x + 2

# y was created as a result of an operation, so it has a grad_fn attribute.
# grad_fn: references a Function that has created the Tensor
print(x) # created by the user -> grad_fn is None
print(y)
print(y.grad_fn)

# Do more operations on y
z = y * y * 3
print(z)
z = z.mean()
print(z)

# Let's compute the gradients with backpropagation
# When we finish our computation we can call .backward() and have all the gradients computed automatically.
# The gradient for this tensor will be accumulated into .grad attribute.
# It is the partial derivate of the function w.r.t. the tensor

print(x.grad)
z.backward()
print(x.grad) # dz/dx

# !!! Careful!!! backward() accumulates the gradient for this tensor into .grad attribute.
# !!! We need to be careful during optimization !!! optimizer.zero_grad()

Stop a tensor from tracking history:

For example during the training loop when we want to update our weights, or after training during evaluation. These operations should not be part of the gradient computation. To prevent this, we can use:

x.requires_grad_(False)
x.detach()
wrap in with torch.no_grad():

# .requires_grad_(...) changes an existing flag in-place.
a = torch.randn(2, 2)
b = (a * a).sum()
print(a.requires_grad)
print(b.grad_fn)

a.requires_grad_(True)
b = (a * a).sum()
print(a.requires_grad)
print(b.grad_fn)

# .detach(): get a new Tensor with the same content but no gradient computation:
a = torch.randn(2, 2, requires_grad=True)
b = a.detach()
print(a.requires_grad)
print(b.requires_grad)

# wrap in 'with torch.no_grad():'
a = torch.randn(2, 2, requires_grad=True)
print(a.requires_grad)
with torch.no_grad():
    b = a ** 2
    print(b.requires_grad)

Gradient Descent Autograd

Linear Regression example:

$f(x) = w * x + b$

here : f(x) = 2 * x

import torch

# Linear regression
# f = w * x  + b
# here : f = 2 * x

X = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8], dtype=torch.float32)
Y = torch.tensor([2, 4, 6, 8, 10, 12, 14, 16], dtype=torch.float32)

w = torch.tensor(0.0, dtype=torch.float32, requires_grad=True)

# model output
def forward(x):
    return w * x

# loss = MSE
def loss(y, y_pred):
    return ((y_pred - y)**2).mean()

X_test = 5.0

print(f'Prediction before training: f({X_test}) = {forward(X_test).item():.3f}')

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz
100%|██████████| 9912422/9912422 [00:00<00:00, 116079447.08it/s]
Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz
100%|██████████| 28881/28881 [00:00<00:00, 120413214.54it/s]
Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz
100%|██████████| 1648877/1648877 [00:00<00:00, 28916457.87it/s]
Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz
100%|██████████| 4542/4542 [00:00<00:00, 5074727.96it/s]
Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw

To be continued...

Supercharge Your Spreadsheet Skills with Python in Microsoft Excel

vamsikrishna71 — Thu, 24 Aug 2023 04:07:38 +0000

Introduction

Microsoft Excel has long been the go-to tool for data analysis, financial modeling, and creating visually appealing spreadsheets. While it offers a wide range of functionalities and features, there are times when you need more flexibility and power to tackle complex tasks. This is where Python, one of the most versatile and widely-used programming languages, comes into play. In recent years, Python has been seamlessly integrated into Microsoft Excel, offering users the ability to supercharge their spreadsheet skills and automate tasks like never before.

In this article, we'll explore the integration of Python into Excel, discuss its benefits, and walk you through some practical examples of how you can use Python to enhance your Excel experience.

The Power of Python in Excel

Python is known for its simplicity and versatility. It excels at data manipulation, analysis, and automation – all skills that are in high demand for Excel users. By integrating Python into Excel, you can:

Automate Repetitive Tasks: Python can be used to automate repetitive and time-consuming tasks in Excel, such as data cleaning, formatting, and report generation. This can save you hours of manual work.

Advanced Data Analysis: Python's rich ecosystem of libraries, such as NumPy, pandas, and matplotlib, allows for advanced data analysis and visualization directly within Excel. You can perform complex calculations and create interactive charts with ease.
Access External Data Sources: Python's extensive libraries can connect to external data sources, such as databases, APIs, and web scraping, and seamlessly import the data into Excel for further analysis.
Custom Functions: You can create custom Python functions and use them in your Excel formulas, extending Excel's capabilities. This is particularly useful for specialized calculations.
Machine Learning: For users interested in machine learning, Python in Excel enables you to build, train, and deploy machine learning models for predictive analysis and decision-making.

How to Use Python in Microsoft Excel

To get started with Python in Excel, you'll need to follow these steps:

Install Python: If you haven't already, install Python on your computer. Make sure to select the option to add Python to your system's PATH during installation.
Install Excel Add-ins: There are several Excel add-ins available that facilitate the integration of Python, such as PyXLL, DataNitro, and xlwings. Choose one that suits your needs and follow the installation instructions.

Python in Excel with PyXLL, the Python Excel Add-in

Create Python Scripts: Once the add-in is installed, you can start writing Python scripts directly in Excel. These scripts can be used to automate tasks, perform data analysis, or create custom functions.
Run Python Code: Execute your Python code within Excel. You can link Python scripts to Excel buttons or macros for easy execution.
Import Python Libraries: You can import external Python libraries into your Excel environment, expanding your capabilities even further.

Practical Examples

Here are some practical examples of how Python can be used in Microsoft Excel:

Data Cleaning: Automate the process of cleaning and transforming messy data by writing Python scripts to remove duplicates, fill missing values, and standardize formats.

Financial Modeling: Build complex financial models using Python's mathematical libraries and Excel's user-friendly interface for inputs and outputs.
Interactive Dashboards: Create interactive dashboards with Python libraries like Plotly or Bokeh, allowing users to explore data dynamically.
Automated Reporting: Generate automated reports by pulling data from various sources using Python and presenting it in Excel-ready formats.
Custom Functions: Write custom Python functions to extend Excel's capabilities, such as calculating specialized financial metrics.

Conclusion

The integration of Python into Microsoft Excel opens up a world of possibilities for users who want to take their spreadsheet skills to the next level. By combining Excel's user-friendly interface with Python's power and versatility, you can automate tasks, perform advanced data analysis, and create custom solutions tailored to your specific needs.

Whether you're a data analyst, financial professional, or anyone who regularly uses Excel, learning to use Python within Excel can significantly enhance your productivity and problem-solving abilities. Embrace this integration, and you'll find yourself working smarter and more efficiently in no time.

Laravel DO's, DONT's and It's alternatives.

vamsikrishna71 — Fri, 26 May 2023 02:22:01 +0000

Laravel is a powerful and widely used PHP framework known for its elegant syntax, robust features, and developer-friendly environment. It has gained significant popularity in the web development community due to its ability to streamline the development process and promote code organization. However, like any tool, Laravel has its own set of best practices and pitfalls that developers should be aware of. In this article, we will explore the DOs and DON'Ts of Laravel and discuss some alternatives that developers can consider.

DOs:

<?php

namespace App\Http\Controllers;

use App\Models\User;
use Illuminate\Http\Request;
use App\Http\Controllers\Controller;

class UserController extends Controller
{
    /**
     * Display the specified user.
     *
     * @param  int  $id
     * @return \Illuminate\Http\Response
     */
    public function show($id)
    {
        $user = User::findOrFail($id);
        return view('users.show', compact('user'));
    }

    /**
     * Update the specified user in storage.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  int  $id
     * @return \Illuminate\Http\Response
     */
    public function update(Request $request, $id)
    {
        $user = User::findOrFail($id);
        $user->update($request->all());
        return redirect()->route('users.show', $id)->with('success', 'User updated successfully.');
    }
}

Follow Laravel conventions: Laravel follows a set of conventions for file and folder structure, naming conventions, and coding styles. Adhering to these conventions ensures consistency and makes it easier for other developers to understand and contribute to your codebase.
Utilize Laravel's ORM (Eloquent): Laravel's ORM, Eloquent, provides a simple and expressive way to interact with databases. Make use of Eloquent's features such as relationships, query scopes, and eager loading to simplify database operations and enhance performance.
Leverage Laravel's built-in security features: Laravel provides various security features out of the box, including CSRF protection, input validation, and encryption. Make sure to use these features to safeguard your application from common security threats.
Take advantage of Laravel's middleware: Middleware allows you to filter HTTP requests and add additional functionality to your application's request lifecycle. Utilize middleware to handle tasks such as authentication, authorization, and request validation.
Use Laravel's artisan command-line interface (CLI): Laravel's CLI tool, Artisan, offers a wide range of commands to generate code, run tests, manage migrations, and more. Familiarize yourself with Artisan's capabilities to boost productivity and automate repetitive tasks.

DON'Ts:

Overuse complex features: Laravel provides numerous advanced features like queues, events, and broadcasting. While these features can be powerful, avoid using them unnecessarily. Overcomplicating your codebase with unnecessary features may lead to maintenance issues and decreased performance.
Ignore Laravel's caching mechanisms: Laravel offers several caching mechanisms such as Memcached and Redis, which can significantly improve your application's performance. Neglecting to use caching when appropriate can result in slower response times and increased server load.
Neglect security best practices: While Laravel provides security features, it's crucial not to solely rely on them. Implement additional security measures like secure password hashing, proper access controls, and regular security updates to protect your application from potential vulnerabilities.
Neglect performance optimization: While Laravel is known for its performance, it's essential to optimize your code and database queries to ensure optimal performance. Avoid inefficient queries, implement caching strategies, and consider using tools like Laravel Debug bar to identify and resolve bottlenecks.

Alternatives to Laravel:

Symfony: Symfony is a robust PHP framework known for its flexibility and scalability. It offers a wide range of components and tools for building large-scale applications. Symfony emphasizes code reusability and follows best practices, making it an excellent choice for complex projects.
CodeIgniter: CodeIgniter is a lightweight PHP framework designed for developers who prefer a simple and straightforward approach. It has a small footprint and requires minimal configuration, making it ideal for small to medium-sized projects that prioritize simplicity and performance.
Yii: Yii is a high-performance PHP framework that focuses on speed and security. It provides features such as caching, authentication, and role-based access control out of the box, making development faster and more secure. Yii is well-suited for building large-scale applications that demand high performance.

In conclusion, Laravel is a powerful PHP framework that offers numerous benefits to developers. By following the DOs and avoiding the DON'Ts outlined above, developers can harness.

Laravel's capabilities effectively. However, it's important to consider alternatives like Symfony, CodeIgniter, and Yii based on the specific requirements of your project. Each alternative has its own strengths and weaknesses, so make sure to evaluate them carefully before making a decision.

Laravel eloquent multiple dependent model

vamsikrishna71 — Sun, 10 Jul 2022 11:45:44 +0000

Laravel eloquent multiple dependent model

How to Implement the Import CSV with Validation of Null Values?

Lets Start

What is CSV ?

CSV or Comma-Separated Values are text files delimited by using commas to separate values, but sometimes also using other characters such as semicolons. Each record consists of one or more fields separated by commas. The use of commas as separators (fields) is the source of the name for this file format. CSV files usually store tabular data (numbers and text) in plain text, in which case each row will have the same number of fields.

How to Import CSV in Laravel 9?

In laravel 9, to make a feature to import data from files in CSV format to a database or export data in CSV format, we can use the laravel-excel package. The ways of implementation will be discussed below.

Import in Laravel 9

Import CSV

    Step 1: Install Laravel
    Step 2: Setup Database
    Step 3: Create Models

Import CSV

Ok, the first feature we created is a feature to import data from a CSV format file to the users table in the database.

CSV to Array trait

Create Traits folder inside the App directory

namespace App\Traits;
trait CsvToArray
{
    /**
     * Converts the  CSV DATA to an Array with uploaded files
     *
     * @param string $filename
     * @param string $delimiter
     * @return array
     */
    public function csvToArray(string $filename = '', string $delimiter = ',')
    {
        if (!file_exists($filename) || !is_readable($filename)) {
            return false;
        }
        $header = null;
        $data = [];
        if (($handle = fopen($filename, 'r')) !== false) {
            while (($row = fgetcsv(
                $handle,
                1000,
                $delimiter
            )) !== false) {
                if (!$header)
                    $header = $row;
                else
                    $data[] = array_combine($header, $row);
            }
            fclose($handle);
        }
        return $data;
    }
}

Multiple Dependent Model

Which is Dependent with one on another data, which distribute along the Primary key of the parent table and foreign key on the child table.

 public function importEmployee(Request $request)
    {
        $request->validate(
            [
                'file' => ['required', 'mimes:csv,txt'],
            ]
        );
        $fileName = time() . '.' . $request->file->getClientOriginalExtension();
        $request->file->move(public_path('uploads'), $fileName);
        $csvFile   = public_path('uploads/' . $fileName);
        $employees = $this->csvToArray($csvFile);
        $user = User::find(Auth::user()->id);

In File upload input make field as accept = csv only.

Here, Dependent model Employee has the data from user to create the new employees.

foreach ($employees as $employee) {
            $user->employee()->create([
                'username'    =>  ' ' ? $employee['email'] : $employee['username'],
                'employeeId'  => $employee['employeeId'],
                'first_name'  => $employee['first_name'],
                'last_name'   => $employee['last_name'],
                'email'       => $employee['email'],
                'department'  => $employee['department'],
                'designation' => $employee['designation'],
                'password'    =>  ' ' ? Hash::make($employee['email']) : Hash::make($employee['password']),
            ]);
        }
        return redirect('employee.details')
            ->with('success', 'Employee updated Successfully');
    }

$user = User::find(Auth::user()->id);

$user->employee()->create

Above Two lines which are most important to understand.

In Employee Model file Employee.php

public function user()
    {
        return $this->belongsTo(
            User::class,
            'user_id',
            'id'
        );
    }

In User Model file User.php

public function employee(){
        return $this->hasMany(Employee::class);
    }

Route file web.php

Route::post('importEmployee', [App\Http\Controllers\EmployeeImportController::class, 'importEmployee']);

Conclusion

In this article, we have both learned how to create an import data with related of multiple model in Eloquent ORM in CSV format of laravel 9.