Forem: Anant Parmar

High-Performance Bulk Insert with relationships

Anant Parmar — Tue, 29 Jul 2025 05:20:38 +0000

Ever tried to bulk insert thousands of records that depend on each other?

If you have, you probably know the pain.

If you haven't, we're about to dive into a problem that'll make you question your life choices, and then show you a neat trick to solve it.

The Problem

Picture this: You're building a project management system and the templating feature requires the ability to save project templates and then create project instances for all clients.

Your project structure looks like this:

Project → Phases → Tasks → Subtasks

We store a similar structure for templates, so we can reuse them across different clients - make sense?

Template Project → Template Phases → Template Tasks → Template Subtasks

For simplicity, let’s assume - A Project template has about 5 phases, 10 tasks per phase, and 3 subtasks per task.

We have 100 clients, and are we’ll create a project for each of them from our project template.

So you're looking at generating:

100 projects × 5 phases × 10 tasks × 3 subtasks = 15,000 related records

Sounds simple, right? Wrong. Here's the catch:

You need the Project ID to create Phases
You need the Phase ID to create Tasks
You need the Task ID to create Subtasks
You want to do this FAST (because reasons)
You want to avoid 15,000+ individual database queries

We’ll explore a scenario with Laravel, but the pattern can apply to any other language or framework.

The "Normal" Approaches (And Why They Hurt)

Approach 1: Individual Inserts

foreach ($clients as $client) {
    $project = Project::create([
        'name' => 'Website Development - ' . $client->name,
        'client_id' => $client->id,
        'start_date' => $startDate,
    ]);

    foreach ($template->phases as $phaseTemplate) {
        $phase = Phase::create([
            'project_id' => $project->id,
            'name' => $phaseTemplate->name,
            'order' => $phaseTemplate->order,
        ]);

        foreach ($phaseTemplate->tasks as $taskTemplate) {
            $task = Task::create([
                'phase_id' => $phase->id,
                'name' => $taskTemplate->name,
                'estimated_hours' => $taskTemplate->estimated_hours,
            ]);

            foreach ($taskTemplate->subtasks as $subtaskTemplate) {
                Subtask::create([
                    'task_id' => $task->id,
                    'name' => $subtaskTemplate->name,
                    'description' => $subtaskTemplate->description,
                ]);
            }
        }
    }
}

Pros: Simple to understand and write

Cons: You're looking at 15,000+ database queries

Approach 2: Memory Mapping

// Step 1: Bulk insert all projects and keep references
$projectData = [];
foreach ($clients as $client) {
    $projectData[] = [
        'name' => 'Website Development - ' . $client->name,
        'client_id' => $client->id,
        'start_date' => $startDate,
        'created_at' => now(),
    ];
}

// Bulk insert projects
Project::insert($projectData);

// Get the inserted projects (this is where it gets tricky)
$insertedProjects = Project::where('name', 'LIKE', 'Website Development%')
    ->where('created_at', '>=', now()->subMinute())
    ->orderBy('id')
    ->get();

// Step 2: Prepare phases with real project IDs
$phaseData = [];
foreach ($insertedProjects as $index => $project) {
    foreach ($template->phases as $phaseTemplate) {
        $phaseData[] = [
            'project_id' => $project->id,
            'name' => $phaseTemplate->name,
            'order' => $phaseTemplate->order,
            'created_at' => now(),
        ];
    }
}

// Bulk insert phases
Phase::insert($phaseData);

// Get inserted phases and map them back...
$insertedPhases = Phase::whereIn('project_id', $insertedProjects->pluck('id'))
    ->where('created_at', '>=', now()->subMinute())
    ->orderBy('project_id', 'order')
    ->get();

// Step 3: Continue this pattern for tasks and subtasks...
// (This gets increasingly complex and error-prone)

Pros:

Uses bulk inserts for better performance than individual creates

Cons:

Complex ID mapping logic that's prone to race conditions
Requires careful ordering and filtering to match inserted records
Gets exponentially more complex with each relationship level

Our Solution: The `temp_identifier` Trick

Here's where we got creative.

What if we could get the benefits of bulk inserts AND handle relationships properly?

The idea is simple:

Add a temporary column to your tables
Use it to track relationships during bulk operations
Clean it up when you're done

Step 1: Add Temporary Columns

ALTER TABLE projects ADD COLUMN temp_identifier BIGINT NULL, ADD COLUMN batch_id VARCHAR(255) NULL;
ALTER TABLE phases ADD COLUMN temp_identifier BIGINT NULL, ADD COLUMN batch_id VARCHAR(255) NULL;  
ALTER TABLE tasks ADD COLUMN temp_identifier BIGINT NULL, ADD COLUMN batch_id VARCHAR(255) NULL;
ALTER TABLE subtasks ADD COLUMN batch_id VARCHAR(255) NULL;

Step 2: Prepare Data with Temporary IDs

$batchId = uniqid(); // Unique identifier for this batch to prevent concurrency issues
$projectData = [];
$phaseData = [];
$taskData = [];
$subtaskData = [];
$projectCounter = 0;
$phaseCounter = 0;
$taskCounter = 0;

foreach ($clients as $client) {
    $tempProjectId = $projectCounter++;

    // Prepare project data
    $projectData[] = [
        'temp_identifier' => $tempProjectId,
        'batch_id' => $batchId,
        'name' => 'Recurring Task - ' . $client->name,
        'client_id' => $client->id,
        'start_date' => $startDate,
        'created_at' => now(),
        'updated_at' => now(),
    ];

    foreach ($template->phases as $phaseTemplate) {
        $tempPhaseId = $phaseCounter++;

        // Prepare phase data
        $phaseData[] = [
            'temp_identifier' => $tempPhaseId,
            'batch_id' => $batchId,
            'project_temp_identifier' => $tempProjectId, // Reference to project's temp_identifier
            'name' => $phaseTemplate->name,
            'order' => $phaseTemplate->order,
            'created_at' => now(),
            'updated_at' => now(),
        ];

        foreach ($phaseTemplate->tasks as $taskTemplate) {
            $tempTaskId = $taskCounter++;

            // Prepare task data
            $taskData[] = [
                'temp_identifier' => $tempTaskId,
                'batch_id' => $batchId,
                'phase_temp_identifier' => $tempPhaseId, // Reference to phase's temp_identifier
                'name' => $taskTemplate->name,
                'estimated_hours' => $taskTemplate->estimated_hours,
                'created_at' => now(),
                'updated_at' => now(),
            ];

            foreach ($taskTemplate->subtasks as $subtaskTemplate) {
                $subtaskData[] = [
                    'batch_id' => $batchId,
                    'task_temp_identifier' => $tempTaskId, // Reference to task's temp_identifier
                    'name' => $subtaskTemplate->name,
                    'description' => $subtaskTemplate->description,
                    'created_at' => now(),
                    'updated_at' => now(),
                ];
            }
        }
    }
}

Step 3: Bulk Insert and Map IDs

// Bulk insert projects
    Project::insert($projectData);

    // Get the mapping of temp_identifier to real ID
    $projectMapping = Project::where('batch_id', $batchId)
        ->whereNotNull('temp_identifier')
        ->pluck('id', 'temp_identifier')
        ->toArray();

    // Update phase data with real project IDs
    foreach ($phaseData as &$phase) {
        $phase['project_id'] = $projectMapping[$phase['project_temp_identifier']] 
            ?? throw new Exception('Project temp_identifier not found');
        unset($phase['project_temp_identifier']);
    }

    // Bulk insert phases
    Phase::insert($phaseData);

    // Get phase mapping
    $phaseMapping = Phase::where('batch_id', $batchId)
        ->whereNotNull('temp_identifier')
        ->pluck('id', 'temp_identifier')
        ->toArray();

    // Update task data with real phase IDs
    foreach ($taskData as &$task) {
        $task['phase_id'] = $phaseMapping[$task['phase_temp_identifier']] 
            ?? throw new Exception('Phase temp_identifier not found');
        unset($task['phase_temp_identifier']);
    }

    // Bulk insert tasks
    Task::insert($taskData);

    // Get task mapping
    $taskMapping = Task::where('batch_id', $batchId)
        ->whereNotNull('temp_identifier')
        ->pluck('id', 'temp_identifier')
        ->toArray();

    // Update subtask data with real task IDs
    foreach ($subtaskData as &$subtask) {
        $subtask['task_id'] = $taskMapping[$subtask['task_temp_identifier']] 
            ?? throw new Exception('Task temp_identifier not found');
        unset($subtask['task_temp_identifier']);
    }

    // Bulk insert subtasks
    Subtask::insert($subtaskData);

This should be wrapped in a DB::transaction with temporarily disabling foriegn key checks if required.

DB::transaction(function () use ($projectData, $phaseData, $taskData, $subtaskData, $batchId) {
    // Temporarily disable foreign key checks for bulk inserts
    DB::statement('SET FOREIGN_KEY_CHECKS=0;');


    // Bulk Operation

    // Re-enable foreign key checks
    DB::statement('SET FOREIGN_KEY_CHECKS=1;');
});

Step 4: Clean Up

Project::where('batch_id', $batchId)->update(['temp_identifier' => null, 'batch_id' => null]);
Phase::where('batch_id', $batchId)->update(['temp_identifier' => null, 'batch_id' => null]);
Task::where('batch_id', $batchId)->update(['temp_identifier' => null, 'batch_id' => null]);
Subtask::where('batch_id', $batchId)->update(['batch_id' => null]);

Pros:

Achieves true bulk insert performance with clean relationship handling
Simple integer lookups eliminate complex field matching
More predictable than approaches relying on insertion order or field matching

Cons:

Requires temporary database columns and schema changes
More complex setup than simpler approaches
Cleanup step needed after completion

When Should You Use This?

This approach works best when:

You're dealing with thousands of related records - multi-level relationships

Don't use it when:

You're inserting just a few hundred records (the overhead isn't worth it)
Your relationships are simple (just use insertGetId loops)
Your team isn't comfortable with the added complexity

Comparison

Let's say you're generating projects for 100 clients from a single template, each with 5 phases, 10 tasks per phase, and 3 subtasks per task:

Approach	Database Queries	Memory Usage	Reliability
Individual Inserts	~20,600	Low	High
Memory Mapping	~7	High	Medium
temp_identifier	~10	High	High

The next time you're staring at a bulk insert problem that seems too complex, remember: sometimes the answer is as simple as adding a temporary column and getting creative with how you use it.

I hope you find this valuable, if you did - share it with your folks who'll find it useful too.

If you have any suggestions/comments please feel free.

Happy coding!

3 essential tools for your JSON objects

Anant Parmar — Thu, 13 Jul 2023 04:45:19 +0000

This is a short listicle

As a developer, it is sometimes difficult to work with JSON. Especially if you are working with third-party APIs. The structure of the object is not in your control. And you have to work with what you get.

Here are 3 tools that I found super helpful in scenarios like this

1. JSONLint

JSONLint is a validator and reformatter for JSON. You can copy paste, or directly type in the editor and it validates and format your messy JSON.

Link: https://jsonlint.com

2. JSON Editor Online

It allows you to transform and compare your JSON. You can Configure schema, inspect and query JSON with ease.

Link: https://jsoneditoronline.org

3. JSON Path Finder

It allows you to find the path of an item in a JSON object. You can expand and select elements in a tree view. And it gives the dot notation of the currently selected piece of data.

Link: https://jsonpathfinder.com

Let me know in the comments if you know something better.

I hope you find this valuable, if you did - awesome 👌 share it with your folks if it's relevant to them. If you have any suggestions/comments please feel free.

Happy coding!

My Socials - LinkedIn | GitHub | Twitter

Using GPT's Function Calling Capability: Stock Information Demo

Anant Parmar — Fri, 16 Jun 2023 10:07:07 +0000

The ability to extract insights from user input has always been an excellent use case for OpenAI APIs. I have seen a lot of people using GPT APIs to perform NLP tasks. From Intent Detection, Classification to NER (Named Entity Recognition) it's quite useful, especially for developers who do not have extensive ML backgrounds.

And this new capability of "Function Calling" takes this use case to a whole new level. In this article, we are going to explore how we can use this feature to get real data about a stock in a few simple lines of Python.

Understanding the core problem

Getting Stocks data is not difficult, it's only a matter of simple API calls.

Here are the tricky parts :

Identifying what the user wants to do
Mapping the users intent to the solution function
Extracting data that our function expects
Chaining the prompts together to respond

Before the availability of the function calling feature, we had to juggle all these things which led to complicated flow. Sure frameworks like langchain can help us simplify this juggling. But the ability of the model itself to do this for us is quite amazing.

I am going to demonstrate this with a simple example :

Answer user's questions about stock prices & other information

Get your API keys

To get the stock price data, we are going to use Polygon,

You can signup there and then access your API keys at: https://polygon.io/dashboard/api-keys

Polygon has a Python client but to keep it more abstract, we are going to use its REST API

If you don't already have your OpenAI API keys, get them from here: https://platform.openai.com/account/api-keys

Let's get into the Code

First, let's install openai and reuests

!pip install openai
!pip install requests

Now, let's setup some global variables and API keys (OpenAI & Polygon)

import openai

global OPENAI_API_KEY
global POLYGON_API_KEY

OPENAI_API_KEY="Your OpenAI API Key goes here" 
POLYGON_API_KEY="Your Polygon API Key goes here" 

openai.api_key = OPENAI_API_KEY

Imports, some more global variables and setup code

import datetime
import requests
import json

global today_str

today = datetime.date.today()
today_str = today.strftime('%Y-%m-%d')

Functions to fetch Stocks data

Let's write a function that we can use to get stock prices data from the polygon API

def get_stock_price_by_date(ticker_symbol, date):
    url = f"https://api.polygon.io/v1/open-close/{ticker_symbol}/{date}?adjusted=true&apiKey={POLYGON_API_KEY}"
    try:
        response = requests.request(method='GET', url=url)
        response.raise_for_status()
        # print(response.text)
        return json.dumps(response.text)
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

You can try calling this function with manual parameter input, to make sure it works as expected

print(get_stock_price_by_date(ticker_symbol="GOOGL", date="2023-06-16"))

{
    "status": "OK",
    "from": "2023-06-15",
    "symbol": "GOOGL",
    "open": 123.14,
    "high": 125.46,
    "low": 122.4,
    "close": 125.09,
    "volume": 3.5146268e+07,
    "afterHours": 124.85,
    "preMarket": 123.48
}

Let's write another function that fetches the data about stock according to the SEC filing

def get_stock_sec_details(ticker_symbol, date = today_str):
    url = f"https://api.polygon.io/v3/reference/tickers/{ticker_symbol}?date={date}&apiKey={POLYGON_API_KEY}"
    try:
        response = requests.request(method='GET', url=url)
        response.raise_for_status()
        # print(response.text)
        return json.dumps(response.text)
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

We can try calling this function to check how it responds

print(get_stock_sec_details(ticker_symbol="AAPL"))

{
    "request_id": "f7d8eb4aac2a75fbb4b8df1f25ab4ead",
    "results": {
        "ticker": "AAPL",
        "name": "Apple Inc.",
        "market": "stocks",
        "locale": "us",
        "primary_exchange": "XNAS",
        "type": "CS",
        "active": true,
        "currency_name": "usd",
        "cik": "0000320193",
        "composite_figi": "BBG000B9XRY4",
        "share_class_figi": "BBG001S5N8V8",
        "market_cap": 2.92569585902e+12,
        "phone_number": "(408) 996-1010",
        "address": {
            "address1": "ONE APPLE PARK WAY",
            "city": "CUPERTINO",
            "state": "CA",
            "postal_code": "95014"
        },
        "description": "Apple designs a wide variety of consumer electronic devices, including smartphones (iPhone), tablets (iPad), PCs (Mac), smartwatches (Apple Watch), and AirPods, among others. In addition, Apple offers its customers a variety of services such as Apple Music, iCloud, Apple Care, Apple TV+, Apple Arcade, Apple Fitness, Apple Card, and Apple Pay, among others. Apple's products include internally developed software and semiconductors, and the firm is well known for its integration of hardware, software, semiconductors, and services. Apple's products are distributed online as well as through company-owned stores and third-party retailers.",
        "sic_code": "3571",
        "sic_description": "ELECTRONIC COMPUTERS",
        "ticker_root": "AAPL",
        "homepage_url": "https://www.apple.com",
        "total_employees": 164000,
        "list_date": "1980-12-12",
        "branding": {
            "logo_url": "https://api.polygon.io/v1/reference/company-branding/d3d3LmFwcGxlLmNvbQ/images/2023-05-01_logo.svg",
            "icon_url": "https://api.polygon.io/v1/reference/company-branding/d3d3LmFwcGxlLmNvbQ/images/2023-05-01_icon.jpeg"
        },
        "share_class_shares_outstanding": 15728700000,
        "weighted_shares_outstanding": 15728702000,
        "round_lot": 100
    },
    "status": "OK"
}

Utilizing Function Calling

Now is the part where we describe our function to OpenAI API

def run_conversation(message):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo-0613",
        messages=[{"role": "user", "content": f"Date today is {today_str}. Provide answer with a date. {message}"}],
        functions=[
            {
                "name": "get_stock_price_by_date",
                "description": "Get stock prices by date",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "ticker_symbol": {
                            "type": "string",
                            "description": "The ticker symbol for the stock listed on NASDAQ exchange",
                        },
                        "date": {
                            "type": "string", 
                            "description": "date string in %Y-%m-%d format"
                        }
                    },
                    "required": ["stock_ticker", "date"],
                },
            },
            {
                "name": "get_stock_sec_details",
                "description": "Get SEC filing details by stock ticker symbol",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "ticker_symbol": {
                            "type": "string",
                            "description": "The ticker symbol for the stock listed on NASDAQ exchange",
                        },
                        "date": {
                            "type": "string", 
                            "description": "date string in %Y-%m-%d format"
                        }
                    },
                    "required": ["stock_ticker"],
                },
            }
        ],
        function_call="auto",
    )

There are 2 differences between a standard chat completion call and this one.

Model: To use the function calling feature we have to use the latest models gpt-3.5-turbo-0613 or gpt-4-0613 (GPT 4)
Additional Parameters to describe the function : functions & function_call

functions expect an array of the function descriptions, you can provide more functions to this array and the model will try to detect which function is most suitable for the given user input
function_call to indicate whether to generate a model-facing (auto response (none a response that will be fed to the model on subsequent call) or a user-facing response that will be presented to the user

Now based on the model's response we'll call the actual function, here you can check which function was found most suitable for the model

Here is how we can do this

## Continuing the `run_conversation` function

internal_message = response["choices"][0]["message"]

    if internal_message.get("function_call"):
        function_name = internal_message["function_call"]["name"]
        function_args = json.loads(internal_message["function_call"]["arguments"])
        function_response = ""

        # Printing which function is detected
        print(function_name)

        # If model detected stock price info is required
        if function_name == "get_stock_price_by_date":
          function_response = get_stock_price_by_date(
              ticker_symbol=function_args.get("ticker_symbol"),
              date=function_args.get("date"),
          )

        # If model detected stock SEC filing info is required
        if function_name == "get_stock_sec_details":
          function_response = get_stock_sec_details(
              ticker_symbol=function_args.get("ticker_symbol")
          )


        second_response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo-0613",
            messages=[
                {"role": "user", "content": message},
                internal_message,
                {
                    "role": "function",
                    "name": function_name,
                    "content": function_response,
                },
            ],
        )
        return second_response

One thing to note here is that the function description (objects within the functions array) will be counted against input tokens

Alright, let's test

print(run_conversation("tell me apple volume 3 days ago")["choices"][0]["message"]["content"])

get_stock_price_by_date

The volume for Apple (AAPL) three days ago, on June 13, 2023, was approximately 54,867,129 shares.

print(run_conversation("how many outstanding google shares are there")["choices"][0]["message"]["content"])

get_stock_sec_details

As of now, there are approximately 5,874,000,000 outstanding Google shares.

So to conclude, by using the function calling feature -

The model can detect which function should be called to fulfill the user's request
It can extract required info from the user's input and provide us with function arguments

You can find the full source code of this article at - https://github.com/anantrp/ai-experiments/blob/main/openai_function_calling_stock_prices_demo.ipynb

You can read more about Function Calling at - https://platform.openai.com/docs/guides/gpt/function-calling

This can go beyond simple retrieval capabilities and we can build actuator functions (functions that can perform actions) such as sending an email, ordering products, taking action on our custom software systems and much more

With Function Calling capability we have great potential to build the AI agents of tomorrow

I hope you find this valuable, if you did - that's awesome 👌 share it with your folks who'll find it useful too. If you have any suggestions/comments please feel free.

Happy coding!

Dealing with incorrect PHP Intelephense errors in your Laravel Project

Anant Parmar — Thu, 15 Jun 2023 08:10:11 +0000

If you are a PHP developer using VS Code, it's almost a no-brainer to use the PHP Intelephense plugin. It provides excellent IntelliSense for your PHP code and makes life easier.

It prompts you to fix any undefined type, function, constant, class constant, method, or property in your code. Which is great, it's the reason why we want to install it. However, if you are someone like me who doesn't want any red wiggly error/warning lines showing up in your code - it can be a bit problematic sometimes.

Let me give you an example - I was using Laravel Livewire for a project recently and I wanted to pass some parameters to my default AppLayout class from the Livewire component.

I know for a fact that this is a perfectly valid code, the layout() method is there in the documentation. The parameters are also correct and the code works without throwing any errors or exceptions.

But it is annoying to see this every time I open the file. To add more to that now all the files, and folders also appear to be red (containing errors) in my VS Code Explorer.

Apart from my being over-sensitive about this, the practical problem is - I don't know whether the warning in the VS Code Explorer is caused by my stupid code or is it because of the layout() method not being recognized.

I searched and found one possible fix to this problem on Stack Overflow.

There are a couple of solutions in the article :

Use Laravel IDE Helper: https://github.com/barryvdh/laravel-ide-helper

This was too heavyweight for me, sure those warnings are annoying but I don't want to install a whole package for that - maybe it is something right for you
If my solution does not work properly probably I'll switch to this in the future

Use the _ide_helper.php and use that within your code like it is explained in this Laracast Video

I like this approach it's not too heavy and still works

I ended up implementing another solution. Create a helper class that targets specifically the methods which are shown in the warning.

Here is how I did it

Create a new Helper class IntelephenseHelper.php in app\Helpers directory

Added the following code

<?php

namespace Illuminate\Contracts\View;

use Illuminate\Contracts\Support\Renderable;

interface View extends Renderable
{
  /** @return static */
  public function layout();
  public function slot();
}

And the warnings were gone!

What it does is quite simple and self-explanatory. It extends the interface and adds the methods which were showing the warnings.

No more red wiggly warnings and no more annoying red file indicators. Life is simpler again.

If you are just starting with Laravel, here is an article I wrote that might be useful for you: How to write clean Controllers in Laravel

I hope you find this valuable, if you did - awesome 👌 share it with your folks if it's relevant to them. If you have any suggestions/comments please feel free.

Happy coding!

How to write clean Controllers in Laravel

Anant Parmar — Thu, 25 May 2023 08:55:10 +0000

As a beginner Laravel developer - writing clean controllers is crucial. In my experience, I have come across a lot of laravel projects where the controllers are 1000s of lines long. It should not have to be this way.

Writing maintainable and efficient controllers isn't hard. You just have to know how to organize your codebase. In this article, we will explore essential tips to help you write cleaner controllers.

3 tips for cleaner laravel controllers :

Smaller Methods with Single Responsibility
Validation with Form Request Class
Eloquent API Resources

We'll use a simple example of an e-commerce product management system to explain each point. So let's dive in!

1 - Smaller Methods with Single Responsibility

A controller method should have a single responsibility. Handling a specific HTTP request and returning the appropriate response.

You should avoid adding complex logic to your controller methods. Decouple the request-handling part of our application from the complex business logic. Your controller should only be responsible for "controlling" the flow of execution.

Let's understand this with an example :

class ProductController extends Controller
{
    // ...

    public function update(UpdateProductRequest $request, $id)
    {
        $product = Product::findOrFail($id);

        $data = $request->validated();

        $product = Product::findOrFail($id);

        if ($request->hasFile('image')) {
            $image = $request->file('image');
            $imagePath = 'uploads/products/' . $image->hashName();
            $image->storeAs('public', $imagePath);
            $product->image = $imagePath;
        }

        $product->name = $request->input('name');
        $product->price = $request->input('price');
        $product->category_id = $request->input('category_id');
        $product->description = $request->input('description');
        $product->save();

        return redirect()->route('products.index')->with('success', 'Product updated successfully.');
    }

    // ...
}

Here update method here is responsible for multiple things. It also handles file uploads and storing images which is not its primary concern. It should only focus on 'updating' product data.

Let's see how we can clean up our update method

class ProductController extends Controller
{
    private $productService;

    public function __construct(ProductService $productService)
    {
        $this->productService = $productService;
    }

    // ...

    public function update(UpdateProductRequest $request, $id): RedirectResponse
    {
        $product = Product::findOrFail($id);

        $data = $request->validated();

        if ($request->hasFile('image')) {
            $image = $request->file('image');
            $data['image'] = $this->productService->storeProductImage($image);
        }

        $this->productService->updateProduct($product, $data);

        return redirect()->route('products.index')->with('success', 'Product updated successfully.');
    }
}

Note that we still have to update the Product model and upload the image. But we do that in a separate class ProductService that handles these specific operations.

Create Service Classes or Actions to handle complex business logic

This is a very simple example but you can imagine a more complex feature that involves probably 5 different operations based on the request and model. It can get very complicated very quickly.

Key Takeaways :

Stick to a single responsibility for each controller action
Keep your methods focused and concise, performing a specific task
Break down complex operations into smaller, reusable methods

2 - Validation with Form Request Class

Validating incoming request data is essential for ensuring data integrity and security. But if you are not careful - the validation logic can get complex making code unreadable.

You can create custom form request classes to keep your controllers clean. This separates validation logic from the controller itself. Laravel provides a great way to overcome this - Request classes.

Let's explore this. First, consider store method of the ProductController. This is your typical validation logic implemented within the controller action.

use Illuminate\Http\Request;
use App\Http\Controllers\Controller;
use App\Models\Product;

class ProductController extends Controller
{
    public function store(Request $request)
    {
        // Validate the request data
        $validatedData = $request->validate([
            'name' => 'required|string|max:255',
            'description' => 'required|string',
            'price' => 'required|numeric|min:0',
            'quantity' => 'required|integer|min:0',
            'category_id' => 'required|exists:categories,id',
            'brand_id' => 'required|exists:brands,id',
            'color' => 'required|string|max:50',
            'size' => 'required|string|max:20',
            'weight' => 'required|numeric|min:0',
            'images' => 'required|array',
            'images.*' => 'required|image|max:2048',
        ]);

        // Create a new product ...        
    }
}

There are a few disadvantages of doing validation this way :

Code bloating and decreased readability
Code duplication across controller actions
Testing challenges and reduced testability

So let's see how you can rewrite this using a Request class and make it cleaner, reusable, and more testable

Creating a request class using the Artisan command:

php artisan make:request CreateProductRequest

This will generate a new CreateProductRequest class in the app/Http/Requests directory. Open the generated file and update its rules method with the validation rules:

namespace App\Http\Requests;

use Illuminate\Foundation\Http\FormRequest;

class CreateProductRequest extends FormRequest
{
    public function rules()
    {
        return [
            'name' => 'required|string|max:255',
            'description' => 'required|string',
            'price' => 'required|numeric|min:0',
            'quantity' => 'required|integer|min:0',
            'category_id' => 'required|exists:categories,id',
            'brand_id' => 'required|exists:brands,id',
            'color' => 'required|string|max:50',
            'size' => 'required|string|max:20',
            'weight' => 'required|numeric|min:0',
            'images' => 'required|array',
            'images.*' => 'required|image|max:2048',
        ];
    }
}

Now, in your ProductController you can use the new request class like so:

use Illuminate\Http\Request;
use App\Http\Controllers\Controller;
use App\Http\Requests\CreateProductRequest;
use App\Models\Product;

class ProductController extends Controller
{
    public function store(CreateProductRequest $request)
    {
        // Retrieve the validated data
        $validatedData = $request->validated();

        // Create a new product ...
    }
}

It's a good idea to reuse the request class for different controller actions if possible. But if you want you can also create a different request class for update action. You can create as many custom request classes as needed.

Key Takeaways:

Separate validation logic from your controller using custom form requests.
Reuse the request class in other controllers/methods

3 - Eloquent API Resources

When you are writing controllers for your API it is very common to format the JSON response.

If the response format that you need is different than your schema, the formatting logic can also make your controllers bloated and hard to read. And sometimes you may want to reuse the same response format in multiple API endpoints.

This is where you can leverage Eloquent API Resources to keep your responses consistent, make your formats reusable and your controllers clean

Let's look at an example of an API response that is formatted within the controller method:

use Illuminate\Http\Request;
use Illuminate\Http\Response;

class ProductController extends Controller
{
    public function index(Request $request)
    {
        $products = Product::with('brand', 'categories', 'features')->get();

        // Transform the product list with required attributes
        $transformedProducts = $products->map(function ($product) {
            return [
                'id' => $this->id,
                'name' => $this->name,
                'price' => $this->price,
                'description' => $this->description,
                'brand_name' => $product->brand->name,
                'brand_image' => $product->brand->image,
                'categories' => $product->categories->map(function ($category) {
                    return [
                        'name' => $category->name,
                        'alias' => $category->alias,
                        'image' => $category->image,
                    ];
                }),
                'features' => $product->features->map(function ($feature) {
                    return [
                        'title' => $feature->title,
                        'description' => $feature->description
                    ];
                })
            ];
        });

        return response()->json(['products' => $transformedProducts], Response::HTTP_OK);
    }
}

Now let's look at how can we extract this logic and put it in a Resource class

First, create ProductResource class

php artisan make:resource Product

Let's move the logic to this class

use Illuminate\Http\Resources\Json\JsonResource;

class ProductResource extends JsonResource
{
    public function toArray($request)
    {
        return [
            'id' => $this->id,
            'name' => $this->name,
            'price' => $this->price,
            'description' => $this->description,
            'brand_name' => $this->brand->name,
            'brand_image' => $this->brand->image,
            'categories' => $this->categories->map(function ($category) {
                return [
                    'name' => $category->name,
                    'alias' => $category->alias,
                    'image' => $category->image,
                ];
            }),
            'features' => $this->features->map(function ($feature) {
                return [
                    'title' => $feature->title,
                    'description' => $feature->description
                ];
            })
        ];
    }
}

Now we can use this in our controller

use Illuminate\Http\Request;
use Illuminate\Http\Response;
use App\Http\Resources\ProductResource;

class ProductController extends Controller
{
    public function index(Request $request)
    {
        $products = Product::with('brand', 'categories', 'features')->get();

        $productResource = ProductResource::collection($products);

        return response()->json(['products' => $productResource], Response::HTTP_OK);
    }
}

As you can see with this approach our controller is much cleaner compared to the original index method. It is also reusable. Let's say you have a customer orders endpoint where want to provide the products along with an order object (a very common API requirement) you can reuse the same resource/collection

Key Takeaways:

Use the Eloquent API Resources to keep controllers more readable
Improve the reusability and maintainability of your code by separating data transformation logic from your controllers

Writing clean controllers in Laravel is essential for building maintainable and efficient applications. Get into the habit of writing cleaner code and your life as a developer will be much easier. As with all good habits, it will take deliberate thinking and practice.

By practicing these principles and incorporating them into your development workflow, you'll be well on your way to writing cleaner and more organized controllers in Laravel.

I have also published a blog on getting started with background jobs in laravel

I hope you find this valuable, if you did - awesome 👌 share it with your folks if it's relevant to them. If you have any suggestions/comments please feel free.

Happy coding!

ChatGPT API Prompting Guide & Best Practices

Anant Parmar — Sat, 20 May 2023 06:32:00 +0000

This is not a programming or ChatGPT API integration tutorial, these are some key ideas and points to keep in mind while using ChatGPT API.

Large Language Models provide unprecedented capabilities to your software products. As developers, it is crucial to learn this tech and integrate it within our projects. Platforms like OpenAI provides API to interact with these models and we can leverage that to enhance our software.

Recently, I completed a fascinating course, "ChatGPT Prompt Engineering for Developers." I learned some incredible techniques to get the most out of OpenAI's ChatGPT API, and I'd love to share those insights with you in this post.

Guidelines

When interacting with ChatGPT, clarity is crucial. Here are some useful strategies:

Separate instructions and input texts: This allows you to test your instructions with various inputs. For example, if you're asking ChatGPT to summarize an article, make the instruction "Summarize the following article:" separate from the article text itself.
Use delimiters: They help specify the boundaries between the instruction and the input, reducing the likelihood of prompt injections. A colon or a new-line can serve as simple but effective delimiters.
Request structured output: If you need to parse ChatGPT's response programmatically, ask it to structure its output in a specific way. For instance, you might request, "List the key points from the following text as bullet points:".
Check input conditions: To reduce the chances of the model producing irrelevant or inaccurate responses (a phenomenon known as "hallucination"), specify any conditions the input must meet. For example, "If the text contains a date, provide the day of the week for that date."
Few-Shots Prompting: If possible, providing a few examples of the desired input-output pattern can guide the model to produce similar results.

Iterative Prompt Development

Crafting the perfect prompt with ChatGPT is indeed an iterative process, and it requires a keen understanding of the objective and a willingness to experiment and learn.

Let's break it down into actionable steps:

Define the Objective: The first step in prompt crafting is to clearly understand the desired output. Ask yourself, what do you want ChatGPT to generate? For example, if your goal is to extract key points from a text, your objective would be a list of the main ideas from the input text.
Write an Initial Prompt: Based on your objective, write an initial prompt. A prompt is a command or question given to ChatGPT to direct its response. For the objective mentioned above, an initial prompt could be, "List the main ideas in the following text:".
Test the Prompt: Now, it's time to test the prompt. Run it through the model and see what kind of output you get. Is it in line with your objective? If not, it's time to iterate.
Analyze the Output: Analyze the output from the model. What aspects of the response align with your objective, and what parts are off target? This will give you clues about how to adjust your prompt.
Refine the Prompt: Based on your analysis, adjust your prompt. For instance, if the model isn't quite capturing the main ideas as you intended, you might refine the prompt to be more specific, such as, "Summarize the following text into bullet points:".
Repeat the Process: Continue to test, analyze, and refine your prompt until it consistently generates the desired output. Remember that this is an iterative process. It might take several rounds of refinement to get it just right.
Generalize the Prompt: Once you have a prompt that works well for a specific case, try to generalize it for other similar cases. This means testing the prompt with a variety of input texts to ensure it works across a broad range of scenarios.

Summarization

Summarization is a powerful use case of ChatGPT API, but it's essential to know how to use it effectively. Here are some guidelines:

Define the purpose: If the summary is going to be used in a specific way, make sure to specify this in the instruction.
Stay focused: Ask the model to focus on specific parts of the input if needed. For instance, if you're only interested in the financial aspects of a business report, you might ask, "Summarize the financial information in the following report:".
Extract instead of summarizing: In some cases, it might be more useful to extract key information instead of summarizing. For instance, you could prompt, "List the names of all the people mentioned in the following text:".

Inferring

ChatGPT can also infer insights from a text, performing tasks such as sentiment analysis, categorization, classification, and labeling. Here are some ways you can leverage this functionality:

Multitask: You can ask the model to perform more than one task in the same instruction, and to generate the output in a specific format. For example, "Analyze the sentiment of the following review and classify it as positive, negative, or neutral:".
Test on multiple examples: It is possible that a prompt that works well on one set of inputs does not perform as well on others. To ensure that your instruction works well across a broad range of inputs, test it with a variety of examples.
Include your own labels/tags: If you have a predefined set of categories, you can include them in the instruction, and ask the model to select the most relevant one(s) for the given input. For instance, "Classify the following text into one of these categories: Technology, Environment, Politics, or Culture:".

Transforming

ChatGPT is also capable of performing a variety of text transformations, including language translations & format transformations.

Language Identification & Translation: You can ask the model to identify the language of an input text, or translate it into another language. For instance, "Translate the following Spanish text into English:".
Tone Transformation: You can transform an input text into a different tone, such as formal, casual, or conversational. For example, "Rewrite the following formal text in a casual tone:".
Format Transformation: The model can transform text from one format to another, such as from JSON to HTML or CSV to JSON. For instance, "Convert the following JSON data into HTML table format:".
Proofreading: You can ask the model to proofread a text, making corrections for grammar, punctuation, and spelling. For example, "Proofread and correct any errors in the following text:".

Expanding

ChatGPT can also expand a short input text into a longer, more detailed piece of writing, such as a blog post, an essay, or an email reply. Here are some things to consider:

Provide context: If the output is being used in a specific context, make sure to provide that context in the prompt. For instance, "Write a reply to the following email, expressing appreciation for the sender's suggestions and agreeing to implement them:".
Disclose AI involvement: If the output is being communicated to a user, it's recommended to disclose that it's AI-generated to maintain transparency.
Adjust the temperature: The temperature parameter controls the model's creativity. A lower temperature (closer to 0) makes the model's output more deterministic, while a higher temperature (closer to 1) allows for more creative responses.

Developing a Chatbot

You can use ChatGPT to create a chatbot with specific behaviors:

Set the role: Use the 'system' role to frame the conversation. This message tells the model how it should behave in response to the 'user' messages. For example, "You are a helpful assistant that always provides detailed responses:".
Provide user context: Include the user's name, details, and any other relevant context in the initial user message. For example, "The user is a beginner programmer asking for help with Python syntax errors:".
Remember the model's limitations: The ChatGPT model cannot remember earlier interactions, so you have to provide previous messages each time you make a request. For example, if the user asked a question in a previous message, include that message in the next request if it's relevant to the ongoing conversation.
Control the temperature: For output that is intended to be presented to the user, you can use a higher temperature for less predictable responses. For output intended to be parsed programmatically, use a lower temperature for more reliable results.

Understanding these prompt engineering concepts can significantly improve your interactions with the ChatGPT API, making your applications more effective and user-friendly.

The purpose of this blog is to give you a glimpse of the course "ChatGPT prompt engineering for developers". The course was packed with useful insights and techniques for anyone working with the ChatGPT API.

Here is the link to the course: https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/

I hope you find this valuable, if did awesome 👌 share it with your folks if it's relevant to them. If you have any suggestions/comments please feel free.

Happy Coding!

Background Jobs in Laravel for Beginners

Anant Parmar — Fri, 12 May 2023 18:45:14 +0000

If you're a Laravel developer understanding background jobs is critical. This technique helps manage time-consuming tasks, such as uploading images, sending emails, or processing data, without affecting user experience.

In this article, we'll walk through the process of implementing background jobs using Laravel's queue system with a simple image upload example. Don't worry, we'll keep it simple and straightforward.

What are Background Jobs?

In any web application, some operations take a considerable amount of time to execute. If these operations are run during a user's request, it could lead to a poor user experience due to long loading times. This is where background jobs come in. They allow these heavy operations to be processed in the background, freeing up the application to handle other requests.

Background Jobs within the Controller

Let's start with an image upload example. In a typical scenario, a user uploads an image, and the application processes the image before storing it. If we did this synchronously, the user would have to wait until the processing is done. But with background jobs, we can improve the user experience.

Here's how you might do this in a controller:

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Illuminate\Support\Facades\Storage;
use Image;

class ImageUploadController extends Controller
{
    public function upload(Request $request)
    {
        $path = $request->file('image')->store('images');

        // Image processing (resize, crop, etc.)
        $image = Image::make(Storage::get($path));
        $image->resize(300, 200);
        Storage::put($path, (string) $image->encode());

        return response()->json(['message' => 'Image uploaded and processed successfully.']);
    }
}

While this works, it's synchronous. The user has to wait for the image processing to finish. This is where Laravel's queue comes in handy.

Implementing Queues with Jobs and Dispatch

Let's move the processing part to a Laravel job. First, we create a job using the artisan command:

php artisan make:job ProcessImage

This will create a new job file under app/Jobs. Here's how we can modify it to process our image:

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Image;
use Illuminate\Support\Facades\Storage;

class ProcessImage implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $path;

    public function __construct($path)
    {
        $this->path = $path;
    }

    public function handle()
    {
        $image = Image::make(Storage::get($this->path));
        $image->resize(300, 200);
        Storage::put($this->path, (string) $image->encode());
    }
}

Now, in our controller, instead of processing the image directly, we dispatch the job:

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use App\Jobs\ProcessImage;

class ImageUploadController extends Controller
{
    public function upload(Request $request)
    {
        $path = $request->file('image')->store('images');

        // Dispatch job to process the image
        ProcessImage::dispatch($path);

        return response()->json(['message' => 'Image uploaded. Processing will be done in the background.']);
    }
}

Implementation Considerations:

Often when you learn some new concept or technique you lack the experience (of course because you just learned it) of using it. So most of the time you wonder -when you should & should not use it; what will be the benefits, what will be the downsides, etc.

Let's explore this further

Advantages of using background jobs:

Improved Performance: By delegating time-consuming tasks to background jobs, your application remains responsive and can handle more user requests simultaneously.
Better User Experience: Users are not kept waiting for tasks like image processing or data crunching to complete. They get instant feedback, which can significantly enhance their experience.
Resource Management: Background jobs allow for better server resource utilization. You can control the number of concurrent jobs, manage priorities, and handle retries and failures.
Better Code Quality: Since the logic of heavy processing is moved to Job class, your controllers are cleaner and more readable. You separate the logic of your Reuest's control from the processing that needs to be done.

Drawbacks of using background jobs:

Complexity: Background jobs can add complexity to your application. You must manage the job queue, handle job failures, and ensure jobs are processed in a timely manner.
Testing Difficulty: Testing asynchronous jobs can be more challenging than testing synchronous code.
Delayed Execution: Since jobs are processed in the background, there can be a delay before a job is executed, especially if the queue is long.

When to Use Background Jobs?

Use background jobs when you need to perform time-consuming tasks that don't need to be completed immediately for the user request to be fulfilled. These can include sending emails, processing images, handling data, and more.

However, if a task must be completed before a response can be sent to the user - for instance, validating user input or querying the database for specific records - it's best to execute it synchronously during the user request.

You can learn more about jobs & queues at Laravel's official documentation: https://laravel.com/docs/queues

In conclusion, Laravel's queue system for background jobs is a powerful tool for improving your web application's performance and user experience. It may take some time to grasp, but once you do, you'll find it's an indispensable part of your Laravel toolkit.

I hope you find this valuable, if did awesome 👌 share it with your folks if it's relevant to them. If you have any suggestions/comments please feel free.

How I built a chat widget with ChatGPT in under an hour

Anant Parmar — Thu, 27 Apr 2023 02:27:07 +0000

What I wanted

For an AI-powered chatbot project, I wanted a simple responsive chat widget that can be integrated into any webpage with a single script tag. I searched for open-source chat widgets but they were either built with some framework or came with a lot of unnecessary stuff.

So, I decided to give ChatGPT a try. I have used it for some of my projects to write tests, docs, and even functional code. But never tried to build something from start to finish

Getting Started

Opened ChatGPT and selected the GPT-4 model (I thought 25 messages would be enough)

Prompt #1 :

write code for a floating chat widget such that it can be embedded with a script tag on any website, it should functions such as onUserRequest and reply

It generated HTML code, but what the heck - let's try it.

And here is what came out :

This was looking like a chat widget but it was HTML a page. And my objective was to have javascript code that I can simply add to any webpage with a script tag.

But it worked! I can type a message and either click send button or hit enter and the functions were properly in place. Time to improvise.

Prompt #2 :

this is an HTML code and i am pretty sure we can not embed it with a script tag

The response was cut off in between so I had to ask for the continuation

This time it created chat-widget.js javascript code (probably could have avoided the first step by improving the prompt)

So, I created index.html with some boilerplate and added the script tag

<script async src="./chat-widget.js"></script>

Here is the output :

Now, I was pretty confident that it can be done shortly. I was basically 10 minutes in and it has generated decent results.

There were a couple of problems though :

The chat dialog element was always open - I wanted it to be open only once the bubble is clicked
I wanted the height to be fixed and the conversations could scroll within the container
I wanted it to take up the whole screen while on a mobile device

Prompt #3 :

this works, now we want to style it in such a way that it appears a bottom chat bubble, and when we click on that it opens a popup. on mobile screen this should take the fullscreen. also style it like a modern chat widget with avatars and some decent styling

And the result was pretty good, I added an emoji as the icon on the chat bubble. I tried several prompts to fix the height but I could not so I fixed the height.

Here is how it was looking after a few minor adjustments :

It was almost there, pretty close to the live chat widgets that we see on websites.

Next, I wanted it to use tailwind (probably not the best choice to load a 2mb+ CSS on a chat widget - but again its an experiment)

Prompt #4:

use tailwind css to style this, use a cdn to inject tailwind, keep the other functionalities as they are

After this, there were only some tweaks to be made, like changing the bubble icon, captions, and some styling with the index.hml page

Final Outcome

Mobile viewport

Desktop viewport

Conclusion

Within about 8 prompts and around 25 minutes of editing/adjustments, I was able to get a completely responsive, pure javascript chat widget that was exactly how I wanted it to be.

ChatGPT especially GPT-4 is amazing 🙌

Here is a fully working demo: https://anantrp.github.io/chat-widget

Here is the full source code: https://github.com/anantrp/chat-widget

I hope you find this valuable, if did awesome 👌 share it with your folks if it's relevant to them. If you have any suggestions/comments please feel free.

Forem: Anant Parmar

High-Performance Bulk Insert with relationships

The Problem

The "Normal" Approaches (And Why They Hurt)

Approach 1: Individual Inserts

Approach 2: Memory Mapping

Our Solution: The temp_identifier Trick

Step 1: Add Temporary Columns

Step 2: Prepare Data with Temporary IDs

Step 3: Bulk Insert and Map IDs

Step 4: Clean Up

When Should You Use This?

Comparison

3 essential tools for your JSON objects

1. JSONLint

2. JSON Editor Online

3. JSON Path Finder

Using GPT's Function Calling Capability: Stock Information Demo

Understanding the core problem

Get your API keys

Let's get into the Code

Functions to fetch Stocks data

Utilizing Function Calling

Alright, let's test

Dealing with incorrect PHP Intelephense errors in your Laravel Project

How to write clean Controllers in Laravel

3 tips for cleaner laravel controllers :

1 - Smaller Methods with Single Responsibility

2 - Validation with Form Request Class

3 - Eloquent API Resources

ChatGPT API Prompting Guide & Best Practices

Guidelines

Iterative Prompt Development

Summarization

Inferring

Transforming

Expanding

Developing a Chatbot

Background Jobs in Laravel for Beginners

What are Background Jobs?

Background Jobs within the Controller

Implementing Queues with Jobs and Dispatch

Implementation Considerations:

Advantages of using background jobs:

Drawbacks of using background jobs:

When to Use Background Jobs?

How I built a chat widget with ChatGPT in under an hour

What I wanted

Getting Started

Prompt #1 :

Prompt #2 :

Prompt #3 :

Prompt #4:

Final Outcome

Mobile viewport

Desktop viewport

Conclusion

Our Solution: The `temp_identifier` Trick