Forem: sai teja

Speed up your apis- part 2

sai teja — Fri, 03 Mar 2023 15:27:08 +0000

https://medium.com/@saiteja180/speed-up-your-apis-part-2-eb16eb333df5

Speed up your apis

sai teja — Fri, 03 Mar 2023 15:26:15 +0000

https://medium.com/@saiteja180/speed-up-your-apis-8f006d8bde61

Handling response from a go routine

sai teja — Thu, 05 Jan 2023 14:21:10 +0000

There are several use cases of Go Routines and one of the use case is firing multiple routines and capture the response.

Example use cases:-

Calling multiple Apis and log/use the response.
Inserting rows into a Table and log the errors if any.

package main

import (
    "fmt"
    "sync"
)

type Resp struct {
    Msg string
}

func main(){

    // unbuffered channel to capture response from the go routine
    ch:=make(chan Resp)

    // sync.WaitGroup type for handling go routines
    var wg sync.WaitGroup

    for i:=1;i<=5;i++{
        // tell WaitGroup to that you are going totrigger a go routine
        wg.Add(1)

        // trigger
        go routine(&wg,i,ch)

    }

    // fire a go routine to run as demon and give a way to read the response
    // and it will wait until all go routines executed successfully
    go func(){

        // wait until all goroutines are executed successfully
        wg.Wait()

        // we have to close the channel else below for loop will wait for data to write but data won't come
        // this will lead to a deadlock (infinite wait)
        close(ch)

    }()

    // read from response channel
    for v:=range ch{
        fmt.Println(v.Msg)
    }

    fmt.Println("signing off ...!!!")

}

func routine(wg *sync.WaitGroup,i int,ch chan Resp){
    // tell waitGroup that you are done with this task
    defer wg.Done()

    //send data to response channel
    ch<-Resp{
        Msg: fmt.Sprintf("completed task %d..!!",i),
    }

}

output:-
completed task 4..!!
completed task 3..!!
completed task 5..!!
completed task 1..!!
completed task 2..!!
signing off ...!!!

please follow medium for interesting things

Sai Teja – Medium

Read writing from Sai Teja on Medium. Software Engineer | Fitness trainer | Random Thinker | https://www.linkedin.com/in/saitejamesala/. Every day, Sai Teja and thousands of other voices read, write, and share important stories on Medium.

medium.com

Limit the Go Routines

sai teja — Wed, 30 Nov 2022 16:21:33 +0000

A easy way to limit the go routines without impacting the performance

Yes,It’s true GO takes less memory to initiate a Go Routine and we can trigger any number of go routines simultaneously but we have to consider
few parameters like with whom it is interacting , what type of data it is handling, will routine run infinitely or it ends after given task in done etc…
And we always get these type of issues only in PROD ENV 😅

In case of handling files concurrently we may ran out of memory it may cause of failure of the system eventually we may lose the data in memory.
hence system shows poor performance.. it is not all acceptable in modern world 🤔.
In case Data Base interaction we can’t create Infinite no of connections to a data base and unnecessarily it will stress the data base and increases the CPU usage.
In case of fire and forget tasks like logging things in background. it will trigger routine for each log and increases the CPU load.

TO Avoid such type of issues we have to utilize the go routines efficiently
i.e limiting the Go Routines (go routine limiter)
In go we can simply use a buffered channel and limit the go routines usage.
Steps:-

decide how many go routines you want to trigger (consider factors like CPU and memory )
load the some dummy data into a buffered channel
trigger the go routine and buffered channel into it
release data from buffered channel once task is done so it will free up space for one task/routine to fire.

package main

import (
    "context"
    "fmt"
    "runtime"
    "time"
)

const maxGoRoutines = 2

// main itself go routine in golang
// #GO Routine - >1
func main() {
    ctx := context.Background()

    // set ctx timeout to exit the worker before main exits
    ctx, cancelFunc := context.WithTimeout(ctx, time.Second*4)
    defer cancelFunc()

    // #GO Routine - >2
    go worker(ctx)
    // easiest way  to hold the main is sleep for some time
    // hard way is we can use ctx with timeout , if we are implementing same in an api then it will run until we exit the api
    time.Sleep(time.Second * 5)

    //total GO Routines count=  #GO Routine(1)+ #GO Routine(2)+ maxGoRoutines


}
func worker(ctx context.Context) {

    // Buffered channel to limit the go routines
    limit := make(chan bool, maxGoRoutines)

    // channel to communicate between pool of go routines and print the cur data
    data := make(chan int, maxGoRoutines)

    // send some data to data channel so go routines can communicate
    data <- 1

    for {
        select {

        case <-ctx.Done():
            fmt.Println("no of go routines used in current execution", runtime.NumGoroutine())
           fmt.Println("exiting worker")
            // close the channel before return
            close(limit)

            return
        default:

            // if buffer is full then it will block the code here itself (it will not allow to trigger another go routine)

            // sending data to the limit channel
            limit <- true

            go func() {
                // sleep for sometime (this will be useful in understanding the limiter)
                time.Sleep(time.Second * 1)
                x := <-data
                fmt.Println(x)

                // increment the value by 1
                data <- x + 1

                // release the data from channel so that we can accommodate another go routine
                <-limit

            }()
        }

    }
}

Explanation:-

GO blocks the code if channel buffer is full and it will wait until someone to free up the space to unblock
In above Code we are loading data to channel and then firing the routine once buffer is full and go routines taking time to finish the tasks it will block the code to fire new routines until at-least one task is done from a pool.
for sake of data communication between used a data channel

Please feel free to add any comments/suggestion/ mistakes etc…👍

please follow medium for interesting things

Sai Teja – Medium

Read writing from Sai Teja on Medium. Software Engineer | Fitness trainer | https://www.linkedin.com/in/saitejamesala/ | https://topmate.io/saiteja180.

medium.com

Fun with GO Concurrency

sai teja — Tue, 29 Nov 2022 05:37:18 +0000

For those People who are fascinated about concurrency in GO.
i created a small funny program to understand the concurrency in Go with intuitive example.

Steps:-

Create a buffered channel to hold at-least values in channels will being utilized by two go routines in further
ch := make(chan int,2)
create two go routines which will consume data from the channel which we created in step 1
go iAcceptOnlyNegativeFolks(ch)
go iAcceptOnlyPositiveFolks(ch)
send initial signal/data to the channel
ch<-1
keep some sleep /use select case / some other strategy to block the main from exiting.
time.Sleep(time.Millisecond*5)

package main

import (
    "fmt"
    "time"
)

func main() {
    ch := make(chan int, 2)

    // initiate +ve & -ve folks
    go iAcceptOnlyNegativeFolks(ch)
    go iAcceptOnlyPositiveFolks(ch)

    // ignite fire between +ve & -ve folks 😅
    ch <- 1

    // this is just simple way to block main else go routines will exit along with main
    time.Sleep(time.Millisecond * 5)

}

func iAcceptOnlyPositiveFolks(ch chan int) {
    for {
        select {
        case x := <-ch:
            if x > 0 {
                fmt.Println(x, " is a positive folk")
                ch <- x + 1
            } else {
                // giving some motivation to -ve folk to become +ve folk
                ch <- x * (-1)
            }
        }
    }
}

func iAcceptOnlyNegativeFolks(ch chan int) {
    for {
        select {
        case x := <-ch:
            if x < 0 {
                fmt.Println(x, " is a negative folk")
                ch <- x - 1
            } else {
                // giving some motivation to +ve folk to become -ve folk
                ch <- x * (-1)
            }
        }
    }
}

sample output 1:-
-1  is a negative folk
-2  is a negative folk
-3  is a negative folk
4  is a positive folk
-5  is a negative folk
-6  is a negative folk
7  is a positive folk
-8  is a negative folk
9  is a positive folk
10  is a positive folk
11  is a positive folk

sample output 2:-
1  is a positive folk
2  is a positive folk
3  is a positive folk
-4  is a negative folk
-5  is a negative folk
6  is a positive folk
7  is a positive folk

Explanation:-

we are creating a buffered channel as a communication medium between two go routines
iAcceptOnlyNegativeFolks & iAcceptOnlyPositiveFolks functions will read data from the channel concurrently (which routine is accessible at that point of time it will read that data from the channel)
After running the above program we will get different out put for every execution because it is running concurrently we don’t know which routine will access the channel data at a point of time.

Please feel free to add any comments/suggestion/ mistakes etc…👍

please follow medium for interesting things

medium.com

Pub Sub model in golang

sai teja — Mon, 28 Nov 2022 08:42:32 +0000

created a basic pub-sub model in GO
please go thru it and let me know if any mistakes/suggestions etc

package main

import (
    "context"
    "fmt"
    "math/rand"
    "time"
)

var Subscribers map[string]chan interface{}

// init empty subscribers
func init() {
    Subscribers = make(map[string]chan interface{})
}

type SubscriberClient struct {
    Name string
    Ch   chan interface{}
}

func main() {

    // register clients to subscribe
    subscriberOne := RegisterClient("one")
    subscriberTwo := RegisterClient("two")

    ctx := context.Background()

    // initiate subscribers 

    // set timeout to exit the subscriber 1
    ctx, cfunc := context.WithTimeout(ctx, time.Second*12)

    // it will send the signal to subscriber to exit
    defer cfunc()
    go subscriberOne.sub(ctx)

    // set timeout to exit the subscriber 2
    ctx, cfunc2 := context.WithTimeout(ctx, time.Second*8)
    defer cfunc2()
    go subscriberTwo.sub(ctx)

    // initiate publisher

    // set timeout to exit the publisher
    ctx, cfunc3 := context.WithTimeout(ctx, time.Second*6)
    defer cfunc3()
    go pub(ctx)

    // i am just keeping ticker to send events only for 6 seconds after that it will exit the main
    // else based on use case publisher can handle this . if you are implementing this in
    // API's then no need for ticker because publisher will run until program exists
    select {
    case <-time.Tick(time.Second * 13):
        fmt.Println("main exiting")
        break
    }
}

// it will return subscriber client with name and channel to read
func RegisterClient(name string) *SubscriberClient {
    // for async we  use buffered channels
    // for sync we use unbuffered channels
    ch := make(chan interface{}, 4)

    // store it channel in our global producer list
    Subscribers[name] = ch

    // return the channel to subscriber so they can read from it
    return &SubscriberClient{Name: name, Ch: ch}
}

func pub(ctx context.Context) {
    for {
        select {
        //when to stop condition (publisher need to handle this ...)
        case <-ctx.Done():
            fmt.Println("publisher exiting")
            return

        // it will publish messages every 2 seconds
        // other way we can keep constraints when to publish
        case <-time.Tick(time.Second * 2):
            publishToAllConsumers(rand.Int())
        }
    }
}

// subscriber
func (s *SubscriberClient) sub(ctx context.Context) {
    for {
        select {
        //when to stop condition (subscriber need to handle this ...)
        case <-ctx.Done():
            fmt.Println(s.Name, "exiting")
            return
        case x := <-s.Ch:
            fmt.Println(x, " value from ", s.Name, " subscriber")
        }
    }
}

// it will publish all registered subscribers (implemented TC:-O(N)// we can enhance it further....)
func publishToAllConsumers(data interface{}) {
    for k, v := range Subscribers {
        fmt.Println("sending data to ", k)
        v <- data
    }
}

please follow medium for interesting things

Sai Teja – Medium

Read writing from Sai Teja on Medium. Software Engineer | Fitness trainer | Random Thinker. Every day, Sai Teja and thousands of other voices read, write, and share important stories on Medium.

medium.com

Data Streaming in Go Lang

sai teja — Fri, 25 Nov 2022 07:50:35 +0000

we may need to do streaming for few things while working with large scale systems like calling multiple API's whenever required and store it in cache if it is frequently used data etc... for these kind of things we may not need to create functions and call them whenever it is needed.Especially when you want to deal data asynchronously.In this case we can simply create a stream and read data from stream whenever it is required.
For readers code will be simple & clean 👍.

updated basic stream... please go thru the code below with explanation and let me know if any doubts / suggestions / mistakes in my code .

package main

import (
    "fmt"
    "math/rand"
)

func main() {
    streamingDataUsingChannelAndGoRoutines()
}
func streamingDataUsingChannelAndGoRoutines() {
    // send signal to stop channel once you are done with streaming the data (optional)(have to handled by stream owner)
    stop := make(chan bool)

    // it will return read only channel for readers
    publisher := stream(stop)

    read(publisher)

}


// it will return read only channel so it will not block the execution / it will not wait to write something into channel/avoids dead lock
func stream(stop <-chan bool) <-chan int {

    // unbuffered channel to stream data
    // this will useful when you want stream live data when ever required
    //ch := make(chan int)


    // buffered channel to stream data
    // this will be useful when you want to store some data in advance  in buffer or cache
    // ans use whenever it is required directly from buffer
    ch := make(chan int,2)

    // this go func will run in background and push data consistently to the channel
    go func() {

        // closing the channel once streaming  gets stop request
        defer close(ch)

        for {
            select {
            case <-stop:
                return
            case ch <- rand.Int(): // eg use case:- we do an api call and store the data here
            }
        }

    }()

    return ch
}
// reader
func read(value <-chan int) {
    // iterating/reading the data for 5 times
    for i := 1; i <= 5; i++ {

        // we can read from channel stream any number of times
        fmt.Println(<-value, <-value, i)

    }

}

please follow medium for interesting things

Sai Teja – Medium

Read writing from Sai Teja on Medium. Software Engineer | Fitness trainer | Random Thinker. Every day, Sai Teja and thousands of other voices read, write, and share important stories on Medium.

medium.com

Handle go routines gracefully

sai teja — Thu, 24 Nov 2022 09:55:16 +0000

Understanding and handling the Go routines is little bit tricky but once you got to know the cream of Go Routine it will be a cake walk of creating and handling the routines 👍
Updated the code with explanation
please go thru it and please feel free to add comments if i miss anything / you want to add something..!!

package main

import (
    "fmt"
    "sync"
    "time"
)

const maxRoutines = 5

func main() {
    HandleUsingWaitGroup()
    HandleUsingChannel()
}

func HandleUsingWaitGroup() {
    var wg sync.WaitGroup

    // add no of go routines to trigger
    wg.Add(maxRoutines)

    for i := 1; i <= maxRoutines; i++ {
        go waitGroupRoutine(&wg, i)
    }

    // it will block the main until all go routines run successfully
    // whenever wg.Done() informs wait group it will reduce the go routine count by 1
    // and it will block until it becomes zero(0)
    wg.Wait()
}
func waitGroupRoutine(wg *sync.WaitGroup, value int) {

    // this will inform  wait group once it is done with the task
    defer wg.Done()

    fmt.Println("i am from waitGroup routine ", value)

}

// this works with buffered and unbuffered channels
func HandleUsingChannel() {
    ch := make(chan int, maxRoutines+1)

    //uncomment below line if you want to test un-buffered channel
    //ch := make(chan int)

    // this channel acts as a gatekeeper for the main func
    exitCh := make(chan bool)

    // this channel acts as a mediator to inform gatekeeper to exit main once all tasks are done
    mediatorCh:=make(chan bool)

    // trigger go routine and wait for the data in background
    go channelRoutine(exitCh,mediatorCh, ch)

    for i := 1; i <= maxRoutines; i++ {

        // push data to channel
        ch <- i

        if i == maxRoutines {

            // tell mediator to inform exit channel to exit the main func
            // and it return the go routine / exit the go routine
            mediatorCh <- true

            // close the channel here and read residual data in routine
            // else we will get a dead lock error because data push to channel is done
            // and nothing to read from it and we are trying to access the same...
            close(ch)
        }

    }

    // it will block the main until all go routines run successfully
    select {
    case <-exitCh:
        close(exitCh)
        fmt.Println("exit from Channel main")
        break

    }
}

func channelRoutine(exitCh chan bool,mediatorCh chan bool, ch <-chan int) {
    for {
        select {
        case <-mediatorCh:
            // we can read from closed channel
            close(mediatorCh)

            // read residual data before exiting
            // if we use buffered channel size of our data length
            //it will have enough space to push data to channel
            // once it pushes everything it will exit and routine will close automatically
            //and to avoid data miss we have to read channel data before returning
            for v := range ch {

                // this will show the difference in logs while comparing buffered and un-buffered channels
                time.Sleep(time.Second * 1)
                fmt.Println(" i am from  channel routine exit state", v)
            }

            // here we are telling exit channel to exit the main
            exitCh<-true
            return
        case x := <-ch:
            fmt.Println(" i am from channel routine", x)


        }
    }
}

please follow medium for interesting things

Sai Teja – Medium

Read writing from Sai Teja on Medium. Software Engineer | Fitness trainer | Random Thinker. Every day, Sai Teja and thousands of other voices read, write, and share important stories on Medium.

medium.com