Forem: Mat

I made SQL easy in Go

Mat — Sun, 02 Feb 2025 10:04:01 +0000

I hate working with SQL. I dont know why but just something about using it in programming languages is ugly to me. Because of this I tend to avoid it at all costs. I am a total SQL noob.

When I've used it in projects in the past its been through libraries like prisma-client-go. This is a really cool package. You define your schema, and run 1 command and you instantly have fully typed Go code to work with. I even used it to create a monitoring mobile app called Cronus.

I used it a lot but as my schema's grew in complexity, so did the generated go files. It currently sits at 61mb. This would often cause my VSCode to lag and become unusable. Theres also the problem that as your schema grows, Intelisense becomes so overwhelming that its basically useless.

I spent the weekend creating a go package that I think is pretty useful. It abstracts away the vast majority of the complexity of working with SQL in Go.

Feel free to check out eazydb

Creating a Client

Not much to change here from the default

c, err := eazydb.NewClient(eazydb.ClientOptions{
    User:     "postgres",
    Password: "postgres",
    Host:     "localhost",
    Port:     "5432",
    Name:     "postgres",
    Type:     eazydb.POSTGRES,
})

Creating a table

This is where you start to see just how easy it is, theres also options to handle adding new fields from your structure automatically.

Any field with a json tag will be added to the Schema

type Users struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
    Age  int    `json:"age"`
}

// Create a table
_, err = c.NewTable("users").Fields(User{}).Key("id", dbtypes.SERIAL).Exec()
if err != nil {
    log.Fatalf("could not create table %v", err)
}

// Shorthand to use later
table := c.Table("users")

Inserting data

Could not be easier, create your data and add it, either 1 at a time or 5000! It'll be handled

// Insert data
users := makeUsers(5000)
metadata, err := table.Add(users).Exec()
if err != nil {
    log.Fatalf("could not insert users table %v", err)
}

Updating a field

Again, very easy. Just define what fields you want updated

// Update a field
Mat := &User{
    Age: 24,
}
metadata, err = table.Update(Mat).Where(
    *eazydb.String("name").Equals("Mat"),
).Exec()

Want to update every field? No problem

// Update a field
metadata, err = table.Update(Mat).Exec()

Deleting rows

Similar to updating

// Delete
metadata, err = table.Delete().Where(
    *eazydb.Int("age").Equals(40),
).Exec()

Want to limit? Eazy just add a MaxRows()

metadata, err = table.Delete().Where(
    *eazydb.Int("age").Equals(40),
).MaxRows(50).Exec()

Fetching data

Probably the best part, like every other operation. Just defined what you want to return. Any field with the json tag will be returned

var resp []User
_, err = table.Get(User{}).Where(
    *eazydb.Int("age").Equals(24),
).Exec(&resp)

A CLI to grab song lyrics

Mat — Sat, 18 May 2024 06:41:03 +0000

I created this tool because I am interested in doing some machine learning and thought that song lyrics would be a good start.

The CLI: https://github.com/mperkins808/lyrical/tree/main

The trouble is, these are rather hard to come by. I luckily managed to stumble across this API https://lyricsovh.docs.apiary.io/# that will return Lyrics for songs. It doesn't seem to be well known as the main way to get lyrics seems to be to use a web scraping API, which isn't ideal.

The CLI is simple.

It uses the spotify API to collect all the song names from an artist.

Once it has a list of songs, it calls the lyricsovh api to get the lyrics for each song.

Finally it outputs the result in a json file, perfect for further processing or parsing into a LLM learning data set.

Running a command

lyrical -i <SPOTIFY_CLIENT_ID> -s <SPOTIY_CLIENT_SECRET> -a "Drake" -d ../songs

The output

2024/05/18 16:12:03 fetched the lyrics for In My Feelings
2024/05/18 16:12:04 fetched the lyrics for Don’t Matter To Me (with Michael Jackson)
2024/05/18 16:12:04 fetched the lyrics for After Dark (feat. Static Major & Ty Dolla $ign)
2024/05/18 16:12:05 fetched the lyrics for Final Fantasy
2024/05/18 16:12:05 fetched the lyrics for March 14
2024/05/18 16:12:06 fetched the lyrics for Free Smoke

And the results. A json array like the following

  {
    "artists": ["Drake", "J. Cole"],
    "album": "For All The Dogs Scary Hours Edition",
    "name": "First Person Shooter (feat. J. Cole)",
    "lyrics": "(Pew, pew-pew)\r\nFirst person shooter mode, we turnin' your song to a funeral\r\nTo them niggas that say they wan' off us, you better be talkin' 'bout workin' in cubicles\r\nYeah, them boys had it locked, but I knew the code\r\nLot of niggas debatin' my numeral\r\nNot the three, not the two, I'm the U-N-O\n\nYeah\n\nNumero U-N-O\n\nMe and Drizzy, this shit like the Super Bowl\n\nMan, this shit damn near big as the—\n\n\n\nBig as the what? Big as the what? Big as the what?\n\nBig as the Super Bowl\n\nBut the difference is it's just two guys playin' shit that they did in the studio\n\nNiggas usually send they verses back to me and they be terrible, just like a two-year old\n\nI love a dinner with some fine women\n\nWhen they start debatin' about who the G.O.A.T\n\nI'm like \"Go 'head, say it then, who the G.O.A.T.?\n\n\"Who the G.O.A.T.? Who the G.O.A.T.? Who the G.O.A.T.?\"\n\n\"Who you bitches really rootin' for?\"\n\nLike a kid that act bad from January to November, nigga, it's just you and Cole\n\nBig as the what? Big as the what? Big as the what?(Ayy)\n\nBig as the Super Bowl\n\n\n\nNiggas so thirsty to put me in beef\n\nDissectin' my words and start lookin' too deep\n\nI look at the tweets and start suckin' my teeth\n\nI'm lettin' it rock 'cause I love the mystique\n\nI still wanna get me a song with YB\n\nCan't trust everything that you saw on IG\n\nJust know if I diss you, I'd make sure you know that I hit you like I'm on your caller ID\n\nI'm namin' the album The Fall Off, it's pretty ironic 'cause it ain't no fall off for me\n\nStill in this bitch gettin' bigger, they waitin' on the kid to come drop like a father to be\n\nLove when they argue the hardest MC\n\nIs it K-Dot? Is it Aubrey? Or me?\n\nWe the big three like we started a league, but right now, I feel like Muhammed Ali\n\nHuh, yeah, yeah, huh-huh, yeah, Muhammed Ali\n\nThe one that they call when they shit ain't connectin' no more, feel like I got a job in IT\n\nRhymin' with me is the biggest mistake\n\nThe Spider-Man meme is me lookin' at Drake\n\nIt's like we recruited your homies to beat demon deacons, we got 'em attending a wake\n\nHate how the gang gotta wait for the boss, man, this shit like a prison escape\n\nEverybody steppers, well fuck it, then everybody breakfast and I'm 'bout to clear up my plate (Huh, huh, huh)\n\nWhen I show up, it's motion picture blockbuster\n\nThe G.O.A.T. with the golden pin, the top toucher\n\nThe spot rusher, sprayed his whole shit up, the crop duster\n\nNot Russia, but apply pressure\n\nTo your cranium, Cole's automatic when aimin' 'em\n\nWith The Boy in the status, a stadium\n\nNigga\n\nPart II\n\n\n\nAyy, I'm 'bout to—, I'm bout to—\n\nI'm 'bout to—, yeah\n\nYeah\n\n\n\nI'm 'bout to click out on this shit\n\nI'm 'bout to click, woah\n\nI'm 'bout to click out on this shit\n\nI'm 'bout to click, woah\n\nI'm down to click down you hoes and make a crime scene\n\nI click the trigger on the stick like a high beam\n\nMan, I was Bentley wheel whippin' when I was nineteen\n\nShe call my number, leave her hangin', she got dry-cleaned\n\nShe got a Android, her messages is lime green\n\nI search one name, and end up seein' twenty tings\n\nNadine, Christine, Justine, Kathleen, Charlene, Pauline, Claudine\n\nMan, I pack 'em in this phone like some sardines\n\nAnd they send me naked pictures, it's the small things\n\nYou niggas is still takin' pictures on a dog stream\n\nMy youngers richer than you rappers and they all stream\n\nI really hate that you been sellin' them some false dreams\n\nMan, if your pub was up for sale, I buy the whole thing\n\nWill they ever give me flowers? Well, of course not\n\nThey don't wanna have that talk, 'cause it's a sore spot\n\nThey know The Boy the one they gotta boycott\n\nI told Jim and Jammer I use a GRAMMY as a door stop\n\nGirl gave me some head because I need it\n\nAnd if I fuck with you, then after I might eat it, wait\n\nNiggas talkin' 'bout when this gon' be repeated\n\nWhat the fuck bro? I'm one away from Michael\n\nNigga, beat it, nigga, beat it, what?\n\n\n\nBeat it, what? Beat it, what? Beat it, what? Beat it, what?\n\nBeat it, what? Beat it, what? Beat it, what? Beat it, what?\n\nBeat it, what? Beat it, what? Beat it, what? Beat it, what?\n\nDon't even pay me back on none them favors, I don't need it"
  }

Creating a Log Based Metric Exporter

Mat — Mon, 25 Mar 2024 07:57:24 +0000

This blog post will show how I created a simple log-based-exporter that measures the occurance of logs in kubernetes containers

Why

Major cloud providers like GCP & AWS offer log based metrics but at scale they can become extremely cost prohibitive. My aim was to create a cheaper solution that converts the logs into prometheus metrics out of the box, with the added benefit that they are measured in real time.

How it works

You specify the logs you want to measure, the containers that they occur in and the namespaces where the containers run. The exporter will automatically discover all matching pods as they appear
The exporter measures the metrics in real time, every time prometheus pings the /metrics endpoint, the gauge's are reset

Generating some logs

This quick little program will spit out copious amounts of logs. I've containerized it and deployed to a local cluster

package main

import (
    "fmt"
    "time"
)

func main() {
    i := 0
    for {
        l := fmt.Sprintf("log entry %v", i)
        fmt.Println(l)

        if i > 10000 {
            i = 0
        }
        i += 1

        time.Sleep(time.Millisecond * 50)
    }
}

Fetching these logs using the kube api

func StreamLogs(ctx context.Context, client *kubernetes.Clientset, ns string, pod string, callback LogCallback) error {
    opts := &corev1.PodLogOptions{
        Follow: true,
    }

    req := client.CoreV1().Pods(ns).GetLogs(pod, opts)
    stream, err := req.Stream(ctx)
    if err != nil {
        return err
    }
    defer stream.Close()

    buffer := make([]byte, 4096) // Adjust buffer size as needed
    for {
        n, readErr := stream.Read(buffer)
        if readErr != nil {
            // Check if it's the end of the stream
            if readErr == context.Canceled {
                log.Println("Stream canceled")
                break
            }
            return readErr
        }

        // Call the callback with the data chunk
        callback(ns, pod, buffer[:n])
    }

    return nil
}

This function is using a callback function to parse the data outside of the function. Heres an example implementation of the callback function.

We're doing a couple of things.

initializing the StreamLogs function
Define a callback that will measure logs as they appear
Incrementing metrics based on the outcome of the measurement

func monitorPodForRule(ctx context.Context, rule rules.Rule, ns string, pod string) {
    select {
    case <-ctx.Done():
        log.Infof("stopped monitoring %s/%s for rule %s", ns, pod, rule.Name)
        return
    default:
        callback := func(ns string, pod string, data []byte) {
            log.Debugf("namespace %s pod %s bytes processed %v", ns, pod, len(data))
            match := measureLogAgainstCondition(data, rule)
            if match {
                m.IncrementMetric(rule, ns, pod)
            } else if os.Getenv("EXPORT_ZERO") == "true" {
                m.SetMetric(rule, ns, pod, 0)
            }
        }

        client, err := kube.GenClient()
        if err != nil {
            log.Error(err)
            return
        }

        err = kube.StreamLogs(ctx, client, ns, pod, callback)
        if err != nil {
            log.Error(err)
        }
    }
}

The measurement is pretty simple, we just iterate through each condition and check for sub string matches

func measureLogAgainstCondition(data []byte, rule rules.Rule) bool {
    for _, cond := range rule.Condition {
        if !strings.Contains(string(data), cond) {
            return false
        }
    }
    return true
}

How to dynamically monitor pods as they appear

Kubernetes has the following hierachy Namespace --> Pod --> Container. Since pod names are random, we need to perform a lookup and find pods that contain the containers we wish to monitor.

This function will do the needed.

func findPodsForRule(ctx context.Context, rule rules.Rule) (map[string][]string, error) {
    log.Infof("finding pods for rule %s", rule.Name)

    // scan namespaces for containers
    client, err := kube.GenClient()
    if err != nil {
        return nil, err
    }

    namespaces, err := kube.ListNamespaces(ctx, client)
    if err != nil {
        return nil, err
    }

    validNamespaces := make([]string, 0)
    for _, ns := range rule.Namespace {
        for _, actualNS := range namespaces {
            if actualNS == ns {
                validNamespaces = append(validNamespaces, ns)
                break
            }
        }
    }

    if len(validNamespaces) == 0 {
        return nil, fmt.Errorf("rule %s no valid namespaces were found", rule.Name)
    }

    log.Infof("rule %s. %v of %v namespaces found", rule.Name, len(validNamespaces), len(rule.Namespace))

    matchingPods := make([]string, 0)
    result := make(map[string][]string)
    for _, ns := range validNamespaces {
        info, err := kube.ListContainers(ctx, client, ns)
        if err != nil {
            log.Errorf("could not list containers for namespace %s", ns)
            continue
        }

        pods, ok := info[ns]
        if !ok {
            log.Errorf("could not list containers for namespace %s", ns)
            continue
        }

        result[ns] = make([]string, 0)

        for podName, containers := range pods {
            for _, container := range containers {
                for _, ruleContainer := range rule.Container {
                    if ruleContainer == container {
                        matchingPods = append(matchingPods, podName)
                        result[ns] = append(result[ns], podName)
                        break
                    }
                }
            }
        }

    }

    if len(matchingPods) == 0 {
        return nil, fmt.Errorf("no matching pods found for rule %s", rule.Name)
    }

    log.Infof("for rule %s found the following namespaces %v", rule.Name, validNamespaces)
    log.Infof("for rule %s found the following pods %v", rule.Name, matchingPods)

    return result, nil
}

The last thing we need for the exporter is a way to bring up and down monitoring for a pod, we'll use context to control this. We'll run a scan every 15 seconds for new pods that match, any pods that no longer exist we'll cancel the monitoring for them, and bring up monitoring for the new pods.

func monitorRule(ctx context.Context, rule rules.Rule) {
    // refresh every 15 seconds
    ticker := time.NewTicker(15 * time.Second)
    defer ticker.Stop()

    for ; true; <-ticker.C {
        if ctx.Err() != nil {
            return
        }

        pods, err := findPodsForRule(ctx, rule)
        if err != nil {
            log.Errorf("error finding pods for rule: %v", err)
            continue
        }

        updateMonitoring(ctx, rule, pods)
    }
}

func updateMonitoring(ctx context.Context, rule rules.Rule, currentPods map[string][]string) {
    monitorState.Lock()
    defer monitorState.Unlock()

    // Create a set of current pods for quick lookup
    currentSet := make(map[string]bool)
    for ns, pods := range currentPods {
        for _, pod := range pods {
            key := podKey(ns, pod)
            currentSet[key] = true

            if _, monitored := monitorState.pods[key]; !monitored {
                // Start monitoring new pod
                podCtx, cancel := context.WithCancel(ctx)
                monitorState.pods[key] = monitoredPod{cancel: cancel}
                metrics.IncActiveMetrics()
                go monitorPodForRule(podCtx, rule, ns, pod)
            }
        }
    }

    // Check for pods that are no longer current and cancel their monitoring
    for key, mp := range monitorState.pods {
        if !currentSet[key] {
            mp.cancel()
            metrics.DecActiveMetrics()
            delete(monitorState.pods, key)
        }
    }
}

Configuration

If you remember the log generating app I made at the top, here I'll create a rule to measure the logs generated from it. In this rule we'll match everytime log entry & 4 are detected in the incoming log stream

[
  {
    "name": "log entry 400",
    "metric": "log_entry_400",
    "namespace": ["default"],
    "container": ["dummy"],
    "condition": ["4", "log entry"]
  }
]

Deploying to Kubernetes

Here is a full working example of a k8s deployment manifest. Once it's deployed we can now see that its picking up the log based metrics.

log_based_metric{metric="log_entry_400",name="log entry 4",namespace="default",pod="dummy-59b956b77c-2h78c"} 0

log_based_metric{metric="log_entry_400",name="log entry 4",namespace="default",pod="dummy-59b956b77c-99rvv"} 73

Thanks for reading!

Don't pay for a Content Management System. Build your own

Mat — Thu, 29 Feb 2024 09:48:46 +0000

Building your SEO is vital for any websites success. You should also be building organic traffic to your site before engaging in digital margeting. It can be prohibitively expensive, especially for something you do as a hobby. I also believe you should determine whether there is demand for your website before investing thousands of dollars in advertising.

cronusmonitoring.com is something that I really enjoy working on, without a monetization strategy in place I'd prefer not to pay $500 a day (Google's suggested bidding price) to drive traffic to the site. So I've turned to blogging as a means to improve SEO.

Ok so I've decided to start writing blogs, what approach should I take?

The website is composed of a React Frontend, A Go Server and a MySQL database. All of this is bundled up nicely into a docker image. If you want to use a similar setup, I've got an awesome-template that you can use.

I've considered the following options and at the end decided to build my own. I'll go through each and discuss their merits

Writing the blogs in the React and then building the image and deploying the update
Using a CMS like Butter
Building my own

Writing the blogs in the React and then building the image and deploying the update

I actually starting with this method and have wrote a handful of blog posts. But this became annoying very quickly. I'd write the post, then I'd have to build the docker image and then I'd have to deploy the updated image. I'd often spot typos I've made and then would have to repeat the process. It could take 30 minutes just to upload a single blog post.

Using a CMS like ButterCMS

It was after I had written a few posts that I knew something needed to change. So I did a google search for content management system and saw ButterCMS pop up. It looks fine but the cost is alot for what you get. I could've opted for the Free Developer Plan but if my website is successful one day I would like to monetize it. The next option is $99/month and you only get 50 posts. Seems like a lot in my opinion.

Building my own

After assesing my options I decided that the best approach was to build my own. This ended up being the best approach, as I managed to build a simple CMS in about an hour or 2. The best part is its free.

I'll now be showing how you can build your own.

We'll be doing the following

Adding markdown file support to React, this is much friendlier to write in than pure HTML
Adding some endpoints so we can upload markdown files and images to our server
Automating deployment using a python script and Github Actions

Adding markdown support to React

Adding the package

npm i react-markdown

Adding to a component

function BlogPost({ title, description, slug }) {
  const [data, setData] = useState("");

  useEffect(() => {
    document.title = title;
    meta("description", description);
    return () => {
      document.head.removeChild(metaSelector("description"));
    };
  }, []);

  useEffect(() => {
    getBlog(slug).then((resp) => {
      if (!resp) return;
      setData(resp);
    });
  }, [name]);

  if (data === "") {
    return <h3>Loading...</h3>;
  }

  return (
    <div className={styles.container}>
      <Markdown remarkPlugins={[remarkGfm]}>{data}</Markdown>
    </div>
  );
}

Fetching the blog posts from the server

Later I'll be showing the implementation of these endpoints. For now I'll show the client using the endpoints

export const getBlog = async (name: string): Promise<string | null> => {
  try {
    const resp = await axios.get(`/api/blog/${name}`);
    if (resp.status === 200) {
      return resp.data;
    }

    return null;
  } catch (error) {
    console.error(error.response?.data.message);
    return null;
  }
};

export interface BlogResponse {
  name: string;
  slug: string;
  description: string;
}

export const listBlogs = async (): Promise<BlogResponse[] | null> => {
  try {
    const resp = await axios.get(`/api/blog`);
    if (resp.status === 200) {
      return resp.data;
    }

    return null;
  } catch (error) {
    console.error(error.response?.data.message);
    return null;
  }
};

Display it all in a Blog page. I've also included a search box

export default function Blog() {
  const [list, setList] = useState<BlogResponse[]>([]);
  const [searchTerm, setSearchTerm] = useState("");

  useEffect(() => {
    listBlogs().then((resp) => {
      if (!resp) return;
      setList(resp);
    });
  }, []);

  const params = useParams();

  const { name } = params;

  const filteredList = list.filter((blog) =>
    blog.name.toLowerCase().includes(searchTerm.toLowerCase())
  );

  if (!name) {
    return (
      <div>
        <div className={styles.container}>
          <h1 className={styles.title}> Blogs </h1>
          <input
            type="text"
            placeholder="Search"
            value={searchTerm}
            onChange={(e) => setSearchTerm(e.target.value)}
            className={styles.searchInput}
          />
          <ul className={styles.listStyle}>
            {filteredList.map((blog, index) => (
              <BlogPreview
                desc={blog.description}
                name={blog.name}
                slug={blog.slug}
                key={`${blog.name}-${index}`}
              />
            ))}
          </ul>
        </div>
      </div>
    );
  }

  if (!list) return;

  return (
    <>
      <div className={styles.container}>
        <BlogPost
          slug={name}
          title={name ? name : list[getIndexOfSlug(list, name)].name}
          description={
            name ? name : list[getIndexOfSlug(list, name)].description
          }
        />
      </div>
    </>
  );
}

const getIndexOfSlug = (list: BlogResponse[], slug: string) => {
  for (let i = 0; i < list.length; i++) {
    if (list[i].slug === slug) return i;
  }
  return 0;
};

Updating the Server to allow creating blog files and also fetching of posts

I'll omit the authentication code. But you should protect your POST endpoints.

First of all we'll need a schema update for a Blog object. If you're unfamiliar with Prisma for Go I recommend you check out goprisma.org.

model Blog {
    slug        String   @id
    name        String
    description String   @default("")
    createdAt   DateTime @default(now())
}

Next, we'll create an endpoint for creating a blog post.

func RestCreateMD(w http.ResponseWriter, r *http.Request) {
    if err := r.ParseMultipartForm(32 << 20); err != nil { // 32MB is the max memory size
        responses.Response(w, http.StatusBadRequest, "error parsing form data")
        return
    }

    name := r.FormValue("name")
    desc := r.FormValue("description")
    slug := Slugify(name)


    file, _, err := r.FormFile("content")
    if err != nil {
        responses.Response(w, http.StatusBadRequest, "error retrieving the file")
        return
    }
    defer file.Close()

    // Read the contents of the file
    fileBytes, err := io.ReadAll(file)
    if err != nil {
        responses.Response(w, http.StatusInternalServerError, "error reading file content")
        return
    }

    // Save the blog post details to the database
    client, _ := util.GeneratePrismaClientWithContext()
    ctx := r.Context()
    defer util.CleanupPrismaClient(client)

    _, err = client.Blog.UpsertOne(db.Blog.Slug.Equals(slug)).Create(
        db.Blog.Slug.Set(slug),
        db.Blog.Name.Set(name),
        db.Blog.Description.Set(desc),
    ).Update(
        db.Blog.Slug.Set(slug),
        db.Blog.Name.Set(name),
        db.Blog.Description.Set(desc),
    ).Exec(ctx)

    if err != nil {
        log.Error(err)
        responses.Response(w, http.StatusInternalServerError, "failed to save MD file")
        return
    }

    // Define the path and filename for the MD file
    docPath := os.Getenv("BLOG_DOC_PATH")
    filename := name + ".md"
    filePath := filepath.Join(docPath, filename)

    // Save the MD file
    if err := os.WriteFile(filePath, fileBytes, 0644); err != nil {
        log.Error(err)
        responses.Response(w, http.StatusInternalServerError, "failed to save MD file")
        return
    }

    // Respond with the created blog post details (or just a success message)
    msg := fmt.Sprintf("Blog post '%s' created successfully", name)
    log.Info(msg)
    responses.Response(w, http.StatusCreated, msg)
}

Similarly, We'll create an endpoint for uploading an image.

func RestCreateImage(w http.ResponseWriter, r *http.Request) {
    if err := r.ParseMultipartForm(10 << 20); err != nil { // 32MB is the max memory size
        responses.Response(w, http.StatusBadRequest, "image is too large")
        return
    }
    file, handler, err := r.FormFile("image")

    if err != nil {
        responses.Response(w, http.StatusInternalServerError, "something went wrong")
        return
    }

    defer file.Close()

    imagePath := os.Getenv("BLOG_IMAGES_PATH")
    path := fmt.Sprintf("%s/%s", imagePath, handler.Filename)

    dst, err := os.Create(path)
    if err != nil {
        responses.Response(w, http.StatusInternalServerError, "something went wrong")
        return
    }

    defer dst.Close()

    if _, err = io.Copy(dst, file); err != nil {
        responses.Response(w, http.StatusInternalServerError, "something went wrong")
        return
    }

    responses.Response(w, http.StatusCreated, fmt.Sprintf("%v uploaded successfully", handler.Filename))
}

Now we need some endpoints for fetching the blog posts and images.

Creating an endpoint to fetch a blog post

func RestGetMD(w http.ResponseWriter, r *http.Request) {
    doc := chi.URLParam(r, "doc")

    if doc == "" {
        responses.Response(w, http.StatusNotFound, fmt.Sprintf("%s not found", doc))
        return
    }

    client, _ := util.GeneratePrismaClientWithContext()
    ctx := r.Context()
    defer util.CleanupPrismaClient(client)

    docDB, err := client.Blog.FindUnique(db.Blog.Slug.Equals(doc)).Exec(ctx)
    if err != nil {
        responses.Response(w, http.StatusNotFound, fmt.Sprintf("%s not found", doc))
        return
    }

    doc = docDB.Name + ".md"

    docPath := os.Getenv("BLOG_DOC_PATH")
    path := fmt.Sprintf("%s/%s", docPath, doc)
    http.ServeFile(w, r, path)
}

Creating an endpoint to fetch an image

func RestGetImage(w http.ResponseWriter, r *http.Request) {

    image := chi.URLParam(r, "image")

    if image == "" {
        responses.Response(w, http.StatusBadRequest, "supply an image")
        return
    }

    // regex is to only allow certain files to be requested and protects against mistakes and malicious requests
    re := regexp.MustCompile(`^/([^.]+)\.(svg|ico|jpg|png)$`)
    matches := re.FindStringSubmatch(r.RequestURI)

    if matches == nil {
        responses.Response(w, http.StatusNotFound, fmt.Sprintf("%s not found", image))
        return
    }

    imagePath := os.Getenv("BLOG_IMAGES_PATH")
    path := fmt.Sprintf("%s/%s", imagePath, image)

    http.ServeFile(w, r, path)

}

Finally we'll need an endpoint to list avaliable blog posts

func RestListMD(w http.ResponseWriter, r *http.Request) {

    client, _ := util.GeneratePrismaClientWithContext()
    ctx := r.Context()
    defer util.CleanupPrismaClient(client)

    files, err := listMarkdownFiles(client, ctx)
    if err != nil {
        log.Errorf("failed to list md files %v", err)
        responses.Response(w, http.StatusInternalServerError, "something went wrong")
        return
    }

    b, _ := json.Marshal(files)

    responses.BuildableResponse(w, http.StatusOK, b)
}

func listMarkdownFiles(client *db.PrismaClient, ctx context.Context) ([]BlogResponse, error) {
    filenames := make([]BlogResponse, 0)

    docs, err := client.Blog.FindMany().Exec(ctx)
    if err != nil {
        return nil, err
    }

    for _, doc := range docs {
        filenames = append(filenames, BlogResponse{
            Slug:        doc.Slug,
            Name:        doc.Name,
            Description: doc.Description,
        })
    }

    return filenames, nil
}

Great, all our functions are setup. All thats left to do is implement them

// Content Management (Blog)
mux.HandleFunc("/api/blog/image/{image}", cms.RestGetImage)
mux.Get("/api/blog/{doc}", cms.RestGetMD)
mux.Get("/api/blog", cms.RestListMD)
mux.Post("/api/blog/image", auth.PriviledgedAuth(cms.RestCreateImage))
mux.Post("/api/blog", auth.PriviledgedAuth(cms.RestCreateMD))

Automating deployment using a python script and Github Actions

This is how I have my blog folder setup. I have a docs directory that contains folders that each have a meta.json file and a .md file.

By the way. The above image is displayed using

![CMS setup in a Go React Server](/api/blog/image/cms-setup.png)

Here's what my meta.json looks like. For now I'm just supplying a description

{
  "description": "Setting up a Content Management System (CMS) is simple. Build your own for free using Go. Improve your workflow and upload new blogs automatically."
}

Next we'll create a simple python script that will iterate through the folder we have created and upload the assets

import json
import os

import requests


URL = "https://cronusmonitoring.com"
KEY = "<INSERT_KEY>"


def main():
    print("uploading blog to Cronus")
    print("Uploading images...")
    upload_images(f'{os.getenv("BLOG_PATH")}/images')
    print("Uploading blog posts...")
    upload_blogs(f'{os.getenv("BLOG_PATH")}/docs')

def upload_blogs(directory):
    for root, dirs, files in os.walk(directory):
        for file in files:
            if file != "meta.json":
                meta_path = os.path.join(root, "meta.json")
                md_path = os.path.join(root, file)
                if os.path.exists(meta_path):
                    with open(meta_path, "r") as meta_file:
                        obj = json.load(meta_file)
                        upload_file(md_path, "blog", obj["description"])


def upload_images(directory):
    for root, dirs, files in os.walk(directory):
        for file in files:
            file_path = os.path.join(root, file)
            upload_file(file_path, "image", "")


def upload_file(file_path, file_type, description):
    # Determine the correct endpoint and field name based on file type
    url = f"{URL}/api/blog"
    if file_type == "image":
        url = f"{url}/image"
        files = {"image": open(file_path, "rb")}
    else:  # Assuming file_type is "blog"
        files = {"content": open(file_path, "rb")}

    # Common data for both requests
    data = {
        "name": os.path.basename(file_path).split(".")[0],
        "description": description,
    }
    headers = {"Authorization": f"Bearer {KEY}"}

    response = requests.post(url, files=files, data=data, headers=headers)

    if response.status_code == 201:
        print(f"Successfully uploaded {file_path}")
    else:
        print(f"Failed to upload {file_path}. Status code: {response.status_code}")


if __name__ == "__main__":
    main()

Next we'll create a Github Action workflow, so that any time we push a change to the repo, the blog is updated on the website

name: Run Upload Script

on:
  push:
    branches:
      - main
    paths:
      - "blog/**/*"

jobs:
  run-script:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run upload.py
        env:
          BLOG_PATH: blog
        run: python blog/upload.py

Lets commit our code to the repo and see our blog appear on the website.

And thats it. Our post is now visible on the website. Thank you for reading.

Connecting local apps to remote servers. An advance Go guide.

Mat — Sun, 25 Feb 2024 04:27:02 +0000

This guide will show how I created a system that allows clients (mobiles in this case) to connect to local servers without exposing them to the internet. My exact use case is that I have a website cronusmonitoring.com that brings monitoring and alerting to mobile devices.

A major limitation with my service so far is that if a user wants to connect their prometheus server, they would need to expose it over the internet so Cronus can communicate with it.

This guide will be using the following software I have created

Enough rambling. Lets get started.

The proposed design

What do we need to get started

Some software that runs on a user's computer that forwards requests to the desired datasource
This software should establish a long lasting websocket with the server
The client, the Cronus mobile app reaches out to my server over HTTPS. These requests need to be translated and forwarded to the websocket.

Creating magicmirror

Establishing a connection to a websocket on a server

    uRemote, err := url.Parse(remote)
    if err != nil {
        log.Fatalf("Failed to parse remote URL: %v", err)
    }

    dialer := websocket.Dialer{HandshakeTimeout: 45 * time.Second}
    header := make(http.Header)
    if apikey != "" {
        header.Set("Authorization", fmt.Sprintf("Bearer %s", apikey))
    }

    connectionURL := fmt.Sprintf("%s?name=%s", uRemote.String(), name)

    conn, res, err := dialer.Dial(connectionURL, header)
    if res != nil && res.StatusCode != http.StatusSwitchingProtocols {
        msg, err := io.ReadAll(res.Body)
        if err != nil {
            msg = []byte("")
        }
        log.Fatalf("failed to connect to remote host: %v. response code %v with response \n%v", remote, res.Status, string(msg))
    }
    if err != nil {
        log.Errorf("Failed to connect to remote host: %v %v. Retrying...", remote, err)
        return false
    }

    defer conn.Close()

We need to encode and decode requests and responses. I've omitted the code for brevity, But what we are achieving is base64 encoding the following objects.

type EncodedRequest struct {
    Method  string              `json:"method"`
    Uri     string              `json:"uri"`
    Body    []byte              `json:"body"`
    Headers map[string][]string `json:"headers"`
}

type EncodedResponse struct {
    Body       []byte              `json:"body"`
    Headers    map[string][]string `json:"headers"`
    StatusCode int                 `json:"status_code"`
}

Now that we can decode messages into a request object. We can make that request.

// handleMessage takes a base64 encoded message, decodes it, and makes a HTTP request.
func HandleMessage(encoded []byte, local string) (string, error) {

    req, err := messages.DecodeRequest(encoded, local)

    if err != nil {
        return "", err
    }

    // Make the HTTP request using http.Client
    client := &http.Client{}
    response, err := client.Do(req)
    if err != nil {
        return "", fmt.Errorf("error making HTTP request: %v", err)
    }

    resp, err := messages.EncodeResponse(response)
    return resp, err

}

Finally, we can encode the response and send it back to the server.

        _, message, err := conn.ReadMessage()
        if err != nil {
            log.Errorf("Error reading message: %v. Attempting to reconnect...", err)
            return false
        }

        resp, err := HandleMessage(message, local)
        if err != nil {
            log.Errorf("Error handling message: %v", err)
            failedRead++
            continue
        }

        err = conn.WriteMessage(websocket.TextMessage, []byte(resp))
        if err != nil {
            log.Errorf("Error writing message: %v", err)
        }

magicmirror is now created, The link at the top will show the full source code for it.

Updating the Server to forward requests to magicmirror

First of all, we need to handle creating the connection. Using socketmanager we accept inbound connections and then store the connection in socketmanager. Once again i've omitted most of the code.

func WSMirrorHandler(w http.ResponseWriter, r *http.Request, upgrader websocket.Upgrader) {
    sm, err := socketmanager.GetSocketManagerFromContext(r.Context())

        ...

    // what the user calls the connection
    name := r.URL.Query().Get("name")

    conn, _ := upgrader.Upgrade(w, r, nil)

        ... 

    uid := fmt.Sprintf("%s-%s", userid, name)

  // adding the connection to socketmanager
    sm.Add(uid, name)
    sm.SetArb(uid, constants.MIRROR_CONNECTION, conn)
}

Now we need to forward HTTP requests to the connection. I have an endpoint that users query and returns data. In this endpoint we'll update it to forward those requests to magicmirror.

Extracting the connection from socketmanager

func GetSocketConnection(sm *socketmanager.SimpleSocketManager, id string, name string) (*websocket.Conn, error) {
    uid := fmt.Sprintf("%s-%s", id, name)
    arb := sm.GetArb(uid, constants.MIRROR_CONNECTION)
    if arb.Err != nil {
        return nil, arb.Err
    }
    conn := arb.Value.(*websocket.Conn)
    return conn, nil
}

Now we can use the connection to make requests

func FetchMirrorData(sm *socketmanager.SimpleSocketManager, req *http.Request, id string, name string) (*http.Response, error) {
    conn, err := GetSocketConnection(sm, id, name)

    if err != nil {
        return nil, err
    }

    encoded, err := encoder.EncodeRequest(req)
    if err != nil {
        return nil, err
    }

    conn.WriteMessage(1, []byte(encoded))

    _, msg, err := conn.ReadMessage()
    if err != nil {
        return nil, fmt.Errorf("failed to read connection error %v", err)
    }

    resp, err := encoder.DecodeResponse(string(msg))
    if err != nil {
        return nil, fmt.Errorf("failed to read connection message %v", err)
    }

    return resp, nil
}

Finally, for the server, lets add a cleanup function to remove inactive sockets from socketmanager

func MirrorHeartBeat(ctx context.Context, dur time.Duration) {
    for {
        sm, err := socketmanager.GetSocketManagerFromContext(ctx)
        if err != nil {
            log.Errorf("mirror heartbeat failed %v", err)
            time.Sleep(dur)
            continue
        }

        active := GetActiveMirrors(sm)
        for i := range active {

            arb := sm.GetArb(active[i], constants.MIRROR_CONNECTION)
            if arb.Err != nil {
                continue
            }
            conn := arb.Value.(*websocket.Conn)
            err := conn.WriteMessage(websocket.PingMessage, []byte{})
            if err != nil {
                sm.Remove(active[i])
            }
        }
        time.Sleep(dur)
    }
}

The code is done. Time to connect

Establishing a connection from my local computer to cronusmonitoring.com

docker run mattyp123/magicmirror --remote wss://cronusmonitoring.com/mirror --apikey <INSERT> --name localprometheus --local http://192.168.0.78:9090

Now we can select it as a datasource

And then run queries against it

And finally, we can view it on the app

Thanks for reading!

Reverse engineering the Grafana API to get the data from a dashboard

Mat — Sun, 18 Feb 2024 04:53:41 +0000

Grafana is fantastic, you can collect dozens of datasources and view them in dashboards. But what if you wanted to extend this to your own tool. Perhaps you're creating a mobile app for monitoring and alerting and thought that Grafana would be a great way to get a bunch of datasources for not much effort.

First things First.

Yes I'm aware that Grafana is open source but the method I used to find the API endpoints is far quicker than digging through hundreds of files in a codebase I'm not familiar with.

The plan

1. Get the dashboard json
Grafana dashboards are stored as json objects and are easily queryable via their API. Theres also documentation on doing just that. Easy stuff.

2. Figure out what api call is used to get data
Not as easy. This is not documented, so we'll have to inspect network requests using the browser to find the requests needed.

3. Combine step 1 and 2
I should be able to get the json for a dashboard and then make a request for the data without any hardcoding of parameters. If the Grafana client can do it, i can do it.

Setup

We'll need a Grafana service account and token to get started. This is easy and theres a guide for it.

Getting the dashboard json

If we look at a dashboard in Grafana. We'll see the uid in the url. Then a simple GET request to /api/dashboards/uid/<uid> will suffice.

For the sake of brevity, I've omitted most of the response. A small dashboard can easily be more than 1000 lines of json. We'll save this json for later.

{
    "dashboard": {
      "panels": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "c8f641f5-80c3-41e2-bf79-d307ae89cf8f"
          },
          "id": 1,
          "targets": [
            {
              "datasource": {
                "type": "prometheus",
                "uid": "c8f641f5-80c3-41e2-bf79-d307ae89cf8f"
              },
              "disableTextWrap": false,
              "editorMode": "builder",
              "expr": "page_requests_total",
              "fullMetaSearch": false,
              "includeNullMetadata": true,
              "instant": false,
              "legendFormat": "__auto",
              "range": true,
              "refId": "A",
              "useBackend": false
            }
          ],
          "title": "Panel Title",
          "type": "timeseries"
        }
      ],
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "title": "New dashboard",
      "uid": "bebca380-068d-463d-9c9c-1bb19cb8d2b3",
    }
  }

How does Grafana use that information to get data for a dashboard?

Lets look at the network requests to get some clues.

We can see that when refreshing the panel there are 2 network requests made. One for annotations and one for query, with the later one looking like the request we need to make. If we right click on the request we can copy it as a curl command. Once again I've removed all the fields we don't actually need.

curl 'http://localhost:3000/api/ds/query?ds_type=prometheus&requestId=Q105' \
  --data-raw '{"queries":[{"datasource":{"type":"prometheus","uid":"c8f641f5-80c3-41e2-bf79-d307ae89cf8f"},"disableTextWrap":false,"editorMode":"builder","expr":"page_requests_total","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"legendFormat":"__auto","range":true,"refId":"A","useBackend":false,"exemplar":false,"requestId":"1A","utcOffsetSec":39600,"interval":"","datasourceId":1,"intervalMs":30000,"maxDataPoints":568}],"from":"1708207962285","to":"1708229562285"}' \
  --compressed

So we've learnt 3 things.

We need to make a call to /api/ds/query?ds_type=prometheus&requestId=Q105
Its a POST request
The request object is pretty complex

To get a better view of the request object, lets view it as json.

{
  "queries": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "c8f641f5-80c3-41e2-bf79-d307ae89cf8f"
      },
      "disableTextWrap": false,
      "editorMode": "builder",
      "expr": "page_requests_total",
      "fullMetaSearch": false,
      "includeNullMetadata": true,
      "instant": false,
      "legendFormat": "__auto",
      "range": true,
      "refId": "A",
      "useBackend": false,
      "exemplar": false,
      "requestId": "1A",
      "utcOffsetSec": 39600,
      "interval": "",
      "datasourceId": 1,
      "intervalMs": 30000,
      "maxDataPoints": 568
    }
  ],
  "from": "1708207962285",
  "to": "1708229562285"
}

It was at this point that I lost hope. How am i meant to create all those fields for each request, with other dashboards surely having different request objects, but thats when I noticed the similarity between the request object and the dashboard json. The queries array is actually just the targets array from the dashboard, and the from and to are timestamps.

Updated game plan

Ok so we know we need to get the dashboard json, then we'll extract out the targets array for a panel, then we'll build the request.

Extract the dashboard uid and panel id from a URL
This function will get the arguments we need from a grafana url.

func ExtractArgs(urlStr string) (string, int) {
    parsedUrl, err := url.Parse(urlStr)
    if err != nil {
        return "", 0
    }

    segs := strings.Split(parsedUrl.Path, "/")
    var uid string
    if len(segs) >= 3 {
        uid = segs[2]
    } else {
        return "", 0
    }

    viewPanel := parsedUrl.Query().Get("viewPanel")
    if viewPanel == "" {
        return "", 0
    }

    id, err := strconv.ParseInt(viewPanel, 0, 0)
    if err != nil {
        return "", 0
    }

    return uid, int(id)
}

Filtering the panels until we find the one we want. then grabbing the targets array.

for i := range dashboard.Dashboard.Panels {
    p := dashboard.Dashboard.Panels[i]
    if p.ID != panelID {
        continue
    }
    targets = append(targets, p.Targets...)
}

Building the request

endTime := time.Now().Unix() * int64(1000)
startTime := start.Unix() * int64(1000)

request := GrafanaDataQueryRequest{
    Queries: targets,
    From:    fmt.Sprint(startTime),
    To:      fmt.Sprint(endTime),
}

b, err := json.Marshal(&request)
if err != nil {
return result, fmt.Errorf("failed to build request object %v", err)
}

query := fmt.Sprintf("%v://%v/api/ds/query", c.baseURL.Scheme, c.baseURL.Host)
req, err := c.NewRequest(http.MethodPost, query, bytes.NewBuffer(b))

From this point on I'll using the package I created grafanadata.

We can simply paste in a URL for a dashboard and it will spit out the raw data for a panel

package main

import (
    "encoding/json"
    "log"
    "os"
    "time"

    "github.com/mperkins808/grafanadata/go/pkg/grafanadata"
)

func main() {

    u := "http://localhost:3000/d/bebca380-068d-463d-9c9c-1bb19cb8d2b3/new-dashboard?orgId=1&viewPanel=2"
    t := "glsa_5N21WQvXza0oWkbqQvjOhII8yJYxGS0G_fbb82943"
    client, err := grafanadata.NewGrafanaClient(u, t)

    if err != nil {
        log.Fatal(err)
    }

    start := time.Now().Add(time.Hour * 24 * -7)
    data, err := client.GetPanelDataFromURL(u, start)

    if err != nil {
        log.Fatal(err)
    }

    log.Default().Println(data)

    b, _ := json.Marshal(&data)
    os.WriteFile("data.json", b, 0644)
}

The results

This panel produced a json file 4000 lines long, heres the important parts. I've also included a grafanadata.ConvertResultToPrometheusFormat() function to convert the grafana response into a prometheus format, this is 1/4th the size and I prefer this format.

{
      "status": 200,
      "frames": [
        {
          "schema": {
            "fields": [
              {

                "labels": {
                  "__name__": "go_memstats_alloc_bytes",
                  "instance": "cronus-saas:4000",
                  "job": "cronus-saas"
                }
              }
            ]
          },
          "data": {
            "values": [
              [
                1707732000000, 1707739200000
              ],
              [
                4493296, 3634360
              ]
            ]
          }
        }
      ]
    }

How I actually use this package

As i mentioned at the top. I'm creating a mobile app for monitoring and alerting and wanted to add Grafana as a datasource. Users can simply copy and paste their Grafana URL and their metrics will be avaliable.

Viewing the metrics on the App

Handling socket connections across functions in Go

Mat — Sun, 11 Feb 2024 02:14:16 +0000

This guide shows how to manage socket connections across different functions using the socketmanager package.

This is a problem that i needed to solve.

Say you have a website and you want to measure the current active users. On the client side its pretty simple, every time the website is loaded, establish a web socket connection to the server.

Sample client side code

function App() {
  useEffect(() => {
    let ws = "wss";

    if (window.location.protocol === "http:") ws = "ws";
    const socket = new WebSocket(
      `${ws}://${window.location.host}/active-users`
    );

    return () => {
      socket.close();
    };
  }, []);

Then on the server side you would have a handler for when the client connects to the server.

Sample server side code

Creating the handler

func WSHandler(w http.ResponseWriter, r *http.Request, upgrader websocket.Upgrader) {

    conn, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        fmt.Println(err.Error())
        return
    }

    defer conn.Close()
    for {
        _, _, err := conn.ReadMessage()
        if err != nil {
            break
        }

        time.Sleep(time.Second)
    }
}

Calling the handler for /active-users

    mux.HandleFunc("/active-users", func(w http.ResponseWriter, r *http.Request) {
        WSHandler(w, r, upgrader)
    })

This is great but theres one problem, You cant break out of the function to measure the active users. Each function WSHandler running is not aware of other WSHandler functions running.

This is where the package i created comes in to save the day! Heres how it works.

Install the package

go get -u github.com/mperkins808/socketmanager/go/pkg/socketmanager

Establish a socketmanager in your main function.

// creating the socket manager
sm := socketmanager.NewSimpleSocketManager()
// optionally create it and add to context instead 
ctx := socketmanager.NewSimpleSocketManager().WithContext(context.Background())

Now we can parse socketmanager as either a direct argument or through context through to our handlers

// directly 
mux.HandleFunc("/ws", func(w http.ResponseWriter, r *http.Request) {
        WSHandler(w, r, sm, upgrader)
    })

// through context 
mux.HandleFunc("/ws", func(w http.ResponseWriter, r *http.Request) {
        WSHandler(w, r.WithContext(ctx), upgrader)
    })

Now lets update our websocket handler so that when there is a new connection this is added to our list of active sockets.

Here everytime a new user connects on the client. Our list of active sockets is increased.

func WSHandler(w http.ResponseWriter, r *http.Request, upgrader websocket.Upgrader) {

    conn, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        fmt.Println(err.Error())
        return
    }
// get socket manager from context
    sm, err := socketmanager.GetSocketManagerFromContext(r.Context())
    if err != nil {
        fmt.Println(err.Error())
        return
    }
    sID := uuid.New().String()
    sm.Add(sID, sID)

    defer sm.Remove(sID)
    defer conn.Close()
    for {
        _, _, err := conn.ReadMessage()
        if err != nil {
            break
        }

        time.Sleep(time.Second)
    }
}

Finally. Lets create a function that runs in the background and logs the number of active users

    go func(ctx context.Context) {
        for {
            sm, err := socketmanager.GetSocketManagerFromContext(ctx)
            if err != nil {
                // handle err
            } else {
                fmt.Printf("active users %v\n", len(sm.GetActiveSockets()))
            }
            time.Sleep(time.Second)
        }
    }(ctx)

The end result

I just used the npm package wscat to act as the client for this tutorial. Then as each client connected, the number of active users increased.

Real use cases

Logging out the active users is nice but its not the reason i created this package. Its main use case is for instructing the client when to fetch more data. Say you have an endpoint that fetches some data, this data can update at any time and you would like the client to fetch the latest data when this occurs.

First step is to establish a websocket connection. Thats what the first code block in this tutorial is doing, but we'll expand it to handle an update message. So when the page loads we'll call handleFetchData(), and then if the update message is received we'll call handleFetchData() again

  useEffect(() => {

    handleFetchData();
    let ws = "wss";
    if (window.location.protocol === "http:") ws = "ws";

    const socket = new WebSocket(
      `${ws}://${window.location.host}/device-ws?id=${props.currentUser.uid}`
    );

    socket.addEventListener("message", (event) => {
      if (event.data === "update") {
        handleFetchData();
      }
    });
    return () => {
      socket.close();
    };
  }, [props.currentUser]);

Now lets update the websocket handler to check if an update is due, if it is then send the update message to the client.

This updated function is doing a few things.

getting the userid (id) from the query
adding that userid to the socket manager
if an update is due, then it sends the update message to that user client

func WSDeviceHandler(w http.ResponseWriter, r *http.Request, upgrader websocket.Upgrader) {
    id := r.URL.Query().Get("id")
    if id == "" {
        return
    }

    conn, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Error(err)
        return
    }
        sm, err := socketmanager.GetSocketManagerFromContext(ctx)
        if err != nil {
       // handle err
    } 
    sID := uuid.New().String()

    sm.Add(id, sID)
    defer sm.Remove(id)
    defer conn.Close()
    for {
        if err != nil {
            return
        }
        if sm.UpdateDue(id) {
            log.Info("update due for ", id)
            sm.SetUpdateDue(id, false)

            err = conn.WriteMessage(1, []byte("update"))
            if err != nil {
                return
            }

        }
        time.Sleep(time.Second)
    }
}

Finally we have another function that determines if an update is due for a user

func DataHandler(w http.ResponseWriter, r *http.Request) {
   ...
   // above work determining if an update is due
   // socketmanager retrieved from context like prior examples 
   sm.SetUpdateDue(id, true)
   ...

What did we achieve

Ability to manage active sockets across many functions
Extract state out of a websocket or http handler
Enable a more reactive client

Real example of how i use the package

I created a IOS app Cronus that brings Prometheus and Google Cloud Monitoring to mobiles. You can view your metrics and alerts wherever you are. The way i have set up the app is that IOS devices need to be added by scanning a QR code on cronusmonitoring.com. Every time a device is scanned and added. The page showing the active devices should automatically updated.

On the backend i have websocket that is measuring the active users on the /devices page. Then when a device is successfully onboarded (through a separate handler), socketmanager tells the websocket that that particular socket needs to send an update message. Finally on the user's page the client pulls their devices data when told to update.

Create a React Go Server in 2 minutes

Mat — Wed, 07 Feb 2024 13:12:55 +0000

I made this awesome template that comes with a well structured go server and a react frontend and i think its worth sharing.

I created this because I reached the limits of Next.js in my projects and was looking for a good Go alternative. Surprisingly this was really difficult to get set up, and examples were hard to come by.

If you have repo cloned it's 2-3 commands (depending if you have dependencies) to get it running.

For development

Running the frontend vite server

make app

Running the backend go server

make server

Thats it! The site will now be reachable on port 3000 by default.

For production

Building docker image

make build

Running the docker image

make run

I can definitely make improvements to the UI of the pages, however from my experience of using things like Vite and React. I end up ripping out all the extras anyway.

The easiest way to send mobile push notifications for developers

Mat — Wed, 31 Jan 2024 07:55:31 +0000

Today i will be showing the easiest way to send push notifications to yourself using Cronus Monitoring and Alerting.

We’ll be using the npm package cronus-alert to send an alert for an event. We’ll then receive this alert on our mobile device by using the Cronus Mobile App.

Download the IOS App.
Go to cronusmonitoring.com and create a free account.
Go to the Devices page. Choose a device name and then click generate. This will produce a QR code that you can scan with your camera app. Ensure that once you have added your device, you enable alerts for it.

Go the API page and create an api key. We’ll use this to authenticate our requests to Cronus.
Create a new folder and add an index.js file.
Import the cronus-alert package

npm i cronus-alert

Lets create a simple function that when called will send an alert

import { CronusAlert, Status } from "cronus-alert"

function main() {
    console.log("Starting up test.")

    const APIKEY = "COPY FROM STEP 4."
    const cronusAlert = new CronusAlert(APIKEY);

    cronusAlert
    .FireAlert(
      "The system is down", // Summary
      "Someone forgot to lock the gate", // Description
      Status.Firing // Optional status
    )
    .then((resp) => {
      console.log(resp.data); // API response
    })
    .catch((err) => {
      console.error(err); // network error
    });
}

main()

Run the file and you’ll see the alert appear on your device.

bun run index.ts

Congrats. You have the integration configured. You can send up to 1000 alerts a month for free.