Forem: Omar Ahmed

[Side Project] Post automated Youtube videos from Reddit

Omar Ahmed — Wed, 05 Oct 2022 18:49:14 +0000

Long introduction

I've always been curious about some videos that I come across on Facebook and YouTube that look a little strange. Like they were just generated by a robot. Think of a video playing with a little box of a guy reacting to it while their reaction is incredibly strange and appears utterly unrelated. They always feature stuff that either doesn't fit the caption or the content itself is unusual.

I finally understand why. These videos are all automated and performed by scripts. It also appears that "Youtube/Facebook automation" is popular right now.

I first became curious about "Youtube automation" after running across the word. I didn't know that some people were building scripts to scrape stuff from websites (like Facebook and Reddit, but largely Reddit, to be honest), process it, and then produce a video from it to put online.

Imagine this thing running, and having a cron-job to run and post a new video every day. Consider your potential earnings and the consistency of this. You simply need to worry about funding your server and maybe improving how the material is processed while being scraped.

There is a lot to say about this, but I'm here to explain how I used Golang to get several Tiktok videos from a subreddit, combine them, and post a video of the "Top 10 funny/cringiest Tiktok videos" on YouTube without knowing what the video's content was.

TLDR

Social media is really weird. Authentic content is becoming harder to find, and BTW! I managed to code a Golang script to scrape Tiktok videos from a Subreddit, combine them into a single video, and upload it to Youtube all automatically.

So how though?

So for starters, the initial idea is to split the big thing into smaller things and blah blah you know the boring engineering stuff. But I seriously wanted to make it to scale. So it made sense to have 3 Golang packages/modules:

Reddit Aggregator:
- This would be responsible for scraping a subreddit and extracting Tiktok videos out of it.
- Will save the results into a MongoDB collection - because schema is overrated -
Video Processor:
- This would grab unpublished results from MongoDB collection:
  - Downloads all videos on the server
  - Process each single video
  - Combine all videos into a single video
- Now obviously this seems a bit too much for a single service, but ... meh
Uploader:
- This will upload the final video to Youtube

TLDR

This is a side project, and I wanted it to be complex enough that Youtube would actually use it. However, spoiler alert: they won't. That is why I divided the code into three Golang packages to run them as microservices serverless functions. Spoiler alert again: they are not.

Reddit Aggregator

The aggregator is straightforward; all it does is use the Reddit API to obtain posts from a subreddit and store them in a MongoDB collection.

https://www.reddit.com/r/<SUBREDDIT>/<POSTS_TYPE>.json?limit=<LIMIT>
https://www.reddit.com/r/linux/hot.json?limit=10

Now not to keep it too simple, every single thing between < > is a variable, that can be provided before the run to make it more robust.

The aggregator will save each post on a DB collection that will look like this:

{
    "_id" : ObjectId("633c37a1740ff26f2433f99c"),
    "hash" : "78a0428f87a9adf7d692e5435466bc5b",
    "title" : "<POST_TITLE>",
    "video" : "<VIDEO_LINK>",
    "created_at" : ISODate("2022-10-04T15:39:45.808+02:00"),
    "updated_at" : ISODate("2022-10-04T15:39:45.808+02:00"),
    "published" : false
}

What are the hash and published fields for you wonder?

Well, hash is a simple MD5 hash of the video link, and on DB this field has a unique index so that we're sure that the aggregator never inserts duplicates. While published is for determining to not return a post/video that was already published to our Youtube channel, this way it will not slip into the process.

Video processor

So, this package will begin by getting a list of unpublished posts from the database, downloading them all to a specific directory, and then starting the processor to process them all.

Downloader

Downloader is using youtube-dl behind the scenes to download every single video, now why not just a simple HTTP GET request and write the file?

youtube-dl provide great support for downloading videos, from various sources.
The goal is to be scalable and to be able to even have multiple aggregators ( facebook aggregator, youtube aggregator, Vimeo... etc )
I already had a wrapper to run a CLI command safely that I did (sniff.go#L118)

Goroutines were required since downloading 10 videos one at a time seemed incredibly slow. I had to run 10 goroutines and wait for them to complete before going on to the next step.

With Goroutines in place, there comes a challenge, more on that will be on here

So to download a single video is just that:

const ydl_command = "youtube-dl %s -q -o %s.%%(ext)s"

func download(url, base, filename string) (string, string, error) {
    dest := path.Join(base, "downloads", filename)
    cmdString := fmt.Sprintf(ydl_command, url, dest)
    args := strings.Split(cmdString, " ")

    cmd := exec.Command(args[0], args[1:]...)
    out, err := helpers.RunCmd(cmd)

    return args[len(args)-1], out, err
}

Processor

ffmpeg is pure evil

The processor is identical to the downloader; it will run through each video and, given that all Tiktok videos are vertical and have a black background on the left and right sides, its sole function will be to create blurry background for the video.

So to achieve that goal, you will have to use ffmpeg which seems more evil than Regex.

Anyway for processing a single video this is the function to do it:


const (
    ffmpeg_quality = "1080k"
    ffmpeg_command = `ffmpeg -i %s -lavfi %s -vb %s -c:v libx264 -crf 20 %s.mp4 -n`
    ffmpeg_filters = `[0:v]scale=ih*16/9:-1,boxblur=luma_radius=min(h\,w)/20:luma_power=1:chroma_radius=min(cw\,ch)/20:chroma_power=1[bg];[bg][0:v]overlay=(W-w)/2:(H-h)/2,crop=h=iw*9/16`
)

func process(video downloader.Video, base string) (string, string, error) {
    dest := path.Join(base, "blurry", video.Post.Hash)
    cmdString := fmt.Sprintf(ffmpeg_command, video.Path, ffmpeg_filters, ffmpeg_quality, dest)
    args := strings.Split(cmdString, " ")

    cmd := exec.Command(args[0], args[1:]...)
    out, err := helpers.RunCmd(cmd)
    if strings.Contains(out, "already exists. Exiting") {
        err = nil
    }

    return args[len(args)-2], out, err
}

For the record, it took me ~1hr to fix an issue running ffmpeg command to learn that I had to change this:

`ffmpeg -i %s -lavfi '%s' -vb %s -c:v libx264 -crf 20 %s.mp4 -n`

to this:

`ffmpeg -i %s -lavfi %s -vb %s -c:v libx264 -crf 20 %s.mp4 -n`

Now of course since the script will be processing 10 videos, this needs to be done using Goroutines to benefit the concurrency. As with the downloader, this will be later explained in the concurrency challenge section

Merger

The merger is the processor's last step, where after adding the blurry background to all of the videos, it will merge them all into a single video. But it does that in steps:

Create a text file that has all of the videos paths in this format

file /some/path/blurry-file1.mp4
file /some/path/blurry-file2.mp4

Then pass that file to ffmpeg so that it combines all videos

ffmpeg -f concat -safe 0 -i all_videos.txt final.mp4

Then remove the temporary all_videos.txt file

func MergeAll(videos []ProcessedVideo, base, output string) error {
    temp, err := createVideosFile(videos, base)
    if err != nil {
        return err
    }

    out, err := runMerge(output, temp)
    if err != nil {
        log.Printf("error running command, output: %q", out)
        return err
    }

    return os.Remove(temp)
}

Uploader

This is the last piece of the puzzle where the final video gets uploaded to youtube, but to avoid having the same title always, there is an option to have a random title deduced from the posts we already have on the database.

Since the goal is to be able to publish a video every single day, you can just get all posts of today and get a single random post, and then make it the video's title.

type Config struct {
    VideoInfoType    string
    VideoTitle       string
    VideoDescription string
    DatabaseURI      string
}

func (c *Config) GetTitle() (string, error) {
    switch c.VideoInfoType {
    case VideoTypeManual:
        return c.VideoTitle, nil
    case VideoTypeRandom:
        db, cancel, err := getDbClient(c)
        if err != nil {
            return "", err
        }
        defer cancel()

        var posts []types.Post
        err = db.FindRandomOfToday(1, &posts)
        if err != nil {
            return "", err
        }
        p := posts[0]
        return fmt.Sprintf("Top 10 best/cringiest Tiktoks today: %s", p.Title), nil
    default:
        return "", fmt.Errorf("unknown video type %v", c.VideoInfoType)
    }
}

Besides that script of course will need client_credentials and API token to be able to use youtube API to upload the video.

upload := &youtube.Video{
  Snippet: &youtube.VideoSnippet{
    Title:       title,
    Description: description,
  },
  Status: &youtube.VideoStatus{
    PrivacyStatus: c.PrivacyStatus,
  },
}

call := service.Videos.Insert([]string{"snippet", "status"}, upload)

file, err := os.Open(c.OutputFile)
check(err)
defer file.Close()

r, err := call.Media(file).Do()
check(err)

Concurrency challenge

The main challenge for me was to be able to just concurrently run tasks I want to run, of which i have 2:

Downloading videos
Processing videos

These are the most expensive tasks I have in terms of time because if they run one by one it will take so much time obviously.

So I wanted to have a way of running a tasks pool, where I pass the number of needed threads, and the pool will just split the tasks across these threads. So I ended up implementing this:

type ThreadElement struct {
    Element interface{}
}

func Threadify(numOfThreads int, elements []ThreadElement, f func(args ...interface{})) {
    length := len(elements)
    each := length / numOfThreads
    acc := length - (numOfThreads * each)

    var wg sync.WaitGroup

    wg.Add(numOfThreads)

    start := 0

    for i := 0; i < numOfThreads; i++ {
        running := each

        if acc > 0 {
            running++
            acc--
        }

        go func(start, i int) {
            for j := 0; j < running; j++ {
                e := elements[start]
                f(e.Element)
                start++
            }

            wg.Done()
        }(start, i)

        start += running
    }

    wg.Wait()
}

Pretty basic right? and for sure there is a better way to do it, but hey it actually worked, and now I can download videos by calling it like this:

var (
  videos   []Video
  elements []helpers.ThreadElement
)

for _, p := range posts {
  elements = append(elements, ThreadElement{
    Element: p,
  })
}

Threadify(3, elements, func(args ...interface{}) {
  p := args[0].(types.Post)
  log.Printf("calling download %q", p.Hash)
  downloadedPath, out, err := download(p.Video, base, p.Hash)
  if err != nil {
    log.Fatalf("error downloading video: %q of this post: %q::: %q\ncommand output: %s", p.Video, p.Hash, err.Error(), out)
  }

  log.Printf("Downloaded video: %q with hash %q on %q\n", p.Title, p.Hash, downloadedPath)
  videos = append(videos, Video{
    Post: p,
    Path: strings.ReplaceAll(downloadedPath, "%(ext)s", "mp4"),
  })
})

fmt.Println("Videos downloaded", videos)

I loved how generic this function is, and that is I just can use it for anything I want to concurrently run, and the way you can pass elements to it, by just kind of mapping it into ThreadElement so that it understands what it will later pass to the callback function.

But still, that looked hard to maintain, and I asked myself there has to be a better way to do this, just out of curiosity i googled and came across ants which seemed exactly right for this kind of functionality I wanted, so I converted the same function to use ants, and it became this:

func Threadify(numOfThreads int, elements []ThreadElement, f func(args ...interface{})) error {
    var wg sync.WaitGroup

    p, err := ants.NewPoolWithFunc(numOfThreads, func(e interface{}) {
        f(e)
        wg.Done()
    })
    if err != nil {
        return err
    }

    defer p.Release()

    for _, e := range elements {
        wg.Add(1)
        err = p.Invoke(e.Element)
        if err != nil {
            return err
        }
    }

    wg.Wait()
    return nil
}

Now it even looks cleaner, and I just don't care anymore about how it works as long as it works, and the most beautiful thing is that the function signature is still the same, which means I could still use it in the same way I did and have it generic just as I wanted initially.

Lessons learned

Finally now after the project is complete, here are the top lessons I learned from:

I shouldn't be surprised to know about "Youtube automation", literally everything is possible nowadays.
Golang is just super easy, Goroutines are easy when you just think of them logically, but also they're easy to produce bugs.
Don't use any dotenv package in Golang, IDK why I did!! it just makes no sense at least in this kind of project.

Finally

I got the idea from this Reddit post huge kudos to the OP, he made it simply in bash, but I just had to be a nerd to do it in Golang :/

There is huge room for improvement indeed, but I'm happy with the results, mission is achieved now to the new side project.

MongoDB aggregations are easy

Omar Ahmed — Sun, 01 Aug 2021 06:01:15 +0000

Building aggregation pipelines sometimes just seems so hard, especially when you're working with NoSQL database such as MongoDB, since there is no defined schema, and there are just so many nested fields

Well in this article i'm not going to go deep into what aggregation query is, since i think MongoDB Docs explained this perfectly, but instead i will go through a kind of seeming-complex aggregation query, which is you probably going to face if you're currently with MongoDB and i will use this example here to just show you how aggregation pipelines are just logically easy, and it's always depends on how you think while building it

The example

Well for the sake of our example, let's consider that you have this silly products collection which has these 2 documents:

{
  "_id": {
    "$oid": "55c30ff62cfa09af198b465a"
  },
  "name": "Awesome Tshirt",
  "currency": "usd",
  "one_size": "xl",
  "variants": [
    {
      "type": "color",
      "base_sku": 132145,
      "items": [
        {
          "color": "Grey Melange",
          "price": 80,
          "sku": 1243252369
        },
        {
          "color": "Bottle Green",
          "price": 90,
          "sku": 1243252368
        },
        {
          "color": "Deep Charcoal Grey",
          "price": 80,
          "sku": 1243252376
        },
        {
          "color": "White",
          "price": 80,
          "sku": 1243252363
        },
        {
          "color": "Black",
          "price": 80,
          "sku": 1243252362
        }
      ]
    }
  ]
},
{
  "_id": {
    "$oid": "55c30ff62cfa09af198b465c"
  },
  "name": "Hacker Tshirt",
  "currency": "usd",
  "one_size": false,
  "variants": [
    {
      "type": "color",
      "base_sku": 132155,
      "items": [
        {
          "color": "Black",
          "price": 100,
          "sku": 87987963
        }
      ]
    },
    {
      "type": "size",
      "base_sku": 342434,
      "items": [
        {
          "size": "sm",
          "price": 100,
          "sku": 97896796
        },
        {
          "size": "xl",
          "price": 100,
          "sku": 43534534
        },
        {
          "size": "xxl",
          "price": 100,
          "sku": 76576532
        }
      ]
    }
  ]
}

and now our goal is to get the price of all grey T-shirts, so since as you can see that each product has it's price vary based on the variant itself, so we have to get both from variants items item object. Yes in this collection its so silly that somehow you have to choose of either having black T-shirt or XXl tshirt, but not both :D

Stage #1

The very first step when you create a pipeline is to always match your query, this to narrow down querying the whole collection to just a limited number of documents that match your criteria

You always want to narrow down the number of documents you're searching within at the very beginning of the aggregation pipeline, this will lead to faster queries

So let's do this, first we want to use only the documents that has color variant, and it has also grey color inside it's variant items. So this is how we're translating this:

    {
        '$match': {
            'variants': {
                '$elemMatch': {
                    'type': 'color', 
                    'items': {
                        '$elemMatch': {
                            'color': /grey/i
                        }
                    }
                }
            }
        }
    }

$elemMatch operator matches documents that contain an array field with at least one element that matches all the specified query criteria.

We're using $elemMatch here to find a variant of type color first, and then we're using it again to find an element of color that contains grey and its case insensitive - notice the regex /grey/i -

So this was our first step, notice that only 1 item will be returned in this case that has ID 55c30ff62cfa09af198b465a, since it is the only one with variants of type color that has grey color

Document 55c30ff62cfa09af198b465c it has variants of type color, but it has only black color

Stage #2

Now we still need to query a nested object that is inside variants (array) and also inside items (array), so its more like this variants -> items -> {color, price}

This seems complicated, and since we're dealing with array of objects here, what could make it easier? .. to just deal with this array as an object instead, so we can just leverage the dot notation syntax in Mongo

Well we can do that by just unwinding the array! simple as that, just flatten this array into objects by using $unwind

$unwind Deconstructs an array field from the input documents to output a document for each element

$unwind is so simple that the only required parameter to pass is path which is the path of the array you want to flatten

  {
    '$unwind': {
      'path': '$variants'
    }
  }

Notice the dollar sign before variants, we have to prefix the field name with it so Mongo can interpret it, it just tells Mongo to inject the actual value of variants

Now by then we will have also 1 document still, since variants array has only 1 element

Notice that variants is now object instead of array

Stage #3

Now what? we still need to query variants.items elements which is the same case as variants before we unwind it. So i guess we will have to flatten variants.items too, so next stage will be

  {
    '$unwind': {
      'path': '$variants.items'
    }
  }

Notice now that we can easily access items with dot notation, since variants is an object and not an array anymore, now these are the the new documents returned after this stage

items is an object now with 1 document per items element which is exactly what we need, but did you notice something strange?

Stage #4

Now we have documents with items.color value that does not contains grey we have Black, White, and Bottle Green as well, why is that?

Well that would be because our first $match stage was only getting the documents that have items with grey color, having this does not necessary means that it will magically just filter other colors from items, this our job to do now

So now we will need to get only the documents that has variants.items.color with greyish color, looks like another $match query, right?

  {
    '$match': {
    'variants.type': 'color', 
    'variants.items.color': /grey/i
  }

It is so simple now, here we're just saying, out of the result of stage 3 we just want the documents which has variants of type color and any item that has grey with case insensitive, that will return us these documents:

Stage #5

Now we have the documents, but we don't really care about all of these properties right? We only care about the color and the price

So since we're having a couple levels of nesting here variants -> items and we only care about price and color properties, then we need to focus/project these properties only and ignore any other stuff we don't need

$project Passes along the documents with the requested fields to the next stage in the pipeline.

$project is so simple also, you'll probably just need to know that:

It takes field name as a key and 1 or 0 as value (1 to show, 0 to hide)
_id is shown by default, unless you specify to hide it
You have to specify at least 1 output field

So our stage implementation will be simple as

  {
    '_id': 0,
    'color': '$variants.items.color',
    'price': '$variants.items.price'
  }

Notice that we passed '_id': 0 because we don't really care about the document ID - at least not in this example, normally you'll need it though - so we just hid it

So now this will be the final result

The aggregation query

db.getCollection('products').aggregate([
  {
    '$match': {
      'variants': {
        '$elemMatch': {
          'type': 'color', 
          'items': {
            '$elemMatch': {
              'color': new RegExp('grey', 'i')
            }
          }
        }
      }
    }
  },
  {
    '$unwind': {
      'path': '$variants'
    }
  },
  {
    '$unwind': {
      'path': '$variants.items'
    }
  },
  {
    '$match': {
      'variants.type': 'color', 
      'variants.items.color': new RegExp('grey', 'i')
    }
  },
  {
    $project:  {
      '_id': 0,
      'color': '$variants.items.color',
      'price': '$variants.items.price'
    }
  }
])

Final words

As you can see it's pretty straight forward, and it's quite easy and seems quite logical too, building aggregation pipelines stages is just like actually talking to rubber duck while coding, instead this time you're just talking to Mongo shell

This post was originally published on my blog blog.mrg.sh

I Don’t Know!

Omar Ahmed — Wed, 28 Jul 2021 23:56:13 +0000

Just finished my working day and wow today was just filled up that it made me wonder, how long I’m gonna last working like this, and most importantly, do I even want this to end? Or do I like it? I don’t know.

Meetings are indeed time-consuming but only if they were with the wrong people, or if they were just useless, thank god that is not my case.

The day started with an ~1hr long meeting, discussing some challenges that are facing us related to our upcoming features, and oh these challenges are indeed complex and a brand new type for me probably I can write about them whenever I get the chance.

Having challenges are quite good training and experience booster, but do you like having constant challenges? Do I like that?

Then my day went on to exploring some emails, checking snd reviewing some PRs, till I was interrupted with some issue, that I kinda didn’t have the time to check it and it was quite important. I checked the card, did some quick investigation (~1hr) and hey, it was not replicating! Smooth eh? Typical bug, the support team will say it exists but you can’t replicate it, no surprise there.

I checked again, went deep this time by checking some corner cases, read the code again tried to be at least sure before pulling out my big gun, but it was in vain. So I had to do it... I called one of our support guys, the one reported it, we hoped in a meeting, that lasted for ~1hr and we come up with this besides exchanging some laughs

Issue is not an issue, and it was something wrong with the client configuration.
There is another different type of issue happening (nice trade eh?)

so that was it, now I had to eat something and get some fresh air, but oh wait... it’s hot outside, so just eat something while watching some YouTube videos.

I finished and went back to a task that I planned on doing, we have an issue and we need to check for a memory leak. So ideas were thrown at my first meeting to try flame graphs, and heap dumps, so here I go trying.

I tried to use perf command to monitor a Nodejs running server, got nothing, just weird ASCII characters on the output, tried parsing them, but it just didn’t look right, so now I just have to try heap… oh message from DevOps what they want? They’re asking about something related to the database, I have to give them a query now, and it’s a kinda complex one. Okay, let’s first start the Vagrant machine, hop into Robo3t to access my local database, and start building a 5 stages aggregation pipeline to return a field value from a nested array of objects.

Now where we’re we? Oh, heap dump, yeah let’s try to use heapdump package to generate a heap dump for my running Nodejs server, as usual pasting some code from the documentation, tweaking it a bit, and now .. a message from scrum master? I need to review these 2 PRs ASAP that belong to another team but they’re touching our team functionality so we can do the release. Ding it, I didn’t notice these PRs so I gotta check them.

Now, what is this variable for? Let me open the repository, look for it, I get it now.. wait.. no I let’s understand the code, oh what’s that? Let me call the developer he knows better, I message him we hop into a meeting and he explains the issue perfectly and we find that this code has a problem and it wouldn’t properly work with a specific corner case. Okay great now I just jest changes on the PR, type a comment summarizing our meeting discussion, and back to where I was.

Now what? Where? Oh, heap dump I left the server running, so I just have to hit a specific endpoint and the heap snapshot will be written to the disk for me to analyze it, let’s do that. Endpoint called, no errors, but the server is restarting … hmmm is this expected? Does this package somehow affect the runtime? IDK but hey doesn’t matter now, let me just see the heap dump.

cat ./heaptest.heapsnapshot

It’s empty! Hmm weird, let me do it again. After 4 times repeating the file is still empty now I’m pissed off it should work, right? Okay, let just google that, I found tutorials, articles, and StackOverflow questions but they’re all the same, maybe I’m missing something.

In the meanwhile, I’m replying to messages and answering questions, but what got me mentally blocked is another PR, but this time a comment I made a while ago and the developer is asking about it. The comment was about some clean code stuff and how it’s better not to pollute a function with custom conditions and instead think of another way of doing it. We discussed and he suggested a perfect approach, so now it was time to continue.

But I lost it already, I’m not as focused as before, so I just did a couple of other tries, and called it day at 6 pm already.

Now what?

This was indeed a busy day, but I’m kinda used to it by now, but Is this okay? I do let know.

This is definitely not bad to make you practice multitasking and make you available at all times, but it comes with a cost on your concentration. I’m not here to propose a solution though, I’m just here to share and for the hope that I will find the solution by just writing this kind of a daily routine.

Do I like being busy? Yeah, it makes my mind spin and thinks, and mostly shifting from thinking on a certain problem to another one, helps me find a solution for the first problem. I got a weird mind IKR?

This article was originally published to my blog blog.mrg.sh

Why you should start your wargames journey?

Omar Ahmed — Wed, 28 Jul 2021 15:30:14 +0000

Lately i been exploring some hacking and offensive security content, so basically it involved me just turning off my room lights, me wearing a hacker hoodie, and watching
John Hammond videos and nodding my head, gasping and a little bit actually trying to mimic what he does on my machine

So wargames bandit was something that John Hammond actually recommended to get familiar with Linux for anyone that wanted to start hacking and become a hacker. I thought oh what is this wargames and started googling

I found it, wargames bandit and i read it and thought, well this is going to be easy, i know Linux i been working on it for about 6 years now, managing and deploying servers and use it everyday since then for my daily life, i even play games on it for god sake

What is wargames bandit?

Well this game is simple, its consists of levels, you start at level 0 and make your way through by finishing the level, basically finishing a level means that you get the password for the next level, and so on

Bandit is made for absolute wargames beginners and it aims to get players familiar with Linux and the wargames style

Why wargames?

Well first because it's fun, and i mean it, it is actually fun, and no matter how you're using Linux, mostly you'll learn a lot by playing it

You will know more about some commands that you might be using everyday like grep, ssh, cat, or find. And even new commands (at least for me) like strings, uniq, bzip2, s_client, and xxd

You will write a simple bash scripts, which is super fun especially if you're like me, someone that loves to automate some tedious work by writing scripts

You will understand some Unix concepts and even interact with text editors like vi for example, and even know how to deal with different shells

Also you will interact with git and running some git commands so you can look for some secret credentials in git branches, commits, or tags

And of course you will know some hacking concepts, and how to protect your server by avoiding some silly mistakes, that we all may be doing

As a developer you'll find this also important, especially if you're interacting with servers on any environment, you'll learn essential security concepts and it will open your eyes to a whole new world

The gist

You should give a try , i started it on a weekend and i found myself finishing the 33 levels in 1 day that's how entertaining this is, and i will probably check other wargame as well maybe natas? who knows!

This article was originally published on my personal blog blog.mrg.sh

Introducing and installing API Platform on your machine

Omar Ahmed — Wed, 10 Oct 2018 14:16:18 +0000

I’m a full stack web developer, who takes API seriously. Its been 2 years now and i’m still doing the same steps to create every API i implemented, sure i was automating some stuff using some frameworks who makes things easy, organised and simple starting with Express and Sails, to using Symfony and finally API Platform!

Two months ago -give or take I’ve a bad memory- I was thinking that i hope one day i’ll automate the API design and implementation to be much faster, maybe running a couple of PHP commands to create a whole entities with its relationships and API routes, and expose them, with a well auto-generated docs for entities based on annotations -maybe- .. well that till a friend of mine told me about this awesome framework which he stumbled upon and “we need to check it out, this 100% gonna be awesome” — he said. And he was right, they did the same thing i wanted to do AND MORE.

pffff!

Enough nonsense, i’m not a good writer anyway. Lets get into installation:

First of all you need to have docker & docker-compose installed on your machine you, you can install it by following the official docker documentation here.

You need to clone API Platform repo, by running this command git clone git@github.com:api-platform/api-platform.git
Then while you’re standing on the main root of the repo. run the following command to actually run the platform containers docker-compose up -d

If you’ve ran into this error or similar

ERROR: for client Cannot start service client: driver failed programming external connectivity on endpoint api-platform_client_1 (7d82cb397e0a882f7474f227541ca43eb804316ece68c8a7dc5ac9af91972202): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

that actually self explanatory it means that port 80 is in use in your current OS, so check which service is using it and kill it or better change its port. In my case this service was apache so a simple sudo service apache2 stop was enough to do the trick.
Everything should be working now, docker-compose up -d should be full with green done statements indicating everything is working. You should be able to head to http://localhost first then navigating to https://localhost and see the cool API Platform PWA client.

But what if its not!
You may run into this issue here, Its okay i ran into it and jcarrier-vp solution did it for me.

You’re now ready to create your API and consider following API Platform awesome documentation its really helpful and they have a cool walk through that will help you get to know it more.