Forem: Juan Pablo Ramirez

New Frontiers

Juan Pablo Ramirez — Thu, 27 Aug 2020 06:24:02 +0000

We are excited to launch an alpha of #space related libraries that will be integrated in our #ResilientOne engine.

We added ow.ly/Rut250BahLe to our portfolio as a clear effort to join the space community.

We hope you can enjoy it and leave comments below.

Coding Hands vs Bouldering

Juan Pablo Ramirez — Tue, 07 Jul 2020 05:39:18 +0000

After years, and I say .. YEARS, I manage to drag myself to a Bouldering session, I can say without shame that I'm really really bad at it, will get better, just keep going, right?

I forgot completely how hard it was on your hands and forearms, so for all of us that we spend hours and hours typing and coding, I completely recommend it.

And yes of course now I can barely type... or use a fork... but hey, will be awesome tomorrow, when my shoulders and arms hurt more and more.

But again, I completely recommend it!. GO have fun, fall and try again.

What you guys do for relax?

COVID-19 and world today

Juan Pablo Ramirez — Mon, 06 Jul 2020 16:44:49 +0000

Has been several months since the world suffer one of the most precise and dedicated zero day exploits of our history.
(For those who does not know what a zero day is go here, but long story short, we were hit by something we did not know that existed, and had no defense against it until was too late, and... made every infrastructure and protocol fail, every security dude worse nightmare).

Literally this little virus spread across our lives and disrupt every industry, causing major collapses across the world, across culture, governments and industries.

As humans, we failed to defend ourselves.

One clear thing that COVID-19 is still doing to us, is pinning the out liars of our world, forcing our hand to face our biggest fears, ourselves.
From the selfish hoarders, at the beginning, or the leaders that did not listen the warnings, the CEO's and the market gurus that lied about their investments and put everybody at risk.

We can lie to each other all we want.. we can hide it behind technicalities, and try to defend our countries, our flags, our political party, but we keep forgetting one thing....

Like a computer Virus, COVID-19 does not discriminate, will affect us and change us if we don't listen and don't respect the rules, that's what has been doing all this time.

I've been a security architect for almost 18 years now, worked in several companies with different levels of training and expertise, teach security DevOps here and there and security architecture for Cloud Architects and the like. When a security breach reaches a client, it's 99.9% Human error, either in the perimeter, software pipeline, bug or whatever else, it's always a human that forgot to check, and a protocol that forgot to verify that, basically I see this happening all the time, seeing it at a massive scale like COVID-19 it's terrifying.

I saw during this time all level of denial, from businesses that could see this happening miles away and decided to face it and put their employees at risk, to governments that decided not to listen to our very own experts.

As human race, we failed, and 2020 has been our defeat year.
We failed to collaborate as a specie, as one world, we failed obeying simple orders, driven by our own selfishness, we failed protecting our elders, as we forgot about them when we did not take care of ourselves.

And the worst part is .. we could avoid this, prevent it from getting it worst, and we are still unable to do so.

Since the beginning the data was there, our expert told us what could happen if we did not take care of this threat sooner. Was simple, very simple rules.
From a computer perspective... is like " Do not believe that a Nigerian prince is sending you money" or " Do not click on that file that says.. 'install-me.exe'" or "send your bank password over an email".

I do believe that from this we are going to recover, be better humans, listen, and prepare our lives to be aware of what we are doing, I certainly hope so, that this zero exploit vulnerability and all the enormous cost in lives is worth something.

To learn to be prepared, to trust, to listen, to forget once and for all that is not just about us individually and is about us as a whole.

Of course this post is a rant, a rant from a security guy that just want to be better every day and that learned so much during this year, so much about me but also about the world around me, but it's seeing so much pain, and bad decisions being made day by day, by the people the suppose to protect us.

I will end this rant with a question.

What you learn this 2020 ? Good or bad...what this hack to society teach you.?

Thanks!
Stay safe, keep safe, keep learning.

Secure Devops Tactics

Juan Pablo Ramirez — Fri, 03 Jul 2020 13:08:52 +0000

So security, if we could agree, it's a catch up game.

I have more than 15 years of security experience and every single time I felt that I was running behind, behind developers, business goals, release times, policy, and many many more reasons.

Trying to reason a deadline or QA, Threat modeling constantly and getting into the CD/CI in the development environment is always a struggle.

Either you have an backlog of the size of Iguazu falls, and can barely catch up, and let's face it, developers are extremely clever to back channel to get faster their feature request from an extremely ever hungrier business development.

And security can wait right? After all nothing can really happen in a few days, or hours and we can always add more firewall and so on.

But that's not the case ever, as we struggle to automate tests, incorporate more checklist and be in every meeting, every decision. One things is obvious, we are an intrinsic part of development and we should become the extra step friend that make it easier and tell everybody to wear a coat because is cold, or take an umbrella because might rain... without being the paranoid or a pain.

For that reason I started a small repo on GitHub , and a meet up weekly, to tell stories, and share advises, small recipes, little pieces of code, that can go a long way in the SecDevOps, helping fellow Security Engineers and Architects to evangelize caution.

Security should be about protecting, oversight, recommendations and thoughtful watch, we are here to help. Not to be an inconvenience or a barrier to business.

HTML to PDF in GO

Juan Pablo Ramirez — Fri, 03 Jul 2020 11:39:03 +0000

Recently I had a project that required to get thousands of URL's and convert them into PDF. After reviewing a few and complicated tools I decided to create my own.

I admit that I'm bias towards GO, as I believe is an extremely versatile, fast and easy to pick up language. I did tried to write the same in Python, but this small tools cost me around 2 hours in GO, while.. in python in my case I would be still looking for examples and code styles.

Another was the package system, I really like the whole concept of sending only a binary thing.

It's simple enough for now, as it just uses chromium browser to do the PDF, I added classes and functions to also read an Excel file, as my project required to do it.

I think that the best part is the command line capabilities, the configuration file and the usage of the makefile to automate the building and the release in github.

I will continue to add functionality to this tool, please , feel free to comment, ask for features, PR and so on.

https://github.com/epyphite/html2pdf

Thanks!!

Distributed Rant Series Part 2

Juan Pablo Ramirez — Fri, 08 Nov 2019 20:09:03 +0000

Continuing with my "discussion" and following the guide that I started in my last post I would like to introduce my own personal choice for structuring go projects.

So what's the long term goal:

Build Client / Server applications.
Has to be web based.
Has implement Security standards
Has to be cross platform (lets cover non mobile OS for now)
Keeping it as simple as possible...

Follow this Github repository to keep track of this project.

So... let's get started.

[ CODE PART STARTS HERE ]

My development environment:

Ubuntu 19.04.
go1.13.1 linux/amd64

For me getting used to GoLang GOPATH (later versions) was complicated, as an old Python developer, projects were all encapsulated in virtualenv and other horrors. ( Don't get me wrong, I love virtualenv)

But I must admit that getting the order that GoLang required from the start boosted my productivity, solve a few issues, and reduce for sure many headaches.

So my environment will start always here:


/home/{user}/projects/src/

Clone my repo

➜ git clone git@github.com:jpramirez/go-web-starterpack.git

Cloning into 'go-web-starterpack'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 4 (delta 0), pack-reused 0
Receiving objects: 100% (4/4), done.
➜  cd go-web-starterpack
➜  go-web-starterpack git:(MakefileInit) ls -ltrh
total 4,0K
-rw-r--r-- 1 shellcheff shellcheff 2,4K nov  8 12:30 Makefile
➜  go-web-starterpack git:(MakefileInit)

I was never a fan of C, C++, Makefile, deps and all that, made my brain hurt just by thinking about it, so many dependencies, so many variables, files, libraries... Why someone will create such a horror!!...

Now.. to be honest the best way to start working with Go for me was a makefile.

Easy, plain, no BS, is somehow pretty standard to any Linux environment.

Lets check that file then


TIME=$(shell date +"%Y%m%d.%H%M%S")
VERSION=0.1.1-alpha-0.8
BINARY_NAME=go-webstarter-go.v1

BINARY_NAME_SERVER=go-webstarter-server.v1


BASE_FOLDER = $(shell pwd)
BUILD_FOLDER  = $(shell pwd)/build

FLAGS_LINUX   = CGO_LDFLAGS="-L./LIB -Wl,-rpath -Wl,\$ORIGIN/LIB" CGO_ENABLED=1 GOOS=linux GOARCH=amd64  
FLAGS_DARWIN  = OSXCROSS_NO_INCLUDE_PATH_WARNINGS=1 MACOSX_DEPLOYMENT_TARGET=10.6 CC=o64-clang CXX=o64-clang++ CGO_ENABLED=0
FLAGS_FREEBSD = GOOS=freebsd GOARCH=amd64 CGO_ENABLED=1
FLAGS_WINDOWS = GOOS=windows GOARCH=amd64 CC=i686-w64-mingw32-gcc CGO_ENABLED=1 

GOFLAGS_WINDOWS = -ldflags -H=windowsgui


init: 
    @mkdir -vp $(BASE_FOLDER)/assets
    @mkdir -vp $(BASE_FOLDER)/build
    @mkdir -vp $(BASE_FOLDER)/cmd
    @mkdir -vp $(BASE_FOLDER)/config
    @mkdir -vp $(BASE_FOLDER)/extras
    @mkdir -vp $(BASE_FOLDER)/pkg/constants
    @mkdir -vp $(BASE_FOLDER)/web
    @mkdir -vp $(BASE_FOLDER)/third-party
    @mkdir -vp $(BASE_FOLDER)/api
    @mkdir -vp $(BASE_FOLDER)/vendor
    @echo "Creating Base Files" 
    @touch $(BASE_FOLDER)/pkg/constants/version.go
    @touch $(BASE_FOLDER)/vendor/packages_windows.txt
    @touch $(BASE_FOLDER)/vendor/packages_linux.txt
    @echo "# README BASE " >> $(BASE_FOLDER)/Readme.md







check-env:
    @mkdir -p $(BUILD_FOLDER)/dist/linux/bin
    @mkdir -p $(BUILD_FOLDER)/dist/windows/bin
    @mkdir -p $(BUILD_FOLDER)/dist/arm/bin
    @mkdir -p $(BUILD_FOLDER)/dist/osx/bin
    cp -R config $(BUILD_FOLDER)/dist/linux/
    cp -R config $(BUILD_FOLDER)/dist/windows/
    cp -R config $(BUILD_FOLDER)/dist/arm/
    cp -R config $(BUILD_FOLDER)/dist/osx/
    cp -R extras $(BUILD_FOLDER)/dist/linux/
    cp -R assets $(BUILD_FOLDER)/dist/linux/


## Linting
lint:
    @echo "[lint] Running linter on codebase"
    @golint ./...


getdeps:
    ./getDeps.sh


versioning:
    ./version.sh ${VERSION} ${TIME}

build/weblayer-linux:
    cd cmd/WebServer && ${FLAGS_LINUX} go build -o ${BUILD_FOLDER}/dist/linux/bin/${BINARY_NAME_SERVER} .


run/dev:
    cd build/dist/linux && bin/${BINARY_NAME_SERVER} --config config/config.json

build/dev: check-env build/weblayer-linux run/dev

clean:
    rm -Rvf build/dist/

startover:
    @rm -fr $(BASE_FOLDER)/assets
    @rm -fr $(BASE_FOLDER)/build
    @rm -fr $(BASE_FOLDER)/cmd
    @rm -fr $(BASE_FOLDER)/config
    @rm -fr $(BASE_FOLDER)/extras
    @rm -fr $(BASE_FOLDER)/pkg
    @rm -fr $(BASE_FOLDER)/web
    @rm -fr $(BASE_FOLDER)/third-party
    @rm -fr $(BASE_FOLDER)/api
    @rm -fr $(BASE_FOLDER)/vendor
    @rm $(BASE_FOLDER)/Readme.md

So a few take away are:

One makefile is all you need to jump start your project, will create the appropriate folders, README and directory structur.

➜  go-web-starterpack git:(MakefileInit) make init
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/assets'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/build'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/cmd'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/config'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/extras'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/pkg'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/pkg/constants'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/web'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/third-party'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/api'
mkdir: created directory '/home/shellcheff/projects/src/github.com/epyphite/go-web-starterpack/vendor'
Creating Base Files
➜  go-web-starterpack git:(MakefileInit) ✗ ls -ltrh
total 48K
-rw-r--r-- 1 shellcheff shellcheff 2,4K nov  8 12:30 Makefile
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 build
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 assets
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 extras
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 config
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 cmd
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 web
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 third-party
drwxr-xr-x 3 shellcheff shellcheff 4,0K nov  8 12:45 pkg
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 api
drwxr-xr-x 2 shellcheff shellcheff 4,0K nov  8 12:45 vendor
-rw-r--r-- 1 shellcheff shellcheff   15 nov  8 12:45 Readme.md
➜  go-web-starterpack git:(MakefileInit) ✗

As you notice we created a few folders and a few files, lets try to explain a little bit.

build

This folder will be use as output of any compilation and building, we can then divide it into distributions, operating systems and flavor ( we get to that later )

assets

In this case assets will hold anything http/web related. Such as css, images, js, vue.js and templates.

extras

Anything "extra", such as tls and ssl certificates, tool configurations, or things needed to run. In our particular project this will be TLS certificates.

config

Configuration file in json for our app. We will get to this later, but lets just say it easy to have a config/config.json available.

cmd

From go standard "Main applications for this project." that means minimal code to run our application.

web

Here I will put all the code related to a webserver implementation, and any webapp. Usually I like to divide webserver.go with webapp.go. But we will get to that later.

third-party

Reserved for any plugin, auxiliary library, or later in this series protobuf libraries.

pkg

THE MAIN LIBS goes here, all the code for our app, implementations, classes, storage drivers etc...

api

Will hold any specifications, like swagger or protobuf (later)

vendor

Coming from python I rather prefer to have my package.txt file. Don't get me wrong vendoring in Go is mature, I just... don't like it enough. And so far for me has been easier and simpler to get a vendor folder with a few txt files listing my dependencies.

Again, this is my preferred way to order my code and projects, makes it easier for me to handle and specially to spin projects fast.

In my next posts I will show you how to use this structure to easily spin a web server, create a configuration files, build and package it :)

[ CODE PART ENDS HERE ]

[ RANT PART STARTS HERE ]

Today is Friday... So will not be a rant directly associated with this post...

I wish you all happy coding, and PM me for comments or fixes :)

Distributed Rant Series Part 1

Juan Pablo Ramirez — Thu, 07 Nov 2019 20:14:21 +0000

In this rant series I would like to discuss a little bit about the new era of software development, users trends, security an son on.

While that verbose rant is going I will introduce a series of guides to design , prepare,code and deploy micro applications using GO.

First Question ... why GO?

Well... Go is simple, very simple to learn, is cross-platform (or at least easier than many) is super fast compare with other cross-platform frameworks, the community is growing exponentially fast.

Building with Go in my opinion has major takeaways.
Simplicity of deployment and roll outs.
Security, as long as you care about the communication layer.
Performance, as many task can be delegated to the client in a more efficient way.

For the sake of order, we will be following , generally this index:

Structuring Go Projects ( What I found to be more easy to follow)
Golang Web with net/http and Gorilla Mux
Templating with GO
Serving Vue.js application (Why Vuejs... well...check later)
Embedding databases ( bbolt with a few disclaimers and twists)
gRPC , protobuf
Client Server Communication
Security, TLS and mtls.
Publishing on the internet, securing applications,

So we have a quite to cover.

Let's narrow a bit the scope for those looking for specifics...

We are going to talk about distributed systems while we code server-less applications, that can serve as both client and server, that can run locally to the user and communicate remotely.

[RANT START HERE]

Why? Well we are in the verge of a big change, or at least we are witnessing a technological revolution.

New generations grow with tech in their cribs. They consume information faster and at a younger age. So it's only reasonable that we need to start considering this to build the next layers of society.

We witness the danger of monopoly in a digitized world. Major vendors control influences, tells us what to do, what to wear, and people are seeking more resilient and dependable services.

New generations are seeing this, and freedom, or what we know as freedom has become an emblem.

But freedom without responsibility, without accountability, is not freedom. True freedom is both, being accountable and own our decisions.

And in this verge is where us, the older generations, and the systems we lay and build, along with the speed and the new technology grown humans can have a chance for creating self sustain , resilient, mature software.

In this "extremely extensive" series will try to give my grain of sand, by teaching what I believe is the root for owning software.

[END OF RANT ]

Hope you enjoy it :)

Audio Processing Services

Juan Pablo Ramirez — Wed, 06 Nov 2019 14:28:53 +0000

For a while now I have been working on different stream processing projects, one of them is related to the audio industry, that for some obscure reason seems to be forgotten.

We hear daily advertisment, Games, E-Learning platforms, and instructional videos but we never think on how much work doing all that audio can cost.

Digging deeper into the pains of audio editing, I discover that at a bareminimun cutting big audio wav files is truly a pain.

For example one of the average work for an audio engineer is to get this huge file where a voice over records and cut silence between sentences, cut gazillions of files and number them to better organize the script or work.

This can be tedious.... and super frustrating, time consuming and many other good adjetives.

For that reason I started to create a microservice that will take a .wav file, use python3 ML, detect silence, cut it, compress it and return it back to the user.

The main idea to cut editing time, but also make use of Golang net/http with tls support (generated with letsecrypt), Python 3 tornado API, templating and embedded database (using bbolt).

You can see a working (super alpha) demo here

https://audio.epyphite.com/

Of course in the case you take a look... be kind, it's in a super alpha state.

Can only accept wav files and should be super fast returning the zip file wiht the chooped audio.

If anybody is interested please PM me, or ask me, will be happy to help.

Thanks!

ULID in GO Lang

Juan Pablo Ramirez — Mon, 14 Oct 2019 02:32:31 +0000

I have digging around for some time in audit-able ready log systems, as a core part of a decentralized environments and the core for trust between hosts. In epyphite (soon to be release) trust and transparency are important. For that reason we needed to come out with a way to generate a Unique Identifier that can be concurrent and millisecond friend. Specially when it comes to decentralized and high reliant systems.

Its very important to be able to have a unique identifier, timestamped and ordered that can point and trace fast to different events.

I came across with this github repo covering ULID that suited in that moment our needs.

https://github.com/oklog/ulid

But we had a problem...
In our environment millisecond are valuable, and by "secure by design" principle, we could not have events sharing the same timestamp ID. Every change is important, and the ability to search fast in between is at the most importance

Searching around I found this great article

https://barkeywolf.consulting/posts/badger-event-store/

So I decided to code those specs into an small library to share on Github.

HERE

Again.. all of this are WIP and they need to be expanded for use and documentation, but ...you will get the idea.

The goal is to be able to generate sub buckets for a millisecond in a consistent, ordered , to have different sub id's that we can use. We do have limitations.. but for the specific use case that we were having this implementation was enough.

Any comments, feel free.
Happy Coding.

News API in Golang

Juan Pablo Ramirez — Sun, 13 Oct 2019 23:39:25 +0000

This was a fast job coding weekend. While putting up a few series of news aggregation services for a new product I cross path with https://newsapi.org, is pretty slick, straight forward and above all free for developers.

They have support for several programming languages but currently I'm a heavy gopher, and it turns out that they don't have support for it.

So... I coded a fast API interface to integrate it into my projects (or use i as a small microservice) It's of course a WIP and I will continue to add features to it.

Github REPO
Hope you like it !