Forem: Mario

DATACENTER IN A SUITCASE - A REAL SMALL EDGE CASE

Mario — Mon, 15 Jan 2024 14:32:21 +0000

Usually when we talk about datacenters, we get the impression it’s going to be big. There are more and more use cases emerging where your datacenter either needs to be portable or is limited in size and power supply. Here we will take a look on how to create a portable datacenter.

THE UNUSUAL CLOUD-NATIVE DEMANDS

In the cloud-native space, we’re shifting more components into unconventional spots — Edge. This means locations that might have limited energy reserve or are prone to outages or are limited in space.. We need systems that are self-sufficient, energy-efficient, and portable.

Consider the idea: building everything inside a suitcase. We’re talking about leveraging ARM processors, not your typical server processors. These ARM units are powerful and energy-efficient—perfect for a low-energy, highly portable setup.

WHY ARM PROCESSORS?

ARM processors, short for Advanced RISC Machines, power nearly every smartphone. They operate on Reduced Instruction Set Computing (RISC) architecture, executing instructions predictably and efficiently.

The licensing model for ARM is open, enabling various companies to develop specialized processors for different use cases. With advancements to 64-bit technology, ARM processors become more feasible in the cloud-native realm.

THE ARM ADVANTAGE

Energy efficiency is a big win with ARM processors. They consume far less energy compared to traditional x86 architectures, making them ideal for portable data centers.

Moreover, ARM processors offer hardware customization, allowing tailored designs for specific needs, enhancing efficiency further. This can also be adopted by hardware vendors due to the flexible licensing.

CHALLENGES AND SOLUTIONS

However, there are challenges. Compatibility with software not optimized for ARM remains a hurdle. Performance per core might lag compared to x86 processors, and standardization issues persist due to the diversity of ARM options.

The compatibility problem will solve over time as more and more software is getting adopted also on ARM architecture.

THE PRACTICAL SETUP

I recently acquired a mini-ITX board from TuringPi, equipped to house ARM computing modules. Sure, there are other boards, but this one had networking and storage features baked in, making it a convenient choice. Plus, the price was reasonable, around $200.

Building this setup wasn’t without its challenges. From USB compatibility issues to missing packages on Ubuntu, each step required troubleshooting and adaptation.

VIRTUALIZATION: THE MISSING LINK

Virtualization plays a crucial role in this setup. While containers run most software, there are cases where VMs are necessary, especially for non-containerized applications. Virtualization provides a layer of control and safety, ensuring system stability.

With the help of KubeVirt we can leverage the OpenSource way to virtualize our ARM based hardware nodes to create VM’s to give the possibility of facilitating a smoother transition to a complete ARM infrastructure.

FUTURE STEPS AND USE CASES

Moving forward, the plan is to explore multi cluster deployment, optimize power sources with solar energy, and upgrade hardware for better performance. Also, an automated installation with the help of PXEBoot and Tinkerbell to Set up the whole infrastructure is something to take into consideration to automate and streamline the management of multiple “Datacenters”

The last question to be answered would be what is the use-case? As already mentioned, we are here focusing on Edge scenarios. Here the possibilities are also quite diverse. Beginning with just a small cluster that needs to run inside a store or a small part of a manufacturing line and shop floor to some true remote cases.

Some of those could range from expedition teams that need on site compute power to work at the location on research up to military use cases too. Those could be having a portable Intel capable station, that can be carried by a single soldier and adapt to location changes, but have the power to support with intel gathering, data analysis and general compute power.

CONCLUSION: A WORK IN PROGRESS

This unconventional approach is still a work in progress. While it’s not production-ready yet, it’s an intriguing concept with promising potential for specific environments and scenarios.

And that’s the gist! While it might sound like a crazy idea at first, the notion of a suitcase-sized data center opens doors to exciting possibilities in the cloud-native landscape.

NodeJS Continuous Deployment in Google Cloud with Kubernetes & Container Builder

Mario — Sun, 17 Mar 2019 23:11:59 +0000

So you want your app to be deployed to your Kubernetes Cluster without caring about any manual step?
I got you covered, it's super simple to create a Continuous Deployment Pipeline with Google Cloud.
For the sake of understanding I choose an NodeJS Express Application, but it also works with react or PHP or any other application layer.

Let's get started:

Because IAM admin

First we need to give the container builder the rights to access our Kubernetes API. Remember this does not give access to a certain cluster. It just allows the cloudbuilder service account to access our Kubernetes Cluster. So jump to the IAM settings page and look for the cloudbuild service account. If it does not exist you might have to enable the cloudbuild API

It should look like this

We need to add the rights to access the Kubernetes API of our clusters so klick on the pen and look for the following.

Prepare the application

I won't go into details on how to setup an express application and introduce testing to it.
I created a repository with the sample application, that we can use

mfahlandt / gcp-continuous-deployment-node-demo

This is an example project to show how you can easily create a continuous deployment to google cloud

NodeJS Continuous Deployment done with Container Builder and Kubernetes Engine

To find all the details on how to use this repository please look refere to the corresponding block post on dev.to

View on GitHub

To give you an overview, we have a basic express app with 2 backend routes to retrieve users or and user by id.
Also we have a test folder that have tests for the two routes in it. These tests are written with the help of chai and mocha.
If you download the repository you can do the following to see if the tests are working.


npm install
npm test

Before the app can run we need the service and the deployment in the Kubernetes Cluster. So let's quickly create a service and a deployment. All of the files you also can find in the repository.


apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: server-production
  labels:
    app: YOUR-PROJECT-ID
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: server
    spec:
      containers:
        - name: server
          image: gcr.io/PROJECT_ID/REPOSITORY:master
          imagePullPolicy: Always
          ports:
            - containerPort: 3000
          env:
            - name: NODE_ENV
              value: "production"

The only important part here is that you change the project id and the repository to the path that the repository will have.

After this we only need a service to expose our app to the internet. So quickly apply the service.


kind: Service
apiVersion: v1
metadata:
  name:  server
spec:
  selector:
    app:  server
  ports:
    - name:  server
      protocol: TCP
      port: 80
      targetPort: 3000
  type: LoadBalancer

Ready to deploy

Now we need to go to the most important part of the whole Setup. The cloudbuild.yaml. There we will define everything for our continuous deployment steps.

The first amazing part will be, that it is possible to put all of the important data in environment variables defined in the build, so you can use the cloud build for different setups.

First we install all of the node dependencies and run the test.


  - name: 'gcr.io/cloud-builders/npm'
    args: ['install']
  - name: 'gcr.io/cloud-builders/npm'
    args: ['run', 'test']

After this we build a docker image with all the repositories files inside and a proper defined environment, so you can easily do a staging deployment as well, or even branch deployment. And we push it to the google image repository.


  - name: 'gcr.io/cloud-builders/docker'
    args:
      - build
      - '--build-arg'
      - 'buildtime_variable=$_NODE_ENV'
      - '-t'
      - gcr.io/$PROJECT_ID/$REPO_NAME:$BUILD_ID
      - '.'
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME:$BUILD_ID']

Also important to see, we tag the image with the unique build id to make use of the apply ability of kubernetes, so the image is actually changed.


  - name: 'gcr.io/cloud-builders/kubectl'
    args:
      - set
      - image
      - deployment
      - $_DEPLOYMENT
      - $_DEPLOYMENT=gcr.io/$PROJECT_ID/$REPO_NAME:$BUILD_ID
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=$_CLUSTER_ZONE'
      - 'CLOUDSDK_CONTAINER_CLUSTER=$_CLUSTER_NAME'

And finally we set the image in the kubernetes cluster. BAM! Commit hook, automated testing, if successful automated deployment, no downtime.

Now we open the container builder trigger and choose where our code is located.

In the last trigger step we now can add the custom variables. This is the first point where we actually define the cluster. So everything is aggregated at one place and ready to go.

Now we just need to commit to the master and the trigger is started.

YIHA now we have continuous deployment, without setting up any extra services like jenkins, ant or chef. Pretty amazing

I'm thinking of creating a tutorials series from zero to hero in cloud are you interested? drop me a comment!

Scaling properly a stateful app like Wordpress with Kubernetes Engine and Cloud SQL in Google Cloud

Mario — Sun, 10 Mar 2019 18:15:43 +0000

There are a lot of examples in the web that show you how you can run Wordpress in Kubernetes. The main issue with this Examples is: Only one Pod running with wordpress and you cannot really scale it.

So i faced the issue, that i needed a highly scalable setup for wordpress and here is what i came up with.

Why is it so hard to scale stateful apps?

These apps write to the disc directly and most of the time you cannot prevent it. This is often the case in PHP based applications that use some kind of plugin system. So files cannot be stored in some kind of bucket but have to be in the filesystem of the application.

Now you say something like but there is a Stateless plugin like https://de.wordpress.org/plugins/wp-stateless/ that writes to cloud buckets. Yes this is true, but it does not store the plugins there or the files, that some plugins might directly write in there folder (sad that this happens but true)

What to do?

We need a couple of things, we want a scalable database, we need some kind of shared filebase for our application and the application itself.

For the sake of shortening we will just use a predefined Wordpress Docker Image, although you should always try to create additions to these Dockerfiles, that fits your own needs. Use them as a base but extend them to your needs.

So we need a shared disc and here we encounter our first problem. We need a ReadWriteMany volume in our Kubernetes Cluster and the problems start. The Cloud providers do not have this.
If you check the Kubernetes documentation
you will see neither GCEPersistantDsik nor AzureDisk nor AWSElasticBlockStore support what we need.
There are options like CloudFileStore in Goolge Cloud or AzureFile but they are way to expensive and to big for our case (We do not need 1TB to store our Wordpress thank you)

NFS to the rescue

But when we look at the list we see the saviour: NFS to the rescue. Let's create the only option we have a ReadWriteOnce Storage connected to our NFS. So we need a Storage Class ideally shared between regions:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: regionalpd-storageclass
provisioner: kubernetes.io/gce-pd
parameters:
 type: pd-standard
 replication-type: regional-pd
 zones: europe-west3-b, europe-west3-c

And we need to create the Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: nfs
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: ""
  volumeName: nfs

Now Let’s create our NFS

apiVersion: v1
kind: Service
metadata:
 name: nfs-server
spec:
 clusterIP: 10.3.240.20
 ports:
   - name: nfs
     port: 2049
   - name: mountd
     port: 20048
   - name: rpcbind
     port: 111
 selector:
   role: nfs-server

Now we add the NFS itself. The good thing here, we can use a predefined service

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: nfs-server
spec:
 replicas: 1
 selector:
   matchLabels:
     role: nfs-server
 template:
   metadata:
     labels:
       role: nfs-server
   spec:
     containers:
       - name: nfs-server
         image: gcr.io/google_containers/volume-nfs:0.8
         ports:
           - name: nfs
             containerPort: 2049
           - name: mountd
             containerPort: 20048
           - name: rpcbind
             containerPort: 111
         securityContext:
           privileged: true
         volumeMounts:
           - mountPath: /exports
             name: nfs
     volumes:
       - name: nfs
         gcePersistentDisk:
           pdName: nfs
           fsType: ext4

CloudSQL so secure so much beauty

Alright so we have a running NFS for our static data. So next big step connect Cloud SQL. So let’s say you already have Setup an Cloud SQL Mysql. How do you connect your pods to it?

We use the SQL proxy for it that comes as a sidecar to our container. The good thing about this is, our MySQL is not exposed and we can use localhost. Amazing isn’t it?

First you have to activate the Cloud SQL Admin API

And you need to create a service account that have actual access to cloud SQL.

Here we create a new role that have the rights for Cloud SQL > Cloud SQL-Client

Download the created private Key this one we need to access the SQL instance.

Now create a database user if you have not already done so

gcloud sql users create [DBUSER] --host=% --instance=[INSTANCE_NAME] --password=[PASSWORD]

And we need the name of the instance, easy:

gcloud sql instances describe [INSTANCE_NAME]

Or in the webinterface you find it here:

Now we save the credentials to our Kubernetes instance:

kubectl create secret generic cloudsql-instance-credentials \
    --from-file=credentials.json=[PROXY_KEY_FILE_PATH]
kubectl create secret generic cloudsql-db-credentials \
    --from-literal=username=[DBUSER] --from-literal=password=[PASSWORD]

So we are ready to Setup our Wordpress aren’t we?

Let’s create the service as a first step:

apiVersion: v1
kind: Service
metadata:
 name: wlp-service
 labels:
   app: wlp-service
spec:
 type: LoadBalancer
 sessionAffinity: ClientIP
 ports:
   - port: 443
     targetPort: 443
     name: https
   - port: 80
     targetPort: 80
     name: http
 selector:
   app: wordpress

Alright now we have the Service Up and running only missing is the pod itself.
Let's Split it up a bit so i can explain

apiVersion: apps/v1
kind: Deployment
metadata:
 name: wordpress
 labels:
   app: wordpress
spec:
 replicas: 2
 strategy:
   type: RollingUpdate
 selector:
   matchLabels:
     app: wordpress
 template:
   metadata:
     labels:
       app: wordpress
   spec:
     containers:
       - name: wordpress
         image: wordpress:7.3-apache
         imagePullPolicy: Always
         env:
           - name: DB_USER
             valueFrom:
               secretKeyRef:
                 name: "cloudsql-db-credentials"
                 key: username
           - name: DB_PASSWORD
             valueFrom:
               secretKeyRef:
                 name: "cloudsql-db-credentials"
                 key: password
         ports:
           - containerPort: 80
             name: wordpress
           - containerPort: 443
             name: ssl

This would be enough to run wordpress, but without the database or the persistent nfs. One by one let's add the cloud sql proxy:

       - name: cloudsql-proxy
         image: gcr.io/cloudsql-docker/gce-proxy:1.11
         command: ["/cloud_sql_proxy",
                   "-instances=[YOUR INSTANCESTRING THAT WE LOOKED UP]=tcp:3306",
                   "-credential_file=/secrets/cloudsql/credentials.json"]
         securityContext:
           runAsUser: 2  # non-root user
           allowPrivilegeEscalation: false
         volumeMounts:
           - name: cloudsql-instance-credentials
             mountPath: /secrets/cloudsql
             readOnly: true
     volumes:
       - name: cloudsql-instance-credentials
         secret:
           secretName: cloudsql-instance-credentials

Cool now we can access our Cloud SQL with localhost :) It basically adds a second container to your pod that proxys everything comming to 3306 to our cloud SQL instance without exposing the traffic to the public net.

And now we want to mount our wp-content directory to the NFS

volumeMounts:
           - name: my-pvc-nfs
             mountPath: "/var/www/html/wp-content"
volumes:
        - name:  my-pvc-nfs
        nfs:
            server: 10.3.240.20
            path: "/"

Now you would start saying, but Mario why the hack do you put in a fixed IP for the NFS. There is a reason. This is the only case that i know where the internal dns is not working properly.

And that's it now we can scale our pods by creating and hpa

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
 name: wordpress
 namespace: default
spec:
 maxReplicas: 10
 metrics:
   - resource:
       name: cpu
       targetAverageUtilization: 50
     type: Resource
 minReplicas: 3
 scaleTargetRef:
   apiVersion: extensions/v1beta1
   kind: Deployment
   name: wordpress

All our wp-content files go to the nfs and is shared between the instances. Yes you are correct the NFS is now our single point of failure but an NFS is way more stable than having just one machine running. If you use Caching like redis or increase the fpm cache you can further reduce the load time.

Cool isn’t it?

Are you interested in basic Kubernetes / Cloud walkthroughs? Just let me know

Move PostgreSQL AWS RDS to Google Cloud SQL

Mario — Sun, 03 Mar 2019 18:25:05 +0000

We have the issue, that we have to move a large postgreSQL database away from Amazon's AWS to Google's GCP.

The Problems where:

Large Database: 160GB+ We only had the Snapshots of AWS

Get a snapshot out of RDS into Storage

To do this we created a new compute engine and connected to it via ssh. We want to get the dump file direct to a new bucket in storage. So we have to enable the bucket as new volume of the machine:

gcloud init

You can either Login or use the Service Account, but keep in mind that the Service Account needs the rights to create a bucket.

gsutil mb gs://my-new-bucket/

Now we have to mount the Bucket to our machine. For this wie use Cloud Storage FUSE, to install it we need the following steps:

export GCSFUSE_REPO=gcsfuse-lsb_release -c -s
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install gcsfuse

And now we can finally mount it

gcsfuse db /mnt/gcs-bucket

So we have a place to store the dump, what next? We have to install the same PostgresQL Version on the Machine as the remote Server is, to get a working pg_dump

echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main 9.5" | sudo tee /etc/apt/sources.list.d/postgresql.list
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
apt-get install postgresql-9.5
sudo apt-get install postgresql-9.5

Now we finally can do the dump:

pg_dump -h yourRDS.rds.amazonaws.com -p 5432 -F c -O -U postgres DATABASE > /mnt/gcs-bucket/db.dump

Depending on how large your Database is this will take quite a while. Whats next, create your SQL Instance on GCP. There is an import function for SQL Files out of the Bucket but sadly not for dumps, so we have to do the restore the hard way.

pg_restore -h YourNewSQLInstanceIP -n public -U postgres-user -d DATABASE -1 /mnt/gcs-bucket/db.dump

This will even take longer, be sure to whitelist the IP of the Compute Engine, that it can have access to the SQL Instance.

I did everything like you told me but i receive weird errors

Something like this?

pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 4198; 0 0 ACL children_of(integer) 
postgrespg_restore: [archiver (db)] could not execute query: ERROR: role "user" does not exist Command was: REVOKE ALL ON FUNCTION children_of(root_id integer) FROM PUBLIC;

Easy to answer, you have missing users on your new Database that are referenced by in the Dump.

How to avoid this?

Easy to answer, create the users. Sadly you can't export them due some regulations RDS that makes it impossible to do a pg_dumpall -g (Only User and Roles)

pg_dumpall -h yourRDS.cd8cncmdv7f0.eu-central-1.rds.amazonaws.com -g  -p 5432  -U postgres > /mnt/gcs-bucket/db_roles.dump

This do not work and you will receive the error

pg_dumpall: query failed: ERROR:  permission denied for relation pg_authid
pg_dumpall: query was: SELECT oid, rolname, rolsuper, rolinherit, rolcreaterole, rolcreatedb, rolcanlogin, rolconnlimit, rolpassword, rolvaliduntil, rolreplication, rolbypassrls, pg_catalog.shobj_description(oid, 'pg_authid') as rolcomment, rolname = current_user AS is_current_user FROM pg_authid ORDER BY 2

Because AWS RDS don't do the query as the superuser and so you cannot export it. However if you create them manually it will work fine

till next time

We need you! Join tech communities!

Mario — Thu, 17 Jan 2019 15:04:51 +0000

All of us benefit in some way of communities. One of the obvious examples is this very plattform here. Without it i would probably not blog. Thanks for it at this point to the dev.to team!

Since 2 and a half year i am the main organizer of the Google Developer Group for cloud related topics in munich and i must admit it’s not easy. I mean i like the event themselves but organizing it in your spare time is challenging. I even run in the issue that it was impossible for me for more than 6 months to organize a meetup. There had been multiple reasons.
We organized a 2 day conference that was completely organized by a joined force of organizers of different meetups / communities called dachfest. It had been a huge success, i will write about it in another blog post. But it took all my spare time that i have from organizing meetups. It would have helped if i had more help organizing the meeetup group with someone else. Here comes the main point of this article.

You probably have been to meetups already or to un-conferences or or or.
The people that are organizing it need your help badly. How to help? And what is in for you? The last question should not be asked, but i know a lot of people do so we will try to answer it as well.

First things first: Where to find a community

There are a lot of possibilities, most of them are based on the area you are in. In Europe / North America the plattform Meetup is used to organize communities to various topics. Also i know of Google Groups that is used to organize or twitter.

Give a talk

It gives me hard times to find speakers (an the meetup have 680 members…). Half of the talks on each meetup is held by me. That brings more overhead. You need to prepare talks for each meetup that takes time, to test research and build the presentation. Also it is kinda boring always seeing myself on stage as well as i’m not the all knowledge person, so basically it narrows the field of talks.
Everyone can give a talk, don’t be shy. No one will kill you or blame you if something does not go as planned. People are happy that it is not themselves up there.
Most meetups do something like lightning talks, short 5-10 Minute talks, try to start there or team up with a colleague to give a talk. We are all happy to have you and we will surly spend some time to go over your talk with you.
what’s in for me: Training! You improve your skill to speak in front of a crowd. This will help you when you talk to customers or with your fellow company people. Also you get references for talks that some conferences want to see before they consider your talks.

Give location and catering

Another important topic is, where to meet. I don’t want to be fixed to one location / company. Communities should be independent and not reliant to company or govs. If you can provide space and some snacks (based on region also beer, obviously we are in bavaria) it is helpful. The easiest way is to contact the organizers directly, most of them are super happy to have you as host for the next meetup, but please be a little flexible with dates and be open to have someone at the meetup present. Because the will most likely happening in the evening or on weekends.
what’s in for me: You have a lot of tech folk at your office, most meetups offer like 5 min of advertising and recruiting talk. You can talk to a cpl of people that have a same problem stack / tech stack

Join a community as organizer / create a community

The most time consuming option for shure. A lot of communities are looking for co-organizers ( Me for example). It’s hard to find reliable people that can give talks organize new meetups do some advertisement for the meetup, find speakers and so on.
Another option here is: There is no meetup / group for my topic. Easy create one! People will join you fast. I created the meetup and after two weeks ~ 100 People joined and i had about 30 people on our first meetup.
There is always a need for you. Just do it!
what’s in for me: Time management and organisation skills!

Ok that all sound fine, but i feel uncomfortable around people

No problem we got you covered: Get involved in online communities like here. Take part in discussions, share your knowledge. Create tutorials / video tutorials. Or one of the coolest things: Contribute to open source.
Our whole world depend on open source projects, help us with a little bit of your time.

Thanks to all the people that are already organizing community work and are contributing to all of the above points: You Rock!

One year home office, things I have learned

Mario — Mon, 07 Jan 2019 10:48:59 +0000

To be honest it's just a little bit more than one year. Actually one year and three months.

Ther first and most important rule above all:

Wear pants, good pants that you would wear in office, every working day.

Something i learned pretty fast. When you get up and you get well dressed for the day you can draw a clear line between work and spare time. So you have a clean start in the day and in the evening you can look forward to get of the "work" clothes on the evening.
Also my wife complained about me looking less and less appealing, so this is a must, you do not want to look like the troll under the bridge.
Positive thing, you are always prepared for surprising video chats coming up.

Setting basic work times, but don't get up and sit on your desk

For myself it was important to create some kind of working day. So i set myself fixed times to start work. I start working between 8.30am and 9.30am. This is also easy for all people that want to reach me, to know when i am available. Important is that there is another BUT. I never get up and sit directly down at my desk, basically i created something like a way to work. So when getting up the first things are, preparing coffee and taking the dog for a walk, afterwards i have breakfast and then start working. This also helps you to create some routine and don't fall in the trap with oversleeping.

Have brakes, don't eat at your desk!

This is probably one of the biggest issues, not scheduling breaks or eat at your working desk. Also an error that i made for a long time and sometimes still do. Eating at your desk or don't make proper brakes. I try to get up from my desk at least every 2 hours for some minutes. Stretch, get something to drink and so on. If you have a portable device, change places. If you want to sit on your couch for some time, do it! But please do breakes and stop working, to refresh your mind. Because of the dog, i have to go outside for at least once in the working hours. If you don't have something similar find an activity to get you away from the working station. It helps getting new energy.

Invest in your equipment

We sit the whole day, so invest in a good Setup that fits you. Buy a proper chair, your back will thank it to you. I underestimated this a long time, now i have to see the back doc every once and awhile. Don't make the same mistake. Ask your employer for support. The company saves a lot of money not having to maintain an office spot for you so it is more than fair if the pay for a good chair and maybe a desk. Also get a keyboard that feels right for you and a proper webcam and microphone / headset, your clients and colleagues will thank you.

Feel so lonely

A big challenge that will not be easy or achievable by all is the separation and missing of human interaction. You won't see a lot of people. For me this is just what i have needed, i like working alone in my house more than working in any kind of office. But you will have less contact with people. I do sports and meet friends beside work which is enough for me. For a lot of people this is not. Honestly i cannot help you here. Some colleagues of mine are in skype the whole day, to have people to talk to.

Oh sweet sweet distraction

Another hard point, distraction. The kitchen could be cleaned, the garden needs some fixing and procrastinate on the interweb ;)
Being home adds more distraction than being in the office. Keep in mind also your family can be distraction. When my wife is home i have the feeling i get less done than the times she is not. How to deal with this. Easiest way for internet distraction get yourself a website blocker that have timing windows so you won't end up on facebook or youtube. For everything else get back to the point where we said define work hours and make it clear to your family that this time is work time.

STOP WORKING!

Also the other side you maybe end up and can't stop working. So set yourself a fixed time each day when you will stop working. I think this is a bigger issue than distraction at least for me. mostly i stop working between 6pm and 7pm and i do not read any more mails after this and i stopped working on weekends (except for high time projects, but that's another story).
You won’t do yourself any good if you do over hours each and every day.

Stay fit

Maybe you are missing out a walk or the fitnessstudio after work. Don’t! Search yourself a sport activity to stay fit, because you will even get out less than in an office job. Good thing is you can go at afternoon to the fitness studio or climbing house. You define your work hours yourself. That’s a huge advantage, use it! I stopped it and it was bad for me, now i go climbing at least 2 times a week in a complete empty hall.

Last words

This is my perspective, so it does not necessarily apply to you. Are you working from home and have other tips? Let me know! Do you think starting working from home drop a comment

Holidays Side Project Time, or maybe NOT?

Mario — Tue, 25 Dec 2018 12:40:51 +0000

On 247 days i wrote code this year and will probably do it on two days more to come. On my twitter timeline and reading in a lot of blog posts i read, holidays are side project time. 4-5 years ago i would have agreed on this but now it's different.

My parents moved from a it's a daytrip distance to it require some organisation and at least 4 days to get there distance two years ago. So we see them only once a year. This makes me feel bad when i remember the days in the past where i wasted my time at my parents with side projects instead of spending some time together. Yes you read correctly i wrote wasted.
Each of us write code in one form or another most of the year and it should be fun and it should stay fun and here is the point: At one point you will not have the joy anymore, at one point it will get a necessity. For me this would be the worst moment in my life, i have to work in a job that brings me no joy anymore.
I can hear the outcry ohh you are not a passionate software developer, but honestly i don't see it. The most import job of all of us is to look out after ourselves and our loved ones. Especially these day are made for it. Most of your loved ones have time and you can get together play board games, go for a walk, watch movies together and most important give your brain a rest from doing the stuff it hast to do all year long.
Another argument i often hear: side projects are the only way i can try new things. This is an argument that makes me mad when i hear it. There is a simple reason for this. As an employer the knowledge of my employees is the most valued possession i have. So if any developer now tells me holidays are the only days in the year to move forward my technical knowledge it means that the developer does not get the chance and the time by his employer to learn new stuff. Ask your employer to get time to try new stuff. If the answer is yes you know your knowledge is valued, if the answer is no, maybe time to look for a new job.

Don't feel bad if you don't work on a side projects on the holidays, even if you don't spend quality time with loved ones. Even if you have spare time and just want to relax by reading a NON tech book, playing games or even the heck, just watch a movie or a series that you missed in all the stress this year.

Step back take a break, you won't miss out anything. Bring your mindset to a state where it says the heck i hope i can get back to work to do coding again.

Now what is your way, are you just now working on a side project for the holidays? Or did you give yourself a day of? Anyways, happy holidays

Fast Knowledge: Kubernetes port forwarding

Mario — Thu, 29 Nov 2018 11:45:50 +0000

When i finally discovered the option in Kubernetes to forward a port of a pod to localhost, it changed the way i started working with my cluster.

I often have my MongoDB, MYSQL or Elasticsearch inside of a Kubernetes cluster, running on pods. Naturally it has no public access point or port forwarding to the outside of the cluster. But sometimes you want to use a Client to access your Database or you want to use your cli locally to find errors.

The solution is

kubectl port-forward localport:podport

This allows you to easily forward your mongoDB Port to localhost for example

kubectl port-forward pods/mongo-0 27017:27017

or

kubectl port-forward deployment/mongo-0 27017:27017

Then you can use use your MongoDB Client (e.g Robo 3T) to connect to localhost:27017 and you are connected to your remote NOSQL Database.

For more information look here

Remove and modify nested documents in MongoDB

Mario — Mon, 01 Oct 2018 13:05:54 +0000

I have a rather complex Document Structure to fulfill the approach of have all the data available when you execute one query. However there is the big issue of modifying it.

So lets have a look at our Object that we want to modify


{
    "_id" : ObjectId("5b0bf696cb5dd80010bea0a3"),
    "domains" : [ 
        {
            "_id" : ObjectId("5b0bf696cb5dd80010bea0a4"),
            "DKIMRecordName" : "mailjet._domainkey.example.de.",
            "DKIMRecordValue" : "k=rsa; p=000000000000000000000000",
            "DKIMStatus" : "Not Checked",
            "Domain" : "example.de",
            "IsCheckInProgress" : false,
            "SPFRecordValue" : "v=spf1 include:spf.mailjet.com ?all",
            "SPFStatus" : "Not Checked",
            "emails" : [ 
                {
                    "_id" : ObjectId("5b0bf696cb5dd80010bea0a5"),
                    "CreatedAt" : ISODate("2018-05-28T12:31:18.000Z"),
                    "DNSID" : "2837371580",
                    "Email" : "no-reply@example.de",
                    "EmailType" : "unknown",
                    "Filename" : "",
                    "ID" : "216556",
                    "IsDefaultSender" : false,
                    "Name" : "",
                    "Status" : "Inactive"
                },
                {
                    "_id" : ObjectId("5b0bf696cb5dd45410bea0a5"),
                    "CreatedAt" : ISODate("2018-05-28T12:31:18.000Z"),
                    "DNSID" : "2837371580",
                    "Email" : "newsletter@example.de",
                    "EmailType" : "unknown",
                    "Filename" : "",
                    "ID" : "216556",
                    "IsDefaultSender" : false,
                    "Name" : "",
                    "Status" : "Inactive"
                }
            ]
        }
}

So if you want to modify one of them we will use the following

db.yourCollection.update({'domains.emails.Email': 'no-reply@example.de'},
{ $set: {'email.$.emails':  
    {
                    "_id" : ObjectId("5b0bf696cb5dd80010bea0a5"),
                    "CreatedAt" : ISODate("2018-05-28T12:31:18.000Z"),
                    "DNSID" : "2837371580",
                    "Email" : "no-reply@example.de",
                    "EmailType" : "unknown",
                    "Filename" : "",
                    "ID" : "216556",
                    "IsDefaultSender" : false,
                    "Name" : "",
                    "Status" : "Active"
                }

  },
{multi: true})

Why do we have to put in the whole Object? Sadly we can not use a second positional $ operator. So we have to provide the full nested Document. Additional it MUST be included as part of the Query document, otherwise it would not recognize it. This would throw the Error "The positional operator did not find the match needed from the query."

Also what we can do is a pull operation:

db.yourCollection.update({'email.emails.Email': 'no-reply@example.de'},
{ $pull: {'email.$.emails':  {'Email': 'no-reply@example.de'}}  },
{multi: true})

This will remove the Child Document from the nested Array.

Be aware, the multi: true will not work and you have to run the query multiple times.

How to move Google Cloud images to other accounts

Mario — Thu, 20 Sep 2018 13:11:36 +0000

I handle multiple accounts for my customers and i have some images that i want to use on some of those accounts.

You now have different options on how to achieve this. The best one would always have a docker file for your images and use this as a baseline, we will look at this option in another article soon.

The easiest option however is, if we already have an image based on one machine to copy it to the other account.

to create images there are several ways to do it, the easiest way is to create it in you CloudConsole.

Compute Engine => Images => [+] Create Image

However if you like the console more do it by the command:

gcloud compute images create IMAGE_NAME --source-disk=SOURCE_DISK

Now that we have an image we want to export it to a cloud storage bucket

gcloud compute images export --destination-uri gs://bucket-name/imagename.tar.gz --image imagename

After this we will find the image as tar.gz file in out Cloud Storage. Now we have to make it available for other users.
To keep it simple we just make it public available.

The trick is here, add a User with Name allUsers and give it Read access.
Now it's freely downloadable.

Now we switch to the Account where we need the image and import it


gcloud compute images create imagename     --source-uri gs://bucket-name/imagename.tar.gz

That's it.

Please tell me if you want to see Video Tutorials for Google Cloud

Cheers

Copy Files from and to Kubernetes pods and Docker container

Mario — Mon, 17 Sep 2018 16:00:35 +0000

I often want to have some database dumps from my mongo also in my local Setup. To achieve this is pretty easy.
You can either copy files or whole folders.
What we need is the kubectl command line tool.

Copy Files from a pod to your machine

kubectl cp {{namespace}}/{{podname}}:path/to/directory /local/path
eg:
kubectl cp mongo-0:/dump /local/dump

Copy Files to a pod

kubectl cp /local/path namespace/podname:path/to/directory 
eg:
kubectl cp /local/dump mongo-0:/dump

The amazing hint on this is, this works as well with docker.
Just change kubectl to docker and it will work as well.

Copy Files from a docker container to your machine

docker cp containerID:/path/to/directory /local/path
eg:
docker cp mongo-0:/dump /local/dump

Copy Files to a docker container

docker cp /local/path containerID:path/to/directory
eg:
docker cp /local/dump mongo-0:dump

Why i write this obvious post? I'm tiered looking up the correct syntax now and then and just looking at my blog post speed things up.