Forem: oluseyeo

Securing Kubernetes: Adding a new hostname or IP address to Kubernetes API Server

oluseyeo — Thu, 30 Nov 2023 00:37:21 +0000

$:# kubectl get ns
Unable to connect to the server: x509: certificate is valid ...
$:#

Introduction:

In the ever-evolving landscape of Kubernetes, securing connections to the API server is paramount. This article presents an approach on how to add a hostname or IP address to the TLS certificate used by the Kubernetes API server to allow continued secure access. Not assigning a static IP address to the control plane from get go, or using a new hostname or different hostname to access the API Server may result in the error highlighted above when attempting to access the cluster.

A quick workaround is to use the --insecure-skip-tls-verify command line argument with kubectl. However, it's crucial to note that this method skips TLS verification, rendering the connection insecure, and thus improper for Production use.

In a short delve into the technical background, it is important for know that the Kubernetes API server employs digital certificates to encrypt incoming and outgoing traffic and authenticate connections. For those interested in understanding the underlying processes, attempting to connect to the API server via kubectl with a hostname or IP address not included in the certificate's Subject Alternative Names (SAN) will result in an error. This error signifies that the certificate is not valid for the specified IP address or hostname.

To resolve this issue, it is necessary to update the certificate to include all relevant IP addresses or hostnames in the list of SANs. This ensures that the API server recognizes and validates connections initiated with the specified addresses or hostnames.

Before You Begin:

Before embarking on any updates, it's imperative to consider the production environment. For nodes actively serving in production, pull them out of the active node pool or gracefully drain live traffic using kubectl drain. This ensures a smooth transition without disrupting critical workloads.

Updating Kubernetes API Server Certificates:

Before proceeding with the certificate update, it is essential to ensure that the IP address of the node is accurately reflected across all pertinent manifest files. These Kubernetes component manifests are centrally located in the directory /etc/kubernetes/manifests/ and undergo real-time updates. Consequently, any modifications made to these files are promptly redeployed. The following steps outline the process:

Update Kubelet config:

$:# vi /etc/default/kubelet
KUBELET_EXTRA_ARGS=--node-ip=x.x.x.x #newIPaddress
$:#

Update Kubernetes manifest

Edit the following Kubernetes manifests files - etcd.yaml and kube-apiserver.yaml. In the etcd.yaml - update the old IP address registered with the new IP address. You can execute the below, and it will update the address wherever it is present in the file.

$:# vi /etc/kubernetes/manifests/etcd.yaml
$:#

Once the file is open, ensure you are escaped and run this command:

:%s/{oldIPaddress}/{newIPaddress}/gc

In the kube-apiserver.yaml file, update the IP address value of the advertise-address and save the file.

$:# vi /etc/kubernetes/manifests/kube-apiserver.yaml
$:#

--
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=x.x.x.x #newIPaddress

With the above steps executed, the manifest files are updated, and now you may proceed with updating the certificate accordingly.

Retrieve the kubeadm Configuration File:

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' --insecure-skip-tls-verify > kubeadm.yaml

Fine-Tune the Configuration File:

Open the saved kubeadm.yaml in a text editor and navigate to the certSANs section under apiServer. Update the new IP address or hostname.

apiServer:
  certSANs:
  - "10.10.10.100"
  - "kubernetes.default"
  - "new-hostname"
  - "X.X.X.X" #newIPaddress
  extraArgs:
    ...

Recreate API Server Certificates:

This process will generate a new certificate and key for the API server, using the specified configuration file. Since the specified configuration file includes a certSANs list, kubeadm will automatically add those SANs when creating the new certificate.

mv /etc/kubernetes/pki/apiserver.{crt,key} ~
kubeadm init phase certs apiserver --config kubeadm.yaml

Restart the kubeapiserver Container:

Depending on the container runtime being used, follow either of the options below

Using Containerd as the runtime:

Run crictl pods | grep kube-apiserver | cut -d' ' -f1 to get the Pod ID for the Kubernetes API server Pod.
Run crictl stop <pod-id> to stop the Pod.
Run crictl rm <pod-id> to remove the Pod.

Using Docker as the runtime:

Run docker ps | grep kube-apiserver | grep -v pause to get the container ID for the container running the Kubernetes API server.
Run docker kill <containerID> to kill the container.

Upon completion of these steps, Kubelet will autonomously initiate the restart of the container, facilitating the adoption of the updated certificate. Following the API server's restart, immediate connectivity will be established, allowing access through either of the newly-added IP addresses or hostnames.

Validation:

This is dual pronged:

Use kubectl to access the cluster and confirm that the node and pod are all accessible.

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get ns

Alternatively, employ this verification method, leveraging OpenSSL on the Kubernetes control plane node. This step entails decoding the certificate and inspecting the list of Subject Alternative Names (SANs) embedded within the certificate:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

Inspect the "X509v3 Subject Alternative Name" line, which lists the DNS names and IP addresses incorporated as Subject Alternative Names (SANs) in the certificate. Executing this procedure should show the newly-added names and IP addresses specified in the modified kubeadm configuration file. If any error persists, revisit the process, checking for common errors such as overlooking the removal of the previous certificate and key or neglecting to include --config kubeadm.yaml in the kubeadm init phase certs command.

Updating the In-Cluster Configuration:

Assuming a smooth execution of all changes, the final step entails updating the kubeadm ConfigMap stored in the cluster. This guarantees that when kubeadm is utilized for a cluster upgrade later, the updated information seamlessly integrates into the cluster

kubeadm config upload from-file --config kubeadm.yaml

Verify the successful application of changes to the configuration with the command:

kubectl -n kube-system get configmap kubeadm-config -o yaml

Conclusion:

In the dynamic world of Kubernetes, ensuring secure access to the API server is a critical aspect of maintaining a robust and reliable infrastructure. By carefully updating and fine-tuning the TLS certificates used by the Kubernetes API server, administrators can seamlessly integrate new IP addresses or hostnames into the certificate's Subject Alternative Names (SANs), thus enabling secure and uninterrupted access. However, it is crucial to follow the outlined steps diligently and verify the changes to guarantee a smooth transition without compromising the security of the Kubernetes cluster.

Thank you for reading, and I hope someone finds this helpful. I am happy to chat further about new knowledge, opportunities and possible corrections. I can be reached on twitter via, @oluseyeo_

Happy Hacking 💥 💥

Securing Kubernetes: Adding a new hostname or IP address to Kubernetes API Server

oluseyeo — Tue, 28 Nov 2023 05:19:19 +0000



$:# kubectl get ns
Unable to connect to the server: x509: certificate is valid ...
$:#

Introduction:

Before You Begin:

Updating Kubernetes API Server Certificates:

Update Kubelet config:



$:# vi /etc/default/kubelet
KUBELET_EXTRA_ARGS=--node-ip=x.x.x.x #newIPaddress
$:#

Update Kubernetes manifest



$:# vi /etc/kubernetes/manifests/etcd.yaml
$:#

Once the file is open, ensure you are escaped and run this command:



:%s/{oldIPaddress}/{newIPaddress}/gc

In the kube-apiserver.yaml file, update the IP address value of the advertise-address and save the file.



$:# vi /etc/kubernetes/manifests/kube-apiserver.yaml
$:#



--
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=x.x.x.x #newIPaddress

With the above steps executed, the manifest files are updated, and now you may proceed with updating the certificate accordingly.

Retrieve the kubeadm Configuration File:



export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' --insecure-skip-tls-verify > kubeadm.yaml

Fine-Tune the Configuration File:

Open the saved kubeadm.yaml in a text editor and navigate to the certSANs section under apiServer. Update the new IP address or hostname.



apiServer:
  certSANs:
  - "10.10.10.100"
  - "kubernetes.default"
  - "new-hostname"
  - "X.X.X.X" #newIPaddress
  extraArgs:
    ...

Recreate API Server Certificates:



mv /etc/kubernetes/pki/apiserver.{crt,key} ~
kubeadm init phase certs apiserver --config kubeadm.yaml

Restart the kubeapiserver Container:

Depending on the container runtime being used, follow either of the options below

Using Containerd as the runtime:

Run crictl pods | grep kube-apiserver | cut -d' ' -f1 to get the Pod ID for the Kubernetes API server Pod.
Run crictl stop <pod-id> to stop the Pod.
Run crictl rm <pod-id> to remove the Pod.

Using Docker as the runtime:

Run docker ps | grep kube-apiserver | grep -v pause to get the container ID for the container running the Kubernetes API server.
Run docker kill <containerID> to kill the container.

Validation:

This is dual pronged:

Use kubectl to access the cluster and confirm that the node and pod are all accessible.



export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get ns

Alternatively, employ this verification method, leveraging OpenSSL on the Kubernetes control plane node. This step entails decoding the certificate and inspecting the list of Subject Alternative Names (SANs) embedded within the certificate:



openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

Updating the In-Cluster Configuration:



kubeadm config upload from-file --config kubeadm.yaml

Verify the successful application of changes to the configuration with the command:



kubectl -n kube-system get configmap kubeadm-config -o yaml

Conclusion:

Thank you for reading, and I hope someone finds this helpful. I am happy to chat further about new knowledge, opportunities and possible corrections. I can be reached on twitter via, @oluseyeo_

Happy Hacking 💥 💥

PASSING THE AWS SOLUTIONS ARCHITECT – ASSOCIATE EXAM SAA-C02 IN 30 DAYS.

oluseyeo — Mon, 29 Nov 2021 12:12:00 +0000

Writing about my experience especially when it has to do with exams is not what I’d normally do. However, following the various feedback I got following my success at the ‘AWS Solutions Architect – Associate’ exam (“SAA-C02”, “CSAA”, or “the Exam”), I am, therefore, compelled to share this article to assist future test-takers in preparation for the Exam.

The AWS Solutions Architect – Associate exam, is one of the most popular certification exams in the suite of proficiency tests offered by AWS - the biggest public cloud service provider in the world. The exam tests for proficiency in and understanding of how the different services of AWS can be architected to build scalable and resilient infrastructure.

Of the associate level Cloud architecture exams I have taken, (Azure, Google Cloud & Alibaba), I found AWS’s – at the time of writing the exam, the most difficult of the four. With better preparation than I did, - 30 days of study in total, I reckon my judgment of the AWS-SAA-C02 would be better.

Why 30 days? Well, I had less time due to work and other life events. Therefore, I had to leverage my wealth of knowledge and experience of the other Cloud computing services to run through my study within the short period.

EXAM RECOMMENDATION

It is highly recommended that test-takers have at least one year of working or designing applications within the Public Cloud services – most preferably, AWS, prior to sitting for the AWS-CSAA exam. For noobs who are just making their foray into the AWS and Cloud Computing space in general, I’d recommend at least 6 months of study, or taking the AWS Cloud Practitioner exam first, as it is better suited as an introductory exam. It is the perfect foundational exam to test for knowledge of key AWS services before dipping two feet in service architecture.

THE ACTION ITSELF

The AWS-CSAA tests for knowledge across (4) major domains namely;

Designing Resilient Architecture
Designing High Performing Architectures
Designing Secure Applications & Architectures
Designing Cost Optimized Architectures

These pillars sum the core of what the AWS- SAA-C02 exam is built on.

For study purposes, however, I shall not bore you with the standardized nomenclatures, and would rather reclassify these domains into core reading areas – made up of services the test is built on.

A popular reference in the Developer (Technology) community is how professionals rely less on official documentation (maybe because they are considered too abstract), but more on shared practical knowledge from seniors and colleagues in the industry. This is what has necessitated the popularity of developer blogs and the greatest of all, Stackoverflow.

My reclassification of the four domains, and concepts to masters are into these 10 areas respectively;

1. SECURITY

IAM Users, Roles, Group & Policy, Securing Root User Account, Role Assignment - STS & metadata, Certificate Manager, Parameter store, Secrets Manager, KMS and encryption, CloudHSM, Macie for PII, Presigned S3 URLs, AWS WAF, GuardDuty & Shield for DDOS. The network layers at which these services operate.

2. NETWORKING

VPC provisioning – how subnets come to play, differences between Security Group & NACL, NAT Gateway and how private subnets connect to it, PrivateLink, Peering, DirectConnect, CloudHub and TransitGateway Connection options for connections between AWS Services and how to connect Onprem resources.
Routing with Route53 – the different routing policies and when to use them.
Load Balancers – how they work in sync with other AWS services, and external traffic routing.

3. COMPUTE & SERVERLESS

Launching EC2, AMI– types and selection, how to save cost with Reserved instances and differentiate from Spot instances. Leveraging Spot Fleets, EC2 Roles, and Placement Groups. Difference between Metadata and UserData.
What Lambda functions are and how to trigger it, differentiating between what EKS and ECS are, and what Fargate does.

4. STORAGE

The storage types and lifecycle method – one-way traffic to Glacier Storage. Object versioning, backup and its rules. How to host static websites with S3 and integrate Cloudfront. Object lock and different modes to meet regulations.
Volume Storage: Differences between Block Storage and File System. The different types of the Elastic File System. How to encrypt and unencrypt attached volumes.

5. DATABASE & CACHING

Categorize DB’s between OLTP and OLAP; SQLs and NoSQLs. Understand the various DB engines available, and Amazon’s proprietary DB, Aurora and the edge it has over other DBs, DB Caching, Difference between Read-Replicas and Secondary Tables, Failover in a Multi-regional setup. How to build highly-available and highly-performant DB that can either scale vertically or horizontally.
How to setup Elastic Cache, the role of DAX on DynamoDBs and the advantage Global Accelerator affords service architecture.

6. BIG DATA & IOT

The roles of Redshift, Athena, ElasticSearch, and Kinesis in Data processing and streaming. Be able to differentiate between the streaming options in Kinesis – near real-time, and real-time, and the list of AWS services Firehouse streams to.

7. AUTOMATION & DECOUPLING WORKFLOWS

SNS, SQS. How DLQ is setup and its purpose to message processing.
Identify the various types of Message ordering, its size, limits per second per category, and the role of SNS in the Pub-Sub message delivery model.
What is CloudFormation, Stacks and its lifecycle. Knowledge of Elastic Beanstalk will come in handy as well.

8. MIGRATION

How to migrate data with the SNOW options – size, and realistic migration timelines. Storage Gateway & DataSync.
Understand how to optimize data transfers from on-prem to the Cloud.

9. MONITORING & LOGGING

Cloud watch events, Cloud Watch Logs and storage options. Differentiation between default, and custom metrics.

10. COMPLIANCE AND GOVERNANCE

What AWS Organizations is, and the role is managing sub-accounts, AWS Tags and Resource Groups.
Auditing with Trusted Advisor, Budget Management with Budgets & Costs explorer, AWS Config and AWS Systems Manager in managing compliance across instances,

NOTE!:

While the above isn’t exhaustive, the questions I encountered during the exam required a clear understanding of the aforementioned services.

In addition, note that AWS is heavy on 3 core components that serves as the base for most of the questions, and without which, your journey towards being a Cloud Solutions architect is considered half-baked.

i. High Availability: The system lifecycle must not have a single point of failure, and should continue to function, even in a degraded state.

ii. Fault Tolerance: The system must operationally function, despite some or complete component failure, that may include a zone/regional blackout.

iii. Cost Optimization: AWS is highly keen on cost-efficient service architectures. Thus, as you architect solutions that are considered highly available and performant, they must be at the lowest price point possible.

STUDY RESOURCES:

I primarily used 3 resources for the exam. Kindly note that this isn’t an exhaustive list, as I flickered through many resources randomly to find the best fit for my study style.

Acloud Guru – Kudos to the ACG team for the course compilation. Both the content, exam tips, and practice tests were helpful to grasping the needed knowledge for the exam.
AWS re:Invent Sessions on YouTube – This was my audio through my daily commute. So I simply made a playlist of re:invent sessions and listened while in transit.
AWS Certified Solutions Architect Study Guide – this book was written by Ben Piper, David Clinton, and was a game-changer for me in understanding some concepts beyond the surface level. One out of many concepts was understanding the Read and Write throughput capacity for DynamoDB. Also noteworthy are the tests at the end of each chapter to check the understanding of each concept.
Last but not least is hands-on experience in the AWS Management console – I urge all test takers to take advantage of the 12 months (1 Year) free tier offering from AWS. With this personal free tier account, you can access most AWS services at no cost, if you stay within the limits of the offering. AWS offers as much as 750 FREE monthly hours for EC2.

You can see the full details of the services covered here

Never discount Practice tests – take them as though it was the real exam to see what area you need to brush up knowledge on.

Post-Exam

Finally, finishing the certification is just the beginning of the AWS journey for me. AWS services are intriguing, and I’m studying further to deepen my knowledge for DevOps and shortly, the Solutions Architect Professional exam. Therefore, the journey to Cloud Infrastructure mastery, particularly, how to help businesses leverage cloud service offerings to improve agility and ultimately gain a business advantage is a life-long learning journey for me.

In Conclusion

In a future blog post, I shall write in more detail on the key knowledge to acquire on each of the pillars that make up the exam. In the interim, I am happy to connect and answer any additional questions you may have about the SAA-C02 exam and/or speak at community sessions - sharing knowledge about Public Cloud Infrastructure. You may also leave your questions and comments below, and I’ll get back to answering them as soon as possible.

You may also connect with me via Twitter @oluseyeo_ or LinkedIn - Oluwaseye

Best of luck, and keep building!

How to Create Relationships with Mongoose and Node.JS

oluseyeo — Tue, 15 Sep 2020 05:53:05 +0000

FOCUS: One-to-many Relationships

NoSQL databases, unlike SQL databases like PostgreSQL, MYSQL etc, which are traditionally built for data relationship management, indexed and referenced across multiple tables, have a poor or almost non-existent support for relationships in her JSON-like built schema. MongoDB, a popular NoSQL database, like others, have inbuilt methods that developers can leverage to build relationships between multiple schemas.

Relationships in MongoDB are built on the JOIN functionality and with the popular NPM module, the Mongoose library, developers can harness its raw power, building complex relationships, and importantly, designing efficient databases to avoid throttling queries, as it would have been done, if working with an SQL database.

In this tutorial, I am going to be touching on the following in details:

Types of relationships & object reference types in MongoDB
Mongoose Populate Method
Mongoose Virtuals

Prerequisites:

It is expected that readers have a good basic grasp of ExpressJS, Mongoose, ES6+ JS & Postman.

Also, the following should be available either as a service or installed and running locally on your PC:

MongoDB or you can choose Atlas, the cloud version of MongoDB.
Mongoose NPM. Simply run [npm i mongoose ] at the root of your project folder.
Postman, to test the endpoints.



"npm i mongoose"

For the purpose of this write-up, I have built a small “Publishing House” project, to walk you through how to achieve any of the methods to be discussed. The Publishing House project assumes Publishers as registered users, who can publish multiple books under their portfolio.

MongoDB as database.
Mongoose library, as the database object document manager (ODM).
ExpressJS to create our routes using async/await ES6+ since we shall be dealing with promises.
Postman will be used to test our endpoints for responses.

Mongoose represents relational data using two major design models, and the choice of model to deploy when planning the database collections of any project is predominantly hinged on the data-size, data accuracy, and frequency of access. Nonetheless, the rule of thumb is, the size of documents stored, is in direct proportion to the speed at which queries are resolved, and ultimately, how performant the database is.

The two models are as follows:

Embedded Data Models [Denormalization]: This is the least recommended form of relationship. Data is simply denormalized by embedding Child (related) documents right into the Parent (main) document. Using our “Publishing project” as an example, this would mean, Publishers, store all published books and related information directly on each publisher’s object.
In a typical One-to-Few document relationship, this would work perfectly as the expected size of documents is not more than 20. However, when working with Child documents of a larger size, this size heavily impairs database performance, causing lags, and difficulty in keeping data synced, ultimately bringing about poor user experience.
Referenced Data Model [Normalization]: When data is normalized, it means documents are separated into different collections, and they share references between each other. In most cases, a single update on the Parent document, with all parameters passed, updates the child documents directly referenced to it. The rest of this tutorial will be focused on the best use case of this method, and how best to organize our database collections and documents in an efficient manner.

Referencing documents between collections can be done via dual approaches, and are as follows:

Child Referencing: A document is considered Child referenced, when the Parent document stores a reference to its child collections, storing its identifiers - in most situations, the id, in an array of similar identifiers on the Parent document. Citing our “Publishing House” project, this would mean, having Publishers store the book._id for each book created, in an array of book id’s, predefined on the Publisher's Schema, and when needed, fetch these child documents using the populate method.

From our Project, see the Publisher's schema below:



const mongoose = require('mongoose');
const {Schema} = require('mongoose');

const publisherSchema = new Schema({
   name: String,
   location: String,
   publishedBooks: [{
      type: Schema.Types.ObjectId,
      ref: 'Book'
   }]
},
{timestamps: true});

module.exports = mongoose.model('Publisher', publisherSchema);

Publisher Schema [Notice published books is an array]

Here is our Book Schema:



const mongoose= require('mongoose');
const {Schema} = require('mongoose');

const bookSchema = new Schema({
   name: String,
   publishYear: Number,
   author: String,
   publisher: {
      type: Schema.Types.ObjectId,
      ref: 'Publisher',
      required: true
   }
},
{timestamps: true});

module.exports = mongoose.model('Book', bookSchema);

Book Schema

The mongoose “populate” method loads the details of each referenced Child documents and returns it alongside each Publisher's document fetched from the DB. Let’s see an example of this using our project.

We start by creating a new Publisher below:



/***
 * @action ADD A NEW PUBLISHER
 * @route http://localhost:3000/addPublisher
 * @method POST
*/
app.post('/addPublisher', async (req, res) => {
   try {
      //validate req.body data before saving
      const publisher = new Publisher(req.body);
      await publisher.save();
      res.status(201).json({success:true, data: publisher });

   } catch (err) {
      res.status(400).json({success: false, message:err.message});
   }
});

Create a new publisher



{
    "success": true,
    "data": {
        "publishedBooks": [],
        "_id": "5f5f8ac71edcc2122cb341c7",
        "name": "Embedded Publishers",
        "location": "Lagos, Nigeria",
        "createdAt": "2020-09-14T15:22:47.183Z",
        "updatedAt": "2020-09-14T15:22:47.183Z",
        "__v": 0
    }
}

A new publisher

Next, the newly created Publisher proceeds to add a new book about to publish to it's DB. The publisher’s _id is passed in as a value to the Publisher’s key on the Book schema before saving, and in the same request loop, right after calling the save method on the new book, the newly created book object returned from the Promise, MUST be passed as a parameter to a push method, called on the Publisher’s key. This would ensure that the book object, is saved on the Publisher's document.

Here's the magic breakdown:



/***
 * @action ADD A NEW BOOK
 * @route http://localhost:3000/addBook
 * @method POST
*/

app.post('/addBook', async (req, res)=>{

   /**
    * @tutorial: steps
    * 1. Authenticate publisher and get user _id.
    * 2. Assign user id from signed in publisher to publisher key.
    * 3. Call save method on Book.
   */

   try {
      //validate data as required

      const book = new Book(req.body);
      // book.publisher = publisher._id; <=== Assign user id from signed in publisher to publisher key
      await book.save();

      /**
       * @tutorial: steps
       * 1. Find the publishing house by Publisher ID.
       * 2. Call Push method on publishedBook key of Publisher.
       * 3. Pass newly created book as value.
       * 4. Call save method.
      */
      const publisher = await Publisher.findById({_id: book.publisher})
      publisher.publishedBooks.push(book);
      await publisher.save();

      //return new book object, after saving it to Publisher
      res.status(200).json({success:true, data: book })

   } catch (err) {
      res.status(400).json({success: false, message:err.message})
   }
})

A Publisher adding a new book to be published to her DB

This is the defined way to saving child document references(id’s) on the publisher’s document. On successful creation, the below is returned when you query the Publisher's id.

PS: The Publisher below created 3 new books.



{
    "publishedBooks": [
        {
            "_id": "5f5f8ced4021061030b0ab68",
            "name": "Learn to Populate virtuals Mongoose",
            "publishYear": 2019,
            "author": "Devangelist"
        },
        {
            "_id": "5f5f8d144021061030b0ab6a",
            "name": "Why GoLang gaining traction",
            "publishYear": 2020,
            "author": "John Doe"
        },
        {
            "_id": "5f5f8d3c4021061030b0ab6b",
            "name": "Developer Impostor syndrome",
            "publishYear": 2021,
            "author": "John Mark"
        }
    ],
    "_id": "5f5f8ac71edcc2122cb341c7",
    "name": "Embedded Publishers",
    "location": "Lagos, Nigeria",
    "createdAt": "2020-09-14T15:22:47.183Z",
    "updatedAt": "2020-09-14T15:33:16.449Z",
    "__v": 3
}

Saved object returns Child array

However, Should the push and save method not be called on the Publisher's document, the Publisher although existing, and the new Book created, will return an empty array of publishedBooks as seen below, when queried.



{
    "success": true,
    "data": {
        "publishedBooks": [],
        "_id": "5f5f8ac71edcc2122cb341c7",
        "name": "Embedded Publishers",
        "location": "Lagos, Nigeria",
        "createdAt": "2020-09-14T15:22:47.183Z",
        "updatedAt": "2020-09-14T15:22:47.183Z",
        "__v": 0
    }
}

Empty Array, when object isn't pushed and saved

Despite the success of the Child Referencing method, its limitation as seen above is that the size of the array of Id’s can get very large quickly, consequently seeing the database lose efficiency and performance overtime as the size of the array grows. MongoDB officially recognizes this as an anti-pattern, and strongly discourages its use for document relationships run at scale.

Parent Referencing: Parent referencing, on the other hand, is a tad different from Child Referencing as described earlier, in that, ONLY Child documents keep a reference to parent documents. This reference is singly kept on each Child document created, defined as an object ID on the Schema. Parent documents, conversely, keep no direct reference but builds one with the help of a Mongoose method called Virtuals.

Mongoose Virtual is a far more sophisticated approach to fetching referenced Child documents, and it importantly, takes up less memory for data storage, as the new key-field Mongoose virtual creates whenever a query is run, doesn’t persist on the Parent document. Occasionally, Virtuals are also referred to as "reverse-populate', as such, when you hear people mention that, don't fret!

Enough with the talk, let's jump into our project code.
First, let's see what our Book Schema looks like below:



const mongoose= require('mongoose');
const {Schema} = require('mongoose');

const bookSchema = new Schema({
   name: String,
   publishYear: Number,
   author: String,
   publisher: {
      type: Schema.Types.ObjectId,
      ref: 'Publisher',
      required: true
   }
},
{timestamps: true})

module.exports = mongoose.model('Book', bookSchema);

Next, which is where the tricky part lies, is our Parent document. Please pay attention to how virtuals are defined and a crucial part of this is the extra options we must set on the Schema, without which no results get returned. These extra options are the toJSON and toObject options. They both default to false, and are core to ensuring that whenever the Parent document is queried when these options are set to True, results are passed to the .json() method on the response call.



const mongoose = require('mongoose');
const {Schema} = require('mongoose');

const publisherSchema = new Schema({
   name: String,
   location: String
},
   {timestamps: true}
);

/**
 * @action Defined Schema Virtual
 * @keys 
 *    1.   The first parameter can be named anything.
 *          It defines the name of the key to be named on the Schema
 * 
 *    2. Options Object
 *       ref: Model name for Child collection
 *       localField: Key for reference id, stored on Child Doc, as named on Parent Doc.
 *       foreignField: Key name that holds localField value on Child Document
 */
publisherSchema.virtual('booksPublished', {
   ref: 'Book', //The Model to use
   localField: '_id', //Find in Model, where localField 
   foreignField: 'publisher', // is equal to foreignField
});

// Set Object and Json property to true. Default is set to false
publisherSchema.set('toObject', { virtuals: true });
publisherSchema.set('toJSON', { virtuals: true });


module.exports = mongoose.model('Publisher', publisherSchema);

Notice that we don’t have a publishedBooks array anymore on the Schema

Defining the virtual object comes next, and the best way to easily remember how to define it, (much easier if you’re from an SQL background), is;

SELECT “name for the virtual field” FROM “ref – Child collection name”, WHERE “localField – Parent key stored on child collection, mostly id” EQUALS “_foreignField – the name of Child schema key, storing parent id, as its value.

With both options above defined, whenever we populate our Publisher after calling the GET method, we are guaranteed to retrieve all books published by each publisher, and for further specificity, as not all the information about a book will be needed, select the keys required from each book and return it in the response body.

See how it is done in our project below:



/***
 * @action GET ALL PUBLISHERS
 * @route http://localhost:3000/publishers
 * @method GET
 */
app.get('/publishers', async (req, res) => {
   try {
      const data = await Publisher.find()
                                 .populate({path: 'booksPublished', select: 'name publishYear author'});
      res.status(200).json({success: true, data});
   } catch (err) {
      res.status(400).json({success: false, message:err.message});
   }
})

Get all Publishers



{
    "success": true,
    "data": [
        {
            "_id": "5f5f546e190dff51041db304",
            "name": "Random Publishers",
            "location": "Kigali, Rwanda",
            "createdAt": "2020-09-14T11:30:54.768Z",
            "updatedAt": "2020-09-14T11:30:54.768Z",
            "__v": 0,
            "booksPublished": [
                {
                    "_id": "5f5f548e190dff51041db305",
                    "name": "Mastering Mongoose with Javascript",
                    "publishYear": 2020,
                    "author": "Devangelist",
                    "publisher": "5f5f546e190dff51041db304"
                },
                {
                    "_id": "5f5f55ca190dff51041db307",
                    "name": "Learning Mongoose Populate method",
                    "publishYear": 2019,
                    "author": "Devangelist",
                    "publisher": "5f5f546e190dff51041db304"
                }
            ],
            "id": "5f5f546e190dff51041db304"
        }
}

Query results from getting all publishers [Notice the booksPublished array]

Summarily, Parent referencing is the best approach to referencing when using the Normalized model method and dealing with a large dataset.

If you made it to this point, thank you for reading through, and I hope you’ve learnt something-[new]. I’m happy to chat further about new knowledge, opportunities and possible corrections. I can be reached on twitter via, @oluseyeo_ or via email at, sodevangelist@gmail.com.

Happy Hacking 💥 💥

TL: DR;

There are two modelling approaches, Embedded and Referenced.
Embed only when your data will be accessed less frequently and you’re mostly only reading data.
For larger IOPS, use referencing model.
Referencing can be done in two ways, Child and Parent referencing.
If Child document size is small, under 100, use Child referencing. This stores child reference key directly on Parent document using the push method.
If the size of Child documents is huge, use the parent referencing option, reverse populating Parent documents using mongoose virtual.

Recommended further reading:
Data Access Patterns
Mongoose Documentation
Denormalization

HTML, CSS & Javascript good enough for a Newbie?

oluseyeo — Sat, 16 May 2020 12:06:25 +0000

HTML, CSS with a sprinkle of Javascript, will likely not land you your first role as a Developer, in 2020.

With this, I welcome you to my first public article.

This tweet made rounds on #techTwitter, and most persons mistook Danny for a gatekeeper who was raising the entry-level bar for noobs.

Smacking a few pixels around, and being able to lay them out properly on a page, to be displayed in browsers about a decade ago, would land you, your first job as a Web-Developer. Unfortunately, it isn't so anymore!
At the time, it wasn’t expedient to know more for starters, as most firms had a training culture for Juniors, and in their ranks also, were senior developers who were committed to training and retraining, which ultimately, would result in your professional growth as a noob.

Circa 5years back, this training culture for junior developers, create a gap that resulted in the Bootcamp boom experienced around the world. This, saw to companies outsourcing a huge chunk of their internal training programs to Bootcamps, trusting their expertise to train committed and passionate noobs to become world-class developers, who can be hired.

This background story isn’t to share the evolution and proliferation of Web programming but to share insights on how events have evolved to this point where, HTML, CSS and a bit of Javascript would not open the doors so easily anymore.

I bet you’re asking, what should I know?

As it stands, there’s no general consensus on what stack or knowledge depth is sufficient, however going by statistics as gleaned from open job positions, it’s expected to know the following depending on your pick of the development-side.

At base, you are to know
⁃ HTML5
⁃ CSS3
⁃ Javascript(ES6) - Arrow functions shouldn't be alien to you.
⁃ GIT (Github/Gitlab)
⁃ Command Prompt

From this point, you can then choose the side to fight for.

If you choose Client-Side Development(Frontend)

⁃ React or Vue or Angular.
⁃ Preferably, a testing framework alongside.

➡️ Read more about the Frontend Developer Roadmap for 2020 here

For Server-Side Development(Backend)

⁃ PHP (Laravel) or NodeJs(Express) or Python(Django)
⁃ Database ( SQL & No-SQL; most tend to pick MongoDB or MySQL)
⁃ Preferably, a testing framework alongside.

➡️ Read more about the Backend Developer Roadmap for 2020 here

Fullstack:
⁃ Everything above.
⁃ AWS or GCP (the basics)

NOTE:
Before jumping into any framework, please understand, NOT cram, the basics of the language.

As I bring this to a close, here's a list of awesome resources that have helped me so far.
PS: This isn't an exhaustive list, neither am I affiliated with any of the tutors

⁃ FreecodeCamp: Always grateful for Quincy's vision to start FreecodeCamp. That is the ultimate learning resource for beginners.

⁃ Dr Angela Yu: Angela, is so resourceful. She simplifies the concept behind every step and I highly recommend her lectures. I look forward to meeting her someday, so I can at least buy her coffee and say, thank you.

⁃ Brad Traversy: Please, follow his YouTube channel if you cannot afford to buy his courses yet. Brad is mindblowing!

⁃ Codecademy: They’ll get your hands dirty with so many projects that will up your skill.

⁃ Maximilian Schwarzmuller: He goes an extra mile to ensure that his student understands the art. He puts up courses on Udemy.

If there are other resources that have been helpful to you, please do not hesitate to suggest them in the comments section.

Conclusively, remember, starting out with an easier language will make it simpler to master complex languages later. As your career progress, you'll get to master not only one but multiple languages, as it'll deepen your versatility and success in any programming environment you find yourself.

With love ❤️

Olu the Devangelist! #Reskill