Forem: JoeHo

Disable anonymous bind for OpenLDAP in Centos7

JoeHo — Thu, 13 Jul 2023 13:34:24 +0000

LDAP bind is a process which the client tries to authenticate themselves to the server. Depends on the server set up, such bind request sent from client may contain no credentials (i.e. anonymous bind).

In this guide, I will share how to configure the LDAP bind feature.

Concept
Before diving into the configuration, it’s better to know the types of LDAP bind.

Anonymous bind
Anonymous bind is that you present no distinguished name (you may treat it as an account name) and password in the bind request, the LDAP server will treat you as an anonymous. Usually, we will combine it with LDAP Access Control (ACL) to prevent anonymous from knowing some sensitive data if you decide to open part of the LDAP data to the public.

Unauthenticated bind
Unauthenticated bind allows you to present distinguished name and no password. By default, it’s disabled, as many applications don’t realize that they can still bind to LDAP server with incorrect password.

Authenticated bind
Authenticated bind requires the client to provide distinguished name and password.

Disable anonymous bind for OpenLDAP
By default, you can query LDAP data as an anonymous

ldapsearch -LLL -x -b "dc=abc,dc=local" '(uid=joe)'

Now, we will disable it. Create a disable_bind_anon.ldif with below content

dn: cn=config
changetype: modify
add: olcDisallows
olcDisallows: bind_anon
Apply the configuration

sudo ldapadd -Y EXTERNAL -H ldapi:/// -f disable_bind_anon.ldif

If we try again, we can no longer query the user

ldapsearch -LLL -x -b "dc=abc,dc=local" '(uid=joe)'

Conclusion
In this guide, we discuss what bind is, types of bind and how to disable anonymous bind.

Original Post: Disable anonymous bind for OpenLDAP in Centos7 | Joe Ho Blog

Install and configure OpenLDAP server in CentOS 7

JoeHo — Fri, 30 Sep 2022 13:38:46 +0000

LDAP, Lightweight Directory Access Protocol is a directory service for us to manage identities and objects easily. One of the most common application is that you can authenticate to a server with LDAP. In Windows world, the server is usually Active Directory. In Linux world, OpenLDAP is widely adopted.

In this guide, I will show you how to install and configure a simple workable LDAP server.

Concept

Before diving into the installation and configuration, it's better to know some terms used in LDAP.

Attribute

An attribute is a characteristic of an object. For example, an email of an account.

Object Class

An object class defines what attributes that object can have. For example, we define an object class, InetOrgPerson, it may contain displayName and mail attributes. Depends on the definition of object class, the attributes specified can be mandatory or optional.

Distinguished Name (DN)

Distinguished Name lets us uniquely identify the object. It is similar to the file path in a reverse order. For example, uid=joeho,OU=People,DC=abc,DC=local is a DN

Entry

An entry is just an object. You define what object class this entry belongs to & each object class defines what attributes this object has. Each entry can belong to multiple object classes and need to have all mandatory attributes specified in all object classes it belongs to.

Schema

A schema contains the definitions of various attributes and object classes.

Domain Component (DC) & Organizational Unit (OU)

They are containers, contains object & let you manage objects in a hierarchy manner. People use them commonly.

OpenLDAP Installation

Install OpenLDAP related packages

sudo yum install openldap* -y
sudo systemctl start slapd
sudo systemctl enable slapd
sudo systemctl status slapd # Check service is started & enabled

OpenLDAP Configuration

Generate OpenLDAP password and save it

sudo slappasswd

Then, we will use ldapmodify to update /etc/openldap/slapd.d/cn=config/olcDatabase={2}hdb.ldif, which is our database config file

We will create a file & customize and paste content below

vi db.ldif

Content you should paste:
You should replace with your customized values

olcSuffix (should be replaced by your domain, e.g. example.com -> dc=example,dc=com)
olcRootDN (should be replaced by your domain admin name, can be any name you prefer, e.g. admin -> cn=admin,dc=abc,dc=local)
olcRootPW (should be the password you generate above)

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=abc,dc=local

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcRootDN
olcRootDN: cn=admin,dc=abc,dc=local

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}xxxxx

Run this command to update.

sudo ldapmodify -Y External -H ldapi:/// -f db.ldif

Configuration of /etc/openldap/slapd.d/cn=config/olcDatabase={2}hdb.ldif should change

Apply some commonly used schema. The 2nd & 3rd schema allow us to create an object with InetOrgPerson & ShadowAccount which we will use to create an user

sudo ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif
sudo ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif
sudo ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif

OpenLDAP Verification

Create objects, Organizational Unit and group

Create a file, entries.ldif, and add below content which

create a user, joe
assign joe to 2 groups, joe & Engineering

dn: dc=abc,dc=local
dc: abc
objectClass: top
objectClass: domain

dn: ou=People,dc=abc,dc=local
objectClass: organizationalUnit
ou: People

dn: ou=Groups,dc=abc,dc=local
objectClass: organizationalUnit
ou: Groups

dn: cn=Engineering,ou=Groups,dc=abc,dc=local
cn: Engineering
objectClass: posixGroup
gidNumber: 20100
memberUid: joe

dn: uid=joe,ou=People,dc=abc,dc=local
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: joe
sn: Ho
givenName: Joe
cn: Joe Ho
displayName: Joe Ho
uidNumber: 20001
gidNumber: 20001
loginShell: /bin/bash
homeDirectory: /home/joe
shadowMin: 0
shadowMax: 2
shadowWarning: 1
userPassword: {CRYPT}x
shadowLastChange: 19261
dn: cn=joe,ou=Groups,dc=abc,dc=local
cn: joe
objectClass: posixGroup
gidNumber: 20001
memberUid: joe

Apply the content

ldapadd -x -W -D "cn=admin,dc=abc,dc=local" -f entries.ldif

Test querying LDAP

Query all entries

ldapsearch -D cn="admin,dc=abc,dc=local" -W -b "dc=abc,dc=local"

Conclusion

We complete the whole set up and are able to create users and manage them in LDAP

Original Post: Install and configure OpenLDAP server in CentOS 7 | Joe Ho Blog

How to store secrets in Azure Databricks

JoeHo — Tue, 27 Sep 2022 01:18:31 +0000

Background

In Azure Databricks, we can write code to perform data transformation on data stored in various Azure Services, e.g. Azure Blob Storage, Azure Synapse. However, as other programs, sometimes, you want to protect credentials used in Azure Databricks, Azure Databricks provides a solid secret management approach to help you achieve that.

Steps

Prepare Databricks command-line interface (CLI) in Azure Cloud Shell

Configure your cloud shell environment

Open Cloud Shell & make sure you select “Bash” for the Cloud Shell Environment.

Set up Virtual Environment

Create Virtual Environment with below command.

# Bash
virtualenv -p /usr/bin/python2.7 databrickscli

Activate your virtual environment

Activate your virtual environment with below command.

# Bash
source databrickscli/bin/activate

Install Databricks CLI

Install Databricks CLI with below command.

# Bash
pip install databricks-cli

Create secret in Azure Databricks

Set up authentication

Before you can create a secret, you need to authenticate as a user of the Azure Databricks, which requires your Azure Databrics workspace’s URL and a token

Get your Azure Databricks workspace’s URL

You can navigate to your Azure Databricks workspace and copy its URL.

Generate Access Token for your Azure Databricks workspace

You can follow below steps to retrieve access token

Launch Databricks workspace
Click 'User Settings'
Click 'Generate New Token'
Configure access token & click 'Generate'
Copy access token

Create Secret Scope

After authentication, you need to first create a secret scope which you may group several secrets.

If your databricks is in Standard plan, you can only create secret scope which will be shared with other users in the same workspace.

# Bash
databricks secrets create-scope --scope <<scope>>

# Example
databricks secrets create-scope --scope storage --initial-manage-principal users # Standard Plan
databricks secrets create-scope --scope storage # Premium plan

Create Secret

You can use below command to create secret under the specified scope.

# Bash
databricks secrets put --scope <<scope>> --key <<key name>>

databricks secrets put --scope storage --key blob #Example

Use Secret in Notebook

You can use secret by below command in notebook.

# Python
dbutils.secrets.get(scope=<<scope>>,key=<<key>>)

dbutils.secrets.get(scope=storage,key=blob) #Example

Original Post: https://joeho.xyz/blog-posts/how-to-store-secrets-in-azure-databricks/

How to connect to Azure Synapse in Azure Databricks

JoeHo — Tue, 27 Sep 2022 00:53:53 +0000

Background

With Azure Databricks, we can easily transform huge size of data in parallel and store the transformed data in different Azure services, one of them is Azure Synapse (formerly SQL DW). Azure Databricks has built-in connector which lets us read and write data easily from Azure Synapse.

Prerequisite

Azure Databricks Workspace
Azure Blob Storage or Azure Data Lake Gen 2
Azure Synapse Instance
Azure Synapse User Credentials

Steps

Configure storage key in notebook session

This will configure your storage credentials in your notebook session, which we will use them to connect to that storage. This storage acts as a staging storage when you read and write data from Azure Synapse. Currently, only Azure Blob Storage and Azure Data Lake Gen 2 are supported, they have slightly different configurations. Below are session configuration for both types of storage.

Azure Blob Storage As Staging Storage

# Python

spark.conf.set(fs.azure.account.key.<<your-storage-account-name>>.blob.core.windows.net, ”<<your-storage-account-access-key>>”)

Azure Data Lake Gen 2 As Staging Storage

# Python

spark.conf.set(fs.azure.account.key.<<your-storage-account-name>>. dfs.core.windows.net, ”<<your-storage-account-access-key>>”)

Configure Azure Synapse connection

Next, we will define below variables

Azure Synapse connection string
Staging storage folder (i.e. where some temporary data will be written to when you read/write data from/to Azure Synapse)
An Azure Synapse Table which you will read/write data from/to Azure Synapse

# Python

# Azure Synapse Connection Configuration
dwDatabase = <<your-database-name>>
dwServer = <<your-sql-sever-name>>
dwUser = <<your-database-account>>
dwPass = <<your-database-account-password>>
dwJdbcPort =  "1433"
dwJdbcExtraOptions = "encrypt=true;trustServerCertificate=true;hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
sqlDwUrl = f"jdbc:sqlserver://{dwServer}:{dwJdbcPort};database={dwDatabase};user={dwUser};password={dwPass};${dwJdbcExtraOptions}"

# Staging Storage Configuration
# Azure Blob Storage
# tempDir = "wasbs://<<container>>@<<your-storage-account-name>>.blob.core.windows.net/<<folder-for-temporary-data>>"

# Azure Data Lake Gen 2
tempDir = "abfss://<<container >>@<<your-storage-account-name>>.dfs.core.windows.net/<<folder-for-temporary-data>>"

# Azure Synapse Table
tableName = <<your-azure-synapse-table-to-read-or-write>>

Read Data from Azure Synapse

Then, we will try to read data from Azure Synapse

# Python
df = spark.read \
  .format("com.databricks.spark.sqldw") \
  .option("url", sqlDwUrl) \
  .option("tempDir", tempDir) \
  .option("forwardSparkAzureStorageCredentials", "true") \
  .option("dbTable", tableName) \
  .load()

Write Data to Azure Synapse

Finally, we will try to write data to Azure Synapse

# Python
df.write \
  .mode('append') \ # Append Data
  .format("com.databricks.spark.sqldw") \
  .option("url", sqlDwUrl) \
  .option("tempDir", tempDir) \
  .option("forwardSparkAzureStorageCredentials", "true") \
  .option("dbTable", tableName) \
  .save()

Bonus: Why do we need staging storage

As mentioned above, staging folder is needed to store some temporary data whenever we read/write data from/to Azure Synapse. Whenever we read/write data, we actually leverage PolyBase to move the data, which staging storage is used to achieve high performance.

Another Bonus: Secret Management in Azure Databricks

The code above put our secrets, e.g. user credentials, storage key in pain text. Actually, we should hide them in the notebooks. You can refer to this guide for more detail.

Original Post: https://joeho.xyz/blog-posts/how-to-connect-to-azure-synapse-in-azure-databricks/