Forem: Samuel Earl

Data Pipelines with Great Expectations | Conclusion

Samuel Earl — Tue, 31 Jan 2023 02:42:25 +0000

That's it! Congratulations!

We were able to configure a Great Expectations project and connect a datasource to it. After that, we created an Expectation Suite that used a batch of data to automatically generate our initial Expectations. We then edited our Expectation Suite to only use a couple of Expectations. Finally, we ran a second batch of data through a Checkpoint, which validated the data in that second batch and uncovered existing data issues.

I hope you understand how GX can help you in your data pipelines. At this point you may want to check out the Great Expectations Docs to learn more.

Thank you for reading!

Data Pipelines with Great Expectations | Step 4: Validate data

Samuel Earl — Tue, 31 Jan 2023 02:42:15 +0000

Set up a Checkpoint

A Checkpoint is a step in a data pipeline that runs an Expectation Suite against a batch of data. New validation results are created each time a Checkpoint validates a batch of data. Checkpoints can also be configured to execute Actions (e.g. send an alert message through Slack or email when a validation fails).

We are going to run our Expectation Suite against the February data in our data folder. In order to do that we need to set up a Checkpoint. In your terminal run the following command from your gx-getting-started directory (make sure to shut down any running Jupyter Notebooks first):

great_expectations checkpoint new getting_started_checkpoint

A new browser tab will open with a Jupyter Notebook and the edit_checkpoint_getting_started_checkpoint.ipynb file inside.

File Location

The file to edit your Checkpoint is located at
gx-getting-started/uncommitted/edit_checkpoint_getting_started_checkpoint.ipynb.

This file contains the configuration code for a Checkpoint. Scroll down to the second code cell. You should see the following code:

my_checkpoint_name = "getting_started_checkpoint" # This was populated from your CLI command.

yaml_config = f"""
name: {my_checkpoint_name}
config_version: 1.0
class_name: SimpleCheckpoint
run_name_template: "%Y%m%d-%H%M%S-my-run-name-template"
validations:
  - batch_request:
    datasource_name: getting_started_datasource
    data_connector_name: default_inferred_data_connector_name
    data_asset_name: yellow_tripdata_sample_2019-02.csv
    data_connector_query:
        index: -1
    expectation_suite_name: getting_started_expectation_suite_taxi.demo
"""
print(yaml_config)

Make sure that the data_asset_name is set to yellow_tripdata_sample_2019-02.csv so we can validate the February data. Now scroll down to the last code cell and uncomment both lines of code. Then run all the cells in the notebook, which will create the Checkpoint configs in your project.

This will open the Data Docs again in a new tab.

Inspect the Validation Results for your Checkpoint

You should see that the first result listed under the “Validation Results” tab shows a failed Expectation Suite (i.e. there is a red “X” in the “Status” column). If you click on that row you will see more details about the failure.

You should see that the Expectation for "values must be greater than or equal to 1 and less than or equal to 6" failed. But why?

In the results of that Expectation you will see a small table with a heading "Sampled Unexpected Values" with a 0 below it. That means that the passenger_count column contained the value 0. Since that Expectation was configured to only allow values greater than or equal 1 and less than or equal to 6 for the passenger_count column, the validation failed. (A taxi ride with 0 passengers does not make sense.)

Data Pipelines with Great Expectations | Step 3: Create Expectations

Samuel Earl — Tue, 31 Jan 2023 02:42:02 +0000

An Expectation is essentially a rule that defines how your data should be validated. An Expectation Suite is a collection of Expectations.

To create our first Expectation Suite, let’s go to our terminal. Stop the previous Jupyter Notebook by pressing Ctrl+C and type the following from inside the gx-getting-started directory:

great_expectations suite new

This will bring up the following prompt:

How would you like to create your Expectation Suite?
    1. Manually, without interacting with a sample Batch of data (default)
    2. Interactively, with a sample Batch of data
    3. Automatically, using a Data Assistant
: 3

You can create your expectations in one of three ways, as outlined in the above prompt. We are going to select option 3 and press Enter, which uses a Data Assistant to automatically generate an Expectation Suite based on the profile of our data.

What is a data profile?

If you were to write a profile of a person, then you would write down a description of that person and their characteristics. A data profile is similar. It is a description of the data and its characteristics (e.g. data types, value ranges, mean, median, mode).

The GX Data Assistant requires a batch of data so it can create a data profile. GX will then automatically generate an Expectation Suite that matches the data profile. That automatic Expectation Suite can be used as a starting point and you can edit each Expectation however you want.

The next prompt asks which dataset we want to use to generate our Expectation Suite:

A batch of data is required to edit the suite - let's help you to specify it.

Which data asset (accessible by data connector "default_inferred_data_connector_name") would you like to use?
    1. yellow_tripdata_sample_2019-01.csv
    2. yellow_tripdata_sample_2019-02.csv
: 1

We will use the January 2019 data to create a data profile and an automatic Expectation Suite. We will then validate any subsequent batches of data with the Expectation Suite that was generated from the January 2019 batch. Select option 1 and press Enter.

That will bring up the next prompt.

Name the new Expectation Suite [yellow_tripdata_sample_2019-01.csv.warning]: getting_started_expectation_suite_taxi.demo

You can name the suite anything you want. Let’s use the same name from GX’s Getting Started Tutorial: getting_started_expectation_suite_taxi.demo. Press Enter.

You should see one last prompt that asks if you want to proceed with creating the Expectation Suite. Type y then press Enter.

A new notebook will open up for you automatically in your browser. (You can close the tab with the previous notebook file.)

File Location

The Expectation Suite notebook file is located here: great_expectations/uncommitted/edit_getting_started_expectation_suite_taxi.demo.ipynb

Creating Expectations in Jupyter Notebooks

In the Jupyter Notebook that opened in your browser, scroll down to the second code cell that contains a list named exclude_column_names. This cell allows you to select the columns that you want to include in the Expectation Suite. We want to create an Expectation Suite based on the number of passengers in each taxi ride, so comment out the "passenger_count" column to include it in the suite. It should look like this:

exclude_column_names = [
    "vendor_id",
    "pickup_datetime",
    "dropoff_datetime",
#     "passenger_count",
    "trip_distance",
    "rate_code_id",
    "store_and_fwd_flag",
    "pickup_location_id",
    "dropoff_location_id",
    "payment_type",
    "fare_amount",
    "extra",
    "mta_tax",
    "tip_amount",
    "tolls_amount",
    "improvement_surcharge",
    "total_amount",
    "congestion_surcharge",
]

Run all the cells in the Jupyter Notebook to save the Expectation Suite and create the configs for your new suite. Another browser tab will automatically open to show the Data Docs for your Expectation Suite.

Viewing Expectations in Data Docs

Data Docs are where you can view and edit your Expectations and see the status of your data validations.

The Data Docs open to a page titled “Expectation Validation Result”. If you click on “Home” in the breadcrumbs you will be taken to a page that has two tabs: “Validation Results” and “Expectation Suites”.

The tab for “Validation Results” is where we just were. It shows a list of all the validations that you have run. Each time you run all the cells in the edit_getting_started_expectation_suite_taxi.demo.ipynb file you are running the Expectation Suite against the dataset that you passed to it (i.e. the dataset that is listed next to the data_asset_name key in the first code cell of the edit_getting_started_expectation_suite_taxi.demo.ipynb file).

Also each time you run your Expectation Suite against your dataset a new entry will be created under the “Validation Results” list on the Data Docs home page. On the Data Docs home page, click the first result in the “Validation Results” list to view the Expectation Validation Result.

On the “Expectation Validation Result” page you will see a heading for “Table-Level Expectations” with a table of Expectations below it. Below that you will see a subheading for passenger_count with a table of Expectations below it.

In each table you can see the “Status”, “Expectation”, and “Observed Value” for each of the Expectations in the suite. Each row in the table will show the results of each Expectation (either a checkmark for success or an "X" for failed along with some other details).

How to edit an Expectation Suite

The Expectation Suite that was generated by the Data Assistant provides a good starting point, but you probably don't want to use all those Expectations in production or maybe you want to edit the configurations for some of your Expectations. Let’s edit our suite to include only the Expectations that we want for our project.

On the “Expectation Validation Result” page, click the button on the left side of the screen labeled “How to Edit This Suite”. Copy the CLI command that appears. Back in your terminal, stop the Jupyter Notebook server (Ctrl+C), paste the CLI command that you copied, and run it:

great_expectations suite edit getting_started_expectation_suite_taxi.demo

You will see the following two prompts:

How would you like to edit your Expectation Suite?
    1. Manually, without interacting with a sample batch of data (default)
    2. Interactively, with a sample batch of data
: 2

Select option 2, which will give us an easier way to edit our Expectation Suite.

A batch of data is required to edit the suite - let's help you to specify it.

Which data asset (accessible by data connector "default_inferred_data_connector_name") would you like to use?
    1. yellow_tripdata_sample_2019-01.csv
    2. yellow_tripdata_sample_2019-02.csv

Type [n] to see the next page or [p] for the previous. When you're ready to select an asset, enter the index.
: 1

Select option 1 to use the same dataset that was used to profile our data.

A new Jupyter Notebook will open up with the edit_getting_started_expectation_suite_taxi.demo.ipynb loaded again. This time the file will have the Expectations that were automatically generated by the Data Assistant and each one will be in their own cells. (You can close any other tabs that have notebooks or Data Docs in them.)

Go ahead and delete both of the Table Expectations (click the “Edit” → “Delete Cells” or click the In in front of the cell that you want to delete and type D twice).

Under the passenger_count Column Expectation, delete all of the Expectations except for the validator.expect_column_values_to_not_be_null() and validator.expect_column_values_to_be_between() Expectations.

Once you have edited your Expectations, follow the same process to save them and open the Data Docs:

Run all the cells in your notebook to save the updates to your Expectation Suite. This will open the Data Docs for your Expectation Suite again.

Troubleshooting Tip

The steps above should overwrite the pre-existing getting_started_expectation_suite_taxi Expectation Suite, but it didn’t do that for me. So I had to go into the great_expectations/expectations folder and delete the getting_started_expectation_suite_taxi folder and then run all the cells in the edit_getting_started_expectation_suite_taxi.demo.ipynb file again. Then it worked.

Now you should only see the Expectations for "values must never be null" and "values must be greater than or equal to 1 and less than or equal to 6" under the passenger_count column.

Expectation Gallery

You can see the full list of Expectations that are available in the Expectation Gallery.

Data Pipelines with Great Expectations | Step 2: Connect to data

Samuel Earl — Tue, 31 Jan 2023 02:41:43 +0000

Before you can validate data you first need to have data that needs to be validated. A datasource is simply the data at some stage in your pipeline that needs to be validated before it can move to the next stage in your pipeline.

While inside your gx-getting-started directory in your terminal, run the following command to create a new datasource:

great_expectations datasource new

The format of GX CLI commands

Most of the GX CLI commands follow this format:
great_expectations <noun> <verb>
For more details you can type one of the following commands in a terminal:
great_expectations --help
great_expectations <noun> --help
great_expectations <noun> <verb> --help

You will be prompted to select a type of datasource:

What data would you like Great Expectations to connect to?
    1. Files on a filesystem (for processing with Pandas or Spark)
    2. Relational database (SQL)
:1

Since we will be using the local data files in our data directory, select option 1 and press Enter.

The next prompt will ask you how you want to process your data files:

What are you processing your files with?
    1. Pandas
    2. PySpark
:1

Select option 1 and press Enter.

The final prompt for your datasource asks where your data is located:

Enter the path of the root directory where the data files are stored. If files are on local disk enter a path relative to your current working directory or an absolute path.
:data

Since we are working inside the gx-getting-started directory, we can type data as the path to our data directory and press Enter.

When you press Enter a Jupyter Notebook will open up in your default browser with a file named datasource_new.ipynb loaded inside. This file will allow you to configure your datasource.

File Location

You can find the datasource_new.ipynb file here: great_expectations/uncommitted/datasource_new.ipynb

Let’s change the name of our datasource. In the Jupyter Notebook, scroll down to the second code cell and change it as follows:

datasource_name = "getting_started_datasource"

Run all the cells in your notebook to save your datasource configs in the great_expectations/great_expectations.yml file. The great_expectations.yml file contains the main project configurations. If you open that file you should see a new entry under the datasources header with the datasource that you just configured.

How are notebook files related to GX project configs

The Jupyter Notebook files inside the uncommitted folder are basically scripts that update the configurations in other parts of the great_expectations folder. The notebook files inside the uncommitted folder are generated when you run the CLI commands. Once a notebook file is generated, GX will open it automatically for you in Jupyter Notebooks. You can change the settings in the notebook file and run all the cells in the notebook to update your configs.

Everything in the uncommitted folder is temporary and is not intended to be saved in version control. However, the rest of the contents in your great_expectations folder should be saved in version control. The great_expectations/.gitignore file already comes with an entry for uncommitted/.

Data Pipelines with Great Expectations | Step 1: Setup

Samuel Earl — Tue, 31 Jan 2023 02:41:22 +0000

GitHub Repo

Before we go any further, here is the finished project that contains the files used in this tutorial: gx-getting-started

Setup your folder structure and virtual environment

Create your folder structure

Create a folder called gx-getting-started.

VS Code Recommendation
I recommend opening your gx-getting-started folder in VS Code. GX uses Jupyter Notebooks to configure your project. For me it is helpful to understand where files are located and what they do. Jupypter Notebooks does not have a good file explorer, which makes it difficult for me to understand where files are located within my project. However, VS Code does have a good file explorer.

If you open your gx-getting-started folder in VS Code you can run any GX CLI commands from VS Code’s built-in terminal and look at the project files in VS Code's file explorer. This might help you understand the structure of GX projects better and it could help you make the connection between the scripts inside the Jupyter Notebook files and your GX project configs. More on that later.

Inside of the gx-getting-started folder create another folder called data and copy the two data files from this data folder.

Create an `environment.yml` file in your project root directory

You can create and activate virtual environments with conda. (See Managing environments.)

Create the following environment.yml file in your gx-getting-started directory:

# /environment.yml

name: gx-env
channels:
  - conda
  - conda-forge
dependencies:
  - python=3.9.5
  - great-expectations==0.15.46

Install Anaconda3

Go to Anaconda's website and install Anaconda for your operating system.
After Anaconda is installed, then make sure that Python3 and Pip3 are on your PATH variable. In a terminal run which python3 and then which pip3. Those commands should return a file path to your Anaconda installation.
conda is Anaconda's package manager. Make sure that your conda installation worked: conda --version. If that returns a version number, then you have installed Anaconda3 and conda correctly.

Create your virtual environment

In a terminal window cd into your project root directory and run:

conda env create -f environment.yml

This will install the virtual environment that is specified in your environment.yml file. If you see a prompt asking you to confirm before proceeding, type y and press Enter to continue creating the environment. Depending on your system configuration, it may take a while for the process to complete.

Verify that the new virtual environment was installed correctly

In your terminal type:

conda env list

You should see gx-env in that list, which is the name specified in your environment.yml file.

Activate your virtual environment

conda activate gx-env

Once your virtual environment has been activated, your command prompt should be prefixed with (gx-env). For example:

(gx-env) ~/gx-getting-started$

NOTE: If you need to deactivate or delete a virtual environment, look at the end of this post for instructions.

Setup a Great Expectations project

Check that Great Expectations is installed

GX should already be available in your virtual environment, so you don’t need to install it. But you can confirm that you have Great Expectations installed by running

great_expectations --version

This should output like this:

great_expectations, version 0.15.46

Create a Data Context

What is a Data Context? In web development, a project is a folder that contains all of the packages, config files, code, etc. to develop and run a web project. A Data Context in GX is similar. It contains all of the packages, config files, code, etc. to configure and run data validations in your pipeline.

To create a Data Context, open your terminal, make sure that you are inside your gx-getting-started folder, and run the following command:

great_expectations init

When you are asked “OK to proceed?” type y and press Enter.

This will create a subdirectory called great_expectations with a bunch of files and folders inside. I will show you some of those files and folders in this tutorial and explain their purpose.

Additional Info

How to deactivate a virtual environment

When you are done working on a particular project you can deactivate the virtual environment with:

conda deactivate

See Deactivating an environment.

How to delete a virtual environment

If you need to delete a virtual environment for any reason, this is how:

First make sure your environment is deactivated. Then run this command:

conda env remove --name gx-env

You can verify that your virtual environment has been deleted by running:

conda env list

You should not see your virtual environment listed.

Data Pipelines with Great Expectations | Introduction

Samuel Earl — Tue, 31 Jan 2023 02:41:10 +0000

My background is mostly in web development, but I am learning data engineering because I am interested in business intelligence and want to level-up my knowledge. One of the things that I have recently discovered is data validation within a data pipeline. Let me share a quick example from a previous job I had that illustrates the need for data validation in a data pipeline.

Example of a faulty data pipeline

We had a web app that allowed our clients to log into an account and view performance data in a dashboard. That dashboard was the main landing page that users would see after they logged in. The dashboard pulled its data from a datasource that was managed by a group of analysts. The analysts would pull data from a database, perform some analyses, and update the data that was used in the dashboard.

The problem was that the analysts would accidentally change the structure of the data sometimes and the dashboard in our web app would fail silently. When we finally discovered the failures, then we had to figure out who changed the data and try to get it fixed before it affected too many of our users. (Clearly we had some automation issues, but this kind of problem exists even with automated data pipelines that are well architected.)

We could have really benefited from using Great Expectations. Let me show you why.

NOTE: Instead of writing out “Great Expectations” I will abbreviate it as “GX” from now on in this tutorial.

What does GX do and where does it fit in a data pipeline?

This is an example of a simple architecture that uses GX:

Simple data pipeline example using GX

GX is used to validate data within a data pipeline. You can place GX Checkpoints at any stage in a pipeline where data needs to be checked for quality before it moves onto the next stage.

When using GX in a data pipeline, if the data passes validation, then it can move to the next stage in the pipeline. If the data does not pass validation, then we need to alert the proper stakeholders (e.g. data engineers) so they can fix the issues before end users are affected.

What will we be doing in this tutorial?

We are going to use some datasets from the NYC taxi data to demonstrate how GX works. Specifically, we are going to use two datasets: January 2019 and February 2019. Each dataset contains 10,000 records and each record represents one taxi ride with multiple columns of data (e.g. pick-up location, drop-off location, payment amount, number of passengers).

In our scenario, we know that the January data is clean and it matches what we expect. So we are going to use that dataset to create some validation rules. Then we are going to run the February dataset (i.e. batch) through our validation rules to see if the February data passes validation.

Sam's Notes | Data Science | How to setup a data science project with JupyterLab and Python

Samuel Earl — Thu, 15 Dec 2022 20:13:18 +0000

Install Anaconda3

Go to Anaconda's website and install Anaconda for your operating system.
After Anaconda is installed, then make sure that Python3 and Pip3 are on your PATH variable. In a terminal run which python3 and then which pip3. Those commands should return a file path to your Anaconda installation.
conda is Anaconda's package manager. Make sure that your conda installation worked: conda --version. If that returns a version number, then you have installed Anaconda3 and conda correctly.

Run JupyterLab inside a virtual environment

(RECOMMENDED)

Why JupyterLab instead of Jupyter Notebooks?
JupyterLab builds on Jupyter Notebooks and includes more features. In other words, JupyterLab is the next generation of Jupyter Notebooks.

Why use virtual environments?
Every data science project you do will require some combination of external libraries, sometimes with specific versions that differ from the specific versions you used for other projects. If you were to have a single Python installation, these libraries would conflict and cause you all sorts of problems.

The standard solution is to use virtual environments, which are sandboxed Python environments that maintain their own versions of Python libraries (and, depending on how you set up the environment, of Python itself).

As a matter of good discipline, you should always work in a virtual environment, and never using the "base" Python installation.

(Source: Data Science from Scratch, pages 37 and 41)

Step 1: Create an `environment.yml` file in your project root directory

You can create and activate virtual environments with conda. (See Managing environments.)

First, create an environment.yml file in your project root directory. Here is an example of an environment.yml file:

name: name-of-virtual-environment
channels:
  - conda
  - conda-forge
dependencies:
  - python=3.9.5
  - pip
  - python-graphviz
  - pip:
    - scikit-learn==0.24.0
    - pandas==1.2.0
    - seaborn==0.11.1
    - imbalanced-learn==0.7.0
    - numpy==1.19.5
    - matplotlib==3.3.3
    - kmodes==0.11.0
    - six==1.15.0
    - pydotplus==2.0.2
    - statsmodels==0.12.1
    - ipykernel==6.15.2
    - jupyterlab==3.4.5

NOTES:

Replace name-of-virtual-environment with the name that you want to give to your virtual environment.
Notice the ipykernel and jupyterlab packages at the end of the above list. I will give more details about those later.

This next environment.yml example is for web development, so you can skip this if you only want to know about data science.

Just as a reference, here is an environment.yml file that could be used with a Python backend for web development, if you are not using Docker as your development environment:

name: name-of-virtual-environment
dependencies:
  - python=3.6
  - pip=21.0
  # You need to list pip as a dependency and then list any
  # packages that need to be installed by pip after all the 
  # dependencies that are to be installed by conda.
  # https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment
  # https://stackoverflow.com/questions/61715343/solving-environment-failed-when-trying-to-set-up-python-virtual-envrironment
  - pip:
    - fastapi
    - uvicorn
    - aiofiles
    - plotly==4.14.3
    - pandas==1.1.5

Step 2: Create your virtual environment

In a terminal window cd into your project root directory and run:

conda env create -f environment.yml

This will install the virtual environment that is specified in your environment.yml file. In case you see a prompt asking you to confirm before proceeding, type y and press Enter to continue creating the environment. Depending on your system configuration, it may take a while for the process to complete.

Step 3: Verify that the new virtual environment was installed correctly

In your terminal type:

conda env list

You should see the name of your virtual environment in that list, which is the name specified in your environment.yml file.

Step 4: Activate your virtual environment

conda activate <name-of-virtual-environment>

NOTE: Replace <name-of-virtual-environment> with the name that you gave to your virtual environment.

Once your virtual environment has been activated, your command prompt should be prefixed with the name of your virtual environment. For example:

(name-of-virtual-environment) ~/data-science-projects$

Step 5: Run JupyterLab inside your virtual environment

Run the following command inside your activated virtual environment:

jupyter-lab

For example:

(name-of-virtual-environment) ~/data-science-projects$ jupyter-lab

When you run jupyter-lab the JupyterLab server will run and an instance of JupyterLab will open up in a browser window. The following are a couple of indicators that you are running JupyterLab from inside a virtual environment:

The output in the terminal where you ran jupyter-lab should have the following two lines (which indicate that JupyterLab is being served from the /path/to/anaconda3/envs/ directory, which is where the virtual environments are stored):

[I 2022-09-07 09:31:23.835 LabApp] JupyterLab extension loaded from /path/to/anaconda3/envs/<name-of-virtual-environment>/lib/python3.9/site-packages/jupyterlab
[I 2022-09-07 09:31:23.835 LabApp] JupyterLab application directory is /path/to/anaconda3/envs/<name-of-virtual-environment>/share/jupyter/lab

When JupyterLab first loads up in your browser, under the "Launcher" tab you will see options for "Python 3 (ipykernel)" under the "Notebook" and "Console" headings.

So what is ipykernel?

The Jupyter Notebook and other frontends automatically ensure that the IPython kernel is available. However, if you want to use a kernel with a different version of Python, or in a virtualenv or conda environment, you’ll need to install that manually. (Source Installing the IPython kernel)

Since we are using JupyterLab, instead of Jupyter Notebooks, we are able to run it inside of a virtual environment in a pretty simple and automated way. I don't think that is possible with Jupyter Notebooks, but I could be wrong.

If you need to use Jupyter Notebooks, then I think you will have to install ipykernel and run Jupyter Notebooks inside a virtual environment a different (and more manual) way. The Preface of the book "Data Science for Marketing Analytics" has a good explanation of the process you would have to use.

How to deactivate a virtual environment

When you are done working on a particular project you can deactivate the virtual environment with:

conda deactivate

See Deactivating an environment

How to delete a virtual environment

First make sure your environment is deactivated. Then run this command:

conda env remove --name <name-of-virtual-environment>

Make sure to replace <name-of-virtual-environment> with the name of the virtual environment that you want to delete.

You can verify that your virtual environment has been deleted by running:

conda env list

You should not see your virtual environment listed.

How to install JupyterLab and run it outside of a virtual environment

(NOT RECOMMENDED)

I do not recommend running JupyterLab outside of a virtual environment. However, I wanted to share an example of how to setup a data science project that does not use a virtual environment to show how it compares to the proper way (above).

Step 1: Install JupyterLab with `pip`:

pip install jupyterlab

Note: If you install JupyterLab with conda or mamba, it is recommended to use the conda-forge channel.

Step 2: Check that JupyterLab installed correctly:

jupyter-lab --version

Step 3: Once installed, launch JupyterLab with:

jupyter-lab

Sam's Notes | Data Science | Intro

Samuel Earl — Thu, 15 Dec 2022 20:13:04 +0000

When I learn something new I often write reference guides for myself and save them in Google Docs. However, instead of keeping my reference guides to myself I thought I would share some things that I have learned and that have helped me. That's what my "Sam's Notes" posts are.

A Basic Chart - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 4

Samuel Earl — Wed, 13 Jul 2022 21:32:43 +0000

We are going to add a D3.js bar chart that will interact with our geospacial data.

The Building a Geospacial App tutorial uses a data visualization library called React-Vis. Apparently React and D3 do not work together very easily, so you will see some data visualization libraries that combine React and D3 to make it easier for developers to incorporate D3 into a React app. Since we are using Svelte we don't have that problem and we can use D3 (along with all of its power and customizability) directly.

Svelte and D3 work really well together. However, to optimize the relationship between Svelte and D3 it is important to understand some basic concepts about these two great pieces of software. Please read this (short) post titled Making Visualizations Literally with Svelte & D3 and then come back here.

Update the `processData()` function

In your index.svelte file, find the processData() function and update it to look like this:

function processData() {
  let data = taxiData.reduce(
    (accu, curr) => {
      let pickupHour = new Date(curr.pickup_datetime).getUTCHours();
      let dropoffHour = new Date(curr.dropoff_datetime).getUTCHours();

      let pickupLongitude = Number(curr.pickup_longitude);
      let pickupLatitude = Number(curr.pickup_latitude);

      if (!isNaN(pickupLongitude) && !isNaN(pickupLatitude)) {
        accu.points.push({
          position: [pickupLongitude, pickupLatitude],
          hour: pickupHour,
          pickup: true
        });
      }

      let dropoffLongitude = Number(curr.dropoff_longitude);
      let dropoffLatitude = Number(curr.dropoff_latitude);

      if (!isNaN(dropoffLongitude) && !isNaN(dropoffLatitude)) {
        accu.points.push({
          position: [dropoffLongitude, dropoffLatitude],
          hour: dropoffHour,
          pickup: false
        });
      }

      let prevPickups = accu.pickupObj[pickupHour] || 0;
      let prevDropoffs = accu.dropoffObj[dropoffHour] || 0;

      accu.pickupObj[pickupHour] = prevPickups + 1;
      accu.dropoffObj[dropoffHour] = prevDropoffs + 1;

      return accu;
    },
    {
      points: [],
      pickupObj: {},
      dropoffObj: {}
    }
  );

  data.pickups = Object.entries(data.pickupObj).map(([hour, count]) => {
    return { hour: Number(hour), x: Number(hour) + 0.5, y: count };
  });
  data.dropoffs = Object.entries(data.dropoffObj).map(([hour, count]) => {
    return { hour: Number(hour), x: Number(hour) + 0.5, y: count };
  });

  taxiDataset = data;
}

Now find the pointsArray declaration and replace this:

let pointsArray = [];

...with this:

let taxiDataset = {
  points: [],
  pickups: [],
  pickupObj: {},
  dropoffs: [],
  dropoffObj: {}
};

Now find the renderLayers reactive property:

$: renderLayers({ data: pointsArray, settings: settings });

...and update it to look like this:

$: renderLayers({ data: taxiDataset.points, settings: settings });

It looks like there is a lot going on in the processData() function. It is just formatting the taxiData into an object that is shaped like the taxiDataset object that you used to replace the pointsArray variable.

The points property holds the scatterplot data, which will show how the distance and time of the trips are correlated.
The pickupObj property holds the number of pickups by hour.
The pickups property holds the x and y values of the pickup bars in the bar chart.
The dropoffObj property holds the number of dropoffs by hour.
The dropoffs property holds the x and y values of the dropoff bars in the bar chart.

Create the bar chart container

Create a new file inside your /src/components directory and name it BarChart.svelte. Paste the following code inside:

<div class="bar-chart">
  <h2>Pickups by hours</h2>
</div>

<style>
  .bar-chart {
    position: absolute;
    left: 10px;
    bottom: 10px;
    z-index: 100;
    width: 500px;
    height: 210px;
    padding: 10px;
    background-color: white;
    border-radius: 3;
    box-shadow: 0 1px 2px rgba(0, 0, 0, 0.1);
    font-size: 12px;
    line-height: 1.833;
  }
</style>

Now in your index.svelte file, import that BarChart component:

import BarChart from "$/components/BarChart.svelte";

...and include it inside the <div> tag in your HTML as the last element/component in the <div> tag, like this:

<div class="deck-container">
  {#if hover.hoveredObject}
    <div class="tooltip" style:transform={`translate(${hover.x}px, ${hover.y}px)`}>
      <div>{hover.label}</div>
    </div>
  {/if}
  <MapStylePicker currentStyle={mapStyle} on:change={handleStyleChange}/>
  <LayerControls {settings} controls={CONTROLS} on:layerSettingsChange={updateLayerSettings} />
  <div id="map" bind:this={mapElement}></div>
  <canvas id="deck-canvas" bind:this={canvasElement}></canvas>
  <BarChart />
</div>

Make sure to save your files and refresh your browser, if necessary, to see the box for your bar chart.

Create the bar chart

First, we are going to create a simple bar chart of pickups by hour.

To do this, we are going to use the taxiDataset.pickups variable we prepared above. This is an array of objects of the form {hour: 0, x: 0.5, y: 246}. Each x is going to be the x-coordinate of the corresponding bar and each y is going to be y-coordinate of the corresponding bar for the pickups we want to plot.

We are going to create the bar chart by coding SVG elements, which are similar to HTML elements, directly into the HTML portion of our BarChart.svelte component.

In your BarChart.svelte file, update the HTML as follows:

TODO: Finish the rest of this tutorial.

Hexagon Overlay - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 3

Samuel Earl — Wed, 13 Jul 2022 21:32:26 +0000

ScatterplotLayer can plot raw points, but to visualize distribution of these points, we need a layer that can aggregate points into a geo grid. HexagonLayer and GridLayer are both aggregation layers that can visualize a distribution heatmap from raw points.

Update the Control Panel

We're going to upgrade our control panel so we can switch from the scatterplot layer to the hexagon layer. Let's do that now, so you can see the changes on the hexagon layer as we build it.

Update the CONTROLS constant in your index.svelte file to include the hexagon controls. CONTROLS should now look like this:

const CONTROLS = {
  showHexagon: {
    displayName: "Show Hexagon",
    type: "boolean",
    value: true
  },
  radius: {
    displayName: "Hexagon Radius",
    type: "range",
    value: 100,
    step: 50,
    min: 50,
    max: 1000
  },
  coverage: {
    displayName: "Hexagon Coverage",
    type: "range",
    value: 1,
    step: 0.1,
    min: 0,
    max: 1
  },
  upperPercentile: {
    displayName: "Hexagon Upper Percentile",
    type: "range",
    value: 100,
    step: 0.1,
    min: 80,
    max: 100
  },
  showScatterplot: {
    displayName: "Show Scatterplot",
    type: "boolean",
    value: true
  },
  radiusScale: {
    displayName: "Scatterplot Radius",
    type: "range",
    value: 10,
    step: 1,
    min: 1,
    max: 100
  },
};

Add Constants for Hexagon Layer

Deck.gl shallow compares layer props to decide how to update attributes. To avoid unnecessary re-calculations, we define constant params outside of the createDataLayers() function.

You can add these constants below your other variable declarations in index.svelte and then we will pass them into the HexagonLayer later on.

const HEATMAP_COLORS = [
  [255, 255, 204],
  [199, 233, 180],
  [127, 205, 187],
  [65, 182, 196],
  [44, 127, 184],
  [37, 52, 148]
];

const LIGHT_SETTINGS = {
  lightsPosition: [-73.8, 40.5, 8000, -74.2, 40.9, 8000],
  ambientRatio: 0.4,
  diffuseRatio: 0.6,
  specularRatio: 0.2,
  lightsStrength: [0.8, 0.0, 0.8, 0.0],
  numberOfLights: 2
};

const ELEVATION_RANGE = [0, 1000];

Add the Hexagon Layer

Now we only need to render the HexagonLayer. Import HexagonLayer inside your <script> tag:

import { HexagonLayer } from "@deck.gl/aggregation-layers";

And add HexagonLayer to the createDataLaters() function:

function createDataLayers(props) {
  const { data, settings } = props;
  return [
    settings.showScatterplot && new ScatterplotLayer({
      id: "scatterplot",
      getPosition: d => d.position,
      getFillColor: d => d.pickup ? PICKUP_COLOR : DROPOFF_COLOR,
      getRadius: d => 5,
      opacity: 0.5,
      pickable: true,
      radiusMinPixels: 0.25,
      radiusMaxPixels: 30,
      data,
      onHover: (hoverProps) => handleHover("scatterplotLayer", hoverProps),
      ...settings
    }),
    settings.showHexagon && new HexagonLayer({
      id: "heatmap",
      colorRange: HEATMAP_COLORS,
      elevationRange: ELEVATION_RANGE,
      elevationScale: 5,
      extruded: true,
      getPosition: d => d.position,
      lightSettings: LIGHT_SETTINGS,
      opacity: 0.8,
      pickable: true,
      data,
      onHover: (hoverProps) => handleHover("hexagonLayer", hoverProps),
      ...settings
    })
  ];
}

And update the handleHover() function to include the hexagon layer type:

function handleHover(layerType, hoverProps) {
  let label;
  if (layerType === "scatterplotLayer") {
    label = hoverProps.object ? (hoverProps.object.pickup ? "Pickup" : "Dropoff") : null;
  }
  if (layerType === "hexagonLayer") {
    let count = hoverProps.object ? hoverProps.object.points.length : null;
    label = `${count} pickups or dropoffs here`;
  }
  // Set the coordinates for the tooltip.
  hover.x = hoverProps.x;
  hover.y = hoverProps.y - 20;
  hover.hoveredObject = hoverProps.object;
  hover.label = label;
}

Now you should see the HexagonLayer along with controls for displaying the data.

In the next tutorial we are going to use D3.js to add a bar chart that will interact with the geospacial data on our map.

Scatterplot Overlay - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 2

Samuel Earl — Wed, 13 Jul 2022 21:32:09 +0000

Add a Scatterplot Overlay with Deck.gl

As cool as having a map is, an empty map is not that useful. Let's see if we can add a Scatterplot overlay with the Taxi data set to the map we created in the previous post.

Deck.gl is a WebGL overlay suite for JavaScript, providing a set of highly performant data visualization overlays.

Deck.gl comes with several prepackaged layers that we can use, in conjunction with our map, to display geospatial data. The simplest one is the ScatterplotLayer, which we will use.

An introduction to Deck.gl

Deck.gl allows you to visualize stacked layers of data. Each layer can be of a different type (we'll see two: scatterplots and hexagons, but there are many more) and can have millions of data points. In the finished visualization, the layers appear on top of each other, and they support interactions like pan, zoom, rotate, as well as mouse hover, clicks, etc.

Deck.gl doesn't have to work with a map but it was designed with geospatial applications in mind. Like the map we designed in the previous post, Deck.gl layers have the same concept of view state and user interactions.

There are a couple of ways to make Deck.gl work with mapbox-gl, but this is how we are going to work with them:

We will have a deck.gl <canvas> element and a mapbox-gl <div> element in the HTML. Then we will make sure that the deck.gl canvas stays on top of the mapbox element. This will allow us to use a static map (i.e. one where the interactive option is set to false), which will get all the view state information from Deck.gl.

Here is the updated code, which adds Deck.gl to our code and causes the map and Deck.gl to work together:

<MapStylePicker currentStyle={mapStyle} on:change={handleStyleChange}/>
<div class="deck-container">
  <div id="map" bind:this={mapElement}></div>
  <canvas id="deck-canvas" bind:this={canvasElement}></canvas>
</div>

<script>
  import { onMount } from "svelte";
  import mapboxgl from "mapbox-gl";
  import { Deck } from "@deck.gl/core";
  import MapStylePicker from "$/components/MapStylePicker.svelte";

  let mapElement;
  let canvasElement;
  let map = null;
  let deck = null;
  let accessToken = import.meta.env.VITE_MAPBOX_API_ACCESS_TOKEN;
  let mapStyle = "mapbox://styles/mapbox/light-v9";
  let viewState = {
    longitude: -118.2443409,
    latitude: 34.0544779,
    zoom: 2,
    pitch: 0,
    bearing: 0,
  };

  onMount(() => {
    createMap();
    createDeck();
  });

  // Create the map.
  function createMap() {
    map = new mapboxgl.Map({
      accessToken: accessToken,
      container: mapElement,
      interactive: false,
      style: mapStyle,
      center: [viewState.longitude, viewState.latitude],
      zoom: viewState.zoom,
      pitch: viewState.pitch,
      bearing: viewState.bearing,
    });
  }

  /**
   * Use the `setStyle` function from `mapbox-gl` to update the map style:
   * https://docs.mapbox.com/mapbox-gl-js/example/setstyle/
   */
  function handleStyleChange(event) {
    map.setStyle(event.target.value);
  }

  // Create the deck.gl instance.
  function createDeck() {
    deck = new Deck({
      canvas: canvasElement,
      width: "100%",
      height: "100%",
      initialViewState: viewState,
      controller: true,
      // Change the map's viewState whenever the view state of deck.gl changes.
      onViewStateChange: ({ viewState }) => {
        map.jumpTo({
          center: [viewState.longitude, viewState.latitude],
          zoom: viewState.zoom,
          bearing: viewState.bearing,
          pitch: viewState.pitch,
        });
      },
    });
  }
</script>

<style>
  .deck-container {
    width: 100%;
    height: 100%;
    position: relative;
  }

  #map {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    background: #e5e9ec;
    overflow: hidden;
  }

  #deck-canvas {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
  }
</style>

This has all the same functionality as our previous version, but Deck.gl has been incorporated. We added the following things:

A couple more HTML elements along with CSS styles for those elements.
The @deck.gl/core library.
A couple of variables for the canvasElement and the deck instance.
A createDeck() function that will create the Deck.gl instance, which gets called during the onMount hook.

The finished code

I am going to show you the finished code for this post and then explain some things about the finished code below.

Add Data

Create a new folder called data in your src directory and create a new file inside your data folder called taxi.js. The taxi.js file will contain location data for each time a taxi either picked up or dropped off passengers on Manhattan Island.

Then go to https://github.com/uber-common/vis-academy/blob/master/src/demos/data/taxi.js, click the Download button, and copy/paste the raw data into your taxi.js file. At the bottom of the taxi.js file you will need to change the module.exports = dataRows.map... statement to export default dataRows.map....

Once that is done, import the taxi data into your index.svelte component:

import taxiData from "$/data/taxi.js";

Now we need to process this data into a usable format. Since we are only going to be working with a ScatterplotLayer for now, we only care about the latitude, longitude, and another bit of data called pickup, which is used for coloring the dots.

We add a let points = []; variable and a processData() function and call it when the component mounts to process the data.

Add Deck.gl Layers

Import the ScatterplotLayer:

import { ScatterplotLayer } from "@deck.gl/layers";

Then add the following code:

// Whenever the layer data is changed and new layers are created, then rerender the layers.
$: renderLayers({ data: pointsArray });

// Use the `deck.setProps()` method to set the layers in deck.gl.
// See https://deck.gl/docs/api-reference/core/deck#layers.
function renderLayers(props) {
  // If `deck` is null then return early to prevent errors.
  if (!deck) return;

  deck.setProps({
    layers: createDataLayers(props)
  });
}

function createDataLayers(props) {
  const {data} = props;
  return new ScatterplotLayer({
    id: "scatterplot",
    getPosition: d => d.position,
    getFillColor: d => [0, 128, 255],
    getRadius: d => 5,
    opacity: 0.5,
    pickable: true,
    radiusMinPixels: 0.25,
    radiusMaxPixels: 30,
    data
  });
}

The $: renderLayers({ data: pointsArray }); reactive property is responsible for watching the pointsArray variable and calling the renderLayers() function whenever the data in the pointsArray variable changes.

The renderLayers() function is responsible for taking the pointsArray data (which is passed as the props parameter) and getting the updated scatterplot layer from createDataLayers(). That updated scatterplot layer is then updated in the browser by calling deck.setProps().

The createDataLayers() function is responsible for creating the updated scatterplot layer and returning it.

Now you should see a working Deck.gl overlay that displays the taxi data as a scatterplot overlay. You should see blue dots laid out across the top of Manhattan Island. The data points are a little small, so you might need to zoom in to see them.

Let's go over just a few properties in the new ScatterplotLayer() code above:

data {Array}
The data for the layer. In this case, it's our Taxi data set.

getPosition {Function}
Function that gets called for each data point. This should return an array of [longitude, latitude].

getFillColor, getRadius {Function}
These also get called for each data point and they return the color and radius, respectively, for each point.

pickable {Bool}
This indicates whether this layer will be interactive.

Polish

Dynamic Color

Right now, our getFillColor accessor returns a constant, but we can have color change depending on the data we pass. Let's display the magenta data points where a taxi picked up a passenger and orange data points where a taxi dropped off a passenger:

// Variable declarations:
const PICKUP_COLOR = [114, 19, 108];
const DROPOFF_COLOR = [243, 185, 72];

...

// Inside the createDataLayers() function:
getFillColor: d => d.pickup ? PICKUP_COLOR : DROPOFF_COLOR,

Control Panel

We are going to create two control elements that will allow us to (1) change the size of the dots in our scatterplot and (2) show or hide the scatterplot.

First, create a LayerControls.svelte component inside the src/components directory and paste this code inside:

<div class="layer-controls">
  <h4>Layer Controls</h4>
  <hr>
  <div class="controls-container">
    {#each Object.keys(settings) as key (key)}
      {#if controls[key] && controls[key].type}
        {#if controls[key].type === "range"}
          <div class="range-input-group">
            <label>{controls[key].displayName}</label>
            <div style="display:inline-block; float:right;">
              {settings[key]}
            </div>
            <div>
              <input
                type="range"
                id={key}
                min={controls[key].min}
                max={controls[key].max}
                step={controls[key].step}
                value={settings[key]}
                on:change={(event) => handleValueChange(key, Number(event.target.value))}
              />            
            </div>
          </div>
        {:else if controls[key].type === "boolean"}
          <div class="checkbox-input-group">
            <label>{controls[key].displayName}</label>
            <input
              type="checkbox"
              id={key}
              checked={settings[key]}
              on:change={(event) => handleValueChange(key, event.target.checked)}
            />
          </div>
        {/if}
      {/if}
      {#if controls[key].displayName === "Hexagon Upper Percentile"}
        <hr>
      {/if}
    {/each}
  </div>
</div>

<script>
  import { createEventDispatcher } from "svelte";

  export let settings;
  export let controls;

  const dispatch = createEventDispatcher();
  let max = 100;

  function handleValueChange(settingName, newValue) {
    // Only update if we have a confirmed change.
    if (settings[settingName] !== newValue) {
      // Create a new object so that shallow-equal detects a change.
      const newSettings = {
        ...settings,
        [settingName]: newValue
      };

      dispatch("layerSettingsChange", newSettings);
    }
  }
</script>

<style>
  .layer-controls {
    border: 1px solid lightgray;
    box-shadow: 2px 2px 2px gray;
    font-family: sans-serif;
    font-size: 12px;
    line-height: 1.833;
    width: 220px;
    position: absolute;
    top: 10px;
    right: 10px;
    padding: 20px;
    z-index: 100;
    background: white;
  }

  h4 {
    margin: 0;
    line-height: 1rem;
  }

  hr {
    border: none;
    border-top: 1px solid gray;
  }

  .checkbox-input-group {
    display: flex;
    justify-content: space-between;
    align-items: center;
    margin-bottom: 5px;
  }
</style>

In the index.svelte file, import the LayerControls component:

import LayerControls from "$/components/LayerControls.svelte";

...and add the LayerControls component to your HTML:

<div class="deck-container">
  <MapStylePicker currentStyle={mapStyle} on:change={handleStyleChange}/>
  <LayerControls {settings} controls={CONTROLS} on:layerSettingsChange={updateLayerSettings} />
  <div id="map" bind:this={mapElement}></div>
  <canvas id="deck-canvas" bind:this={canvasElement}></canvas>
</div>

Now add the following two variables below your other variable declarations and the updateLayerSettings() function below your other functions inside your <script> tag:

const CONTROLS = {
  radiusScale: {
    displayName: "Scatterplot Radius",
    type: "range",
    value: 10,
    step: 1,
    min: 1,
    max: 100
  },
  showScatterplot: {
    displayName: "Show Scatterplot",
    type: "boolean",
    value: true
  },
};
// The reduce function will return an initial value of `{radiusScale: 30, showScatterplot: true}`.
// Each time a user changes a layer control, the properties in this `settings` object will be updated and the layer data's appearance will be updated.
let settings = Object.keys(CONTROLS).reduce(
  (accu, key) => ({
    ...accu,
    [key]: CONTROLS[key].value
  }),
  {}
);

function updateLayerSettings(event) {
  settings = event.detail;
}

Now update the renderLayers() reactive property and the createDataLayers() function so the layer data will be updated when a user changes any of the control settings:

$: renderLayers({ data: pointsArray, settings: settings });

...

function createDataLayers(props) {
  const { data, settings } = props;
  return [
    settings.showScatterplot && new ScatterplotLayer({
      id: "scatterplot",
      getPosition: d => d.position,
      getFillColor: d => d.pickup ? PICKUP_COLOR : DROPOFF_COLOR,
      getRadius: d => 5,
      opacity: 0.5,
      pickable: true,
      radiusMinPixels: 0.25,
      radiusMaxPixels: 30,
      data,
      ...settings
    })
  ];
}

Now, you should have a control panel that will allow you to control display aspects of your scatterplot layer.

Mouseover Interaction

We are going to add a mouseover event to our scatterplot that will display a small tooltip with either the term "Pickup" or "Dropoff" for each data point.

These steps are similar to the control panel steps above, but the user events flow in the opposite direction. Instead of capturing a user event in HTML components and passing it to the WebGL layer, we capture a user event in the WebGL layer and pass it to HTML elements.

All of this code will be in the index.svelte file. First, add the tooltip code to the HTML:

<div class="deck-container">
  {#if hover.hoveredObject}
    <div class="tooltip" style:transform={`translate(${hover.x}px, ${hover.y}px)`}>
      <div>{hover.label}</div>
    </div>
  {/if}
  <MapStylePicker currentStyle={mapStyle} on:change={handleStyleChange}/>
  <LayerControls {settings} controls={CONTROLS} on:layerSettingsChange={updateLayerSettings} />
  <div id="map" bind:this={mapElement}></div>
  <canvas id="deck-canvas" bind:this={canvasElement}></canvas>
</div>

Then add the CSS for the tooltip:

.tooltip {
  position: absolute;
  padding: 4px;
  background: rgba(0, 0, 0, 0.8);
  color: #fff;
  max-width: 300px;
  font-size: 10px;
  z-index: 9;
  pointer-events: none;
}

Add the following variable below your other variable declarations:

let hover = {
  x: 0,
  y: 0,
  hoveredObject: null
};

Now we need a way to capture an onHover event when a user hovers over a data point in our scatterplot layer. In the createDataLayers() function, add the onHover listener like I have done in the code below:

function createDataLayers(props) {
  const { data, settings } = props;
  return [
    settings.showScatterplot && new ScatterplotLayer({
      id: "scatterplot",
      getPosition: d => d.position,
      getFillColor: d => d.pickup ? PICKUP_COLOR : DROPOFF_COLOR,
      getRadius: d => 5,
      opacity: 0.5,
      pickable: true,
      radiusMinPixels: 0.25,
      radiusMaxPixels: 30,
      data,
      onHover: (hoverProps) => handleHover("scatterplotLayer", hoverProps),
      ...settings
    })
  ];
}

...and add a handleHover() function at the bottom of the <script> tag:

function handleHover(layerType, hoverProps) {
  let label;
  if (layerType === "scatterplotLayer") {
    label = hoverProps.object ? (hoverProps.object.pickup ? "Pickup" : "Dropoff") : null;
  }
  // Set the coordinates for the tooltip.
  hover.x = hoverProps.x;
  hover.y = hoverProps.y - 20;
  hover.hoveredObject = hoverProps.object;
  hover.label = label;
}

That's it! You should now see a tooltip appear next to any data points that you hover over.

In the next post you will see how to create a layer that will show a frequency distribution of the data points.

Start with a Map - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 1

Samuel Earl — Wed, 13 Jul 2022 21:31:00 +0000

The tutorial titled Building a Geospacial App from Vis Academy is written using React. I wanted to use Deck.gl in a Svelte app (specifically SvelteKit), so I am rewriting that tutorial using SvelteKit. I also borrowed some ideas from this tutorial: Large Scale Geospatial Visualization with Deck.gl, Mapbox-gl and Vue.js. Here it goes.

What is Deck.gl?

Deck.gl is an opensource visualization framework created by Uber. It allows you to create things like maps with data layers over the top of the maps. Deck.gl is built to handle large datasets while providing high-performance visualizations on the web. Deck.gl is part of the vis.gl opensource visualization framework suite.

Setup

SvelteKit

Create your SvelteKit project following the instructions on their website: SvelteKit.

In your root folder create a .env file and add this entry:

VITE_MAPBOX_API_ACCESS_TOKEN=you will paste your Mapbox API token here

Get a Mapbox API token

Mapbox GL JS is a client-side JavaScript library for building web maps and web applications with Mapbox's modern mapping technology. You can reference the API documentation here. Mapbox also has a pretty extensive examples page here where you can learn to do a lot of cool stuff.

You need a free Mapbox token in order to get the map to load. Head over to Mapbox and create an account (or sign in if you already have an account). Once you are signed in, click the "Account" navigation link. You should see a section in your account labelled "Default public token". Copy that token and paste it in your .env file.

Create the Map

The mapbox-gl npm package is the MapboxGL mapping library. This package makes it super easy to drop a mapping component into your application. Install mapbox-gl with this command:

npm install mapbox-gl

I will show you the map code first and then I will explain it afterwards.

// src/routes/index.svelte

<div id="map" bind:this={mapElement}></div>

<script>
  import { onMount } from "svelte";
  import mapboxgl from "mapbox-gl";

  let mapElement;
  let map = null;
  let accessToken = import.meta.env.VITE_MAPBOX_API_ACCESS_TOKEN;
  let mapStyle = "mapbox://styles/mapbox/light-v9";
  let viewState = {
    longitude: -118.2443409,
    latitude: 34.0544779,
    zoom: 2,
    pitch: 0,
    bearing: 0,
  };

  onMount(() => {
    createMap();
  });

  function createMap() {
    map = new mapboxgl.Map({
      accessToken: accessToken,
      container: mapElement,
      interactive: true,
      style: mapStyle,
      center: [viewState.longitude, viewState.latitude],
      zoom: viewState.zoom,
      pitch: viewState.pitch,
      bearing: viewState.bearing,
    });
  }
</script>

<style>
  #map {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    background: #e5e9ec;
    overflow: hidden;
  }
</style>

In the HTML
As you can see we created an element that will be used to display our map: <div id="map" bind:this={mapElement}></div>.

In the <script> tag
There is nothing too complex here. We simply set some configuration variables and wrote a function that creates a new map with those configuration variables. You can read through the code to get an idea of what it all does.

In the <style> tag
These styles are simply setting the position and size of the map that is rendered in the browser.

Changing Map Styles

I will show you how to add a select box that will be populated with some additional styles and which can be used to change the appearance of the map.

Create a components folder inside your src directory. Then create a MapStylePicker.svelte file inside your components folder and paste this code inside:

// src/components/MapStylePicker.svelte

<select
  class="map-style-picker"
  bind:value={currentStyle}
  on:change
>
  {#each MAPBOX_DEFAULT_MAPSTYLES as style (style.value)}
    <option value={style.value}>
      {style.label}
    </option>
  {/each}
</select>

<script>
  export let currentStyle;

  const MAPBOX_DEFAULT_MAPSTYLES = [
    { label: "Streets V10", value: "mapbox://styles/mapbox/streets-v10" },
    { label: "Outdoors V10", value: "mapbox://styles/mapbox/outdoors-v10" },
    { label: "Light V9", value: "mapbox://styles/mapbox/light-v9" },
    { label: "Dark V9", value: "mapbox://styles/mapbox/dark-v9" },
    { label: "Satellite V9", value: "mapbox://styles/mapbox/satellite-v9" },
    {
      label: "Satellite Streets V10",
      value: "mapbox://styles/mapbox/satellite-streets-v10"
    },
    {
      label: "Navigation Preview Day V4",
      value: "mapbox://styles/mapbox/navigation-preview-day-v4"
    },
    {
      label: "Navitation Preview Night V4",
      value: "mapbox://styles/mapbox/navigation-preview-night-v4"
    },
    {
      label: "Navigation Guidance Day V4",
      value: "mapbox://styles/mapbox/navigation-guidance-day-v4"
    },
    {
      label: "Navigation Guidance Night V4",
      value: "mapbox://styles/mapbox/navigation-guidance-night-v4"
    }
  ];
</script>

<style>
  .map-style-picker {
    background-color: #fff;
    position: absolute;
    top: 20px;
    left: 20px;
    z-index: 100;
  }
</style>

The MapStylePicker component is just a <select> element with some style options hard coded inside.

Now import MapStylePicker into your index file. Here is the updated code:

<MapStylePicker currentStyle={mapStyle} on:change={handleStyleChange}/>
<div id="map" bind:this={mapElement}></div>

<script>
  import { onMount } from "svelte";
  import mapboxgl from "mapbox-gl";
  import MapStylePicker from "$/components/MapStylePicker.svelte";

  let mapElement;
  let map = null;
  let accessToken = import.meta.env.VITE_MAPBOX_API_ACCESS_TOKEN;
  let mapStyle = "mapbox://styles/mapbox/light-v9";
  let viewState = {
    longitude: -118.2443409,
    latitude: 34.0544779,
    zoom: 2,
    pitch: 0,
    bearing: 0,
  };

  onMount(() => {
    createMap();
  });

  function createMap() {
    map = new mapboxgl.Map({
      accessToken: accessToken,
      container: mapElement,
      interactive: true,
      style: mapStyle,
      center: [viewState.longitude, viewState.latitude],
      zoom: viewState.zoom,
      pitch: viewState.pitch,
      bearing: viewState.bearing,
    });
  }

  function handleStyleChange(event) {
    map.setStyle(event.target.value);
  }
</script>

<style>
  #map {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    background: #e5e9ec;
    overflow: hidden;
  }
</style>

The main thing to notice here is the handleStyleChange() function. It uses the setStyle function from mapbox-gl to update the map style. (See https://docs.mapbox.com/mapbox-gl-js/example/setstyle/.)

Now you should be able to change the styles of the map.

If you feel creative, you can also create your own styles in Mapbox Studio and put their url (starting with http://api.mapbox.com/styles/) in the state instead.

`onViewportChange` Callback

The "Building a Geospacial App" tutorial has a section that shows how you can add an onViewportChange callback to the map to enable user interactions (e.g. panning, zooming, rotating, etc). However, that interactivity code appears to be enabled by default now (or at least in non-React environments it is enabled by default). So I am going to skip to the next section.

Adding Polish

This section of the "Building a Geospacial App" tutorial also seems unnecessary. That tutorial specifies the width and height of the map with window.innerWidth and window.innerHeight, which would require a callback in the JavaScript code to update the dimensions of the map. But I am using CSS to set the dimensions of the map, so there is no need for a callback function.

That's it for this part of the tutorial. In the next tutorial I will show you how to add a data layer on top of your map.

Forem: Samuel Earl

Data Pipelines with Great Expectations | Conclusion

Data Pipelines with Great Expectations | Step 4: Validate data

Set up a Checkpoint

Inspect the Validation Results for your Checkpoint

Data Pipelines with Great Expectations | Step 3: Create Expectations

Creating Expectations in Jupyter Notebooks

Viewing Expectations in Data Docs

How to edit an Expectation Suite

Expectation Gallery

Data Pipelines with Great Expectations | Step 2: Connect to data

Data Pipelines with Great Expectations | Step 1: Setup

GitHub Repo

Setup your folder structure and virtual environment

Create your folder structure

Create an environment.yml file in your project root directory

Install Anaconda3

Create your virtual environment

Verify that the new virtual environment was installed correctly

Activate your virtual environment

Setup a Great Expectations project

Check that Great Expectations is installed

Create a Data Context

Additional Info

How to deactivate a virtual environment

How to delete a virtual environment

Data Pipelines with Great Expectations | Introduction

What does GX do and where does it fit in a data pipeline?

What will we be doing in this tutorial?

Sam's Notes | Data Science | How to setup a data science project with JupyterLab and Python

Install Anaconda3

Run JupyterLab inside a virtual environment

(RECOMMENDED)

Step 1: Create an environment.yml file in your project root directory

Step 2: Create your virtual environment

Step 3: Verify that the new virtual environment was installed correctly

Step 4: Activate your virtual environment

Step 5: Run JupyterLab inside your virtual environment

How to deactivate a virtual environment

How to delete a virtual environment

How to install JupyterLab and run it outside of a virtual environment

(NOT RECOMMENDED)

Step 1: Install JupyterLab with pip:

Step 2: Check that JupyterLab installed correctly:

Step 3: Once installed, launch JupyterLab with:

Sam's Notes | Data Science | Intro

A Basic Chart - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 4

Update the processData() function

Create the bar chart container

Create the bar chart

Hexagon Overlay - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 3

Update the Control Panel

Add Constants for Hexagon Layer

Add the Hexagon Layer

Next

Scatterplot Overlay - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 2

Add a Scatterplot Overlay with Deck.gl

An introduction to Deck.gl

The finished code

Add Data

Add Deck.gl Layers

Polish

Dynamic Color

Control Panel

Mouseover Interaction

Next

Start with a Map - Building a Geospacial App with SvelteKit, Deck.gl, and Mapbox - Part 1

What is Deck.gl?

Setup

SvelteKit

Get a Mapbox API token

Create the Map

Changing Map Styles

onViewportChange Callback

Adding Polish

Next

Create an `environment.yml` file in your project root directory

Step 1: Create an `environment.yml` file in your project root directory

Step 1: Install JupyterLab with `pip`:

Update the `processData()` function

`onViewportChange` Callback