Forem: Ahmad Faiyaz

Test blog

Ahmad Faiyaz — Mon, 12 Dec 2022 20:52:12 +0000

terfdsdfsf

Building an end to end Machine Learning Pipeline in Bigquery

Ahmad Faiyaz — Fri, 13 Mar 2020 10:41:16 +0000

Google BigQuery is one of the more advanced data warehouses in the market, and has out of the box support for building and training ML models using SQL like statements without requiring any code. This is extremely powerful, however managing end to end ML pipelines in this way can be fragile and requires manual steps to updating training and prediction.

In this article we walk through building a simple end to end BigQuery ML pipeline using the open-source framework Dataform to help us manage the end to end process of data preparation, training and prediction.

Google BigQuery provides some Machine Learning algorithms such as Linear regression, Binary logistic regression etc. To find out more about the models that BigQuery supports, check out the documentation.

A typical workflow for building a machine learning model looks like:

Data exploration
Data pre-processing (data transformation)
Model training
Model evaluation on test dataset
Prediction/Inference on real dataset

In this article I am going to follow the tutorial from Google Cloud documentation to create a machine learning model with Google BigQuery, please read the official documentation for understanding the technical details.

For managing our end to end pipeline, we are going to use Dataform to help us version control our queries and manage pipeline execution order. Dataform makes it easy for us to version control our BigQuery code and execute complex pipelines with just a few commands.

All of the code for this example is available in the demo repository here for you to follow along with. To run this example in your own BigQuery project you perform the following steps.

Clone the repo:

git clone https://github.com/dataform-co/bigquery-ml-pipeline.git

Install the Dataform CLI using npm or yarn:

npm i -g @dataform/cli

Set up the Dataform project:

cd bigquery-ml-pipeline && dataform install

For instructions on authenticating BigQuery so that you can run the queries, you can follow the Dataform documentation here.

Data exploration

Data exploration is usually done in jupyter notebooks or some dashboard solution: for example Looker or Google Data Studio. In this step one needs to find out which datasets are required, and which columns should be used as features for model training. Following the Google Cloud tutorial, I am going to use the public dataset named Census Adult Income (bigquery-public-data.ml_datasets.census_adult_income), which contains these columns:

Data pre processing

I need to split the dataset into three sections: training, evaluation and prediction. To do this using Dataform, I will create a new sqlx file with the code block below. This query extracts data on census respondents, including education_num, which represents the respondent's level of education, and workclass, which represents the type of work the respondent performs. This query excludes several categories that duplicate data: for example, the columns education and education_num in the census_adult_income table express the same data in different formats, so this query excludes the education column. The dataframe column uses the excluded functional_weight column to label 80% of the data source for training, and reserves the remaining data for evaluation and prediction. The query creates a table containing these columns, so that I can use them to perform training and prediction later.

census_input.sqlx

config {
  type: "table"
}

SELECT
  age,
  workclass,
  native_country,
  marital_status,
  education_num,
  occupation,
  race,
  hours_per_week,
  income_bracket,
  CASE
    WHEN MOD(functional_weight, 10) < 8 THEN 'training'
    WHEN MOD(functional_weight, 10) = 8 THEN 'evaluation'
    WHEN MOD(functional_weight, 10) = 9 THEN 'prediction'
  END AS dataframe
FROM
  `bigquery-public-data.ml_datasets.census_adult_income`

Model training

Now it's time to create a model using the training dataset. I am creating a new file for this model creation. You can find more about the model options from here.

model_train.sqlx

config {
  type: 'operations',
  hasOutput: true,
  tags: ["training"]
}

CREATE
OR REPLACE MODEL ${self()} OPTIONS (
  model_type = 'LOGISTIC_REG',
  auto_class_weights = TRUE,
  input_label_cols = ['income_bracket']
) AS
SELECT
  *
FROM
  ${ref('census_input')}
WHERE
  dataframe = 'training'

You’ll notice a few non-SQL constructs above, these are Dataform features. Our configuration block at the top tells Dataform that this is an operation (a custom SQL statement) that generates an output relation. We can use the self() function in dataform to get the fully qualified name of this relation that we should create, and we can use the ref() function to select from the preprocessed dataset that we created in the previous step.

I have also added a tag, which will be useful to run only this step with or without dependencies using the command: dataform run --tags training --include-deps or on Dataform web. Check the Dataform documentation on tags.

Model evaluation

Continuing with the standard ML workflow, I will evaluate my model with the evaluation dataset.
As I am creating a ML model pipeline, it is important not to run the prediction step if my model’s accuracy (for this model, I will consider accuracy less than .8 is bad) is not good enough. To solve this in the pipeline, I will use Dataform’s assertion feature.

Bigquery’s ML.EVALUATE function returns a row with column accuracy on which I can run the assertion.

model_evaluate.sqlx

config {
  type: 'assertion',
  tags: ["evaluate"]
}

SELECT
  *
FROM
  ML.EVALUATE (
    MODEL ${ref('model_train')},
    (
      SELECT
        *
      FROM
        ${ref('census_input')}
      WHERE
        dataframe = 'evaluation'
    )
  )
WHERE accuracy < .8

To train the model and run the accuracy test, use the following command:

dataform run --actions model_evaluate --include-deps

Run prediction

As I have created the model and evaluated, now I want to run prediction on a real datasets. Let's do it.

predict.sqlx

config {
  type: 'table',
  dependencies: ['model_evaluate'],
  tags: ["predict"]
}

SELECT
  *
FROM
  ML.PREDICT (
    MODEL ${ref('model_train')},
    (
      SELECT
        *
      FROM
        ${ref('census_input')}
      WHERE
        dataframe = 'prediction'
    )
  )

This prediction step depends on the evaluation step, so I have added it as a dependency.

So let's take a look at the dependency graph:

Now I can run the whole ML pipeline as a schedule within Dataform, written entirely in SQL and executed on BigQuery’s CPUs.

How we use MobX at Dataform to solve our frontend application state problems

Ahmad Faiyaz — Wed, 16 Oct 2019 11:40:51 +0000

Having a state management library on a React based single page application is quite useful, especially if the application is complex in nature, for example, if we want to share states between two react components which are neither siblings nor child. But even if you use a state management library, it might not solve the application state in a clean and expected way.

What library did we use before?

We initially used our in-house developed state management tool, which I will refer to as Goggle Store in this whole article. Goggle Store follows object oriented style, where you need to create state entity and state entities have a flat structure. And the store implementation was type-safe.

What problems did we face with Goggle Store?

As an early stage startup, we couldn’t invest a lot of development time on this in house Goggle store. So we have little to no documentation for the store.
Goggle store uses React’s “forceUpdate” method to re-render react components on state change, which made our React app rendering kinda inefficient. Also forceUpdate usage is discouraged on React’s documentation.
We have to do “console.log” based debugging to check current state of the application with Goggle store.
Not having control over mutating the state on Goggle store, means one can set values in any component by directly calling entity.set(x) which makes hard to keep track of where state is mutated. We had to search the whole code base to find out where set method is being called.
Goggle Store doesn’t have caching mechanism for some state combination. For example, on our Dataform web application, you can switch git branches, so if you open some directories on Branch A, then switch to Branch B open some other directories, then move to Branch A again, we couldn’t show the directories you opened last time due to lack of scoped state caching mechanism.
Goggle Store code structure doesn’t enforce state dependency, so one can add a state entity to the store and make it independent even though it is supposed to be dependent on other state(s). We found many bugs related to this issue, as the developer forgot to reset value on some state changes, which led to inconsistent information on the UI. After having all those issues above, we finally decided to move from Goggle store to another store library, which should solve the above problems and make our life easier.

We chose MobX

We did some R&D with two state management libraries named Redux and MobX. With Redux, we couldn’t have an object oriented structure: it seems best practice for Redux is to have flat store structure. Another thing about Redux is that it requires lots of boilerplate code to work with React, which seems annoying. And last but not least, we couldn’t find a solution to our caching and state dependency problem with Redux.
As a result, we decided on using MobX for our application because of its derivation feature, such as computed values and reactions. Also with MobX we can follow object oriented paradigm and it requires less boilerplate code to work with React. We turned on enforceActions flag so that one can mutate state only inside an action. We have turned mobx-logger on so that one can see how state changes. But MobX didn’t solve our caching and state dependency enforcement issue. To solve those issues we have introduced a state dependency tree.

State Dependency Tree

We grouped our state entities in a store, and created a dependency tree. Our entity structure with Goggle Store (simplified) is like this:

We converted the state like a tree on MobX below:

So the code implementation looks like:

import {action, computed, observable, runInAction} from 'mobx';
import Loadable from './loadable';
export default class Loadable<T> {
  // our state entity class
  public static create<T>(val?: T) {
    return new Loadable<T>(val);
  }
  @observable private value: T;
  @observable private loading: boolean = false;
  constructor(val?: T) {
    this.set(val);
  }
  public isLoading() {
    return this.loading;
  }
  public val() {
    return this.value;
  }
  public set(value: T) {
    this.loading = false;
    this.value = value;
  }
  public setLoading(loading: boolean) {
    this.loading = loading;
  }
}
interface IProject {
  projectName: string;
  projectId: string;
}
export class RootStore {
  @observable public currentProjectId: string = null;
  @observable public projectsList = Loadable.create<IProject[]>();
  public readonly projectStoreMap = new Map<string, ProjectStore>();
  public projectStore(projectId: string) {
    if (!this.projectStoreMap.has(projectId)) {
      const project = this.projectsList
        .val()
        .find(project => project.projectId === projectId);
      if (!project) {
        throw new Error('Project not found');
      }
      this.projectStoreMap.set(projectId, new ProjectStore(project));
    }
    return this.projectStoreMap.get(projectId);
  }
  @computed public get currentProjectStore() {
    return this.projectStore(this.currentProjectId);
  }
  @action public setCurrentProjectId(projectId: string) {
    this.currentProjectId = projectId;
  }
  @action.bound
  public async fetchProjectsList() {
    this.projectsList.setLoading(true);
    const response = await ApiService.get().projectList({});
    runInAction('fetchProjectsListSuccess', () =>
      this.projectsList.set(response.projects)
    );
  }
}
interface IBranch {
  branchName: string;
}
class ProjectStore {
  public readonly currentProject: IProject;
  @observable public branchList = Loadable.create<IBranch[]>();
  @observable public currentBranchName: string = null;
  public readonly branchStoreMap = new Map<string, BranchStore>();
  constructor(project: IProject) {
    this.currentProject = project;
  }
  public branchStore(branchName: string) {
    if (!this.branchStoreMap.has(branchName)) {
      const branch = this.branchList
        .val()
        .find(branch => branch.branchName === branchName);
      if (!branch) {
        throw new Error('Branch not found');
      }
      this.branchStoreMap.set(branchName, new BranchStore(branch));
    }
    return this.branchStoreMap.get(branchName);
  }
  @computed public get currentBranchStore() {
    return this.branchStore(this.currentBranchName);
  }
  @action public setCurrentBranchName(branchName: string) {
    this.currentBranchName = branchName;
  }
  @action.bound
  public async fetchBranchList() {
    this.branchList.setLoading(true);
    const response = await ApiService.get().branchList({
      projectId: this.currentProject.projectId,
    });
    runInAction('fetchBranchListSuccess', () =>
      this.branchList.set(response.branches)
    );
  }
}
const rootStore = new RootStore();

We have utilized the computed value feature to add state dependency. So the developer doesn’t need to know which state entity they need to change. And as we have grouped entities together in a domain based store object, we can now cache the states for which we are using ES6 map, please take a look at line 46-57 for further understanding.

Conclusion

In software development world, no library is good at everything, which is also true for MobX. For example: its documentation, dev-tools are not rich like Redux but so far it is solving our problems. Many people don’t know about MobX as Redux is quite popular in react world. But I think, MobX can also be a great state management solution for many react developers.