Forem: Pipedream

Automate customer support with Dialogflow

Pipedream — Fri, 03 Jun 2022 15:46:25 +0000

In this week's Speed Run, we learn how to connect Dialogflow with Pipedream and generate dynamic responses to customer support requests without code.

With Dialogflow, we can build chat bots that use Natural Language Processing or NLP for short.

Just give Dialogflow examples of training phrases and how to respond to them, and it will automatically train and build a dedicated bot for you.

Then we use Pipedream to customize responses and reply with real dynamic data.

In this example, we automatically reply with the current stock in our inventory.

Real time data check notifications with MongoDB

Pipedream — Thu, 12 May 2022 15:58:50 +0000

Sometimes error logs don't pick up serious logic problems in your apps and your database's integrity is compromised.

This week's Speed Run shows you how to quickly build a data integrity monitoring system to alert you whenever you have malformed or incorrect data in your MongoDB database.

"Check ups" were coined by Ryan Laughlin at RailsConf 2018. You can watch his entire presentation on how they helped him and his team at SplitWise to detect logic bugs in their code:

This same pattern will work with relational databases as well, like MySQL, Postgres and others.

We'll send ourselves a Slack message if impossible data is present in our database, so we can investigate and fix before more data causes downstream issues in our apps.

Send delayed welcome emails to new users with Postmark

Pipedream — Fri, 06 May 2022 16:32:56 +0000

This week we released the ability to delay your workflows with both a pre-built action and a helper function in your Node.js code steps.

In this weeks Speed Run, we give a 5 minute example on how to use Delay to create an email drip campaign after new users sign up for your app.

We integrate with Postmark, an email provider in this workflow and delay it a full 30 minutes before sending an email to the customer with a Calendly link to book a call with us.

You can use delay in Node.js steps as well, using the built in function $.flow.delay(miliseconds).

Read more about delaying workflows in our docs:
https://pipedream.com/docs/workflows/built-in-functions/#delay

How to send promotion emails with Shopify Orders at random, with a Pipedream workflow

Pipedream — Fri, 29 Apr 2022 14:35:35 +0000

Discount codes are a great way to build customer loyalty. Even better, adding a layer of chance adds an element of excitement to the order.

In this episode we show you how to generate promotional emails from your Shopify orders without adding any apps to your store.

You can control how chances each order has of triggering a discount code in an email. Or you can remove this random portion altogether for a post-purchase email message.

Here's the random number generator code step you can use in your workflows:

export default defineComponent({
  props: {
    minimum: { type: 'string' },
    maximum: { type: 'string' }
  },
  async run({ steps, $ }) {
    const max = parseInt(this.maximum);
    const min = parseInt(this.minimum);

    return Math.floor(Math.random() * (max - min + 1) + min);
  },
})

Learn more and get connected!

🔨 Start building at https://pipedream.com
📣 Read our blog https://pipedream.com/blog
💬 Join our community https://pipedream.com/community

Connect a Webflow Form to an Airtable Database

Pipedream — Fri, 22 Apr 2022 15:09:44 +0000

Add interactivity to your Webflow website by collecting contact information from your customers using a Pipedream workflow - in 5 minutes without code!

Contact forms are a vital part of building websites, they're the first layer of interaction between you and your audience, and we'll show you how to start building your own database of contacts using Airtable.

Airtable is a great way to store data and view it in different ways.

After you have stored your contacts in Airtable, you can customize the view to make a Trello type Kanban board, and much more.

We'll show you how to connect your Webflow forms to a Pipedream workflow, so you can leverage the power of data collection from your customers.

Have an idea for a Speed Run? Let us know!

⚡️ Request a Pipedream Speed Run ⚡️

jotform.com

We're always looking for new Speed Run ideas to feature on the show, fill out the 3 question form below for a chance to have a shout out & and your idea brought to life with a Pipedream workflow!

Learn more and get connected!

🔨 Start building at https://pipedream.com
📣 Read our blog https://pipedream.com/blog
💬 Join our community https://pipedream.com/community

Trigger an Algolia Crawler to reindex on Github Repository Releases

Pipedream — Fri, 15 Apr 2022 15:59:40 +0000

Here at Pipedream, you may have noticed that our documentation is powered by Algolia.

In this Speed Run, we'll automate the reindexing of the Pipedream docs on every Github release.

By automating the crawler, we ensure that the search bar in the docs is always up to date with the latest content.

By default, Algolia will only crawl our site once per week. By creating a Pipedream workflow to automate this process, we'll know on every new docs release that the search engine will be up to date!

In this episode we're going to show you how to do this without any code, and leverage a brand new feature here at Pipedream: pre-built native HTTP actions, right in the app.

Learn more and get connected!

🔨 Start building at https://pipedream.com
📣 Read our blog https://pipedream.com/blog
💬 Join our community https://pipedream.com/community

Request to have your idea come to life (and get a shout out too) in a Pipedream Speed Run video:

https://www.jotform.com/220694247042049

Automate a weekly new Twitter Followers Shoutout with Pipedream

Pipedream — Fri, 08 Apr 2022 14:44:55 +0000

Engage your Twitter audience with an automatic weekly shout out to your new followers. We'll show you how to build this workflow in 5 minutes on Pipedream.

In this episode we're featuring a brand new pre-built action for Pipedream Data Stores to store a list of new Twitter followers on your account in a given week.Learn more and get connected!

🔨 Start building at https://pipedream.com
📣 Read our blog https://pipedream.com/blog
💬 Join our community https://pipedream.com/community

Request to have your idea come to life (and get a shout out too) in a Pipedream Speed Run video:

https://www.jotform.com/220694247042049

Build a Slack music playlist with YouTube channels

Pipedream — Fri, 01 Apr 2022 13:35:51 +0000

In this speed run we'll create an automated new music discovery tool by piping new music videos fresh from YouTube straight into your Slack workspace.

We'll create a Pipedream workflow that will notify you whenever new videos are uploaded to your favorite channels - all in less than 5 minutes:

Learn more and get connected
🔨 Start building at https://pipedream.com

📣 Read our blog https://pipedream.com/blog

💬 Join our community https://pipedream.com/community

Real Time Shopify Partner Revenue Tracker

Pipedream — Fri, 25 Mar 2022 12:36:07 +0000

In this Speed Run we'll be building a real time physical count of our Shopify Partner revenue with an official Shopify Counter.

This counter is perfect for feedback on how your apps are earning for you 💵 and give that extra motivation to keep building!

We'll also cover a brand new feature in Pipedream called Data Stores. Data Stores can hold data between workflow runs, which become our database for keeping track of total revenue for the physical Smiirl counter.

Want to see a specific workflow tutorial? We're taking submissions!

🔨 Start building at https://pipedream.com
📣 Read our blog https://pipedream.com/blog
💬 Join our community https://pipedream.com/community

Automatically commit your Dev.to articles to your personal GitHub repo

Pipedream — Fri, 18 Mar 2022 15:06:19 +0000

As you know, Dev.to is one of the biggest Developer communities on the web.

One of the killer features that is that your posts are natively in Markdown with frontmatter that's common static site generators like Jekyll.

Every post you make on Dev.to is actually compatible with static site generators.

In this tutorial, I'll show you how to use Pipedream to build a workflow that will automatically sync your Dev.to articles to your GitHub repository.

That way your posts are backed up in version control, and you can automatically publish them onto your own statically generated site built with Netlify or Vercel.

Get the latest count of confirmed cases, recoveries and deaths from the COVID-19 Coronavirus via an HTTP API

Pipedream — Thu, 06 Feb 2020 00:54:30 +0000

A few days ago I started searching for an API to programmatically get the latest data about the global coronavirus outbreak. I came across several visualization and scraping projects, but I couldn't find a simple way to query the latest data programmatically — so I created an HTTP API that returns the latest data in JSON format using a Pipedream workflow.

The API fetches the latest number of confirmed cases, recoveries and deaths from a public Google Sheet published by the team at the Center for Systems Science and Engineering (CSSE) at John Hopkins University, and returns both the raw regional breakouts as well as summary stats in JSON format. It also caches the data for up to 5 minutes to improve performance. The raw data is aggregated by the team at the CSSE from multiple sources including the WHO, CDC, ECDC, NHC and DXY, and updates are published to Google Sheets multiple times per day.

Using the API

To use the API, just make an HTTP request to the following endpoint URL:

https://coronavirus.m.pipedream.net/

You can test it by loading the URL in your browser or app, by running curl https://coronavirus.m.pipedream.net/ in a terminal, or by copying and running this workflow.

The API returns:

Global, Mainland China and Non-Mainland China summary stats (count of cases, recoveries and deaths)
Raw data (counts by region as published in the Google Sheet)
Metadata (including when data was last published and the cache status)

Note: Data is cached using $checkpoint to improve performance. The cache is updated if it's more than 5 minutes old (view the workflow code or read more below).

Sample API Response

Following is a sample of the data returned by the API. Note: the rawData array is truncated in this example to only show a single result — query the endpoint URL to retrieve the full response.

{
  "apiSourceCode": "https://pipedream.com/@/p_G6CLVM",
  "cache": {
    "lastUpdated": "2 minutes ago",
    "expires": "in 3 minutes",
    "lastUpdatedTimestamp": 1580925783250,
    "expiresTimestamp": 1580926083250
  },
  "summaryStats": {
    "global": {
      "confirmed": 24630,
      "recovered": 1029,
      "deaths": 494
    },
    "mainlandChina": {
      "confirmed": 24405,
      "recovered": 1020,
      "deaths": 492
    },
    "nonMainlandChina": {
      "confirmed": 225,
      "recovered": 9,
      "deaths": 2
    }
  },
  "rawData": [
    {
      "Province/State": "Hubei",
      "Country/Region": "Mainland China",
      "Last Update": "2/5/20 16:43",
      "Confirmed": "16678",
      "Deaths": "479",
      "Recovered": "538"
    },
  ],
  "dataSource": {
    "googleSpreadsheetId": "1wQVypefm946ch4XDp37uZ-wartW4V7ILdg-qYiDXUHM",
    "range": "Feb05_1220PM!A1:F1000",
    "dataLastPublished": "44 minutes ago",
    "googleSheetLastModified": "2020-02-05T17:27:39.593Z",
    "publishedBy": "John Hopkins University Center for Systems Science and Engineering",
    "ref": "https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6"
  }
}e

Capabilities

The workflow behind this API uses the following capabilities and it runs on Pipedream for free (view, copy, modify and run the workflow code):

HTTP trigger
Node.js code steps
Google and Moment.js npm pacakges (to use any npm package, just require it — no package.json or npm install required)
Auth managed by Pipedream (for Google Sheets and Drive)
$checkpoint (maintains state across workflow executions)
$respond() (returns an HTTP response to a client)
Step exports (provides observability into data and enables workflows to pass data to later steps via the steps object)

How It Works

The HTTP API works by triggering the Node.js code in this workflow on every request to https://coronavirus.m.pipedream.net/. The workflow consists of multiple steps to fetch and cache new data, transform and aggregate it, and finally respond to the client.

Next, I'll explain some of the key steps in the workflow.

steps.trigger

When you select an HTTP / Webhook trigger, Pipedream automatically generates a unique endpoint URL to trigger your workflow code. Since I'm sharing the endpoint URL publicly (https://coronavirus.m.pipedream.net/), anyone can make a request to execute the code and get the response.

steps.filter_favicon_requests

The first code step filters out duplicate requests caused by web browsers requesting a favicon.ico asset when loading the endpoint URL — if the the triggering URL contains favicon.ico, the workflow execution ends early and no additional steps or code are executed.

if (steps.trigger.event.url.indexOf('favicon.ico') !== -1) {
  $end('Terminating workfow for favicon request')
}

steps.get_data

Next, we either refresh the cache with the latest data from both Google Sheets and Google Drive (using the googleapis npm package with auth managed by Pipedream), or we return the data cached to this.$checkpoint.

First, we initialize this.checkpoint if it is undefined:

if (typeof this.$checkpoint === 'undefined') {
  // initialize this.$checkpoint
  this.$checkpoint = {}
  this.$checkpoint.data = {}
  this.$checkpoint.ts = 0
  this.$checkpoint.lastModified = ''
  this.$checkpoint.range = ''
  this.$checkpoint.spreadsheetId = ''
  this.$checkpoint.lastPublished = 0
}

Then we determine whether the cache should be updated. There are two processes that will trigger a cache refresh:

I'm running a separate workflow on a schedule to update the cache every 4 minutes. That workflow simply makes a request to the endpoint URL for this workflow and passes refresh as the value of the query parameter action, and the environment variable process.env.CSSE_NCOV_REFRESH_TOKEN as the value for of the query parameter token. The goal for this secondary process is to improve performance for end users, since it's slower to fetch and process live data from Google Sheets (ideally, the only time the cache is updated is via this out of band process).
However, if a user attempts to retrieve data by making a request to the API and the cache is older than 5 minutes, then that should also trigger a real-time lookup to Google Sheets will be triggered (this should only happen if #1 above fails)

this.dataExpiry = 5 * 60 * 1000

if (((new Date().getTime() - this.$checkpoint.ts) > (this.dataExpiry)) || 
    (event.query.action === 'refresh' && event.query.token === 
     process.env.CSSE_NCOV_REFRESH_TOKEN)) {
  this.updateCache = true
} else {
  this.updateCache = false
}

Note: I'm using the step export this.updateCache to determine whether to fetch new data because I'm going to reference this value in a later step (I'll be able to reference this value as steps.get_data.udpateCache). Using step exports also provides default observability into exported data, so I can easily see what condition was triggered for each event:

Select an event to inspect the execution and exports for each step

Finally, if this.updateCache is true, then we fetch the latest data using the googleapis npm package and save it to this.$checkpoint (which maintains state across workflow executions). Otherwise, we simply return the value of this.$checkpoint.

if (this.updateCache === true) {
  // fetch the latest data from the Google Sheet
  console.log('Fetching new data')
  const {google} = require('googleapis') 

  const auth = new google.auth.OAuth2()
  auth.setCredentials({ 
    access_token: auths.google_sheets.oauth_access_token 
  })
  const sheets = await google.sheets({
    version: 'v4', 
    auth
  });

  this.$checkpoint.spreadsheetId = params.spreadsheetId

  let response = await sheets.spreadsheets.values.get({
    spreadsheetId: this.$checkpoint.spreadsheetId,
    range: params.range
  })
  this.$checkpoint.data = response.data
  this.$checkpoint.ts = new Date().getTime()
  
  // get the date/time the file was last modified
  auth.setCredentials({ 
    access_token: auths.google_drive.oauth_access_token 
  })
  const drive = await google.drive({version: 'v3', auth});
  this.$checkpoint.lastModified = (await drive.files.get({
    fileId: this.$checkpoint.spreadsheetId,
    fields: params.fields
  })).data.modifiedTime

  // check if the tab name was updated
  // which indicates new data was published
  if (response.data.range !== this.$checkpoint.range) {
    this.$checkpoint.range = response.data.range
    this.$checkpoint.lastPublished = this.$checkpoint.lastModified
  }
} else {
  console.log('Return cached data')
}

return this.$checkpoint

Note: I connected my Google Sheets and Drive accounts to this step and used the auths object in code to securely pass the oauth access token to Google's API for authentication. E.g.,

const auth = new google.auth.OAuth2()
auth.setCredentials({ 
  access_token: auths.google_sheets.oauth_access_token 
})

If you copy the workflow into your account to modify and run it yourself, then you'll need to connect your own accounts.

steps.transform_data

The data returned from Google Sheets in the previous step is an array of arrays representing the rows and columns of data in the sheet. This step makes the data more ergonomic by transforming the data into an array of JSON objects, with each value matched with its respective key (based on the value in the header).

const transformedData = [], originalData = steps.get_data.$return_value.data.values
let rowCount = 0

originalData.forEach(function(row){
  if (rowCount > 0) {
    let obj = {}
    for (let i=0; i<row.length; i++) {
      obj[originalData[0][i]] = row[i] 
    }
    transformedData.push(obj)
  }
  rowCount++
})

return transformedData

steps.summarize_data

This step returns a JSON object with the total count of confirmed cases, recoveries and deaths, as well as subtotals for Mainland China and non-Mainland China. The data is cached to this.$checkpoint and it uses the updateCache export from steps.get_data to determine whether to update the cache, or return previously cached data.

if (steps.get_data.updateCache === true) {
  console.log('updating cached stats')

  // initialize the stats object
  const stats = {
    global: { confirmed: 0, recovered: 0, deaths: 0 },
    mainlandChina: { confirmed: 0, recovered: 0, deaths: 0 },
    nonMainlandChina: { confirmed: 0, recovered: 0, deaths: 0 },
  }

  function incrementTotals(statsObj, regionObj) {
    statsObj.confirmed += parseInt(regionObj.Confirmed)
    statsObj.recovered += parseInt(regionObj.Recovered)
    statsObj.deaths += parseInt(regionObj.Deaths)
    return statsObj
  }

  steps.transform_data.$return_value.forEach(function(region){
    // increment global totals
    stats.global = incrementTotals(stats.global, region)
    if (region['Country/Region'] === 'Mainland China') {
      // calculate totals for mainland china
      stats.mainlandChina = incrementTotals(stats.mainlandChina, region)
    } else {
      // calculate totals for non-mainland china regions 
      stats.nonMainlandChina = incrementTotals(stats.nonMainlandChina, region)
    }
  })

  this.$checkpoint = stats
} else {
  console.log('using cached stats')
}

return this.$checkpoint

steps.respond_to_client

Finally, we construct the body based on data exported and returned from previous steps, and use the moment.js npm package to provide human readable relative dates/times. We use $respond() to issue the response, set the content-type header to application/json and JSON.stringify() the data before returning it as the response body.

const moment = require('moment')

const body = {}
const lastUpdatedTimestamp = steps.get_data.$return_value.ts
const expiresTimestamp = steps.get_data.dataExpiry 
                         + steps.get_data.$return_value.ts

body.apiSourceCode = `https://pipedream.com/@/${steps.trigger.context.workflow_id}`
body.cache = {
  lastUpdated: moment(lastUpdatedTimestamp).fromNow(),
  expires: moment(expiresTimestamp).fromNow(),
  lastUpdatedTimestamp,
  expiresTimestamp
}
body.summaryStats = steps.summarize_data.$return_value
body.rawData = steps.transform_data.$return_value
body.dataSource = {
  googleSpreadsheetId: steps.get_data.$return_value.spreadsheetId,
  range: steps.get_data.$return_value.range,
  googleSheetLastModified: steps.get_data.$return_value.lastModified,
  dataLastPublished: moment(steps.get_data.$return_value.lastPublished).fromNow(),
  dataPublishedBy: `John Hopkins University Center for Systems Science and Engineering`,
  ref: `https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6`
}

await $respond({
  immediate: true,
  status: 200,
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify(body)
})

Feedback

Try out the public endpoint at https://coronavirus.m.pipedream.net/ or copy, modify and run it yourself for free on Pipedream. Please let us know if you have any feedback – you can join our public Slack at https://pipedream.com/community. And be sure to check out the great work the team at CSSE is doing.

Use any API in seconds with auth managed by Pipedream

Pipedream — Sat, 25 Jan 2020 03:25:52 +0000

Pipedream makes it easy to write Node.js and use any API in seconds! We securely manage auth for 100+ apps (with more added daily) – just connect your app accounts and use the auths object in code to reference tokens and keys.

Github HTTP API

For example, to use Github's HTTP API, just connect your account using a browser-based oauth flow and and pass auths.github.access_token as the value of the Bearer token (check out the octokit section below for an example of how to use auths with an npm package).

const axios = require('axios')

const response = await axios({
  url: `https://api.github.com/user`,
  headers: {
    Authorization: `Bearer ${auths.github.oauth_access_token}`,
  }, 
})

return response.data

Here's a GIF that demonstrates how to use the Github API in 30-seconds:

In ~30-seconds, I add a code step with Github auth, paste in the code above, connect my account, query the API and inspect the results. Copy this template to try it yourself .

Github via Octokit

You're not limited to using Pipedream managed auth with HTTP APIs – it works with npm packages too! Here's an example of the same API request as above, but using Github's octokit package:

const Octokit = require('@octokit/rest')

const octokit = new Octokit({
  auth: auths.github.oauth_access_token
})

return (await octokit.users.getAuthenticated()).data

Pipedream transparently installs any npm packages you require in your code when you deploy. To try it yourself, copy this template, enable steps.octokit, connect your account and run the code.

Scaffolding API + Auth Code

Pipedream also supports actions, which are re-usable code steps. Actions provides scaffolding for popular APIs and functions. To scaffold a generic API for an app, just click the plus (+) button , select the app and choose Run Node.js with [App Name]:

To scaffold a specific API for an app, just click the plus (+) button , select the app and choose the action that you want to scaffold. Try editing the code and confirm that you want to customize it:

Triggering Your Code

Trigger your code:

Manually
On an interval or cron schedule
On HTTP request (Pipedream will generate a unique URL — any requests to this URL will trigger your code)
On an email (Pipedream will generate a unique email address — any emails sent to this address will trigger your code)
Via the Pipedream Node.js or Ruby SDKs

Getting Started

To get started, just sign in with your Google or Github account at https://pipedream.com. Check out the project templates at https://pipedream.com/explore, and join our public Slack at https://pipedream.com/community!