Forem: Jessica Garson

Choosing between an index-level API, a query, an aggregation, or ES|QL in Elasticsearch

Jessica Garson — Mon, 17 Jun 2024 18:03:47 +0000

When getting started with Elasticsearch, something that needs to be clarified is figuring out when to use an index-level API, a query, an aggregation, or ES|QL can be tricky. This blog post aims to walk you through when to use which.

At a high level, you can think about the differences as follows:

Index-level APIs help manage your index. They allow you to create, delete, and modify your index settings, mappings, and aliases.
Queries help retrieve data that meets specified criteria using a JSON-based query language (Query DSL).
Aggregations perform functions such as calculations and grouping data. They are accommodating for data analysis.
ES|QL is a procedural piped query language with SQL-like syntax, useful for data filtering and analytics.

You can run the examples from this blog post inside Elastic’s Dev Tools Console. In the Dev Tools Console, you can directly invoke Elasticsearch’s REST APIs without needing to supply additional authentication parameters inside the Dev Tools Console.

When to use an index-level API

An Elasticsearch index is a data structure containing a set of documents. Each document in an index contains key-value pairs that store your data. An index-level API works with the index as a whole instead of individual documents or a cluster. Index-level APIs enable you to manage your index, settings, aliases, mappings, or templates. The documentation on the subject provides a complete list of index-level APIs.

Examples of times when you would use an index-level API include:

Creating a new index.
Deleting an index.
Cloning an index.
Creating an alias for an index.

Creating a new index

To create a new index you would run the following command:

PUT /new_index

The response that gets returned lets you know that an index called new_index has been created.

{
 "acknowledged": true,
 "shards_acknowledged": true,
 "index": "new_index"
}

Deleting an index

For testing purposes, it's common to create multiple indexes. Having the ability to delete these indexes can be very useful in cleaning up these indexes.

To delete an index, you would use the following syntax:

DELETE /new_index

The output confirms that the index has been deleted successfully.

{
 "acknowledged": true
}

Cloning an index

Cloning an index can be helpful for backup and recovery or data archiving purposes. Our documentation on the subject provides more information on the clone index API.

First, you will want to prevent write operations on the index using the add index block API.

PUT /new_index/_block/write

The output confirms that you have added a block to prevent further write operations on your index.

{
 "acknowledged": true,
 "shards_acknowledged": false,
 "indices": []
}

To clone an index called new_index_cloned from new_index, you would use the following syntax:

POST /new_index/_clone/new_index_cloned

The output indicates that you have created a new index called new_index_cloned.

{
 "acknowledged": true,
 "shards_acknowledged": true,
 "index": "new_index_cloned"
}

Creating aliases

Creating aliases can be helpful for index management. They allow you to refer to an index by a more intuitive and usually shorter name. The following snippet creates an alias for new_index called new.

POST /_aliases
{
 "actions": [
   {
     "add": {
       "index": "new_index",
       "alias": "new"
     }
   }
 ]
}

The output confirms that an alias has been created.

{
 "acknowledged": true
}

When to use queries

While index-level queries help manage your index as a whole, queries help search and retrieve documents that meet the criteria you define. The language used for creating queries in Elasticsearch is called Query DSL (Domain-Specific Language). Query DSL employs JSON. It is beneficial for getting back documents that match the specifications you create inside your query. Some of the most used queries include match, term, and range queries. You also can combine queries to reach a greater granularity using a boolean query.

Match query

A match query in Elasticsearch retrieves documents that correlate with a given value. Match queries are handy for full-text searches since they return text containing a specific phrase or value.

If you had an index containing information on the Boston Celtics games and were looking for games in which the Celtics had a plus-minus score of -19, you would use the following query.

GET /celtics/_search
{
 "query": {
   "match": {
     "PLUS_MINUS": "-19"
   }
 }
}

A basketball team's plus-minus score is a statistic that measures the point differential when a specific team is on the court. It calculates the difference between the team's points and those scored by its opponents while that team is playing. A positive plus-minus indicates the team outscored its opponents, while a negative plus-minus indicates they were outscored.

The result that would get returned contains the one game where the Boston Celtics had a plus-minus score of -19:

{
 "took": 1,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 1,
     "relation": "eq"
   },
   "max_score": 1,
   "hits": [
     {
       "_index": "celtics",
       "_id": "0022300646",
       "_score": 1,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300646",
         "GAME_DATE": "2024-01-27",
         "MATCHUP": "BOS vs. LAC",
         "WL": "L",
         "MIN": 240,
         "PTS": 96,
         "FGM": 36,
         "FGA": 100,
         "FG_PCT": 0.36,
         "FG3M": 10,
         "FG3A": 40,
         "FG3_PCT": 0.25,
         "FTM": 14,
         "FTA": 16,
         "FT_PCT": 0.875,
         "OREB": 18,
         "DREB": 34,
         "REB": 52,
         "AST": 21,
         "STL": 2,
         "BLK": 9,
         "TOV": 11,
         "PF": 13,
         "PLUS_MINUS": -19
       }
     }
   ]
 }
}

Term query

A term query returns an exact match of a specific term, which can be helpful when working with structured data. The following example searches for information about a game on a particular date, April 29th, 2024.

GET /celtics/_search
{
 "query": {
   "term": {
     "GAME_DATE": "2024-04-29"
   }
 }
}

You will get back information about the game that took place on 2024-04-29.

{
 "took": 2,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 1,
     "relation": "eq"
   },
   "max_score": 1,
   "hits": [
     {
       "_index": "celtics",
       "_id": "0042300104",
       "_score": 1,
       "_source": {
         "SEASON_ID": "42023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0042300104",
         "GAME_DATE": "2024-04-29",
         "MATCHUP": "BOS @ MIA",
         "WL": "W",
         "MIN": 240,
         "PTS": 102,
         "FGM": 36,
         "FGA": 86,
         "FG_PCT": 0.419,
         "FG3M": 14,
         "FG3A": 37,
         "FG3_PCT": 0.378,
         "FTM": 16,
         "FTA": 18,
         "FT_PCT": 0.889,
         "OREB": 11,
         "DREB": 35,
         "REB": 46,
         "AST": 21,
         "STL": 5,
         "BLK": 3,
         "TOV": 10,
         "PF": 20,
         "PLUS_MINUS": 14
       }
     }
   ]
 }
}

Range query

A range query in Elasticsearch retrieves documents containing terms within a specified range. While working with range queries, it is helpful to note that gte stands for greater than or equal to, and lte stands for less than or equal to.

The following example looks for Celtics games with total points between 145 and 150.

GET /celtics/_search
{
 "query": {
   "range": {
     "PTS": {
       "gte": 145,
       "lte": 150
     }
   }
 }
}

Two games that fit into this range get returned:

{
 "took": 1,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 2,
     "relation": "eq"
   },
   "max_score": 1,
   "hits": [
     {
       "_index": "celtics",
       "_id": "0022300542",
       "_score": 1,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300542",
         "GAME_DATE": "2024-01-13",
         "MATCHUP": "BOS vs. HOU",
         "WL": "W",
         "MIN": 239,
         "PTS": 145,
         "FGM": 51,
         "FGA": 95,
         "FG_PCT": 0.537,
         "FG3M": 24,
         "FG3A": 47,
         "FG3_PCT": 0.511,
         "FTM": 19,
         "FTA": 25,
         "FT_PCT": 0.76,
         "OREB": 10,
         "DREB": 40,
         "REB": 50,
         "AST": 26,
         "STL": 7,
         "BLK": 8,
         "TOV": 11,
         "PF": 21,
         "PLUS_MINUS": 32
       }
     },
     {
       "_index": "celtics",
       "_id": "0022300389",
       "_score": 1,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300389",
         "GAME_DATE": "2023-12-23",
         "MATCHUP": "BOS @ LAC",
         "WL": "W",
         "MIN": 241,
         "PTS": 145,
         "FGM": 49,
         "FGA": 94,
         "FG_PCT": 0.521,
         "FG3M": 25,
         "FG3A": 53,
         "FG3_PCT": 0.472,
         "FTM": 22,
         "FTA": 28,
         "FT_PCT": 0.786,
         "OREB": 15,
         "DREB": 36,
         "REB": 51,
         "AST": 33,
         "STL": 4,
         "BLK": 5,
         "TOV": 9,
         "PF": 19,
         "PLUS_MINUS": 37
       }
     }
   ]
 }
}

To combine different queries, you can use a boolean query. It returns documents that match the boolean combinations of other queries. A boolean query must contain at least one of the conditions of must, filter, should, or must not.

The following example searches for Celtics games in which they had a plus-minus score of 10, scored between 100 and 130 points, and did not lose the game.

GET /celtics/_search
{
 "query": {
   "bool": {
     "must": [
       {
         "match": {
           "PLUS_MINUS": "10"
         }
       },
       {
         "range": {
           "PTS": {
             "gte": 100,
             "lte": 130
           }
         }
       }
     ],
     "must_not": [
       {
         "term": {
           "WL": "L"
         }
       }
     ]
   }
 }

Four games meet the parameters of the above query will return the following result:

{
 "took": 0,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 4,
     "relation": "eq"
   },
   "max_score": 2,
   "hits": [
     {
       "_index": "celtics",
       "_id": "0022300920",
       "_score": 2,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300920",
         "GAME_DATE": "2024-03-09",
         "MATCHUP": "BOS @ PHX",
         "WL": "W",
         "MIN": 240,
         "PTS": 117,
         "FGM": 46,
         "FGA": 94,
         "FG_PCT": 0.489,
         "FG3M": 15,
         "FG3A": 39,
         "FG3_PCT": 0.385,
         "FTM": 10,
         "FTA": 13,
         "FT_PCT": 0.769,
         "OREB": 13,
         "DREB": 30,
         "REB": 43,
         "AST": 29,
         "STL": 7,
         "BLK": 4,
         "TOV": 12,
         "PF": 12,
         "PLUS_MINUS": 10
       }
     },
     {
       "_index": "celtics",
       "_id": "0022300246",
       "_score": 2,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300246",
         "GAME_DATE": "2023-11-26",
         "MATCHUP": "BOS vs. ATL",
         "WL": "W",
         "MIN": 239,
         "PTS": 113,
         "FGM": 42,
         "FGA": 95,
         "FG_PCT": 0.442,
         "FG3M": 13,
         "FG3A": 47,
         "FG3_PCT": 0.277,
         "FTM": 16,
         "FTA": 20,
         "FT_PCT": 0.8,
         "OREB": 18,
         "DREB": 40,
         "REB": 58,
         "AST": 24,
         "STL": 9,
         "BLK": 3,
         "TOV": 12,
         "PF": 19,
         "PLUS_MINUS": 10
       }
     },
     {
       "_index": "celtics",
       "_id": "0022300194",
       "_score": 2,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300194",
         "GAME_DATE": "2023-11-15",
         "MATCHUP": "BOS @ PHI",
         "WL": "W",
         "MIN": 239,
         "PTS": 117,
         "FGM": 42,
         "FGA": 88,
         "FG_PCT": 0.477,
         "FG3M": 18,
         "FG3A": 50,
         "FG3_PCT": 0.36,
         "FTM": 15,
         "FTA": 19,
         "FT_PCT": 0.789,
         "OREB": 12,
         "DREB": 33,
         "REB": 45,
         "AST": 23,
         "STL": 7,
         "BLK": 8,
         "TOV": 9,
         "PF": 15,
         "PLUS_MINUS": 10
       }
     },
     {
       "_index": "celtics",
       "_id": "0022300136",
       "_score": 2,
       "_source": {
         "SEASON_ID": "22023",
         "TEAM_ID": 1610612738,
         "TEAM_ABBREVIATION": "BOS",
         "TEAM_NAME": "Boston Celtics",
         "GAME_ID": "0022300136",
         "GAME_DATE": "2023-11-04",
         "MATCHUP": "BOS @ BKN",
         "WL": "W",
         "MIN": 240,
         "PTS": 124,
         "FGM": 43,
         "FGA": 90,
         "FG_PCT": 0.478,
         "FG3M": 15,
         "FG3A": 45,
         "FG3_PCT": 0.333,
         "FTM": 23,
         "FTA": 27,
         "FT_PCT": 0.852,
         "OREB": 10,
         "DREB": 40,
         "REB": 50,
         "AST": 22,
         "STL": 4,
         "BLK": 6,
         "TOV": 11,
         "PF": 17,
         "PLUS_MINUS": 10
       }
     }
   ]
 }
}

When to use aggregations

Aggregations in Elasticsearch allow you to summarize data by creating metrics and using summary statistics. They are beneficial for analytics. There are three types of aggregations in Elasticsearch: metric, bucket, and pipeline. You can also nest aggregations as well.

Metric aggregation

A metric aggregation performs calculations such as a sum or average on a field value.

The following query calculates the average of the total number of points contained in the index.

GET /celtics/_search
{
 "size": 0,
 "aggs": {
   "total_points": {
     "avg": {
       "field": "PTS"
     }
   }
 }
}

A result containing a section titled aggregations, which includes the average number of points, will be returned.

{
 "took": 0,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 85,
     "relation": "eq"
   },
   "max_score": null,
   "hits": []
 },
 "aggregations": {
   "total_points": {
     "value": 120.2
   }
 }
}

Bucket aggregations

A bucket aggregation groups documents into buckets according to specified criteria. These buckets, or bins, categorize data based on field values, ranges, or other criteria.

To group the Celtics games by months you would use the following query:

GET /celtics/_search
{
 "size": 0,
 "aggs": {
   "games_over_time": {
     "date_histogram": {
       "field": "GAME_DATE",
       "calendar_interval": "month"
     }
   }
 }
}

In the aggregations section of the JSON response, there is a date histogram called games_over_time, grouping documents by month. doc_count indicates the number of documents per month.

{
 "took": 5,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 85,
     "relation": "eq"
   },
   "max_score": null,
   "hits": []
 },
 "aggregations": {
   "games_over_time": {
     "buckets": [
       {
         "key_as_string": "2023-10-01T00:00:00.000Z",
         "key": 1696118400000,
         "doc_count": 3
       },
       {
         "key_as_string": "2023-11-01T00:00:00.000Z",
         "key": 1698796800000,
         "doc_count": 15
       },
       {
         "key_as_string": "2023-12-01T00:00:00.000Z",
         "key": 1701388800000,
         "doc_count": 14
       },
       {
         "key_as_string": "2024-01-01T00:00:00.000Z",
         "key": 1704067200000,
         "doc_count": 16
       },
       {
         "key_as_string": "2024-02-01T00:00:00.000Z",
         "key": 1706745600000,
         "doc_count": 10
       },
       {
         "key_as_string": "2024-03-01T00:00:00.000Z",
         "key": 1709251200000,
         "doc_count": 16
       },
       {
         "key_as_string": "2024-04-01T00:00:00.000Z",
         "key": 1711929600000,
         "doc_count": 11
       }
     ]
   }
 }
}

Pipeline aggregations

Pipeline aggregations in Elasticsearch perform calculations on the results of other aggregations, allowing for complex data processing and analytics.

You can create a query that calculates the cumulative sum of points scored. First, you will need a date histogram to bucket the documents by date, then a cumulative sum pipeline aggregation to calculate the cumulative total. The following query uses a pipeline aggregation to view the total number of points scored by the Celtics per month.

GET /celtics/_search
{
 "size": 0,
 "aggs": {
   "games_over_time": {
     "date_histogram": {
       "field": "GAME_DATE",
       "calendar_interval": "month"
     },
     "aggs": {
       "total_points": {
         "sum": {
           "field": "PTS"
         }
       },
       "cumulative_points": {
         "cumulative_sum": {
           "buckets_path": "total_points"
         }
       }
     }
   }
 }
}

The aggregations section of the response contains groupings of the total number of points scored by the Celtics organized by month. You can also see the total number of games per month as well.

{
 "took": 2,
 "timed_out": false,
 "_shards": {
   "total": 1,
   "successful": 1,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": {
     "value": 85,
     "relation": "eq"
   },
   "max_score": null,
   "hits": []
 },
 "aggregations": {
   "games_over_time": {
     "buckets": [
       {
         "key_as_string": "2023-10-01T00:00:00.000Z",
         "key": 1696118400000,
         "doc_count": 3,
         "total_points": {
           "value": 353
         },
         "cumulative_points": {
           "value": 353
         }
       },
       {
         "key_as_string": "2023-11-01T00:00:00.000Z",
         "key": 1698796800000,
         "doc_count": 15,
         "total_points": {
           "value": 1740
         },
         "cumulative_points": {
           "value": 2093
         }
       },
       {
         "key_as_string": "2023-12-01T00:00:00.000Z",
         "key": 1701388800000,
         "doc_count": 14,
         "total_points": {
           "value": 1771
         },
         "cumulative_points": {
           "value": 3864
         }
       },
       {
         "key_as_string": "2024-01-01T00:00:00.000Z",
         "key": 1704067200000,
         "doc_count": 16,
         "total_points": {
           "value": 1915
         },
         "cumulative_points": {
           "value": 5779
         }
       },
       {
         "key_as_string": "2024-02-01T00:00:00.000Z",
         "key": 1706745600000,
         "doc_count": 10,
         "total_points": {
           "value": 1220
         },
         "cumulative_points": {
           "value": 6999
         }
       },
       {
         "key_as_string": "2024-03-01T00:00:00.000Z",
         "key": 1709251200000,
         "doc_count": 16,
         "total_points": {
           "value": 1947
         },
         "cumulative_points": {
           "value": 8946
         }
       },
       {
         "key_as_string": "2024-04-01T00:00:00.000Z",
         "key": 1711929600000,
         "doc_count": 11,
         "total_points": {
           "value": 1271
         },
         "cumulative_points": {
           "value": 10217
         }
       }
     ]
   }
 }
}

Nested aggregations

One advanced feature of aggregations is the ability to nest them, which allows you to perform multilevel data analysis. For example, you can first group your data by a particular field and perform further aggregations within those groups.

An example of a nested aggregation is an aggregation that first calculates the average of points scored and groups by game month and then by the game result (whether they won or lost).

Since the WL field is a text field, field data is disabled for text fields by default because text fields are not optimized for operations that require per-document field data like aggregations and sorting. To fix this, you can use a keyword field instead. Since WL is already a text field, you can add a .keyword subfield.

GET /celtics/_search
{
  "size": 0,
  "aggs": {
    "games_by_month": {
      "date_histogram": {
        "field": "GAME_DATE",
        "calendar_interval": "month"
      },
      "aggs": {
        "results": {
          "terms": {
            "field": "WL.keyword"
          },
          "aggs": {
            "average_points": {
              "avg": {
                "field": "PTS"
              }
            }
          }
        }
      }
    }
  }
}

The response will provide the average points scored each month, further broken down by wins and losses, allowing you to analyze performance trends over time.

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 85,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "games_by_month": {
      "buckets": [
        {
          "key_as_string": "2023-10-01T00:00:00.000Z",
          "key": 1696118400000,
          "doc_count": 3,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 3,
                "average_points": {
                  "value": 117.66666666666667
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2023-11-01T00:00:00.000Z",
          "key": 1698796800000,
          "doc_count": 15,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 11,
                "average_points": {
                  "value": 119.45454545454545
                }
              },
              {
                "key": "L",
                "doc_count": 4,
                "average_points": {
                  "value": 106.5
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2023-12-01T00:00:00.000Z",
          "key": 1701388800000,
          "doc_count": 14,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 12,
                "average_points": {
                  "value": 127.75
                }
              },
              {
                "key": "L",
                "doc_count": 2,
                "average_points": {
                  "value": 119
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2024-01-01T00:00:00.000Z",
          "key": 1704067200000,
          "doc_count": 16,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 11,
                "average_points": {
                  "value": 123.9090909090909
                }
              },
              {
                "key": "L",
                "doc_count": 5,
                "average_points": {
                  "value": 110.4
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2024-02-01T00:00:00.000Z",
          "key": 1706745600000,
          "doc_count": 10,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 9,
                "average_points": {
                  "value": 123.88888888888889
                }
              },
              {
                "key": "L",
                "doc_count": 1,
                "average_points": {
                  "value": 105
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2024-03-01T00:00:00.000Z",
          "key": 1709251200000,
          "doc_count": 16,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 12,
                "average_points": {
                  "value": 124.5
                }
              },
              {
                "key": "L",
                "doc_count": 4,
                "average_points": {
                  "value": 113.25
                }
              }
            ]
          }
        },
        {
          "key_as_string": "2024-04-01T00:00:00.000Z",
          "key": 1711929600000,
          "doc_count": 11,
          "results": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "W",
                "doc_count": 9,
                "average_points": {
                  "value": 117.88888888888889
                }
              },
              {
                "key": "L",
                "doc_count": 2,
                "average_points": {
                  "value": 105
                }
              }
            ]
          }
        }
      ]
    }
  }
}

When does ES|QL come in?

While the structure of a JSON query language is something that you get used to over time, it can be challenging at first. ES|QL is a procedural piped query language with SQL-like syntax. One key feature is that you can define the order in which you wish to return data. The pipes in ES|QL look like |, enabling you to manipulate and transform data step by step. The main uses of ES|QL are for analytics, data manipulation, and other transformations. It also can help work with visualizations in general.

Filtering data with ES|QL

ES|QL allows you to quickly filter data from a dataset to return the results you are looking for. The following query first specifies the desired format for the returned data. The default is JSON, but you can define formats such as txt or csv. For readability, txt was chosen. The FROM field contains an index called celtics. The query then narrows the index down to fields containing the date of the game and whether the team won or lost. It also limits the results to the last ten games from the dataset.

POST _query?format=txt
{
 "query": """
   FROM celtics
   | KEEP GAME_DATE, WL
   | LIMIT 10
 """

The results you get back contain a table-like structure that includes the date of the game and the result, with W being a win and L being a loss.

      GAME_DATE        |      WL      
------------------------+---------------
2024-04-29T00:00:00.000Z|W             
2024-04-27T00:00:00.000Z|W             
2024-04-24T00:00:00.000Z|L             
2024-04-21T00:00:00.000Z|W             
2024-04-14T00:00:00.000Z|W             
2024-04-12T00:00:00.000Z|W             
2024-04-11T00:00:00.000Z|L             
2024-04-07T00:00:00.000Z|W             
2024-04-05T00:00:00.000Z|W             
2024-04-03T00:00:00.000Z|W

SQL-like syntax

You can also utilize the SQL-like syntax for additional filtering capabilities. The following query narrows the index down to games that the Celtics won and returns the fields for the date of the game and the matchup, which includes the two teams that played. It also limits it to the last ten results contained in the index.

POST _query?format=txt
{
 "query": """
   FROM celtics
   | WHERE WL == "W"
   | KEEP GAME_DATE, MATCHUP
   | LIMIT 10
 """
}

The results you get back contain a table-like structure that includes the date of the game and the matchup, which consists of a summary of the two teams that played the given game.

      GAME_DATE        |    MATCHUP   
------------------------+---------------
2024-04-29T00:00:00.000Z|BOS @ MIA     
2024-04-27T00:00:00.000Z|BOS @ MIA     
2024-04-21T00:00:00.000Z|BOS vs. MIA   
2024-04-14T00:00:00.000Z|BOS vs. WAS   
2024-04-12T00:00:00.000Z|BOS vs. CHA   
2024-04-07T00:00:00.000Z|BOS vs. POR   
2024-04-05T00:00:00.000Z|BOS vs. SAC   
2024-04-03T00:00:00.000Z|BOS vs. OKC   
2024-04-01T00:00:00.000Z|BOS @ CHA     
2024-03-30T00:00:00.000Z|BOS @ NOP

Aggregations in ES|QL

You can use ES|QL to quickly find statistics about a given field. The following query uses the exact data for the Celtics to find the average field goal and field goal percentage from the three-point line. It limits the average to include one line to summarize the data contained in the index.

In basketball, the field goal percentage is a key statistic that measures a player or team's efficiency in making shots. It is calculated by dividing the number of successful field goals by the total number of field goal attempts. The three-point field goal percentage is a version of this metric, but it focuses instead only on shots taken from beyond the three-point line.

POST _query?format=txt
{
 "query": """
   FROM celtics
   | STATS AVG(FG_PCT), AVG(FG3_PCT)
   | LIMIT 1
 """
}

The result contains a table-like structure that includes the index's average field goal percentage and field goal percentage from the three-point line.

     AVG(FG_PCT)    |   AVG(FG3_PCT)   
-------------------+-------------------
0.48734117781414704|0.38770588215659646

Working with visualizations

One of ES|QL's main advantages is its ability to work with visualization in Kibana and reduce the need to switch between different tools. You can use it to explore your data while creating or modifying visualizations seamlessly. Additionally, you can create alerts based on conditions defined with ES|QL.

Conclusion

Typically, an index-level API helps manage your index, a query to find specific data, and an aggregation to perform calculations or obtain statistics about your data. ES|QL, a piped query language, allows you to filter, transform, and analyze structured and unstructured data in Elasticsearch more intuitively than with a JSON query. Let us know if you built anything based on this blog or if you have questions on our Discuss forums and the community Slack channel.

Additional resources

If you are getting started with Elastic, these resources may be helpful.

Harnessing Screams with Tidal Looper

Jessica Garson — Sun, 12 May 2024 19:38:45 +0000

Since 2017, I have performed as Messica Arson, my live coded noise project. A defining feature of this work is the use of looping screams, which adds a potent rawness to my sound. Over the years, I've used many different technologies, from playing audio files in Audacity to a custom script entitled scream.py that allows me to create vocal samples on the fly quickly.

Since then, I've been working more and more with TidalCycles. TidalCycles is an open-source live coding framework for creating patterns written in Haskell. TidalCycles uses SuperCollider on the backend, another language I've been using for live coding. Recently, I started using Tidal Looper for live vocal processing. This blog post will walk you through what you need to get started with vocal looping with Tidal Looper.

The code for this blog post is available in the corresponding GitHub repository. It includes the file I use to start Tidal Cycles for shows featuring the use of Tidal Looper, as well as a code from my live performance at Murder by Modulation, an event hosted by the New York Modular Society during the fall at Trans Pecos.

What you need to get started

First, you will need to have SuperCollider and TidalCycles installed locally. Then, to get started with vocal looping, you will need a microphone and a way to connect to your computer.

My current setup is as follows:

Motu M4
Shure SM-58 with a switch - The switch is necessary because I often use feedback in my work, and having control over when I use it can make my performance more dynamic.

Before getting started you will need to make sure your audio input is set to use your audio interface or the method through which your microphone is connected to your computer.

Why Tidal Looper

Tidal Looper allows you to sample live on the fly natively within TidalCycles. You don’t have to switch between different frameworks and programs. Tidal Looper aims to recreate the capabilities of a looping pedal, with recording, a replace mode, and an overdub mode. You also have the full patterning capabilities of TidalCycles on the loops you record.

Installing Tidal Looper

The steps to installing Tidal Looper are as follows:

Download the Tidal Looper repository.
Install Tidal Looper as SuperCollider Quark by selecting where it says "Language," then selecting where it says "Quarks," and finally selecting "Install a folder."
In SuperCollider, create a new file called start_looper.scd that contains the following:

~dirt = SuperDirt.start

~looper = TidalLooper(~dirt)

For sets where I’m using Tidal Looper this is how I load SuperCollider before starting my set.

Many adjust their startup script so that Tidal Looper automatically loads when they open SuperCollider. Directions on how to do this can be found in the README of Tidal Looper. Since I use various live coding frameworks for different sets (sometimes multiple in the same set), this doesn't quite work for me.

Using Tidal Looper

Now, you can start creating sounds with Tidal Looper. For this section, I’ll walk you through a track created and performed live in the fall.

I usually start in my live sets by defining the cycles per second or tempo of the sounds. I'll typically begin by setting the cps to 2, meaning each cycle of the rhythm or pattern you're programming will last half a second. This setting effectively determines the speed at which your musical patterns progress and loop. I often increase the cps to 3 during the peak of my live sets, which means that the tempo of the music or rhythm is defined so that each cycle of the loop takes one-third of a second.

setcps 2

I often start my sets by screaming. The line once $ s "looper" plays the first scream while I’m recording it. I can use this scream throughout my performance.

once $ s "looper"

I frequently use TidalCycles to create drum beats and synth sounds that are almost danceable. However, to add a unique twist and throw the rhythm off balance, I incorporate recordings of my screams recorded on the fly with Tidal Looper. This technique introduces an unexpected element contrasting with the otherwise rhythmic and melodic flow.

d1 defines a music pattern. This pattern plays a bass drum sample (bd) and then a sequence involving the samples bubble, loop repeated three times, and control. The # operator is used to apply effects: leslie 0.3 applies a Leslie speaker effect at level 0.3, and enhance 0.5 applies an enhancement effect at level 0.5. The sample loop is the scream recorded using Tidal Looper in the first line of the code.

d1 $ "bd drum! [bubble loop*3 control]" # leslie 0.3 # enhance 0.5

I’ll often like to make use of the overdub or olooper mode in Tidal Looper. This mode is similar to an overdub in a looping pedal where new sounds can be added on top of existing loops. I use this to create a more full sound.

d2 uses qtrigger 1 to quantize the start of the pattern, ensuring it starts at the beginning of a cycle. Stack layers multiple samples: olooper created from the overdub function of Tidal Looper, loop - the original scream recorded at the beginning of the set, scream and gabba repeated seven times, playing them simultaneously. The sample scream is a sound I've recorded using scream.py and have loaded into Tidal Cycles. I'll use this technique to create a loud rhythmic beat mixed in with a chorus of screams.

d2 $ qtrigger 1 $ stack [s "olooper", s "loop", s "scream", s "gabba*7"]

I enjoy blending ambient sounds with my screams. This combination produces a harsh yet calming effect. This approach allows me to explore the dynamic interplay between tranquility and turbulence in my soundscapes. The distortion and reverb applied to this pattern allow me to add a level of physicality to my sound.

In d3, I play the seawolf sample three times, interspersed with a variant of the loopsample recorded at the beginning of the set and two screams. It also includes effects: distort 1.2 significantly distorts the sound, and room 0.5 applies a reverb effect at level 0.5.

d3 $ s "seawolf*3 {loop! ~ scream:2}" # distort 1.2 # room 0.5

In d4, I use Tidal Cycles patterning capabilities to create a more exciting sound. This code block utilizes the function every 3 (fast 2) to double the speed of the pattern every third cycle. It specifies a sequence of note indices with n "0 1 [~ 2] 3" and dictates the samples to be played, which include arpy and scream repeated six times. The samples feel and less are layered simultaneously. Additionally, legato 0.3 is employed to smooth the transitions between notes, enhancing the fluidity of the sound.

d4 $ every 3 (fast 2) $ n "0 1 [~ 2] 3" # sound "arpy scream*6 {feel less}" # legato 0.3

In d5, I created a pattern with the sample fire played on repeat, a rest (~), and then a group with sid played three times and 808 played twice. The effect squiz 0.5 compresses the sound, and decay 0.3 shortens the sound's decay time.

d5 $ "fire! ~ [sid*3 808*2]" # squiz 0.5 # decay 0.3

One of my favorite tricks in Tidal Cycles is to chop up and slice my screams. d6 uses chop 16 to cut the pattern into 16 pieces. It plays a variation of scream followed by the original scream loop recorded with Tidal Looper loop and another variant of scream. detune 0.3 slightly detunes the samples.

d6 $ chop 16 $ "scream:3 loop*3 scream:4" # detune 0.3

What does this sound like

You can hear a jam using the code from this blog post:

Conclusion

By using TidalCycles alongside Tidal Looper, it becomes a canvas for sonic experimentation. While this is the setup I use for live vocal processing of my screams in some of my performances, you can use the same process to record instruments or other vocal work. Something I love about the livecoding community is its supportive nature, so if you run into any trouble, you can reach out on Tidal Club or Discord.

This blog post was made possible by a TidalCycles microgrant, with thanks to the Open Collective contributors

Export your Bookmarks with Flask, OAuth 2.0, and Render

Jessica Garson — Thu, 29 Sep 2022 23:40:01 +0000

Earlier this year, I built a sample that allows you to plot your Bookmarks. However, I recently received an email asking if I knew of a service that will enable you to export your Bookmarks. So, I decided to revamp this sample to export your Bookmarks into a CSV. This example uses Python and Flask to create a website where you can export a CSV of your bookmarks.

The code is currently live using Render for hosting, and you can find the full version of the code on our GitHub.

What do you need to get started with the Twitter API

You need to sign up for access to the Twitter API. You will also create a project in the developer portal and an App containing the credentials required to use the Twitter API. Additionally, you will need to have OAuth 2.0 turned on in your App’s authentication settings. The developer playground is a great place to start using the Twitter API.

Setting up your templates

In your code editor, you will want to create three files, base.html, index.html, and next.html inside a folder called templates.

The base.html file is where you set up your site's style and HTML formatting. All the other templates you make will extend from this template.

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Export your Bookmarks!{% block title %}{% endblock %}</title>
    <link
      href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
      rel="stylesheet"
    />
    <link
      href="https://getbootstrap.com/examples/jumbotron-narrow/jumbotron-narrow.css"
      rel="stylesheet"
    />
  </head>

  <body>
    <div class="container">
      <h1>Export your Bookmarks!</h1>
      {% block containercontent %}{% endblock %}
      <hr />
      <div class="footer">
        <p class="text-muted">
          <small
            >This is a basic demo created by the
            <a href="https://twitter.com/TwitterDev">@TwitterDev</a> team. It is
            written in Python, using
            <a href="http://flask.pocoo.org/">Flask</a>. Basic CSS is provided by
            <a href="http://getbootstrap.com/">Bootstrap</a>. The source code
            for this project can be found at
            <a href="https://github.com/twitterdev/export-bookmarks"
              >github/twitterdev/export-bookmarks</a
            >.</small
          >
        </p>
      </div>
    </div>
  </body>
</html>

The index.html serves as the landing page for your site and walks you and your users through what you can expect from the demo site.

{% extends "base.html" %}
{% block containercontent %}
<p class="lead">This application allows you to export your recent Bookmarked Tweets.</p>
<div class="hero-unit">
    <p>To get started, we'll need you to authorize our demo App (Hello world twitter bot) to access your bookmarks on Twitter using OAuth 2.0. Learn more information about <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">OAuth 2.0.</a></p>
    <p>Here's how it works:</p>
    <ol>
        <li>We will redirect you to Twitter.com for you to authorize our app.</li>
        <li>While at Twitter.com, you will authorize our App to access your Bookmarks.</li>
        <li>After you click "Export CSV", you should get back a CSV of your most recent Bookmarks.</li>
    </ol>
</div>
<p>Let's get started!</p>
<a href="/start" class="btn btn-small btn-success">Start</a>
{% endblock %}

The next.html template is where you can set up to download the CSV after clicking on a button. In the HTML for the button, you will find a link to next <a href=next>. This is where you will later connect to a route called next where you set the logic to download a CSV.

{% extends "base.html" %}
{% block containercontent %}
<p class="lead">This application allows you to export your recent Bookmarked Tweets.</p>
<div class="hero-unit">
    <p>To get started, we'll need you to authorize our demo App (Hello world twitter bot) to access your bookmarks on Twitter using OAuth 2.0. Learn more information about <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">OAuth 2.0.</a></p>
    <p>Here's how it works:</p>
    <ol>
        <li>We will redirect you to Twitter.com for you to authorize our app.</li>
        <li>While at Twitter.com, you will authorize our App to access your Bookmarks.</li>
        <li>After you click "Export CSV", you should get back a CSV of your most recent Bookmarks.</li>
    </ol>
</div>
<p>Let's get started!</p>
<a href="/start" class="btn btn-small btn-success">Start</a>
{% endblock %}

Installing the required packages

In your terminal, you can install the packages you need with the following command:

pip3 requests pandas requests_oauthlib flask

We will be using requests to make HTTP requests to the Twitter API, pandas for shaping the data before you export it, requests_oauthlib to help connect to OAuth 2.0, and flask to set up a web framework for you to interact with a browser before downloading the export of your Bookmarks.

Creating your app.py file

Most of the logic for your application will take place in your app.py file. So, in your code editor, create a file called app.py.

You will first want to import all the packages you will use at the top of the file. The imports will include the packages you installed in the previous step and helper utilities such as base64, hashlib, and os, which help access environment variables and connect to OAuth 2.0.

import base64
import hashlib
import os
import re
import requests
import pandas as pd

from requests_oauthlib import OAuth2Session
from flask import (
    Flask,
    request,
    redirect,
    session,
    render_template,
    make_response,
)

As you do at the beginning of most Flask applications, you will want to set a variable called app and connect a random secret key.

app = Flask(__name__)
app.secret_key = os.urandom(50)

Now that you’ve set up your flask app, you will want to add some code that helps you connect to OAuth 2.0. OAuth 2.0 is an industry-standard authorization protocol that allows for greater control over an application’s scope, and authorization flows across multiple devices. Our implementation includes an Authorization Code Flow with PKCE.

With OAuth 2.0, you can set the scopes, which are your App's permissions, and these scopes will be present to the user when they log in.

You can obtain your client ID and client secret from your App’s keys and tokens inside the developer portal. The redirect URI for running the code locally should be http://127.0.0.1:5000/oauth/callback. When you deploy this to a server, you will want to add a redirect for the site you use for deployment.

client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")
redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

scopes = ["tweet.read", "users.read", "bookmark.read"]

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)
code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()

code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")

Inside your terminal, you can set environment variables for CLIENT_ID, CLIENT_SECRET, and REDIRECT_URI.

export CLIENT_ID=’your_client_id’
export CLIENT_SECRET=’your_client_secret’
export REDIRECT_URI=’http://127.0.0.1:5000/oauth/callback’

Now you can create a function called get_bookmarks which makes an HTTP request to the manage Bookmarks endpoint. This request also includes a field for a date the Tweet was created, created_at, which has to be requested separately as a tweet.field to allow for more flexibility.

def get_bookmarks(user_id, token):
    print("Making a request to the bookmarks endpoint")
    params = {"tweet.fields": "created_at"}
    return requests.request(
        "GET",
        "https://api.twitter.com/2/users/{}/bookmarks".format(user_id),
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
        params=params,
    )

At this point, you will want to set up the first page a user visits when they come to your site. For this site, you will display the index.html you set up earlier.

@app.route("/")
def hello():
    return render_template("index.html")

After the user presses the start button, they will be taken to Twitter to log into their Twitter account and see your app's permissions which you set in the scopes.

@app.route("/start")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)

Once the user has confirmed, you will get an access token that can be used to make requests on behalf of the user. For example, since you need the user's ID to use the manage Bookmarks endpoint, you can use authenticated user lookup endpoint to get their user ID. Afterward, you can request the manage Bookmarks endpoint to get the user’s Bookmarks and store them in a tabular fashion using a panda’s DataFrame.

@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )
    print(token)
    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]
    bookmarks = get_bookmarks(user_id, token).json()
    global df
    df = pd.DataFrame(bookmarks["data"])
    return render_template("next.html")

After the user authenticates, they will go to a page where they can download their Bookmarks when they click the export button.

@app.route("/oauth/next")
def export():
    resp = make_response(df.to_csv())
    resp.headers["Content-Disposition"] = "attachment; filename=export.csv"
    resp.headers["Content-Type"] = "text/csv"
    return resp


if __name__ == "__main__":
    app.run(debug=True)

To run the file locally, run the following line in your terminal:

python3 app.py

Deployment

I currently have this project deployed using Render. I used a similar method described in their documentation on flask deployment. You will have to add your new REDIRECT_URI to your environment variables inside of Render and in the developer portal in your application’s Authentication Settings. Additionally, you will want to add your Client ID and Client Secret to your environment variables.

Next steps

Hopefully, this tutorial can serve as a starting point for getting started with OAuth 2.0, Flask, or web development. In addition, this code sample can be easily extended to select the fields and expansions you want from your CSV or connect to Google Sheets.

Be sure to inform us on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything!

Creating bots with OAuth 2.0 Authorization Code Flow with PKCE and V2 of the Twitter API

Jessica Garson — Fri, 23 Sep 2022 17:30:09 +0000

Bots on Twitter are a large part of what makes the conversation on Twitter engaging and unique. Some bots enhance the Twitter experience, like Thread Reader, and some bots provide vital information, such as Acts of Congress. Additionally, there are other bots, such as New New York Times, which Tweets every time the New York Times uses a new word.

Making bots on Twitter is a great way to start programming and using the Twitter API. However, in the past year, we’ve made some improvements to our API, such as launching a new manage Tweets endpoint and an OAuth 2.0 Authorization Code Flow with PKCE. Making bots using these new features is possible, but it is a bit different than it has been in the past.

A new paradigm for making bots

One change with OAuth 2.0 is that the bot account will need to log in and give consent to your App before you can get an access token. As someone who is typically a backend-focused developer, I found this a bit harder to get my head around. How I solved this problem as a Python developer by creating a Flask application to handle the login flow.

With OAuth 2.0, your access token, the credential you use to request v2 endpoints, stays valid for two hours. However, since bots run automatically, you will need to figure out how to handle refreshing tokens and save them to a database or in memory.

Prerequisites

A developer account. If you don’t already have access to the Twitter API, you can sign up for a developer account.
A Project in the developer portal
An App containing the credentials required to use the Twitter API
OAuth 2.0 turned on in your App’s authentication settings

Steps to making a bot

Step 1: Set up the profile for your bot, including adding a profile picture and banner image and setting up a label to indicate your bot is a bot.
Step 2: While logged in as your bot, you can make your bot's first call using our API playground.
Step 3: Create a web application that handles the OAuth 2.0 log-in flow for your bot
Step 4: Determine how you’ll handle refreshing tokens for your bot and where you will store your token information.
Step 5: Figure out how often your bot will Tweet. For example, my bots typically Tweet twice a day, but it could be more frequent or less frequent depending on the bots you create.

Resources

Check out our tutorial, which walks you through how to make a bot with OAuth 2.0 using Python. This tutorial walks you through how to make a bot that Tweets dog facts like this bot.
We have an example bot starter project built using Flask deployed to Glitch that can be remixed. This code powers this bot that Tweets noun/verb pairings.

Key learnings

As I’ve been making bots using OAuth 2.0, I’ve learned the following lessons along the way:

Using Python’s persistent in-memory storage as a key-value store

When I started making bots using OAuth 2.0, I used Redis as a key-value store for handling my tokens. I found that I could use the dbm Python package as a lightweight, cost-effective way of storing my tokens in a key-value store.

Handling refresh tokens

While refresh tokens remain valid for six months to allow flexibility to change the timing, it might be best to generate a new one each time the bot posts a Tweet.

Determining the timing for bots in the application itself

When I’ve previously made bots, I’ve typically used an external service such as Render’s cron job service or Google’s cloud scheduler to handle the timing of my bots. But for @noun__verb, I used flask-apscheduler to set up the timing. This might be better for smaller projects, but you may want to consider a cloud service as you scale.

Next steps

Hopefully, this post inspires you to create Twitter bots using OAuth 2.0. Be sure to inform us on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this guide inspires you to create anything!

Showcasing developer built tools with the Twitter Toolbox

Jessica Garson — Fri, 09 Sep 2022 15:35:01 +0000

As a Developer Advocate at Twitter, I have the privilege of seeing many tools developers have created using the Twitter API. I’m consistently impressed and inspired by what developers have made using the Twitter API. However, when I’d hear from other developers that they were unsure what they could build with the Twitter API.

So, earlier this year, we launched Twitter Toolbox, a hub where you easily discover, learn more about, and quickly sign-up for third-party developer tools that enhance your Twitter experience. Tools are bots, on-Twitter enhancements, or off-Twitter experiences that you invent. This is especially exciting because this is the first time that we’ve featured tools created by developers in such a prominent way.

Categories of the Toolbox

We currently have three types of tools in the Toolbox:

Expression tools - Discover new and creative ways to compose, organize, and manage your Tweets. These tools include Chirr, Thread Reader, and Buffer.
Safety tools - Helping you find new or more efficient ways to filter unwanted content, mentions, and profiles with our safety tools, Block Party and Bodyguard.
Measurement tools - Measure your impact and explore new ways to connect with your audience using Ilo, Blackmagic, Direcon, Followerwonk, and Tweepsmap.

Next steps

If you are feeling inspired to start creating with the Twitter API, you can sign up for a developer account to get started with the API. You can also start making API calls on our developer playground.

We hope to expand the number of applications we have in the Toolbox in the future and continue to add new tools. Let us know in the comments below if you have any tools you think we should consider for this program.

Plotting Bookmarks with Flask, Matplotlib, and OAuth 2.0

Jessica Garson — Thu, 30 Jun 2022 01:51:50 +0000

I don’t Bookmark Tweets very often, and I was wondering how constantly I use the Bookmark functionality on Twitter. To figure out exactly how often I bookmark Tweets, I built a demo application in Python using Flask.

This application will authorize your account using the OAuth 2.0 Authorization Code Flow with PKCE and plot up to your last 100 Bookmarked Tweets. The full code sample can be found on our GitHub.

Step 1: Getting set up with the Twitter API

If you don’t already have access to the Twitter API, you can sign up for a developer account. Once you have an approved developer account, you must create a Project in the developer portal. Each Project contains an App with which you can generate the credentials required to use the Twitter API. You will also need to have OAuth 2.0 turned on in your App’s settings. In our platform overview, you can learn more about getting started with the Twitter API.

Step 2: Setting up your environment

After obtaining access to the Twitter API, you are ready to set up your requirements.txt file.

First, you’ll need to create a directory for this project in your terminal and navigate into it. After making a requirements.txt file, you can open it in your chosen code editor. In your terminal, run the following command:

mkdir plot-bookmarks
cd plot-bookmarks
touch requirements.txt

Add these packages to requirements.txt file

certifi==2022.6.15
charset-normalizer==2.0.12
click==8.1.3
cycler==0.11.0
Flask==2.1.2
fonttools==4.33.3
gunicorn==20.1.0
idna==3.3
importlib-metadata==4.11.4
itsdangerous==2.1.2
Jinja2==3.1.2
kiwisolver==1.4.3
MarkupSafe==2.1.1
matplotlib==3.5.2
numpy
oauthlib==3.2.0
packaging==21.3
pandas==1.3.5
Pillow==9.1.1
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
requests==2.28.0
requests-oauthlib==1.3.1
six==1.16.0
urllib3==1.26.9
Werkzeug==2.1.2
zipp==3.8.0

To install all the packages in your requirements.txt file, in your terminal, run the following command:

pip install -r requirements.txt

Step 3: Creating your templates

Now that you have your environment set, you are ready to define your templates. Templates in Flask allow you to define the HTML of your pages. For this example, you’ll define two templates, base.html, and index.html.

Inside of the plot-bookmarks directory, you will want to make a new directory in your terminal and two files for templates (base.html and index.html). In your terminal, run the following command:

mkdir templates
cd templates
touch base.html
touch index.html

The base.html file contains the framework for all HTML pages in the application and can be extended to apply to other pages.

Add the following code to base.html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Plot Bookmarks!{% block title %}{% endblock %}</title>
    <link
      rel="stylesheet"
      href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css"
    />
  </head>

  <body>
    <div class="container">
      <h1>Plot Bookmarks by Date</h1>
      {% block containercontent %}{% endblock %}
      <hr />
      <div class="footer">
        <p class="text-muted">
          <small
            >This is a basic demo created by the
            <a href="https://twitter.com/TwitterDev">@TwitterDev</a> team. It is
            written in Python, using
            <a href="http://flask.pocoo.org/">Flask</a>. Basic CSS is provided by
            <a href="http://getbootstrap.com/">Bootstrap</a>. The source code
            for this project can be found at
            <a href="https://github.com/twitterdev/plot-bookmarks"
              >github/twitterdev/plot-bookmarks</a
            >.</small
          >
        </p>
      </div>
    </div>
  </body>
</html>

The index.html template extends from the base.html template and will be used in your code later on to introduce the application to the user. Add this code to index.html.

{% extends "base.html" %}
{% block containercontent %}
<p class="lead">This app lets you plot your Bookmarked Tweets by Date.</p>
<div class="hero-unit">
    <p>To get started, we'll need you to authorize Hello world twitter app to access your bookmarks on Twitter using OAuth 2.0. Learn more information about <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">OAuth 2.0</a></p>
    <p>Here's how it works:</p>
    <ol>
        <li>We will redirect you to Twitter.com for you to authorize our app.</li>
        <li>While at Twitter.com, you will authorize bookmarks-to-Pinterest to access your bookmarks. Upon authorization, we will get your recent 100 bookmarks.</li>
        <li>Next we will display your Bookmarked Tweets by date.</li>
    </ol>
</div>
<p>Ready to give this a shot?</p>
<a href="/start" class="btn btn-small btn-success">Let's Begin</a>
{% endblock %}

Step 4: Creating your app.py file

Now that you have your templates set, you will create an app.py file which will be the main file for your application.

To get started, navigate to the plot-bookmarks directory in your terminal and create an app.py file by running the following command:

cd ..
touch app.py

You will also set up environment variables to ensure you aren’t passing in your secrets directory. To achieve this, you will define environment variables. In your terminal, run the following command:

export CLIENT_ID=’xxxxxxxxxxxxxx’
export CLIENT_SECRET=’xxxxxxxxxxx’
export REDIRECT_URI=’http://127.0.0.1:5000/oauth/callback’

You can obtain your own OAuth 2.0 Client ID and Secret inside of your App’s settings in the developer portal. Use your Client ID and Secret instead of the xxxxxxxxxxxxxx.

I am currently using the type of App as a public client, a single-page App. Inside your App’s OAuth settings in the Developer Portal, ensure that you also have the above redirect URI defined for locally testing your application.

You are ready to start editing your app.py file in your chosen code editor. First, you will import the packages you’ll need for your application. You will use base64, hashlib, and re to create your code verifier and code challenge. The os package will be used to set environment variables and randomize values. You will use the requests package to make HTTP requests to the Twitter API and pandas to create a data frame so you can plot values. You will use io and the matplotlib features for creating your bookmarks plot and flask for creating the web framework.

Add this code to your app.py file.

import base64
import hashlib
import os
import re
import requests
import pandas as pd
import io

from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
from requests_oauthlib import OAuth2Session
from flask import Flask, request, redirect, session, render_template, Response

First, you will need to set a variable for your app, which is the start of every Flask app. After, you’ll need to create a secret key for your app to be a random string using the os package.

Add this code to below your import statements.

app = Flask(__name__)

app.secret_key = os.urandom(50)

To bring the environment variables into your code, you can use the os package to get the variables you set in your terminal.

Add this code after the app = Flask(__name__) code block.

client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")

Additionally, you will obtain the environment variable you set for your redirect URI and create a variable for the authorization and token URLs for generating an OAuth 2.0 token.

Add this code below the client_secret variable.

redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

Scopes are what sets the permissions for an application. With OAuth 2.0, there are more fine-grained permissions, so you only need to ask the user for what you need. For this application, you will need the following scopes:

tweet.read, which gives you access to read Tweets.
users.read allowing you to obtain information about users
bookmark.read giving you access to an authenticated user’s bookmarks

Add this code below the token_url variable.

scopes = ["tweet.read", "users.read", "bookmark.read"]

Since Twitter’s implementation of OAuth 2.0 is PKCE compliant, you will need to set a code verifier, a secure random string. The code verifier is used to create the code challenge. The code challenge is a base64 encoded string of the SHA256 hash of the code verifier.

Add this code below the scopes variable.

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")

At this point, you’ll want to set a function for getting an authenticated user’s Bookmarks back from the Twitter API. For debugging, you’ll want to add a print statement so that you know when you are requesting your server. This function uses the requests package to request the Bookmarks lookup endpoint. Since you will want to use the date of the Bookmark for the plot later, make sure that you pass in a tweet.field of created_at as a param.

def get_bookmarks(user_id, token):
    print("Making a request to the bookmarks endpoint")
    params = {"tweet.fields": "created_at"}
    return requests.request(
        "GET",
        "https://api.twitter.com/2/users/{}/bookmarks".format(user_id),
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
        params=params,
    )

Now, you’ll want to set up the landing page of your application. This will render the template from the index.html file you created earlier. The @app.route("/") indicates this is the first page at the beginning of the application a user will be taken to.

@app.route("/")
def hello():
    return render_template("index.html")

After clicking the “Let’s Begin” button on the application you will be directed to a page for the user to log into Twitter and authorize your App to make requests on the behalf of the user.

@app.route("/start")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)

After the user logs in, they will be directed to the callback. The callback will generate an object called token that contains a series of tokens that includes an access token. The access token is what is used to make a request to the Bookmarks lookup endpoint. You will print out this token object to debug on the server.

@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )
    print(token)

After getting a token, you will want to make a request to the authenticated user lookup endpoint. Since the Bookmarks lookup endpoint requires you to have the ID of the user you are obtaining bookmarks on behalf of. You will pass the user_id and token object into the get_bookmarks function you defined earlier in your code. You’ll save your Bookmarks into a JSON object so that you can easily convert into a data frame.

    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]
    bookmarks = get_bookmarks(user_id, token).json()

Now you can create a data frame in Pandas for your Bookmarks, and adjust the date so that it is easy to work with. You’ll also plot a count of Bookmarks per day. To do so, you can use the groupby function of Pandas. You will define your labels as the date the Tweet was created and the count as the values.

    df = pd.DataFrame(bookmarks["data"])
    df["created_at"] = pd.to_datetime(df["created_at"])
    df["created_at"] = df["created_at"].dt.date
    counts = pd.DataFrame({"count": 
    df.groupby(["created_at"]).size()}).reset_index()
    labels = counts["created_at"]
    values = counts["count"]

To plot the labels and values you will use Matplotlib. Once you have your graph you can save it and render in a view.

    fig = Figure()
    ax = fig.subplots()
    ax.set_title("Number of Bookmarks by Date")
    ax.bar(labels, values, color="orange", width=26)
    output = io.BytesIO()
    FigureCanvas(fig).print_png(output)
    return Response(output.getvalue(), mimetype="image/png")

if __name__ == "__main__":
    app.run()

To run the file locally, run the following line in your terminal:

python app.py

For further testing or if you run into any errors you may want to consider turning on Flask’s debug mode.

Step 5: Deployment

I currently have this project deployed using Render. I used a similar method to what’s described in their documentation on flask deployment.

To serve the correct URI redirect, I had to deploy the application first to figure out what my URL would be and set it up in my environment variables in Render and the developer portal.

The redirect URI for my application was as follows:

https://plot-bookmarks.onrender.com/oauth/callback

I also set up environment variables for Client ID and Client Secret in my environment variable in my settings in Render.

Next steps

Hopefully, this can be a starting point for you to get started with the Bookmarks endpoint, plotting in Python and OAuth 2.0. This code sample can be easily extended to be part of a complete application that includes more graphs and dynamic content.

Be sure to inform us on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Handling Refresh Tokens in the OAuth 2.0 Authorization Code Flow with PKCE with Flask

Jessica Garson — Fri, 17 Jun 2022 20:42:05 +0000

We recently released OAuth 2.0 Authorization Code Flow with PKCE for use with most of our v2 endpoints. As a backend-focused Python developer, I struggled with the flow since much of the code I like to write focuses on automation tasks.

This tutorial will walk you through critical concepts related to refresh tokens using v2 of the Twitter API. If you don’t already have access to the Twitter API, you can sign up for an account. You will also need to have OAuth 2.0 turned on in your App’s settings.

Tokens are only valid for two hours

I was used to working with OAuth 1.0a, where keys and tokens are available for use until they are revoked. This is not the case with OAuth 2.0 Authorization Code Flow with PKCE. Access Tokens generated using this flow are only valid for 2 hours.

Refresh tokens

Inside the token object generated from the consent flow is a refresh token. A refresh token allows an application to obtain a new access token without prompting the user to log in again.

To generate a refresh token, you must set a scope for offline access. If I was using the manage Tweets endpoint and I wanted to Tweet on behalf of a user every six months, I’d use the following line of Python code:

scopes = ["tweet.read", "users.read", "tweet.write", "offline.access"]

How long are refresh tokens valid for?

Refresh tokens stay valid for six months, so you will want to refresh them at least that often or more regularly.

Generating tokens

To generate a token using OAuth 2.0 Authorization Code Flow with PKCE you can use a method similar to this example that allows you to look up an authenticated user’s bookmarks. Since the authenticated user will need to log in directly, and you will need to parse the response, I found creating a Flask application allowed for a more automated process.

The following code will generate a token and save the token into a dictionary.

You will be working with the following libraries:

requests for making HTTP requests
requests_oauthlib for working with OAuth 2.0
os for parsing environment variables and creating random strings
re, base64 and hashlib to create the code challenge and code verifier
flask for creating a web framework

If you don’t already have flask, requests_oauthlib and requests installed you will need to install these libraries.

You will first need to import the following libraries:

import base64
import hashlib
import os
import re
import requests

from requests_oauthlib import OAuth2Session
from flask import Flask, request, redirect, session, url_for

First, you will need to set a variable for your app, which is the start of every Flask app. Additionally, you will need to set a secret key for your app to be a random string.

app = Flask(__name__)
app.secret_key = os.urandom(50)

The two main credentials you will need to authenticate with OAuth 2.0 Authorization Code Flow with PKCE are Client ID and Client Secret. You can set these as environment variables to ensure security.

client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")

You will also need to set a variable for your redirect URL, this should be the same value as your callback URL in your App’s settings in the Developer Portal.

redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

To define the permissions of your App, you will need to set the scopes of your application. Check out the authentication mapping guide to determine what scopes you will need based on the endpoints you are using. The scope offline.access allows you to generate refresh tokens. The scopes for this demo are tweet.read and users.read, which gives you access to read Tweets and obtain information about users.

scopes = ["tweet.read", "users.read", "offline.access"]

You will need to set a code verifier which is a secure random string. The code verifier is used to create the code challenge.

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier

With PKCE the code challenge is a base64 encoded string of the SHA256 hash of the code verifier.

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")

Now you can start with creating a login page that will be the first page you will visit when you run your application. @app.route("/") indicates it’s the first landing page. This page will be the page that an authenticated user logs into.

@app.route("/")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)

After the user logs in, they will be directed to the callback. In the callback you will generate an object called token that contains a series of tokens that includes an access token and a refresh token.

@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )

Now that you’ve generated a token object, you can now make a request to the authenticated user lookup, to obtain a user ID that can be used to access many of our v2 Users endpoints such as manage Tweets or manage blocks.
The access_token variable inside of the token object is the bearer token you’d use to connect to any of the endpoints that support OAuth 2.0 Authorization Code Flow with PKCE.

    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]

Now that you’ve created a token and saved into a dictionary you can now access the token dictionary and create a refreshed token through this process. This code should be added to your callback method. This refreshed_token will contain a new access token as well.

tokens = {"new_token": token}
    t = tokens["new_token"]
    refreshed_token = twitter.refresh_token(
          client_id=client_id,
          client_secret=client_secret,
          token_url=token_url,
          refresh_token=t["refresh_token"],
        )
    tokens.update({"new_token": refreshed_token})
    return "You should now have a refreshed token"


if __name__ == "__main__":
    app.run()

Full code

Here is the full code that you can save as app.py

import base64
import hashlib
import os
import re
import requests

from requests_oauthlib import OAuth2Session
from flask import Flask, request, redirect, session, url_for


app = Flask(__name__)
app.secret_key = os.urandom(50)


client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")

redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

scopes = ["tweet.read", "users.read", "offline.access"]

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")


@app.route("/")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)


@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )
    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]
    tokens = {"new_token": token}
    t = tokens["new_token"]
    refreshed_token = twitter.refresh_token(
          client_id=client_id,
          client_secret=client_secret,
          token_url=token_url,
          refresh_token=t["refresh_token"],
        )
    tokens.update({"new_token": refreshed_token})
    return "You should now have a refreshed token"


if __name__ == "__main__":
    app.run()

To run the file locally you can run the following line:

python app.py

Next steps

Hopefully, this can be a starting point for you to get started with generating refresh tokens. As a next step, if you are using Flask you may want to consider using a schedular to update your refresh tokens regularly in an automated fashion. Additionally you may want to consider saving your tokens to a database in a secure fashion. This code sample can also be extended to allow you to connect to any of the endpoints that support v2 and can be deployed to a server as part of a more complete application.

Be sure to let us know on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Tweet to Super Followers with Postman, OAuth 2.0 and Manage Tweets

Jessica Garson — Mon, 28 Feb 2022 17:12:56 +0000

We recently launched Manage Tweets with a feature to Tweet directly to Super Follows. This tutorial will walk you through how to get started using the Twitter API to Tweet to your subscribers using OAuth 2.0 and Postman.

What is Super Follows?

Super Follows is an opportunity to earn monthly revenue by sharing subscriber-only content with your most engaged followers on Twitter. Through Super Follows, creators like you can create a monthly subscription of $2.99, $4.99, or $9.99 a month, where you could give access to exclusive content or insights into your creative process for creating your bot. In addition, super Followers enjoy extra special access to unfiltered thoughts, early previews, and subscriber-only conversations from their favorite accounts.

How to Apply to Super Follows

You will need to have Super Follows enabled on your account to follow this tutorial. To apply to set up a Super Follows subscription, you will need to follow these steps:

Swipe open the sidebar on your home timeline
Tap on “Monetization
”Select “Super Follows”

After you apply, our team will review your application shortly. You will also need to sign up for a developer account if you don’t already have one.

Setting up Postman

You will first want to download Postman if you have not already done so. After you have Postman enabled, you will want to download the Twitter API v2 collection. To learn more about getting started with the Postman and the Twitter API, check out our documentation on the subject.

Generating an OAuth 2.0 Token

For this tutorial, you will use OAuth 2.0 with PKCE to make requests on behalf of the Super Followers enabled account. To authenticate using OAuth 2.0, you must create an OAuth 2.0 token using the following step. First, In the “Manage Tweets” folder of the Twitter API v2 collection, go to the tab entitled “Authorization” and select “OAuth 2.0”.

In this tab, be sure to follow these steps:

Name your token
Select the Grant Type as Authorization Code (with PKCE)
Update the parameters:
- Auth URL - https://twitter.com/i/oauth2/authorize
- Access Token URL - https://api.twitter.com/2/oauth2/token
- Callback URL - https://www.example.com (matching the callback URL I set in my auth settings page in the Developer Portal)
- Client ID - Cut and paste OAuth 2.0 client ID from the Developer Portal
Review the proper scopes for Manage Tweets on this page.
Update the scopes with the following values: tweet.read users.read tweet.write
Populate the field state with “State”
Click where it says “Generate Token”
Press the save icon to save the folder changes.

You may get a message that you are not logged into Twitter. If you get this error, you will need to log in to the Twitter account inside of Postman you are trying to post on behalf of.

Tweet to your Super Followers

In the Postman Collection, you should find a sample request for “Manage Tweets.” Select the POST request called “Create a Tweet.”

You will want to go into that request and change the “Authorization” tab of the request and select “Inherit auth from parent” to ensure you are using the token you created in the last step.

Next, update the “Body” tab of the request to the following:

{
   "text": "Hello, Super Followers!",
   "for_super_followers_only": true
}

You can now press the blue button that says “Send” and see a response that looks like this one.

{
   "data": {
       "id": "1491508991335272449",
       "text": "Hello, Super Followers!"
   }
}

Next steps

Hopefully, this can be a starting point for you to get started with Tweeting programmatically to Super Followers using Postman and OAuth 2.0. To generate a code sample in the language of your choosing, click on the button in Postman that says “Code." Additionally, you may want to write some language to schedule specific Tweets on a schedule to your subscribers and run your code on a server.

Be sure to let us know on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Getting started with data visualization with Dash and recent search counts

Jessica Garson — Mon, 13 Sep 2021 15:26:04 +0000

Let’s say you want to create a webpage where you can visualize how often you are Tweeting. For example, you could use the recent Tweets counts endpoint and Dash to create a website that automatically refreshes to show how much you are Tweeting throughout the week. This tutorial assumes a basic understanding of Python and Python 3 installed on your machine.

What is Dash?

Dash allows you to quickly set up a Flask app to create a web framework for data visualizations using Python, R, Julia, or .NET. For this tutorial, you’ll be using Dash with Python.

First, you’ll need to install dash with the following command in your terminal.

pip3 install dash

Additionally, you also want to install dash-bootstrap components to customize the look and feel of our page.

pip3 install dash-bootstrap-components

Getting started with the Twitter API

With this tutorial, you’ll be using the Tweet counts endpoint that returns the frequency of your Tweets over the last seven days.

Before you can use the Twitter API v2, you will need a developer account. Once you have an approved developer account, you will need to create a Project in the developer portal. Each Project contains an App, with which you can generate the credentials required to use the Twitter API. You can learn more about getting started with the Twitter API in the getting started section of our documentation.

To connect to the Twitter API, we’ll be using the requests package of Python. You will need to install this and pandas to help us get our data into the right shape. You can use dash-bootstrap-components to customize the look and feel of your webpage a bit further.

pip3 install requests
pip3 install pandas

Setting up your app.py file

To start writing code to create your web application, you will need to set up an app.py file. In your terminal, you will want to make a python file entitled app.py.

touch app.py

Importing packages

Dash utilizes the plotting library Plotly. Plotly is an open-source plotting package, and you can check out the documentation, which might help debug some issues you may encounter.

In the code editor of your choosing, open up your app.py file. At the top of the file, you will need to define the packages you want to import.

import dash
import dash_bootstrap_components as dbc
import plotly.express as px
import pandas as pd
import os
import requests
from dash import dcc
from dash import html

Creating the app variable

After importing the packages, you will need to create a variable called app. In this variable, you are initializing the app you are making and pulling in the bootstrap theme of journal. Using a bootstrap theme is just one way that you can start customizing your page.

app = dash.Dash(external_stylesheets=[dbc.themes.JOURNAL])
server = app.server

Defining the endpoint URL and setting up your query parameters
Before connecting to the Twitter API, you can set the URL to the endpoint you are calling. For this example, you will be calling the recent Tweet counts endpoint, and you will save that as a variable called search_url.

search_url = "https://api.twitter.com/2/tweets/counts/recent"

Next, you’ll want to set up a dictionary for query_params to define the query and granularity of the counts you’ll get back. In the parameter entitled query, you can specify the rules of what will get returned from the Twitter API. In the example below, the query is set to be all Tweets from the handle @jessicagarson. You can swap this out for your handle if you wish. To learn more about creating a query, check out our documentation on the subject.

With the granularity parameter, you can adjust the timing of the counts you’ll have returned. For example, you will get back hourly data instead of daily if you don’t specify ‘day’ in the granularity parameter.

query_params = {'query': 'from:jessicagarson','granularity': 'day'}

Authenticating to the Twitter API

You will want to set up an environment variable in your terminal to save your Bearer Token to your request. Then, you need to replace your-bearer-token with your bearer token. Be sure to keep the single quotes around your Bearer Token. You can get your bearer token from the keys and tokens section of your App inside the developer portal.

export BEARER_TOKEN='your-bearer-token'

Define a variable set to bearer_token in your code editor, and use the os module to get the bearer_token environment variable set in the terminal.

bearer_token = os.environ.get('BEARER_TOKEN')

Pass your Bearer Token as a parameter to create the headers to authenticate to the Twitter API as part of a method.

def bearer_oauth(r):
    r.headers["Authorization"] = f"Bearer {bearer_token}"
    r.headers["User-Agent"] = "GettingStartedDash"
    return r

Connecting to the Twitter API

Create a function to connect to the Twitter API and get back a JSON response if you’ve successfully connected. This function has some additional error handling and will print out the response code you get to help with debugging.

def connect_to_endpoint(url, tweet_fields):
    response = requests.request("GET", url, auth=bearer_oauth, params=tweet_fields)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(
            "Request returned an error: {} {}".format(
                response.status_code, response.text
            )
        )
    return response.json()

Now, you can call the function you created with the URL for the Tweet counts endpoint, our authentication headers, and the query parameters you defined earlier.

json_response = connect_to_endpoint(search_url, query_params)

Creating a data frame

Next, you can convert the JSON object returned from the Twitter API into a pandas data frame. Using a data frame allows you to work with the data more robustly and gets our data into the correct format to create a plot with Dash. For our purposes, you’ll only need to work with the data object of the JSON payload you get from the Twitter API, so you can convert only that into a data frame using pandas.

df = pd.DataFrame(json_response['data'])

You’ll also need to convert the date to a universal date object which you can do with the to_datetime method in pandas.

df['start'] = pd.to_datetime(df['start'])

Since you’ll only need the start and tweet_count columns, you can create a new data frame called final that selects only these columns.

final = df[['start', 'tweet_count']]

Now you are ready to create a line graph with our final data frame and set our x-axis and y-axis up.

fig = px.line(final, x="start", y="tweet_count")

Setting up your website

You can now start creating our website. You can begin by setting up the colors you want to use by creating a dictionary for the hex colors for our background and the text of our line graph.

colors = {"background": "#FFFFFF", "text": "#1DA1F2"}

fig.update_layout(
    plot_bgcolor=colors["background"],
    paper_bgcolor=colors["background"],
    font_color=colors["text"],
)

At this point, you are ready to tie it all together by setting up your app layout. You will first need to define your background color, followed by setting up your title as an H1 HTML element and a brief description of your graph. Additionally, you will need to set your chart with and unique id and set the figure equal to the fig variable you defined earlier.

app.layout = html.Div(
    style={"backgroundColor": colors["background"]},
    children=[
        html.H1(
            children="Tweets by Date",
            style={"textAlign": "center", "color": colors["text"]},
        ),
        html.Div(
            children="An example using Dash and the Twitter API recent search counts to see how much I've been Tweeting this week",
            style={"textAlign": "center", "color": colors["text"]},
        ),
        dcc.Graph(id="example-twitter", figure=fig),
    ],
)

Finally, you will need to set the main statement at the end of your file to run in debug mode when it sets up a local server for you to run and view the code.

if __name__ == "__main__":
    app.run_server(debug=True)

Running the Python file and deployment

Now that you’ve written the code needed for your Python file, you can run it locally. In your terminal, run the following command.

python3 app.py

Now in your terminal, you should see a response that looks similar to the following.

200
Dash is running on http://127.0.0.1:8050/

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
200

If you go to http://127.0.0.1:8050/ in our web browser. You should now see a graph that looks similar to the one below.

Deployment

This application is deployed to Glitch and can be remixed. This application is boosted using Glitch's pro plan to pull data more often.

Next steps

Hopefully, this tutorial can be a starting place to work with Dash and the Twitter API. You can view the full version of the code in our GitHub repository for this code sample. You can also check out a version deployed to Glitch.

As a next step, you may want to customize your query further to see how much you Tweet about a specific topic or how many times a particular term has been posted throughout the week. You may also want to add data visualization using a different Twitter API endpoint, such as searching for themes around what type of Tweets you like.

Be sure to let us know on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Getting started with R and v2 of the Twitter API

Jessica Garson — Thu, 25 Mar 2021 18:08:21 +0000

This tutorial will walk through what you need to get started with the programming language R and the Twitter API v2. Using R to connect to the user lookup endpoint, I’ll show how to work with JSON returned from the Twitter API. User lookup is a GET method and returns information about a user or group of users, specified by a user ID or a username.

If you aren’t familiar, R is one of the most popular languages for common Data Science tasks like time-series analysis, modeling, visualization, and other data analysis, and is often used in conjunction with the Twitter API. With the user lookup endpoint, you can use the user object to determine a correlation between the number of followers a person has and the sentiment score of their bio. The user object may also be used to map a group of accounts based on the location publicly listed in their profiles.

Getting started with the Twitter API

Before you can use the Twitter API v2, you will need a developer account.

Once you have an approved developer account, you will need to first create a Project. Projects allow you to organize your work based on how you intend to use the Twitter API, so you can effectively manage your access to the API, and monitor your usage.

Each Project contains an App, with which you can generate the credentials required to use the Twitter API. You can learn more about how to get started with the Twitter API, in the getting started section of our documentation.

Getting your R environment set up

First, you need to download R which you can do on the cran website.

After, to set up an environment to work with R, you can use R studio, the R extension pack for Visual Studio Code, or a Jupyter notebook if you come from the Python world.

Setting up your environment variable

For the code examples, I’m going to be showing today, you will want to create an environment variable for your bearer token. The bearer token is what allows you to authenticate to the Twitter API and start making requests. First, replace “your-bearer-token” with your own bearer token, which can be obtained from the keys and tokens section of your App in the developer portal. You’ll need to run this line of code in the console before you start writing a script.

Sys.setenv(BEARER_TOKEN = "your-bearer-token")

Making your request

You can use the package httr to make HTTP requests to the Twitter API. If you haven’t already installed this, please install the package in your console. You will also need to install jsonlite to work with our JSON object and dplyr for data manipulation.

install.packages("httr")
install.packages("jsonlite")
install.packages("dplyr")

You can now begin writing your R script to connect to the API. At the top of the file, call the packages httr, jsonlite, and dplyr.

require(httr)
require(jsonlite)
require(dplyr)

The first step in your code sample is to get set up to authenticate to the Twitter API. Grab the bearer token you set, and pass that into your headers for authentication.

bearer_token <- Sys.getenv("BEARER_TOKEN")
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))

Once you got your authentication set up, define the parameters of your request. By default, you will get back the id, name, and username of each user you get back. You can make adjustments to this payload by adding additional fields and expansions. For this example, you will want the profile bio of the user which is found in the description and an expansion that contains the pinned Tweet of the user.

params <- list(`user.fields` = 'description',
               `expansions` = 'pinned_tweet_id')

Now you are ready to format your URL with the handle you are looking to get more information about. Use the readline method to allow this sample to be reusable. After you type the handle you want to look at, format your URL to contain the handle you define.

handle <- readline('What handle do you want to look at? ')
url_handle <-
  sprintf('https://api.twitter.com/2/users/by?usernames=%s', handle)

At this point, use the httr package to make a GET request to the URL you just created, pass in our authentication credential via the header, and pass in the parameters you defined. You can save the response as a text object in the variable obj and print this out to view the result of the request you made.

response <-
  httr::GET(url = url_handle,
            httr::add_headers(.headers = headers),
            query = params)
obj <- httr::content(response, as = "text")
print(obj)

Working with our JSON payload

One of my favorite ways to work with a JSON is to use a data frame, which allows you to easily access complex nested data. To do this, use the fromJSON method of the jsonlite package to flatten your file to allow the fields to be in the same object. Then, pass that object into a data frame. Now you are ready to view this data frame.

json_data <- fromJSON(obj, flatten = TRUE) %>% as.data.frame
View(json_data)

You can access the fields of the data from the data frame and pass them into a string that has the handle, the username, and bio.

final <-
  sprintf(
    "Handle: %s\nBio: %s\nPinned Tweet: %s",
    json_data$data.username,
    json_data$data.description,
    json_data$includes.tweets.text
  )

Use cat instead of print to view the object with newlines between each field you are pulling.

cat(final)

If you had multiple handles you made requests for, you could easily use a loop to access each data frame element.

Conclusion

Hopefully, this tutorial can be a starting place to work with R and the Twitter API. As a next step, you may want to look at our R samples for recent search, Tweet lookup and user lookup in our v2 sample code. Be sure to let us know on the forums if you run into any troubles along the way or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Sending a response from the Twitter API to a spreadsheet in Google Sheets

Jessica Garson — Tue, 22 Dec 2020 22:24:46 +0000

Liquid syntax error: 'raw' tag was never closed

5 ways to convert a JSON object to a CSV

Jessica Garson — Tue, 24 Nov 2020 23:07:50 +0000

While working with APIs, you may need to convert a JSON object to a CSV. Since CSVs are a widely used, versatile format that can easily be imported into an Excel spreadsheet, database, or data visualization software. Turning a JSON payload into a CSV is a great way to explore an API for the first time since you can see each field individually. This blog post will walk you through five strategies to convert a JSON response generated from a request made using the recent search endpoint from v2 of the Twitter API.

What you need to get started

To get started, you will need to have an approved developer account, and have activated the new developer portal experience. Access to the Twitter API is available with active keys and tokens for a developer App. As a reminder, to create an App to use v2 of the Twitter API, you must first create a Project in the developer portal, and then you will be prompted to create your App. You will also need a bearer token from your App, which can be retrieved in the Twitter developer portal.

1. From the command line using csvkit

Using the command line to turn a JSON object into a CSV can come in handy if you want to explore the payload of an API for the first time. It allows you to quickly make a request and see the results in a tabular structure.

First, set up an environment variable in your terminal to save your bearer token to your request. You need to replace your-bearer-token with your own bearer token. Be sure to keep the single quotes around your bearer token.

export BEARER_TOKEN='your-bearer-token'

Now, install csvkit, a collection of tools that allows you to work with CSVs from the command line. You need to have Python and pip installed on your machine to complete this step.

pip3 install csvkit

Next, use curl to make an HTTP request and save the output as a JSON file.

curl 'https://api.twitter.com/2/tweets/search/recent?query=from:TwitterDev' -H "Authorization: Bearer $BEARER_TOKEN" > response.json

Since the key of the JSON object that gets returned from the Twitter API payload is data, you need to set that as the key using the -k flag and send that to a CSV. This returns the data object of the payload which contains the Tweets and IDs.

in2csv -k data response.json > response_csvkit.csv

2. Using the web

Using the web is a good method to get started for newer developers. It also works well if you need to create a CSV quickly. However, this method can fall short if you need to do additional data cleaning and manipulation.

After you set up your bearer token as an environment variable, as outlined in the last section, make a request using curl. This method is similar to how you did it in the last step.

curl 'https://api.twitter.com/2/tweets/search/recent?query=from:TwitterDev' -H "Authorization: Bearer $BEARER_TOKEN"

Cut and paste the JSON response into this website which automatically converts your payload into a CSV. This site was made by Eric Mill.

3. Using requests and pandas in Python

Using Python and pandas is a good framework for getting set up to perform any additional data analysis, data cleaning, and manipulation, or for combining Twitter data with another data source.

First, pip install the packages requests and pandas which you'll use to make your requests and to work with the data from the JSON object. From here, import the packages you'll be using. Since os is part of the standard library you do not need to install it prior.

import requests
import pandas as pd
import os

Next, use the same method outlined previously to set your bearer token as an environment variable. From here, format your bearer as a dictionary to create the headers needed to authenticate to the Twitter API.

bearer_token = os.environ.get('BEARER_TOKEN')
headers = {"Authorization": "Bearer {}".format(bearer_token)}

Now, create a variable to save the URL you are looking to hit and make a request to the endpoint using the requests package. Make sure to add the .json() method to the end of your request to ensure you get a JSON object back.

url = "https://api.twitter.com/2/tweets/search/recent?query=from:TwitterDev"
response = requests.request("GET", url, headers=headers).json()

To create a CSV, create a data frame for the data object which contains the ID and text of the Tweets and send that data frame to a CSV.

df = pd.DataFrame(response['data'])
df.to_csv('response_python.csv')

4. In R using httr

Similar to Python, you have the control to quickly and easily manipulate your data making this a great method for data analysis, data cleaning, or for combining datasets together.

In the console of your R environment, you first want to set your environment variable. I use R studio as my main environment for R.

Sys.setenv(BEARER_TOKEN = "your-bearer-token")

To make HTTP requests, use the package httr. After installing this package, make sure you add a require statement as follows.

require(httr)

Next, grab the bearer token you set, and pass that into your headers for authentication. You need to define the query you are adding to your request as a parameter to pass in as you make your request.

bearer_token <- Sys.getenv("BEARER_TOKEN")
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))
params <- list(`query` = 'from:TwitterDev')

You are now ready to make your request and pass in your headers and query parameters into the request.

response <-
  httr::GET(url = 'https://api.twitter.com/2/tweets/search/recent',
            httr::add_headers(.headers = headers),
            query = params)

At this point, create a data frame for the results from recent search and send the data frame to a CSV.

recent_search_body <-
  content(
    response,
    as = 'parsed',
    type = 'application/json',
    simplifyDataFrame = TRUE
  )
write.csv(recent_search_body, 'response_r.csv')

5. In Node.js using axios and json2csv

Using Node.js allows you the control to work your data in a robust way. Similar to csvkit, json2csv can also be used as a command-line module giving you optionality. This method works particularly well if you are looking to create a backend of a website that has the option of downloading a dataset a CSV.

To achieve this same task in JavaScript, use axios to connect to the Twitter API, json2csv to convert your JSON payload to a CSV, and fs to write a file. First, install the required packages using npm and call them as follows. Similar to os in Python, fs is part of the standard library and you do need to install it first.

const axios = require('axios');
const { parse } = require('json2csv');
const fs = require('fs');

After you set your environment variables on the command line, as you did with csvkit, create a variable for your bearer token and configure your request. Once you have the right parameters, pass your config variable to axios so you are ready to make an HTTP request to the right endpoint.

const bearer_token = process.env.BEARER_TOKEN;
const config = {
  method: 'get',
  url: 'https://api.twitter.com/2/tweets/search/recent?query=from:TwitterDev',
  headers: {
    'Authorization': `Bearer ${bearer_token}`
  }
};
axios(config)

Now, write a promise that will save the response data in a variable called res, set the fields you want for the CSV you are writing, and parse the dataobject of the response into a CSV format. From there, use fs to write the CSV file. Print out the csv object and any errors that may come up to help you troubleshoot if needed.

.then(function (response) {
  const res = response.data;
  const fields = ['id', 'text'];
  const opts = { fields };
  const csv = parse(res.data, opts);
  fs.writeFile('response_nodejs.csv', csv, { flag: 'a+' }, err => {})
  console.log(csv);
})
.catch(function (error) {
  console.log(error);
});

Conclusion

Please reply in the comments with your favorite method for converting JSON to CSV, if it isn't included. The sample code provided above can be found on our GitHub repository. Be sure to let us know on the forums or at @TwitterDev if you run into any troubles along the way or if this list inspires you to create anything new.