Forem: X

Get a video URL from Twitter API v2

Andy Piper — Mon, 15 Aug 2022 14:09:00 +0000

Twitter API v2 has been available for quite a while now, but it's an evolving process as we bring more endpoints and features to the new environment.

One data field that was not initially available at launch was the URL for videos attached to Tweets. This information was added to the data expansions earlier this year, so if you are making a request to the API today, you can now add query parameters to have multiple video variant URLs returned as part of an attached expansion.

If you're into the details, you can always check what functionality is available in the API by grabbing the latest OpenAPI specification document, which lives at https://api.twitter.com/2/openapi.json - you can look at the Changelog for a more high-level list of updates

Twitter API v2 has many improvements over the legacy version, one of them being the concepts of fields and expansions. These are query parameters that control which elements of data are returned on an API call. To get the video information, the data we want to return is part of the media object, contained in variants, which is an array of available display formats.

Building a query

I'll pick a recent Tweet with an attached video from my own timeline at random:

// Detect dark theme var iframe = document.getElementById('tweet-1557438090897137665-590'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1557438090897137665&theme=dark" }

The Tweet ID here is 1557438090897137665 (I got this by taking it from the URL of the Tweet on Twitter, but I could have picked it out from an API query). If we request that Tweet via the API, by default only the the basic fields will be returned (for a Tweet, these are text and id). In this case, we want to get the video URLs that are attached. In oder to do this, we need the following combination of fields and expansions:

tweet.fields - I need to ask for any attachments (the concept here is that videos are attached to Tweets, and so are images, polls, etc)
- expansions - the name of the expansion is attachments.media_keys

These two parameters will now tell the API to return any media object associated with the Tweet inside the includes section of the response, but by default, the default fields for the media object are media_key (the value that links the data to the Tweet itself), and type (video, image, etc). So we need to add third parameter:

media.fields - the video data is in the variants array. It may also be useful to request alt_text (if the user has set alternative text for accessibility), preview_image_url, duration_ms, and possibly also public_metrics so that we can see how many times the video has been viewed.

The final request we've built includes the URL parameters ?tweet.fields=attachments&expansions=attachments.media_keys&media.fields=variants,alt_text,preview_image_url,duration_ms,public_metrics. These parameters can be used on any API calls that are expected to return Tweet objects, and will expand the media to include appropriate video information when available.

Show us some code!

The Twitter team released some beta SDKs for the new API, starting with packages for JavaScript and TypeScript in Node, and also Java.

// Detect dark theme var iframe = document.getElementById('tweet-1503863301276827651-748'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1503863301276827651&theme=dark" }

If you don't code in these languages (or even if you do, but you're familiar with a different SDK), the community has created a range of great libraries to help you code against the Twitter API!

Let's go ahead and use the TypeScript SDK to run our query. You'll need a Twitter Developer Account (if you don't have one yet, sign up here, it's free!), and to create a project containing an app to get a Bearer Token. You'll also need to install the Twitter SDK (add "twitter-api-sdk": "^1.1.0" to your package.json file)

Here's the code to get the same Tweet we looked at before, using the same set of parameters to the API.

import { Client } from "twitter-api-sdk";

if (!process.env.BEARER_TOKEN) throw new Error("Please add your BEARER_TOKEN as an environment variable in the Secrets section");

const client = new Client(process.env.BEARER_TOKEN as string);

async function main() {
  const { data, includes } = await client.tweets.findTweetsById({
    "ids": [
      "1557438090897137665"
    ],
    "tweet.fields": [
      "attachments"
    ],
    "expansions": [
      "attachments.media_keys"
    ],
    "media.fields": [
      "alt_text",
      "duration_ms",
      "preview_image_url",
      "public_metrics",
      "variants"
    ]
  });

  if (!data) throw new Error("Couldn't retrieve Tweet");

  console.log("\nTweet text:\t" + data[0].text);
  console.log("Media:\t\t" + includes.media[0].type);
  console.log("Alt text:\t" + includes.media[0].alt_text);
  console.log("Views:\t\t" + includes.media[0].public_metrics.view_count);

  console.table(includes.media[0].variants, ['bit_rate', 'url']);

  // dump whole JSON data objects
  // console.log("data", JSON.stringify(data, null, 1));
  // console.log("includes", JSON.stringify(includes, null, 1));

}

main();

Let's go through this code. We start out by bringing in the Twitter SDK, and then we read the value of the bearer token in from the environment variables (you can do this via setting up secrets in a cloud runtime, or create a local shell environment variable e.g. export BEARER_TOKEN=your-bearer-token-value at the command line).

Inside our main function, we know the ID of the Tweet we want, so we use client.tweets.findTweetsById to call the GET /2/tweets/:id API, passing in our Tweet ID. We could easily choose to use a search API, or we could request multiple Tweets at once, if we wanted to do so. Note that when we make the call, we request both the data and the includes (since the video information is included under includes, where attachments are added in the response)

We know the parameters we need to get the values, so we add those to the request.

Once the data is returned, we can grab just the values we want to show. In this example we're using console.table to quickly throw the URLs and additional fields into a table, but we could choose the handle it differently. To debug, we can simply dump out the whole JSON format data and includes, but that can stay commented out if we are confident with the data format.

Tweet text: @biglesp @Jolly_Device It blinks! The module that is… not coded yet. https://t.co/ErJABKMFYL
Media:      video
Alt text:   undefined
Views:      13
┌─────────┬──────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ (index) │ bit_rate │                                                     url                                                      │
├─────────┼──────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│    0    │  256000  │    'https://video.twimg.com/ext_tw_video/1557437992389722112/pu/vid/480x270/XRx8UwJoOLZS1AE4.mp4?tag=14'     │
│    1    │  832000  │    'https://video.twimg.com/ext_tw_video/1557437992389722112/pu/vid/640x360/kbjqEmwvIA1_3zFq.mp4?tag=14'     │
│    2    │ 2176000  │    'https://video.twimg.com/ext_tw_video/1557437992389722112/pu/vid/1280x720/1nqh8h8iFoHno66E.mp4?tag=14'    │
│    3    │          │ 'https://video.twimg.com/ext_tw_video/1557437992389722112/pu/pl/e2nunEhaXNarlcHp.m3u8?tag=14&container=fmp4' │
│    4    │ 10368000 │   'https://video.twimg.com/ext_tw_video/1557437992389722112/pu/vid/1920x1080/QXFoJWg4begKhDss.mp4?tag=14'    │
└─────────┴──────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

I didn't add alt text for this video, which was not very user-friendly of me - I'll do better!

That's it - that's the sample. We looked at:

where video information exists in the v2 API response
which API parameters are needed (the expansions and fields added to the request - there are others to experiment with)
quickly getting started with the beta Twitter SDK for TypeScript

I've created a template on Replit that can be forked and modified - add a valid BEARER_TOKEN value to the project secrets. You can also find more examples of using the Twitter SDKs on Replit.

Hope you found this helpful - let us know in the comments!

Evolving the Twitter API

Andy Piper — Tue, 05 Jul 2022 16:30:07 +0000

Have you ever wanted to request an API feature, or ask for additional information in the Twitter API? Read on for information about how you can be a part of the Open Evolution process.

Sign up to build what's next, with a Twitter Developer account

Building in the open

One of the things we have been doing since the inception of Twitter API v2 is building in the open - showing the developer community what we are working on, and asking what we could be doing differently or better. For me, this is one of the most exciting parts of working in Developer Relations - talking to the people who use the platform and product every day, and representing their thoughts and needs back to the engineering teams.

We started out with Twitter Developer Labs, which was the label for our early exploration of the new concepts (data structures and API specifications) that are now foundational in v2. Labs was an early access program that enabled us to test out the new formats, and also to check that our new systems scaled.

We've also been building in the open with features like the Spaces API. Spaces are a product feature that were added while we were working on the new Twitter API, and we were able to run sessions where we asked the community what data and features would be most important in the API, and to respond to that feedback. Most recently, I also hosted two series of Spaces back in March, to talk about potential future additions to the developer platform, around Tweet formats and timeline customisation.

// Detect dark theme var iframe = document.getElementById('tweet-1497258071709696003-296'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1497258071709696003&theme=dark" }

We're grateful to everyone that joined those Spaces! We had some great conversations, and we are continuing to build on what we learned.

Follow @TwitterDev to watch out for future opportunities to join live discussions with us. There's also a monthly Developer Town Hall session where you can learn about the latest updates, and ask questions about the API.

Twitter API Open Evolution

There's also a way to directly influence the technical direction of the API: the Open Evolution process.

It's important to note that this is specific to the Twitter Developer Platform (the API - not to the main Twitter app or website); and also, it is only for the current version of the API. Previous versions may remain supported for use, but new features will not be considered for addition.

You can find the repository and workflow on GitHub.

xdevplatform / open-evolution

Open evolution proposals for the Twitter API

Twitter API Open Evolution

This repository allows you to submit proposals to change aspects of the Twitter public API. You can read more about this process below, and in the announcement post on the Twitter Developer Community forums.

Submitting a proposal

Anyone can submit a proposal for consideration by Twitter. Proposals must conform to a defined template, and must respect the general principles of the Twitter API platform.

Before submitting a proposal, we strongly encourage you to start a discussion by opening an issue on GitHub, and to seek consensus and input from the broader developer community.

Workflow

Fork this repository.
Make a copy of proposal-template.md, and rename it so that the title describes the proposal (it can have the same title as your proposal). Use a format like 000-your-proposal-name.md, where your-proposal-name is a descriptive name (e.g. the proposal title), keeping 000 for the numeric part.
Fill…

View on GitHub

How to submit a proposal

A proposal needs to conform to a specific template, provided in the repository; and it also needs to align with the general principles of the Twitter API platform, the fundamental rules of how we approach design of the public / external Twitter API. You'll see that these are around security, consistency, and a few other features - make sure to read the principles in detail, to understand the background.

If you're not sure about how your proposal would fit, you can open a conversation via GitHub Issues or on the developer forums to get things started, and to get feedback from other members of the community. The core process is driven by a Pull Request using the template, to define the requirement and proposal.

The process is:

fork the Open Evolution repository;
complete the template with your proposal, named so that it describes the suggested addition or change;
submit a Pull Request containing the proposal;
await a review from our team (we aim to get back to you within 15 working days if possible, although this may vary slightly with availability).

Proposals can be approved, or rejected. If the review process results in approval, this means that the changes are agreed in principle, but does not mean that they will be immediately implemented, and they may have to wait for a future developer platform revision.

As an example of how this can go, one of the Twitter Developer Insiders, @IgorBrigadir, made a great proposal requesting improvements to the way that Retweets are represented in the API. You do not have to be an Insider to get involved in the process, though! We welcome all proposals that meet the criteria described in the repository, and we love to learn more about how you use the API and what you might want to build with it.

Resources and next steps

Learn about the Chirp Developer Challenge, and enter for your chance to improve the Twitter experience for users, and to win prizes
Read our DEV posts and explore the #twitter topic for examples of how to use Twitter API v2
Check out our code examples on GitHub
Find a local Twitter community group via our website
Follow us on Twitter @TwitterDev, and you can find other members of the community in our developer forums.

Handling Refresh Tokens in the OAuth 2.0 Authorization Code Flow with PKCE with Flask

Jessica Garson — Fri, 17 Jun 2022 20:42:05 +0000

We recently released OAuth 2.0 Authorization Code Flow with PKCE for use with most of our v2 endpoints. As a backend-focused Python developer, I struggled with the flow since much of the code I like to write focuses on automation tasks.

This tutorial will walk you through critical concepts related to refresh tokens using v2 of the Twitter API. If you don’t already have access to the Twitter API, you can sign up for an account. You will also need to have OAuth 2.0 turned on in your App’s settings.

Tokens are only valid for two hours

I was used to working with OAuth 1.0a, where keys and tokens are available for use until they are revoked. This is not the case with OAuth 2.0 Authorization Code Flow with PKCE. Access Tokens generated using this flow are only valid for 2 hours.

Refresh tokens

Inside the token object generated from the consent flow is a refresh token. A refresh token allows an application to obtain a new access token without prompting the user to log in again.

To generate a refresh token, you must set a scope for offline access. If I was using the manage Tweets endpoint and I wanted to Tweet on behalf of a user every six months, I’d use the following line of Python code:

scopes = ["tweet.read", "users.read", "tweet.write", "offline.access"]

How long are refresh tokens valid for?

Refresh tokens stay valid for six months, so you will want to refresh them at least that often or more regularly.

Generating tokens

To generate a token using OAuth 2.0 Authorization Code Flow with PKCE you can use a method similar to this example that allows you to look up an authenticated user’s bookmarks. Since the authenticated user will need to log in directly, and you will need to parse the response, I found creating a Flask application allowed for a more automated process.

The following code will generate a token and save the token into a dictionary.

You will be working with the following libraries:

requests for making HTTP requests
requests_oauthlib for working with OAuth 2.0
os for parsing environment variables and creating random strings
re, base64 and hashlib to create the code challenge and code verifier
flask for creating a web framework

If you don’t already have flask, requests_oauthlib and requests installed you will need to install these libraries.

You will first need to import the following libraries:

import base64
import hashlib
import os
import re
import requests

from requests_oauthlib import OAuth2Session
from flask import Flask, request, redirect, session, url_for

First, you will need to set a variable for your app, which is the start of every Flask app. Additionally, you will need to set a secret key for your app to be a random string.

app = Flask(__name__)
app.secret_key = os.urandom(50)

The two main credentials you will need to authenticate with OAuth 2.0 Authorization Code Flow with PKCE are Client ID and Client Secret. You can set these as environment variables to ensure security.

client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")

You will also need to set a variable for your redirect URL, this should be the same value as your callback URL in your App’s settings in the Developer Portal.

redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

To define the permissions of your App, you will need to set the scopes of your application. Check out the authentication mapping guide to determine what scopes you will need based on the endpoints you are using. The scope offline.access allows you to generate refresh tokens. The scopes for this demo are tweet.read and users.read, which gives you access to read Tweets and obtain information about users.

scopes = ["tweet.read", "users.read", "offline.access"]

You will need to set a code verifier which is a secure random string. The code verifier is used to create the code challenge.

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier

With PKCE the code challenge is a base64 encoded string of the SHA256 hash of the code verifier.

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")

Now you can start with creating a login page that will be the first page you will visit when you run your application. @app.route("/") indicates it’s the first landing page. This page will be the page that an authenticated user logs into.

@app.route("/")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)

After the user logs in, they will be directed to the callback. In the callback you will generate an object called token that contains a series of tokens that includes an access token and a refresh token.

@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )

Now that you’ve generated a token object, you can now make a request to the authenticated user lookup, to obtain a user ID that can be used to access many of our v2 Users endpoints such as manage Tweets or manage blocks.
The access_token variable inside of the token object is the bearer token you’d use to connect to any of the endpoints that support OAuth 2.0 Authorization Code Flow with PKCE.

    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]

Now that you’ve created a token and saved into a dictionary you can now access the token dictionary and create a refreshed token through this process. This code should be added to your callback method. This refreshed_token will contain a new access token as well.

tokens = {"new_token": token}
    t = tokens["new_token"]
    refreshed_token = twitter.refresh_token(
          client_id=client_id,
          client_secret=client_secret,
          token_url=token_url,
          refresh_token=t["refresh_token"],
        )
    tokens.update({"new_token": refreshed_token})
    return "You should now have a refreshed token"


if __name__ == "__main__":
    app.run()

Full code

Here is the full code that you can save as app.py

import base64
import hashlib
import os
import re
import requests

from requests_oauthlib import OAuth2Session
from flask import Flask, request, redirect, session, url_for


app = Flask(__name__)
app.secret_key = os.urandom(50)


client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")

redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

scopes = ["tweet.read", "users.read", "offline.access"]

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")


@app.route("/")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)


@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )
    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]
    tokens = {"new_token": token}
    t = tokens["new_token"]
    refreshed_token = twitter.refresh_token(
          client_id=client_id,
          client_secret=client_secret,
          token_url=token_url,
          refresh_token=t["refresh_token"],
        )
    tokens.update({"new_token": refreshed_token})
    return "You should now have a refreshed token"


if __name__ == "__main__":
    app.run()

To run the file locally you can run the following line:

python app.py

Next steps

Hopefully, this can be a starting point for you to get started with generating refresh tokens. As a next step, if you are using Flask you may want to consider using a schedular to update your refresh tokens regularly in an automated fashion. Additionally you may want to consider saving your tokens to a database in a secure fashion. This code sample can also be extended to allow you to connect to any of the endpoints that support v2 and can be deployed to a server as part of a more complete application.

Be sure to let us know on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

A guide to working with the Twitter API v2 in Java using twitter-api-java-sdk

Suhem Parack — Fri, 17 Jun 2022 17:54:00 +0000

twitter-api-java-sdk is Twitter's official SDK for Java that supports the Twitter API v2 and the academic research product track. In this guide, we will learn how to use the various functionality available in the Twitter API v2, using this twitter-api-java-sdk.

Prerequisite

In order to work with the Twitter API, you need to have a developer account and your API keys and tokens to connect to the API. Learn more on how to obtain these keys and tokens here.

In order to work with twitter-api-java-sdk, make sure you have Java installed on your machine. If you use Maven, add the following dependency:

<dependency>
  <groupId>com.twitter</groupId>
  <artifactId>twitter-api-java-sdk</artifactId>
  <version>2.0.0</version>
</dependency>

For all the examples in this section, you will first need to create an instance of the TwitterApi class and pass it your bearer token. Once you have the instance set up, you will be ready to start using the various functions in this library.

TwitterApi apiInstance = new TwitterApi();
TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
apiInstance.setTwitterCredentials(credentials);

1. Searching for Tweets from the last 7 days

In order to search Tweets from the last 7 days, you can use the tweetsRecentSearch method. You will have to pass it a search query to specify the data that you are looking for. In the example below, we are searching for Tweets from the last 7 days that contain the term covid and we are excluding retweets using -is:retweet.

By default, in your response, you will only get the Tweet ID and Tweet text for each Tweet. If you need additional Tweet fields such as context_annotations, created_at (the time the tweet was created) etc., you can specifiy those fields using the tweetFields parameter, as shown in the example 2. Learn more about available fields here. By default, a request returns 10 Tweets. If you want more than 10 Tweets per request, you can specify that using the maxResults parameter. The maximum Tweets per request for this recent search endpoint is 100.

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        int maxResults = 100;

        TweetSearchResponse result = apiInstance.tweets().tweetsRecentSearch(query, null, null, null, null, maxResults,
                null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                System.out.println(tweet.getText());
            }

        }

    }

}

2. Searching for Tweets from the full-archive of public Tweets

If you have access to the academic research product track, you can get Tweets older than 7 days i.e. from the full archive of publicly available Tweets. In order to get these Tweets, make sure to use the bearer token from an app connected to your academic project in the Twitter developer portal. You can use the tweetsFullarchiveSearch method and pass it the search query. As shown in the example below, you can also specify additional Tweet fields such as created_at, lang etc. that you'd like returned for your request to the Twitter API.

Note: Not all Tweets have context_annotations associated with them, which is why, in the sample below, we only print the context_annotations if those are available for a Tweet.

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "context_annotations"));

        String query = "covid -is:retweet";

        int maxResults = 500;

        TweetSearchResponse result = apiInstance.tweets().tweetsFullarchiveSearch(query, null, null, null, null,
                maxResults, null, null, null, null, tweetFields, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                if (tweet.getContextAnnotations() != null) {
                    for (ContextAnnotation contextAnnotation : tweet.getContextAnnotations()) {
                        System.out.println(contextAnnotation.getDomain().getName());
                        System.out.println(contextAnnotation.getEntity().getName());
                    }
                }
            }

        }

    }

}

3. Getting Tweets from the full-archive of public Tweets for a specific time-frame

If you want to get Tweets from the full-archive for a specific time-period, you can specify the time-period using the startTime and endTime parameters, as shown in the example below:

import java.time.OffsetDateTime;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        int maxResults = 100;

        OffsetDateTime startTime = OffsetDateTime.parse("2020-01-01T00:00:00Z");

        OffsetDateTime endTime = OffsetDateTime.parse("2021-01-01T11:59:59Z");

        TweetSearchResponse result = apiInstance.tweets().tweetsFullarchiveSearch(query, startTime, endTime, null, null,
                maxResults, null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                System.out.println(tweet.getText());
            }

        }

    }

}

4. Writing Tweets to a text file

This example shows how you can write the Tweet IDs for each Tweet obtained for a search result, to a text file. Make sure to replace the fileName with the a name of your chosing. If you wish to write other fields to the text file, make sure to adjust the script below accordingly.

import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.io.UnsupportedEncodingException;
import java.time.OffsetDateTime;
import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException, FileNotFoundException, UnsupportedEncodingException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        int maxResults = 100;

        String fileName = "tweet_ids.txt";

        TweetSearchResponse result = apiInstance.tweets().tweetsFullarchiveSearch(query, null, null, null, null,
                maxResults, null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            PrintWriter writer = new PrintWriter(fileName, "UTF-8");
            for (Tweet tweet : result.getData()) {
                writer.println(tweet.getId());
            }
            writer.close();
        }

    }

}

5. Getting Tweets with user information, for each Tweet

In the examples so far, we have seen how to get Tweet information for the Tweets that we have searched for. However, if you need additional information associated with the Tweet, such as the information of the user Tweeting it, you can use expansions and pass it to the tweetsRecentSearch method.

In this example, we want user information, which is why the value for expansions that we pass is author_id. Additionally, we will specify the userFields that we want returned such as profile_image_url, description (aka user's bio) etc. Learn more about available fields here.

From the response, we will get the users list from the includes object. Then, we will look up the user information for each Tweet, using the author_id of the Tweet.

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        // Replace with your own bearer token below
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "conversation_id"));

        Set<String> userFields = new HashSet<>(Arrays.asList("description", "profile_image_url"));

        Set<String> expansions = new HashSet<>(Arrays.asList("author_id"));

        // Replace with your own query below
        String query = "covid -is:retweet";

        TweetSearchResponse result = apiInstance.tweets().tweetsRecentSearch(query, null, null, null, null, 100, null,
                null, null, expansions, tweetFields, userFields, null, null, null);

        // Create a map with User ID and User Object
        Map<String, User> userMap = new HashMap<>();
        if (result.getIncludes() != null && result.getIncludes().getUsers() != null) {
            List<User> users = result.getIncludes().getUsers();
            for (User user : users) {
                userMap.put(user.getId(), user);
            }
        }

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                User user = userMap.get(tweet.getAuthorId());
                System.out.println(tweet.getId());
                System.out.println(user.getUsername());
                System.out.println(user.getDescription());
            }

        }

    }

}

6. Searching for geo-tagged Tweets

Some Tweets have geographic information associated with them. The search endpoints in the Twitter API v2 support operators such as place, place_country, bounding_box, point_radius etc. (these operators are currently only available in the academic research product track) that allow you to get these geo-tagged Tweets.

In the example below, we want all Tweets that are from the country US, which is why we specify it in our search query using the place_country:US operator. Now because by default, only the Tweet ID and Tweet text are returned, we have to specify the fact that we want the geo information in our response as well. We do this by setting the expansions to geo.place_id in the tweetsRecentSearch method and we also specify the geo fields that we are looking for.

In our response, we get the list of places from the includes object, and we match on the place_id to get the relevant geo information associated with the Tweet (the place name in the example below)

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "geo"));

        Set<String> placeFields = new HashSet<>(Arrays.asList("name", "place_type"));

        Set<String> expansions = new HashSet<>(Arrays.asList("geo.place_id"));

        String query = "covid -is:retweet place_country:US";

        TweetSearchResponse result = apiInstance.tweets().tweetsRecentSearch(query, null, null, null, null, 100, null,
                null, null, expansions, tweetFields, null, null, placeFields, null);

        // Create a map with Place ID and Place Object
        Map<String, Place> placeMap = new HashMap<>();
        if (result.getIncludes() != null && result.getIncludes().getPlaces() != null) {
            List<Place> places = result.getIncludes().getPlaces();
            for (Place place : places) {
                placeMap.put(place.getId(), place);
            }
        }

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                Place place = placeMap.get(tweet.getGeo().getPlaceId());
                System.out.println(tweet.getId());
                System.out.println(place.getName());
            }

        }

    }

}

7. Getting Tweet counts (volume) for a search query

So far, we have seen examples of how to get Tweets for a search query. However, if you want to get the count of Tweets (volume) for a search term, you can use the tweetCountsRecentSearch function. You can pass it the search query and specify the granularity for the data aggregation e.g. if you want the daily count, hourly count or 15-minute count for a search term.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        TweetCountsResponse result = apiInstance.tweets().tweetCountsRecentSearch(query, null, null, null, null, null,
                null, Granularity.DAY);

        if (result.getData() != null) {
            for (SearchCount searchCount : result.getData()) {
                System.out.println(searchCount.getTweetCount());
            }

        }

    }

}

8. Getting a user's timeline

In order to get the most recent 3200 Tweets from a user's timeline, we can use the usersIdTweets method and pass it the user id. We can also specify additional fields that we want using the tweetFields and expansions (as shown in the examples above).

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        int maxResults = 100;

        GenericTweetsTimelineResponse result = apiInstance.tweets().usersIdTweets(userID, null, null, maxResults, null,
                null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
            }
        }

    }

}

9. Getting a user's mentions

In order to get the most recent 3200 mentions for a user, we can use the get_users_musersIdMentionsentions method and pass it the user id. We can also specify additional fields that we want using the tweetFields and expansions (as shown in the examples above).

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        int maxResults = 100;

        GenericTweetsTimelineResponse result = apiInstance.tweets().usersIdMentions(userID, null, null, maxResults,
                null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
            }
        }

    }

}

10. Getting a user's followers

In order to get the followers of a user, we can use the usersIdFollowers function and pass it the user ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        GenericMultipleUsersLookupResponse result = apiInstance.users().usersIdFollowers(userID, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

11. Getting users that a user follows

In order to get the list of users that a user follows, we can use the usersIdFollowing function and pass it the user ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        UsersFollowingLookupResponse result = apiInstance.users().usersIdFollowing(userID, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

12. Getting users that like a Tweet

In order to get the list of users that like a Tweet, we can use the tweetsIdLikingUsers function and pass it the Tweet ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String tweetId = "20";

        GenericMultipleUsersLookupResponse result = apiInstance.users().tweetsIdLikingUsers(tweetId, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

13. Getting users that retweeted a Tweet

In order to get the list of users that retweeted a Tweet, we can use the tweetsIdRetweetingUsers function and pass it the Tweet ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String tweetId = "20";

        GenericMultipleUsersLookupResponse result = apiInstance.users().tweetsIdRetweetingUsers(tweetId, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

14. Getting Tweets that a user liked

In order to get the list of Tweets that a user liked, we can use the usersIdLikedTweets function and pass it the User ID. If we want additional fields for the Tweet object such as context_annotations, created_at etc. we can specify those using the tweetFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userId = "2244994945";

        UsersIdLikedTweetsResponse result = apiInstance.tweets().usersIdLikedTweets(userId, null, null, null, null,
                null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
            }
        }

    }

}

15. Lookup Tweets using Tweet IDs

In some cases, if we have a list of Tweet IDs and want to build the Tweet dataset for these Tweet IDs, we can use the findTweetsById function and pass it the list of Tweet IDs. By default, only the Tweet ID and Tweet text are returned. If we want additional Tweet fields, we can specify those using tweetFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        List<String> tweetIds = Arrays.asList("1460323737035677698", "1519781379172495360", "1519781381693353984");

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "context_annotations"));

        MultiTweetLookupResponse result = apiInstance.tweets().findTweetsById(tweetIds, null, tweetFields, null, null,
                null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                System.out.println(tweet.getLang());
                System.out.println(tweet.getCreatedAt().toString());
            }
        }

    }

}

These are some common examples of working with the Twitter API v2 using twitter-api-java-sdk.

If you have any questions or feedback about this guide, please feel free to reach out to me on Twitter!

Tweet to Super Followers with Postman, OAuth 2.0 and Manage Tweets

Jessica Garson — Mon, 28 Feb 2022 17:12:56 +0000

We recently launched Manage Tweets with a feature to Tweet directly to Super Follows. This tutorial will walk you through how to get started using the Twitter API to Tweet to your subscribers using OAuth 2.0 and Postman.

What is Super Follows?

Super Follows is an opportunity to earn monthly revenue by sharing subscriber-only content with your most engaged followers on Twitter. Through Super Follows, creators like you can create a monthly subscription of $2.99, $4.99, or $9.99 a month, where you could give access to exclusive content or insights into your creative process for creating your bot. In addition, super Followers enjoy extra special access to unfiltered thoughts, early previews, and subscriber-only conversations from their favorite accounts.

How to Apply to Super Follows

You will need to have Super Follows enabled on your account to follow this tutorial. To apply to set up a Super Follows subscription, you will need to follow these steps:

Swipe open the sidebar on your home timeline
Tap on “Monetization
”Select “Super Follows”

After you apply, our team will review your application shortly. You will also need to sign up for a developer account if you don’t already have one.

Setting up Postman

You will first want to download Postman if you have not already done so. After you have Postman enabled, you will want to download the Twitter API v2 collection. To learn more about getting started with the Postman and the Twitter API, check out our documentation on the subject.

Generating an OAuth 2.0 Token

For this tutorial, you will use OAuth 2.0 with PKCE to make requests on behalf of the Super Followers enabled account. To authenticate using OAuth 2.0, you must create an OAuth 2.0 token using the following step. First, In the “Manage Tweets” folder of the Twitter API v2 collection, go to the tab entitled “Authorization” and select “OAuth 2.0”.

In this tab, be sure to follow these steps:

Name your token
Select the Grant Type as Authorization Code (with PKCE)
Update the parameters:
- Auth URL - https://twitter.com/i/oauth2/authorize
- Access Token URL - https://api.twitter.com/2/oauth2/token
- Callback URL - https://www.example.com (matching the callback URL I set in my auth settings page in the Developer Portal)
- Client ID - Cut and paste OAuth 2.0 client ID from the Developer Portal
Review the proper scopes for Manage Tweets on this page.
Update the scopes with the following values: tweet.read users.read tweet.write
Populate the field state with “State”
Click where it says “Generate Token”
Press the save icon to save the folder changes.

You may get a message that you are not logged into Twitter. If you get this error, you will need to log in to the Twitter account inside of Postman you are trying to post on behalf of.

Tweet to your Super Followers

In the Postman Collection, you should find a sample request for “Manage Tweets.” Select the POST request called “Create a Tweet.”

You will want to go into that request and change the “Authorization” tab of the request and select “Inherit auth from parent” to ensure you are using the token you created in the last step.

Next, update the “Body” tab of the request to the following:

{
   "text": "Hello, Super Followers!",
   "for_super_followers_only": true
}

You can now press the blue button that says “Send” and see a response that looks like this one.

{
   "data": {
       "id": "1491508991335272449",
       "text": "Hello, Super Followers!"
   }
}

Next steps

Hopefully, this can be a starting point for you to get started with Tweeting programmatically to Super Followers using Postman and OAuth 2.0. To generate a code sample in the language of your choosing, click on the button in Postman that says “Code." Additionally, you may want to write some language to schedule specific Tweets on a schedule to your subscribers and run your code on a server.

Be sure to let us know on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Retrieve a list of user mentions from a thread of Tweet replies

Aurelia Specker — Tue, 25 Jan 2022 17:10:07 +0000

The code for this tutorial is available on GitHub.

Have you ever wanted a quick way to retrieve all usernames mentioned in a thread of replies on Twitter? This Python script does just that using the Twitter API.

Set up

Clone the repository here.
Ensure you have a Twitter developer account with access to GET /2/tweets/search/all. If you don't have access to this endpoint, follow the instructions below to use the recent search endpoint instead. More information on getting access to the API is available here.
In the root directory, rename .env_example to .env and add the credentials for your developer App in between the "". Remember to add .env to your .gitignore file to keep your credentials secure.

Running the code

Use the following command to run the code:

$ python3 replies.py -t <TWEET-ID>

The following flags are available to use:

Flags
`-t`	Tweet ID	Required	ID of the Tweet for which you want to pull data.
`-h`	Help	Optional	View a list of available flags and a description for each of these.
`-s`	Start time	Optional	The oldest UTC timestamp from which the replies will be provided. Format: YYYY-MM-DDTHH:mm:ssZ; for example: 2021-12-04T01:30:00Z. If unspecified, will default to returning replies from up to 30 days ago.
`-e`	End time	Optional	The newest, most recent UTC timestamp to which the replies will be provided. Format: YYYY-MM-DDTHH:mm:ssZ; for example: 2021-12-04T01:30:00Z. If unspecified, will default to [now - 30 seconds].

Output

Returned in the command life interface

Ordered dictionary containing a list of all usernames present in the replies and the number of times each username was mentioned. Note: the user who authored the original Tweet is not included.
Number of replies to the Tweet.
Number of usernames mentioned in these replies.
Number of requests made to the Search Tweets endpoint to pull all replies.

Returned in a new file

A new file, replies.txt, will be created. This file will contain a list of Tweet IDs for the replies.

What if I don't have access to Twitter's full-archive search endpoint?

If you only have access to the recent search endpoint, you can still use this script. Simply make the following two modifications to replies.py:

On line 63, change the endpoint to:

search_url = "https://api.twitter.com/2/tweets/search/recent"

On line 49, make sure the max_results parameter is set to:

"max_results": "500"

When you run the code, if you specify a start time and/or end time using the -s and -e flags, these must be within the last seven days, otherwise you will get back a 400 Bad Request error.

You can read more about Search Tweets and the difference between the recent search and full-archive search endpoints here: https://developer.twitter.com/en/docs/twitter-api/tweets/search/introduction

Use case examples

The above can be helpful in the following scenarios:

You've asked a question on Twitter and want to quickly retrieve a list of all accounts mentioned in the replies.
Identify a list of influencers, as mentioned in a conversation thread.

Getting the follower count from Twitter API v2

Andy Piper — Thu, 20 Jan 2022 15:18:23 +0000

Version 2 of the Twitter API has been the primary way to access the developer platform since November, but, with around 10 years of blog tutorials about v1.1, there is a lot less information shared by the developer community about the new version. Let's start to change that!

Here's a great question I just answered in the Twitter tag on Stack Overflow:

"How do I find the number of followers for a Twitter user in API v2?"

You can see the full question (and my answer) embedded below. I thought I'd post here in order to expand on my response a little.

Jan 19 '22 Comments: 1 Answers: 2

I am surprised to see that there is no way to simply get the number of followers for one Twitter account.

I found many answers (Follower count number in Twitter, How to obtain follower count in Twitter API 1.1?, How to get followers count from twitter…

Open Full Question

The answer is quite simple, but not everyone is familiar with the new data formats yet. In v2, we have reduced the number of fields returned by default, and you can opt to select the information that is useful for your application, using the fields and expansions parameters.

The follower and following counts for a user are part of the public_metrics fields in the User object.

The API endpoint and parameters are in the format

https://api.twitter.com/2/users/[ID]?user.fields=public_metrics,[any other fields]

Below is an example of the output, using the twurl command line tool (I wrote a post about twurl here). In this case, twurl handles the authentication etc, but you could also use the Twitter API v2 Postman Collection, or the API Explorer from the API Tools available on the developer website, or a library of your choice.

$ twurl -j "/2/users/786491?user.fields=public_metrics,created_at"
{
  "data": {
    "id": "786491",
    "username": "andypiper",
    "created_at": "2007-02-21T15:14:48.000Z",
    "name": "andypiper.xyz",
    "public_metrics": {
      "followers_count": 16570,
      "following_count": 3247,
      "tweet_count": 134651,
      "listed_count": 826
    }
  }
}

In this call, we've requested user ID 786491 (which is me! give me a follow, if you feel like it). By default without any parameters, calls to the users lookup endpoint only return the id, name and username values. Here, we added the user.fields to also ask for the public_metrics - followers, following, number of Tweets I've posted, and the number of lists the account has been added to.

So the answer to the original question: to get the follower count, ask for the public_metrics field in the User object, and access data.public_metrics.followers_count in the response.

Let's take it one step further. Expansions and fields are powerful. Using an expansion, we can also ask for my current pinned Tweet and more details about the contents of that Tweet, in a single API call:

$ twurl -j "/2/users/786491?user.fields=public_metrics,created_at,pinned_tweet_id&expansions=pinned_tweet_id&tweet.fields=created_at,public_metrics,source,context_annotations,entities"
{
  "data": {
    "public_metrics": {
      "followers_count": 16571,
      "following_count": 3247,
      "tweet_count": 134654,
      "listed_count": 826
    },
    "pinned_tweet_id": "1482011997814374406",
    "id": "786491",
    "username": "andypiper",
    "created_at": "2007-02-21T15:14:48.000Z",
    "name": "andypiper.xyz"
  },
  "includes": {
    "tweets": [
      {
        "public_metrics": {
          "retweet_count": 31,
          "reply_count": 5,
          "like_count": 547,
          "quote_count": 1
        },
        "entities": {
          "urls": [
            {
              "start": 113,
              "end": 136,
              "url": "https://t.co/lCaoNcXcyL",
              "expanded_url": "https://dev.to/andypiper/what-resources-help-you-to-learn-a-new-api-2cc9",
              "display_url": "dev.to/andypiper/what…",
              "images": [
                {
                  "url": "https://pbs.twimg.com/news_img/1482011998338654212/0NiDLEms?format=jpg&name=orig",
                  "width": 1000,
                  "height": 500
                },
                {
                  "url": "https://pbs.twimg.com/news_img/1482011998338654212/0NiDLEms?format=jpg&name=150x150",
                  "width": 150,
                  "height": 150
                }
              ],
              "status": 200,
              "title": "What resources help you to learn a new API?",
              "description": "In my day job, when I'm not learning things by reading and sharing here on DEV, I'm a developer...",
              "unwound_url": "https://dev.to/andypiper/what-resources-help-you-to-learn-a-new-api-2cc9"
            }
          ],
          "annotations": [
            {
              "start": 97,
              "end": 103,
              "probability": 0.9056,
              "type": "Product",
              "normalized_text": "Twitter"
            }
          ]
        },
        "text": "I'd love to know more about how you learn to use new APIs, and how I can help you to learn about Twitter API v2. https://t.co/lCaoNcXcyL",
        "id": "1482011997814374406",
        "source": "Twitter Web App",
        "context_annotations": [
          {
            "domain": {
              "id": "46",
              "name": "Brand Category",
              "description": "Categories within Brand Verticals that narrow down the scope of Brands"
            },
            "entity": {
              "id": "781974596752842752",
              "name": "Services"
            }
          },
          {
            "domain": {
              "id": "47",
              "name": "Brand",
              "description": "Brands and Companies"
            },
            "entity": {
              "id": "10045225402",
              "name": "Twitter"
            }
          }
        ],
        "created_at": "2022-01-14T15:29:29.000Z"
      }
    ]
  }
}

🤯

OK, that's a lot more information! Let's break it down.

Here, I did the following:

added the user.field to request pinned_tweet_id, which returned "pinned_tweet_id": "1482011997814374406"
also, I asked for this to be expanded in the response (expansions=pinned_tweet_id). That gets expanded inside of the includes array in the response.
by default, the API only returns the id and text fields for a Tweet object, so I also specified that I wanted more information. I asked for tweet.fields and specified created_at, public_metrics, source, context_annotations, and entities
- created_at: creation time of the Tweet, the UTC time when I posted it.
- public_metrics: similar to metrics on the User object, we get data about the numbers of retweets, replies, likes and quotes of this Tweet.
- source: the name of the App that was used to post the Tweet (the Twitter website in this case)
- context_annotations: OK, this is super cool. We get back an array of information (domain/entity pairs) that cover what Twitter has detected this Tweet to be about. In this case I was posting about the Twitter API, so it pulled out that I'm talking about a brand and company (Twitter) in the Tweet. There are also domains and entities for things like movies, sports, products, award shows, and so on. You can even search for Tweets by entity value using the v2 API - very powerful.
- entities: this is also an improvement / additional set of data over v1.1. The entities identified in my original Tweet are broken out and expanded. In this case, there was a shortened URL to a blog post here on DEV. This is expanded to show the full URL, and what is displayed in the Tweet, as well as the URL to the image that is displayed in the website summary large image card (and the dimensions of that image), and the description from that card.

Read more about the new Twitter data formats and Tweet payloads on the Developer Blog. Talk to us via the Twitter Developer Community forums.

Share what you learn about Twitter API v2 here on DEV! Use the twitter tag. It will help others to learn, too.

Finally, if you found this interesting or helpful, I'd love to hear from you in the comments, or on Twitter. Happy coding! 💻

A comprehensive guide for using the Twitter API v2 with Tweepy in Python

Suhem Parack — Wed, 29 Sep 2021 14:57:33 +0000

Tweepy is a popular package in Python used by students, researchers and developers for interacting with the Twitter API. Recently, the version 4.0 of this package was released that supports the Twitter API v2 and the academic research product track. In this guide, we will learn how to use the various functionality available in the Twitter API v2, using Tweepy.

Note: At the time of writing this guide, the streaming endpoints in the Twitter API v2 (such as filtered stream and sampled stream) are not supported yet in Tweepy, but will be added in the future.

The complete documentation for Tweepy can be found here

Prerequisite

In order to work with the Twitter API, you need to have a developer account and your API keys and tokens to connect to the API. Learn more on how to obtain these keys and tokens here.

In order to work with Tweepy, make sure you have Python installed on your machine. Then, in the terminal run:

pip3 install tweepy

Note: If you already have Tweepy installed, you can upgrade it to the latest version i.e. 4.0 by running:

pip3 install tweepy --upgrade

For all the examples in this section, you will first need to import the tweepy library, then you will need to initialize the client by passing it your bearer token. Once you have the client initialized, you will be ready to start using the various functions in this library.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

1. Searching for Tweets from the last 7 days

In order to search Tweets from the last 7 days, you can use the search_recent_tweets function available in Tweepy. You will have to pass it a search query to specify the data that you are looking for. In the example below, we are searching for Tweets from the last days from the Twitter handle suhemparack and we are excluding retweets using -is:retweet.

By default, in your response, you will only get the Tweet ID and Tweet text for each Tweet. If you need additional Tweet fields such as context_annotations, created_at (the time the tweet was created) etc., you can specifiy those fields using the tweet_fields parameter, as shown in the example below. Learn more about available fields here. By default, a request returns 10 Tweets. If you want more than 10 Tweets per request, you can specify that using the max_results parameter. The maximum Tweets per request is 100.

Note: Not all Tweets have context_annotations associated with them, which is why, in the sample below, we only print the context_annotations if those are available for a Tweet.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'from:suhemparack -is:retweet'

tweets = client.search_recent_tweets(query=query, tweet_fields=['context_annotations', 'created_at'], max_results=100)

for tweet in tweets.data:
    print(tweet.text)
    if len(tweet.context_annotations) > 0:
        print(tweet.context_annotations)

2. Searching for Tweets from the full-archive of public Tweets

If you have access to the academic research product track, you can get Tweets older than 7 days i.e. from the full archive of publicly available Tweets. In order to get these Tweets, make sure to use the bearer token from an app connected to your academic project in the Twitter developer portal. You can use the search_all_tweets method and pass it the search query. As shown in the example above, you can also specify additional Tweet fields etc. that you'd like returned for your request to the Twitter API.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'from:suhemparack -is:retweet'

tweets = client.search_all_tweets(query=query, tweet_fields=['context_annotations', 'created_at'], max_results=100)

for tweet in tweets.data:
    print(tweet.text)
    if len(tweet.context_annotations) > 0:
        print(tweet.context_annotations)

3. Getting Tweets from the full-archive of public Tweets for a specific time-frame

If you want to get Tweets from the full-archive for a specific time-period, you can specify the time-period using the start_time and end_time parameters, as shown in the example below:

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'from:suhemparack -is:retweet'

# Replace with time period of your choice
start_time = '2020-01-01T00:00:00Z'

# Replace with time period of your choice
end_time = '2020-08-01T00:00:00Z'

tweets = client.search_all_tweets(query=query, tweet_fields=['context_annotations', 'created_at'],
                                  start_time=start_time,
                                  end_time=end_time, max_results=100)

for tweet in tweets.data:
    print(tweet.text)
    print(tweet.created_at)

4. Getting more than 100 Tweets at a time using paginator

As seen in the examples above, you can get upto 100 Tweets per request. If you need more than 100 Tweets, you will have to use the paginator and specify the limit i.e. the total number of Tweets that you want.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'covid -is:retweet'

# Replace the limit=1000 with the maximum number of Tweets you want
for tweet in tweepy.Paginator(client.search_recent_tweets, query=query,
                              tweet_fields=['context_annotations', 'created_at'], max_results=100).flatten(limit=1000):
    print(tweet.id)

5. Writing Tweets to a text file

This example shows how you can write the Tweet IDs for each Tweet obtained for a search result, to a text file. Make sure to replace the file_name with the a name of your chosing. If you wish to write other fields to the text file, make sure to adjust the script below accordingly.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'covid -is:retweet'

# Name and path of the file where you want the Tweets written to
file_name = 'tweets.txt'

with open(file_name, 'a+') as filehandle:
    for tweet in tweepy.Paginator(client.search_recent_tweets, query=query,
                                  tweet_fields=['context_annotations', 'created_at'], max_results=100).flatten(
            limit=1000):
        filehandle.write('%s\n' % tweet.id)

6. Getting Tweets with user information, for each Tweet

In this example, we want user information, which is why the value for expansions that we pass is author_id. Additionally, we will specify the user_fields that we want returned such as profile_image_url etc. Learn more about available fields here.

From the response, we will get the users list from the includes object. Then, we will look up the user information for each Tweet, using the author_id of the Tweet.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'covid -is:retweet'

tweets = client.search_recent_tweets(query=query, tweet_fields=['context_annotations', 'created_at'],
                                     user_fields=['profile_image_url'], expansions='author_id', max_results=100)

# Get users list from the includes object
users = {u["id"]: u for u in tweets.includes['users']}

for tweet in tweets.data:
    if users[tweet.author_id]:
        user = users[tweet.author_id]
        print(user.profile_image_url)

7. Getting Tweets with media information, for each Tweet

In this example, we want media information, which is why the value for expansions that we pass is attachments.media_keys. Additionally, we will specify the media_fields that we want returned such as preview_image_url etc. Learn more about available fields here.

From the response, we will get the media list from the includes object. Then, we will look up the media information for each Tweet, using the media_keys from the attachements for the Tweet.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'covid -is:retweet has:media'

tweets = client.search_recent_tweets(query=query, tweet_fields=['context_annotations', 'created_at'],
                                     media_fields=['preview_image_url'], expansions='attachments.media_keys',
                                     max_results=100)

# Get list of media from the includes object
media = {m["media_key"]: m for m in tweets.includes['media']}

for tweet in tweets.data:
    attachments = tweet.data['attachments']
    media_keys = attachments['media_keys']
    print(tweet.text)
    if media[media_keys[0]].preview_image_url:
        print(media[media_keys[0]].preview_image_url)

8. Searching for geo-tagged Tweets

In the example below, we want all Tweets that are from the country US, which is why we specify it in our search query using the place_country:US operator. Now because by default, only the Tweet ID and Tweet text are returned, we have to specify the fact that we want the geo information in our response as well. We do this by passing the expansions='geo.place_id' to the search_all_tweets function and we also specify the geo fields that we are looking for using place_fields=['place_type','geo'].

In our response, we get the list of places from the includes object, and we match on the place_id to get the relevant geo information associated with the Tweet (the place.full_name in the example below)

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'covid -is:retweet place_country:US'

tweets = client.search_all_tweets(query=query, tweet_fields=['context_annotations', 'created_at', 'geo'],
                                  place_fields=['place_type', 'geo'], expansions='geo.place_id', max_results=100)

# Get list of places from includes object
places = {p["id"]: p for p in tweets.includes['places']}

for tweet in tweets.data:
    print(tweet.id)
    print(tweet.text)
    if places[tweet.geo['place_id']]:
        place = places[tweet.geo['place_id']]
        print(place.full_name)

9. Getting Tweet counts (volume) for a search query

So far, we have seen examples of how to get Tweets for a search query. However, if you want to get the count of Tweets (volume) for a search term, you can use the get_recent_tweets_count function. You can pass it the search query and specify the granularity for the data aggregation e.g. if you want the daily count, hourly count or 15-minute count for a search term.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'covid -is:retweet'

counts = client.get_recent_tweets_count(query=query, granularity='day')

for count in counts.data:
    print(count)

The response will look something like this:

{'end': '2021-09-23T00:00:00.000Z', 'start': '2021-09-22T03:01:44.000Z', 'tweet_count': 142282}

10. Getting a user's timeline

In order to get the most recent 3200 Tweets from a user's timeline, we can use the get_users_tweets method and pass it the user id. We can also specify additional fields that we want using the tweet_fields and expansions (as shown in the examples above).

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace user ID
id = '2244994945'

tweets = client.get_users_tweets(id=id, tweet_fields=['context_annotations','created_at','geo'])

for tweet in tweets.data:
    print(tweet)

11. Getting a user's mentions

In order to get the most recent 3200 mentions for a user, we can use the get_users_mentions method and pass it the user id. We can also specify additional fields that we want using the tweet_fields and expansions (as shown in the examples above).

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace user ID
id = '2244994945'

tweets = client.get_users_mentions(id=id, tweet_fields=['context_annotations','created_at','geo'])

for tweet in tweets.data:
    print(tweet)

12. Getting a user's followers

In order to get the followers of a user, we can use the get_users_followers function and pass it the user ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the user_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace user ID
id = '2244994945'

users = client.get_users_followers(id=id, user_fields=['profile_image_url'])

for user in users.data:
    print(user.id)

13. Getting users that a user follows

In order to get the list of users that a user follows, we can use the get_users_following function and pass it the user ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the user_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace user ID
id = '2244994945'

users = client.get_users_following(id=id, user_fields=['profile_image_url'])

for user in users.data:
    print(user.id)

14. Getting users that like a Tweet

In order to get the list of users that like a Tweet, we can use the get_liking_users function and pass it the Tweet ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the user_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace Tweet ID
id = '1441054496931541004'

users = client.get_liking_users(id=id, user_fields=['profile_image_url'])

for user in users.data:
    print(user)

15. Getting users that retweeted a Tweet

In order to get the list of users that retweeted a Tweet, we can use the get_retweeters function and pass it the Tweet ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the user_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace Tweet ID
id = '1441054496931541004'

users = client.get_retweeters(id=id, user_fields=['profile_image_url'])

for user in users.data:
    print(user)

16. Getting Tweets that a user liked

In order to get the list of Tweets that a user liked, we can use the get_liked_tweets function and pass it the User ID. If we want additional fields for the Tweet object such as context_annotations, created_at etc. we can specify those using the tweet_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace User ID
id = '2244994945'

tweets = client.get_liked_tweets(id=id, tweet_fields=['context_annotations','created_at','geo'])

for tweet in tweets.data:
    print(tweet)

17. Lookup Tweets using Tweet IDs

In some cases, if we have a list of Tweet IDs and want to build the Tweet dataset for these Tweet IDs, we can use the get_tweets function and pass it the list of Tweet IDs. By default, only the Tweet ID and Tweet text are returned. If we want additional Tweet fields, we can specify those using tweet_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace Tweet IDs
ids = ['1409935014725177344', '1409931481552543749', '1441054496931541004']

tweets = client.get_tweets(ids=ids, tweet_fields=['context_annotations','created_at','geo'])

for tweet in tweets.data:
    print(tweet)

18. Lookup Users using User IDs

In some cases, if we have a list of User IDs and want to build the User dataset for these User IDs, we can use the get_users function and pass it the list of User IDs. If we want additional User fields, we can specify those using user_fields parameter.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace User IDs
ids = ['2244994945']

users = client.get_users(ids=ids, user_fields=['profile_image_url'])

for user in users.data:
    print(user.profile_image_url)

19. Create a Tweet

If you want to create a Tweet with Tweepy using the Twitter API v2, you will need to make sure that you have your consumer key and consumer secret, along with your access token and access token secret, that are created with Read and Write permissions. You will use these keys and token when initializing the client as shown below.

import tweepy

client = tweepy.Client(consumer_key='REPLACE_ME',
                       consumer_secret='REPLACE_ME',
                       access_token='REPLACE_ME',
                       access_token_secret='REPLACE_ME')

# Replace the text with whatever you want to Tweet about
response = client.create_tweet(text='hello world')

print(response)

20. Retweet a Tweet

If you want to retweet a Tweet with Tweepy using the Twitter API v2, you will need to make sure that you have your consumer key and consumer secret, along with your access token and access token secret, that are created with Read and Write permissions (similar to the previous example). You will use these keys and token when initializing the client as shown below.

import tweepy

client = tweepy.Client(consumer_key='REPLACE_ME',
                       consumer_secret='REPLACE_ME',
                       access_token='REPLACE_ME',
                       access_token_secret='REPLACE_ME')

# Replace the Tweet ID that you want to retweet
response = client.retweet(1409931481552543749)

print(response)

21. Replying to a Tweet

If you want to reply to a Tweet with Tweepy using the Twitter API v2, in the create_tweet method, you can simply pass the in_reply_to_tweet_id with the ID of the Tweet that you want to reply to, as shown below:

import tweepy

client = tweepy.Client(consumer_key='REPLACE_ME',
                       consumer_secret='REPLACE_ME',
                       access_token='REPLACE_ME',
                       access_token_secret='REPLACE_ME')

# Replace the text with whatever you want to Tweet about
response = client.create_tweet(text='hello world reply', in_reply_to_tweet_id=12345678)

print(response)

These are some common examples of working with the Twitter API v2 using Tweepy. As Tweepy adds support for additional endpoints, we will update this post to include examples of using those endpoints.

If you have any questions or feedback about this guide, please feel free to reach out to me on Twitter!

Also, special shoutout to Harmon and the contributors of the Tweepy package for supporting the Twitter API v2 and the academic product track.

Getting started with data visualization with Dash and recent search counts

Jessica Garson — Mon, 13 Sep 2021 15:26:04 +0000

Let’s say you want to create a webpage where you can visualize how often you are Tweeting. For example, you could use the recent Tweets counts endpoint and Dash to create a website that automatically refreshes to show how much you are Tweeting throughout the week. This tutorial assumes a basic understanding of Python and Python 3 installed on your machine.

What is Dash?

Dash allows you to quickly set up a Flask app to create a web framework for data visualizations using Python, R, Julia, or .NET. For this tutorial, you’ll be using Dash with Python.

First, you’ll need to install dash with the following command in your terminal.

pip3 install dash

Additionally, you also want to install dash-bootstrap components to customize the look and feel of our page.

pip3 install dash-bootstrap-components

Getting started with the Twitter API

With this tutorial, you’ll be using the Tweet counts endpoint that returns the frequency of your Tweets over the last seven days.

Before you can use the Twitter API v2, you will need a developer account. Once you have an approved developer account, you will need to create a Project in the developer portal. Each Project contains an App, with which you can generate the credentials required to use the Twitter API. You can learn more about getting started with the Twitter API in the getting started section of our documentation.

To connect to the Twitter API, we’ll be using the requests package of Python. You will need to install this and pandas to help us get our data into the right shape. You can use dash-bootstrap-components to customize the look and feel of your webpage a bit further.

pip3 install requests
pip3 install pandas

Setting up your app.py file

To start writing code to create your web application, you will need to set up an app.py file. In your terminal, you will want to make a python file entitled app.py.

touch app.py

Importing packages

Dash utilizes the plotting library Plotly. Plotly is an open-source plotting package, and you can check out the documentation, which might help debug some issues you may encounter.

In the code editor of your choosing, open up your app.py file. At the top of the file, you will need to define the packages you want to import.

import dash
import dash_bootstrap_components as dbc
import plotly.express as px
import pandas as pd
import os
import requests
from dash import dcc
from dash import html

Creating the app variable

After importing the packages, you will need to create a variable called app. In this variable, you are initializing the app you are making and pulling in the bootstrap theme of journal. Using a bootstrap theme is just one way that you can start customizing your page.

app = dash.Dash(external_stylesheets=[dbc.themes.JOURNAL])
server = app.server

Defining the endpoint URL and setting up your query parameters
Before connecting to the Twitter API, you can set the URL to the endpoint you are calling. For this example, you will be calling the recent Tweet counts endpoint, and you will save that as a variable called search_url.

search_url = "https://api.twitter.com/2/tweets/counts/recent"

Next, you’ll want to set up a dictionary for query_params to define the query and granularity of the counts you’ll get back. In the parameter entitled query, you can specify the rules of what will get returned from the Twitter API. In the example below, the query is set to be all Tweets from the handle @jessicagarson. You can swap this out for your handle if you wish. To learn more about creating a query, check out our documentation on the subject.

With the granularity parameter, you can adjust the timing of the counts you’ll have returned. For example, you will get back hourly data instead of daily if you don’t specify ‘day’ in the granularity parameter.

query_params = {'query': 'from:jessicagarson','granularity': 'day'}

Authenticating to the Twitter API

You will want to set up an environment variable in your terminal to save your Bearer Token to your request. Then, you need to replace your-bearer-token with your bearer token. Be sure to keep the single quotes around your Bearer Token. You can get your bearer token from the keys and tokens section of your App inside the developer portal.

export BEARER_TOKEN='your-bearer-token'

Define a variable set to bearer_token in your code editor, and use the os module to get the bearer_token environment variable set in the terminal.

bearer_token = os.environ.get('BEARER_TOKEN')

Pass your Bearer Token as a parameter to create the headers to authenticate to the Twitter API as part of a method.

def bearer_oauth(r):
    r.headers["Authorization"] = f"Bearer {bearer_token}"
    r.headers["User-Agent"] = "GettingStartedDash"
    return r

Connecting to the Twitter API

Create a function to connect to the Twitter API and get back a JSON response if you’ve successfully connected. This function has some additional error handling and will print out the response code you get to help with debugging.

def connect_to_endpoint(url, tweet_fields):
    response = requests.request("GET", url, auth=bearer_oauth, params=tweet_fields)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(
            "Request returned an error: {} {}".format(
                response.status_code, response.text
            )
        )
    return response.json()

Now, you can call the function you created with the URL for the Tweet counts endpoint, our authentication headers, and the query parameters you defined earlier.

json_response = connect_to_endpoint(search_url, query_params)

Creating a data frame

Next, you can convert the JSON object returned from the Twitter API into a pandas data frame. Using a data frame allows you to work with the data more robustly and gets our data into the correct format to create a plot with Dash. For our purposes, you’ll only need to work with the data object of the JSON payload you get from the Twitter API, so you can convert only that into a data frame using pandas.

df = pd.DataFrame(json_response['data'])

You’ll also need to convert the date to a universal date object which you can do with the to_datetime method in pandas.

df['start'] = pd.to_datetime(df['start'])

Since you’ll only need the start and tweet_count columns, you can create a new data frame called final that selects only these columns.

final = df[['start', 'tweet_count']]

Now you are ready to create a line graph with our final data frame and set our x-axis and y-axis up.

fig = px.line(final, x="start", y="tweet_count")

Setting up your website

You can now start creating our website. You can begin by setting up the colors you want to use by creating a dictionary for the hex colors for our background and the text of our line graph.

colors = {"background": "#FFFFFF", "text": "#1DA1F2"}

fig.update_layout(
    plot_bgcolor=colors["background"],
    paper_bgcolor=colors["background"],
    font_color=colors["text"],
)

At this point, you are ready to tie it all together by setting up your app layout. You will first need to define your background color, followed by setting up your title as an H1 HTML element and a brief description of your graph. Additionally, you will need to set your chart with and unique id and set the figure equal to the fig variable you defined earlier.

app.layout = html.Div(
    style={"backgroundColor": colors["background"]},
    children=[
        html.H1(
            children="Tweets by Date",
            style={"textAlign": "center", "color": colors["text"]},
        ),
        html.Div(
            children="An example using Dash and the Twitter API recent search counts to see how much I've been Tweeting this week",
            style={"textAlign": "center", "color": colors["text"]},
        ),
        dcc.Graph(id="example-twitter", figure=fig),
    ],
)

Finally, you will need to set the main statement at the end of your file to run in debug mode when it sets up a local server for you to run and view the code.

if __name__ == "__main__":
    app.run_server(debug=True)

Running the Python file and deployment

Now that you’ve written the code needed for your Python file, you can run it locally. In your terminal, run the following command.

python3 app.py

Now in your terminal, you should see a response that looks similar to the following.

200
Dash is running on http://127.0.0.1:8050/

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
200

If you go to http://127.0.0.1:8050/ in our web browser. You should now see a graph that looks similar to the one below.

Deployment

This application is deployed to Glitch and can be remixed. This application is boosted using Glitch's pro plan to pull data more often.

Next steps

Hopefully, this tutorial can be a starting place to work with Dash and the Twitter API. You can view the full version of the code in our GitHub repository for this code sample. You can also check out a version deployed to Glitch.

As a next step, you may want to customize your query further to see how much you Tweet about a specific topic or how many times a particular term has been posted throughout the week. You may also want to add data visualization using a different Twitter API endpoint, such as searching for themes around what type of Tweets you like.

Be sure to let us know on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Getting started with R and v2 of the Twitter API

Jessica Garson — Thu, 25 Mar 2021 18:08:21 +0000

This tutorial will walk through what you need to get started with the programming language R and the Twitter API v2. Using R to connect to the user lookup endpoint, I’ll show how to work with JSON returned from the Twitter API. User lookup is a GET method and returns information about a user or group of users, specified by a user ID or a username.

If you aren’t familiar, R is one of the most popular languages for common Data Science tasks like time-series analysis, modeling, visualization, and other data analysis, and is often used in conjunction with the Twitter API. With the user lookup endpoint, you can use the user object to determine a correlation between the number of followers a person has and the sentiment score of their bio. The user object may also be used to map a group of accounts based on the location publicly listed in their profiles.

Getting started with the Twitter API

Before you can use the Twitter API v2, you will need a developer account.

Once you have an approved developer account, you will need to first create a Project. Projects allow you to organize your work based on how you intend to use the Twitter API, so you can effectively manage your access to the API, and monitor your usage.

Each Project contains an App, with which you can generate the credentials required to use the Twitter API. You can learn more about how to get started with the Twitter API, in the getting started section of our documentation.

Getting your R environment set up

First, you need to download R which you can do on the cran website.

After, to set up an environment to work with R, you can use R studio, the R extension pack for Visual Studio Code, or a Jupyter notebook if you come from the Python world.

Setting up your environment variable

For the code examples, I’m going to be showing today, you will want to create an environment variable for your bearer token. The bearer token is what allows you to authenticate to the Twitter API and start making requests. First, replace “your-bearer-token” with your own bearer token, which can be obtained from the keys and tokens section of your App in the developer portal. You’ll need to run this line of code in the console before you start writing a script.

Sys.setenv(BEARER_TOKEN = "your-bearer-token")

Making your request

You can use the package httr to make HTTP requests to the Twitter API. If you haven’t already installed this, please install the package in your console. You will also need to install jsonlite to work with our JSON object and dplyr for data manipulation.

install.packages("httr")
install.packages("jsonlite")
install.packages("dplyr")

You can now begin writing your R script to connect to the API. At the top of the file, call the packages httr, jsonlite, and dplyr.

require(httr)
require(jsonlite)
require(dplyr)

The first step in your code sample is to get set up to authenticate to the Twitter API. Grab the bearer token you set, and pass that into your headers for authentication.

bearer_token <- Sys.getenv("BEARER_TOKEN")
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))

Once you got your authentication set up, define the parameters of your request. By default, you will get back the id, name, and username of each user you get back. You can make adjustments to this payload by adding additional fields and expansions. For this example, you will want the profile bio of the user which is found in the description and an expansion that contains the pinned Tweet of the user.

params <- list(`user.fields` = 'description',
               `expansions` = 'pinned_tweet_id')

Now you are ready to format your URL with the handle you are looking to get more information about. Use the readline method to allow this sample to be reusable. After you type the handle you want to look at, format your URL to contain the handle you define.

handle <- readline('What handle do you want to look at? ')
url_handle <-
  sprintf('https://api.twitter.com/2/users/by?usernames=%s', handle)

At this point, use the httr package to make a GET request to the URL you just created, pass in our authentication credential via the header, and pass in the parameters you defined. You can save the response as a text object in the variable obj and print this out to view the result of the request you made.

response <-
  httr::GET(url = url_handle,
            httr::add_headers(.headers = headers),
            query = params)
obj <- httr::content(response, as = "text")
print(obj)

Working with our JSON payload

One of my favorite ways to work with a JSON is to use a data frame, which allows you to easily access complex nested data. To do this, use the fromJSON method of the jsonlite package to flatten your file to allow the fields to be in the same object. Then, pass that object into a data frame. Now you are ready to view this data frame.

json_data <- fromJSON(obj, flatten = TRUE) %>% as.data.frame
View(json_data)

You can access the fields of the data from the data frame and pass them into a string that has the handle, the username, and bio.

final <-
  sprintf(
    "Handle: %s\nBio: %s\nPinned Tweet: %s",
    json_data$data.username,
    json_data$data.description,
    json_data$includes.tweets.text
  )

Use cat instead of print to view the object with newlines between each field you are pulling.

cat(final)

If you had multiple handles you made requests for, you could easily use a loop to access each data frame element.

Conclusion

Hopefully, this tutorial can be a starting place to work with R and the Twitter API. As a next step, you may want to look at our R samples for recent search, Tweet lookup and user lookup in our v2 sample code. Be sure to let us know on the forums if you run into any troubles along the way or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Getting historical Tweets using the full-archive search endpoint

Suhem Parack — Tue, 26 Jan 2021 18:55:09 +0000

Introduction

The full-archive search endpoint lets you search the entire history of public Twitter data, which dates back to the first Tweet in March 2006. It works similarly to the recent search endpoint, used to search Tweets for events from the last 7 days; both endpoints return Tweets related to the topics you are interested in, based on your search query. Full-archive search helps address one of the most common use cases for academic researchers, who might use this for longitudinal studies, or analyzing a past topic or event. Note: v2 full-archive search is currently available only on the newly launched Academic Research product track. See the prerequisite section below to learn more.

This tutorial provides a step-by-step guide for researchers who wish to use this endpoint to search the complete history of public Twitter data. It will also demonstrate the different ways to build a dataset, such as by retrieving geo-tagged Tweets, and how to page through the available Tweets for a query.

Prerequisites

Currently this endpoint is only available as part of the Academic Research product track. In order to use this endpoint, you must apply for access. Learn more about the application and requirements for this track here.

Connect an app to the academic project

Once you are approved to use the Academic Research product track, you will see your Academic Project in the developer portal. From the Projects and Apps section, click on ‘Add App’ to connect your app to the Project.

Then, you can either choose an existing app and connect it to your project (as shown below)

Or you can create a new app, give it a name and click complete, to connect a new app to your academic project.

This will give you your API keys and Bearer token that you can then use to connect to the full-archive search endpoint.

Note: The keys in the screenshot above are hidden, but in your own developer portal, you will be able to see the actual values for the API key, API secret key and Bearer token. Save these keys and the bearer token because you will need those for calling the full-archive search endpoint.

If you want to learn more or have questions, check our documentation about Projects and Apps here.

Connecting to the full-archive search endpoint

The cURL command below shows how you can get historical Tweets from @twitterdev handle:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=from:twitterdev' --header 'Authorization: Bearer XXXXX'

Replace the XXXXX with your own bearer token and paste it in your terminal. You will see the response JSON.

By default, only 10 most recent Tweets will be returned. If you want more than 10 Tweets per request, you can use the max_results parameter and set it to a maximum of 500 Tweets per request, as shown below:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=from:twitterdev&max_results=500' --header 'Authorization: Bearer XXXXX'

Building queries

As you can see in the example calls above, using the query parameter, you can specify the data that you want to search for. As an example, if you wanted to get all Tweets that contain the word covid or the word coronavirus, you can use the OR operator within brackets, and your query can be (covid OR coronavirus) and thus your API call will be:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=(covid%20OR%20coronavirus)&max_results=500' --header 'Authorization: Bearer XXXXX'

Similarly, if you want all Tweets that contain the words covid19 that are not retweets, you can use the is:retweet operator with the logical NOT (represented by -), so your query can be covid19 -is:retweet and your API call will be:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=covid19%20-is:retweet&max_results=500' --header 'Authorization: Bearer XXXXX'

Check out this guide for a complete list of operators that are supported in the full-archive search endpoint.

Using the start_time and end_time parameters to get historical Tweets

When using the full-archive search endpoint, by default Tweets from the last 30 days will be returned. If you want to get Tweets that are older than 30 days, you can use the start_time and end_time parameters in your API call. These parameters must be in a valid RFC3339 date-time format e.g. 2020-12-21T13:00:00.00Z. Thus, if you want to get all Tweets from the Twitterdev account for the month of December 2020, your API call will be:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=from:Twitterdev&start_time=2020-12-01T00:00:00.00Z&end_time=2021-01-01T00:00:00.00Z' --header 'Authorization: Bearer XXXXX'

Getting geo-tagged historical Tweets

Geo-tagged Tweets are Tweets that have geographic information associated with them such as city, state, country etc.

Using has:geo operator

If you want to get Tweets that have geo data, you can use the has:geo operator. For example, the following cURL request will get only those Tweets from the Twitterdev handle that have geo data:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=from:twitterdev%20has:geo' --header 'Authorization: Bearer XXXXX'

Using place_country operator

Similarly, you can limit Tweets that have geo data, to a specific country, using the place_country operator. The cURL command below will get all Tweets from the TwitterDev handle from the United States:

curl --request GET 'https://api.twitter.com/2/tweets/search/all?query=from:twitterdev%20place_country:US' --header 'Authorization: Bearer XXXXX'

The country is specified above using the ISO alpha-2 character code. Valid ISO codes can be found here.

Getting more than 500 historical Tweets using the next_token

As mentioned above, by default you can only get up to 500 Tweets per request for a query to the full-archive search endpoint. If there are more than 500 Tweets available for your query, your json response will include a next_token which you can append to your API call in order to get the next available Tweets for this query. This next_token is available in the meta object of your JSON response, which looks something:

{
   "newest_id": "12345678...",
   "oldest_id": "12345678...",
   "result_count": 500,
   "next_token": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}

Hence, to get the next available Tweets, use the next_token value from this meta object and use the value as the value for the next_token in your API call to the full-archive search endpoint as shown below (You will use your own bearer token and the value that you get for the next token for your previous API call)

curl --request GET 'https://api.twitter.com/2/tweets/search/all?max_results=500&query=covid&next_token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' --header 'Authorization: Bearer XXXXX'

This way, you can keep checking if a next_token is available and if you have not reached your desired number of Tweets to be collected, you can keep calling the full-archive endpoint with the new next_token for each request. Keep in mind that full-archive search endpoint counts towards your total Tweet cap i.e. the number of Tweets per month that you can get from the Twitter API, so be mindful of your code logic when paging through the results in order to make sure you do not end up inadvertently exhausting your Tweet cap.

Below are some resources that can help you when using the full-archive search endpoint. We would love to hear your feedback. Reach out to us on @TwitterDev or on our community forums with questions about this endpoint.

Resources

Sending a response from the Twitter API to a spreadsheet in Google Sheets

Jessica Garson — Tue, 22 Dec 2020 22:24:46 +0000

Liquid syntax error: 'raw' tag was never closed