Forem: Scott Chamberlain

Python, ast, and redbaron

Scott Chamberlain — Tue, 18 Apr 2023 00:00:00 +0000

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub.

There were a few catches though.

First, I couldn’t simply get any string out of each Python script - they needed to be strings specficied by a specific function parameter, and match a regex (e.g., start with ‘abc’).

Second, the script paths lack the GitHub repository root name. This name was part of the function name - so I needed to get access to the function that the path was specified within, and then parse the function name to get the repository name.

The obvious solution I thought was the ast library.

ast library

I started by using ast. The ast.NodeVisitor class seemed like it would do the trick.

An example script (“my_script.py”):

def hello(path, stuff=None):
    return path

if __name__ == " __main__":
    print(hello(path="hello/world.py", stuff="hello mars"))


import ast

class CollectStrings(ast.NodeVisitor):
    def visit_Module(self, node):
        self.out = set()
        self.generic_visit(node)
        return list(filter(lambda w: w.startswith("hello") and w.endswith(".py"), self.out))

    def visit_Str(self, node):
        self.out.add(node.s)

file = "my_script.py"
with open(file, "r") as f:
    body = ast.parse(f.read())

coll = CollectStrings()
coll.visit(body)
## ['hello/world.py']

That worked great at fetching paths - only because all the paths I was looking for started with the same text and all have the same file extension.

HOWEVER - I also needed the function name that the path argument was called from. I tried to make this work with ast.NodeVisitor but couldn’t get it to work.

I eventually got frustrated enough and figured there must be some libraries that build on top of ast that make it easier to work with ast’s in Python.

redbaron

Enter redbaron. I found this library pretty quickly upon searching for a library building on top of ast.

Another example script (“their_script.py”):

def hello(path, stuff=None):
    return path

def goodbye(path, stuff=None):
    return path

def world():
    path_str = hello(path="src/world.py", stuff="hello mars")
    other_path_str = goodbye(path="src/world.py", stuff="hello saturn")

    return path_str, other_path_str

if __name__ == " __main__":
    print(world())


import re
from redbaron import RedBaron

file = "their_script.py"
with open(file, "r") as src:
  red = RedBaron(src.read())

red
## 0 def hello(path, stuff=None):
## return path
##     
##     
##     
## 1 def goodbye(path, stuff=None):
## return path
##     
##     
##     
## 2 def world():
## path_str = hello(path="src/world.py", stuff="hello mars")
## other_path_str = goodbye(path="src/world.py", stuff="hello saturn")
##     
## return path_str, other_path_str
##     
##     
##     
## 3 if __name__ == " __main__":
## print(world())
##

Even just the resulting object you get from parsing something is useful:

And with .help() you get a very detailed map of the structure of the thing you’re trying to navigate (only printing first 20 lines):

red.help()
## 0 -----------------------------------------------------
## DefNode()
## # identifiers: def, def_, defnode, funcdef, funcdef_
## # default test value: name
## async=False
## name='hello'
## return_annotation ->
## None
## decorators ->
## arguments ->
## * DefArgumentNode()
## # identifiers: def_argument, def_argument_, defargument, defargumentnode
## target ->
## NameNode() ...
## annotation ->
## None
## value ->
## None
## * DefArgumentNode()
## # identifiers: def_argument, def_argument_, defargument, defargumentnode
...

Looking at the result from red.help() I can then use .find_all() to find certain nodes in the ast.

nodes = red.find_all("AtomtrailersNode")
nodes = list(filter(lambda w: "hello" in w.dumps(), nodes))
nodes
## [hello(path="src/world.py", stuff="hello mars"), goodbye(path="src/world.py", stuff="hello saturn")]

Then I can write some okay code to extract out the function name, and ugly code to get the string supplied to the path parameter. Then f-string those together to get the path I’m after.

paths = []
for node in nodes:
    fxn_name = node.name.value
    command = re.search("src/.*\\.py", node.dumps()).group()
    paths.append(f"{fxn_name}/{command}")

for path in paths:
    print(path)
## hello/src/world.py
## goodbye/src/world.py

Not super proud of this but gets the job done for my use case - and when you’re not making open source for others, you don’t need to worry about other use cases :)

I’ll definitely try to learn how to properly extract stuff using redbaron - but it got me to answer much faster than the ast library.

CRAN Checks API and Badges

Scott Chamberlain — Thu, 02 Jun 2022 00:00:00 +0000

TL;DR

In 6 months (end of November 2022) the CRAN Checks API https://cranchecks.info/ will be gone
You can still get badges at https://badges.cranchecks.info
You can use the new badges like:

[![cran checks](https://badges.cranchecks.info/worst/dplyr.svg)](https://cran.r-project.org/web/checks/check_results_dplyr.html)

Find more details at https://github.com/sckott/cchecksbadges

Sunsetting the CRAN Checks API

If you contribute an R package to CRAN, you may use badges from the CRAN checks API at https://cranchecks.info/. The CRAN Checks API has been operating since about September 2017 (I think).

The API has a number of routes, but really people only use the badges.

Given this usage pattern, and not wanting to pay for a server anymore, I’ve decided to make the badges available on a static endpoint that doesn’t cost me anything. There are costs of course - but they’re on Github and Netlify (thanks y’all!).

The new static site version

The static site is created using GitHub Actions.

For a static site you need to create files for any route you want to support - so the code for the static site creates 19 routes x No. of CRAN packages = approx. 360,000 svg files.

The badges will be updated once a day - the same schedule as the API.

I had to use Netlify because Github pages (as far as I know) doesn’t provide ssl certs for custom domains and my domain host doesn’t provide free Lets Encrypt certs - whereas Netlify does.

Some example routes you can look at

https://badges.cranchecks.info/summary/taxize.svg https://badges.cranchecks.info/worst/dplyr.svg https://badges.cranchecks.info/flavor/r-devel-linux-x86_64-fedora-clang/DT.svg

If you find any issues with the badges at https://badges.cranchecks.info open an issue.

List comprehension vs. filter vs. key lookup

Scott Chamberlain — Mon, 18 Apr 2022 00:00:00 +0000

I was working on a work task last week, and needed to filter out one instance of a class from a list of class instances. No matter how you do this speed doesn’t matter too much if you’re doing this operation once or a few times.

However, I this operation needs to be done about 100K times each time the script runs - so speed definitely does matter in this case.

First, I naively started off with using filter(). When that lead to waiting more than ten minutes, I read that list comprehensions are faster.

Second, I tried list comprehensions. Also waited more then ten minutes and gave up again.

Last, I thought perhaps it would work to make a dictionary where the keys are the things I need to filter on and the values the class instances. This was the answer! Super fast.

Now some examples of what I’m talking about.

Make a class called Stuff.

class Stuff:
  def __init__ (self, letter):
    self.letter = letter
    super(Stuff, self). __init__ ()
  def __repr__ (self):
        return "<{} ({})>".format(self. __class__. __name__ , self.letter)

x = Stuff('s')
x
#> <Stuff (s)>
x.letter
#> 's'

Make a list of instances of the class Stuff

import string

lst = []
for x in string.ascii_lowercase:
    lst.append(Stuff(x))
lst
#> [<Stuff (a)>, <Stuff (b)>, <Stuff (c)>, <Stuff (d)>, <Stuff (e)>, <Stuff (f)>, <Stuff (g)>, <Stuff (h)>, <Stuff (i)>, <Stuff (j)>, <Stuff (k)>, <Stuff (l)>, <Stuff (m)>, <Stuff (n)>, <Stuff (o)>, <Stuff (p)>, <Stuff (q)>, <Stuff (r)>, <Stuff (s)>, <Stuff (t)>, <Stuff (u)>, <Stuff (v)>, <Stuff (w)>, <Stuff (x)>, <Stuff (y)>, <Stuff (z)>]
len(lst)
#> 26

List comprehension: This is how I did the list comprehension method. Filter the list lst where some attibute matched some value.

[x for x in lst if x.letter == 'f']
#> [<Stuff (f)>]

Filter: This is how I did the filter method. Filter the list lst where some attibute matched some value.

list(filter(lambda x: x.letter == 'f', lst))
#> [<Stuff (f)>]

And here’s the dictionary approach. Here, I first make a dictionary via dict(zip()) where the keys are some attribute of each instance. You can lookup by key.

A major difference/drawback of this approach is that it only works if there’s only one match per key because dictionaries don’t allow duplicate keys.

lst_map = dict(zip([w.letter for w in lst], lst))
lst_map['f']
#> <Stuff (f)>

And better yet, use .get() so you don’t run into a KeyError when the key doesn’t exist

lst_map.get('f')
#> <Stuff (f)>
lst_map.get('5')
#> (returns None)

What about the timings:

from timeit import timeit
n = 100000
time_list_comp = timeit("[x for x in lst if x.letter == 'f']", number=n, globals=globals())
time_filter = timeit("list(filter(lambda x: x.letter == 'f', lst))", number=n, globals=globals())
time_dict = timeit("lst_map['f']", number=n, globals=globals())
time_dict_get = timeit("lst_map.get('f')", number=n, globals=globals())

round(time_list_comp, 3)
#> 0.088
round(time_filter, 3)
#> 0.134
round(time_dict, 3)
#> 0.002
round(time_dict_get, 3)
#> 0.003

For bracketed lookup, the list comprehension is 39 times slower, and the filter is 59 times slower.

For the get() lookup, the list comprehension is 26 times slower, and the filter is 39 times slower.

Notes on Python

Scott Chamberlain — Mon, 07 Feb 2022 00:00:00 +0000

It’s been interesting switching jobs with respect to programming languages. I used to write 95% R - now I write 95% Python.

I have been using Python for many years, but not seriously or getting paid either. I’ve learned alot in the first 6 months.

Some Python things learned:

Functions and methods

I used to think functions and methods were the same thing. But during the last 6 months I learned that functions and methods are not the same. Well, they’re not that different. A function outside a class is just called a function while a function inside a class is called a method. They could be exactly the same and do the same thing, but one is outside a class and the other inside a class.

class Stuff(object):
  def things():
    return 5

def things():
    return 5

Stuff.things()
# 5
things()
# 5

.sort and sorted

.sort called on an object changes the object in place and sorted() creates a new object.

x = [4,1,7,2,6,5,3]
z = x.sort()
# nothing returned, z = None
sorted(x)
# [1, 2, 3, 4, 5, 6, 7]

context managers

I’d surely used a context manager in Python before but didn’t realize what was happening. The code base I work in uses many with statements and these are used with context managers like:

with EXPRESSION as TARGET:
    SUITE

In the above case I often create a connection to a database using the with statement, then once the block is exited, the connection is cleaned up.

imports

Coming from R, it’s so nice in Python to be able to import specific functions, classes, etc. rather than having to load an entire file or package in R. In addition, as in Python imports is really nice to have.

Python

from x import y
import pandas as pd

require(x)
library(x)

Sometimes I run into circular import issues in Python (as I did in R), which I’ve yet to find a neat solution to sorting out.

Mocking HTTP redirects

Scott Chamberlain — Sat, 27 Nov 2021 00:00:00 +0000

You’ve experienced an HTTP redirect (or URL redirect, or URL forwarding) even if you haven’t noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don’t show all the HTTP requests going on in the background, some of which are redirects. Redirection is used for various reasons, including to prevent broken links when web pages are moved, for privacy protection, to allow multiple domains to refer to a single web page, and more.

The easiest way to know if you hit a redirect is to look at the HTTP status code. Status codes in the 3xx series are mostly about URL redirection. The most common you will see are 301 (moved permanently), 302 (moved temporarily), and 303 (see other URI; usually in a “Location” response header).

When making HTTP requests in R, redirects are generally handled automatically by the three HTTP clients (curl, crul, httr). That is, if a 300 series code is detected, all three clients will go to the next URI and so on until there are no more redirects. Automatically following redirects may not be default behavior elsewhere (e.g., crul command line tool doesn’t follow redirects by default), so beware.

HTTP redirects become more tricky when we have to mock them in unit tests or other similar situations. I’ll cover the various tools for doing this in R.

Redirects

First, I’ll show how redirects work with three major HTTP clients:

curl

library(curl)
h <- curl::new_handle()
handle_setopt(h, followlocation=0L)
out <- curl_fetch_memory("https://hb.opencpu.org/redirect/3", handle = h)
curl::parse_headers(out$headers, multiple = TRUE)

crul

library(crul)
con <- HttpClient$new("https://hb.opencpu.org/redirect/3")
res <- con$get()
length(res$response_headers_all)
#> [1] 4

httr

library(httr)
z <- GET("https://hb.opencpu.org/redirect/3")
length(z$all_headers)
#> [1] 4

Mocking redirects

If you want to mock HTTP redirects, you can do so with the webmockr package. Why would you want to mock redirects?

Here’s one use case: Say you have a library/package interacting with a web resource that you interact with. You want to add some unit tests for a route that responds with one or more redirects. You’d prefer not to perform real HTTP requests against the remote service for one reason or another (e.g., extreme rate limiting); and some would say it’s best not to test with real HTTP requests b/c you want to test the functionality of the package, NOT the remote server with which it interacts.

In the following, we re-create what happens in real HTTP requests - but just status codes and the location response header.

library(webmockr)
library(crul)
webmockr::enable()

Make a single stub with each redirect response with to_return()

stub_request("get", "https://hb.opencpu.org/redirect/3") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/2")) %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/1")) %>%
  to_return(status = 302, headers = list(location = "/get")) %>%
  to_return(status = 200, headers = list(location = "hooray, all done!"))

Then make four different requests to https://hb.opencpu.org/redirect/3:

con <- crul::HttpClient$new(url = "https://hb.opencpu.org")
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/2"
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/1"
con$get("redirect/3")$response_headers
#> $location
#> [1] "/get"
con$get("redirect/3")$response_headers
#> $location
#> [1] "hooray, all done!"

This isn’t ideal because it doesn’t reflect how the real HTTP request equivalent happens.

Alernatively, you could set it up like this, with four separate stubs:

stub_request("get", "https://hb.opencpu.org/redirect/3") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/2"))

stub_request("get", "https://hb.opencpu.org/relative-redirect/2") %>%
  to_return(status = 302, headers = list(location = "/relative-redirect/1"))

stub_request("get", "https://hb.opencpu.org/relative-redirect/1") %>%
  to_return(status = 302, headers = list(location = "/get"))

stub_request("get", "https://hb.opencpu.org/get") %>%
  to_return(status = 200, headers = list(location = "hooray, all done!"))

Then make each request in turn to each successive URL:

con <- crul::HttpClient$new(url = "https://hb.opencpu.org")
con$get("redirect/3")$response_headers
#> $location
#> [1] "/relative-redirect/2"
con$get("relative-redirect/2")$response_headers
#> $location
#> [1] "/relative-redirect/1"
con$get("relative-redirect/1")$response_headers
#> $location
#> [1] "/get"
con$get("get")$response_headers
#> $location
#> [1] "hooray, all done!"

Faking real redirects

vcr is built on top of webmockr, but instead of returning stubbed responses and not allowing real HTTP requests, vcr stores real HTTP request/response and uses them on all subsequent matching HTTP requests.

I wrote this back in March 2021 - and was waiting to figure out how to deal with redirects in vcr before finishing this post - see vcr issue #220. I still have and may never get to that issue. If you are interested, please do stop by vcr and make a pull request to get it fixed. The major issue is that vcr stores only the first HTTP response in a redirect chain, rather than the last HTTP response - as I would expect.

API client design: how to deal with lots of parameters?

Scott Chamberlain — Mon, 21 Dec 2020 00:00:00 +0000

In February this year I wroute about how many parameters functions should have, looking at some other languages, with a detailed look at R. On a related topic …

As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

The general programming wisdom I’ve seen is that a function should have no more than 3-4 parameters (e.g., this long SO thread, or this one). So should one do anything different from a normal function when that function is connecting to a web API route with a lot of parameters? I’ve not found very much spilled ink on this exact topic, but I’ll discuss what I have found.

Use cases?

A Software Engineering StackExchange thread How to handle many arguments in an API wrapper? had a couple ideas. One idea is to consider use cases, and then make separate functions covering those use cases. This might work, but I haven’t explored it thoroughly for a real API route yet. Pondering on it though I doubt this would work since you’d have to pre-emptively think about all the different scenarios users might dream up, which seems like a fools errand.

Builder pattern

Another concept brought up in the thread mentioned above was the Builder pattern. It’s hard for me to understand the idea in abstract - here’s a nicer discussion of this in Ruby.

This is a good concept to know about, but I don’t think is appropriate for the issue at hand, of how to handle many API parameters.

Named parameter map

Another idea in that thread was to use a named parameter map. In R this would look something like this (imagine a lot more parameters though):

foo <- function(args) {
  get("/some-api-route", args)
}
api_args <- list(query = "*:*", limit = 10)
my_args <- modifyList(api_args, list(query = "bears", limit = 300))
foo(my_args)

That is, the above would replace this:

foo <- function(query = "*:*", limit = 10) {
  get("/some-api-route", list(query = query, limit = limit))
}
foo(query = "*:*", limit = 10)

So in the first code block the function no longer has a lot of parameters in it. The drawback of this in R (and I’m sure is similar in other languages) is that users lose the autocomplete helpers that most modern IDE’s and text editors have - helping users type less and quickly get a tip on what each parameter is intended to do and importantly (if the developer has documented the function well) what types the parameters expects and what (if any) options there are to pass to the parameter.

Of course a user can “just” read the docs to figure out what each parameter expects, but it sure can save a lot of time if the help is right there in the tooltips of the IDE/text editor. In addition, in R there’s automated checking that parameters in functions are also documented, which is nice for making sure parameters and docs don’t get out of sync. You’d lose this by using a parameter map - though you could document the parameter map - and perhaps wire together some custom code to make sure the parameters in the parameter map are all handled by the function. This does seem like a lot of fuss though compared to simply having the parameters in the function itself.

This approach probably becomes more attractive if a client has many functions that take the same parameters - in which a named parameter map could handle the parameters and any logic behind checking those parameters.

Include no parameters in the function

In other words: Pass all parameters on to the API w/o including any of them in the function - i.e., let the API handle any problems in parameters.

Another approach I’ve not seen written about but that I’ve seen in code is having a rather lite wrapper around an API and letting the API itself sort out any problems due to user inputs.

An example is the gh R package, a client for the GitHub API. For query parameters you can pass in named parameters through the ellipsis ..., all of which are passed as query parameters. The gh package does no checking of these parameters (that I know of); simply passes them to the GitHub API. The GitHub API happens to apparently ignore invalid (silently drop) parameters and invalid valuses of parameters (here, “stuff” is an invalid value for the page parameter).

x <- gh("GET /users/{username}/repos", username = "gaborcsardi", page = "stuff")
length(x)
#> 30

I don’t hate this solution, but I don’t love it either. This approach is highly dependent on a well designed API that fails gracefully, with informative error messages and with correct status codes, etc. I would say most APIs are not as nice as GitHub’s, at least in the scientific API space in which I work.

One plus side of this approach is the R package gh only has one parameter (...) to handle all query parameters, so you do solve the too many query parameters problem.

Another upside to this approach is you do not have to keep up with any changes in parameters on each API route - for example, an API route could drop one parameter, and add another, and the R client wouldn’t have to change anything (assuming the change in parameters wasn’t associated with a change that breaks code in the client).

A major downside of this approach is that the user often has to mount a time-consuming expedition to figure out what parameters are accepted. Some API clients may document them, and some will simply direct users to the web APIs docs. I think this part alone makes this solution (include no parameters in the function) not a good one since the user experience can be so bad if the documentation is not good. And all developers know its much easier for their docs to get out of date than their code.

Include some parameters in the function

Another approach is to define some query parameters in the function, and handle all others via R’s ellipsis (...) - or similar in other languages. I’ve seen this relatively often and have used it myself. It’s often used when there’s a clear smaller set of important parameters - those can be put in the function as named parameters. And then there’s a long tail of other parameters that the maintainer thinks are not likely to be used very often. Those can be looked up by the user in the API docs for whatever API the client interacts with.

An example of this is the rOpenSci package rtweet - a client for the Twitter API. In the search_tweets() function there are a half dozen or so named parameters in the function, but then the ellipsis handles all other parameters.

The drawback to this approach is that no two APIs behave the same way. In the case of Twitter they silently ignore/drop parameters they do not support (same as the GitHub API, see above). For example:

library(rtweet)
search_tweets("hillaryclinton", n = 3, foo = "bar")

Works just fine even though foo is absolutely not a parameter supported by the Twitter API. They must ignore parameters they don’t support. This is same behavior as the GitHub API we saw above.

In the case of Twitter and GitHub one might want to raise errors on unsupported parameters client side in rtweet to avoid any use confusion of parameters being silently dropped.

Grouping similar parameters together

Many threads (e.g., this one) suggest that similar parameters could be grouped together to reduce the number of parameters passed to a function. For example, if a function has the parameters latitude and longitude you could group those into a single parameter called e.g., coordinates.

# Original function, each parameter separate
foo <- function(latitude, longitude) {
    # do something with latitude/longitude
    latitude
    longitude
}

# Modified function, grouping the two parameters into one
foo <- function(coordinates) {
    # do something with latitude/longitude
    coordinates$latitude
    coordinates$longitude
}

Though I’ve not tried this approach myself, it might be a good compromise between a function not handling any parameters (i.e., just passing all to the web API unhandled) and handling every parameter individually.

Closing thoughts

The benefit of documenting API query parameters in a client package is that you can tell the user what each parameters expects in language they can understand. That is, if you simply direct users to the docs for the web API with which the client interacts, the API docs could be not very good and/or specify types expected that the user may not understand. In addition, there may be edge cases or similar with some parameters that are not documented in the API docs but that you can document in the client docs for each parameter.

I would say the vast majority of web API clients I use that do succeed in having very few parameters also have docs in which it’s a nightmare trying to figure out what parameters each method accepts. That is, the pursuit of very few parameters at least is correlated with a very poor user experience - in my experience.

stories behind archived packages

Scott Chamberlain — Thu, 10 Sep 2020 00:00:00 +0000

Code is often arranged in packages for any given language. Packagesare often cataloged in a package registry of some kind: NPM fornode, crates.io for Rust, etc. For R, that registry is eitherCRAN or Bioconductor(for the most part).

CRAN has the concept of an archived package. That is, the namespacefor a package (foo) is still in the registry (and can not be used again),but the package is archived - no longer gets updated and checksI think are no longer performed.

We rarely hear the stories behind how software gets laid to rest. What arethe most common reasons for software to be abandoned?

My CRAN archived packages

First, my archived CRAN packages:

library(pkgsearch)
library(dplyr)
library(data.table)
library(tibble)
x = cran_events(releases = FALSE, archivals = TRUE, limit = 4000L)
res = lapply(x, function(w)
  tibble(pkg=w$name, maintainer=w$package$Maintainer))
df = rbindlist(res, use.names = TRUE, fill = TRUE)
df = as_tibble(df)
scott = filter(df, grepl("chamberlain", maintainer, ignore.case = TRUE)) %>%
  select(pkg)
scott
#> # A tibble: 16 x 1
#> pkg        
#> <chr>      
#> 1 rjsonapi   
#> 2 rdpla      
#> 3 seaaroundus
#> 4 crevents   
#> 5 etseed     
#> 6 rtimes     
#> 7 rsunlight  
#> 8 nneo       
#> 9 binomen    
#> 10 solr       
#> 11 enigma     
#> 12 alm        
#> 13 ropensnp   
#> 14 govdat     
#> 15 spoccutils 
#> 16 rgauges

Apparently I have 16 archived packages on CRAN.

Stories

The following are brief stories of why each package was archived on CRAN.

spoccutils : was a package of utility functions that didn’t quite fit within the scope of another package spocc. It was renamed to mapr. I suppose I could have asked CRAN to change the name, hmmm
rgauges: was a client for the Gaug.es website analytics API - we started the package to gather data on visitors to the rOpenSci website. Eventually we stopped using Gaug.es and then it didn’t make sense to maintain the package, so it was archived.
alm: was a client for a generic article-level metrics web service framework called Lagotto. In the early days of rOpenSci we were quite engaged with the community of folks working on article-level metrics. If I remember correctly, Lagotto usage slowed down and wasn’t used much so I gave up on maintaining alm
ropensnp : was a client for the service OpenSNP. There were other sources of SNP data that I thought would be nice to access all from one package; so a new package (rsnps) was created and incorporated functions for OpenSNP and ropensnp was archived.
solr : was a package client for the Apache Solr database. At some point it got a major overhual and I decided to change the package name to solrium.
binomen: aim of the package was functions for creating taxonomic classes/objects and functions for manipulating taxonomic data, sort of like dplyr. Evolution of ideas in binomen gave way to a new package taxa, now maintained by Zach Foster.
nneo: was a client for APIs for NEON data. At some point I stumbled upon someone else from NEON making essentially the same package, so I archived mine.
etseed: was a package for interacting with the distributed key-value store etcd. I was on a kick at the time of making R packages for databases, and saw a missing package I thought. After getting familiar with etcd, I realized I would probably never use it myself, and further, it probably didn’t make sense to interact with etcd from R anyway.
crevents: Crossref mints DOIs for scholarly articles (among other works). A neat service they started was collecting and making searchable the “events” on DOIs - that is, the links pointing to DOIs, e.g., from Twitter, etc. The service at some point became very unreliable (was often down), so the package was archived.
seaaroundus: Seaaroundus maintains fisheries and fisheries-related data. I was helping maintain an R package for the API, but it was a difficult one to maintain, and most users simply sent emails requesting dumps of data anyway - so the package was archived.
rdpla: The Digital Public Library of America is a very cool organization that is similar to Crossref in a way, in that they centralize metadata about “things”; metadata on museum collections for DPLA and scholarly works for Crossref. I figured many researchers would enjoy being able to easily get metadata from DPLA for research on museum collections. In the end not many people used the package.
rjsonapi: JSON:API is a cool idea - a sort of specification for building APIs in JSON. REST APIs are incredibly variable - this is an attempto standardize it a bit. I thought perhaps JSON:API would be adopted widely and that an R client would be useful for consuming JSON:API services - however, I’ve seen only very few APIs using JSON:API.

The following four packages were all R clients for sources of government open data - see the organization rOpenGov for R packages on government data.

govdat : was split into two packages (rsunlight and rtimes) and govdat was archived
rsunlight: was a client for many APIs of the organization Sunlight Labs - part of the reason for archiving this package was the disintegration of Sunlight Labs, which made the previously sensible organization of many APIs into one R package not sensible anymore. Also, government data was considered out of scope for our work at rOpenSci.
rtimes: was a client for a number of the government data APIs from the New York Times. One reason for abandoning this package was that NYT almost never responded to questions/feedback on their APIs. Another reason was the aforementioned focus of rOpenSci.
enigma: was a client for the Enigma API - the company I think was first focused on making open government data easier to access - as well as data on companies. I didn’t really use the package at all though, and there wasn’t much usage of the package, so archived it.

As a summary of the lists above, a list of the major reasons each package was archived:

Not used
- rjsonapi
- rdpla
- etseed
- enigma
- rgauges
- rtimes
Bad/retired service
- rtimes
- seaaroundus
- crevents
- alm
Name change
- solr
- ropensnp
- spoccutils
Evolution to new package
- binomen
- govdat
Duplicated work
- nneo
Out of scope
- rsunlight

taxizedb: an update

Scott Chamberlain — Mon, 17 Aug 2020 00:00:00 +0000

taxizedb arose from pain in using taxize when dealing with large amounts of data in a single request or doing a lot of requests of any data size. taxize works with remote data sources on the web, so there’s a number of issues that can slow the response down: internet speed, server response speed (was a response already cached or not; or do they even use caching), etc.

The idea with taxizedb was to allow users to do the same things as taxize allows, but much faster by accessing the entire database for a data source on their own computer. The previous versions of taxizedb used a variety of different databases (MySQL/MariaDB, PostgreSQL, SQLite), so the technical barrier to entry was pretty high. In the newest version just released, we’ve drastically simplified the database situation, among other things.

taxadb was developed as an alternative approach to taxizedb and should also be considered when dealing with taxonomic names. It takes a different approach for the data, with tabular files hosted on GitHub releases, but is similar in that after downloading the data is put into a SQL database, SQLite by default (with other options for databases). taxadb user facing functions are different from those in taxizedb, and largely don’t overlap.

taxizedb quick links:

Install the latest version, if you don’t get v0.2.0 with

install.packages("taxizedb")

then use

install.packages("taxizedb", repos = "https://dev.ropensci.org")

Load the package

library(taxizedb)

All SQLite!

SQLite is shipped in nearly every device these days, so taxizedb now uses only SQLite for the database backend for each data source. Every person that installs taxizedb should have SQLite already installed. In addition, there’s no usernames/passwords/ports to setup with. How we’ve accomplished this is partly through automation:

NCBI: SQLite built within R from tabular files
ITIS: they provide a SQLite dump
Plantlist: is no longer updated; we build a SQLite manually from csv files
COL: a SQLite is built once a day via GitHub Actions
GBIF: a SQLite is built once a day via GitHub Actions
Wikidata: SQLite built within R from a tabular file
World Flora Online: SQLite built within R from a tabular file

Some of the databases have indices to speed up queries, making them a bit larger relative to no indices, but these days most people likely are willing to use up a little more disk space on their computer to have faster queries.

New data sources

Three new data sources were added:

NCBI taxonomy - all of this work was done by Zebulun Arendsee
World Flora Online - the replacement for The Plant List
Wikidata - the table wikidata-taxon-info, extracted taxon objects from Wikidata, last updated April 2018, on Zenodo

New functions: taxize equivalents

Three new high level functions matching those in taxize were added: children, classification, downstream. The taxize version of those functions are still good for smaller requests, but with larger requests, its probably best to use taxizedb. The most common problem where taxize becomes frustrating is with downstream where a user wants all species within a high taxonomic rank like phylum. The original work for these functions was done by Zebulun Arendsee.

Here’s a comparison of taxize vs. taxizedb with downstream - getting all species within the genus Bombus (bumble bees)

id_tx <- taxize::get_tsn("Bombus")
system.time(taxize::downstream(id_tx, db = "itis", downto = "species"))
#> user system elapsed 
#> 2.144 0.130 20.533

id_txdb <- taxizedb::name2taxid('Bombus', db = "itis")
system.time(taxizedb::downstream(id_txdb, db = "itis", downto = "species"))
#> user system elapsed 
#> 0.132 0.051 0.186

In addition, three new “mapping” functions were added that are similar to those in taxize, but with different names: name2taxid (scientific or common name to taxonomy ID); taxid2name (taxonomy ID to scientific name); taxid2rank (taxonomy ID to rank).

We saw name2taxid above. Below we get the taxonomic ID for COL, ITIS and GBIF for Bombus

name2taxid('Bombus', db = "col")
#> [1] "3993765"
name2taxid('Bombus', db = "itis")
#> [1] "154397"
name2taxid('Bombus', db = "gbif")
#> [1] "1340278"

Get the scientific name from a taxonomic ID

taxid2name(3993765, db = "col")
#> [1] "Bombus"
taxid2rank(3993765, db = "col")
#> [1] "genus"

These functions are quite fast too:

x <- taxize::names_list(rank = "species", size = 10000L)
system.time(name2taxid(x, db = "gbif", out_type = "summary"))
#> user system elapsed 
#> 0.074 0.014 0.092

Thoughts?

Get in touch if you have any feedback at https://github.com/ropensci/taxizedb/issues

how many parameters?

Scott Chamberlain — Mon, 10 Feb 2020 08:00:00 +0000

Functions can have no parameters, or have a lot of parameters, or somewherein between. How many parameters is too many? Does it even matter how manyparameters there are in a function?

There’s AFAIK no “correct” answer to this question. And surely the “bestpractice” varies among programming languages. What do folks say aboutthis and what should we be doing in R?

From other languages

Many of the blog posts and SO posts on this topic cite the bookClean Code by “Uncle Bob”. I’ve not read the book, butit sounds worth a read.

Some of the arguments go like: too many arguments can …

makes it easier to pass arguments in the wrong order
reduce code readability
make it harder to test a function; it’s difficult/time consuming totest all various combinations of arguments work together

An analysis was done in 2018 of php open source projects, and theyfound that the most common number of parameters was 5; functions with 10parameters or more were found in <20% of projects.

On the other side, some argue thatyou shouldn’t worry so much about the correctnumber of parameters, but rather make sure that all the parameters make sense,and are documented and tested.

To the extreme, a number of people quote the Clean Code book:

The ideal number of arguments for a function is zero

Some general threads on this topic:

Data

Data for this post, created below, is in the github repo sckott/howmanyparams.

What about R?

What do the data show in the R language? Just like the blog post on php above,let’s have a look at a lot of R packages to get a general data informedconsensus on how many parameters are used per function.

It’s incredibly likely that there is a better way to do what I’ve donebelow; but this is my hacky way of getting to the data.

What I’ve done in words:

Get a list of all available package names on CRAN
Install about half of them (didn’t do all cause it takes time and I don’t think I need all 15K packages to get a good answer)
List the exported functions in each package
Count the arguments (parameters) per function in each package
Visualize the results

I ended up using 82489 functions across 4777 packages

Load packages

library(plyr)
library(dplyr)
library(tibble)
library(ggplot2)

Use a different path from my actual R library location to not pollutemy current setup, and put binaries into a temporary directory so they are cleaned up on exiting R.

path <- "/some/path"
binaries <- file.path(tempdir(), "binaries")
dir.create(path)
dir.create(binaries)
.libPaths(path)
.libPaths() # check that the path was set

Function do_one() to run on each package:

try to load the package
if not found install it
get a vector of the exported functions in the package
count how many arguments each function has, make a data.frame
unload the package namespace

do_one <- function(pkg) {
  if (!requireNamespace(pkg))
    install.packages(pkg, quiet=TRUE, verbose=FALSE, destdir = binaries)
  on.exit(unloadNamespace(pkg))
  funs <- paste0(pkg, "::", getNamespaceExports(pkg))
  enframe(vapply(funs, function(w) {
    tt <- tryCatch(parse(text = w), error = function(e) e)
    if (!inherits(tt, "error")) length(suppressWarnings(formals(eval(tt)))) else 0
  }, numeric(1)))
}
do_one_safe <- failwith(tibble(), do_one)

Get a list of packages. First time running through I used available.packages() whichgets you all available packages. After installing packages though, I usedinstalled.packages() to get the list of packages I already installed.

# pkg_names <- unname(available.packages()[,"Package"])
pkg_names <- unname(installed.packages()[,"Package"])

Run each package through the do_one() function. This had to be stopped andre-started a few times. This failed for quite a few packages - I wasn’t trying to get every single package, just a large set of packages to get an idea of what packages do on average.

tbls <- stats::setNames(lapply(pkg_names, do_one_safe), pkg_names)

Combine list of data.frame’s into one data.frame

df <- dplyr::bind_rows(tbls, .id = "pkg")
readr::write_csv(df, "params_per_fxn.csv")

note: you can get this data at sckott/howmanyparams

df <- readr::read_csv("~/params_per_fxn.csv")

Visualize

All functions across all packages

ggplot(df, aes(x = value)) +
  geom_histogram(bins = 30) +
  scale_x_continuous(limits = c(0, 30)) +
  theme_grey(base_size = 15)

The mean number of arguments per function across all packages was 4.4,and the most common value was 3. The maximum number of arguments was209, and there were 5306 functions(or 6.4%) with zeroparameters.

Mean params across functions for each pkg

df_means <- group_by(df, pkg) %>% 
  summarise(mean_params = mean(value, na.rm=TRUE)) %>% 
  ungroup()
# arrange(df_means, desc(mean_params))
ggplot(df_means, aes(x = mean_params)) +
  geom_histogram(bins = 50) +
  scale_x_continuous(limits = c(0, 30)) +
  theme_grey(base_size = 15)

Taking the mean within each package first pulls the number of arguments to the right some,with a mean of 5 arguments, and the most common value at 4.

Thoughts

In terms of getting around the too many arguments thing, there’s talk ofusing global variables, which seems like generally a bad idea; unless perhapsthey are environment variables that are meant to be set by the user innon-interactive environments, etc.

Other solutions are to use ... in R, or similarly **kwargs or *args in Python (ref.), orthe newly added ... in Ruby (ref). With this approach you could have very few parametersdefined in the function, and then internally within the function handle any parameterfiltering, etc. The downside of this in R is that you don’t get the automatedchecks making sure all function arguments are documented, and there’s no documentedarguments that don’t exist in the function.

I’m not suggesting a solution is needed though; there’s probably no right answer, but rather lots of opinions.

Having said that, the average R function does use about 4 arguments, so if you keep your functions to around 4 arguments you’ll be approaching the sort ofconsensus of a large number of R developers.

Last, I should admit that some of the functions in my packages have quite a lotof parameters - which was sort of the motivation for this post - that is, to explorewhat most functions do. For example, brranching::phylomatic has 13 parameters,three functions in the crevents package have 24 parameters … and I wonderabout these types of functions. Should I refactor? Or is it good enough to makesure these functions are thoroughly documented and tested?

finding truffles

Scott Chamberlain — Thu, 30 Jan 2020 08:00:00 +0000

The bad thing about making software is that you can sometimes make it easierfor someone to shoot themselves in the foot. The good thing about softwareis that you can make more software to help them not shoot a foot off.

The R package vcr, an R port of the Ruby library of the same name,records and plays back HTTP requests. Some HTTP requests can have secrets (e.g.,passwords, API keys, etc.) in their requests and/or responses. These secretscan then accidentally end up on the Internet, where bad people may find them.These secrets are sometimes called “truffles”.

There’s a suite of tools out there for finding these truffles (e.g.,truffleHog, gitsecrets) that use tools like regex and entropy.

Despite there being existing tools, users tend to use things that arebuilt in the language(s) they know; that are easy to incorporate into their existing workflows. Towards this end, I’ve been working on a newR package trufflesniffer.

trufflesniffer doesn’t do any fancy entropy stuff, and doesn’t try tofind secrets without any informed knowledge. Rather, the user suppliesthe secrets that they want to look for and trufflesniffer looks forthem. In the future I’d look to see if it can be used withoutany user inputs.

terminology:

sniff: search for a secret

links:

Install

remotes::install_github("ropenscilabs/trufflesniffer")


library(trufflesniffer)

package

sniff through a whole package

foo <- function(key = NULL) {
  if (is.null(key)) key <- "mysecretkey"
}
package.skeleton(name = "mypkg", list = "foo", path = tempdir())
pkgpath <- file.path(tempdir(), "mypkg")
list.files(pkgpath, recursive=TRUE)
#> [1] "DESCRIPTION" "man/foo.Rd" "man/mypkg-package.Rd"
#> [4] "NAMESPACE" "R/foo.R" "Read-and-delete-me"

# check the package
sniff_secrets_pkg(dir = pkgpath, secrets = c("mysecretkey"))
#> $mysecretkey
#> $mysecretkey$foo.R
#> [1] 3

fixtures

sniff specifically in a package’s test fixtures.

Create a package

foo <- function(key = NULL) {
  if (is.null(key)) key <- "a2s323223asd423adsf4"
}
package.skeleton("herpkg", list = "foo", path = tempdir())
pkgpath <- file.path(tempdir(), "herpkg")
dir.create(file.path(pkgpath, "tests/testthat"), recursive = TRUE)
dir.create(file.path(pkgpath, "tests/fixtures"), recursive = TRUE)
cat("library(vcr)
vcr::vcr_configure('../fixtures', 
  filter_sensitive_data = list('<<mytoken>>' = Sys.getenv('MY_KEY'))
)\n", file = file.path(pkgpath, "tests/testthat/helper-herpkg.R"))
cat("a2s323223asd423adsf4\n", 
  file = file.path(pkgpath, "tests/fixtures/foo.yml"))
# check that you have a pkg at herpkg
list.files(pkgpath)
#> [1] "DESCRIPTION" "man" "NAMESPACE"         
#> [4] "R" "Read-and-delete-me" "tests"
list.files(file.path(pkgpath, "tests/testthat"))
#> [1] "helper-herpkg.R"
cat(readLines(file.path(pkgpath, "tests/testthat/helper-herpkg.R")),
  sep = "\n")
#> library(vcr)
#> vcr::vcr_configure('../fixtures', 
#> filter_sensitive_data = list('<<mytoken>>' = Sys.getenv('MY_KEY'))
#> )
list.files(file.path(pkgpath, "tests/fixtures"))
#> [1] "foo.yml"
readLines(file.path(pkgpath, "tests/fixtures/foo.yml"))
#> [1] "a2s323223asd423adsf4"

Check the package

Sys.setenv('MY_KEY' = 'a2s323223asd423adsf4')
sniff_secrets_fixtures(pkgpath)
#> $MY_KEY
#> $MY_KEY$foo.yml
#> [1] 1

sniffer

The function sniffer() wraps the function sniff_secrets_fixtures() andpretty prints to optimize non-interactive use. Run from within R or from the command line non-interactively.

Example where a secret is found:

sniffer(pkgpath)

Example where a secret is not found:

Sys.unsetenv('MY_KEY')
sniffer(pkgpath)

To do

There’s more to do. trufflesniffer hasn’t been tested thoroughly yet; I’ll domore testing to make the experience better. In addition, it’d probably be best to integrate this into the R vcr package so that the user doesn’t have totake an extra step to make sure they aren’t going to put any secrets onthe web.

ack: trufflesniffer uses R packages cli and crayon

text mining, apis, and parsing api logs

Scott Chamberlain — Thu, 21 Mar 2019 07:00:00 +0000

Acquiring full text articles

fulltext is an R package I maintain to obtain full text versions of research articlesfor text mining.

It’s a hard problem, with a sphagetti web of code. One of the hard problems is figuring out what the URL is for the full text version of an article. Publishersdo not have consistent URL patterns through time, and so you can not set rules once and never revisit them.

The Crossref API has links available to full text versions for publishers that choose to share them. However, even if publishers choose to share their full text links, they may be out of date or completelywrong (not actually lead to the full text).

There’s a variety of other APIs out there for getting links to articles, but none really hit the spot, which lead to the creation of the ftdoi API.

the ftdoi API

The ftdoi API is a web API, with it’s main goal for getting a best guess atthe URL for the full text version of an article from it’s DOI (this is done via the /api/doi/{doi}/ route). The API gives back URLs for all those possible, includingpdf, xml, and html. Most publishers only give full text as PDF, but when XML isavailable we give those URLs as well.

The API uses the rules maintained in the pubpatternsrepo. The rules are only rough guidelines though and often require at least one step of making a web request to the publishers site or another site, that’s NOT specified in the pubpatterns rules. For example, the Biorxiv filehas notes about how to get the parts necessary for the full URL, but the actual logic to do so in in the API code base here.

The ftdoi API caches responses for requests for 24 hrs, so even if a request takes 5 secondsor so to process, at least for the next 24 hrs it will be nearly instantaneous. We don’t want to cache indefinitely because URLs may be changed at any time by the publishers.

The fulltext package uses the ftdoi API internally, mostly hidden from users, to get a full text URL.

But why an API?

Why not just have a set of rules in the fulltext R package, and go from there?An API was relatively easy for me to stand up, and i think it has many benefits:can be used by anything/anyone, not just R users; updates to publisher specificrules for generating URLs can evolve independently of fulltext; the logscan be used as a tool to improve the API.

what do people actually want?

The ftdoi API has been running for a while now, maybe a year or so, and I’ve been collecting logs. Seems smart to look at the logs to determine what publishers users are requesting articles from that the ftdoi API does not currently support,so that the API can hopefully add those publishers. For obvious reasons, I can’t sharethe log data.

Load packages and define file path.

library(rcrossref)
library(dplyr)
library(rex)
logs <- "~/pubpatterns_api_calls.log"

The logs look like (IP addresses removed, some user agents shortened):

[28/Nov/2018:20:09:49 +0000] GET /api/members/ HTTP/2.0 200 4844 Mozilla/5.0 ...
[28/Nov/2018:20:23:15 +0000] GET /api/members/317/ HTTP/2.0 200 228 Mozilla/5.0 ...
[29/Nov/2018:01:52:58 +0000] GET /api/members/19/ HTTP/1.1 400 65 fulltext/1.1.0
[29/Nov/2018:01:52:58 +0000] GET /api/members/2308/ HTTP/1.1 400 65 fulltext/1.1.0
[29/Nov/2018:01:52:59 +0000] GET /api/members/239/ HTTP/1.1 400 65 fulltext/1.1.0
[29/Nov/2018:01:53:00 +0000] GET /api/members/2581/ HTTP/1.1 400 65 fulltext/1.1.0
[29/Nov/2018:01:53:00 +0000] GET /api/members/27/ HTTP/1.1 400 65 fulltext/1.1.0
[29/Nov/2018:01:53:01 +0000] GET /api/members/297/ HTTP/1.1 200 336 fulltext/1.1.0
[29/Nov/2018:01:53:01 +0000] GET /api/members/7995/ HTTP/1.1 400 65 fulltext/1.1.0
[29/Nov/2018:10:46:53 +0000] GET /api/members/unknown/ HTTP/1.1 400 65 fulltext/1.1.0.9130

Use the awesome rex package from Kevin Ushey et al. to parse the logs, pulling outjust the Crossref member ID in the API request, as well as the HTTP status code. Thereare of course other API requests in the logs, but we’re only interested here in the onesto the /api/doi/{doi}/ route.

df <- tbl_df(scan(logs, what = "character", sep = "\n") %>%
  re_matches(
    rex(
      "/api/members/",
        capture(name = "route",
          one_or_more(numbers)
        ),
      "/",

      space, space, "HTTP/", numbers, ".", numbers, space,

      capture(name = "status_code",
        one_or_more(numbers)
      )
    )
  ))
df$route <- as.numeric(df$route)
df
#> # A tibble: 896,035 x 2
#> route status_code
#> <dbl> <chr>      
#> 1 NA <NA>       
#> 2 317 200        
#> 3 19 400        
#> 4 2308 400        
#> 5 239 400        
#> 6 2581 400        
#> 7 27 400        
#> 8 297 200        
#> 9 7995 400        
#> 10 NA <NA>       
#> # … with 896,025 more rows

Filter to those requests that resulted in a 400 HTTP status code, that is, they resulted in no returned data, indicating that we likely do not have a mapping for that Crossref member.

res <- df %>%
  filter(status_code == 400) %>%
  select(route) %>% 
  group_by(route) %>%
  summarize(count = n()) %>% 
  arrange(desc(count))
res
#> # A tibble: 530 x 2
#> route count
#> <dbl> <int>
#> 1 10 345045
#> 2 530 7165
#> 3 286 3062
#> 4 276 2975
#> 5 239 2493
#> 6 8611 1085
#> 7 56 853
#> 8 235 722
#> 9 382 706
#> 10 175 590
#> # … with 520 more rows

Add crossref metadata

(members <- rcrossref::cr_members(res$route))
#> $meta
#> NULL
#> 
#> $data
#> # A tibble: 530 x 56
#> id primary_name location last_status_che… total.dois current.dois
#> <int> <chr> <chr> <date> <chr> <chr>       
#> 1 10 American Me… 330 N. … 2019-03-20 600092 14714       
#> 2 530 FapUNIFESP … FAP-UNI… 2019-03-20 353338 38339       
#> 3 286 Oxford Univ… Academi… 2019-03-20 3696643 289338      
#> 4 276 Ovid Techno… 100 Riv… 2019-03-20 2059352 167272      
#> 5 239 BMJ BMA Hou… 2019-03-20 891239 61267       
#> 6 8611 AME Publish… c/o NAN… 2019-03-20 20067 15666       
#> 7 56 Cambridge U… The Edi… 2019-03-20 1529029 84018       
#> 8 235 American So… 1752 N … 2019-03-20 178890 13984       
#> 9 382 Joule Inc. 1031 Ba… 2019-03-20 12666 1868        
#> 10 175 The Royal S… 6 Carlt… 2019-03-20 89219 7262        
#> # … with 520 more rows, and 50 more variables: backfile.dois <chr>,
#> # prefixes <chr>, coverge.affiliations.current <chr>,
#> # coverge.similarity.checking.current <chr>,
#> # coverge.funders.backfile <chr>, coverge.licenses.backfile <chr>,
#> # coverge.funders.current <chr>, coverge.affiliations.backfile <chr>,
#> # coverge.resource.links.backfile <chr>, coverge.orcids.backfile <chr>,
#> # coverge.update.policies.current <chr>,
#> # coverge.open.references.backfile <chr>, coverge.orcids.current <chr>,
#> # coverge.similarity.checking.backfile <chr>,
#> # coverge.references.backfile <chr>,
#> # coverge.award.numbers.backfile <chr>,
#> # coverge.update.policies.backfile <chr>,
#> # coverge.licenses.current <chr>, coverge.award.numbers.current <chr>,
#> # coverge.abstracts.backfile <chr>,
#> # coverge.resource.links.current <chr>, coverge.abstracts.current <chr>,
#> # coverge.open.references.current <chr>,
#> # coverge.references.current <chr>,
#> # flags.deposits.abstracts.current <chr>,
#> # flags.deposits.orcids.current <chr>, flags.deposits <chr>,
#> # flags.deposits.affiliations.backfile <chr>,
#> # flags.deposits.update.policies.backfile <chr>,
#> # flags.deposits.similarity.checking.backfile <chr>,
#> # flags.deposits.award.numbers.current <chr>,
#> # flags.deposits.resource.links.current <chr>,
#> # flags.deposits.articles <chr>,
#> # flags.deposits.affiliations.current <chr>,
#> # flags.deposits.funders.current <chr>,
#> # flags.deposits.references.backfile <chr>,
#> # flags.deposits.abstracts.backfile <chr>,
#> # flags.deposits.licenses.backfile <chr>,
#> # flags.deposits.award.numbers.backfile <chr>,
#> # flags.deposits.open.references.backfile <chr>,
#> # flags.deposits.open.references.current <chr>,
#> # flags.deposits.references.current <chr>,
#> # flags.deposits.resource.links.backfile <chr>,
#> # flags.deposits.orcids.backfile <chr>,
#> # flags.deposits.funders.backfile <chr>,
#> # flags.deposits.update.policies.current <chr>,
#> # flags.deposits.similarity.checking.current <chr>,
#> # flags.deposits.licenses.current <chr>, names <chr>, tokens <chr>
#> 
#> $facets
#> NULL

Add Crossref member names to the log data.

alldat <- left_join(res, select(members$data, id, primary_name),
  by = c("route" = "id"))
alldat
#> # A tibble: 530 x 3
#> route count primary_name                             
#> <dbl> <int> <chr>                                    
#> 1 10 345045 American Medical Association (AMA)       
#> 2 530 7165 FapUNIFESP (SciELO)                      
#> 3 286 3062 Oxford University Press (OUP)            
#> 4 276 2975 Ovid Technologies (Wolters Kluwer Health)
#> 5 239 2493 BMJ                                      
#> 6 8611 1085 AME Publishing Company                   
#> 7 56 853 Cambridge University Press (CUP)         
#> 8 235 722 American Society for Microbiology        
#> 9 382 706 Joule Inc.                               
#> 10 175 590 The Royal Society                        
#> # … with 520 more rows

Theres A LOT of requests to the American Medical Association. Coming ina distant second is FapUNIFESP (SciELO), then the Oxford University Press,Ovid Technologies (Wolters Kluwer Health), BMJ, and AME Publishing Company, all with greater than 1000 requests.

These are some clear leads for publishers to work into the ftdoi API, workingmy way down the data.frame to less frequently requested publishers.

more work to do

I’ve got a good list of publishers which I know users want URLs for, so I can get started implementing rules/etc. for those publishers. And I can repeat this process from time to time to add more publishers in high demand.

Exceptions in control flow in R

Scott Chamberlain — Mon, 04 Mar 2019 08:00:00 +0000

I was listening to a Bike Shed podcast episode 189, "It's Gonna Work, Definitely, No Problems Whatsoever", and starting at 27:44 there was a conversation about exception handling. Specifically it was about exception handling in control flow when doing web API requests. This topic piqued my interest straight away as I do a lot of API stuff (making and wrapping).

The part of the conversation that I want to address is their conclusion that exceptions in control flow are an anti-pattern. Seems this is a general pattern in programming languages, e.g., this SO thread. But on the contrary there are some languages in which exceptions in control flow are considered normal behavior; e.g., Python (this, this).

My first reaction to this was one of vehement disagreement because in my experience wrapping web APIs raising exceptions on HTTP status codes of 400 and 500 series is the norm, in at least R and Ruby. They argued that there are better ways of handling these cases. After a whg with my gut reaction.

Let's take a step back first and look at some concepts before diving further into this.
ile I thought maybe the topic is worth thinking harder about rather than goin

control flow

Control flow in programming is

the order in which individual statements ... are executed or evaluated ... a control flow statement is a statement, the execution of which results in a choice being made as to which of two or more paths to follow.

Control flow in R has an overview of control flow in R. Control flow constructs in R include if/else, for, while, repeat, break, next.

exceptions

Exceptions are

anomalous or exceptional conditions requiring special processing - often changing the normal flow of program execution (source: wikiex)

In R, exception handling can be done with try, tryCatch, withCallingHandlers and others. Often warning() is used to signal to the user what happened, but does not stop execution, and can be suppressed with suppressWarnings(). To stop execution, stop() is used.

back to web API exceptions and control flow in R

Note the word exceptional above in our definition of exceptions. The BikeShed pod hosts were surprised to see exceptions raised with bad API requests because they didn't think a bad API request was exceptional, but rather an expected result given certain conditions (e.g., an HTTP 400 series client error means the client did something wrong and perhaps the server gave back a useful error message to help fix the request).

They observed that most Ruby API wrappers did have the behavior of rasing an exception on a 400/500 series API status, but they disagreed with this approach.

In R world, most API wrappers in my experience also follow the pattern of raising an exception stopping the code flow on a 400/500 series HTTP error.

What would it look like to not stop code execution flow when 400/500 series errors are returned from web API requests? What would need to change from the current setup? How would users be affected?

A typical R function that makes a web API request looks like the following:

foo = function(path, query = list()) {
  conn = crul::HttpClient$new("https://httpbin.org")
  res = conn$get(path = path, query = query)
  res$raise_for_status()
  res$parse("UTF-8")
}

On a successful request all is good and we get back the JSON payload

foo(path = "get", query = list(apple = "pink lady"))
#> [1] "{\n  \"args\": {\n    \"\": \"pink lady\"\n  }, ...

When there is a 400/500 series code the line res$raise_for_status() throws an error, stopping execution

foo(path = "status/400")
#> Error: Bad Request (HTTP 400)

Instead of raising an error we could throw a warning and proceed to the next step

bar = function(path, query = list()) {
  conn = crul::HttpClient$new("https://httpbin.org")
  res = conn$get(path = path, query = query)
  if (res$status_code >= 400) {
    warning(sprintf("HTTP %s %s", res$status_code, res$status_http()$explanation))
  }
  res$parse("UTF-8")
}

bar(path = "status/400")
#> [1] ""
#> Warning message:
#> In bar(path = "status/400") :
#>  HTTP 400 Bad request syntax or unsupported method

This is fine, but there's a few scenarios in which this will be problematic:

Many APIs DO NOT return the same content-type on a 400 series error, and even more common on 500 series errors. In fact, often JSON APIs return an HTML error page, which may or may not contain a meaningul message, instead of the same content type as a successful response (e.g., JSON).
Rather then simply parsing the response res$parse("UTF-8"), the downstream code may be more complex (e.g., selecting particular fields/keys), and may fail out (and in R, this often means useless error messages for the user).

If we take their advice and don't fail out on 400/500 series codes, what would that look like? One could do something like:

hello_world <- function(path, query = list()) {
  conn = crul::HttpClient$new("https://httpbin.org")
  res = conn$get(path = path, query = query)
  if (res$status_code >= 400) {
    warning(sprintf("HTTP %s %s", res$status_code, res$status_http()$explanation))
  }
  res
}

We still get the warning on an error

hello_world(path = "status/400")
#> Warning message:
#> In hello_world(path = "status/400") :
#>   HTTP 400 Bad request syntax or unsupported method

But also we return the response object (HttpResponse from the crul package in this case):

#> <crul response>
#>   url: https://httpbin.org/status/400
#>   request_headers:
#>     User-Agent: libcurl/7.54.0 r-curl/3.3 crul/0.7.0
#>     Accept-Encoding: gzip, deflate
#>     Accept: application/json, text/xml, application/xml, */*
#>   response_headers:
#>     status: HTTP/1.1 400 BAD REQUEST
#>     access-control-allow-credentials: true
#>     access-control-allow-origin: *
#>     content-type: text/html; charset=utf-8
#>     date: Mon, 04 Mar 2019 17:49:39 GMT
#>     server: nginx
#>     content-length: 0
#>     connection: keep-alive
#>   status: 400

Now the user can explore the response body, response headers, etc. and decide on their own what to do instead of the function failing out and returning nothing.

This approach is fine if your users are more advanced, but most packages/libraries are probably trying to give back a data object that users are familiar with. In R, that is clearly the data.frame. When there is a 400/500 series error, one option is to return an empty data.frame and throw a warning about the error, hopefully with enough information for the user to fix the request. This is probably best for naive users, but any package has some more advanced users that would benefit from more information; and more information will help a naive user + the maintainer debug a problem eaiser.

The next more complicated option would be a list that can have the same format regardless of errors or not:

func <- function() {
  res <- hello_world(path = "status/400")
  mssg <- sprintf("HTTP %s %s", res$status_code, res$status_http()$explanation)
  list(data = res$parse("UTF-8"), error = mssg)
}

gives

func()
#> $data
#> [1] ""
#> 
#> $error
#> [1] "HTTP 400 Bad request syntax or unsupported method"

Or possibly something more complex where you can build in accessors to make it easy to get data the user expects, but also dig into the HTTP response object itself if needed:

Response <- R6::R6Class("Response",
  public = list(
    x = NULL,
    initialize = function(resp) self$x <- resp,
    data = function() self$x$parse("UTF-8"),
    error = function() {
      sprintf("HTTP %s %s", self$x$status_code, self$x$status_http()$explanation)
    }
  )
)
myfunc <- function() {
  res <- hello_world(path = "status/400")
  Response$new(res)
}

Which gives:

out <- myfunc()
# the HTTP message
out$error()
#> [1] "HTTP 400 Bad request syntax or unsupported method"
# the response body, parsed
out$data()
#> [1] ""
# the full HTTP response object
out$x

what about users handling exceptions on their side?

If one sticks swith erroring out of excecution flow with 400/500 series errors, the user can still handle it on their end. For example, if they are using a function in a loop/appply type call, they can use tryCatch or similar and check for an error and proceed one of two or more ways depending on the error or successful request. Of course this assumes that the user knows how to do this.

Additionally, this means that each user will handle errors in different ways, possibly making mistakes in the process - arguing for the developer of the package to handle exceptions instead.

it's too complex, just fail out

One reason I like to fail out on 400/500 series errors in my packages is that there is often significant data munging of the response. Failing out makes my life easier as I don't have to worry about what to do with HTTP responses that fail. In the world I run in of smallish APIs for science/research, API failure behavior often is not very good; it's typically unpredictable, changes from time to time, and failure response bodies are often just their HTML failure page, leading to brittle code for parsing that HTML as that HTML can change often. It'd be great if every API was as good as Github's for example, but we'll never be in that place.

performance considerations

In reading about exceptions in control flow, there's a common thread about performance (e.g., c++, Ruby 1, Ruby 2). That is, if throwing exceptions is a slow procedure, that's one reason to avoid them. But if exceptions aren't slow then that's not a great argument for avoiding them.

I haven't seen anything on performance an exceptions in R, though I'm sure there's something out there.

Even if exceptions are a slowish procedure, there is an argument to be made that failing early also saves time; that is, if you get a 400/500 series error you aren't then spending time with downstream processing of the response. However, then the user has less information. Trade-offs all the way down.

conclusion

I'm not sure if I'll change anything in packages I maintain or not. I'll keep thinking aobut this and ask around to gauge others opinions on this. Part of me wants to follow the avoid exceptions path, but I worry about two things. First, the complexity increases for me as the developer. If I don't fail out, then I have to deal with parsing somehow every response. It's not as simple as giving back the HTTP response; I ideally want to give users a data structure they are familiar with, i.e., a data.frame. Second, for the user, if I give back a list or an R6 object, that increases complexity on their side. Is the benefit of more information worth the cost of more complexity for the user? I've no idea.