Forem: Dave Parr

Moving Starpilot to GraphQL

Dave Parr — Tue, 16 Jan 2024 00:00:00 +0000

After building Starpilot at the end of last year and using it for a little. I became pretty frustrated with the time it took to create the vectorstore. Specifically, the time it took to read the relevant repo data from GitHub’s REST api via PyGithub. I outlined the choices I made in the original blog post, but to summarise: PyGithub wraps the GitHub REST api and is used to read the repos that are starred by a user on GitHub, and then pass this to a function that iterates over each repo for each piece of data I want to extract. I’m sure that for many (most) use cases this isn’t an issue, but for getting 9 pieces of data for lists of sometimes hundreds or even thousands of repos, it’s the wrong tool for the job. There is overhead to set up the connection, then execute the call, and wait for the response to come back. The total number of calls is roughly:

9[pieces of information] * 1000[number of repos] + 1[list of user starred repos]

This is slow. To be fair I wasn’t surprised by this. I actually expected it and just accepted it so I could get onto making the tool somewhat useful. Now I know it is useful though, it’s time to optimise. For my list of 800 starred repos it takes about 20 minutes on my desktop. Luckily, I’ve found a way to 10x the speed of this process down to 2 minutes for the same list on the same machine using GraphQL.

How I did it

GraphQL

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. It provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools. GraphQL is perfect for the needs of this project as:

GitHub GraphQL Explorer

GitHub provides a GraphQL api. The Github GraphQL api allows you to structure a query for a specific set of information for a repo, then use that structure to return the same set of of structured information for each repo that a specific user has starred, all in one call. GitHub also provides a GraphQL explorer to help you build your queries. However after a while I found this limiting, mostly because the page renders so that only around a 3rd of the screen is the playground with a mad amount of space for the header, footer and sidebar. Therefore I installed:

Insomnia

Insomnia is a desktop editor for developing APIs. I liked it because the workspace is more fully featured, allowing you to save queries and sync them to your cloud account for storage, and handles authentication and suppliying variables. Because part of the GraphQL specification is that each API can be introspected it also behaved exactly like the GitHub Explorer and provided autocomplete, documentation and error linting for the queries I was writing. This was a huge help in getting the queries right. GitHub’s GraphQL interface is HUUUUGE. With Insomnia I was able to build up the query I needed in stages, testing each part as I went (Insomnia the app, not the sleep problem). However, I still needed to integrate this into my code. Most tutorials seem to suggest that the thing to do is just take the query, wrap it in """ myQuery here""" and pass it to requests, but that seemed like a perfect way to accidentally break 90 lines of code with an unlintable typo. I wanted to use a python library that would allow me to build the query in a more structured way that is harder to break by accident, and also allow me to pass variables to the query. I found:

`graphql-query`

denisart/graphql-query is a Python package to build GraphQL queries from the core abstract pieces of the GraphQL language. I found this really useful as I learnt more about GraphQL and how it’s mental model is formed of fields with potentially optional arguments, nodes, edges and fragments. It also helped to make editing in VS Code directly a lot easier as it supports type hinting, so you can’t put an Argument or Field in the wrong place. As I iterated over the last tweaks to the query I found I actually didn’t need to go back to Insomnia as much. However, graphql-query doesn’t actually act as a client to the GraphQL api, it just builds the query. For that I needed:

`gql`

gql is a Python library for handling GraphQL queries in Python through a client. This became particularly useful as a (reasonable) limitation of GitHub’s GraphQL API is that when returning back a repo type node, you can only return up to 100 per call. Therefore I needed to paginate through the results. I would have liked it if gql had a way to handle this for me, but I couldn’t find it if it does, so I wrote a small bit of simple logic to handle that. The slightly tricky thing about pagination on this node is that you need to handle it with a GraphQL cursor. This is a string that is returned with each node that you can use to tell the API where to start the next page of results from. I found that the easiest way to handle this was to use a while loop and pass the cursor back to the query as a variable. When the query return is empty for a last cursor, the loop breaks, and the results are returned. I did debate a more complex approach that involved counting the number of expected repos first, modulo 100, and then using that to determine the number of pages to request, but I decided that was overkill for my use case.

Other tweaks

Because the data that is returned from the API comes as a dict for each repo, I modified the old function that iterated over each repo to get all the data for a specific repo. This function also contained the logic to write the data to disk for iteration over by langchain, so it seemed a suitable change. As I’d also learnt some more about jq and how it is used in langchain I realised that instead of creating a json file with 2 dicts, one for metadata {...} and one for content {...} I realised I could just write the data structure as a single dict, and then use jq to extract the data I needed via tha optional metadata_func argument in langchain.document_loaders.JSONLoader. I got tripped up for a while as I had mistakenly written a behaviour into metadata_func that effectively re-imputed a None value into the description value after the format_repo() function had taken the value and key out of the file saved to disk, which was then passed to Chroma via the JSONLoader. I managed to debug that however, and then I implemented the suggestion from the helpful error message to use langchain.vectorstores.utils.filter_complex_metadata() as a (slightly redundant) safeguard. I’d already got it wrong once so…

The tweaks that didn’t make it

A version of the GraphQL query that I used originally went the whole hog and also returned the README.md/README.rst from the repo. I’m certain that many of the readmes have great context that will enhance the embedding in the vectorstore, however I found that returning all that data significantly eroded the performance gains. To the extent that though I had 10x speeded up the process of reading the data from GitHub, I was now spending 10x longer generating the embedding via langchain.embeddings.GPT4ALLEmbeddings. I know that the OpenAI embedding model seems to be much faster from some trials I’ve done, but I’m not (yet) happy to migrate over as it increases the cost of running starpilot. However, not to the extent that it’s unviable (probably still less than a cup of coffee), but I’m not sure I’m ready to commit to that yet. Maybe as I work towards building an agent architecture for starpilot I’ll revisit this. When I do, I’ll probably implement a way to summarise the use case from the readmes and strip out the things like installation instructions. That summary would then be the part used in the embedding.

Conclusion

Using Github’s GraphQL API has allowed me to 10x the speed of reading the data from GitHub. Developing for GraphQL is maybe even less of a hassle than for REST APIs, as the introspection allows you to build complex queries in a more structured way with a better linting and autocomplete driven experience up front. I’m not sure that means all API’s should be GraphQL. It’s pretty clear the benefits come when the data that is being returned (or operations being performed) are being done on data objects with a huge amount of structure. The other benefit is being able to batch up calls on that structure. The introspection aspects of it also help with learning, developer experience and good documentation, however arguably a chunk of this idea is already available on REST APIs via OpenAPI/Swagger, so it’s not a unique selling point.

Developing GraphQL in Python also seems pretty robust, with a number of other options I didn’t include for GraphQL clients in the language ecosystem, as well as other tools that I didn’t need for starpilot like GraphQL servers. I had worried it would be pretty under tooled, and that GraphQL client side was a very “JavaScript” thing to do. Luckily that wasn’t the case.

In the future either on this project or another I’d like to explore graphql-query ’s support for fragments to dynamically extend the query render at call time (e.g. to include the readme data in the query based on a boolean flag in the function signature). I’d also maybe look at gql’s support for creating a Domain Specific Language (DSL) for a given GraphQL schema. I only spotted that after I had finished my work with graphql-query so I didn’t spend time resolving the same problem, but if I need to write more GraphQL in Python I’ll definitely look at that.

Hopefully this speed up will make starpilot more approachable for a lot of people. I know I’ve been irritated in my own development cycle iterating over star pilot read and waiting for the data to come back, so at the very least I’ve saved myself some time and learnt a bunch in the process.

How do you use your VSCode profile?

Dave Parr — Fri, 12 Jan 2024 14:03:32 +0000

I've been using VScode profiles for a while, but I fee like my profiles are turnung into a mess.

I started by just making a python profile. Then I made a profile from that one named data science. Then I made one with markdown tools and cspell for writing blog posts and note taking. Then I made one for rust. And one for writing django projects. Plus theres the built in default one.

The problems I'm now getting are that I wonder why my editor in data science view isn't type checking the way I expected it to, and it's because I tweaked those settings in Python, not Data Science. and I've found a great tool that I've added to my Django profile that helps with lots of python things, but I can only add it upwards to my default profile, not my python profile or the data science one, without having to switch it in, refind and install it, then switch back to what I was doing.

So the question is: Have you found a way to manage your profiles that you are happy with?

I feel the problem is that the profile is a 'leaky abstraction'. It allows a lot of customisability to do what I've done, but that ends up with a muddle of profiles split by language OR task.

I feel like a neat solution is to adopt profiles of

default
python
rust
R

web development
data science
writing

What do you all think?

Copilot for your GitHub stars

Dave Parr — Sun, 19 Nov 2023 17:08:15 +0000

How do you use your GitHub stars?

I'd guess if you've been programming for a few years you've probably hit the star button at the top of a few of your favourite repos. I know some people I follow have done it thousands of times. Do you go back to them though? Do you review them for inspiration for your next project or go to them when you're stuck on a partictular problem?

Inspiration

I've always assumed I would use them but I never have. I found myself doing some research recently into how to build software that uses LLMs, with the deliberate goal of building an as yet undefined side-project. I wanted to build something I hadn't built before, something that was hopefully a little original, and maybe even useful! So yet again I was starring repos like LangChain and Chroma, swearing this time would be different.

As I was running through blog posts and diligently smashing the star buttons I realised that I had just hit on exactly what I wanted to try. I wanted to bring my GitHub stars right into my editor. I wanted to be able to have them next to me as I was working and get a sensible set of suggestions on what might be useful for my needs at that moment, and I had just been starring the exact repos that could make this happen!

The original idea

Use a dataset of your personal stars to inform retrival augmented generation for a question and answer large language model deployed in a command line interface

I thought this would be useful for a few reasons:

By having it in the CLI its available right in my editor, and to every project.
By having a set of your personal stars the suggestions are already curated by your interests and preferences. Mine are all Python and R librarys, wierd data bases and charting libraries. Yours might mostly be Ruby gems, or web frameworks, or tools for embedded systems.
By using a large language model the tool might be more capable of understanding the intentention of your goals, for instance the query "Suggest how to build a web app" might be able to infer that you'd likely want a front end component, a backend component and a data storage component, and might even deal with servers and deployment.
By using large language models the tool might be more capable of semantic search rather than keyword matching which suits this problem as there is no strong standard on how a library describes it self through it's topics, description and documentation.

Semantic vs keyword

Keyword

A keyword search looks for the exact letters in a string, or potentially a partial match. As an example the query "Data Science" would find things that exactly matched the charcters in the string "Data Science" and maybe also ["Data", "Science", "DS"].

Semantic

A semantic search looks for the conceptual similarity between things, so in this context "Data Science" would find things that matched the vector embedding of "Data Science" as well as maybe also the vector embeddings that are associated with ["Machine Learning", "Artificial Intelligence"]

And so I ran poetry new starpilot:

DaveParr / starpilot

Use your GitHub stars for great good!

Starpilot is like copilot, but for GitHub stars.

I've been starring repos for years thinking "This will definitely be useful later".

However I never really went back to them.

Starpilot is a retrival augmented generation CLI tool for rediscovering your GitHub stars.

Starpilot helps this problem by allowing you to rediscover GitHub repos you had previously starred that are relevant to your current project.

Here's some more details about the motivation for and state of the project.

Installation

This project is in early development and is not yet available on PyPi

Fork repo
Clone repo
cd starpilot
poetry install

You will need to have a .env file with

a GitHub personal access token saved to a .env file in the root of the project. This should have the user> read:user scope permission.
a OpenAI API key saved to a .env file in the root of the project.

GITHUB_API_KEY="ghp_..."
OPENAI_API_KEY="sk-..."

…

View on GitHub

Why retrival augmented generation matters

Retrieval Augmented Generation (RAG) is a technique used by large language models to cope with some of the limitations inherent in what are also sometimes referred to as 'Foundational' models.

When a model like GPT3 is trained, it is fed large amounts of textual data written by humans. These get translated into 'weights' in a nueral net. To overly simplify, these weights tell the model what the next most likely text is that follows the text it has already been shown.

However, these models don't know much about what has happened recently, what other programming resources really exist rather than what just sounds like it should exist, or where to exactly get a specific repo or webpage.

Retrieval augmented generation solves this by allowing you to feed the large language model with known real, up to date and relevant information.

Vectorstores

A type of data base called a vectorstore is commonly used for this because they are deliberately optimised towards a similarity search use case. They achieve this in a few ways:

Vectorstores store what you pass them as a 'vector embedding'. A vector embedding takes data (like text or images) and converts them to a list like representation of numbers.
Vectorstores keep similar vector embeddings close together in memory. This means that they are as fast as possible at returning lots of documents that have similar semantic meaning, because they are all clustered together.
Vectorstores have APIs that are specifically designed for these use cases, with querying methods that lean towards semantic searches more than sql queries, and loading techniques that integrate tightly into other systems that generate these vector embeddings from large language models.

Designing a system

With this set of goals and new knowledge I got to work working out which puzzle pieces I needed and how to fit them together. This time I did go through my stars (and a few other things), though maybe this is for the last time!

I figured I could get started using 4 main open source repos. My first commit to my pyproject.toml used these projects:

Typer

tiangolo / typer

Typer, build great CLIs. Easy to code. Based on Python type hints.

Typer, build great CLIs. Easy to code. Based on Python type hints.

Documentation: https://typer.tiangolo.com

Source Code: https://github.com/tiangolo/typer

Typer is a library for building CLI applications that users will love using and developers will love creating. Based on Python 3.6+ type hints.

The key features are:

Intuitive to write: Great editor support. Completion everywhere. Less time debugging. Designed to be easy to use and learn. Less time reading docs.
Easy to use: It's easy to use for the final users. Automatic help, and automatic completion for all shells.
Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
Start simple: The simplest example adds only 2 lines of code to your app: 1 import, 1 function call.
Grow large: Grow in complexity as much as you want, create arbitrarily complex trees of commands and groups of subcommands, with options and…

View on GitHub

typer is a pretty trendy framework for building CLI tools in python right now. It embraces typing, uses function decorators to magically turn your functions into CLI commands, and has relatively clear documention.

I chose typer specifically because:

I wanted to see what the hype was about
I think typing helps write better code
I found the documentation really helpful to get started easily

langchain-ai / langchain

⚡ Building applications with LLMs through composability ⚡

🦜️🔗 LangChain

⚡ Building applications with LLMs through composability ⚡

Looking for the JS/TS library? Check out LangChain.js.

To help you ship LangChain apps to production faster, check out LangSmith LangSmith is a unified developer platform for building, testing, and monitoring LLM applications Fill out this form to get off the waitlist or speak with our sales team.

Quick Install

With pip:

pip install langchain

With conda:

conda install langchain -c conda-forge

🤔 What is LangChain?

LangChain is a framework for developing applications powered by language models. It enables applications that:

Are context-aware: connect a language model to sources of context (prompt instructions, few shot examples, content to ground its response in, etc.)
Reason: rely on a language model to reason (about how to answer based on provided context, what actions to take, etc.)

This framework consists of several parts.

LangChain Libraries: The Python and…

View on GitHub

langchain is the most mature and well embraced large language model orchestration framework. Langchain itself doesn't supply you with any specific llm or vector store or embedding approach. Instead it is deliberately 'vendor agnostic'. It provides a common set of APIs and abstractions across a staggering number of vector data bases, large language models and embedding engines.

I chose langchain because:

It is the most established tool in a brand new space
I wasn't really sure which suppliers of vectorstores and large languge models made the most sense for my use case
I found the documentation really helpful to get started

Chroma

chroma-core / chroma

the AI-native open-source embedding database

Chroma - the open-source embedding database.
The fastest way to build Python or JavaScript LLM apps with memory

| | Docs | Homepage

pip install chromadb # python client
# for javascript, npm install chromadb!
# for client-server mode, chroma run --path /chroma_db_path

The core API is only 4 functions (run our 💡 Google Colab or Replit template):

import chromadb
# setup Chroma in-memory, for easy prototyping. Can add persistence easily!
client = chromadb.Client()
# Create collection. get_collection, get_or_create_collection, delete_collection also available!
collection = client.create_collection("all-my-documents")

# Add docs to the collection. Can also update and delete. Row-based API coming soon!
collection.add(
    documents=["This is document1", "This is document2"], # we handle tokenization, embedding, and indexing automatically. You can skip that and add your own embeddings as well
    metadatas=[{"source":

…

View on GitHub

chroma is a vectorstore that has great support from Langchain. There are many others as well but Chroma won out at this stage because:

I can run chroma as an 'embedded' data store, e.g. it runs locally on the users machine
chroma was the most often used vectorstore in the Langchain docs for RAG tasks
It was trivially easy to set up to the point at which I was convinced reading the tutorials that they had to have made a mistake

GPT4All

nomic-ai / gpt4all

gpt4all: open-source LLM chatbots that you can run anywhere

GPT4All

Open-source large language models that run locally on your CPU and nearly any GPU

GPT4All Website and Models

GPT4All Documentation

Discord

🦜️🔗 Official Langchain Backend

GPT4All is made possible by our compute partner Paperspace

Run on an M1 macOS Device (not sped up!)

GPT4All: An ecosystem of open-source on-edge large language models.

Important

GPT4All v2.5.0 and newer only supports models in GGUF format (.gguf). Models used with a previous version of GPT4All (.bin extension) will no longer work.

GPT4All is an ecosystem to run powerful and customized large language models that work locally on consumer grade CPUs and any GPU. Note that your CPU needs to support AVX or AVX2 instructions.

Learn more in the documentation.

A GPT4All model is a 3GB - 8GB file that you can download and plug into the GPT4All open-source ecosystem software. Nomic AI supports and maintains this software ecosystem to…

View on GitHub

gpt4all provides a set of LLM models and embedding engines that are also well supported by Langchain. gpt4all was appealing because:

It runs locally on 'normal' machines
It seems well supported and maintained
It is open about what data it was trained on and what data it will use to train on

PyGithub

PyGithub / PyGithub

Typed interactions with the GitHub API v3

PyGitHub

PyGitHub is a Python library to access the GitHub REST API This library enables you to manage GitHub resources such as repositories, user profiles, and organizations in your Python applications.

Install

pip install PyGithub

Simple Demo

from github import Github

# Authentication is defined via github.Auth
from github import Auth

# using an access token
auth = Auth.Token("access_token")

# First create a Github instance:

# Public Web Github
g = Github(auth=auth)

# Github Enterprise with custom hostname
g = Github(base_url="https://{hostname}/api/v3", auth=auth)

# Then play with your Github objects:
for repo in g.get_user().get_repos():
    print(repo.name)

# To close connections after use
g.close()

Documentation

More information can be found on the PyGitHub documentation site.

Development

Contributing

Long-term discussion and bug…

View on GitHub

Soon after this I realised that pygithub would be an easy way to go to GitHub to get the information I needed and bring it back into starpilot to load into the vectorstore. I had initially thought I might be able to use the GitHub Document Loader built into langchain, though once I sat down to really work it out I realised that this doesn't give access to a users stars, so I needed an alternative.

The other way to build

There were alternatives in all these choices. I think these are all totally viable parts to build effectively the same system:

Click

pallets / click

Python composable command line interface toolkit

$ click_

Click is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary. It's the "Command Line Interface Creation Kit". It's highly configurable but comes with sensible defaults out of the box.

It aims to make the process of writing command line tools quick and fun while also preventing any frustration caused by the inability to implement an intended CLI API.

Click in three points:

Arbitrary nesting of commands
Automatic help page generation
Supports lazy loading of subcommands at runtime

Installing

Install and update using pip:

$ pip install -U click

A Simple Example

import click
@click.command()
@click.option("--count", default=1, help="Number of greetings.")
@click.option("--name", prompt="Your name", help="The person to greet.")
def hello

…

View on GitHub

I actually am using click, sort of. typer is built ontop of click, but to be honest I didn't really know that before I'd mostly decided. click looks like a really great project, but it wasn't as clear how to get started.

run-llama / llama_index

LlamaIndex (formerly GPT Index) is a data framework for your LLM applications

🗂️ LlamaIndex 🦙

LlamaIndex (GPT Index) is a data framework for your LLM application.

PyPI:

LlamaIndex: https://pypi.org/project/llama-index/.
GPT Index (duplicate): https://pypi.org/project/gpt-index/.

LlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.

Documentation: https://docs.llamaindex.ai/en/stable/.

Twitter: https://twitter.com/llama_index.

Discord: https://discord.gg/dGcwcsnxhU.

Ecosystem

LlamaHub (community library of data loaders): https://llamahub.ai
LlamaLab (cutting-edge AGI projects using LlamaIndex): https://github.com/run-llama/llama-lab

🚀 Overview

NOTE: This README is not updated as frequently as the documentation. Please check out the documentation above for the latest updates!

Context

LLMs are a phenomenal piece of technology for knowledge generation and reasoning. They are pre-trained on large amounts of publicly available data.
How do we best augment LLMs with our own private data?

We need a comprehensive toolkit to help perform this data augmentation for LLMs.

Proposed Solution

That's where LlamaIndex comes in. LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools:

Offers data connectors to ingest…

View on GitHub

llama_index is probably a great project, but I only found it late in my thinking on this project. If I start a different project it's suitable for any time soon I'm definately going to try it out as a comparison.

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.

Faiss

Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU. It is developed primarily at Meta's Fundamental AI Research group.

News

See CHANGELOG.md for detailed information about latest features.

Introduction

Faiss contains several methods for similarity search. It assumes that the instances are represented as vectors and are identified by an integer, and that the vectors can be compared with L2 (Euclidean) distances or dot products. Vectors that are similar to a query vector are those that have the lowest L2 distance or the highest dot product with the query vector. It also…

View on GitHub

I'd used faiss in a tutorial on vectorstores before. It didn't strike me as hugely intuitive to use or as simple to set up (it's recommended installation path is via conda). I also don't particularly like Facebook so I'm happy to use an alternative.

openai / openai-python

The official Python library for the OpenAI API

OpenAI Python API library

The OpenAI Python library provides convenient access to the OpenAI REST API from any Python 3.7+ application. The library includes type definitions for all request params and response fields and offers both synchronous and asynchronous clients powered by httpx.

It is generated from our OpenAPI specification with Stainless.

Documentation

The API documentation can be found here.

Installation

Important

The SDK was rewritten in v1, which was released November 6th 2023. See the v1 migration guide, which includes scripts to automatically update your code.

pip install openai

Usage

The full API of this library can be found in api.md.

import os
from openai import OpenAI
client = OpenAI(
    # This is the default and can be omitted
    api_key=os.environ.get("OPENAI_API_KEY"),
)

chat_completion = client.chat.completions.create(
    messages=[

…

View on GitHub

I'd used openai for a handful of tutorials and notebook experiments already and been very happy with it. However for a project like this I wasn't really sure what the operational costs would be, and if they would be worth it for the benefit the tool provides. That combined with the requirement to have network connectivity while using the tool pushed me towards experimenting with alternatives. Luckily with langchain I should be able to provide it as an optional backend in the future?

What state is `starpilot` now?

"actively developed", "v0.1.0", "untested" and "it runs on my machine" are good descriptions of the project right now.

I've spent a few evenings this month on it, and see myself at least spending a few more on it next month. The API is getting breaking changes almost everytime I open the project. It's got 0 real tests. It should get some soon though. It requires a few manual installation steps that are documented in README.md but haven't yet even been attempted on another machine other than the one I'm on right now.

It also doesn't yet achieve exactly what I want it to, but I see no reason yet that it can't with some more development time.

Current features

`starpilot read MyCoolUserName`

This will connect to Github and read the starred repos of the user MyCoolUserName. Then it will go to each of those repos and get the topics and descriptions (and optionally the readmes) and load these into chroma which is persisted on the local hard drive.

`starpilot shoot "insert topic here"`

This will spin up the chroma database and perform a semantic similarity search on the string given in the command, then return the documents that seem to be the most relevant.

`starpilot fortuneteller "Insert a question here"`

This will perform the exact same search as the shoot command, but then spin up a large language model and pass the results into the large language model for processing. It then returns the documents it found as well as the response from the LLM

So....

That's where this project is at. I've learnt a tonne about the available tools and relevant techniques in this space already, which was really the main goal of starting to begin with!

That said the progress I've made so far only makes me more curious about what else can be done with this and what else can be solved towards the vision of "Making your GitHub stars more valuable in your daily coding". Here's some ideas that I've found exciting while getting my hands dirty that might show up in the future. These are along with the obvious things like any testing at all, a simpler way to set up the project on your machine, better error handling, a more sensible way to update the vectorstore than drop everything and rebuild each time, etc.

Inspecting the current projects description (both it's loose goals as well as more specific things like what packages it already uses) so that things that are already used aren't suggested and are instead used to inform the response.
Dynamically creating a GitHub list of similar starred repos for your user (though that would probably rely on this suggestion to extend the GitHub API) so that you naturally have some ways of saving and sharing your starred repos that solve a specific problem between sessions in your terminal
Building starpilot into a research agent that can perform actions such as installing the selected suggestion into the current project or be sent to GitHub to find new projects that solve the current goals that you haven't starred yet

What do you think?

Does this sound like something intersting to you, maybe even something useful? Did this just spark inspiration in you for a new project? Does this actually already exist somewhere and I'm just being an idiot? Let me know :)

My GitHub profile shows my popular dev.to posts and GitHub repos automatically

Dave Parr — Wed, 05 Aug 2020 15:41:23 +0000

This is my github profile:

DaveParr / DaveParr

My GitHub Readme Profile

Popular Repos

Popular Blogs

article	public_reactions_count	comments_count	page_views_count
I made my dev.to content into a website to find a new job	36	3	518
Why use AWS Lambda for Data Science?	33	3	560
Posting straight from .Rmd to dev.to (for real this time)	18	0	173
Local gitlab runners, ‘no such image’, docker and disk space	9	0	600
Writing R packages, fast	8	0	14

View on GitHub

Background

I made it because I saw this repo by zhiiyang

zhiiiyang / zhiiiyang

It is a self-updating personal README showing my latest tweet and reply.

My name is Zhi (pronounced as Z). Welcome to my GitHub page! Here are highlights of my recent projects.

📱 A Phone Shiny app OTworkout to track your workout from GitHub.
🤖 A Twitter bot mutsignatures using Python and AWS Lambda.
📊 #TidyTuesday Data Visualization Gallery.
📦 Two Bioconductor packages: HiLDA and selectKSigs.
👇 You're viewing my lastest tweet deployed on GitHub actions.

Follow me on Twitter 💬 | Connect me on LinkedIn 👔 | Check out my website 🔗

View on GitHub

Thanks to @mokkapps for this article which uses the twitter updating
action for discovering it.

How I Built A Self-Updating README On My Github Profile

Michael Hoffmann ・ Jul 15 '20 ・ 2 min read

#github #profile #javascript #developer

Motivation

It took my a little while to decide what to do with my profile. I wanted
something data driven, and I wanted something to show-off my programming
successes. So I decided to show my most popular repos, and also to show
my most popular dev.to posts! I noticed that zhiiiyang made extensive
use of r-lib/actions, and I’ve
found that it was really valuable for my project too!

Method

Repos

The repos script was where I started. Building from zhiiiyang’s work, I
built a GitHub
workflow
to call a script I had written.

The script was quite
simple.
It gets my repos from the GitHub API through the
gh package, and tidies the return using
hoist to grab the important bits. It then filters and pivots the data
into a simple plot.

The most interesting part was where zhiiiyang added the output as a
commit by the action itself. The authentication for the action is
actually allowed by this section:

env:
    GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}

I did stumble a little over the correct path to the plot image. It turns
out that you can use the relative location (./graph.png) which will
work if you are viewing the README from within the repo, but to make
it work when the README is displayed from my user page you have to use
the absolute path
(https://github.com/daveparr/daveparr/blob/main/graph.png).

Posts

The posts was actually pretty easy once I’d got used to how GitHub
Action operate, and also how the GitHub README profiles worked.

The key part was actually a feature I recently developed for my own
package.

DaveParr / dev.to.ol

dev.to.ol helps R users publish to dev.to

.to.ol

The goal of dev.to.ol is to help R users publish to dev.to

Installation

You can install the dev.to.ol from github with remotes:

remotes::install.github("DaveParr/dev.to.ol")

Workflow

Create your article

The create_new_article function will give you the front mattter boilerplate for an article .Rmd file. Optionally supplying a file name will create a new file with the front matter at the start.

create_new_article(title = "my title")

Write your article!

This is the fun bit. Mark your great ideas down in an .Rmd!

Post your article

Once the .Rmd is written, you can post it to dev.to with post_new_article

post_new_article("./my-great-article.Rmd")

Check your articles

There are two functions to check the posted articles on dev.to, published and unpublished. Both will return a ‘tidy’ data set by default.

get_users_articles()
Using DEVTO in .Renviron
The API returned the expected success

…

View on GitHub

dev.to.ol is a package to help R users manage their dev.to content. In
particular I had recently finished the functions that return data from
the API about your published articles. The key to this is to have the
DEV.TO api key that you want to use set as an encrypted secret in the
repo. once that is set, my package can read it if it’s set as an
environmental variable along with the GITHUB_PAT.

env:
    DEVTO: ${{ secrets.DEVTO }}
    GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}

Because my package returns a tidy data.frame like object, it was trivial
to munge it down to just what I wanted to show, and then format it
neatly with knitr. I also went all in on the r-lib/actions examples,
and now not only generate the new data for both the chart and blogs
during the GitHub action, but also do the full compile from .Rmd to
.md using the setup-pandoc
action.

Lessons in automation

I’d been meaning to explore GitHub Actions for a little while, and I
found a few things out that I’m going to be considering in the future as
I develop this an other projects.

Pricing vs performance on macOS and Linux

It’s free to a point, and then you need to pay. I had a look at the
pricing
plan
and noticed that your runtime impacts your pricing. Most of the examples
from r-lib/actions run from macos-latest, as does zhiiiyang’s project.
In GitHub Actions pricing a minute of macOS runtime is worth 10
minutes of linux runtime. I ran on macOS for a while too, but
eventually thought that it might be a smart idea for a long running
personal project to convert to a linux run time, though now I’ve done it
I’m debating going back to mac.

The ‘problem’ is that I did not write this process to be fast, or light.
It’s a silly hobby project to over-automate because I can. Therefore, on
Linux I am now actually compiling the packages on each installation
AND installing libcurl for the api
calls.
The cost savings I make from not picking the faster run time in this
case are approximately negated by the increase in actual run time. By
eye, that part of the job runs at about 10 minutes now, where as on mac
it was about 10 times faster as the pre-compiled binaries could just be
downloaded and would run ‘out of the box’. I’ll probably change it back
in a little while after I’ve checked how variable it can be.

Potential solutions could include some form of caching (which I’ve heard
is maybe supported?) or running the action in my own docker image with
the pre-compiled, though TBH that sounds like work, and this is supposed
to be fun :P

r-lib actions for the Rmd

I really like the idea of compiling the README.md for a package from
the README.Rmd we often use in R. I’ve often forgotten in my own work
to do that key step before a push, and having a relatively simple
automation backed into where my repos live will likely be something I
use in the future. The best part of this trick is that r-libs/actions
does the most irritating part of ‘making Pandoc work’ for me. So I can
just profit!

Successs!

I really liked hacking this out. I got to put my dev.to.ol package to
another practical lesson and learn about GitHub Actions. Feel free to
re-create this on your profiles, either by grabbing bits or by just
lifting the whole thing. One of the reasons I built it the way I did is
so it could be relatively portable between users, and maybe solve a
problem for more than just me. So if you get this deployed on your
profile, or get stuck, I’d love to hear from you!

How to calculate a Pokemons 'power level' using kmeans

Dave Parr — Fri, 17 Jul 2020 15:29:19 +0000

library(pokedex)
library(tidyverse)
library(tidymodels)
library(showtext)
font_add_google("Press Start 2P")
showtext_auto()

theme_set(theme_pokedex())

knitr::opts_chunk$set(fig.showtext=TRUE)

The Heros Journey

Pokemon games have a very familiar cycle. You start with one of 3
Pokemon. You adventure out with your new buddy, facing tougher and
tougher Pokemon, in greater variety. Many of your Pokemon evolve over
time, and eventually you find the end-game legendaries in a climactic
battle of titans!

You can even see this story in the data. Here is the base_experience
of all the Pokemon, identified by game generation. The base_experience
is the basic amount of experience that is gained by the winner of a
battle from a specific species. e.g. If you beat a Bulbasaur, your
Pokemon gains experience based on a formula which uses Bulbasaurs
base_experience. Because of that we can see it as a proxy value for
how powerful a Pokemon is. If it’s more powerful, it will be harder to
beat, and so reward you with more experience when you win.

pokemon %>%
  ggplot(aes(x = id, y = base_experience, colour = generation_id)) +
  geom_point() +
  labs(title = "Base Experience for each Pokemon")

In each generation you can see a few attributes:

power level - 3(ish) tiers, grouped in different vertical lanes
progression - a (generally) increasing trend in the base power value
the up-ticks at the start and the end of each gen are the starters top-tier evolution (Woo Blastoise!) and the Legendaries at the end-game

Lets see if we can group each Pokemon into a power_level. A
categorical grouping which relates it to other Pokemon with similar
base_experience

Grouping and counting

Maybe we can explicitly describe the power levels of each tier of
Pokemon with a simple process? Can we group each Pokemon by evolutionary
chain, and then count each Pokemons order within the group?

pokemon %>%
  group_by(evolution_chain_id) %>%
  mutate(power_level = row_number()) -> pokemon_group_count

ggplot(pokemon_group_count,
       aes(
    x = id,
    y = base_experience,
    colour = as_factor(power_level)
  )) +
  geom_point() +
  labs(title = "Power level by group count",
       colour = "power_level")

Sort of? Generally we capture the pattern, but we don’t actually get it
very right. First off, there are more than 3 evolutionary tiers in our
engineered feature. We can also see there are some Pokemon are
classified as a power_level higher than 1, but still in the lowest
group on this list. Why might have caused this?

pokemon_group_count %>%
  filter(power_level > 3) %>%
  pull(evolution_chain_id) -> pokemon_group_count_mistakes

pokemon_group_count %>%
  filter(evolution_chain_id %in% pokemon_group_count_mistakes) %>%
  select(id,
         name,
         base_experience,
         generation_id,
         evolution_chain_id,
         power_level)  %>%
  arrange(evolution_chain_id) %>%
  select(id, name, base_experience, evolution_chain_id, power_level) %>% 
  knitr::kable()

id	name	base_experience	evolution_chain_id	power_level
43	Oddish	64	18	1
44	Gloom	138	18	2
45	Vileplume	221	18	3
182	Bellossom	221	18	4
60	Poliwag	60	26	1
61	Poliwhirl	135	26	2
62	Poliwrath	230	26	3
186	Politoed	225	26	4
106	Hitmonlee	159	47	1
107	Hitmonchan	159	47	2
236	Tyrogue	42	47	3
237	Hitmontop	159	47	4
133	Eevee	65	67	1
134	Vaporeon	184	67	2
135	Jolteon	184	67	3
136	Flareon	184	67	4
196	Espeon	184	67	5
197	Umbreon	184	67	6
470	Leafeon	184	67	7
471	Glaceon	184	67	8
700	Sylveon	184	67	9
265	Wurmple	56	135	1
266	Silcoon	72	135	2
267	Beautifly	178	135	3
268	Cascoon	72	135	4
269	Dustox	173	135	5
280	Ralts	40	140	1
281	Kirlia	97	140	2
282	Gardevoir	233	140	3
475	Gallade	233	140	4
789	Cosmog	40	413	1
790	Cosmoem	140	413	2
791	Solgaleo	306	413	3
792	Lunala	306	413	4

So there are some clear problems with this approach. In Gen 1 we had
some branching evolution with the Eevee family. Not only was this family
expanded in multiple generations to eventually 8 variations, but we also
saw more branching evolution trees as well. We also got ‘baby’ Pokemon.
Pokemon that are actually pre-cursors to other Pokemon, but are listed
later in the Pokedex.

Luckily there is another variable we can use, that should be a whole lot
better.

`evolves_from_species`

Each Pokemon that evolves from another Pokemon has the Pokemon they
evolve froms id as the value in the evolves_from_species_id
variable. Maybe we can use that to break up the Pokemon into their
‘power levels’.

The following code is not my best work, however I spent some time on a
more recursive strategy, but it was honestly miles more confusing. For
the purposes of a silly example for a blog, I think this is
prefferable.

pokemon %>%
  mutate(power_level = case_when(is.na(evolves_from_species_id) ~ 1)) %>%
  filter(power_level == 1) -> pokemon_1

pokemon %>%
  mutate(power_level = case_when(evolves_from_species_id %in% pokemon_1$id ~ 2)) %>%
  filter(power_level == 2) -> pokemon_2

pokemon %>%
  mutate(power_level = case_when(evolves_from_species_id %in% pokemon_2$id ~ 3)) %>%
  filter(power_level == 3) -> pokemon_3

bind_rows(pokemon_1, pokemon_2, pokemon_3) %>%
  arrange(id) %>%
  mutate(power_level = as_factor(power_level)) -> pokemon_evolves_from

pokemon_evolves_from %>%
  ggplot(aes(x = id, y = base_experience, colour = power_level)) + 
  geom_point() +
  labs(title = "Power level by evolves_from")

Hmm, that’s actually worse? Lets focus on the Pokemon labelled
power_level 1, but are up where we would expect level 3 Pokemon to be.

pokemon_evolves_from %>% 
  filter(base_experience > 200 & power_level == 1) %>% 
  select(id, name, base_experience) %>% 
  knitr::kable()

id	name	base_experience
144	Articuno	261
145	Zapdos	261
146	Moltres	261
150	Mewtwo	306
151	Mew	270
243	Raikou	261
244	Entei	261
245	Suicune	261
249	Lugia	306
250	Ho-Oh	306
251	Celebi	270
377	Regirock	261
378	Regice	261
379	Registeel	261
380	Latias	270
381	Latios	270
382	Kyogre	302
383	Groudon	302
384	Rayquaza	306
385	Jirachi	270
386	Deoxys	270
480	Uxie	261
481	Mesprit	261
482	Azelf	261
483	Dialga	306
484	Palkia	306
485	Heatran	270
486	Regigigas	302
487	Giratina	306
488	Cresselia	270
489	Phione	216
490	Manaphy	270
491	Darkrai	270
492	Shaymin	270
493	Arceus	324
494	Victini	270
531	Audino	390
638	Cobalion	261
639	Terrakion	261
640	Virizion	261
641	Tornadus	261
642	Thundurus	261
643	Reshiram	306
644	Zekrom	306
645	Landorus	270
646	Kyurem	297
647	Keldeo	261
648	Meloetta	270
649	Genesect	270
716	Xerneas	306
717	Yveltal	306
718	Zygarde	270
719	Diancie	270
720	Hoopa	270
721	Volcanion	270
785	Tapu Koko	257
786	Tapu Lele	257
787	Tapu Bulu	257
788	Tapu Fini	257
793	Nihilego	257
794	Buzzwole	257
795	Pheromosa	257
796	Xurkitree	257
797	Celesteela	257
798	Kartana	257
799	Guzzlord	257
800	Necrozma	270
801	Magearna	270
802	Marshadow	270
805	Stakataka	257
806	Blacephalon	257
807	Zeraora	270

So these Pokemon are nearly all ‘Legendary’. They are big end-game
Pokemon, with real rarity in game. They also don’t evolve from, or to,
anything, so our rule doesn’t classify them effectively.

pokemon_evolves_from %>% 
  filter(base_experience < 200 & base_experience > 100 & power_level == 1) %>% 
  select(id, name, base_experience) %>% 
  knitr::kable()

id	name	base_experience
83	Farfetch’d	132
115	Kangaskhan	172
127	Pinsir	175
128	Tauros	172
131	Lapras	187
132	Ditto	101
142	Aerodactyl	180
201	Unown	118
203	Girafarig	159
206	Dunsparce	145
213	Shuckle	177
214	Heracross	175
222	Corsola	144
225	Delibird	116
227	Skarmory	163
234	Stantler	163
241	Miltank	172
302	Sableye	133
303	Mawile	133
311	Plusle	142
312	Minun	142
313	Volbeat	151
314	Illumise	151
324	Torkoal	165
327	Spinda	126
335	Zangoose	160
336	Seviper	160
337	Lunatone	161
338	Solrock	161
351	Castform	147
352	Kecleon	154
357	Tropius	161
359	Absol	163
369	Relicanth	170
370	Luvdisc	116
417	Pachirisu	142
440	Happiny	110
441	Chatot	144
442	Spiritomb	170
455	Carnivine	159
479	Rotom	154
538	Throh	163
539	Sawk	163
550	Basculin	161
556	Maractus	161
561	Sigilyph	172
587	Emolga	150
594	Alomomola	165
615	Cryogonal	180
618	Stunfisk	165
621	Druddigon	170
626	Bouffalant	172
631	Heatmor	169
632	Durant	169
676	Furfrou	165
701	Hawlucha	175
702	Dedenne	151
707	Klefki	165
741	Oricorio	167
764	Comfey	170
765	Oranguru	172
766	Passimian	172
771	Pyukumuku	144
772	Type: Null	107
774	Minior	154
775	Komala	168
776	Turtonator	170
777	Togedemaru	152
778	Mimikyu	167
779	Bruxish	166
780	Drampa	170
781	Dhelmise	181
803	Poipole	189

These Pokemon are not end game, but they either have very short
evolution trees (2 Pokemon long), or no evolution tree at all.

So our ‘group count’ process doesn’t work well, and neither does our
‘evolves_from_species’ process. We’re going to have to to learn some
new moves.

TM01 (e.g. Tidy Models 01)

Clustering is the process of using machine learning to derive a
categorical variable from data. The simplest form of clustering that
seems relevant to our problem is k-means. Seeing as we have a pretty
good intuition that 3 groups implicitly exist in this data, and a
clear visualisation supporting us, lets cut straight to asking R for 3
groups. k-means clustering aims to divide the data into a known number
of groups which is set in the centers argument, and doesn’t require
any data to be fed to it as examples of what makes up a ‘group’. That
sentence might be a little confusing, as we do obviously give it some
data. What we don’t give it is a training data set which has examples
of what Pokemon are supposed to be in a given group, labelled with the
group they are supposed to be in, e.g. Squirtle is in group 1, Wartortle
is in group 2, Blastoise is in group 3, and then give it unlabelled data
to classify, e.g. “What group is Charmeleon in?”. k-means will figure
out how to split the continuous variable base_experience into 3
groups.

set.seed(68)

pokemon %>% 
  select(base_experience)  %>% 
  kmeans(centers = 3) %>% 
  augment(pokemon) -> pokemon_cluster

How kmeans works

First, we set centroids to be at random positions in the data. To make
sure this doesn’t effect consistency in my article I’ve used set.seed
so k-means starts looking for the centres of our clusters from the same
position each time. A ‘centroid’ can be seen as a ‘centre point’ for
each cluster. We have the same number of centroids set as the value set
in the centers argument. Each data point is then assigned to it’s
closest centroid.

The Sum of Squared Errors (SSE) from the centroids is then used as an
objective function towards a local minimum.

This is the core concept of how k-means calculates a solution. The Sum
of Squared Errors are calculated like this:

\sum^n_{i-1}(x_i-\bar{x})^2

What this means is for each group, the distance from the centroid to
each observation is measured, and then squared. Then each of those
squared distances, one per observation in the group, is totalled.

The centroid is then moved to the average value of it’s group. Because
the centroids are now no longer in the same position as when the SSE was
calculated, the SSE is now recalculated for all observations, to each of
the new centroid positions. This means that some observations are now
closer to a different centroid, and so get assigned to a different
cluster.

Then the centroids are moved again to the new average of the new
cluster. Each cluster then gets new distances calculated. This will go
on until termination when, each centroid is in the average position of
the cluster, and each observation in the cluster is closest to the
centroid it is currently assigned to.

That’s a slightly wordy description of a complicated process. I
recommend that you have a look at the k-means explanation in tidy
models to really
cement the concept. It also contains the most adorable animation of a
statistical concept in existance.

How to use it

Here, I’ve simply selected the one column of data, base_experience,
and piped it into kmeans, which is part of base R. This returns a
super un-tidy list like object of class "kmeans", however, with
tidymodels we can easily make it usable.

augment helps us to match the output of the kmeans function back to
our data for easier processing. The output of kmeans doesn’t actually
contain any of the other information about our data, it only got given
one column remember? augment goes through the return of kmeans,
finds the relevant bit, and matches it neatly back to our original data
ready for plotting in one step!

pokemon_cluster %>% 
  ggplot(aes(x = id, y = base_experience, colour = .cluster)) +
  geom_point() +
  labs(title = "Power level by kmeans clustering")

This is a lot better. It gives us really clear groups, in exactly
where we expected them. It’s also tonnes simpler code!

Lets check some of our boundary positions, just to make sure it makes
sense.

pokemon_cluster %>% 
  filter(.cluster == 1 & base_experience > 100) %>% 
  select(id, name, base_experience) %>% 
  knitr::kable()

id	name	base_experience
132	Ditto	101
440	Happiny	110
699	Aurorus	104
762	Steenee	102
772	Type: Null	107

pokemon_cluster %>% 
  filter(.cluster == 2 & base_experience > 200) %>% 
  select(id, name, base_experience) %>% 
  knitr::kable()

id	name	base_experience
189	Jumpluff	207

Ditto and Type: Null are just a plain weird Pokemon due to their
abilities messing with their type. Happiny is a baby type, but from a
family with an insane base_experience. Jumpluff is an awkward edge
case. Technically the 3rd evolution, it’s still got an extremely low
base_experience. Potentially this is for game balancing as they are
relatively regularly encountered? Aurorus and Steenee I do not have a
good hypothesis for.

Generally I think that this is a pretty good solution. There are maybe a
few edge cases that are open to interpretation, but that’s just what we
get sometimes with machine learning. Lacking a labelled training data
set, we can’t compute a confusion matrix or ROC-curve.

Conclusion

We’ve found a situation in the ‘real’ world where we know from context
there is a categorical relationship, but from the data available it’s
not possible to classify that precisely. However we can create this
categorisation using machine learning! Even better, we can use the
tidymodels package to help us do it quickly and cleanly.

The Missingno Experiment and Multiple Form Pokemon

Dave Parr — Wed, 01 Jul 2020 17:42:42 +0000

Wild missingno appeared!

Missingno is the patron Pokemon of data science. You’re just casually
surfing up and down your data, doing some sweet coding, when suddenly a
bunch of missing and corrupted data gets in you way, and you suddenly
have a bunch of random items in your bag for no reason. OK, well maybe I
just have a messy bag.

The valuable part of this metaphor is the part where you battle
Missingno, and win. I’ve been doing this with my Pokedex project
recently, to try and iron out what data I can rely on from my data
source, and what’s a bit patchy.

library(pokedex)
library(tidyverse)
library(naniar)
library(skimr)

Go, Skimr!

Skimr gives us a text based summary view. As well as the basics on data
set size, it also shows us some statistical values, but most valuably it
describes how many values are missing, and in what columns.

pokemon %>% 
  skimr::skim()


Name	Piped data
Number of rows	807
Number of columns	24
_______________________
Column type frequency:
character	8
list	1
numeric	15
________________________
Group variables	None

Data summary

Variable type: character

skim_variable	n_missing	complete_rate	min	max	n_unique
identifier	0	1.00	3	20	807
type_1	0	1.00	3	8	18
type_2	402	0.50	3	8	18
name	0	1.00	3	12	807
genus	0	1.00	11	21	589
color	20	0.98	3	6	10
shape	20	0.98	4	9	14
habitat	422	0.48	3	13	9

Variable type: list

skim_variable	n_missing	complete_rate	n_unique	min_length	max_length
flavour_text	0	1	807	1	1

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
id	0	1.00	404.00	233.11	1.0	202.5	404	605.5	807.0	▇▇▇▇▇
species_id	0	1.00	404.00	233.11	1.0	202.5	404	605.5	807.0	▇▇▇▇▇
height	0	1.00	1.16	1.08	0.1	0.6	1	1.5	14.5	▇▁▁▁▁
weight	0	1.00	61.77	111.52	0.1	9.0	27	63.0	999.9	▇▁▁▁▁
base_experience	0	1.00	144.85	74.95	36.0	66.0	151	179.5	608.0	▇▇▁▁▁
is_default	0	1.00	1.00	0.00	1.0	1.0	1	1.0	1.0	▁▁▇▁▁
hp	0	1.00	68.75	26.03	1.0	50.0	65	80.0	255.0	▃▇▁▁▁
attack	0	1.00	76.09	29.54	5.0	55.0	75	95.0	181.0	▂▇▆▂▁
defense	0	1.00	71.73	29.73	5.0	50.0	67	89.0	230.0	▅▇▂▁▁
special_attack	0	1.00	69.49	29.44	10.0	45.0	65	90.0	173.0	▃▇▅▂▁
special_defense	0	1.00	70.01	27.29	20.0	50.0	65	85.0	230.0	▇▇▂▁▁
speed	0	1.00	65.83	27.74	5.0	45.0	65	85.0	160.0	▃▇▆▂▁
generation_id	20	0.98	3.67	1.94	1.0	2.0	4	5.0	7.0	▇▅▃▅▅
evolves_from_species_id	426	0.47	364.35	232.43	1.0	156.0	345	570.0	803.0	▇▆▅▆▅
evolution_chain_id	20	0.98	195.96	124.57	1.0	84.0	187	303.0	427.0	▇▆▅▆▅

I was expecting some missing data in type_2, and
evolves_from_species_id, but I wasn’t expecting only half of habitat
to be there. Either I broke something in my data pipeline, or the data
wasn’t there to begin with. colour, shape, generation_id and
evolution_chain_id are all missing 20 entries each, which is a bit or
a coincidence. I wonder if they are all missing from the same Pokemon?

Visdat I choose you!

visdat is a package that helps you visualise missing data and data
types.

visdat::vis_dat(pokemon)

This clearly shows us the data types in each column, and where values
are missing in context. It looks like habitat might just not be
available after a certain time. It also looks like colour, shape,
generation_id and evolution_chain_id looks like they are maybe all
missing from the same individual Pokemon?

Go, Naniar!

Naniar helps us check through plots where relationships between
missing values and other variables might occur. Lets check first if
there is a relationship between generation_id and evolution_chain_id

pokemon %>%
  ggplot(aes(generation_id, evolution_chain_id)) +
  geom_miss_point()

This plot might need a little explanation. For the Not Missing blue
values, this is a normal geom_point(). However, where the values are
marked as Missing pink they are deliberately moved below the (0,0)
mark for the axis they are missing values for, then they ‘jitter’, to
avoid over-plotting. The little cluster at the far bottom left in a line
marks that for all values where evolution_chain_id being missing,
generation_id is also missing. Let’s have a look at the
evolves_from_species_id variable just to help us understand.

pokemon %>%
  ggplot(aes(evolves_from_species_id, generation_id)) +
  geom_miss_point()

This is showing that in every game generation (Red/Blue, X/Y, etc.) that
there are Pokemon that have an evolves_from_species_id, i.e. they have
a precursor Pokemon, and that there are also Pokemon that don’t have a
precursor. Just what we see in the games. It’s also showing that have
neither generation_id or evolves_from_species_id.

Who is that Pokemon?

Now we know the characteristics of the missing data we are interested
in, we can pull them out easily. Especially with the newly released
across() function

missing_cols <- c("color", "shape", "generation_id", "evolves_from_species_id")
pokemon %>% 
  filter(across(missing_cols, ~is.na(.x))) %>% 
  select(name, identifier, missing_cols) -> missing_pokes

missing_pokes %>% knitr::kable()

name	identifier	color	shape	generation_id	evolves_from_species_id
Deoxys	deoxys-normal	NA	NA	NA	NA
Wormadam	wormadam-plant	NA	NA	NA	NA
Giratina	giratina-altered	NA	NA	NA	NA
Shaymin	shaymin-land	NA	NA	NA	NA
Basculin	basculin-red-striped	NA	NA	NA	NA
Darmanitan	darmanitan-standard	NA	NA	NA	NA
Tornadus	tornadus-incarnate	NA	NA	NA	NA
Thundurus	thundurus-incarnate	NA	NA	NA	NA
Landorus	landorus-incarnate	NA	NA	NA	NA
Keldeo	keldeo-ordinary	NA	NA	NA	NA
Meloetta	meloetta-aria	NA	NA	NA	NA
Meowstic	meowstic-male	NA	NA	NA	NA
Aegislash	aegislash-shield	NA	NA	NA	NA
Pumpkaboo	pumpkaboo-average	NA	NA	NA	NA
Gourgeist	gourgeist-average	NA	NA	NA	NA
Oricorio	oricorio-baile	NA	NA	NA	NA
Lycanroc	lycanroc-midday	NA	NA	NA	NA
Wishiwashi	wishiwashi-solo	NA	NA	NA	NA
Minior	minior-red-meteor	NA	NA	NA	NA
Mimikyu	mimikyu-disguised	NA	NA	NA	NA

So it looks like in the current version of the package, these Pokemon
all have ‘complex’ identifiers. This is because these Pokemon all have
different forms. Some vary by colour like
Basculin
which can be Red or Blue striped, others have ability transformations,
like
Aegislash
or which game it was caught in like
Deoxys.

missing_pokes %>% 
  pull(name) %>% 
  stringr::str_to_lower(.) -> missing_pokes_name_list

pokedex$pokemon_species %>%
  filter(
    stringr::str_to_lower(identifier) %in% missing_pokes_name_list
    ) %>% 
  select(identifier, generation_id, evolves_from_species_id, shape_id, color_id) %>% 
  knitr::kable()

identifier	generation_id	evolves_from_species_id	shape_id	color_id
deoxys	3	NA	12	8
wormadam	4	412	5	5
giratina	4	NA	10	1
shaymin	4	NA	8	5
basculin	5	NA	3	5
darmanitan	5	554	8	8
tornadus	5	NA	4	5
thundurus	5	NA	4	2
landorus	5	NA	4	3
keldeo	5	NA	8	10
meloetta	5	NA	12	9
meowstic	6	677	6	2
aegislash	6	680	5	3
pumpkaboo	6	NA	1	3
gourgeist	6	710	5	3
oricorio	7	NA	9	8
lycanroc	7	744	8	3
wishiwashi	7	NA	3	2
minior	7	NA	1	3
mimikyu	7	NA	2	10

If we go back to the raw source data, we can see that the data is
actually there for most cases, it just didn’t join properly because in
the source data, they are identified by the simple name, in lower case,
and in this version of the
package
this data is joined on id AND the column that actually has the complex
name. Also, because shape and color link through this data, they are
missed as well!

You defeated wild missingno!

This is all based on my Pokedex R data package, which I’m just about to
fix :)

DaveParr / pokedex

an R data package for pokemon

Why did I make this dev.to API wrapper?

Dave Parr — Tue, 30 Jun 2020 14:26:12 +0000

dev.to.ol is 0.0.1!

DaveParr / dev.to.ol

dev.to.ol helps R users publish to dev.to

.to.ol

The goal of dev.to.ol is to help R users publish to dev.to

Installation

You can install the dev.to.ol from github with remotes:

remotes::install.github("DaveParr/dev.to.ol")

Workflow

Create your article

The create_new_article function will give you the front mattter boilerplate for an article .Rmd file. Optionally supplying a file name will create a new file with the front matter at the start.

create_new_article(title = "my title")

Write your article!

This is the fun bit. Mark your great ideas down in an .Rmd!

Post your article

Once the .Rmd is written, you can post it to dev.to with post_new_article

post_new_article("./my-great-article.Rmd")

Check your articles

There are two functions to check the posted articles on dev.to, published and unpublished. Both will return a ‘tidy’ data set by default.

get_users_articles()
Using DEVTO in .Renviron
The API returned the expected success

…

View on GitHub

What’s in the box?

dev.to.ol has the minimum set of viable functions I think it needs to
operate, and (just) enough testing to keep it stable. The api is
starting to come together and there seems to be at least 1 other person
who cares enough about this project to talk about it to someone else, so
I thought it might be handy to get a little more together about things.
I also wanted to make up a celebration to share a little about why I
made it and what dev.to means to me.

Are there other similar boxes?

R is in many ways a literate programming language. Markdown is pretty
baked into most of our ecosystem through R
Markdown, as is
LaTeX. I think because of this,
and the frequent usage in academic publishing, R users have developed
tooling for blogging pretty heavily. We have
blogdown and
distill which are becoming very
widely used and feature rich, and @maelle has written a pretty
comprehensive guide to all the different approaches in the Scientific
Blogging with R Markdown
course including her own
solution for Wordpress,
goodpress. As Data Scientists
(emphasis on the science), we need to share our work reproducibly, and
in a way that encourages understanding and review.

So why have another box?

I was looking for something to keep me sane while I was furloughed.

Also, I had poked around dev.to about a year before, and was impressed
by what I took to be it’s un-official mission statement of “We aren’t
Medium, we’re what Medium should have been”.

Medium Was Never Meant to Be a Part of the Developer Ecosystem

Ben Halpern ・ Jun 3 '19 ・ 5 min read

#meta

I’m sure there is a lot more to it than that, especially with the recent
Forem
announcement, but personally, that was a hook I perceived which really
stuck me.

However, at the time the in built editor was “fine”. Fine enough, but
not great. I’m also used to R Notebooks, where I write my code next to
my prose, and compile the whole thing in one. It seemed valuable to
bring that ability to my dev.to posts. I also was aware of all the
alternatives above, but dev.to offered 2 things that were different to
the above options:

Hosted and managed service

Don’t get me wrong, I have nothing but love for JAMstack and static
sites in general. I maintain a template for making them for SatRdays
conferences.
However, I don’t love the process of doing it. In this case I absolutely
believe in the cause, which is why I continue to volunteer on the
project. However, the process of interacting with static sites doesn’t
actually make me happy.

I was looking for the simplest way I could casually blog, which ideally
required no extra work beyond writing articles (to start with).

Community

I wanted at least a few people to read my articles, and I wanted to read
theirs. I’m sure I could have accomplished this with more work in SEO,
cultivating a social media network and all that jazz, but again: I just
wanted to write some stuff that some people might read, and read their
stuff. dev.to had a discovery feed, it had active users, it had
community. This was just what I was looking for. I also kind of hate
twitter (yes, I still use it when I ‘have’ to).

Motivation

Initially my work flow sucked. I wrote an .Rmd in RStudio. Compiled it
to a GitHub flavoured .md. Copy and pasted the output into the editor,
added the meta data, then uploaded all the images. If I found I’d made a
mistake, I’d either do the right thing, which was edit the .Rmd,
recompile, re-copy-pasta, or I’d more often do the quick thing which is
enter the browser editor and fix the typo.

Inception

As I was chewing over the best way to make this work gooder (I am a
programmer after all), a few things happened simultaneously.

Most of the R users on the site syndicate

There are some really great R users on here already:

.ltag__user__id__26527 .follow-action-button { background-color: #5D535E !important; color: #9A9EAB !important; border-color: #5D535E !important; }

Julia Silge

I’m an international keynote speaker and real-world practitioner focused on data analysis and machine learning. I love making beautiful charts, the statistical programming language R, and Jane Austen.

.ltag__user__id__93212 .follow-action-button { background-color: #0D4D4B !important; color: #FFFFFF !important; border-color: #0D4D4B !important; }

Scott Chamberlain

co-founder of rOpenSci

.ltag__user__id__133005 .follow-action-button { background-color: #F69289 !important; color: #FFFFFF !important; border-color: #F69289 !important; }

Maëlle Salmon

.ltag__user__id__21644 .follow-action-button { background-color: #000000 !important; color: #ffffff !important; border-color: #000000 !important; }

Colin Fay

Nerd. R, JS, & Docker. Loves building things, loves breaking things.

I found out that most of the R users here I follow are actually
re-syndicating through rss:

Julia Silge

@juliasilge

@ma_salmon @DaveParr @_ColinFay @ThePracticalDev Under Settings ➡️ Publishing from RSS, you can add the RSS feed from your blog. The posts will go to your Dashboard as drafts. I do need to edit them slightly before publishing (for example, switching out for their shortcodes for a few things).

14:39 PM - 13 May 2020

That’s great for them, but I was actually trying to avoid my own site!

Dev.to has an API

I was poking around older posts and found out that dev.to has an
API. At the time I thought I might play with
webhooks to boot ‘saved’ articles into pocket to work on my e-reader (at
some point I still might). However, in this case it was kind of perfect.
An API driven work flow between .Rmd in RStudio and the hosted dev.to
community. I could see it in my head, and it’s not like I was busy in
May…

Posting from .Rmd to dev.to

Dave Parr ・ May 20 '20 ・ 4 min read

#rstats #projectbenatar #showdev #markdown

Evolution

After proving it ‘might’ be doable, I did some work fleshing out the
‘best’ way to do it.

Posting straight from .Rmd to dev.to (for real this time)

Dave Parr ・ May 24 '20 ・ 4 min read

#rstats #projectbenatar #showdev #meta

So I started some super-basic minimum viable functions. How do I get the
post to turn up on dev.to? How do I make the meta-data work with the
post? What happens if I’ve published and need to correct a typo? All
solvable, all import features, and all now implemented and tested. I got
to learn a lot about testing API wrappers with vcr thanks to @sckott .

Testing my dev.to API package with testthat, webmockr and vcr

Dave Parr ・ Jun 3 '20 ・ 6 min read

#rstats #projectbenatar #showdev #testing

The functions in the package changed a bit as I went through user
testing by using it to write my posts on dev.to that you can see here. I
was also able to generate content pretty quick because I could write
about developing the functions that I was testing at the same time by
writing the articles. Virtuous cycles! (/ black holes)

At this point I was also starting to look for new work, and was
wondering about if I had made the choice backwards. Maybe I did need my
own site after all. Somewhere to put my CV, and that looked a bit more
professional, and maybe didn’t have so much clutter of articles mixed
with my own questions and a bunch of content from other people. Then I
discovered the @stackbit integration!

You can now generate self-hostable static blogs right from your DEV content via Stackbit

Ben Halpern ・ Sep 26 '19 ・ 4 min read

#meta #projectbenatar #webdev #changelog

This meant that I could hand off all the hosting and styling and
management and deploys to them, but still get a hugo repo which I can
tailor how I want, such as removing the #meta and #help posts, which
wouldn’t be useful to a recruiter, but also automatically put my content
into a presentable site, with a few contextual links to thing like my
GitHub and LinkedIn under my personal daveparr.info
domain.

I made my dev.to content into a website to find a new job

Dave Parr ・ May 25 '20 ・ 2 min read

#projectbenatar #showdev #hugo #stackbit

Future

The primary future goal is finding a good way to reference images in the
article in a way that dev.to can use. It’s likely that this will end up
being github itself. Additionally better testing is probably on the
cards as I go. The functions and package API have stabilised enough now
that this can be comprehensive, and getting some CI/CD would be nice
too. I’m also planning on working on smarter ways to run analytics on
the data you can get back from the API about your posts. Maybe even an
inbuilt shiny app?

I hope some others of you might find the package useful, and maybe this
might motivate you to share the work you might already be doing in R, or
even pick up R as a new language!

Photo by Erwan Hesry via
Unsplash

Nativefier is bonkers

Dave Parr — Fri, 26 Jun 2020 21:26:59 +0000

I made an electron app in 4 lines...

nativefier "http://musicforprogramming.net/" -n "musicforprogramming"
cd musicforprogramming-linux-x64/
sudo chmod +x musicforprogramming
./musicforprogramming

Motivation

I like music, but don't like music in browser tabs. Basically because i have it open all the time, I want to find and control it easily, and I don't want it cluttering up an area that might be soley focused on work.

I discovered some neat apps for google play, and wanted to see if there was one for my other go to, musicforprogramming. There wasn't, so I just casually googled how to convert a page into an electron app and OMFG!

Solution

nativefier / nativefier

Make any web page a desktop application

Note: Nativefier is unmaintained, please see #1577.

Nativefier

You want to make a native-looking wrapper for WhatsApp Web (or any web page).

nativefier 'web.whatsapp.com'

You're done.

Introduction

Nativefier is a command-line tool to easily create a “desktop app” for any web site with minimal fuss. Apps are wrapped by Electron (which uses Chromium under the hood) in an OS executable (.app, .exe, etc) usable on Windows, macOS and Linux.

I built this because I grew tired of having to Alt-Tab to my browser and then search through numerous open tabs when using Messenger or Whatsapp Web (HN thread). Nativefier features:

Automatically retrieval of app icon / name
Injection of custom JS & CSS
Many more, see the API docs or nativefier --help

Installation

Install Nativefier globally with npm install -g nativefier . Requirements:

macOS 10.13+ / Windows / Linux
Node.js ≥ 16.9…

View on GitHub

This post and this one helped iron out some kinks and now I can launch programming music right from my VS code terminal!

Extra credit

alias musicforprogramming="~/Dev/musicforprogramming-linux-x64/musicforprogramming

Investigating interactions between dev.to and stackbit

Dave Parr — Fri, 26 Jun 2020 10:18:47 +0000

Is the `description` also the `excerpt`?

I was poking around in the dev.to api, and
had my stackbit cms
integration
open in another window and I noticed something I hadn’t before. My first
ever post had a neat ‘excerpt’ that wasn’t actually part of the main
body of the post. My other posts didn’t. In the yaml there is actually
a key called excerpt with the string in question. The API also has a
value that can be part of the response which is a string labelled
“Description”
which I hadn’t integrated into my
dev.to.ol package yet. I have
now. I must have written that first post through the editor built into
the website. Is this field no longer supported? I seem to be able to set
it through the API, but I can’t see the info anywhere on the current
site.

So, if I’m correct, this post should show up with a neat little
description on stackbit, but not on dev.to.

UPDATE: Apparently not. So what does the description field do then?

Should liquid tags work in stackbit?

I like the liquid tags, in some cases. I was even debating writing an R
function to convert a url into the relevant liquid tag through string
parsing, but I’ve spotted that liquid tags aren’t actually supported by
stackbit? See the end of this post on dev, and this copy of it on my
stackbit site, where the link seems large and broken. Are there plans to
support this on stack bit? Or do you already, and I’ve just not spotted
how to ‘turn it on’?

n.b. if you are on my personal site daveparr.info, this might be
confusing, but it’s actually generated from my content on dev.to via
an integration with stack bit. You can see more about this from the
links in the footer VVV

Why use AWS Lambda for Data Science?

Dave Parr — Thu, 25 Jun 2020 14:23:43 +0000

Motivation

Serverless is a way to deploy code, without having to manage the
infrastructure underneath it. In AWS terms this means there is an
compute instance that runs your code, except you don’t control it, and
that might be a good thing. It only exists when it’s asked for by
something else, and therefore you only pay for the work it does. If you
need to do work concurrently you get a new instance, which also goes
away as soon as you don’t need it, so it’s scaleable.

For a data scientist this is an interesting prospect for a number of
reasons. The first is keeping you hands clean. Not all people in this
role come from a ‘operations’ background. Many of us are analysts first,
and graduate into the role. However, that shouldn’t mean we don’t ‘own
our deployments’. However, it also means that we might not have the
background, time or inclination to really get into the nitty-gritty.
Managed infrastructure, that can scale seamlessly out of the box is a
nice middle ground. We can still manage our own deployments, but theres
less to worry about than owning your own EC2 instances, let alone a
fleet of them. The way that the instances themselves die off is also
valuable. We may be doing work that requires 24/7 processing, but often,
we aren’t. Why pay for a box which might have 50% required utilisation
time, or even less?

Limits

Just like in everything there is a balance. There are physical limits
to this process. I’ve had success deploying data science assets in this
architecture, but if you can’t fit your job in these limits, this
already isn’t for you. Sure, data science can be giant machine
learning models on huge hardware with massive data volumes, but we have
to be honest and acknowledge that it isn’t always. K.I.S.S. should apply
to everything.

If you can get good enough business results with a linear regression,
don’t put in 99% more effort to train the new neural network hotness to
get a 2% increase in performance. Simplicity in calculation, deployment,
and explainability matter.

“No ML is easier to manage than no ML” ©
[@julsimon](https://twitter.com/julsimon/status/1124383078313537536)

Getting started

from scipy import stats
import numpy as np

np.random.seed(12345678)

x = np.random.random(10)
y = 1.6*x + np.random.random(10)

slope, intercept, r_value, p_value, std_err = 
  stats.linregress(x, y)

This is a nonsense linear regression. IMHO it’s a data science ‘Hello
World’. Let’s make it an AWS Lambda serverless function.

+ import json
from scipy import stats
import numpy as np
+ def lambda_handler(event, context):
  np.random.seed(12345678)

x = np.random.random(10)
y = 1.6*x + np.random.random(10)

slope, intercept, r_value, p_value, std_err = stats.linregress(x, y) 
+   return_body = {
  +       "m": slope, "c": intercept,"r2": r_value ** 2, 
  +       "p": p_value, "se": std_err
  +   }
+   return {"body": json.dumps(return_body)}

These changes achieve 3 things:

Turning a script into a function
Supplying the function arguments event and context
Formatting the return as json

These are required as AWS Lambda needs a function. This is so that its
event driven architecture can feed in data through event, and so
that it’s json formatted data can both be received by your function,
and then also the response be returned by that function into the rest of
the system.

You can then open up the AWS console in a browser, navigate to the
Lambda service, and then copy and paste this into this screen:

You can then hit run and…

What happened? Well, because it’s a managed instance, the function
doesn’t know what scipy is. It’s not installed on the cloud, it was
installed on your machine…

Layers

AWS lambda doesn’t pip install ..... Seeing as these run on compute
instances that turn up when needed, and are destroyed when not needed,
with no attached storage, you need to find a way to tell AWS what your
dependencies are, or you’ll just have to write super-pure base Python!
Well, that may not be strictly true. json is built in by default to
every instance, so is boto3, but what about our data science buddies?
numpy, scipy are published by
aws as layers. Layers are bundles of code, that contain the dependencies you need to run the functions you write.
So in this case we can open the ‘layers’ view in AWS and attach these to our function.

Now that you’ve attached all your dependencies with layers, go ahead and
run your function again.

Success! So now you know the basics of how to put some Python data
science into practice on AWS Lambda.

This is a companion post to my talk on using data science in AWS lambda.
If you’re keen to know more, and can’t wait for me to write it all up
here. You can get the gist of the whole talks from this repo 😄

DaveParr / snakes_and_lambdas

Talk for pydata on python datascience lambdas

Snakes and Lambdas

A presentation on using aws lambda for data science tasks.

Foundation

understand aws ecosystem from a base 'cloud concepts' POV
some python experience
data science concepts: linear regression, time series analysis, anomaly detection

Solves

productionising python data science
micro-service for data science
local code -> cloud deployment workflow

Requires

to generate the slides, use revelation

Delivered at

Contributions

I wrote and compiled all the material and the experience that drove it, however I have also been able to use a vast wealth of other peoples resources they have shared on the topic. These are clearly linked and I encourage you to go and dive deeper into those resources. Without those resources I would not have been able to implement any of the material described here.

View on GitHub

Building my first Django project with CSS and Static Files

Dave Parr — Thu, 25 Jun 2020 13:14:11 +0000

I'm working through Django for Beginners by William S. Vincent. Until now we've had some pretty barebones Times New Roman style UI, however that's all about to change! I think.

I'm really appreciating this incremental iteration of concepts in each new Chapter. Each time I start a new app for the chapter it's engraining the muscle memory and concept recall into me. I'm able to run through the first 5 lines of code at the CLI nearly from memory. It's also a great mechanism to help you if you get stuck on a wierd error and don't know what to do. You know the next chapter will have a clean slate. Great idea William.

The repetition isn't just the set-up though.

“Now we can add the functionality for individual blog pages. How do we do that? We need to create a new view, url, and template. I hope you’re noticing a pattern in development with Django now!”

I sure am.

I'm glad that the book also doesn't go into non-django areas, but still points readers to places to look for more info. The last chapter pointed me towards some resources to understand more about databases, and this one does similar with CSS.

It was interesting to see that the approach used to identify a blog post for navigation in the URL patterns looked kinda regex-ish:

urlpatterns = [
    path('post/<int:pk>/', BlogDetailView.as_view(), name='post_detail'),
    path('', BlogListView.as_view(), name='home'),
]

<int:pk> is the part that identifies the blog post required for the url to link from the ListView to the DetailView. It was interesting to find out there was something like a unique identifier for each post baked into it in the background. I was wondering about how this primary key is treated in bigger projects. Is it common to keep this approach of integer ordered primary keys, or do they get replaced in larger, more complex projects with hashes or other unique ids?

3 minimal features for my dev.to api wrapper

Dave Parr — Wed, 17 Jun 2020 13:02:10 +0000

I’ve made 3 very small features for my open source R package wrapping
the dev.to API.

`create_new_article`

Now that the main requirements for the file to post are stabilising,
I’ve written a quick and dirty function to make a boilerplate article:

create_new_article <-
  function(title,
           series = 'series',
           tags = '["tag1", "tag2"]',
           file = "") {
    boilerplate_frontmatter <-
      glue::glue(
        '---\ntitle: "{title}"\noutput: github_document\nseries: "{series}"\ntags: {tags}\n---'
      )

    cat(boilerplate_frontmatter, file = file)
  }

This will use the glue package to put
the strings in the function argument into the right place in the
boilerplate YAML front matter. If then uses cat to either print that
to screen, or to create a new file with it, if a file path is supplied.

`main_image`

If there is a main_image parameter in the YAML front matter that is
a url of an image, that image will be set as the cover image of the
post. I got this one from a photo by Julian
Dufort
on
Unsplash.
I believe that if you put an unsplash URL into this field that goes
directly to the image it is within their
license, though if anyone knows
something to the contrary please let me know. It’s the first time I have
used this service, despite hearing about it for years.

Collapse spaces tags

I recently fooled myself for a good ten minutes into thinking that there
was a problem with my API code yesterday when I kept getting a 422
response to putting a new article up, when in fact it was that I had a
space character in one of my tags. Now the post_new_article function
collapses any spaces it encounters in tags. Achieving this was a breeze
with purrr and
stringr:

purrr::map(file_frontmatter$tags, stringr::str_remove_all, " ")

This little nugget takes the list of tags, and then maps the function
str_remove_all across all the spaces. This isn’t at all exposed to the
user, as it’s non-negotiable from the API side anyway :)

Forem: Dave Parr

Moving Starpilot to GraphQL

How I did it

GraphQL

GitHub GraphQL Explorer

Insomnia

graphql-query

gql

Other tweaks

The tweaks that didn’t make it

Conclusion

How do you use your VSCode profile?

Copilot for your GitHub stars

Inspiration

The original idea

Semantic vs keyword

Keyword

Semantic

DaveParr / starpilot

Use your GitHub stars for great good!

Starpilot is like copilot, but for GitHub stars.

Installation

Why retrival augmented generation matters

Vectorstores

Designing a system

Typer

tiangolo / typer

Typer, build great CLIs. Easy to code. Based on Python type hints.

langchain-ai / langchain

⚡ Building applications with LLMs through composability ⚡

🦜️🔗 LangChain

Quick Install

🤔 What is LangChain?

Chroma

chroma-core / chroma

the AI-native open-source embedding database

GPT4All

nomic-ai / gpt4all

gpt4all: open-source LLM chatbots that you can run anywhere

GPT4All

GPT4All: An ecosystem of open-source on-edge large language models.

PyGithub

PyGithub / PyGithub

Typed interactions with the GitHub API v3

PyGitHub

Install

Simple Demo

Documentation

Development

Contributing

The other way to build

Click

pallets / click

Python composable command line interface toolkit

$ click_

Installing

A Simple Example

run-llama / llama_index

LlamaIndex (formerly GPT Index) is a data framework for your LLM applications

🗂️ LlamaIndex 🦙

Ecosystem

🚀 Overview

Context

Proposed Solution

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.

Faiss

News

Introduction

openai / openai-python

The official Python library for the OpenAI API

OpenAI Python API library

Documentation

Installation

Usage

What state is starpilot now?

Current features

starpilot read MyCoolUserName

starpilot shoot "insert topic here"

starpilot fortuneteller "Insert a question here"

`graphql-query`

`gql`

What state is `starpilot` now?

`starpilot read MyCoolUserName`

`starpilot shoot "insert topic here"`

`starpilot fortuneteller "Insert a question here"`

`evolves_from_species`