The latest articles on Forem by artakulov (@artakulov). https://forem.com/artakulov https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3862175%2Fe8444fd6-f6fb-4873-a2a0-add6ae03c14d.jpeg https://forem.com/artakulov en artakulov Sun, 05 Apr 2026 11:47:23 +0000 https://forem.com/artakulov/what-i-learned-processing-50-federal-data-sources-with-nodejs-2eg6 https://forem.com/artakulov/what-i-learned-processing-50-federal-data-sources-with-nodejs-2eg6 <p>I've been working on a project that pulls environmental data from federal agencies — EPA, FEMA, USGS, CDC, Census, and about 45 others.</p> <p>Some things I ran into that might save you time:</p> <h2> Federal APIs are wild </h2> <p>No two agencies use the same format. EPA gives you XML. USGS gives you tab-separated files from the 90s. FEMA has a decent REST API but paginated at 1,000 records. Census has its own query language.</p> <p>I ended up writing a separate parser for each source. No universal adapter worked.</p> <h2> Streaming EJS at scale </h2> <p>I needed to render ~280K static HTML pages from templates. The first approach (one EJS render per file, sequential) took 14 hours. Switched to streaming writes with a worker pool — got it down to ~5 minutes. The bottleneck was always disk I/O, not rendering.</p> <h2> Cloudflare R2 is underrated for static sites </h2> <p>Hosting 280K HTML files on traditional hosting is painful. R2 + Workers turned out to be perfect — no file count limits, edge-cached globally, and the free tier covers a lot.</p> <h2> Rate limits will find you </h2> <p>Every federal API has different rate limits, and most don't document them well. EPA ECHO silently returns empty results after ~300 requests/minute. USGS returns 503s. I ended up building a generic retry queue with exponential backoff that handles all of them.</p> <p>Would love to hear from others who've worked with government data APIs. What's the worst format you've had to parse?</p> api data node performance