<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Francois LAGIER</title>
    <description>The latest articles on Forem by Francois LAGIER (@francoislagier).</description>
    <link>https://forem.com/francoislagier</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F105679%2Ffca35a64-3885-4e8f-8a92-457ebc953f9a.png</url>
      <title>Forem: Francois LAGIER</title>
      <link>https://forem.com/francoislagier</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/francoislagier"/>
    <language>en</language>
    <item>
      <title>How are people keeping track of their #AWS Reserved Instances purchases?</title>
      <dc:creator>Francois LAGIER</dc:creator>
      <pubDate>Mon, 24 Jun 2019 22:19:36 +0000</pubDate>
      <link>https://forem.com/francoislagier/how-are-people-keeping-track-of-their-aws-reserved-instances-purchases-c94</link>
      <guid>https://forem.com/francoislagier/how-are-people-keeping-track-of-their-aws-reserved-instances-purchases-c94</guid>
      <description>&lt;p&gt;I'm curious to learn how folks keep track of their RIs. &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What number do you keep an eye on and how? 

&lt;ul&gt;
&lt;li&gt;RI Utilizations? RI Coverage?&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;li&gt;Do you use any scripts or tools to do so?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Looking forward to learning new tricks&lt;/p&gt;

</description>
      <category>aws</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Chicago AWS Summit: Things to do, eat and see. Recommendations from a local south-sider.</title>
      <dc:creator>Francois LAGIER</dc:creator>
      <pubDate>Wed, 22 May 2019 18:03:28 +0000</pubDate>
      <link>https://forem.com/cloudforecast/chicago-aws-summit-things-to-do-eat-and-see-recommendations-from-a-local-south-sider-4kdd</link>
      <guid>https://forem.com/cloudforecast/chicago-aws-summit-things-to-do-eat-and-see-recommendations-from-a-local-south-sider-4kdd</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published by co-founder &lt;a href="https://twitter.com/toeknee123" rel="noopener noreferrer"&gt;Tony Chan&lt;/a&gt; on &lt;a href="https://cloudforecast.io/blog" rel="noopener noreferrer"&gt;cloudforecast.io/blog&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Heading to the AWS Summit in Chicago on May 30th and need recommendations on things to eat, see or drink near McCormick Place? &lt;/p&gt;

&lt;p&gt;For those who have been to McCormmick place, you already know the area is lacking quality restaurant, bars or things to do. The area surrounding McCormick place, South Loop and Motor Row, is still “up and coming”, but it’s still 3-5 years out from having more viable options for people attending a conference. Do a quick google map search for “restaurants” around the area and you’ll see what I mean. &lt;/p&gt;

&lt;p&gt;I have listed a few of my favorite places near McCormick Place, but I recommend venturing off to the surrounding neighborhoods for more options and a unique experience (Bridgeport, Pilsen, Chinatown). I grew up in Bridgeport, was a tour guide in Chinatown in college and still reside in the area. I hope my local knowledge of the “South-side” will make you feel a bit better venturing into uncharted territory for many. This is definitely not a complete list, but it's a starting point of a select few places.&lt;/p&gt;

&lt;p&gt;If you need more recommendations or have other places to add, feel free to reach out to me directly, &lt;a href="//mailto:tony@cloudforecaset.io"&gt;tony@cloudforecaset.io&lt;/a&gt;. Happy to help in any way! &lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fcloudforecast.io%2Fblog%2Fassets%2Fimages%2Fposts%2Fsouthloop.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fcloudforecast.io%2Fblog%2Fassets%2Fimages%2Fposts%2Fsouthloop.png" alt="South Loop and Motor Row"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Walking distance, &amp;lt;15-20 minute walk. (South Loop and Motor Row).
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Places to eat&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pizannos&lt;/strong&gt; - Sit down restaurant. One of the best restaurants for Chicago style thin crust. They also have deep dish if that is what you are looking for. Crust is buttery and corn meal based similar to Lou’s and Pequods. &lt;a href="https://goo.gl/maps/S7D2xnx9q9BqvDSj9" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14 Parish&lt;/strong&gt; - Sit down restaurant. Carribean food within &amp;lt;5 min walk of McCormick Place. This will be a good option if you need a spot for a business meeting. &lt;a href="https://goo.gl/maps/fhqetANyijKh43ZA6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Opart Thai&lt;/strong&gt; - Sit down restaurant. Considered one of the best Thai restaurants in Chicago and a neighborhood staple.  Another good option if you need a spot for a business meeting. &lt;a href="https://goo.gl/maps/WsTphXpCZ2yq4mvH6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chef Lucianos&lt;/strong&gt; - Quick bite. Hole in the wall restaurant for quick take out or lunch. A lot of their dishes are Indian inspired and unique. I recommend the fried chicken or Chicken curry. &lt;a href="https://goo.gl/maps/fdtnQxZh5gtSMFR2A" rel="noopener noreferrer"&gt;Google Maps link&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Places for a drink&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Weather Mark Tavern&lt;/strong&gt; - Nautical themed dive bar. Great drink menu and daily food specials. &lt;a href="https://goo.gl/maps/sZG9Zqw3AuuSADrGA" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reggies Chicago&lt;/strong&gt; - Another dive bar and neighborhood staple. Live music everyday with a rooftop patio. Very grungy vibe, but a very fun bar.&lt;a href="https://goo.gl/maps/6978caWmbavTcYya7" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fat Pour:&lt;/strong&gt;  Relatively new place for the area. Wide beer selection but gets pretty busy during conferences due to lack of options around the area. &lt;a href="https://goo.gl/maps/gEdXLgo9hMpipGGV9" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Other options:&lt;/strong&gt; Vu Rooftop bar, Woven and Bound Bar, First Draft’
Further South Loop Options: First Draft, Kaseys Tavern &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Coffee shops to work out of&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Spoke and Bird&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;TeaPotBrew Bakery&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fcloudforecast.io%2Fblog%2Fassets%2Fimages%2Fposts%2Fcbp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fcloudforecast.io%2Fblog%2Fassets%2Fimages%2Fposts%2Fcbp.png" alt="South Loop and Motor Row"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  &amp;lt;15 min rideshare distance (Chinatown, Bridgeport and Pilsen)
&lt;/h3&gt;

&lt;p&gt;&lt;em&gt;Side note: You can get to Chinatown via the 21 Cermak bus which should take less than 10 minutes. Alternatively, it’s about a 20 minute walk.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Places to eat&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Chinatown:&lt;/strong&gt; Chi Cafe - Sit down restaurant. Hong Kong style chinese food. Great stop for a quick lunch. I recommend the Sizziling Beef in Sake Sauce. &lt;a href="https://goo.gl/maps/SbZoT5ySK6d3i6CR9" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chinatown:&lt;/strong&gt; Yummy Yummy Noodles - Sit down restaurant. Small family owned place that is known for their noodle soups. &lt;a href="https://goo.gl/maps/mfgRHQ9ep7mpZXNP6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chinatown:&lt;/strong&gt; Ming Hin - Sit down restaurant. One of the better places for dim sum. Picture menu and ipads for easy ordering. &lt;a href="https://goo.gl/maps/aghZ7YhgnV3Tmhri7" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Other options in Chinatown&lt;/strong&gt;: Dolo, Qing Xiang Yuan Dumplings, Happy Lamb Hot Pot &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bridgeport:&lt;/strong&gt; Ricobenes - Quick bite. Chicago south side staple. Known for their breaded steak sandwiches, vesuvio chicken sandwiches and pizza. &lt;a href="https://goo.gl/maps/bYKt7GF9a9GcgQDX7" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bridgeport:&lt;/strong&gt; Phil’s Pizza - Sit down restaurant. One of the best places in the city for Chicago tavern style pizza. Opens at 4pm. &lt;a href="https://goo.gl/maps/SNQGsYwfAmwnstMy6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Other options in Bridgeport:&lt;/strong&gt; Freddies, The Duck Inn, Nana’s. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Pilsen:&lt;/strong&gt; Carnitas don Pedro - Quick bite. Hole in the wall restaurant service the best Michoacana style carnitas in the city. Pig cooked in it’s own fat, what can go wrong? Cash only. I recommend a mix plate of Carnitas to try out a little bit of everything. Not open for dinner. &lt;a href="https://goo.gl/maps/TckVpxTeb19Vjzse6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Pilsen:&lt;/strong&gt; &lt;a href="https://goo.gl/maps/ZXKHbbKxCHuJH8Zn9" rel="noopener noreferrer"&gt;Tortillería y Taquerías Atotonilco&lt;/a&gt; or &lt;a href="https://goo.gl/maps/K1nTQ7fHhUYMsqv97" rel="noopener noreferrer"&gt;El Milagro&lt;/a&gt; for tacos.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Other options in Pilsen:&lt;/strong&gt; HaiSous, S.K.Y., Monnie Burkes or La Vaca. &lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Places for a drink&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bridgeport:&lt;/strong&gt; Marias and Kimski - Fun little dive bar with a wide beer selection and korean/polish food. Great option for a business meeting and drinks. &lt;a href="https://goo.gl/maps/WnA1FY7j9VMCcX8E6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bridgeport:&lt;/strong&gt; Marz Brewery - One of the best breweries in the city with great food. Their beers are unique and everything here is delicious. It’s off the beaten path, but worth going to if you need to find a place for drinks. &lt;a href="https://goo.gl/maps/cHJqiek1sh26ReEz7" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pilsen:&lt;/strong&gt; Moody Tongue - 12 layer Chocolate cake and beers. Enough said. &lt;a href="https://goo.gl/maps/Uctm3cTLcLZ8EYie6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Coffee shops to work out of:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bridgeport:&lt;/strong&gt; Bridgeport Coffee &lt;a href="https://goo.gl/maps/xob7QHMEiZ8JcLA68" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bridgeport:&lt;/strong&gt; Red Line Cafe &lt;a href="https://goo.gl/maps/Uctm3cTLcLZ8EYie6" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bridgeport:&lt;/strong&gt; Jackalope &lt;a href="https://goo.gl/maps/RnVTfbHNM6h5jwW48" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pilsen:&lt;/strong&gt; La Catrina Cafe &lt;a href="https://goo.gl/maps/zLHL6jsbjovG2A8f8" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pilsen:&lt;/strong&gt; Cafe Jumping Bean &lt;a href="https://goo.gl/maps/wZGJQEBJP84aNED49" rel="noopener noreferrer"&gt;Google Maps Link&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>aws</category>
      <category>awssummitchicago</category>
    </item>
    <item>
      <title>Interview with GitLab CEO, Sid Sijbrandij - Keys to Developing a Self-Hosted Version of Your App.</title>
      <dc:creator>Francois LAGIER</dc:creator>
      <pubDate>Thu, 09 May 2019 18:46:35 +0000</pubDate>
      <link>https://forem.com/cloudforecast/interview-with-gitlab-ceo-sid-sijbrandij-keys-to-developing-a-self-hosted-version-of-your-app-j53</link>
      <guid>https://forem.com/cloudforecast/interview-with-gitlab-ceo-sid-sijbrandij-keys-to-developing-a-self-hosted-version-of-your-app-j53</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://cloudforecast.io/blog/"&gt;cloudforecast.io/blog&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We had the opportunity to chat with &lt;a href="https://twitter.com/sytses"&gt;Sid&lt;/a&gt;, the CEO of &lt;a href="https://gitlab.com/?utm_source=cloudforecast"&gt;GitLab&lt;/a&gt;, on what it takes to offer a self-hosted version of your software a few weeks ago. &lt;a href="https://gitlab.com/?utm_source=cloudforecast"&gt;GitLab&lt;/a&gt; is a single application for the entire software development lifecycle. From project planning and source code management to CI/CD, monitoring, and security. &lt;/p&gt;

&lt;p&gt;In addition to a SaaS product, &lt;a href="https://gitlab.com/?utm_source=cloudforecast"&gt;GitLab&lt;/a&gt; offers a self-managed/self-hosted version of their app that has been a great business model for them. We’ve been working through the idea of offering a self-hosted version of &lt;a href="https://www.cloudforecast.io/?utm_source=blog"&gt;CloudForecast&lt;/a&gt; and knew that Sid would be a great resource to learn from.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR summary:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Leverage third-party software to deploy faster:&lt;/strong&gt; Look into third party software such as  &lt;a href="https://www.replicated.com/?utm_source=cloudforecast"&gt;Replicated&lt;/a&gt; to help speed up your deployment process. &lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Support is important:&lt;/strong&gt; Keep it simple and include support for all your pricing tiers. &lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Offer a freemium version of your app:&lt;/strong&gt; Consider a free version of your software with a few limitations in place. That limitation can be a 30-45 day free trial and flexibility to extend further. This will help you get your foot in the door and create sales opportunities.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Focus your messaging around data security:&lt;/strong&gt; “Data does not leave your network” is the single most important thing you need to focus on while building a self-hosted version of your software. &lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Build a product that saves time:&lt;/strong&gt; Engineers do not want to fiddle with tools; they want to build software. Focusing on an out of the box experience that can get them results in 5 minutes or less is very important. &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here are a few highlights from our conversation. We hope this can help your company if you are considering building a self-hosted version of your app. The full interview video is available after this post. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;There are a lot of technical challenges involved with building an self-hosted version of your software. We found a third party company called Replicated, that makes it easy for SaaS and software companies to make the move into selling modern, on-prem software.&lt;/strong&gt; &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What are your thoughts with using software like &lt;a href="https://www.replicated.com/?utm_source=cloudforecast"&gt;Replicated&lt;/a&gt;?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Faster is better, especially if you are not sure if there is a giant market. It makes a lot of sense to go that route. I know the people from &lt;a href="https://www.replicated.com/?utm_source=cloudforecast"&gt;Replicated&lt;/a&gt; - they are good and they know their stuff. When we first started packaging GitLab, Replicated was not there yet or we were not aware of them at the time. &lt;/p&gt;

&lt;p&gt;There are a few lessons that we ran into while doing it on our own and things you need to consider. First of all, it’s A LOT of work to package up your stuff. We have an entire distribution team that is focused on this. We based our packaging on a technology from Chef called Omnibus. The end user experience is great, the team did a great job, but it is a significant investment. That speaks for using a third party tool like &lt;a href="https://www.replicated.com/?utm_source=cloudforecast"&gt;Replicated&lt;/a&gt; to speed up your process. &lt;/p&gt;

&lt;p&gt;Few things to be aware of in the Enterprise space, not everyone uses Docker and Kubernetes. That might be different for your customers. But, we found that a lot of our customers do not like running them in production. They might have other tools that assume a boring Unix app. Examples: "Oh! I want to provision it as expected user", or "I want to run an agent on the system”. They cannot accept a black box but they want more flexibility. With that being said, I think time is the most important thing you have in a startup which makes going with a solution like &lt;a href="https://www.replicated.com/?utm_source=cloudforecast"&gt;Replicated&lt;/a&gt; makes sense. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How should we consider pricing if we were to go with a self-hosted version of CloudForecast? What are the lessons you’ve learned while trying to figure out a pricing model for GitLab?&lt;/strong&gt; &lt;/p&gt;

&lt;p&gt;We're frequently bought by companies that don't do a consumption pricing model. They want a predictable amount and they know what they will be charged at the end of the month.&lt;/p&gt;

&lt;p&gt;They get a budget for it and they have to put in a dollar number when they request it. Making it variable makes it difficult to purchase. It might not make it difficult though if you can figure out a way to lump it into the existing cloud provider bill. Cloud providers are starting to offer marketplaces that allow you to bill for your software. In that case, your offering of variable pricing can lump right into cloud provider bill. &lt;/p&gt;

&lt;p&gt;A boring solution would to have a few tiers with fixed pricing based on their monthly spend with their cloud provider. It’s not ideal, but it makes it a lot easier to purchase because they know exactly what they will be billed using your software. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do you all handle and price support out for GitLab? Any lessons there with approaches you’ve tried out?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You should never sell without support. You should always include support because you’ll end up with a situation like, "Well, I can't use it I have a problem with it and they say well tough luck you didn't buy support.". The reverse can also be true with ONLY selling support. We did that with GitLab in the beginning. People didn’t have any problems and they cancelled their support contracts. It made sense for them since they were able to use the product with no issues. &lt;/p&gt;

&lt;p&gt;To handle both scenarios, we just include support if they want support or not. They will pay the same price for either/or. I am a huge proponent of building it in no matter what. There is one subscription pricing and it includes support. There is a scenario with support with a good, better, best model. Example for us: the best model means that you’ll get better support that is more timely, more extensive and with more hand holding.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the sales process like selling into enterprise companies? Who should we be talking? Any tips on ways we can start selling into these companies?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;GitLab is an open core company so lots of people that we get to sell into start with our open source offering. This has really helped us get our foot in the door early on and helps speeds up the sales cycle because they are using your product. Typical sales cycle on average for GitLab: Less than 10k is about 60 days. 10k - 100k is about 90 days. 100k and more is about 120 days. &lt;/p&gt;

&lt;p&gt;If you don’t want to open source your software, you should consider a free-tier version of CloudForecast. There is a whole big thing called “Shadow IT”, where people don’t like going through the process and want to run something that can help them. They’ll frequently use free tools since so they don’t have to expense it. &lt;/p&gt;

&lt;p&gt;However, you still have to consider that they might not be able to use the free SaaS version since they cannot share confidential data or credentials. If you had a on-premise, free version of your software, they can use your tool and still be good stewards of the private data. You become an approved vendor since the data is never leaving the network, which becomes less of a problem. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;To follow up on a free version of your product, would you recommend maybe offering a limited version of your software? What are some variants you’ve seen that has worked well?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;There can be many variants to a free version of your product. For GitLab, we’re open core, so our free version will always be free and then we charge for additional features on top. With a lot of proprietary software, the limited version might be a fixed “time limit” and you cannot continue after X days. It doesn’t have to be 30 days and it can be a whole year as an extreme example. &lt;/p&gt;

&lt;p&gt;With someone who has a problem of spending too much with their cloud provider, their boss has tasked them to solve that problem right away. They have to go through a purchasing process of 30-60 days to get the vendor approved when they need a solution right now. They might not object to a 30-45 day free trial to see how the product can work for them. From what I’ve seen, that seems to be the industry standard and it makes a lot of sense. For GitLab, we’ve settled to offer 30 days and if people ask for an extension, we are happy to oblige and extend their free trial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What are some messaging and important features to consider that resonates to decision maker looking for an on-prem solution?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The most important thing to highlight with messaging is “Data does not leave your network”. Maybe with CloudForecast, focus on getting your first cost report in less than five minutes should be key. You also want to highlight average percentage someone can save. Highlighting pricing is also very important. We’ve found that the pricing page is the most popular page on any website. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;With your experience at Gitlab, what are your customers looking for and considering as an on-prem solution?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;GitLab wins because we're a single application for the entire DevOps life cycle. We save them time because they don't have to combine a chain of other applications together and spend a lot of time integrating them all.  All this leads to a subpar experience anyways since they are all still separate apps. &lt;/p&gt;

&lt;p&gt;You’ll start to understand quickly that people have very limited time. They have this assignment to save cost and they need to get results quickly. People want to make software, not string 50 different DevOps tool to get it. That out-of-the-box experience that can get you results in 5 minutes or less is very important. I’d encourage you to show a video of you downloading and installing your software, them putting in their AWS credentials and the result is getting a super cool cost report. That is insightful. If you can make that video in less than three minutes, that is awesome.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do you have a rough idea how long it takes from signing up to installing it and having a working GitLab on your servers? Is there any manual processes involved or is it fully automated process that takes less than 10 minutes?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Five minutes including registering the subdomain is what we are shooting for. In previous years I went online and I live streamed myself doing it. I learned a lot from that.  &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not a customer and interested in trying us out?&lt;/strong&gt; Sign up today and get started with a risk-free 30 day free-trial with us: &lt;a href="https://app.cloudforecast.io/users/sign_up?utm_source=blog"&gt;Start 30 Day Free-trial&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Please reach out to us if you have any feedback for us or have suggestions on what we should build next.&lt;/strong&gt; We would love to hear from you:  &lt;a href="//mailto:hello@cloudforecast.io"&gt;hello@cloudforecast.io&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;iframe width="710" height="399" src="https://www.youtube.com/embed/Lo0bejtOnQc"&gt;
&lt;/iframe&gt;
&lt;/p&gt;

</description>
      <category>aws</category>
      <category>devops</category>
      <category>gitlab</category>
      <category>marketing</category>
    </item>
    <item>
      <title>Watch out for Unexpected S3 cost when using Athena.</title>
      <dc:creator>Francois LAGIER</dc:creator>
      <pubDate>Mon, 29 Apr 2019 17:39:59 +0000</pubDate>
      <link>https://forem.com/cloudforecast/watch-out-for-unexpected-s3-cost-when-using-athena-5hdm</link>
      <guid>https://forem.com/cloudforecast/watch-out-for-unexpected-s3-cost-when-using-athena-5hdm</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://cloudforecast.io/blog/unexpected-s3-cost-when-using-athena/"&gt;cloudforecast.io/blog&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Following up on my last blog post (&lt;a href="https://cloudforecast.io/blog/using-parquet-on-athena-to-save-money-on-aws/"&gt;Using Parquet on Athena to Save Money on AWS&lt;/a&gt;), I wanted to share another thought about Athena, specifically how the S3 bucket is being used by Athena to store query results.&lt;/p&gt;

&lt;p&gt;On the first use of Athena, AWS will automatically create a new bucket to store the query results (bucket name &lt;code&gt;aws-athena-query-results-&amp;lt;ACCOUNTID&amp;gt;-&amp;lt;REGION&amp;gt;&lt;/code&gt;). Athena will store a raw result file (&lt;code&gt;QueryId.csv&lt;/code&gt;) and a metadata file (&lt;code&gt;QueryId.csv.metadata&lt;/code&gt;). &lt;/p&gt;

&lt;p&gt;By storing the data using the &lt;code&gt;QueryId&lt;/code&gt;, it allows you to access previous query's result without re-running them (saving you money since you don't need to rescan the data).&lt;/p&gt;

&lt;p&gt;However, you are the owner of the bucket and therefore responsible for the storage on this bucket and here is a couple of reasons why it could cost you a LOT of money:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;#1 All the queries are being stored! ALL OF THEM!&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
Athena store every query results in the bucket. Query data will just accumulate forever costing more and more $$$.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;#2 Your data may be compressed but the results are not&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
S3 is storing the results in raw CSV. Your data may be compressed (GZIP, Snappy, ...) but the results will be in raw CSV. As an example, I ran an &lt;em&gt;accidental&lt;/em&gt; &lt;code&gt;SELECT * FROM flights.parquet_snappy_data&lt;/code&gt; on a 84M dataset using Parquet which resulted on a &lt;strong&gt;977MB&lt;/strong&gt; file on S3. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to fix this?&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
&lt;em&gt;It's actually pretty easy&lt;/em&gt;. If (and only if) you don't plan to re-use old query results, make sure to setup &lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html"&gt;Lifecycle&lt;/a&gt; on your bucket using a Transition or Expiration actions. For example, you could delete query results after 1 or 7 days. At CloudForecast, we actually don't persist &lt;code&gt;QueryIds&lt;/code&gt; since it's not useful to us so we expire the S3 files after 1 day.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Feel free to reach out if you have any questions at  &lt;a href="//mailto:francois@cloudforecast.io"&gt;francois@cloudforecast.io&lt;/a&gt;  or by Twitter:  &lt;a href="https://twitter.com/francoislagier"&gt;@francoislagier&lt;/a&gt;. Also, follow our journey  &lt;a href="https://twitter.com/cloudforecast"&gt;@cloudforecast&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>finops</category>
      <category>athena</category>
      <category>s3</category>
    </item>
    <item>
      <title>Using Parquet on Athena to Save Money on AWS</title>
      <dc:creator>Francois LAGIER</dc:creator>
      <pubDate>Mon, 15 Apr 2019 17:00:45 +0000</pubDate>
      <link>https://forem.com/cloudforecast/using-parquet-on-athena-to-save-money-on-aws-3fac</link>
      <guid>https://forem.com/cloudforecast/using-parquet-on-athena-to-save-money-on-aws-3fac</guid>
      <description>

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://cloudforecast.io/blog/"&gt;cloudforecast.io/blog&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;To show you how you can optimize your Athena query and save money, we will use the ‘2018 Flight On-Time Performance’ dataset from the Bureau of Transportation Statistics (&lt;a href="bts.gov"&gt;bts.gov&lt;/a&gt;). We will also drop a few interesting facts about US Airports ✈️queried from the dataset while using Athena.&lt;/p&gt;

&lt;p&gt;All datasets and queries used in this post can be found in our &lt;a href="https://github.com/CloudForecast/blog-posts"&gt; Github repo&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Let's start with some terminology&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Amazon Athena&lt;/strong&gt; is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Pricing is based on the amount of data scanned by each query.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Apache Parquet&lt;/strong&gt; is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If pricing is based on the amount of data scanned, you should always optimize your dataset to process the least amount of data using one of the following techniques: compressing, partitioning and using a columnar file format. We will demonstrate the benefits of compression and using a columnar format. &lt;/p&gt;

&lt;p&gt;Using compressions will reduce the amount of data scanned by Athena, and also reduce your S3 storage. It's a Win-Win for your AWS bill. Supported formats: &lt;em&gt;GZIP, LZO, SNAPPY (Parquet) and ZLIB.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Instead of using a row-level approach, columnar format is storing data by columns. This allows  Athena to only query and process the required columns and ignore the rest. If you want to learn more about columnar, check out &lt;a href="%5Bhttps://en.wikipedia.org/wiki/RCFile%5D(https://en.wikipedia.org/wiki/RCFile)"&gt;Wikipedia&lt;/a&gt; and/or the "&lt;a href="https://towardsdatascience.com/the-beauty-of-column-oriented-data-2945c0c9f560"&gt;The beauty of  column-oriented  data&lt;/a&gt;" article by Maxim Zaks&lt;/p&gt;

&lt;h4&gt;
  
  
  Getting familiar with the data
&lt;/h4&gt;

&lt;p&gt;While using the  ‘2018 Flight On-Time Performance’ dataset, we will use three different types of files to compare performance with the data processed: CSV, GZip, and Parquet files.&lt;/p&gt;

&lt;p&gt;The files downloaded from the Bureau of Transportation Statistics are simple CSV files with 23 columns (such as FlightDate, Airline, Flight #, Origin, Destination, Delay, Cancelled, ...)&lt;/p&gt;

&lt;p&gt;Same data, 3 versions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Raw CSV:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Description&lt;/em&gt;: No compression, just a plain set of CSV files
&lt;/li&gt;
&lt;li&gt;Files: 12 ~55MB files (one for each month)&lt;/li&gt;
&lt;li&gt;Total dataset size: ~666MBs&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;GZip CSV&lt;/strong&gt;: 

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Description&lt;/em&gt;: Simple CSV files compressed using GZip to compress them. &lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Files&lt;/em&gt;: 12 ~10MB Gzipped CSV files (one for each month).&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Total dataset size&lt;/em&gt;: ~126MBs&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Compressed Parquet&lt;/strong&gt;: 

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Description&lt;/em&gt;: We converted to the CSV file to parquet using Spark. The same process could also be done with (&lt;a href="https://aws.amazon.com/glue/"&gt;AWS Glue&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Files:&lt;/em&gt; 12 ~8MB Parquet file using the default compression (&lt;a href="https://github.com/google/snappy"&gt;Snappy&lt;/a&gt;). &lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Total dataset size:&lt;/em&gt; ~84MBs&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Find the three dataset versions on our &lt;a href="https://github.com/CloudForecast/blog-posts/tree/master/athena-parquet"&gt;Github repo&lt;/a&gt;.&lt;/p&gt;

&lt;h4&gt;
  
  
  Creating the various tables
&lt;/h4&gt;

&lt;p&gt;Since the various formats and/or compressions are different, each &lt;code&gt;CREATE&lt;/code&gt; statement needs to indicate to Athena which format/compression it should use. &lt;/p&gt;

&lt;h5&gt;
  
  
  Raw CSVs
&lt;/h5&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CREATE EXTERNAL TABLE IF NOT EXISTS flights.raw_data (&lt;br&gt;
    &lt;code&gt;year&lt;/code&gt; SMALLINT,&lt;br&gt;
    &lt;code&gt;month&lt;/code&gt; SMALLINT,&lt;br&gt;
    &lt;code&gt;day_of_month&lt;/code&gt; SMALLINT,&lt;br&gt;
    &lt;code&gt;flight_date&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;op_unique_carrier&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;flight_num&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;origin&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;destination&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;crs_dep_time&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;dep_time&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;dep_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;taxi_out&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;wheels_off&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;arr_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;cancelled&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;cancellation_code&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;diverted&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;air_time&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;carrier_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;weather_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;nas_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;security_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;late_aircraft_delay&lt;/code&gt; DOUBLE&lt;br&gt;
)&lt;br&gt;
ROW FORMAT DELIMITED&lt;br&gt;
FIELDS TERMINATED BY ','&lt;br&gt;
ESCAPED BY '\"'&lt;br&gt;
LINES TERMINATED BY '\n'&lt;br&gt;
LOCATION 's3://INSERT_BUCKET_NAME/raw'&lt;br&gt;
TBLPROPERTIES (&lt;br&gt;
    'skip.header.line.count'='1',&lt;br&gt;
    'serialization.null.format'=''&lt;br&gt;
);&lt;br&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h5&gt;
&lt;br&gt;
  &lt;br&gt;
  &lt;br&gt;
  Gzipped CSVs&lt;br&gt;
&lt;/h5&gt;

&lt;p&gt;Athena automatically detects the gzip format (based on the ".gz" suffix) so we can re-use the query from above. See the full query &lt;a href="https://github.com/CloudForecast/blog-posts/tree/master/athena-parquet"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;h5&gt;
  
  
  Parquet Files using Snappy
&lt;/h5&gt;

&lt;p&gt;Using both &lt;code&gt;STORED AS PARQUET&lt;/code&gt; and &lt;code&gt;"parquet.compress"="SNAPPY"&lt;/code&gt;, Athena will be able to process our data flawlessly. &lt;/p&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CREATE EXTERNAL TABLE IF NOT EXISTS flights.parquet_snappy_data (&lt;br&gt;
    &lt;code&gt;year&lt;/code&gt; SMALLINT,&lt;br&gt;
    &lt;code&gt;month&lt;/code&gt; SMALLINT,&lt;br&gt;
    &lt;code&gt;day_of_month&lt;/code&gt; SMALLINT,&lt;br&gt;
    &lt;code&gt;flight_date&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;op_unique_carrier&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;flight_num&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;origin&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;destination&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;crs_dep_time&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;dep_time&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;dep_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;taxi_out&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;wheels_off&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;arr_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;cancelled&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;cancellation_code&lt;/code&gt; STRING,&lt;br&gt;
    &lt;code&gt;diverted&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;air_time&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;carrier_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;weather_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;nas_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;security_delay&lt;/code&gt; DOUBLE,&lt;br&gt;
    &lt;code&gt;late_aircraft_delay&lt;/code&gt; DOUBLE&lt;br&gt;
)&lt;br&gt;
STORED AS PARQUET&lt;br&gt;
LOCATION 's3://INSERT_BUCKET_NAME/parquet'&lt;br&gt;
tblproperties ("parquet.compress"="SNAPPY")&lt;br&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h4&gt;
&lt;br&gt;
  &lt;br&gt;
  &lt;br&gt;
  Picking a couple of queries to play with&lt;br&gt;
&lt;/h4&gt;

&lt;h5&gt;
  
  
  Query #1: Top route in the U.S. in 2018 alongside the number of airlines providing flights for it.
&lt;/h5&gt;

&lt;p&gt;&lt;strong&gt;Spoiler Alert&lt;/strong&gt;: It's Chicago O'Hare International to New York &lt;em&gt;LaGuardia&lt;/em&gt; with 6 unique airlines offering the route&lt;/p&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SELECT &lt;br&gt;
    origin, &lt;br&gt;
    destination, &lt;br&gt;
    count(*) as total_flights, &lt;br&gt;
    count(distinct op_unique_carrier) as uniq_airlines&lt;br&gt;
FROM flights.parquet_snappy_data&lt;br&gt;
GROUP BY origin, destination&lt;br&gt;
ORDER BY total_flights DESC&lt;br&gt;
LIMIT 1&lt;br&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h5&gt;
&lt;br&gt;
  &lt;br&gt;
  &lt;br&gt;
  Query #2: Best airport to fly out from to avoid any major delay (&amp;gt; 30 minutes)&lt;br&gt;
&lt;/h5&gt;

&lt;p&gt;&lt;strong&gt;Spoiler Alert&lt;/strong&gt;: Honolulu International Airport has less than 4% flight delayed, giving you one more reason to visit Hawaii. Newark Airport had an average of 16.2% flights delayed (&amp;gt;30 minutes). &lt;/p&gt;


&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SELECT&lt;br&gt;
    origin,&lt;br&gt;
    percent_delayed as "% Delayed (&amp;gt; 30)",&lt;br&gt;
    total_flights as "Total Flights"&lt;br&gt;
FROM (&lt;br&gt;
    SELECT&lt;br&gt;
        origin,&lt;br&gt;
        DENSE_RANK() OVER (ORDER BY percent_delayed desc) AS worst_rank,&lt;br&gt;
        DENSE_RANK() OVER (ORDER BY percent_delayed asc) AS top_rank,&lt;br&gt;
        percent_delayed,&lt;br&gt;
        total_flights&lt;br&gt;
    FROM (&lt;br&gt;
        SELECT&lt;br&gt;
            origin,&lt;br&gt;
            sum(CAST (dep_delay &amp;gt; 30 as DOUBLE)) / count(&lt;em&gt;) as percent_delayed,&lt;br&gt;
            count(&lt;/em&gt;) as total_flights&lt;br&gt;
        FROM flights.parquet_snappy_data&lt;br&gt;
        GROUP BY 1&lt;br&gt;
        HAVING COUNT(*) &amp;gt; 10000&lt;br&gt;
    ) as t1&lt;br&gt;
) as t2&lt;br&gt;
WHERE top_rank &amp;lt;= 1&lt;br&gt;
OR worst_rank &amp;lt;= 3&lt;br&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;h4&gt;
&lt;br&gt;
  &lt;br&gt;
  &lt;br&gt;
  Comparing performances&lt;br&gt;
&lt;/h4&gt;

&lt;p&gt;For this category, I ran each query 10 times and averaged the numbers. We will use the &lt;em&gt;Raw&lt;/em&gt; dataset as our baseline. &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Query #1&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Raw&lt;/em&gt;: 665.71 MB scanned in 2.60 seconds. $0.00333 per query &lt;/li&gt;
&lt;li&gt;
&lt;em&gt;GZip&lt;/em&gt;: 125.54 MB scanned in 2.08 seconds. $0.00063 per query (-81% savings).&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Parquet&lt;/em&gt;: 8.29 MB scanned in 1.26 seconds. $0.000050 per query (&lt;strong&gt;-98% savings&lt;/strong&gt;).&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Query #2&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;Raw&lt;/em&gt;: 665.71 MB scanned in 2.60 seconds. $0.00333 per query.&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;GZip&lt;/em&gt;: 125.54 MB scanned in 2.08 seconds. $0.00063 per query (-81% savings).&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;Parquet&lt;/em&gt;: 8.29 MB scanned in 0.81 seconds. $0.000062 per query (&lt;strong&gt;-98% savings&lt;/strong&gt;).&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Conclusions
&lt;/h4&gt;

&lt;p&gt;Thanks to Parquet’s columnar format, Athena is only reading the columns that are needed from the query. This reduces the query time by more than &lt;strong&gt;50+%&lt;/strong&gt; and reduces the query price by &lt;strong&gt;98%&lt;/strong&gt;. &lt;/p&gt;

&lt;p&gt;&lt;em&gt;If you have any questions about Athena, Parquet or CloudForecast, feel free to ping me via email &lt;a href="mailto:francois@cloudforecast.io"&gt;francois@cloudforecast.io&lt;/a&gt; or by Twitter: &lt;a href="https://twitter.com/francoislagier"&gt;@francoislagier&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;


</description>
      <category>aws</category>
      <category>apacheparquet</category>
      <category>athena</category>
    </item>
    <item>
      <title>How we saved over $1,000 by building CloudForecast.io with Serverless and AWS Lambda.</title>
      <dc:creator>Francois LAGIER</dc:creator>
      <pubDate>Wed, 03 Oct 2018 15:58:58 +0000</pubDate>
      <link>https://forem.com/cloudforecast/how-we-saved-over-1000-by-building-cloudforecastio-with-serverless-and-aws-lambda-2nn8</link>
      <guid>https://forem.com/cloudforecast/how-we-saved-over-1000-by-building-cloudforecastio-with-serverless-and-aws-lambda-2nn8</guid>
      <description>&lt;p&gt;In a previous blog &lt;a href="https://medium.com/cloudforecast/benefit-from-what-we-learned-spending-2-million-annually-on-aws-f747e6a3bbb3"&gt;post&lt;/a&gt;, we described our journey to launch &lt;a href="https://www.cloudforecast.io/?utm_source=medium&amp;amp;utm_medium=blog&amp;amp;utm_campaign=serverless"&gt;CloudForecast&lt;br&gt;
&lt;/a&gt; and some of the questions every startup face. In this post, we will answer the question “What’s the right way to build this product with our resources?”&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The “$100 Startup” concept&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I’ve been obsessed with a book called “$100 Startup” by Chris Guillebeau for over a year now. In his book, Chris advises his readers to get their product out fast while being cost-effective to limit the damage if your idea fails (&lt;em&gt;definitely possible, unfortunately!&lt;/em&gt;). We try to apply this advice on every side of our business including our technical decisions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Our goals/requirements&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;With the $100 startup concept in mind, we started by listing our requirements that were important to us while building CloudForecast:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ship it out fast&lt;/strong&gt;: We were excited to build CloudForecast to help companies save $$$ on AWS. We wanted to put it in front of our clients ASAP to keep our excitement going. Spending time debugging config, deploy process, etc. will not make the end product better. We were looking for an out of the box solution that we can grow with and allowed us to focus on our MVP the right away.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost effective&lt;/strong&gt;: As an early startup, we wanted to avoid fixed costs by building a system that will scale seamlessly based on our client base.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;AWS Lambda + Serverless = Easy + Focus + Cost Effective&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;With our next steps, we brainstormed on how can we achieve our goals and requirements. We discussed managing our own instances, using containers and half a dozen other ideas but we wanted something simple so we decided to use a Serverless solution (or FaaS). While there are few downsides (e.g cold starts, …), we believe going Serverless better suited our use case (zero administration, pay-per-execution with no idle cost and auto-scaling).&lt;/p&gt;

&lt;p&gt;For the dev setup and deploy process, we decided to use &lt;a href="https://serverless.com/"&gt;Serverless Framework&lt;/a&gt; with AWS Lambdas for the following reasons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;We can focus on writing our product and let Serverless manage the rest (permission, event management)&lt;/li&gt;
&lt;li&gt;Easy to configure and easy to deploy&lt;/li&gt;
&lt;li&gt;Serverless supports multiple platforms (GCP, AWS, … ) which could prevent headaches in the future. We picked AWS Lambdas to start with since our clients data will be stored in S3. It was a logical decision in order to reduce the reduce the network cost&lt;/li&gt;
&lt;li&gt;Great documentation and community (Github, Gitter, Slack)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Our original requirements were fairly simple: We needed 4 AWS lambda functions each on their own cron job, and each lambda would need to talk to various AWS products (RDS, DynamoDb, SQS et al.). All that with an easy way to manage multiple environments (dev vs prod) easily with an easy/effective way to manage resource permissions.&lt;/p&gt;

&lt;p&gt;Here is how we do it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;We used 4 &lt;a href="https://serverless.com/framework/docs/providers/aws/guide/functions/"&gt;functions&lt;/a&gt; with &lt;a href="https://serverless.com/framework/docs/providers/aws/guide/variables#reference-variables-in-other-files"&gt;external JSON files&lt;/a&gt; variables based on the environment:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--kTbRVJDy--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://cdn-images-1.medium.com/max/1600/0%2AE58m7w8y2BYeDgV9" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--kTbRVJDy--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://cdn-images-1.medium.com/max/1600/0%2AE58m7w8y2BYeDgV9" alt=""&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Used the &lt;a href="https://serverless.com/framework/docs/providers/aws/guide/iam/#custom-iam-roles"&gt;iamRoleStatements&lt;/a&gt; to configure all the permissions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--hDuonfSd--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://cdn-images-1.medium.com/max/1600/0%2Ax0fjcElVTbIeVz_3" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--hDuonfSd--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://cdn-images-1.medium.com/max/1600/0%2Ax0fjcElVTbIeVz_3" alt=""&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With these two code snippets, we were able to set up most of our architecture. 4 functions that will interact with each other through SNS. Two functions will run check on a cron schedule (via the ‘schedule.rate’ params) to check a file need to be reprocessed and trigger another function via an SNS. This configuration will be able to scale effortlessly while keeping our cost under control. We are able to fully silo-ed our environments using iamRoleStatements to configure the permissions. We used an external variable to manage environment variables from a simple YAML file.&lt;/p&gt;

&lt;p&gt;We considered running a couple of small instances to do a job which would cost us at least $1k for the year. Meanwhile, the AWS Lambda cost is meaningless since we are only running tens of thousands of functions a day.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What’s Next?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Using Serverless helped us get &lt;a href="https://www.cloudforecast.io/?utm_source=medium&amp;amp;utm_medium=blog&amp;amp;utm_campaign=serverless"&gt;CloudForecast.io&lt;/a&gt; off the ground quickly. We saved our first clients thousands of dollars by giving them visibility into their AWS spend.&lt;/p&gt;

&lt;p&gt;We are in the process of integrating AWS Athena into CloudForecast for our next project. Integrating with AWS Athena will enable new features that will allow end users to dive deeper into their AWS spend. Every company has unique spending patterns, and this will allow them to fully understand where their $$$ is going. We will write more about our Athena integration soon.&lt;/p&gt;

&lt;p&gt;If you have any questions related to this post or what we do at &lt;a href="https://www.cloudforecast.io/?utm_source=medium&amp;amp;utm_medium=blog&amp;amp;utm_campaign=serverless"&gt;CloudForecast.io&lt;/a&gt;, feel free to reach out to me at &lt;a href="//mailto:francois@cloudforecast.io"&gt;francois@cloudforecast.io&lt;/a&gt;. We would love to hear from you!&lt;/p&gt;

&lt;p&gt;-Francois &amp;amp; the CloudForecast Team&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Special Thanks to &lt;a href="https://dev.to/kylegalbraith"&gt;Kyle Galbraith&lt;/a&gt; for reviewing this draft. Check out his weekly Newsletter, &lt;a href="https://www.kylegalbraith.com/learn-by-doing/"&gt;Learn By Doing&lt;/a&gt;. Learn by Doing is a free weekly curated cloud, blockchain, and coding newsletter delivered to your inbox every week.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>startup</category>
      <category>lambda</category>
      <category>cloud</category>
    </item>
  </channel>
</rss>
