Wow, so many conferences lately! Fortunately for me, PuppetConf was local so I didn’t need to catch any flights or deal with hotel hassle, it was just a 2 block walk from home each day.
My focus for this conference was learning more about how people are using code-driven infrastructures similar to ours in the OpenStack Infrastructure project and meeting up with some colleagues, several of whom I’ve only communicated with online. I succeeded on both counts and it ended up being a great conference for me.
There was a keynote by Gene Kim, he is one of the authors of the “devops novel” The Phoenix Project, which I first learned about from my colleague Robert Collins. His talk focused around the book, as The Phoenix Project: Lessons Learned. In spite of having read the book, it was great to hear from Kim on the topic more directly as he talked about technical debt and outlined his 4 top lessons learned:
- The business value of DevOps is higher than we thought.
- DevOps Is As Good For Dev… …As It Is For Ops
- The Need For High-Trust Management (can’t bog people down)
- DevOps is not just for the unicorns… DevOps is for horses, too. (ie – not just for tech stars like Facebook)
The next keynote was by Kate Matsudaira of Popforms who gave a talk titled Trust Me. I wasn’t sure what to expect with this one, but I was pleasantly surprised. She covered some of what one may call “soft skills” in the tech industry, including helping others, supporting your colleagues and in general being a resourceful person who people enjoy working with. Over the years I’ve seen far too much of people assuming these skills aren’t valuable, even as people look around and identify folks with these skills as the colleagues they like working with the most. Huge thanks to Kate for bringing attention to these skills. She also talked a lot about building trust within your organization as it can often be hard for managers to do evaluations of employees who have the freedom to work unobstructed (as we want!) and mechanisms to build that trust, including reporting what you do to your boss and team mates. Slides from her talk available here: Keynote: Trust Me slides
After the keynote I headed over to Evan Scheessele’s talk on Infrastructure-as-Code with Puppet Enterprise in the Cloud. He works in HP’s Printing & Personal Systems division and shared the evolution and use of a code-driven infrastructure on HP Cloud along with Puppet Enterprise. The driving vision in his organization was boiled down to a series of points:
- Infrastructure as “Cattle” not “Pets”
- Modern configuration-management means: Executable Documentation
- “Infrastructure as Code”
- Focus on the production-pattern, and automate it end-to-end
- Everything is consistently reproducible
He also went application-specific, discussing their use of Jenkins, and hiera and puppetdb in PE. It was a great talk and a pleasure to catch up with him afterwards. Slides available here.
My talk was on How to Open Source Your Puppet Configuration and I brought along Monty Taylor and James E. Blair stick puppets I made to demonstrate the rationale of running our infrastructure as an open source project. I walked the audience through some of the benefits of making Puppet configurations public (or at least public within an organization), the importance of licensing and documentation and some steps for splitting up your configuration so it’s understandable and consumable by others. My slides are here.
On Wednesday I attended Gareth Rushgrove’s talk on Continuous Integration for Infrastructure as Code. He skipped over a lot of the more common individual testing mechanisms (puppet-lint, puppet-syntax, rspec-puppet, beaker) and dove into higher level view things like testing of images and containers and test-driven infrastructure (analogous to test-driven development). Through his talk he gave several examples of how this is accomplished, from use of Serverspec, the need to write tests before infrastructure, writing tests to enforce policy and pulling data from PuppetDB to run tests. Slides here.
After lunch I headed over to Chris Hoge’s talk about Understanding OpenStack Deployments with the Puppet modules available. In spite of all my work with OpenStack, I haven’t had a very close look at these modules, so it was nice meeting up with him and Colleen Murphy from the puppet-openstack team. In his talk he walked the audience through some of the basic design decisions of OpenStack and then pulled in examples of how the Puppet modules for OpenStack are used to bring this all together. Slides here.
Two talks I’ll have to catch once the videos are online, Continuous Infrastructure: Modern Puppet for the Jenkins Project – R.Tyler Croy, Jenkins (slides) and Infrastructure as Software – Dustin J. Mitchell, Mozilla, Inc. (slides). Both of these are open source infrastructures that I mentioned during my own talk! I wish I had taken the opportunity while we were all in one spot to meet together, fortunately I was able to chat with R.Tyler Croy prior to my talk, but his talk conflicted with mine and Dustin’s with the OpenStack talk.
In all, this was a very valuable event. I learned some interesting new things about how others are using code-driven infrastructures and I made some great connections.
More photos from PuppetConf here: https://www.flickr.com/photos/pleia2/sets/72157648049231891/
This past week I headed to Florida to present at Fossetcon and thought it would be a great opportunity to do a formal review of a new tool recently released by the OpenStack Infrastructure team (well, mostly James E. Blair): Gertty.
The description of this tool is as follows:
As compared to the web interface, the main advantages are:
- Workflow — the interface is designed to support a workflow similar to reading network news or mail. In particular, it is designed to deal with a large number of review requests across a large number of projects.
- Offline Use — Gertty syncs information about changes in subscribed projects to a local database and local git repos. All review operations are performed against that database and then synced back to Gerrit.
- Speed — user actions modify locally cached content and need not wait for server interaction.
- Convenience — because Gertty downloads all changes to local git repos, a single command instructs it to checkout a change into that repo for detailed examination or testing of larger changes.
For me the two big ones were CLI-based workflow and offline use, I could review patches while on a plane or on terrible hotel wifi!
I highly recommend reading the announcement email to learn more about the features, but to get going here’s a quick rundown for the currently released version 1.0.2:
First, you’ll need to set a password in Gerrit so you can use the REST API. Do that by logging into Gerrit and going to https://review.openstack.org/#/settings/http-password
pip install gertty
wget https://git.openstack.org/cgit/stackforge/gertty/plain/examples/openstack-gertty.yaml -O ~/.gertty.yaml
Edit ~/.gertty.yaml and update anything that says “CHANGEME”
A couple things worthy of note:
- Be aware that by default, uses ~/git/ for the git-root, I had to change this in my ~/.gertty.yaml so it didn’t touch my existing ~/git/ directory.
- You can also run it in a venv, as described on the pypi page.
Now run gertty from your terminal!
When you first load it up, you get a welcome screen with some hints on how to use it, including the all important “press F1 for help”:
Note: I use xfce4-terminal and F1 is bound to terminal help, see the Xfce FAQ to learn how to disable this so you can actually read the Gertty help and don’t have to ask on IRC how to do simple things like I did ;)
As instructed, from here you hit “L” to list projects, this is the page where you can subscribe to them:
You subscribe to projects by pressing “s” and they will show up as bright white, then you can navigate into them to list open reviews:
Go to a review you want to look at and hit enter, bringing up the review screen. This should look very familiar, just text only. I’ve expanded my standard 80×24 terminal window here so you can get a good look at what the full screen looks like:
Navigate down to < Diff > to see the diff. This is pretty cool, instead of showing it on separate pages like the web UI, it shows you a unified page with all of the file diffs, so you just need to scroll through them to see them all:
Finally, you review! Select < Review > back on the main review page and it will pop up a screen that allows you to select your +2, +1, -1, etc and add a message:
Your reviews are synced along with everything else when Gertty knows it’s online and can pull down review updates and upload your changes. At any time you can look at the top right of your screen to see how many pending sync requests it has.
When you want to quit, CTRL-q
I highly recommend giving it a spin. Feel free to ask questions about usage in #openstack-infra and bugs are tracked in Storyboard here: https://storyboard.openstack.org/#!/project/698. The code lives in a stackforge repo at: http://git.openstack.org/cgit/stackforge/gertty
As I wrote in my last post I attended Fossetcon this past weekend. The core of the event kicked off on Friday with a keynote by Iris Gardner on how Diversity Creates Innovation and the work that the CODE2040 organization is doing to help talented minorities succeed in technology. I first heard about this organization back in 2013 at OSCON, so it was great to hear more about their recent successes with their summer Fellows Program. It was also great to hear that their criteria for talent not only included coding skills, but also sought out a passion for engineering and leadership skills.
After a break, I went to see PJ Hagerty give his talk, Meetup Groups: Act Locally – Think Globally. I’ve been running open source related groups for over a decade, so I’ve been in this space for quite a long time and was hoping to get some new tips, PJ didn’t disappoint! He led off with the need to break out of the small “pizza and a presentation by a regular” grind, which is indeed important to growing a group and making people show up. Some of his suggestions for doing this included:
- Seek out students to attend and participate in the group, they can be some of your most motivated attendees and will bring friends
- Seek out experienced programmers (and technologists) not necessarily in your specific field to give more agnostic talks about general programming/tech practices
- Do cross-technology meetups – a PHP and Ruby night! Maybe Linux and BSD?
- Bring in guest speakers from out of town (if they’re close enough, many will come for the price of gas and/or train/bus ticket – I would!)
- Send members to regional conferences… or run your own conference
- Get kids involved
- Host an OpenHack event
I’ll have to see what my co-conspiratorsorganizers at some local groups think of these ideas, it certainly would be fun to spice up some of the groups I regularly attend.
From there I went to MySQL Server Performance Tuning 101 by Ligaya Turmelle. Her talk centered around the fact that MySQL tuning is not simple, but went through a variety of mechanisms to tune it in different ways for specific cases you may run into. Perhaps most useful to me were her tips for gathering usage statistics from MySQL, I was unfamiliar with many of the metrics she pulled out. Very cool stuff.
After lunch and some booth duty, I headed over to Crash Course in Open Source Cloud Computing presented by Mark Hinkle. Now, I work on OpenStack (referred to as the “Boy Band” of cloud infrastructures in the talk – hah!), so my view of the cloud world is certainly influenced by that perspective. It was great to see a whirlwind tour of other and related technologies in the open source ecosystem.
The closing keynote for the day was by Deb Nicholson, Style or substance? Free Software is Totally the 80’s. She gave a bit of a history of free software and speculated as to whether our movement would be characterized by a shallow portrayal of “unconferences and penguin swag” (like 80s neon clothes and extravagance) or how free software communities are changing the world (like groups in the 80s who were really seeking social change or the fall of the Berlin wall). Her hope is that by stepping back and taking a look at our community that perhaps we could shape how our movement is remembered and focus on what is important to our future.
Saturday I had more booth duty with my colleague Yolanda Robla who came in from Spain to do a talk on Continuous integration automation. We were joined by another colleague from HP, Mark Atwood, who dropped by the conference for his talk How to Get One of These Awesome Open Source Jobs – one of my favorites.
The opening keynote on Saturday was Considering the Future of Copyleft by Bradley Kuhn. I always enjoy going to his talks because I’m considerably more optimistic about the health and future of free software, so his strong copyleft stance makes me stop and consider where I truly stand and what that means. He worries that an ecosystem of permissive licenses (like Apache, MIT, BSD) will lead to companies doing the least possible for free software and keeping all their secret sauces secret, diluting the ecosystem and making it less valuable for future consumers of free software since they’ll need the proprietary components. I’m more hopeful than that, particularly as I see real free software folks starting to get jobs in major companies and staying true to their free software roots. Indeed, these days I spend a vast majority of my time working on Apache-licensed software for a large company who pays me to do the work. Slides from his talk are here, I highly recommend having a browse: http://ebb.org/bkuhn/talks/FOSSETCON-2014/copyleft-future.html
After some more boothing, I headed over to Apache Mesos and Aurora, An Operating System For The Datacenter by David Lester. Again, being on the OpenStack bandwagon these past few years I haven’t had a lot of time to explore the ecosystem elsewhere, and I learned that this is some pretty cool stuff! Lester works for Twitter and talked some about how Twitter and other companies in the community are using both the Mesos and Aurora tools to build their efficient, fault tolerant datacenters and how it’s lead to impressive improvements in the reliability of their infrastructures. He also did a really great job explaining the concepts of both, hooray for diagrams. I kind of want to play with them now.
Introduction to The ELK Stack: Elasticsearch, Logstash & Kibana by Aaron Mildenstein was my next stop. We run an ELK stack in the OpenStack Infrastructure, but I’ve not been very involved in the management of that, instead focusing on how we’re using it in elastic-recheck so I hoped this talk would fill in some of the fundamentals for me. It did do that so I was happy with that, but I have to admit that I was pretty disappointed to see demos of plugins that required a paid license.
As the day wound down, I finally had my talk: Code Review for Systems Administrators.
I love giving this talk. I’m really proud of the infrastructure that has been built for OpenStack and it’s one that I’m happy and excited to work with every day – in part because we do things through code review. Even better, my excitement during this presentation seemed contagious, with an audience that seemed really engaged with the topic and impressed. Huge thanks to everyone who came and particularly to those who asked questions and took time to chat with me after. Slides from my talk are available here: fossetcon-code-review-for-sysadmins/
And then we were at the end! The conference wrapped up with a closing keynote on Open Source Is More than Code by Jordan Sissel. I really loved this talk. I’ve known for some time that the logstash community was one of the friendlier ones, with their mantra of “If a newbie has a bad time, it’s a bug.” This talk dove further into that ethos in their community and how it’s impacted how members of the project handle unhappy users. He also talked about improvements made to documentation (both inline in code and formal documentation) and how they’ve tried to “break away from text” some and put more human interaction in their community so people don’t feel so isolated and dehumanized by a text only environment (though I do find this is where I’m personally most comfortable, not everyone feels that way). I hope more projects will look to the logstash community as a good example of how we all can do better, I know I have some work to do when it comes to support.
Thanks again to conference staff for making this event such a fun one, particularly as it was their first year!