Puppet validator Jenkins plugin

For the last couple of months I’ve mainly been working on automating our product delivery process. From building artifacts with Maven to packaging them as RPM together with generated Puppet modules for configuration, provisioning virtual machines through The Foreman the whole chain is now almost covered.

I delivered a couple of Maven and Jenkins plugins along the way and the Puppet Validator Jenkins Plugin is a simplified version of one of the Jenkins plugins I wrote. It does was its name says it does: the plugin scans the project workspace and validates the Puppet classes it finds against the Puppet parser tool (post-build step).

Pretty convenient when you don’t want to push broken configuration artifacts to whatever repository (Maven, The Pulp, …) they’re supposed to end up in. The source code is available on GitHub.

Enjoy.

How to make good teams great, part two

This is the second part of How to make good teams great. First part is over there.

Feed your brain

In a rapidly changing environment - as the IT world is - you want to stay on top of things and you can’t only rely on techniques, languages, frameworks & technologies you played around with at school or in your current projects.

You must feed your brain with new ideas, new cool stuff (even old cool stuff by the way) and there are various places you can get brain food: conferences (if you or your company can afford it), bar camps & user groups (often free), brown bag sessions over lunch at the office, …

Say “well done”

Give your co-workers credit for a well done job. It does not necessarily need to be for big accomplishments only and you should also communicate on smaller achievements. Kudos can be given via an email sent to the Kudos mailing list or can take the form of a small present you know will be appreciated (a chocolate bar for instance).

Keep in mind that not everyone likes to be a star (even for a minute) and it is important that one gets kudos only if she/he wants.

In my team we use one of those EASY buttons (that says “that was easy” when you press it) to tell the floor someone has achieved something (small or big). It is not exactly the same as saying “well done” but still is a nice way of showing some pride.

Eat your own dog food

As a developer you should already be familiar with the quick feedback that a suite of unit tests provides but that is still pretty “low level”. Feedback is important but quick feedback is even more important since it helps you catch bugs early and fix them early (at a lower cost).

So what about testing the software as a whole? What about using it yourself? Eating your own dog food is all about using the software you develop instead of simply developing, shipping the product to the testers and waiting for feedback. By doing so you also see your software with your end users’ eyes (may it be a tester - internal - or a customer - external) and have a better understanding of their needs.

Experimentation time - motivation through innovation

By giving the developers the opportunity - read time - to explore and implement new ideas a company is making an investment that may or may not pay back. But a lot of products were born during experimentation time (FedEx days at Atlassian, Google Fridays, …) and if a company can afford to set a couple of hours aside for their employees to think out of the box it should go for it.

Do what you want time

The do what you want time differs from experimentation time in the sense that it is more about assigning some time (like 20%) for the developers to improve or implement a feature not yet in the backlog in an existing project.

You can find the slides from Sven’s presentation at Jfokus here.

Book circle at work, my own little story

If there was only one thing I wanted to take with me when I changed job in May 2011 it definitely was the book circles a colleague of mine was organizing twice a year. I simply love books and everything that come with them: knowledge, stories, smells, bookshelves, …

But as I started to set such an activity up at my new work place I realized it was much easier to just wait for an email announcing the upcoming titles, pick one up and then attend the sessions than actually make it happen.

As of today we are about to finish our second book and the third one is on the line. Time to reflect on my own little experience I suppose and share my dos and don’ts.

Work your plan

When presenting the idea to your manager you must come up with a strong case in order to justify both the time and the financial investments of the activity and you must show your manager you already have worked the why, when and how questions.

If knowledge sharing, skills development, team building activities are already in your company’s goals towards employees satisfaction things are going to be easier. If not those are three benefits from running a book circle you can for sure sell to any open minded manager.

One step at a time

In the best of world you would probably want to offer all employees in the company a large list of books to choose from and have everyone participate in the discussions. But that simply does not work out of the box.

Start small with 3 book titles and your teammates (8 persons is a pretty good number: you are pretty much sure you can run an interesting discussion even when a couple of your colleagues are home ill) and get things rolling with a couple of books, from choosing the title, ordering the books, reading them and discussing them, before targeting a bigger audience.

Take a step back

The biggest mistake I made (in my opinion) when I started up with the idea was to choose a book that I thought was a perfect match for a bunch of developers: Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages from the Pragmatic Bookshelf.

But it turned out not to be a very good choice for a first circle. It was hard to get the discussion start and flow and even though the topics looked familiar and in line with a developer’s job I must admit discussing Erlang when no one in the room has experience with the language is a challenging tasks. I will probably bring the book back in a circle in the future when everyone - including myself - feels more comfortable with the ideas behind the book circle and has a better understanding of why we are doing it for and how.

Pragmatic Thinking and Learning: Refactor Your Wetware

We gave up after 4 languages and had a fresh start with Pragmatic Thinking and Learning: Refactor Your Wetware (also from the Pragmatic Bookshelf). From the first session (discussing chapters 2 and 3) we could already feel it was a better choice and it only got better chapter after chapter with nice and constructive discussions going on.

Set the rules

For the first circles you need to come up with a complete solution. You need to define a couple of book titles (3 is a good number), organize a vote, order a copy of the chosen book for everyone, work the reading schedule out and book sessions accordingly. You also need to act as a moderator during the discussions and have a set of points and topics you can use to get the conversation rolling.

Once everyone is familiar with the activity you can start involving others in the logistic, in the moderator role, … In the end you should be able to run several sessions in parallel on the same or different books. That’s what I am aiming for at least.

Current setup

  • Reading Pragmatic Thinking and Learning: Refactor Your Wetware, one chapter a week
  • Discussion every second Friday over lunch (the company buys wraps or sandwiches)
  • One hour meeting starting with lunch and chitchat (10-15 minutes) and then discussion on the chapters
  • Participants (developers & manager) are adding interesting book titles to the wish list on the Wiki page I created for the activity

To be continued…

How to make good teams great, part one

As we are implementing new development processes in the department I work for teams are going through a lot of changes. Attending Sven Peters’ (@svenpet) talk on 7 things: how to make good teams great at Jfokus was an obvious decision and I’m glad I went. It was really nice to hear that some of the actions we are taking right now have proven to be successful and appreciated by teams in other companies.

I’m not going to write about the 7 things at once but will break them down in two posts. Let’s start with It’s flow time and Report robot.

It’s flow time

Ever heard of context switching? It is probably one of the biggest problem of so called knowledge workers who - according to various studies - waste 20% to 30% of their time switching from one context to another. And that problem is even more important in an open space where one can easily be disturbed: a colleague or a manager stopping by your desk for a quick question (notice the quick there: sounds familiar?), someone just walking by, …

Unfortunately there is no quick question or task and you will always waste those 20% to 30% of your time (that’s almost 3 hours of a normal day at the office) switching context.

Do Not Disturb!

In such work environments it is therefore all about finding the right balance between productivity and collaboration. Collaboration is the idea behind open offices but productivity is not and you need to build virtual fences in order to secure high productivity periods - as if you were working in a closed office - over the day or week.

  • do not disturb time - together with the team members agree on a day (or half a day) when it is not allowed for an outsider to break the flow. The team needs to communicate when it’s alright to disturb and when it’s not. It could be done with a red flag on the desks or with the team members all wearing the same and explicit t-shirt for instance. All you need is basically a visual clue that people can easily interpret.
  • support guy of the day - one of the team member is responsible for isolating the team from the outside world and handles all the questions, meetings and so on. It is a good idea to rotate among the team members so it’s not always the same person being disturbed.

Andy Hunt dedicates a whole chapter on the subject - Manage Focus - in Pragmatic Thinking and Learning: Refactor Your Wetware and suggests team to establish rules of engagement to manage interruptions.

Report robot

Constant feedback is one of the key factor of successful agile teams and there are a great deal of tools that can help feeding the team with relevant information.

In your day to day activities you probably use a wiki to share documentation and knowledge. You probably (and you better do) use a bug tracking system to manage defects in the code your write. You also might use a build system to automate builds and deployments.

When you think about it the list is much longer than that and it is almost impossible for a single person to keep track of all the data the different tools provide. If your company (read manager) understands the issue and is willing to invest some money in it maybe it’s time to build information radiators!

Information radiators are central locations (a web page, a flat screen TV close to the team) where the feedback from all those tools can be presented in a concise and visual manner: charts, green and red lights, numbers, … for anything that adds value to the team productivity and spirit.

An overview of the NoSQL world

A few sessions at the Disruptive Code 2010 were dedicated to the “NoSQL solutions” trendy topic and I was really looking forward —I have to admit— to what Adam Skogman from SpringSource and Eric Evans from Rackspace had to say on the subject.

Let's back up the Internet

In the last 5 years the amount of data produced worldwide (texts, images, audio, …) has drastically increased from 161 exabytes to 988 exabytes—one EB being one million TB or one billion GB—and with that come some new challenges (storage capacity, availability, …) that cannot be entirely solved by SQL solutions and relational databases.

Not entirely because “NoSQL” does not mean no SQL at all but Not Only SQL really and a mixed architecture is probably what the final solution to a problem will look like depending on the needs.

What are the problems?

The amount of data is growing at an exponential rate and a relational database (like MySQL) is not really a distributed solution. Even though reads can be performed on slaves, writes most likely have to be done on a master—where would the consistency be otherwise?—which becomes a bottleneck in a transaction intensive system.

Relational models are rather statics and once the model has been defined you better not have to change it. From my personal experience I can say that adding a column to a table that gets 10 millions (roughly) extra rows a day without downtime is a rather complex and costly (just in the human resources involved) operation. And when you need to keep 5 years of data in that same table (which would be 18.3 billions rows) and still have a good performance on reads and writes, it gets even more complex.

When you need to store, as fast as possible, large quantities of data which structure has to be somehow flexible you’ll definitely have to have a look at the following solutions.

What are the solutions?

There are 4 kinds of NoSQL models at the moment:

  • key/value
  • column
  • document
  • graph

Key/value stores design is domain driven. Entities that are tightly coupled go to the same bucket (a customer and her shopping cart for instance) but different instances (customer A and customer B) don’t have to be in the same bucket. With a key/value store like Redis a throughput of 110,000 database operations per second can be achieved when MySQL shows a good 15,000—good because it’s still pretty good.

Most key/value stores provide an indexing mechanism and/or a search engine usually based on Lucene, Solr or Elastic Search.

As for the other solutions we can mention Apache Cassandra from Facebook, Google Bigtable and Hadoop HBase (column models), CouchDB and MongoDB (document model) and Neo4J (graph database).

Graph models seems like a pretty interesting topic—even though performance is not what you can expect from a key/value store for instance—and I will definitely have a closer look at Neo4J.

High performance websites with ads [aftonbladet.se]

Today was the first day of the disruptive code conference in Stockholm and I’m taking notes, a lot of notes. I unfortunately don’t have time to blog live from the event - too busy listening to some really good talks - so I am taking another approach: for the next couple of days, I’ll write about the sessions I have been attending.

Let’s get started then with notes I took during a session ran by Tobias Järlund CTO at AftonBladet, the biggest Swedish newspaper (so he claims at least) : high performance websites, with ads (don’t let third parties make you slow) which was based on AftonBladet’s experience and experiments on website, embedding third party ads, optimization.

Facts and figures

www.aftonbladet.se lately (due to the elections in Sweden) had all time high traffic with 2.4 millions visitors a day on its front page which, as it has always been, is really, really, … really long and contains no less than 20 advertising spots.

Web performance optimization, iframes vs. JavaScript

“The server side is not the bottleneck” and, according to Tobias, optimization should rather be performed on the client side.

There are plenty of solutions available (iframes and/or JavaScript based) and AftonBladet decided, after evaluating the pros and cons of some of them, to go for friendly iframes that have the following “properties” (but not only):

  • may load in parallel
  • possible to late-load
  • full access to the DOM
  • support all types of ads

The setup

  1. create an iframe to a small static cacheable HTML page from the same domain
  2. inside the iframe, use the JavaScript variable inDapIF = true to tell the ad it is loaded inside a friendly iframe
  3. inside the iframe, create a script element with the ad url as the source (load the ad just like a normal JavaScript ad)
  4. when the ad has finished loading, resize the iframe to fit the dimensions of the ad

Even if “using iframes is the best option today” to solve that kind of problems, be careful when implementing such solutions since rendering iframes is quite an expensive operation.

Tweaks

  • don’t load all ads at once, use queues or delays
  • set default sizes (1 by 1 pixel for instance) and then resize
  • try not to block the “onload” event (the page loading speed is now taken into consideration by Google for ranking)

The future

  • <frag> tag (Google Chrome) to mark a piece of code that should be loaded in parallel, not blocking (not implemented)
  • postMessage(…) implemented in all modern browsers for messaging between iframes and the main page

Results

It’s certainly pretty cool when you get the chance to try out solutions and experiment different techniques but it’s even better when you measure the impact of the changes you make on your users’ experience!

By implementing friendly iframes Aftonbladet drastically improved the loading speed of the front page (editorial content loaded at the same time or even before the ads) and readers liked it…

  • visits per week up 7%
  • page views per visit up 16%
  • “latest news” section (which used to show up really late in the rendering process) CTR (click through rate) up 35%
  • average page load time down 30%

… and the most remarkable thing being that all those changes have not had any impact on the ads CTR

Install libstdc++.so.5 on Ubuntu 10.04 64-bit

I recently got a new laptop and, as often in that case, had to install a fresh Ubuntu 10.04 64-bit (to replace a Windows XP) and all the software and tools I need as a developer.

One of the software I had to install was the Tibco Enterprise Messaging Service (an implementation of the Java Message Service, JMS). The installation went fine (I had to deactivate all the visual effects in order to see the content of the dialog boxes though - under System > Preferences > Appearance) but I had some troubles starting EMS.

/opt/tibco/ems/5.0/bin/tibemsd

/opt/tibco/ems/5.0/bin/tibemsd: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

EMS requires libstdc++.so.5 but only libstdc++.so.6 is nowadays bundled with Ubuntu. After some Googling I finally came across a solution that worked.

wget http://security.ubuntu.com/ubuntu/pool/universe/i/ia32-libs/ia32-libs_2.7ubuntu6.1_amd64.deb
dpkg-deb -x ia32-libs_2.7ubuntu6.1_amd64.deb ia32-libs
sudo cp ia32-libs/usr/lib32/libstdc++.so.5.0.7 /usr/lib32/
cd /usr/lib32
sudo ln -s libstdc++.so.5.0.7 libstdc++.so.5

If you have another solution that you’d like to share, please leave a comment!

Recursively search files for a string with 'find'

When looking for a particular string, manually going through hundreds of files is out of the question! One single Unix command does it for you.

find . -iname '*.*' -exec grep -nH 'Where the hell is my string?' {} \;