Release process - from startup to professional

The release process is one of the most underestimated processes in a growing company. Most startups choose between two versions of release management: none or fully automatic. The latter one mostly fits ruby/rails driven startups due the availablity of capistrano or git based hosting companies. The "none" variant means, in our case, deployment via svn checkouts on the server(s). You could say this is release management too, but if you are honest to yourself ...

Startups tend to grow and so you will encounter some point in time when you need to scale the processes within the startup. The release process is one of them, as you get more servers, more developers and more features per development cycle (I tend not to say sprint or release cycle for getting the agile and classically managed in one boat). In addition to the management complexity the software complexity will grow with every feature you do. Phrases like rollback, dependency management and multi-environment come up in tech meetings and the "management by in-flight magazine" comes up with continous deployment to lower the time to market dramatically (sorry, no link or reference for the anti-pattern - the original article seems to be gone, but I guess you can imagine what it means? The CEO or Product Developer reads something in a magazin while traveling and enters the tech department to announce the immediate resource allocation on the new hot topic XYZ without any preperation or even sense). I tend to call this DDD (dream driven development) resulting in RDD (rage driven development) on the tech side.

When you come from a web agency (like me) and start designing a new process for such a company you might tend to rely on a classic release management process. Starting from the way how feature tasks are assigned up to the moment where you build and deploy the piece of software on the server cluster. I have been proven wrong since I stated the attempt to build such a process. The new goal should be to mix up the two worlds, aiming for maximum flexibility within a given level of process security. Cutting it into smaller pieces to take the changes step by step makes it easier for the developers and for the management.

I'll do another post about choosing a software stack to support the release management in a few days.

Indexing octopress article in elasticsearch

I was not satisfied with the simple search plugin that came with octopress (although it is suitable for many blogs), so I started looking for a simple search solution without any branding and with all the features I wanted. I came across Solr, but it was oversized for my concerns. I did not want to create a complex scheme and push the documents with brackground jobs or something similar. This is where elasticsearch came into play. The very dynamic nature fitted very well and the easy to use http interface was exactly what I needed.

I've hacked a simple indexer plugin and a simple javascript file to query the index and display the results in a badly styled overlay (I'm no CSS guy at all).

Here comes the plugin, which is a simple jekyll generator. I does not check for anything, but if you are able to blog with octopress you should be able to cope with the sources and customize them to fit your needs. You'll need to add those two config entries for your index url (which should be protected by http-basic auth) and your index name.

The javascript is also dead simple. You'll need a jsonp request if your elasticsearch server is hosted on another domain or port (which is very likely). The script is nothing crazy. Most of the code is rather bad, using jQuery selectors directly and not using any scope related stuff. The code might work better as a jQuery plugin or something more reusable. Anyway, it works for me as a first step. Feel free to enhance it and drop me a line over twitter so I can link to your solution.

bind function missing in iOS 5 JavaScript

While developing a new, jQuery based, frontend, we encountered some strange behavior while testing a release of the app on the iPad 2 running iOS 5.1.1. Between a bunch of commits the iPad's JavaScript just stopped to do anything. Not a single event was fired, nothing happend.

The debugging tools on the iPad are not that great and so was the debugging process. We found out, that the Konquerer Browser uses the same engine as iOS and the bug was reproducable in Konqueror, too. Git bisect came into play and we found the commit that killed the iPad. We changed the modenizr full build to a custom slim build. This kicked out the "Touch Event" part of the build and this was the reason for all the trouble. The "Touch Event" part defines the bind() function if it is missing in the JavaScript engine. We borrowed the function from modenizr and made the iPad work again.

Sidenote: iOS 6 beta 4, which has a new JavaScript Engine, works without any problems.

Hetzner vServer on ArchLinux (german)

This post is written in german language, as most people who are interested in it are customers of the german hosting provider "Hetzner".

Lange Suche

Ich bin erst seit Kurzem eine Fan von ArchLinux. Vorher war ich bei SuSE, Fedora, Debian, Ubuntu, Gentoo (Stage 1) und PC-BSD. Das letzte Ubuntu Release, 12.04, hat mich enttäuscht und dazu getrieben mir eine echte Alternative zu suchen. Ich war vorher viel im Gentoo und BSD Umfeld unterwegs und mochte den Ansatz der vollkommenen Anpassbarkeit sehr. Der Schritt zu ArchLinux war damit nicht so weit entfernt. Doch nun begann die Suche erst wirklich. Einen Hosting Provider, am liebsten mit deutschem Standort, und vertretbaren Preisen war nicht so schnell zu finden. AWS, Linode und Rackspace bieten ArchLinux direkt an, jedoch sind alle mit 19,99 USD für 512Mbyte RAM vServer mehr einfach viel zu teuer und wenig attraktiv.

Leider bieten viele deutsche Hosting Provider nur Virtuozzo-basierte Lösungen an, so z. B. Strato und Hosteurope. Die Shared-Kernel Architektur verbietet es dann irgendwas Anderes als die angebotenen Distributionen zu verwenden.

Hetzner to the rescue

Hetzner war der erste (und soweit ich weiss auch einzige) Anbieter mit KVM/QEMU Architektur. Zusätzlich bietet Hetzner seinen Kunden an, den Server im sog. Rescue System auszuliefern. Damit bekommt man ein schlankes Debian gebootet, von dem man seinen Server wunderbar bearbeiten kann.

Leider stimmen die Tutorials auf https://wiki.archlinux.de/title/Arch_Linux_auf_einem_Root-Server und http://www.uninformativ.de/?ndo=single&newsid=138 nicht mehr zu 100%, so dass ich meine Erfahrung hier noch mal schnell zusammen schreibe.

Der Arch Linux Wiki Artikel arbeitet mit einem Software Raid, was meiner Meinung nach für einen vServer wenig Sinn macht, und einer pacman basierten Installation. Hetzner verfügt jedoch über eine Web-KVM Konsole, die es uns erlaubt eine normale Installation durchzuführen. Die Installationsanweisung auf uninformativ geht eben diesen Weg.

Das uninformativ Tutorial kann man bis zur Installation von grub gut durcharbeiten. Leider ist der klassische grub unter dem aktuellen Rescue System von Hetzner nicht mehr verfügbar. Da die aktuelle netinstall Version von Arch keine alte Version von grub mehr supportet und die Version 0.9 nun nur noch im AUR verfügbar ist, sollte man hier besser auf syslinux setzen oder sich mit der manuellen Einrichung von grub2 anfreunden.

Der Rest läuft wie im Artikel von uninformativ beschrieben.

Stateful web applications pt.2

This is a followup of my first article about stateful web applications, which you can find here, you may want to read it first.

Push state - poll state

We are still talking about stateful web applications. In my previous article I've talked about a challenge - response method to keep client and server side in sync. This method had some drawbacks but was simple to achieve.

The push state - poll state method is more complex concerning the setup, but way simple when it comes to the development part. Instead of using a single, serial channel of communication, we break up things and use two separate channels for our needs. This simple change gives us the possibility to acquire more change state requests as we could to with the sequential version (meaning the challenge - response method).

The setup

You will need something that accepts a high number of concurrent request. I would not recommend an Apache httpd for that. I've tried node.js and it worked. (nginx might fit as well, depending on your software stack.) This server, let's call it "change-bucket", gets all the change state requests from all clients via PUT request. It responds with some 2xx state, maybe 204 or 201. This is just the info, that the put-request reached the server. As you might imagine, this might lead to an eventual consistency state. The change-bucket passed the change request to a shared queue, which could be a redis server or something like ApacheMQ, RabbitMQ, or similar. The queue server maintains one list per application (of which multiple per user can exists side by side). By the way: redis is awsome for such a job, because it supports list operations natively - even blocking ones - and produces less overhead than a real message queue. The last thing you need is something like memcache. You can use redis or a real memcache server - depending on your setup. The key is: it must be fast and at least persistent as long as it has a power supply.

Now, that we have all change requests of the user in a queue, we need to get them out and change the server-side application state and reflect those changes to the client. This is also a point in this setup, where you can decide which way to go.

We will now switch perspectives and take a view from the client side now. This helps us to identifiy the point where those two ways will come together again.

The client

The client need a poll mechanism. You know, something like setTimeout() with a propper tail recursion. I would not recommend using call(), as it is of lower priority in the most JavaScript Engines. The intervall depends on your technology stack, but I recommend to make it possible to set it to 250ms or less. This enables you to fake a real-time feeling on the customer's side. This poll request must go somewhere and it needs to receive something. What does it receive? The full application state, where does it go? Now we reached the other side of our two way split.

Path 1 - blocking list operations

This way is of moderate complexity and lacks of scalable performance. You will need only one additional piece for your architecture puzzle. You need an endpoint for the client to ask for the latest application state and you will need to make this endpoint fetch the stack of change requests from redis. This fetch operation must be blocking to prevent a concurrency issue. The concurrency issue comes from requesting the same endpoint twice. The first requests has not persisted the new application state yet and the second request loads the old state from the memcache server. The last one who persists the state wins. In this case we would get a not recoverable, inconsistent state. Another problem: you will need to solve the unsuccessful put-requests in the change-bucket with e.g. a retry loop in the client.

Path2 - everything non-blocking

In this setup you will need another, daemon like instance to solve the problem of concurrency. We need to decouple the polling process from persisting the new state. We could do this by a cronjob or a node.js server or something completly different. The important aspect is, that this cronjob just takes data from the queue and merges it into the new state. The polling mechanism may see the same state twice, but that is OK as far as we got rid of our concurrency issues.

Conclusion

We've build a circle, a round-trip for our application state. You might now have the feeling, that this is a shit load of overhead regarding the task of "just" persisting an application state, but that is OK. It is a rather complex setup with it's own issues and I must admit, that it works only in theory at the moment.

Future

One of the big advantages of this setup is the possibility to scale it. If the polling mechnism reached it's scale, you can easily exchange the polling with persistent socket connections. If you do so, you might not even need a single change-bucket, but you could use the same socket to do both parts of the communication over a single architectual piece.

Stateful web applications pt.1

Preface

In modern web application we mostly use two sided setups. One is a client - which will be a browser in this case - the other is some sort of server with some sort of application. The languages used on the server side do not matter at all, but the client side language will be JavaScript.

It is all about state

The application has a state. This might mean some sort of dataset which the client works on or something which is brought to the client by the server and is fully or partly persisted to enable the client further requests in the same data context. This topic is highly connected to caching those datasets and working on those cached results rather than working on live datasets.

HTTP is stateless, which means that we must rely on other techniques to synchronize the server and the client state. You might call for (web-)sockets now, but not all companies are able to cope with a change of their infrastructure to use sockets for the number of users that are concurrently browsing the app.

Challenge - Response

One way of getting this done is a challenge - response communication protocoll between the server and the client. The client always adds a token to identifiy itself on the server side. In addition, when the client sends a request it adds a ticket id to it. The server processes the response either in realtime or asynchronous. When doing it in realtime, the server responds directly to the client request. The async request must trigger polling mechanism where the server answers at some time with the requested ticket id. This type of communication works well, as long as your requests do not depend on each other. If you need state changes in a certain order, you need to add some kind of counter to it. The server needs to keep up with the counter and cannot process counter number 5 before counter number 4 was processed successfully.

This might lead to a deadlock situation, where counter 4 is lost in space for some reason and the server got stuck at holding back anything above counter 4.

Another drawback might be the danger to slip into some sort of interpretation of the client state. The server gives you a bunch of values to set some of the states that were asked for in the last response. You might be forced to start guessing the state of former components based on the partial response.

Too abstract? Ok, here is an example. You open some sort of info box and it's content must be loaded from a server via an ajax call. The response comes back and you display your text. This box is one of three in a left column UI element. The UI concept says, that not more than one box should be open at the same time. OK, what happens next is common sense at the moment, but from my point of view it is a great danger. The client decides, that, on opening another box, the actual box has to be closed. This state never reaches the server. The client interpreted, that he must close it, because another box want be gain the state "open".

Lets get one step further. We have another, dependent view element that also requires a server request. The answer to that request could possibly open one of the three boxes. As the server's answer enforces the client to do that, the actual state of the client (which is unknown to the server) gets lost. The client need to interpred the result again and concludes, that the active open box must be closed to achieve the requirements of the former server response.

Where is the drawback now, you might ask. Try to let the client send a server request based on data that cannot be interpreted correctly be the client:

• You have a entity to show via an id.


• You need to show meta data based on the id.


• You have a default setting on the root level of your application ("/")


• You do not want to show the meta-data to the default id, but another one

If you just deliver the id, the client is not able to decide if it's default or not. The amount of "please client: if default do that, if not do this" raises and will kill you/your app some day.

The next article will be about using a "Push / Poll State Sync" approach.

Building plugins for PHPStorm sucks - somehow

've started building my own plugins for Symfony2 on PHPStorm a while ago and yes, they make awesome IDEs.
If you ever wondered why there are so few plugins - compared to Eclipse - you might want to have a look at the available documentation for developing plugins in IntelliJ Idea based products. Sorry guys, but this is crap. There is no real API documentation anywhere but in the source code offered in form of the IntelliJ Community Edition. Yes, this might lead you to the assumption, that most of it is usable as a good starting point. Nope.

You might know the concept of extension points from Eclipse. It is an awesome concept and does also find it's place in Jetbrains' IDEs. The lack of documentation, which extension point is intended to be used for what kind of feature makes it nearly impossible to get started without asking rather "dumb" querstions in their forums.

I've tried to create a "Clickable Routes" plugin for supporting Symfony2 developers. It should enable you to click on a route string whithin a twig template's path() call. I've tried to use the same PsiReferenceContributor stuff which I successfully used before for my "Clickable Views" plugin. Unfortunatly twig is a template language and therefore handled differently in PHPStorm. The tokens of the template are not delivered to the contributor's extension point. I had to find this out myself as my post in Jetbrains' devnet was left unanswered by their developers.

This means I just can provide clickable routes in PHP files, but not in twig files. This sucks.

fish - my new favorite shell

Thanks to @hochchristoph, I found my new default shell for my ubuntu systems. It's "fish". You can find it here: http://ridiculousfish.com/shell/.

And for the git users using __git_ps1... Here is my ~/.config/fish/config.fish, based on the info I found here.

PHPStorm and Symfony2

I've been working with Symfony 2 a lot these days and I saw the wonderful work, which Robert did with his Eclipse plugins. I've started to write a bunch of plugins for PHPStorm, which - I hope so - will ease the way of developing Symfony 2 apps in PHPStorm.

You can find them here: http://xenji.github.com/phpstorm-symfony2-plugin/

What is so bad about ... PHP

or web technology? I hear so many people ranting these days. About PHP, about MongoDB and so many other things. I got bored about it, as anything said about it is not new. PHP has been the way it is for over a decade, so why do you fuck with it ... right now? It's the same thing with Java, but Java has prooven itself as an, so called, enterprise technology, which lifts it's status above criticism? PHP has been there, most likely before ruby got famous and before things like django came up to the status at which they reside now. All of them are good in solving problems we never had before they came up. We built websites without MVC, without Doctrine, Symfony2, Rails or ActiveRecord or Document Databases. Don't get me wrong, I like the NoSQL movement! But as it says: "not only" - means for me "in both directions". The thing is, that we've reached a scale, which needs fresh ideas. Rails was a good one, django and Symfony2, too. Esp. Symfony2 showed the PHP world how you can use PHP in an enterprise manner. All you people ranting about PHP, look at this framework, tell me if it does not solve the same kind of problem, which you solve with Rails or whatever tool you use! What you like or dislike is not important for solving a problem. Use the right tool for your problem. These words are old, maybe older than me. They still count. By the way: PHP moved to github. That is your chance to contribute and change the things you do not like - instead of just complaining about them.

GitLab - a private github-like solution

I'm always looking for new solutions in the private code and project hosting category. I recently stumbled upon GitLab, which seems to me like an awesome piece of software. The feature set feel a bit like github, which is very cool. The installation is a bit tricky, as you need to set up a full ruby on rails stack. The requirements talk about ruby 1.9.2 or greater, which demands a custom build on Debian 6 (which my private server run on) or the use of rvm. rvm made some trouble when I tried to use it so I switched to the custom build. The rest of the installation manual runs like a charm, really just copy and paste.

I took the nginx version using the unix socket. The only glitch in the manual is the name of the unix socket in the upstream {} block. This is set to /tmp/gitlab.socket, but it means /home/gitlab/gitlab/tmp/gitlab.socket.


One of the key features, from my point of view, is the permission system. It is not very fine grained, but educated enough to visualize e.g. customer-client relationships or help you to manage your university courses when you give your students some programming homework.


To get an impression of the UI and the features you can take a look at their homepage and their demo. The project's code is hosted on github. 

Xubuntu 12.04 on my Sony Vaio VPCS13V9E

This is my installation report about installing the new Xubuntu 12.04 beta 2 on my Sony Vaio VPCS13V9E/B. This is the fourth attempt to install a different linux distribution on my Vaio. Before Xubuntu 12.04 came Linux Mint 12, which was a total failure, and Fedora Core 16. The SELinux seemed contra-productive for a developer notebook to me. Ubuntu 10.10 and 11.04 were total failures, too. The EDID, needed for the graphics, were not present and had to be extracted on a windows machine. The text-based installation was the only way to cope with the problem.

Now comes (X)ubuntu 12.04, my personal highlight so far. I've used the default installation image, downloaded from the official website. I decided to use a classic CD, the attempt to use an usb key instead failed because it just did not boot from it.

The installation went fine and without problems, even the graphics were good to go. After the reboot I found a system with well recognized hardware. WLAN, Bluetooth, network, sound, touchpad with multitouch (two-finger-scroll) and nvidia graphics were all good. Even the HDMI port and the sound over HDMI worked properly from the start.

The problems left:

• UMTS - The Qualcom Gobi 2000 is always a problem
• Keyboard backlight
• Adaptive screen brightness
• Screen brightness controls

The Gobi 2000 problem is easy to solve, as long as you have the windows drivers at hand. The gobi card has to be initialized with the proper firmware for your carrier (mine is O2 germany, which I will use as example). If you have a windows dual boot, just copy the folder C:\Program Files (x86)\QUALCOMM to an usb key. Otherwise download the firmware from the Sony support website and install it via wine. In the QUALCOMM folder you'll find a folder named "Images", which contains  numbered subfolders (0-12) and a folder named UMTS. Now you have to find out which number is correct for your service provider (Mine is 6, I just google'd it and found it on the first result page). Create the folder /lib/firmware/gobi if it does not exist and copy the content of the UMTS folder and the content of the folder with you corresponding provider number to /lib/firmware/gobi.
The next step is to install the gobi-loader, which enables the system to load the firmware into your UMTS card. Use "apt-get install gobi-loader" to install it. Afterwards we need to make sure, that the kernel loads the modules on boot, which we need to get the dialup connection ready. Do a "sudo vim /etc/modules" and add two new lines "qcserial" and "usb_wwan" at the end of the file. Save it and leave vim. To test it without rebooting your notebook, put the hardware switch to it's off state and do a "sudo modprobe qcserial" and "sudo modprobe usb_wwan". Then turn it back on and type "dmesg" in your shell. You should see the QUALCOMM lines in the output. Go to your network settings. You should now be able to create a new mobile broadband connection according to your carrier settings. For O2 germany you need to discard the settings that are in the preset and enter "internet" as APN.
Please note that it will always take some time for the gobi loader to get the firmware into the card's memory. This means, that it will take 10-15 seconds for the connection to appear in the network manager.

The keyboard backlight is eady to solve, too. Just edit your /etc/rc.local and add this line somewhere before the "exit 0" line:
echo 1 | sudo tee /sys/devices/platform/sony-laptop/kbd_backlight
This will enable the backlight. If you want to turn it off for any reason just change the "echo 1" to "echo 0".


The screen brightness stuff is still without a real solution. I'v read articles on the web that enable the brightness controls via the fn-shortcuts, but they are made of a bunch of custom scripts, which I did not check yet. The adaptive brightness has no solution known to me. If you find one, please leave a comment.

Follow up (April, 8th 19:32):
I've found a similar article dealing with another VAIO notebook on an older Ubuntu version (9.04). Unfortunately the hint regarding nvclock does not work on newer nvidia cards - they are not supported yet.

git bash prompt

There are a few styles flying around the internet and I just wanted to contribute mine. It is based on the one I've found here: http://mediadoneright.com/content/ultimate-git-ps1-bash-prompt - with some minor modifications:

• The prompt's color scheme is made for white letters on black background
• The path is shortened
• Beside you branch the short sha1 of your HEAD is displayed
• Replaced the time with user@host