After that, I’ll try to go back to school, doing a Master in Computer Science (because I have no choices in fact) in either the EPFL or the Universities of Bern, Neuchâtel and Fribourg. I see this as being an exit door to what I’m currently doing for a living and an opening door to a new world of opportunities which must not be related to the everywhere Internet.
Many people of my family or friends seem to not follow my moves and not understanding and seeing the meaning behind them. I can tell you, there is nothing particular I’m looking for except the thing I know the most, myself.
Going back to the Alp has a strong meaning, the work there is repetitive. You have to milk, do the cheese every single day no matter how you are, what the weather is or if something went wrong. You have to do it, working in rythm with the animals, the nature, the weather, everything surrounding you. It directs, inhibits my eager of discoveries, knowledge, questions, novelties, forcing me to focus on what matters. And what matters isn’t related to myself, it’s what matter for the whole, the bigger stuff I’m part of.
To be honest with you, working as a Freelance is a way to not be part of anything.
Now the studies, what’s hidden behind this. I’m only describing the idea I may have now which won’t be what it’s gonna bring while doing it. It’s again about going back to something, somewhere. Being humble enough to say, I don’t know shit and I need to be told because I’m ready to listen and learn, instead of pretending I know what I, in fact, ignore. Also, if we learned a lot at the engineering school the focus was really on practice and my theroritical basics as been missed. Sometimes I can feel that lack and it’s a gap that can be filed.
I had the opportunity to give some lessons at an engineering school and it might be something I could do in the future once I’ve learned how to be nice and to answer kindly to stupid and repetitive questions. They are days where I feel more like Linus than Richard when it comes to explaining something very simple to smart people. We need both of them but not everyone likes when you hurt their feelings.
And the most important thing, maybe is because I can. I don’t know how hard it will be but I know I can try without worring about money or people that would be affected by this. (Put here a little “forever alone” rage comic head)
You are who you are, nothing you do, own or make define you.
]]>Let’s build a web server (aka HTTP server).
#!/usr/bin/env php
<?php
$socket = stream_socket_server(
'tcp://0.0.0.0:8000', $errno, $errstr
);
if (!$socket) {
echo "$errstr ($errno)", PHP_EOL;
} else {
while ($conn = stream_socket_accept($socket, -1)) {
$request = '';
while (substr($request, -4) !== "\r\n\r\n") {
$request .= fread($conn, 1024);
}
$response = <<<EOS
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12
Hello World!
EOS;
fwrite($conn, $response);
fclose($conn);
}
}
This is pretty much it. As PHP isn’t multi-threaded maybe the libevent library can help
but I didn’t manage to install it. Or you could fork it
using the pcntl_fork way. It actually works, badly but works.
Let’s create a WSGI for PHP. Maybe not.
I’m now wondering who would use this for real production stuff. It’s not close from what exists in the other web servers like Unicorn, Gunicorn, … even if they can leverage threads, native code and existing libraries more easily I guess. Not an expert though.
The code is on Github: pure PHP web servers (non-concurrent and concurrent)
]]>
So you check the basics like GET and POST arguments. And maybe you check the values coming for the cookies as well. But do you check the file uploading? You know that type arguments sent. This how a request look like:
POST /… … Content-Type: multipart/form-data; boundary=boundary --boundary Content-Disposition: form-data; name="key" value --boundary Content-Disposition: form-data; name="file"; filename="image.png" Content-Type: image/png …Your PNG image here… --boundary--
The Content-Type field is guessed by the browser and most of the time on the file extension. But it can decide to send whatever it wants like application/octet-stream which just says that it’s a binary file.
import magic
from webob import Request
def application(environ, start_response):
req = Request(environ, charset="utf-8")
# sent from the browser
uploaded_file = req.POST.get("file")
# what we actually got
guessed_type = uploaded_file.type
magic_type = magic.from_buffer(uploaded_file.read(1024),
mime=True)
uploaded_file.seek(0)
"… whatever else …"
This way you can really white list the file you users are uploading and prevent them for sending you .mov, .doc or whatever when they think they are uploading a new avatar for their profile. True story.
]]>At the end this operation will happen once, but during the current development we might be running them a lot while adding new data sets to it.
Let’s dive, on how we import the data into the new database:
LOAD DATA INFILE "/tmp/data.dat" INTO TABLE table_name;
That’s very fast, and can be faster if you allow more space for the InnoDB logs.
Now, where does that tmp/data.dat file come from what does it look like. This file has been generated by another request ran on the old database.
SELECT id, name, some_value FROM old_table INTO OUTFILE "/tmp/data.dat";
This is a trivial one, recreating a new table with three columns. By default the output format is a tab separated file where NULL has been replaced by \N.
If your old data base — like it’s our case — is still in Latin 1 or 15 (in Europe), using iconv on the .dat file can convert it to, for example, utf-8.
iconv \
-f iso-8859-15 \
-t utf-8 \
/tmp/data.dat > /tmp/data_utf8.dat
mv /tmp/data{_utf8,}.datWe got tricked by some cp-1252 characters mixed with the Latin 15 ones, beware of those.
And to finish this, two tipps if your old data are too messed up.
You are using text fields in the old data but will use a dedicated table now one so you should use a temporary table for this kind of mapping, i.e. country.
You cannot get all the data at once and need to compute stuff for some rows. In our case moving the pictures to a dedicated table and linking them back afterwards. So we load the data that need to be updated into a temporary table and then perform on update SQL statement to update them all at once, they work like natural joins.
]]>First things to know is that pages are owned by users not applications and you can only interact with facebook as an application on the behalf of a user. First thing first, you need to create an application.
To act on a Page from an application, the administrator of a Page your application want to administrate must give you the permission to manage his pages. This permission is called, no surprises here: manage_pages.
Once the permission is granted, an access_token can be retrieved from https://graph.facebook.com/me/accounts and used from the application to interact with the page.
Nowadays Facebook applications should be made using mostly the JavaScript SDK to login, logout and so forth. That said, ours gonna be static only as we don’t need more.
Okay, now you have an application and a page with its access_token. Let’s have fun with the Graph API. Frankly, there isn’t much I can say against the Graph API. It’s a nice example of REST API build upon a very complex data schema.
Let’s submit something.
<?php
$fb = new Facebook(array(
appId => null,
secret => null
));
$message = 'Hello World!';
$access_token = /* token for the page */;
$fb->api(
'/pageNameOrId/feed',
'POST',
compact('access_token', 'message')
);This a simple message, you can make it way more complex. The message will appear as being posted by your application, so pick a cool name.
]]>I like them because they made that framework so much flexible in a way you can plug anything at almost any places or add — what they call — filters to certain key methods or functions to tweak their behaviour at your needs.
Let’s start with the mentioned filters. A filter will wrap the body of a (static) function or method and let you act before / after the initial work, very much like monkeypatching.
<?php
namespace monkey;
class Math extends \lithium\core\StaticObject {
static function add($a, $b) {
$params = compact('a', 'b');
return static::_filter(__FUNCTION__, $params, function($self, $params) {
extract($params);
return $a + $b;
});
}
}
Math::add(2, 2); // 4As you can see we are using PHP 5.3 features like namespace, closures and late static binding.
monkey namespace.static:: (which is close to self::) but will always be the class where it’s written and not any sub classes (that’s what self:: is for)Just in case you didn’t know, PHP 5.2 is not supported anymore. At least before its next release. Jamais deux sans trois. (never two times without a third time).
Now, let’s hack this.
<?php
use lithium\util\collection\Filters;
Filters::apply('\monkey\Math', 'add', function ($self, $params, $chain) {
extract($params);
$ret = $chain->next($self, $params, $chain);
fwrite(STDERR, "$a + $b = $ret");
return $ret;
});
use monkey\Math;
Math::add(1, 2); // 3
// stderr: 1 + 2 = 3
For those who don’t know, stderr is the error output on UNIX. You can pipe / redirect it to another location using 2>.
Inside lithium this kind of filter are mostly used in the Dispatcher. The Dispatcher is the class that handles all the requests, matching a route with an existing controller. So you can do exactly what you would do with WSGI/Rack/… middlewares.
Methods can aslo be written in a way you can filter them afterwards.
<?php
namespace monkey;
class Tweeter extends \lithium\core\Object {
function tweet($message) {
$params = compact('message');
return $this->_filter(__METHOD__, $params, function ($self, $params) {
echo 'tweet: ', $params['message'], "\n";
});
}
}
$t = new Tweeter();
$t->tweet(’Hello’); // Hello
$t->applyFilter('tweet', function ($self, $params, $chain) {
$params['message'] .= ' ^YB';
return $chain->next($self, $params, $chain);
});
$t->tweet('Hello') // Hello ^YBPardon my examples to be simplistic. A great usage of all this are tests. By performing surgerical modification inside your objects, it becomes very easy to tweak everything without bloating the initial object much. I agree that filterable methods or functions aren’t as readable as normal ones.
And because on major thing you do in Object Oriented Programming is to link objects, instances together here comes the last pattern, kind of, you may find into Lithium. I dunno exactly if it’s Inversion of Control or Dependencies Injection. I’ve stayed away from Java a too long time to know.
<?php
namespace monkey;
class MyObject extends \lithium\core\Object {
protected $_classes = array(
'math' => '\monkey\Math'
);
protected $_autoConfig = array('classes');
public function plus1($n) {
$math = $this->_classes['math'];
return $math::add($n, 1);
}
}
$obj = new MyOject(array('classes' => array(
'math' => '\ape\Math'
)));
$obj->plus1(2); // 3I always feel silly having such useless example but I hope that you get the point. MyObject isn’t linked to a particular Math class but can use a different one if needed with no code modifications.
By the way, Symfony2 uses something similar with a _class suffix in the options.
From a JavaScript, Ruby or even Python point of view, this is the poor’s man monkeypatching and it involves a really heavy syntax for few gains. As I’m still complaining for every array I have to type, this is no big deal.
Don’t forget to join #li3 on freenode.
]]>One year ago, I was leaving Eboutic.ch and started being my own boss. Being your own boss means nothing much as you’ll still be working for someone. That someone will just change more often that it was before.
I really never asked me this question before: “Why would someone hire you for?” What I would be doing there was more clear to my mind but the why wasn’t. And the situation is mostly unpleasant.
If you’re doing well and have no problems in either deadline or staffing, then why would you hire a freelancer? No reasons. Obviously, it starts with a messed up situation. Brilliant!
Your clients are in trouble and they need someone to just fix it. To just fix it, nothing more because the urge of a deadline or an already exploded budget (that your salary will blow up even more) doesn’t allow more time. Time is cruxial, always.
Cruxial because, quite often, they need you right on. It is also true with companies but they don’t care much if you don’t work on first day, or week, or maybe month from my experience.
Let’s recap, you’ll join a project that might be late, with no more budget and with no controls of what have been done and no way to have an impact on it because you’re not supposed to do so.
I didn’t sign for this, really!
I might be lucky with the actual project where I’ve got the opportunity to learn a lot on human relationships and project managment, being put in the center of those. And also to teach some stuff, to set up good practices and tools. It’s hard and painful as well but at least here is somewhere I can put my guts in.
There is always something to learn out of anything and being freelance gives me the opportunity to do that more often.
]]>One of my goals going there was: to let go. I let everything aside: work, flat, computer, loves and worries. None of these define who you are.
My work usually doesn't require any particular rythm. I can start when I'm ready, stop when I'm bored (or it's finished), eat whenever I'm hungry. Dealing with cows that have to be milked twice a day, milk, cheese fabrication, weather and whatever occurs was a real shift. It's also a daily routine where 80% of what you do has to be done that day at a particular time. This repeatition was far from being boring, surprisingly. You start to master some operations, getting better at them, getting faster, parallelizing what can be. At the beginning everything was very intense in how much attention was required and as the time goes none of them remained as something hard to do. Even cleaning the shit out of where the 43 cows were resting during the day was something that became easy.
What about now? I don't know yet, I'll get back on tracks and catch up with projects I was involved in and try to write about stuff I'm doing to gain visibility. Ain't it right?
As a conclusion, this was a huge experience like many others I did in the past. Surprisingly not the one that created the biggest shock of all. I start learning to go through changes it seems.
]]>No more air conditioning, wheel chairs, dual screens, coffee break, cantine, daily standup, weekly meeting, headset, skype calls or who’s-the-hell-broke-the-build mornings. It’d be great to get that body to finally do something (gym doesn’t count and I’m not really into it) and let that full of not-that-important-stuff brain relax a bit.
Via sab.ch, I found some farmers looking for a hand this summer, applied and accepted. This means I’ll know how to milk cows and make cheese by this automn, will let any computer-related activities aside for a while and concentrate on a work that is more meaningful, at least seems to.
I’ll be up there, send me a postcard.
This ends mid-October and I don’t know yet what I’ll do then. Freelancing as I do since December is the logical way but as I know me, I don’t know what’s ahead and it’s a good feeling.
]]>Let’s take that simple piece of PHP code :
<?php $req = "http://pipes.yahoo.com/pipes/pipe.run?_id=…&_render=php"; $data = unserialize(file_get_contents($req));
It grabs some external content and might take some time to achieve this depending of Yahoo! Pipes. This is where the problem starts to itch. When the server gonna call that page it’ll do mostly nothing but waiting for the external data to arrive.
Using Apache, a process or a thread is spawned, waiting on external data and, more importantly, consuming memory and CPU by just waiting.
Going asynchronous would mean being able to do other stuff, answer to other requests while waiting.
Unfortunately, I don’t know any solutions for that problem in PHP. PHP is designed to be executed as fast as possible and not for doing heavy operations. Big websites that are using PHP use it in general as pure front-end language (for example, the before-Google YouTube).
They are a couple of solutions to go async. Some come from a specific language that is asynchronous by design, like JavaScript (with node.js) or Erlang (to name a few). Ruby can achieve this by using EventMachine and Python (which I explored more) has a couple of solutions (Twisted, Tornado (coming from FriendFeed, bought by Facebook), Eventlet, Gevent, …). Take this dummy WSGI code:
from time import sleep
def application(environ, start_response):
start_response("200 OK",
[("Content-Type", "text/plain")])
sleep(1)
return ["Hello, world!"]
The external stuff you’re waiting on is simulated by a sleep, which makes it easy to understand. Up to you to fetch external data that will take a fixed time to arrive (which is the same).
When you run this particular code, a process/thread will be dedicated for that during one second blocking other incoming ones, right? You’ll need as many workers/threads/processes as requests to handle them all in one second. Suboptimal since those particular processes aren’t doing anything special.
Take a look at some tries I made.
Eventlet and Gevent, the more convenient to use imho, will run each incoming requests into a coroutine (like a sub process) enabling the main process to do something else when some time is available. The only thing to change in the code above is where the sleep function comes from.
from eventlet import sleep from gevent import sleep
And, of course, to run it using the appropriate server (the one from the library, Gunicorn or Spawning (Eventlet only)). Those libraries are doing monkeypatching so no code has to be modified in order to become asynchronous. I.e. using urllib.urlopen to fetch external data will not block the whole process. No code changes are required.
Twisted or Tornado, for example, are using a reactor model, which forces you to handle asynchronous code with callbacks. It’s closer to the metal but the learning curve might gives you some headaches. I do love Twisted, but it’s sometimes just too much, really.
Python has another project, running on top of Python Stackless, called Syncless dedicated to async code. It looks quite promising.
Ruby 1.9 will get coroutines under the name of Fibers, until that EventMachine seems the way to go. It looks like super clean Twisted too me. I hope not hurting too much feeling by saying that. (BTW, there is a great article from the SuperFeedr guys: Ruby Fibers may confuse)
One very interesting project, and very young as well, is node.js. Basically server-side JavaScript. This one is asynchronous by design and totally awesome if you love JavaScript or coming out of hell if you don’t like it. I do like it.
With the web evolving the way it does, where data is coming from multiple sources, heavy or special tasks are delegated to specialized units (SOA), I clearly see this asynchronous idea as an ongoing paradigm shift. Today’s bottleneck is the I/O, most often represented by the database.
Don’t forget that the backend performances aim only to serve more people, faster but 80-90 % of the time is spent on the client side. I’m a frontend guy after all.
]]>