Aaron Patterson is one of the most respected and appreciated developers in Ruby world. Member of Ruby and Ruby on Rails core Team he is known for his great professional skills and even greater personality. Our CEO Valentine Zavadskiy and Dmitry Zhlobo from Datarockets have met with Aaron at Ruby Conf to ask him all sorts of questions from future of Ruby to his favorite barbecue recipe.
– Aaron, you are currently working at Github. Could you please tell us what you do in Github with your experience and background.
– Sure. I work for Github and pretty much my recent project has been working on Ruby’s garbage collector. I’m on a System Platform Team and we do low-level stuff. But my focus recently has mostly been on Ruby GC. That’s what I have been doing there. Basically hacking on that.
– Less Ruby and more C?
– More C than Ruby, yeah. I mean, I still have to take care of Ruby stuff, you know. But I’d say lately most of my day I have been doing C coding. Unfortunately.
– That is why you probably learn Japanese, right?
– Yeah. Actually, when I first started learning Ruby, maybe 2006 I think, and googled for code examples, I found this blog post, which was all in Japanese. I could see the code, but I couldn’t read anything. I wanted to know what they are saying, which is why I started learning Japanese. So it’s been 11 years and I can finally read a blog post. I can read that blog post, it just doesn’t matter anymore.
– Was it good? Was it worth it?
– No. I spent 11 years to realize it was no good.
– I actually have a general question: how did you get into programming? Because you are obviously doing some C right now and you were doing Ruby. So what is the story behind it?
– I started getting into programming when I was in High School. Well, actually, I guess before that. We had programming classes. Well, not really classes, we just had a computer in a classroom in elementary school. And I’d play with that. So it was like my first exposure, but I really started doing programming in High School, I guess. Actually even younger. I had my very first computer in 1988 and I the very first program I wrote was in Basic programming. I had a book and in the back of the book I had a really long program about how to predict the weather. Like you take data and try to predict what it will be in future. Like basic machine learning. So I wrote this program. I typed it in, it was maybe six pages long and it didn’t work. And I checked everything and I typed it exactly as in the book and it didn’t work at all. And I was like “This is stupid. This is not going anywhere.” So that was my first exposure.
– Let’s talk about GitHub. Is it public, the version that Rails are running?
– Yes, it is public. Right now we are running Rails 3.2 in production, we are in the middle of upgrading to 4 and eventually getting on to edge. So I assume we will be up to 4.2 fairly soon within the next couple months or so. I think we have like 10 failing tests left. After so we are very close.
– Just remove them!
– Yeah, exactly, just remove them and we are good.
– Would you call Github still Rails Application?
– Oh yeah, for sure. It’s a huge Rails application. I mean we have a bunch of systems around that too, but the main web application is definitely Rails app, for sure. We have a monolith, but we also have services too. We don’t run Git on our web-server. We have a backend server and we do communication between web-server and that. So it’s between monolith and microservices. it’s more like macroservices. It’s kind of weird, we call it monolith, but, I guess strategically, we have certain thing extracted from the application. But it is definitely Rails app for sure.
– And generally are you fine keep it monolith?
– There have been a lot of arguments at work about whether we should keep it as a monolith or not. Like the main issue we are having around it or the argument for going to microservices is the speed of deployment. The way we work is: when somebody fixes a bug, they send a pull request and somebody reviews it, they say “yes” or “no”, then that person has to deploy the code to production. So we have so many engineers that we are deploying every ten minutes or so at work. So the problem is that we have so many engineers that and even waiting 10 minutes is a way too long. You have to get in line and queue up to deploy your changes and just that amount of time multiplied by a number of engineers is too much time. So one solution we are thinking is if we had smaller services, then it’s like “OK, I need to fix a bug, but that bug is in this one, so I can go do that”. And a line for deploying that app is smaller than it is for some other one.
– Do you think is it possible to rewrite it to microservices? How hard would it be?
– I don’t know. Don’t think we would rewrite it. We were just saying “let’s extract certain things”. Like, for example, authentication code. Maybe move that off to a particular service. Something like that. Take a look at the code and see where we can slice it up and put it into different services.
– Are there any conversation about moving from Ruby and Rails to something like Java?
– No, I don’t think so. I mean we have a lot of C code, C++ code. I mean we have a lot of code at work. But most of it is Ruby. And I don’t know how we’d move away from that, from the main application. In fact one of our backend services, some of our legacy code actually used to be in Erlang server, but it turned out we couldn’t actually scale that well. So we went from Erlang to Ruby and now it’s a Ruby server. So the weird thing is that our main web-server has to do communications with this other server and the protocol between them is in Erlang. But now they are both written in Ruby, they are both Ruby applications doing protocol in Erlang. So it is a little bit crazy.
– I know you like a barbeque.
– Yes.
– Even you have own company buying meat.
– Yes, I do.
– Could you share with us your favorite barbeque recipe?
– I love making bacon. It’s super good. Basically just garlic, bay leaves, brown sugar, salt and you just take a pork belly and you put in all the stuff, keep it in a fridge for a couple of days and you smoke it for maybe three hours or so. And then you have bacon from that. It’s so good.
– Would you like to have a special hashtag for trying this out? Like #aaronbacon? Summertime is coming, some people will cook it and let you know.
– Yeah, that’s great.
– Let’s get back to technical questions. Rack 2. We need to be prepared, it’s coming. Do you need some help with this?
– Sure, let’s talk about HTTP/2. I’m gonna speak about HTTP/2 at Rails Conf I think. The problem is that I think it is actually kind of depressing, what I’m gonna say. I’ve been looking into upgrading stuff to HTTP/2, but it turns out that if you use a proxy server, you can have the proxy server do most of it. You don’t need to change your Rails app on the backend. You can just run HTTP/2 in front of it. The main thing you are missing out of is doing push. However, speaking with guy named Kazuho, he created H2O, I don’t know whether you know this web-server. It’s the best HTTP/2 server. It’s super fast and it is a really good web-server. He actually works for Fastly now, so they pay him to work on this web server. Fastly uses H2O for their stuff. Anyway, he came up with an idea that he proposed to make a standard and what it is, it is a new response code and I think it is 101 or 110. It is called an early response. So it is a normal HTTP/1 response code, but it is essentially the same thing as the push, an HTTP/2 push. So what you do you have a proxy and it understand this response code and it basically takes that and translates it into HTTP/2 push for you. So the advantage is you can have your app server here running H1, speaking to a proxy server speaking HTTP/2 and it understands that you want to push stuff. So you can have essentially the best from both worlds. You can run your H1 server without breaking anything and still be able to do pushes out. The other cool thing is that H2O will do a fingerprinting of your assets, so if you say I want to push out some CSS, you don’t want to push it out to that person every single time, right? They may have it. They may actually have it already and you are wasting. So H2O can take that evolve for you. So you don’t have to maintain that state in your application. So what I’m think for, as far as Rack is concerned, is that what we need to do is we need to have a way to send multiple responses from one request. But just be H1 responses. So we don’t need to break anybody’s applications or anything. We can keep the responses the same, essentially keep the API the same, but get 90% of HTTP/2 support.
– But this would like process using proxy server, right?
– Yes, exactly. That a huge downside that we have to have a proxy server in front of, but the upside is that we don’t have to break everybody’s application in order to get the support.
– Do you have any data how many people are using proxy server these days? 95%?
– I think everybody is using proxy servers these days, except for the main people who are on Heroku. So that’s the problem, but Heroku runs its own proxy. You app is actually behind the proxy. So I have been talking to Heroku engineers and I basically said to them: “Look, what you should do is change your proxy to HTTP/2 and then have your proxy support this early response”. And now all apps that run on Heroku can get HTTP/2 support for free. So I think one of their main problems is that they terminate as SSL-proxy, so that the app doesn’t know that it is running SSL. But well, actually it doesn’t matter. But if the proxy understands this early response code, it doesn’t need to know that it is a cell.
– Is there any chance that this response code will become a part of specification?
– Definitely, yeah. Kazuho has already made a IETF proposal, so it just needs to through committee review. But it is definitely becoming a specification. What I think will be even more cool is that if browser started supporting it too. So you could say like “Hey, browser, here is something, go grab that early.” Then you can have HTTP/1 server doing similar stuff to HTTP/2, but anyway, it’s exciting.
– But is Rack ready for sending multiple requests?
– I’ve been thinking about this. We could do it, but essentially it would be just a huge hack. So it’s ready just to be hacky code, all we need to do really is to take the hash put another lambda and then you call out lambda. That’s it. Annoying, but we will see.
– Otherwise we have to break all the apps.
– Exactly. That is what my main concern is. I was thinking about this, we can have a way better API, but it is like breaking so many people’s applications. I don’t want to do that. Nobody will be happy about that, right? Nobody will want to upgrade.
– Nobody would like to have another version of Ruby that breaks API.
– I don’t want to have Python 3.
– Let’s talk about Rails security team. What is your responsibility in Rails security team?
– Well, my responsibility: I read tickets and fix bugs and write stuff. I mean, everybody on the Security Team basically does the same thing. We all share the responsibilities. We just take in tickets, fix them and make releases.
– How did you get to it?
– Well, nobody wants to be on the team. It’s a terrible team to be on. You don’t want to be on a team, but I think what happened was because we had a very large backlog of security issues and I was like “Hey, we need to do a release. We really need to take care of these things, right?” Theт it was like “Congratulations, you are on security team now.” And I’m like “Oh, man!” So that is basically how I got into the team. I was just on a core team already and I accidentally showed interest in the security stuff and then “OK, now you are on the team”. That is what happens. I mean being on the Security Team is not a good thing. You don’t want to do this.
– Do they require you to stay online 24/7?
– Actually, we take all our security reports through Hacker1 and we have been working to expand the size of the Security Team. We used to be only like 3 or 4 people, but now I think we are up to maybe 10. So what we have actually done is Github is giving time to some of their security engineers at work and now they are on security team. They are not on Rails Core Team, but they are on Hacker1. They have access to Hacker1 stuff, so they respond. Like if anybody send in an issue, they are part of the response team and they come up with patches and stuff. But it is up to Rails core team to do the actual releases. But at least we get help from somebody else on the team. So we are able to spread a load out a bit, but no, I’m not required to be online or anything like that.
– And did you ever have a nightmare, when zero-day will revealed and you have to fix that?
– No, not at all. That hasn’t happened. The worst thing that we had was a while back, when we had remote code exploit with YAML. And of course, that thing is going to hurt me every freaking time. All the time it kills me. And the thing is that people misunderstood what the error was and they started thinking “Oh, YAML is unsafe”. What it was is that Rails had code that did eval. And if you sent the right YAML, it would execute eval. So basically YAML decoder uses Ruby, so it’s implemented in Ruby, so if you take a hash from YAML and you create a new hash in Ruby, it allocates a new object and then it calls hash equals on it. So there was an object in Rails that had a hash equals method on it that would do an eval. So people would send a YAML thing and it would create a new one of those and then would do an eval and would run what they wanted to. And then… It’s really hard to explain it people, they just see like YAML RCE. Kind of, but not really. Even now people bring this up once in a while and I’m like “Oh, man!”. That was not fun.
– That reminds me that back 2012 Github was hacked. It wasn’t actually a security issue, but it was like that. It was stripping parameters like you had to pass attr_accessible. And nobody was doing this, because everybody is so lazy. Including Github, yes. It probably was bigger than YAML exposure. I don’t know why. That’s why you probably were hired by Github. They didn’t want to have things like that in future.
– Does your word mean more than DHH’s word related to security. For example there was a bug or maybe more a feature of Rails, where you can send a Javascript template from the server to the client. There was a possibility to steal CSRF token or something like this. I think the first reaction of DHH was like “you shouldn’t just send forms in your Javascript templates.”
– I don’t know. I think maybe he is not into the security problems as much. That’s why we have a security team. To say “no, actually we do need to take care of this”. But what we typically do is a lot of times the security team will explain why it is so bad and then we try to come up with what we actually want that feature to accomplish. And then we figure out how to do that in a secure manner. Unfortunately it means that we may release a feature that is not secure. But I mean in happens in all stuff.
– Web Assembly. It is coming. What languages will we be using for it?
– Oh, I’m excited, we will be running Web Assembly everywhere. I’m going to be running Linux in Firefox. That is going to be my operating system: just Firefox running with Web Assembly. Web Assembly is awesome. I think it’s very cool.
– Is there any conversations? Will it be next big thing in Ruby? We have opal, which translates Ruby to Javascript. Because in Ruby we are doing a lot of Javascript programming, so there should be some opportunity for us to go to frontend.
– I think this will be awesome. I haven’t looked into it that far, but you could totally do it. It would be a fun project. I mean, turning Ruby into any other Assembly. Except this would be running in a browser, which will be nice. I hate writing in Javascript. I’m sorry it is the worst language there really is.
– Who knows, you never write in binary code.
– That’s true. It is probably worse.
– I know that you are playing with Arduino and all this kind of stuff these days. I guess it is because some Ruby Team are running an mruby. Could you tell more about this?
– Yeah, a little bit. I mean, I’ve only played with mruby a little bit, but the problem with mruby and embedded hardware is that Arduino doesn’t have enough memory to store an mruby program. I was looking into this. The chips I’m using at home only have like 14k of memory and you can’t put a memory program in 14k, so unfortunately I can’t use those there, but I think mruby is an interesting thing, especially for command line tools and stuff. It seems to fill in similar spaces as Go, like for command line utilities, but I’m more interested in MRI I think.
– I was using mruby, when I was running Sinatra application and my router was 8mb of RAM. It was exciting; I didn’t expect it would work. I take Sinatra and it was working!
– Wow! That’s awesome.
– OK, my last question. I know that a lot of young people are listening to us. I mean Junior Developers. There was an event yesterday, there were a lot of students there. I know this is a hard question to answer, but I will still ask you: How to become the best developer?
– You need to practice. In my opinion there is no easy way to do it, you just have to practice over and over. Read a lot of books. I can recommend a lot of books depending on how junior you are or where you are in your career. But if you are just learning programming, I love Chris Pine’s book ‘Learning Program Ruby’. But if you are getting more midway into your career I think Michael Feather’s ‘Working Effectively with Legacy Code’ should work. That book is amazing. It changed my programming life. People should get that book. That book is all in Java and C++, but the techniques are all applicable to any particular language. So I think that is the one thing you need to do is not get focused on one specific language. Look at different languages and what you can learn from other languages and bring that to what you are doing at work. “GC Handbook” is good. That is what I have been reading a lot lately, since I’m doing GC stuff. Compilers book too, if you are getting a little more advanced. Those are good books as well. But I think it is most important to just practice. Try challenging yourself, do something new and don’t get bugged down in one particular language. That is what I would recommend to people. Learn an object-oriented language, learn a functional language and learn how to bring the best of those to what you are doing on a daily basis.
– Since we are talking about books. Are there any other books not technical ones, which you would recommend?
– I only read technical books.
– Or probably podcasts or blogs?
– I’m so boring! I actually read cookbooks.
– What is your favorite one?
– I read all the ‘America’s Test Kitchen’ stuff. I read their website, I read all their books. I love those books.
– How about Ruby cookbook?
– Yeah, that is a great cookbook. Typically in my free time I try to read books in Japanese, trying to practice.
– And blog posts, right?
– Yeah and blog posts. Now finally blog posts.
– Thank you for being here. We are really excited to meet you.
– Thank you very much.
(Featured on Awesome Ruby)
2 Comments
Great interview, guys!
Thank you.