ARDUINO SERVERS – OR WHY WERE SLOW
– January 16th, 2012
Hej everyone,
this is a very personal post to explain and send an apology to the community for our very sloppy server response over the last month or so. I will also rise a case for how the website works and for how we are envisioning our newcoming webservices.
I know some of you will appreciate this explanations, since there have been comments in the forum and requests for someone’s toes to be cut (probably mine). It is not that we haven’t been doing anything about it, we just didn’t communicate properly about what we were doing. Mostly because we were too busy getting it done. Therefore I apologize, I am a grown up bearded man with no problems saying I am sorry.
As many of you know, the Arduino project is growing quick. We are not experiencing the exponential growth we saw during 2010, but it respects a somehow lineal growth that makes my hair get a little grayer every time I look at the statistics on how our website is used. We are about to reach 1.000.000 unique visitors per month, adding up to over 53.000.000 hits on the website each month. At our previous server (because we upgraded last week) the 8 cores were 90% of the time at 100% of performance. Since we were renting a VPS server at ServInt, we could go over the use of the 8 cores. In other words, we could easily reach a peak of 9.5 virtual cores in weekdays.
This started to happen during late November 2011. Something unexpected since the end of the year has traditionally been a month when our stats would slow down until March, the time when student projects magically happen at very many education centers around the world. But, somehow, the deals made last year throughout the Arduino distribution network, brought much more traffic to the website and took me a little by surprise.
When I first noticed this issue, I took two decisions:
1) we need to make more periodic backups of the website, to assure we have everything there whenever we migrate the website
2) we need to upgrade the server asap
About the backups
I wrote a script to make daily backups of the website, it will put the DB down for a max of 10m early in the European morning. That is the message that some of our most committed forum participants experience when they go on answering questions at 6AM (man that is commitment).
You guys might come to the website at the time the script is backing up the whole database. In the future we will make this work differently. We are making a whole new backup server that will be making copies of everything while it happens and it will not stop the server of operating as it is now.
You might think … but ServInt is an internationally recognized provider, isn’t it giving you a proper backup tool? Our provider is in fact giving us a great backup system for free, but we want to have our own on top … you never know when it will get handy. We have 7 years of interaction in the forum and website that we don’t want to loose. Our knowledge base (our as in yours and ours) has beenbuilt collaboratively among the Arduino users since 2005 and we have the responsibility of keeping it alive as well as safe so that it doesn’t get lost.
About the upgrade
We have finally moved away from a VPS to our own machine!! This is exciting, the demand of the Arduino community is so high that we have been in the situation where we have to hire two machines in a data center to have the community up and running. It also means that we had to make a migration as we never did before.
I spent over a week discussing with ServInt how to get this done. I got promised they could make it in 30m. I did an upgrade all by myself early 2011 with only 5m downtime, so I thought they were being conservative in their approach. I like when people don’t try to lie to you to get you in, so I believed they could make our upgrade with 30m downtime.
I scheduled it for January 7th 4AM CET, a really good time, since it is the time we have the lowest traffic rate. Servint came in a day earlier informing they had run out of stock on the machine I wanted for us all to use. It took me a couple of hours to react to this and order the next machine in their line (more expensive, but more resources in it) what delayed the upgrade to happen on Sunday the 8th at 4AM CET, the second best time.
Unfortunately, the worst prediction from ServInt wasn’t as bad as reality. Our server was so badly overloaded that it took them over 20h to make the full migration. I assisted to three shift changes on the technicians side and I also witnessed how they accidentally turn off the server off during a 6h interval, when they shouldn’t have done it. This translated in many of you experiencing the server to be down when it shouldn’t. I got a formal apology from their side, we are good customers to them and they treated us nicely. I appreciated their honesty and took for granted they wouldn’t fail a second time. Despite the issues we faced, we have had a really good relationship to this provider over the last 3 years or so, and I am willing to keep it going as long as they keep the good work from now on.
About the Forum being dead slow
The forum makes use of a database. Every request you guys make, specific searches or posts or whatever, translates in a database query that uses CPU time. It is not about harddrive and it is not about memory … it is about CPU. All the requests made by you last the CPU, and the more visitors we get the more CPU time it consumes. Since we are still growing, the consumption of CPU power is our main concern, as everyone likes to interact with the forum. Correct me if I am wrong, but it is a helpful resource, both for looking for advice, but also for recreational purposes … people come, comment, joke, complaint … and we like it to be like that.
So the upgrade was necessary to keep up with the forum’s growth. But if something can go wrong, it might do so … and in this case it did. The forum’s database got a field modified what broke the Personal Messaging feature. At the same time, there were so many files uploaded in the forum’s folders, that we reached a protection quota I established about a year ago … both issues came at the very same time, probably with one day difference, but you experienced it as if it was the same source of problems. Of course, for the server to render that error would also take CPU time, slowing down the server’s reactions to your queries.
Both issues are now fixed and should not bother you for quite some time. The forum has about 100GB in uploads, personal pictures, etc. But the new machine has harddrive enough to handle this.
And now what?
Now, we will keep on working on things as usual. We will move the forum to a machine by itself at some point this year. We will implement database replication to totally cut away those downtimes in the European mornings and we will make sure the backups of the folders are taking as little CPU as possible using differential backup systems (that we aren’t using yet). I will work in bringing in some of the features many of you have requested for the forum over the last months and we will try to better integrate the different parts in the website with each other (yes, some more graphic design will help).
In the meanwhile I have to credit Cristian (aka Mr. Vacuum) for is very helpful contribution to the Forum and Identification system, and all of you for your patience during this last month or so. We will work in anticipating things much better from now on.
Remember
If you want to contribute to the Arduino website, you can use thePlayground, which is an open wiki, but you can also be part of the Forum. You can help by volunteering to moderate a forum in your language (we can create boards in any language in the universe, just ask) or translating the Arduino reference to your mother tongue. Send an email to web [at] arduino DOT cc or comment on this board in the forum with your suggestions.
CONTROL AN LCD WITH A 595 SHIFT REGISTER
– January 16th, 2012
[Carlo Denaro] is sharing a smart solution to save digital pins while controlling an LCD, using Shift Register 74HC595. A simple yet useful project with skecth&libraries, datasheets and Fritzing schematics.
via [grayhats.org]
I AM PRINTER
– January 12th, 2012
Some arty news from K3, Malmo University, Sweden. Three of the students at the Master course in Interaction Design (Scott, Baris and Marcus), in collaboration with the Swedish poet Pär Thörn, created the interactive art installation “I am the poet”. A machine and event dedicated to gather twitter messages including the words “I am” and print them in an endless paper role. They used Arduino to hack an office printer and produce the stream of thoughts in real time.
I AM PRINTER, (c) 2011 by Scott Meadows, Baris Serim and Marcus Ghaly
This project was better explained by the authors themselves:
“I am” printer is a label printer that has been hacked to print a continuous stream of poem by editing the latest twitter feeds that include the phrase “I am” [...] Continuous and automatic printing of the verses, much like a ticker tape, emphasizes the mechanical editing of the poem. Text of the poem is constantly regenerated using algorithms that control the printer.The work has been displayed in Galleri 21, Malmö together with a projection from Marcus and the web application I am developed by Scott. During the exhibition some 80 verses have been generated, printed and later handed out to visitors. Together with other works, the poem is a piece of cultural magazine Pequod’s 2011 April issue.
Besides the student work, Magnus Sjöholm ha written an article about the process that originated this project. He interviewed Erling Björgvinsson, a post-doc at the Medea Research Studio, and professor at the mentioned Masters’ program.
Pär Thörn came up with some ideas and the students began to develop them. They explored digital flows and wrote software and code that made queries for sentences starting with “I am”. Google, the first choice, turned out to be too limiting. Twitter worked better, but if you create poetry with the same speed as the world tweets “I am …”, it would become completely unreadable. A certain percentage of tweets had to be skipped to slow the pace. In addition, we excluded all retweets, all links, and names.
If you want to know more about this project, you should check the following references:
- http://medea.mah.se/2012/01/i-am-and-other-global-twitteralities/ –> the English version of Malmo University Press academic journal explaining the story and interviewing Erling
- http://www.brssrm.com/i-am-printer/ –> Baris website showing some nice pictures of I am Printer
- http://iampoem.net/ –> the never ending twitter poetry
- http://pequod.nu/2011/04/par-thorn-i-am/ –> the page where Pär, the poet with the idea, explains the concept (in Swedish)
CIRCUIT MILLING WITH ROLAND IMODELA, NOW AVAILABLE ON ARDUINO STORE
– January 10th, 2012
[Enrico Bassi] from FablabTorino brought to the extent the use of the Roland iModela, using the little (and inexpensive) desktop milling machine for pcb milling. (in the picture we milled the overexposed Fritzing Parking Assistant).
He wrote some Scuola Lessons (here’s the first) to explain how he realized the pcb, starting from Fritzing.
Roland iModela is now on sell on the Arduino Store.
source [FablabTorino]
TOM IGOE IN CONFERENZA A LUGANO
– January 7th, 2012
Tom Igoe parlerà Lunedì prossimo a Lugano (09 gennaio 2012 – Dipartimento ambiente costruzioni e design, Aula TC102, Blocco C, Campus Trevano, ore 18.00 – controllate il link alla fine della pagina per eventuali cambiamenti all’ultimo)
Tom Igoe, professore di physical computing e networking presso NYU Tisch School of the Arts a New York, illustrerà la sua esperienza e i suoi progetti nel campo della formazione del physical computing per il design e le arti multimediali in un seminario aperto al pubblico organizzato nell’ambito del programma Master of Advanced Studies in Interaction Design SUPSI. Sarà presente all’incontro anche Massimo Banzi, docente di interaction design presso il MAS, e co-fondatore, insieme con Tom Igoe, del progetto Arduino, il micro-controller open source per la progettazione di artefatti interattivi.
via [Supsi]
THE EVOLUTION OF CONTROLLERS [INFOGRAPHICS]
– January 4th, 2012
CHRISTMAS LIGHTS E-MAIL NOTIFIER
– December 20th, 2011
So you are busy this Christmas day, and are one of those people who have been dragged away from their laptops.
Fear not, you’ll know the exact number of emails even while you are away from your laptop.
The guys over at Make have decided to guide you through the whole experience. In the example, they use an Arduino board hooked up to the internet using an ethernet connection, but there’s no reason it can’t work wirelessly – you just need the right board.
Its a simple-to-build exciting festive project!
Hooked up to the web, a bit of code on your mail server and the board itself allow the Ardunio to count your unread emails. If the number’s changed, it throws a signal to a power switch which controls the AC power connected to your lights.Voila! When you get a new message, your Christmas lights let you know.
[Via: Gizmodo]
ITP WINTER SHOW
– December 18th, 2011
For the past couple weeks I’ve been wrapped up in finals at ITP, and in producing the ITP winter show 2011, along with my colleague Marianne Petit. I lost count of all the great Arduino projects in the show, but probably somewhere around 70% of the 110 projects in the show have a little blue board somewhere inside. If you’re in New York City today or tomorrow, come on down, it’s a good time, with lots of creative projects covering a range of application areas from healthcare to fashion to video sculpture to politics, data visualization, assistive technologies, and more.
Here are a few selected at random that are using Arduino:
- FOLKBOX is a device that allows a person with limited left-hand dexterity to play the acoustic guitar, by Justin Lange.
- Glute-o-licious is a virtual cycling experience in which the user controls the speed of a first-person video by riding a stationary bike, by Courtney Mitchell
- HeartRacer is a portable exercise-based video game that plugs into your TV and is powered by your heartbeat, by Nick Santaniello
- Kinetic Sculpture 5 is composed of five pendulums that the viewer can control. Simply wave your hand over the sensors to “play” the piece, by Ben Light
- Metrochange is a charity donation platform using New York City subway cards, by Genevieve Hoffman, Paul May, and Stepan Boltalin
- RFID Beat Box is an instrument that lets you create and play your own music using RFID tags, by Danne Woo and Stefanie Kleinman
- Nostalgia is an interactive art piece where you can view a memory of the artist’s past by typing, by Yoonjo Choi
Here is a full list of the projects. Come and see them all!
DIY SILHOUETTE SKETCHER
– December 14th, 2011
Spanish artist and Fine Arts professor Rubén Tortosa partnered with Computer Science professor Miguel Sánchez, both from Universitat Politècnica de València, to create an interactive arts installation that captures visitors’ silhouettes using a Kinect depth-camera and print them on the wall using an Arduino and a couple of stepper motors. Image processing uses OpenCV library and Daniel Shiffman’s Kinect library for Processing. Arduino code is tasked to act as an interface between the PC and a couple of microstepping motor drivers.
Check the other videos in Miguel’s profile to fully understand how this works. He promised to make better documentation as soon as he has the time, but he couldn’t help sending his proof of concept to our blog as soon as it was done.
Thanks Miguel for the videos, this is beautiful!
PORTABLE, DIY DISCO DANCE FLOOR
– December 9th, 2011
[Rave Rover's Chris Williamson] made a portable DIY dancefloor, sharing instructions and schematics.
Like with many projects similar to this, an Arduino board controls pretty much everything. The floor is dominated by powerful LED lights, which respond to a Serial Peripheral Interface (SPI). A small computer is also inside (complete with Wi-fi), along with a car radio hooked up to speakers. Oh, and thanks to additional wheelchair motors, the floor can be wheeled away to wherever it’s needed.Chris managed to build the dance floor in just one month, and documented how to do it on Instructables for anyone who wants to make their own. And now you’ve seen this, would you really want to throw a party without one?
0 comentarios:
Publicar un comentario