Geeky Peek

Ok, well here is the pass/fail of how we are doing in operating our service.

The good news, if you are a non-Hotmail user, the average user experienced just over 1 hour of outage across the month. Thus we almost hit the target I shared with you before (for non-hotmail users )!

The bad news, if you are a Hotmail user we had a 53 hour outage and you had a really rough month. But we have now moved to POP which will remove the risk of long-lasting Hotmail outages in the future! I recognize that many of you Hotmail users are in ICU in terms of your Peek experience, so we are going to double down to ensure delivery for you is 100% available for a few months.

The details for February:
Non-hotmail users – 66 minutes of outage per customer, 99.87% availability
Hotmail users – 54 hours of outage, 92% availability (again really sorry)

The outages:
1. We had some errors on our side reading emails from Hotmail. Luckily they had launched POP3 on Feb 11 so we cutover to it on Feb 15/16.

2. Our servers that retrieve Yahoo emails had intermittent issues for about 2 hours on Feb 12. About half of Yahoo users were impacted.

3. On Feb 15th one server hit capacity and we had intermittent issues with sending, registering and occasionally receiving. About 15% of users were impacted over the 1 hour period.

4. Our connection to Qualcomm (who is a go-between for us and T-Mobile) died for 26 minutes on Feb 24. Full outage for that 26 minutes, but luckily it may have been small enough of a window for any of you to really notice it.

Again, I know I’ve repeated a bunch of times but the combination of Amazon AWS and Java has been amazing for uptime and scaling. As we’ve been adding customers we’ve been able to very, very easily add more servers and handle the growing load.

Not sure if many of you have perused the 37signals lineup of products, but I can vouch that basecamp is one of the best project management tools around. This technology makes project management democratic & spreads the load. No longer does one person sit on top, collecting status reports and spitting out status reports… each person can update the site with their own messages and people from across the company can quickly understand status. It also decreases the need for complex/expensive project plans using MS Project.

I also just read the book they wrote, Getting Real. It encapsulates a lot of our beliefs in how we run our company, principally:
-small is beautiful
-working product/code over specs
-Build Less to underdo your enemy
-Use forums for support so you talk to customers directly
-Publicize and be open about your screwups (btw we had a 15 minute outage last night)

Its a great read for anybody involved in software and its creation.

RIP
1965 – Feb. 11 2009
Closed Email Systems

Hotmail followed through on its January announcement that it would make POP3 available on Feb 11. We cut over to it on Feb 15th with very few issues!!

Starting Peek in 2007 we have seen all the providers forced into openness by the big movements of: proliferation of mobile email and Google’s push towards openness.

Google launched Gmail embracing accessibility and quickly thereafter included POP and later IMAP access to your inbox. AOL followed suit also offering IMAP & POP3 accessibility. Late last year Yahoo said it would make all of its services gradually available. While they haven’t followed through on making POP generally available, they are pretty open to providing access to 3rd parties like us. And last but not least, the most closed off of them all, Microsoft comes through and provides POP3 access.

So there you have it folks… your email is now accessible!

But… unfortunately POP3 is ancient technology and the world is trying to move on. Don’t get me wrong… its a nice first step that one can now even get their email!!! But we’re almost in 2010 – IMAP type functionality and push-email are requisite! Nobody wants to wait for email.

Well Google has stepped up the race here. IMAP-IDLE, a push email interface, is available publicly. So my ask is that all of you continue to put pressure on the other providers (AOL, Yahoo, Hotmail) to provide publicly accessible, push email APIs so you can have the email experience you deserve!

Our community is erupting with homegrown Peek services. I’m amazed!!!

Firstly, http://charleschilders.com/ent/ has developed a pile of new services for the Peek (well for email) including:
-a translation service
-a bible lookup service
-a UPS tracker

You just email to ent@retroforth.org with the right command in the subject and you get stuff back! How cool is that?

Another user has created http://www.peekfeed.com/, an rss feed for the Peek! Very, very cool indeed!

There’s also the guys at peekservice.com have been rolling along and have news, weather, etc and are looking at some other cool stuff.

Also, mbmcormick, has launched a cool google search engine tool. Email to “gsms@mccormicktech.com” with the search in the subject and it will return you everything.

And finally Mikey94025 has a cool little google local search app – email g@mikey94025.com with your query in the subject line. Be sure to include either a zipcode or the city and state abbreviation at the end. For instance:
* “mcdonalds 94131″ searches zipcode 94131 for nearby McDonalds restaurants.

I love this stuff everybody!

All,

The competition is officially started. For all of you developers who have reached out to me in the past few days please go to contest.geekypeek.com to officially sign-up.

We will provide more technical details in the contest forums over the next few days as well, though there should be enough to get you started.

thank you all

Peek & Xtify

Deep in the labs of Peek HQ we have been busily cooking up a Peeky storm, a storm of new Peek. We thought we’d give you all a glance at what is coming next. The battery life on this thing is amazing, the colours are new and fresh and you should see the newest feature we added on…

BTW – that’s one of our rockstar interns Matt.

There has been a lot of discussion on the boards on upgrading and I appreciate all of you who have been the early adopters of our upgrade tools.

Here is an FAQ of sorts:

1. When I plug in the Peek cable it asks to install a driver? What do I do?
We install a driver during the upgrade process so you don’t need to. But if you want to, please google PL2303 driver and download and install the appropriate driver for your operating system. This will be called a Prolific USB to serial driver.

2. Why do I need this special Peek cable?
In the words of our device guru Tom –
“The mechanical interface is a MicroUSB connector and the electrical interface is RS-232 Serial. The “special” cable has a Serial-to-USB converter in the fat end. “

3. Where do I buy this special cable?
At the Peek Boutique.

4. I can’t get this thing to run, what do I do?
-make sure you aren’t running 64 bit windows, Mac, Windows on Mac or Linux… sorry doesn’t work on those versions yet
-reboot and try again, on some versions of the OS the installation of the PL2303 driver needs a reboot

5. I still can’t get this update tool to run. What do I do?
-Ask for help at the boards (boards.getpeek.com).
-Find somewhere nearby where someone can do the update for you (also on the boards)
-Mail it to us and we will do the update for you (email feedback at getpeek to arrange this)
-Email gabe at getpeek for other help

6. It all works now, how can I thank you?
Feel free to post here about how awesome the new 1.08 build is. Of course I’ve been on 1.09 for a while now (it is in pre-pre-pre-pre-pre beta right now) and it is even more kick-ass. Or you can always email me…

As I posted on the boards we did some maintenance work on the database on the weekend.

A few of our developers have been worried about latency of the Amazon AWS network and our ability to run our large database on it. Well, with our recent maintenance we have pretty much proven that Amazon performs quite well for a large production service like ours.

We recently held a meeting with a bunch of other tech companies in New York that are similar to us (inclusive insiders club only… sorry). It is amazing how ubiquitous Amazon AWS is among the New York, start-up circle. All but one of the companies there used Amazon. Some of them run massive volumes of transactions and storage and Amazon performs quite nicely across a broad array of storage/queuing technologies – MySQL, PostGres, SimpleDB, Erlang, etc.

One of the companies there did not use Amazon, they used the classic giant SAN in a co-lo to run their database. They ran into problems with the SAN and ended up paying buckets to some hardware dude to help solve their issues…. proving my case in point as to why we use Amazon.

Not that we don’t have our own hiccups. We recently brought in a dba, Ronald Bradford. Through him we were able to reduce our database query load by half over the weekend! The amount we were able to optimize was pretty unbelievable.

So the morale of the story here is not just “use Amazon”. Find a broad roster of experts and use them. Amazon are experts at scale and hosting.. I am not. Ronald Bradford is an expert on databases… I am not. I am good at these things, but the difference between good & expertise can be extremely valuable (at the right price).

Here is Amol announcing the Peek Xtify contest at the New York Tech Meetup this week.

We have a crazy hacker at our work… his name is Daniel. Back in the day we hired him because of his renown hacking skills, he was one of the contributors for YPop. Daniel’s hacking skills allow us to use him in many ways since he can both:

1. build cool stuff… fast
2. solve hard problems

So we use him often as the crazy system admin who solves zany outages or we use him to hack up some cool stuff for the Peek.

I’ll give you two recent examples. The first is texting. One day he realized how to do texting… late on a Friday. On the Monday morning it was working. 4 days later, the Friday, we launched texting!

Another example was our outage on Saturday. When getting our tech team setup I asked some friends who work at big telephone companies how they operate and monitor their systems for outages/incidences. The explanation he gave me was this:
-monkey watches system
-something goes wrong, monkey squawks and squeals and tells the gorilla (sysadmin) to fix it
-gorilla grunts and groans, can’t solve anything… then phones developer company
-gorilla gets the monkey on developer side
-….

Eventually somebody useful enters the picture.

In our world we don’t have time or money to have all those layers. We need to solve issues fast. Luckily we have a hacker who can solve complex problems.

So I guess what I’m trying to tell you other techies… hire more pure hackers.