Geeky Peek

I’m very proud to announce that 3jam and Peek have entered an agreement. 3jam is going to deliver improved & enhanced texting services for the Peek!

Some of you may have noticed that 3jam has launched a really cool platform today that competes with Google Voice. I think its a pretty cutting edge service, and I’m pretty excited to partner with these guys and bring some innovative texting services to the Peek.

Here is the TechCrunch article on 3jam’s launch. And here is a great presentation on prezi explaining how 3Jam’s service works.

I’m actually using this on my Peek now, I don’t even need a phone anymore. I can have people send texts or leave voicemails and I get them as emails on my Peek. Its like the complete phoneless solution… Peek + 3jam!

Texting is the fastest growing feature on the Peek, growing exponentially the past few months. After all $20/month for unlimited email + texting is a pretty crazy bargain if you are much of a texter. In the past we have sent texts using carrier email to SMS gateways. Unfortunately some carriers rate limit or just straight up lose some of these messages. So we’ve been looking for a partner who could help us not only improve our current texting experience but also enhance our texting service with cool new features.

You guys should get excited, we are going to do some really cool stuff together.

We’ve done it! We completed a fondly named project called Operation Lockdown. Surprisingly “Lockdown” was a wide sweeping security & privacy blitz.

From the onset, let me say that we do not store any credit card information. We actually get PayPal to store it for us. Presumably they have more resources to become masters of the credit card fraud/security universe. So that’s always been a nice weight off of our shoulders, not having to spend a lot of time worrying about some computer in a bunker with people’s credit card data.

So the other ultra-sensitive information we have are your emails. Us nor any of you would be happy if your emails got stolen. Well we were passing the usual security scans & audits, we wanted to go further and “Lockdown” our system.

The world of Amazon is a bit tricky for security. We have a large number of servers. By default all of our servers can talk to each other, which is how we had things setup. We of course follow the best practice of closing off our servers to the outside world, but we wanted to go one step further and not allow our servers to talk to each other unless necessary.

In Amazon AWS this is managed by security groups. And annoyingly you can’t change security groups once you’ve launched a server. So we basically had to rebuild our network from scratch. That made Lockdown a very difficult and time consuming project.

The auditor is coming in this week to verify the results, but I thought you’d all want to know that your emails are a wee-bit safer today.

One of our uber-geeks here at Peek, Nathan, pointed out that these guys – www.makerbot.com – opened up literally beside my house in Brooklyn!

In my view, 3D Printing will either be one of the most transformative technologies or one of the biggest disappointments of our era. It is full of promise to democratize manufacturing so that every neighbourhood or even person can have their own little manufacturing machine in their basement. Need new cups… don’t go to WalMart, make it yourself!

I think we’ll need to get one for the office here, imagine being able to spit out new Peek prototypes day and night. Whip up a 3D model, feed it into the magic MakerBot and boom… you’ve got a prototype. We’d end up with like 200 models of Peeks within a month.

Ok so we’re actually hiring two roles – a sys admin, and a test lead. Both roles are very exciting. And if you’re a good person who enjoys working hard you will love working here. Seriously, we’re a really good group of folks to work with. So here are the two real job posts – please email me to apply (dan getpeek com). You can be a recent grad or a really good new grad. Also NY or Bay Area is fine.

Operations/Systems Engineer

At Peek we are looking to grow our network operations team. The operations team is the core of our business, the candidate will be tasked with the crucial task of keeping our systems running as a well-oiled machine. Your goal will be high-availability and rapid-scaling.

This person will manage the server systems (mostly Ubuntu Linux), MySQL databases, and application deployment. Ensuring the systems are in good operating condition will be an important role, and keeping up with the latest practices in network operations will be crucial. You will be expected to have vision and come up with ways to scale our system and drive availability.

Key Responsibilities:
1.Setup and management of virtual servers in the Amazon EC2 environment.
2.MySQL database administration and tuning for high performance
3.System performance analysis
4.System problem identification and troubleshooting
5.Work on maintenance windows and outages during flexible hours
6.Dream up better ways to manage our systems and deploy system administration tools (Nagios, Zenoss, Puppet, scripts, etc)

Important Skills & Experience
1.Knowledge of variety of Linux operating systems.
2.Outstanding knowledge of IP networks, firewalls, email systems and web servers.
3.Good understanding relational databases, specific experience with MySQL is a plus.
4.Efficient management of distributed and redundant systems.
5.Experience with JBoss Application Server is a plus.
6.Knowledge of system security concepts and their application.
7.Good communication skills, ability to present architecture designs and research results to the rest of the team.
8.University degree in computer science or applicable engineering degree and more than 1 year of work experience in an operations environment.

More than anything you need to be a great colleague – enjoyable to work with, smart, engaging, a good work attitude and a good sense of humour. No jerks allowed.

Location – New York or the Bay Area, California

Software Test Manager

In order to achieve our goals of being the simplest and easiest to use mobile email service we require somebody to drive the quality of our core services. We pride ourselves in driving service reliability and customer satisfaction through rigorous, consistent, and repeatable testing.

This role will coordinate testing initiatives across Peek’s internal work streams – the device, our email service and more. The quality manager will define a grand vision for testing, come up with a test architecture and test tools, execute test cases, track bugs to completion, drive fixes against requirements, and ensure quality standards are met in development. It is expected that the manager will be the source of quality initiatives, tools and best practices to continue to expand the breadth of quality.

Key Responsibilities
1.Manage the all aspects of Peek’s Quality partners/testers/vendors; ensure bugs are found, tracked, prioritized and assigned to releases.
2.Manage the quality process for software going from development to live. Ensure the software meets quality standards before going into production.
3.Enhance existing quality processes through the development of best practices, test plan reviews, and automated tools development.
4.Owner of system test architecture, test tools, test plans and test results, this person will report to key stakeholders on test progress and timelines to ensure all members are aligned on testing efforts.
5.Organization of testing resources coordinating testing efforts of features vs. regression. Balance and attention to competing timelines

Important Skills & Experience
1.Prior experience in testing.
2.Knowledge of software processes & best practices, especially in testing, tools development, and testing automation.
3.Experience in an independent, results-driven environment while balancing projects in multiple work streams.
4.Incredible communication skills.
5.Ability to lead vendors and cross-departmental projects.
6.Degree in engineering or computer science with 2+ years in software.

More than anything you need to be a great colleague – enjoyable to work with, smart, engaging, a good work attitude and a good sense of humour. No jerks allowed.

Location – New York or the Bay Area, California

With our ever-growing customer base we are bolstering our sys admin team – nominally the job is titled “Guy Who Keeps Stuff Running”. We are looking for somebody who is pretty technical with a bit of experience (1-2 years is fine). This person will need a whole host of standard sys admin qualities such as:

-Spontaneously erupts in moments of anger, especially at perceived incompetence
-’Tux’ tattoo somewhere on their body

-Has long debates with friends about the merits of yum vs apt-get (just kidding, you shouldn’t have friends)
-Owns at least one Man O’War t-shirt (Dragon Force might also be acceptable if you can defend the choice)

You must be able to recite the words to this song

-Uses IRC as their main means to communicate with the outside world
-Checks weather every morning using a home-grown PHP script that sends texts to your device
-Must wear cell phone on belt clip
-Borderline alcohol/nicotine addiction considered a plus
-Works best at 3am when the phone rings with something wrong, since you love being on call 24×7
-We must be able to find several online flame wars where you completely lost the plot over scripting technologies
-Finished 3.68/4 years of university
-Thinks John Allspaw’s awesome 10 deploys per day presentation full of swearing was the “I have a dream…” moment of his kind

Tears roll as you watch this

Seriously, we run LAMJ on Amazon AWS (Ubuntu, MySQL, Amazon EC2 & S3, & JBOSS). Send me an email if interested (dan at getpeek dot com) and kickass.

Check out this link – youtellus.getpeek.com

We’ve been using it to understand feedback from our customers and the market. So far it is pretty clear in telling us to focus on adding more to the user experience of receiving emails, than to say add new functionality like calendar, IM, etc. The two highest ranked features right now are synching inboxes and Peeks, as well as colour coding the multiple emails accounts so folks understand which emails belong to which accounts. As a simplicity, anti-featuritis kind of guy that appeals to me greatly.

Check it out and feel free to give us feedback.

One of our nodes needs a rather large cache, right now it runs at 2.6GBs but it needs to be running at around 6GBs.

My old 32-bit brain, 1990s-upbringing has been screaming to me “Don’t go past 4GBs heap size in the JVM!!!”, with a post-apocalyptic vision of Java garbage collection bringing the Peek world to an end.

Well, it seems that the new Millennium Generation has grown accustomed to 64-bits and brought in an era where the JVM can go to much larger heap sizes (16 exabytes!). So we decided to test this out and increased the heap size to 15GBs in our test environment! 15GBs of heap size… unheard of (well to me)!

So we’ll be rolling this out to production, though I’m still scared to death about garbage collection. I’d love to hear some reassuring stories from any of you about your mammoth heaps running in production, or some tips/advice on optimizing ‘generations’ and garbage collection.

Reading an interesting book right now called “Software Testing Fundamentals” by Marnie L. Hutcheson.

Marnie outlined perfectly my thoughts on quality assurance and documentation with the following line:
“Documentation is the biggest single impediment to software quality today” (note that the book uses the word ‘paper’ instead of documentation).

In fact the book also outlines that research shows that “Documentation is the single biggest bug repository in software development” – via design errors, ambiguities, not thinking, not telling people whats in the doc’t, etc, etc.

In university in my 4th year engineering class I remember learning that specifications were crucial because it allows you to build the program twice – first when you spec it, second when you build it. Wow! I think that mode of thinking has a time and place, but generally not for consumer software products.

Using a more iterative development process it is almost impossible to keep documentation up-to-date, reviewed and in accord with the code. And lets not forget the knowledge transfer that has to come with that document.

On top of that I actually hypothesize that documentation leads to a decrease in collaboration. The mentality of “Read the F’n Manual” can become pervasive instead of strong communicating within the team. The flow of communication can go from a process like:

“Developer tells tester and system designer about new change… tester and designer update test & use cases”

To a document-driven process like:

“Developer tells designer about change. Designer updates document and reviews it with the developer. Designer then has a meeting with tester to review the document changes, likely a week later bundled in with some other changes.”

Annoyingly though, documentation is a necessary evil, knowledge can’t be housed in one person’s head. So, what is one to do (from the book and from my own experience)?

1. Use diagrams/modeling instead of word-written specs – helps reduce ambiguity and shear volume of documentation.
2. Use interconnected tools that tie together the crucial elements of the dev’t process. Code check-ins should tie to bugs, tasks should tie to use cases, etc – tools like Trac + SVN (thats what we use)
3. Use collaborative, portal software – Redmine, Basecamp, Pivotal Tracker, FogBugz, etc (we use Trac + Basecamp)
4. Enforce strong collaboration throughout the team – meetings (yikes)

In general though I refute the idea of building it twice, once with documentation, once with code. I’ll always favour an approach that starts with light requirements and then goes right to an actual work product – Rapid Prototyping followed by iterative development of the real product.

At Peek we did exactly this.

1. First we built an email client that used a server connected to gmail. The email client worked on the T-Mobile Dash (HTC phone). We played with it and gave it to some trial users to get some feedback.

2. Then we started the dev’t with our first build two weeks in, using a weekly cadence of releases (more often in crunch periods).

Ok, well thats a long rant. And I know I promised to talk about ops processes instead of dev’t processes, so I’ll get busy on that!

The world of databases has changed a lot in the past 5-ish years. 5 years go the approach to data was basically “buy a huge honking RAID and put all data in it with a $50k Oracle license”. I worked in teleco where that approach was extremely entrenched. Sun + Oracle.

Well since then things have changed dramatically. Amazon S3, MapReduce/Hadoop, better distributed caching, and new distributed databases like couchdb have come along. Now a days, its almost as if the best practice is to avoid the database at all costs in favour of more distributed systems!

In the early days of Peek we went with the database approach, albeit using MySQL and hosting it on Amazon EC2 (not a RAID). Well, here’s the gotcha of that approach. Because of Amazon I can only throw so much hardware at the problem, I’m on their Very Large instance now and thats as big as I can get! So I lose the growth of a RAID. And since I chose a database that doesn’t distribute very well, I can’t distribute and grow horizontally easily.

So now we’re in the midst of re-designing our data layer to store a lot more things in distributed file systems like S3.

But in the interim we still have to keep the database running fast and furiously. So I have to start doing something I’ve often ignored in the past, I have to optimize the database!

As a very small company we don’t have an in-house dba. We all have some db knowledge, but its not deep expertise. So we’ve got a db guru who worked at MySQL for many years, Ronald Bradford 42sql.com, who helps us out. Generally he’s been a lifesaver. So armed with some support from Ronald and the trusty High Performance MySQL book written by the Percona guys I set about with the first baby steps in making our database run faster.

The first step is to deeply understand your traffic and data needs. We have very spiky writing patterns (people receive emails in clumps), and pretty constant reads. Our reads outnumber our writes about 3:1 statistically. On top of that we don’t need to be absolute in our ACID requirements of transactionality. We can always re-fetch emails that fail to store. The last note is that we use the MySQL InnoDB engine.

So we made a few big changes:
1. We got more aggressive about cleaning our database and increased our delete script amounts.
2. We changed innodb_trx_flush_commit to level 2, which favours performance over lossiness of writes/consistency of data.
3. We added another 1GB of memory to the innodb buffers to help with the spikiness of our traffic patterns.
4. We decreased the connection pool size on each node. We found we had a lot of OS Waiting due to context switches.

These changes had an extremely sizable impact. We used to have 120 threads running to handle 500+ queries per second. Now we have 20 threads running to handle the same volume of queries.

And we’ve still got a few more changes to make as well, increasing innodb log size to allow longer delays until it writes to physical disk, adding more memory to the buffer and playing with dirty page percentages.

There is a very broad Linux for ARM movement around the world that is ongoing that has me excited these days as we use an ARM7 Texas Instrument’s processor.

http://www.linux-arm.org/

This is the list of the over 2300 devices in the world that have been ported to Linux for ARM variants:
http://www.arm.linux.org.uk/developer/machines/

There is even a flavour of Linux for our ARM7 processor (we use ARM7TDMI):
http://www.uclinux.org/pub/uClinux/ports/arm7tdmi/
http://www.linux.org/apps/AppId_8588.html

So armed with this new fountain of information I’m hoping one or more of you out there in the world can get Linux running on the Peek. Feel free to post here on how far you get.

And here’s the deal, if one of you does make it happen you can have a little mini-consulting gig with us to tell us how you did it and how we could build a Mobile Linux Peek.

And here’s one more link to inspire some of you out there, a list of hacks on other consumer electronic devices and how the hacks have been done http://dev.eyebeam.org/projects/reware/wiki/Reware