Geeky Peek

New York Times produced an interesting article on privacy today.

Amazingly, I think the internet has progressed massively in terms of security and privacy. There is still far to go, but there are tons of organizations and standards out there that try to enforce trust and privacy with your data. And if you don’t trust other organizations there are all kinds of privacy tools available – delete cookies, change your IP, etc.

But the world of mobile – wow! Nothing belongs to you and you can do nothing about it. While the internet has been built democratically by thousands of organizations (and amazingly dominated by a low-gov’t trust libertarian viewpoint out of Silicon Valley), the world of telecommunications has been built by dominant teleco monopolies. At somepoint along the way the following was de facto decided:

1. When you communicate over a mobile network the government can look at whatever you say whenever you want.

2. When you communicate over a mobile network, the corporations involved in the communication can basically do whatever they want with that data.

Without a lot of public debate, this has become par for the course, and we’ve already, very rapidly, gotten use to it. And its a problem.

Most recently in Iran the problem with gov’t “sniffing” came to people’s attention. In Iran, the government was able to monitor all people’s communications on their cell phones, disrupt protests and arrest protestors.

Amazingly and pretty much unaware to users of the networks, the governments in almost all countries have the exact same capability. In India we have had our launch delayed significantly because of Lawful Intercept requirements. Literally, the Indian government has a tap into all teleco data with all carriers, with the ability to refuse the launch of a product or service until Lawful Intercept requirements are met. Can you imagine this kind of model on the internet? Could you imagine if you had to apply to the gov’t regulator to put a website up or to install a router?

BlackBerry made a big deal about this a year or so, fighting similar delays and also not wanting to sacrifice its ultra secure connection for corporate customers. Here’s a link talking about BlackBerry actually being threatened to be pulled by Tata by the Government of India – http://tech2.in.com/india/news/smart/blackberry-blackout-by-indian-govt/30901/0

In the US, its no different. We’ve had customer data taken by law officials multiple, mulitiple times. We were warned by the FBI that in short order we’d have a full time staff member just to handle their data requests! Can you imagine a start-up of our size employing somebody full-time just to handle police requests? I’m pretty certain that the criminal element is not that large a member of Peek’s base that we don’t need to be providing 5 day a week full-time support. I mean its flattering that the various government agencies think we’re that successful but come on…

I know some of you probably sit there and say “Hey, whats the big deal? I don’t protest against the US gov’t, and we’re in a nice stable country… who cares?” Well I’ll try to give an example that hits a bit more home.

Like sending provocative texts (sexts) to your “amour”? Well, so do the engineers at the carriers, who can basically sit there and read your texts all they want. I’ve helped deploy SMS routing applications all over the world and read “sexts” in more languages than I care to imagine. Guess what, the government also gets that data feed, and many of their engineers can read that on a whim. Oh by the way, folks in marketing, they also get those texts. They want to slice and dice how and why people use texting so they can offer more compelling marketing (ads, promos, price plans, etc).

Well, a friend of mine has started an Android project to hopefully help give people a choice – http://openideals.com/guardian/ Basically the idea is to port the Android OS into something that sends end to end encrypted messages that can’t be snooped on.

I think its a pretty powerful concept, much like when PGP came out once upon a time ago for email. Its funny, when I think back to the 90s and early ’00s. Basically BT had a monopoly across Europe which got spun-off and privatized into the O2 group. The O2 group became a dominant carrier across Europe, but BT provided a lot of the internet lifeline (backhaul, etc). The O2 group had a very stringent security policy around emails, everybody had to use PGP for secure emails. So somehow big corporations recognize the need for security, but consumers don’t? Why?

Which is what sort of drives me nuts about privacy and security in mobility. Not to be all big brother/anarchist, but it is a tool heavily relied on by those in power, but not provided to the masses. CEOs, gov’t personnel, etc all have the ability to secure and encrypt their communications… shielded away from prying eyes. They buy BlackBerrys, they build additional encryption, and so on. But somehow this basic right can’t be afforded to the customers of the service. And really, shouldn’t the openness be the other way around? Don’t we need the ability to trace through big decisions made by corporate CEOs and gov’t officials?

I’ve followed this space pretty closely for a while. Mobile money has been one of those technologies that has been long promised but never realized. Does anybody remember the early years of the 2000s when everybody thought we’d be buy from coke machines via our cellphone?

Working in the Caribbean, Mobile Money was also seen to be a fantastic opportunity. In Jamaica and Haiti, huge portions of their economy are based on remittances (i.e. foreigners sending money back to the country… usually family members who have gotten work abroad). The goal was to find a way to transfer money internationally using cellphones. Digicel got as far as being able to do international “top-ups” (from New York top-up your cousin in Jamaica), but never really quite got a full “remittance” product out there that could compete with Western Union and the likes. If you can snag even a small percentage of transactions you can make a lot of money.

Well, it turns out the idea of Mobile Money is starting to actually happen. There is a confluence of both devices for handling money as well applications/payment gateways to support those devices. The news has been littered with articles & press releases recently that are starting to connect the dots.

The first article that caught my eye was this one:

http://news.bbc.co.uk/2/hi/technology/8516079.stm

The $15 price point on the phone is pretty impressive. Even in Haiti I think our lowest cost phones were $20-$25 (Alcatel). But even more so than the $15 price point, the phones are embedded with Mobile Money functionality.

In many parts of the world 40% or more of the population is “unbanked”. I.e. money is not kept in their bank accounts. When somebody puts $20-$50 of “credit” on their pay as you go account, it is basically a very lightweight bank account. So mobile operators are seeing opportunity to provide banking-like services – bill pay, money transfer (btwn phone accounts), purchasing, remittance, etc. Its an incredible opportunity as the teleco can skim lucrative transaction fees that have been limited to the very few monopolistic banks/financial service companies (i.e. Western Union).

BTW – I love this song by K’Naan on Western Union – http://www.last.fm/music/K%27naan/Troubadour/15+Minutes+Away

On top of mobile companies and telecos getting into the game, we are seeing Paypal and the high-tech payment industry attempt to cut into the payments game with richer payment infrastructure. This Wired article caught my eye – The Future of Money, talking about Paypal’s repurposed payment APIs and how it has helped create an explosion of payment applications, like their TwitPay example. Note that we use these APIs at Peek for our billing needs.

And further more if you look at Jack Dorsey’s (Twitter founder) payment add-on for smartphones Square Up, you’ll see its possible to basically turn some smartphones into payment machines. This means mobile professionals can easily accept payments on a $100-$300 phone, instead of say on a much more expensive, tethered moneris terminal.

So I think the mobile money revolution has finally arrived! Smaller devices for money transfers and bill pay exist, while richer devices for accepting payments are gaining popularity. On top of that companies like Paypal are making it more and more possible to plug into the payment eco-sphere in a safe/secure way.

The last question is if Peek should join the foray. Being a device used heavily by the SMB market I could see processing payments as a powerful application on the Peek. Imagine if say a locksmith on a job could just enter in credit card details and find out if the payment was accepted. Pretty cool, right? Food for thought…

Its been a while since I did an ops and outages update. Remember our goal at Peek is to be above 99.9% uptime, or roughly less than 1 hour outage per month.

Generally uptime has been much, much better than the early hey-days of Peek. Here’s the past few months:
October – 99.9%, about 40 minutes of outage per customer
November -99.92%, about 30 minutes of outage per customer
December – 99.1% about 6.5 hours of outage per customer… we did a big upgrade and also battled some capacity problems with Christmas sales. The holidays were not so cheery for me.
January – 99.92% about 30 minutes of outage per customer
February – 99.99% about 6 minutes of outage per customer

The number one problem for uptime is actually ourselves. Deployments historically have been a problem. I think part of the problem is that despite practicing agile dev’t, our deployments are relatively “big-bang”. I.e. we plan a big release and try to launch it several months later (with tons of iterations along the way).

I think a big goal of this year would be to start embracing some of the more modern Agile Web Ops/IT practices like:

• Smaller more frequent releases
• Investing in “rollback”-ability instead of relying so heavily on QA
• More dark-launching capabilities
• Better metrics so problem deployments can be detected quicker

Another area for improvement is Time To Detect. Detection time is the amount of time from when an issue occurs till the point that somebody is notified and starts solving the issue. Classically our detection time was about 40-50% of our overall outage time! Seems horrid, though I know detection is a very difficult problem for many tech ops organizations. We generally resolve an issue very fast, but sometimes things fail in new ways that your detection system has problems detecting.

So to fight this we deployed some serious cutting-edge technology… no wait… bleeding edge technology. Something so advanced, it’ll blow your brain away. We got a “person”, i.e. a real human being, to pick up a Peek twice a day and use it (well, they run a set of tests). If it doesn’t work work, they come tap us on the shoulder! How crazy is that? The funny part is, it really works! Sometimes there are parts of the system that are less used by customers. For instance reading PDFs. If our pdf convertors aren’t working in some weird way (for instance converts incorrectly but still produces stuff), it is not very quickly reported by customers and virtually undetectable systematically. But a manual test picks it up perfectly!

Anyways, we’re doing well at hitting 3 9’s so the goal of 2010 is to move up to 4 9’s, 99.99%, or about 10 minutes of downtime per user. So look forward to even better Peek service.

damn you team USA. Free month of service if you buy a Peek now, for the rest of the game if you use the promo MARTYISGOD.

And now so is HOCKEYNO1 8-(

But at least we kicked butt in ice dancing tonight.

Well, with some good timing, consumer reports just posted the latest usage numbers for mobile data.

http://www.macrumors.com/2010/02/12/average-iphone-consumer-data-usage-pegged-at-five-times-that-of-blackberry/

iPhone – 273MBs
BlackBerry – 54MBs
Peeks – 3MBs (!!)

Mike Lazardis echoed my sentiment as well with this line at Barcelona this week “manufacturers had better start building more efficient applications and more efficient services. There is no real way to get around this.”

Funny enough, the BoyGeniusReport hit back with some stellar, well thought out feedback (sarcasm) -
http://www.boygeniusreport.com/2010/02/16/rims-mike-lazaridis-wants-manufacturers-to-limit-your-data-usage/

A few thoughts, well potentially in the US and EU we may get to 4G and LTE relatively fast, in the rest of the world carriers will upgrade less quickly due to the nature of their markets. These markets (lets take Haiti for one) still need access to data applications such as email (such as all the workers right now in the field). Email cannot be iPhone like, sucking up all the bandwidth and straining a network that is built to run on users who pay $10-$15/month. It simply won’t work.

Secondly, even in the US where 4G and LTE are ‘close’ we are still 3-5 years away from seeing them happen. I think most users in New York and the Bay Area would agree to just about anything to have their iPhone’s work, even if images were slightly more compressed or 1% of web pages got ruined by compression. Congestion and capacity is a problem RIGHT NOW, why not apply easy to deploy technology to fix the current problem?

Lastly, you can efficiently optimize data across mobile networks, without impacting user experience. The Bolt Browser is one very good example here. Email is another, by way of example iPhone wastes tons of overhead by using IMAP Idle directly from the phone. Who wants to pay for overhead? Email can still be a fantastic experience when its optimized. I get that BlackBerry does a poor job with attachments, but because they did one thing wrong doesn’t mean the whole model is broken.

Remember in the world of scaling/managing capacity there are always three answers:
1. Make a bigger pipe (BGR’s suggestion – vertical scaling)
2. Add more pipes (horizontal scaling)
3. Become more efficient (RIM’s suggestion – diagonal scaling)

Why would anybody limit themselves in terms of solving this very difficult capacity problem? We should be doing all three!
Why wouldn’t the industry work together on #3? You can really cheaply gain capacity via efficiency.
Wouldn’t users be happy to not pay for dumb protocol overhead? I mean I loooove paying for shit like TCP retries because some idiot iPhone app builder doesn’t know how to work offline mode properly.

Bitstream released their new browser, the bolt browser in December.

I think this re-enforce my point about gateways and ‘middle men’. Bitstream doesn’t just sell their browser, but they sell a ‘gateway’ that shrinks data usage by 24:1. Its a pretty powerful value prop for any of the carriers. I know this because bitstream used to try to sell to me at Digicel. Optimizing traffic can save a lot of money for the carriers.

In the world of mobile, the major applications such as browsing, email, calendar, forms, business apps, etc – should go through an intelligent gateway to optimize payload.

One more note here, gateways also help incredibly with offline experience. I have a G1 and so many of my apps work poorly or don’t work when the device cannot connect to the network. I don’t know the global numbers, but 30-40% of Peeks are offline at any given point.

Some Yankee joker in our marketing department thinks its heeeelarious to run a pro-USA promotion on a system that was built and is maintained mostly by Canadians.

Did you see this insidious post, Go For Gold, on PeekSpeaks. How dare they!!!!

Well my friends… actually none of you are really my friends for the next two weeks. I am now decidedly anti-American for the remainder of the Olympics. Canada is going to win the medal count, the world order is about to be heavily disrupted. First our banks bought all your banks, and now we’re going to beat you in the Olympics. Next thing you know you’ll be eating poutine and steamies, with french swear words interlacing your speech…. taberhuite, that is not bain cool yo!!!

And forget that all of us Canadian GeekyPeeks left Canada to move to the US, thats not an important fact right now. Also forget that most of wouldn’t leave the US either (well I just moved back). YOU’RE GOING DOWN U-S-A, YOU’RE GOING DOWN.

“Ohhhhhh Canada, our home and naaaative land”

So, since I have the keys to our handy-dandy little promotion system – I have setup a counter promotion. For every gold medal that Team Canada wins you get a free week of service. And get this, I am so confident that our hockey team is physically and mentally superior that if your hockey team beats our hockey team I will set the system up to give away free Peeks for an hour. I will also shoot myself. In fact, screw that… if your hockey team scores more than 2 goals on St. Martin Brodeur, his holiness himself… amen, than I will give a free month of service for the period of the game where the score is > two goals! Here are the three promos I will setup:
CNUCKSGOLD
HOCKEYNO1
STMARTYISGOD

Never bet against St. Marty

Inbound, raw, uncompressed traffic to the Peek Servers is roughly 280MBs per customer per month.

We compress and optimize that to right around 3MBs per customer per month. Woah. Thats the magic and innards of Peek’s basically entire business model, right there.

Folks at Apple & Google – use a frickin’ gateway with some basic capabilities and you’ll save your carrier partners billions and billions of dollars globally. It’ll cost you next to nothing to build. You only need the tiniest amount of compression on the top products (email, calendar, widgets, etc) and it’ll make a landslide of difference.

You don’t even need to go as far as us (we consulted with PhDs and experts in compression). Just use gzip.

Hi, I’m Nick, one of the Geekypeeks.

My job at Peek is in the realm of that most eloquent acronym “OSS/BSS” (I’ve chosen for myself the title of “OSS BOSS”). This term has always been somewhat cloudy to me, as its definition can be expanded or shrunk so as to include anything, or nothing. While the term arose in a telecom context around billing, rate plans, and so on, it’s sturdy enough to absorb a bigger definition. What Dan always tells me is that it means I have dominion over “the systems that support the business”. Our “business” is our email/texting service and hardware, aka the Peek. The system that bills customers, manages inventory and SIM cards, serves up data to the finance team, tracks customer lifecycle, and so on, those are things that constitute a “support” role in the grander scheme of things. That’s me.

I haven’t been doing OSS/BSS for long, and am certainly not an expert in it, but the primary challenges of my job are not pure OSS/BSS problems. I do not often get to dwell in the rarefied air of enterprise product catalogs and real-time IP packet rating. Fledgling startups are a swamp of pragmatic concerns — making sure deadlines and budgets are met, managing expectations, communicating occasionally confusing operational procedures to other staff, firefighting, laborious integration projects with elephant-slow vendors, and so on. Thus, the challenge of startup OSS/BSS is to manage oneself in order to support others.

As a result I deal directly with almost everyone in the company. My role is to enable their individual mandates technologically, whatever it may be. In that sense I am a requirements bucket for the company, and my role in some ways resembles a kind of technical caregiver. By this standard, the only benchmark worth a damn is coworker satisfaction. Help me help you.

A setup like this could lead to intra-departmental conflicts and territorial behaviour over limited resources. The basis for business decisions are often built on shifting soil, and it is difficult for organizations to avoid this problem, especially the “flat” organizational structures that startups tend to favour. As Dávila said, “Every non-hierarchical society is divided in two.”

But it turns out that Peek has a fantastic office culture, the best I’ve ever worked in by a country mile, and as a result of everyone’s shared understanding of company goals and general respect level for each other, this has not ended up a problem at all. It’s rather creepy, actually. Could it be that coworker chemistry matters? There’s something to it, anyway.

Anyway, I’ve read Dan’s unhinged screeds with some interest over the past year and I hope to use this space to flesh out some of my loftier ideas, not only about OSS/BSS but about tech, engineering, and life.

In my spare time I am a soccer goalie, a co-ed, rec league superstar Goaltending is an exceptionally tough sport mentally. There is no such thing as a goal that wasn’t your fault. Every goal is a failure. Goalies have to accept failure and being beaten as a fact of life. Growing up, acceptance of failure is generally not part of how we are taught life. “if you study hard you will get an A+”, “practice makes perfect”, “set goals and achieve them”, and all the other optimism/success oriented values of our society do not apply to goaltenders. The philosophy of goaltending has to be one of managing failure. I think strikers have to have the exact opposite philosophy…. they have to succeed and are constantly trying to optimize success.

To me, this is also the exact attitude of software development. There is no such thing as a bug that isn’t your fault. And you will have bugs, sometimes embarrassing ones that cause big outages. Much like the goaltender, us poor techies are often the ‘last man’. So ultimately we are accountable, we are the ones who make the mistakes and have to fess up.

And I think this is why there is a huge dropout rate in computer science and shortly thereafter. Is a 21 year old mentally mature enough to deal with constant failure in his life? Or will they blame the requirements, the due date, the money, the hours, the lack of sleep, the constant changing priorities, their boss, the dev’t env’t, the production env’t, the build process, etc. There are few things worse in soccer than a goalie who can’t accept failure, the maniac screaming at his defense on every goal. Similar for the developer – “it works perfectly on my machine” is an attitude that simply won’t work.

And I think thats why project mgmt styles like ‘agile’, ‘tdd’, ‘extreme’, etc have exploded and done so well. They manage failure & risk along the way (and maybe CMM, waterfall, etc are in the optimize success bucket?).

So for all of you who are looking to ’succeed’ in life – don’t become a goalie or a programmer (or a salesman or somebody trying to raise money or a whole litany of other jobs).