Beyond growth pains

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

Beyond growth pains: A Q & A session with some of the global movers of Web 2.0

The social media we have come to know and love has very few prominent players. For growth in that market and developing your platform to giant proportions – with users other than close relatives and distant cousins. Who, better than the whizzes themselves to tell us how?

I sat in on a Q & A session with panelists from Twitter, Yahoo!, Mozilla Foundation and Simplegeo. Boy won’t you be glad I did because below are some of the things they covered.

Q: “What has been your hardest challenge, in your career, and how did you overcome it?”

A: (Joe Stump – Simplegeo) “Finding people I can bounce ideas off of, who have similar challenges at the same level in their growth“

A: (John Resig – Clear Left) “Understanding that the startup life wasn’t for me. It became most rewarding with a community.”

Q: “Is moving away from a small company and going corporate, the death of creativity?”

A: (Dustin Diaz – Twitter) “Twitter is always flowing with ideas. We all have different ideas and the constant challenge is to ways to implement the thinking of different people.”

Aside: Joe Stump had something relevant to say, which was unrelated to the question but makes sense to put here. “My number one rule is to hire people on the assumption that they great and potentially smarter than me.”

Q: “Are we creating a digital divide through building products that highly sophisticated and require more and more bandwidth?”

A: (John Resig – Clear Left) “There is a mobile digital divide that I’m realizing, most things now being developed android phones and the iPhone.”
The panelists earlier noted that the computer as we now use it is moving away from the need of an Operating System. The panelists assert that they spend less time using some of the functionality they once needed Operating Systems for. More people with team members that work remotely use Google Docs and other web based equivalents.

Some questions also came from delegates and this one in particular by Toby Shapshak I found really worth sharing.

Q: “With the browser becoming bigger and essentially what the internet is becoming, how do you fit that into mobile phones for use on cellphone screens?”

A: (Jonathan Snook – Yahoo!) “Get to the core of what you are trying to build and deliver that to your user. Essentially, products developed for the web have to be delivered differently for mobile phones.”

The session was quite eye-opening considering that while developing a product you also have to think of the business element related to the product.

Mongezi Mtati

Scale to get big

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

It’s going to be a flood of data as more people connect more often with their mobiles. Says Joe Stump (co-founder of SimpleGeo, previously main dev bod at Digg): “Each smartphone has six-plus sensors, and it’s not long before they add barometers and temperature sensors and more. Data production is following Moore’s Law.”

He did a simple calculation, working out what happens if you were to tag the phones (just time and location) once every minute for the 500 million Facebook users.
Just this little addition would add 37.2GB of data every minute to the piles that already need to be crunched.
He asks: “How are we going to store, scale and serve this mess?”
His main point is: scaling != performance.
Performance is more about i/o, and not so much in your choice of language. Choose Ruby, choose php, it makes little total impact to large-scale systems, he insists.
Mostly, scaling is a specialisation.
“The more traffic you get, the more specialised your infrastructure needs to be,” he says. The key is automation – bits should be able to be called or started or attached automatically. Use the cloud, but treat everything in the cloud as ephemeral. It can and will just disappear. Expect it.

He discussed the two approaches to scaling – namely out, and up.
If you scale out, you spread load across lots of boxes. If you scale up you get a bigger, faster box. Less complex infrastructure, but a really powerful box can cost millions of bucks – only workable if your service is making big money already.

Other gems of wisdom:
* Partition your data from the very beginning
* Make use of queues – very important part of consistency of user experience.
* Caching is critical – especially in supporting queues. Write a record to cache while it’s processed by queue so that user experience stays OK.

These are lessons learned from long years worrying about things like: how do you handle objects such as the front page story on Digg when it’s getting millions of hits?
His other key advice is about people:
“It takes a lot of people to build, scale and maintain infrastructure – you will grow from one or two to 15 or more.” The human management issues become tricky here: “The first two or three devs on board are going to question every decision management makes.”
A good thought: “Look for a trait in developers: laziness. You want someone who looks for a quicker, better way.”

As your site (and dev team) grows, he advises looking to lower barriers to entry for more junior devs. “Get your codebase to a position where you don’t need to hire a Jedi. Jedis are rare. Jedis are expensive.”
He recommends breaking teams up. 4-6 people work well, at 8 it starts breaking. Get a Jedi, and make them the team leader. Note: team leader, not manager. They should act more like a sports team’s captain. Create frameworks (authentication, error handling) to lower barriers to entry as new coders come on.
And use code repositories. Full stop.

He is very passionate about promote ownership in the codebase, so that individuals work on three of four areas and have responsibility for them.
“As you scale and your code bases grow, from 50,000 lines of code to 400,000 lines, no-one can be effective across the whole base,” he says.
Before you start, design the software – don’t just start coding. He is a big fan of stubbing out the API on a whiteboard.

When it comes to testing – automation is good, and use several methods. If you fix something, make sure you run a test on the old version and make sure it fails it. Apply patch, and make sure it now passes.
Documentation. Build time into your planning for documentation. Even if old and stale it adds historical context, maybe helping you understand later why you made a particular decision.
Do peer reviews. “I’ve never sat in on any peer review and didn’t see at least one show-stopping bug.”

There are a number of ways to scale up using powerful technologies. “When I left Digg we were handling 37,000 requests a second,” he says. Now at SimpleGeo, he runs 15 nodes in one Cassandra cluster, 12 nodes in other cluster.
The numbers will go up (if you are even remotely successful). The technology is getting faster and faster, handling volumes that would have been unthinkable before. “You can get 1500 writes a second on a decent SQL box. A couple of years ago if you asked me if I’d need that, I would have laughed,” says Stump. Right now he is putting 5,000 to 7,000 writes/sec on a Cassandra cluster.

Most South African web developers, even those working for the relative giants like see only a fraction of these volumes – but one thing is sure. Africa is developing its Internet community fast – it won’t be long before servers talking to thousands of users are talking to millions.

Roger Hislop

Boiling the Ocean

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

During his talk “Boiling The Ocean: how a VoIP mobile business is changing the shape of the Mobile Industry” on how Internet Solutions has progressed since he was asked to start it up and take on the main Mobile Providers, Justin Spratt delved into just a few of the obstacles that he had to overcome in making that happen.

It all came about from them wanting to scratch the itch of mobile costs being too high in SA. With variable costs being around 5c per minute and charges around R2 they saw a clear gap to fill. Deciding to do this through VOIP over an enterprises existing WiFi network, they were able to come up with a very viable solution. In order to do this their requirements were to start with extremely talented people, strive for technical excellence (“Nobody does it better than us” – JS) and making sure that the leadership was given a long enough leash to be able to adapt quickly and effectively.

When it came to building the business, creating the core technology was easy. Their first version vastly outperformed the mobile operators. One of the biggest issues was the diversity of Mobile OS. With Blackberry not opening their API and Nokia having 3 OS versions – each not upgradable, this was no easy feat. [On a side not he made a prediction that Blackberry will be in serious decline in the next few years… This is because the customer is not the center of their development. Also that Android will soon overtake the iPhone due to its openness.] With Microsoft being years behind the game they have chosen to focus on writing software for Android, iPhone and Blackberry only.

When looking at moving the main routing through a central place they originally looked in India but soon moved to Israel because of the language barrier.

Deciding to run on a model where companies only pay for what they use and IS covers the setup cost they have managed to bring their prices down, starting at 50c per minute.

As Take-Aways Justin left us with 3 words: Usability, Adapt and Love. For him these were the most important ingredients in their success. Creating the customer at the center of their development was absolutely critical as well as being adaptable as situations and roll-outs differ. Love was needed to be inherent in the product and that “People want to see caring in your eyes”…

For more details, find Justin Spratt’s presentation slides here.

Roger Norton

Mesh Potatos Become Hot Potatos

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

The Tech track at Tech4Africa kicked off with a presentation by Steve Song, a Shuttleworth Foundation fellow and a founder of Village Telco project.
His main theme was the high cost of access to basic communications, and the solution developed that can help solve this problem. Which is always a good thing in a solution.
An eye-opening statistic: the bottom 75% income band in Africa spends 50% of their disposable income on mobile services (in SA it’s a touch lower, at 40%, although probably because people earn a bit more here than because our mobile operators are particularly cheap).
If you’re an economist you get to sit back smugly, take a sip of Merlot and say, “See, you see how much they value it? Clearly it delivers the incentive they believe makes it worth the price.” If you’re not an economist you kick them in the head.
If you’re an African leader you wake the hell up, and realise that without driving down the cost of telephone and data access, your people will struggle and struggle to advance (and hence build businesses that pay you tax revenues that leaders can in turn plunder for new presidential palaces, you bastards).
In Steve’s more considered words, “What might an enlightened African leader should say: ‘It’s the cost of access, stupid’.”
His main contention in developing businesses is that to enable innovation, you must lower the cost of failure. When failure is expensive, the lesson learned is to stop wasting money.
With greater communication, more sharing of knowledge and greater transparency across markets comes immediate and marked growth in GDP. A cheaper way to provide telephony services in Africa is vital, and since the mobile operators (and government regulators) are not coming to the party, other options must be found.
His talk then turned to an old Linksys wireless router, the WRT54G. Popular, now getting a little long in the tooth, but based on a Linux core in its firmware. Open Source types compelled Linksys to open up the code under GPL, and they dug in.
They hacked it, and discovered it was a pretty powerful box, with hardware (RAM, CPU, etc) more advanced than its as-shipped specs needed. A whole new industry of hacked WRT54G routers grew up… enthusiasts wrote books about it, codified the OS as OpenWRT.
You can unplug the stub antenna, and add a directional (such as a cheap, home-made cantenna), and talk reliably over many kilometres.
You could hook a bunch together over in a mesh over an area, plug a phone into it, hack it a bit more, and end up with the Mesh Potato.
Mesh, as in the Open Source mesh protocol developed substantially by a woman called Elektra in Germany, POTS as in Plain Old Telephone System (it works with old skool analogue telephone handsets) and an ATA (Analogue Telephone Adapter) to packetise the voice.
Mesh Potato. Get it? It’s cleverer in Spain, where they call them patatas.
The hardware for the box was designed by an Australian hardware engineer called Mr. David Rowe, and then mass produced.
Hacking together some code is easy enough, you may think, but a piece of hardware is an entirely different proposition.
Luckily, not true.
An interesting development, says Song: “Hardware is the new software. You can bring a unique hardware product to market quite quickly and cheaply.”
The Mesh Potato team finalised the design, did some prototypes for testing, then went into production with a manufacturer in China, who was only to happy to give them a good unit price in exchange for the know-how (yay for open source goodness and commodity technology like WiFi and open standards like SIP!).
The final device will be a fairly cute little white box that’s weatherproof (rain, dust and sun) with hardened ports that won’t fry its little heart if some klutz plugs the wrong thing in.
The mesh then connects up a number of standard handsets that can be pulled out the dumpster, allowing a non-profit to set up a local telephone network quickly and exceptionally cheaply.
The little network can be autonomous, or you can add a “server” that’s being developed that has a VoIP connection upstream to a gateway to allow interconnect to national operators.
Song recons they got a 400m range at a setup they ran at the Afrikaburn festival/party/freakshow in the Karoo in the Western Cape a few months back, and with a new version planned using 802.11n, they’re looking for even longer range.
The non-profit part is key, because these organisations can operate a network in many African countries free if they don’t charge. A small business can be set up to be a commercial venture, but then they’d need an ECS class licence (in SA, similar elsewhere).
The box was developed by a team that kicked around ideas and designs across South Africa, Kenya, Tanzania, Uganda.
Song says they’re so far not getting hostility from the incumbent mobile and landline operators, but probably because they’re flying well under the radar. The tolerance may change when the Mesh Potato starts to become widespread.
It’s an interesting future. In the past the only people with the ability to deliver a viable telephone network would have to do it on a national basis. Deep pockets. Exclusive licences. Monopolies. Price gouging. In incumbent operators’ defence, the price to pay for doing a national network is that it would have to meet minimum standards. Expensive, reliable, standardised technology. Think SS7.
That changes with repurposed commodity technology.
Now a village or area in a township can set up a telephone network that services just them cheaply and easily. If it falls over, they know exactly whose door to go knock on. It does the needed job. It’s cheap. It can be grown, and grown.
Rather like ISPs providing VoIP services to small businesses will eat into national operators, so small consumer meshes will eat into national operators.
Is this the democratisation and commoditisation of telephony? These small potatos?

(NOTE: Blogging for the event is done with little time and bandwidth. Apologies for any factual errors, but c’est la vie).

By Roger Hislop

When Developers Get Antsy

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

It’s not often in SA that you get to hear a group of developers chew the fat. Well, you get it all the time. But not these developers. Tech4Africa 2010 offered an “Intimate Q&A” panel with Andy Budd (Clear Left), Dustin Diaz (Twitter, previously Google/Gmail), Joe Stump (SimplyGeo, previously Digg), John Resig (Mozilla Foundation (jQuery), Jonathan Snook (Yahoo!). These are guys that are at the coalface of the biggest, most successful Web development projects in the world.

It was a lively, and very funny discussion. First up were some general words of wisdom from Joe Stump, developer extraordinaire (see separate post on his presentation on scaling Web environments to global audience). He’s from the Valley… and he says his greatest asset there is his network. Many great developers, many great companies, many great brains, all sharing information and supporting others. Being commercial competitors doesn’t mean technology people shouldn’t help each other. Developers in Africa should build their networks, build their connections, and don’t be shy to ask for help and to share. Right. Enough serious stuff.

Some great quotable quotes:

* Can we ban the use of the word “Cloud”? Can we maybe use the word “Internet”?

* Question: Are frameworks stopping people investigating the depths of jscript? Answer: No-one wants to investigate the depths of jscript.

* Question: Will Flash be killed by HTML5?
Answer 1: Flash is the Cobol of the Internet.
Answer 2: It won’t go away for a long time, particularly for video.

* There are so many security holes in Flash, and people are driving buses through them.

* We’re getting clients saying, “Can you HTML5 our site?” and I think, “What the hell are you talking about?”

* The website is not the service, its just a gateway to the service.

* In the future, the browser is going to have more direct access to the hardware. The browser will become the OS, with more power and features.
(Ed’s note: What happens when the browser is so powerful and hardware-connected it will replace the OS. Will we then need a small, lightweight browser in the big fat OS-browser?)

* To go global you have to work on a baseline User Interface – don’t just develop for latest browsers, computers and phones.

Roger Hislop

The next big thing is content

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

The race to lay fibre optics cables into Africa is on, with around 10 cables either landed or planned by 2012*. What does this mean for business, consumers, rural areas, and communications service providers themselves?

An interesting debate emerged under a session about “The promise of fibre, the last mile, wireless and ubiquitous bandwith”. MTN, Vodacom, Infraco and Seacom spoke about their competing interests and how collaboration is crucial for the future. So far, mobile operators are collaborating on laying cables.

But that’s pretty much where it stops.

Everyone on the panel –Duncan McLeod (Techcentral), Stafford Masie (Seacom), Angela Gahagan (MTN Business), Pieter Uys (Vodacom),Dave Smith (Broadband Infraco)- recognized that the proliferation of cables coming into Africa means that costs will inevitably come down – it’s just a matter of time. And that introduces issues of sustainability for businesses currently working in this space. What are they going to do when competition drives down the price of infrastructure costs and their revenue models are threatened?

Seacom’s Stafford Massie stuck his neck out and said it’s time to move the game on from discussions about infrastructure. He suggested that providers are currently engaging international content owners in a very predatory fashion – wanting exclusivity in their deals – because they’re still obsessed with ownership of infrastructure (see individual cables and towers, placed right next to each other). However, content owners (such as Google, Youtube, Facebook and others) want more openness and are not prepared to be locked into MTN exclusive or Vodacom exclusive deals. Stafford suggests that for the future, the discussion should be about how operators can diversify and monetise new business models.
Massie believes that the way to make money in this space is in content and application services with a pan-African open access platform.
Content however, is a different game to infrastructure. Content models work much better with co-operation, alliances and openness, rather than secretive competition. You have to be more innovative about how you make your money. The telecoms giants in south Africa are possibly too comfortable in their dominant ownership of the space and may well see themselves losing large chunks of revenue to more open and creative players over time.

Angela Gahagan of MTN says they do have value added services, and products that focus on cloud computing, but insists that without the infrastructure, you can’t look at the add-ons. Pieter Uys of Vodacom was equally defensive about the need for infrastructure. There was recognition that providers still need to deliver on terrestrial infrastructure and establish additional footprint to get to underserviced areas. However, apart from SEACOM (represented by Massie) none of the big guns seemed to be able to lift their eyes beyond the current model of getting more subscribers and traffic through “owning” the infrastructure. Perhaps this is a reflection of the bedeviled ICT regulatory environment in South Africa.

Both the panel and participants spoke about how looser regulation allows for a whole different approach to ICT innovation. Kenya has managed to achieve far more Internet penetration and telecoms services, largely because Kenya’s ICT space isn’t highly regulated, allowing plenty of opportunity for innovation and entrepreneurship. In discussing leadership, participants said they have tried to work with politicians but so far it just hasn’t worked, so what next? Stafford Massie encouraged people (especially individuals who have the government’s ear) to try again – he said that politicians are not technological people. Things have changed so much in the last 2 years that it’s worth going back to politicians to show them examples of how technology can transform peoples’ lives.

More details on this topic on this podcast with Stafford Massie.

* The current and planned cables include South Asia Telecom Cable (SEACOM), SAT-3, Main-one, Glo-one, East African Submarine Cable System (EASSY), The Eastern African Marine Systems (TEAMS), and West Africa Cable System (WACS).

Samantha Fleming

Leila C. Janah inspires delegates to change the world

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

The keynote speaker for the first day of Tech4Africa 2010 was Leila Chirayath Janah from Samasource, a woman that every one of us should aspire to be like. She gave up a life in America to come to Africa to teach.

Out of this she founded Samasource, which allows people living in poverty to cross the digital divide. Leila surprised the audience with some astounding stats on education and literacy levels in Africa, which are higher than we all think. The real problem is lack of work, unemployment drives young people to join terrorist organisations in droves because it is the only way to earn money to feed their families.

Local trade and crafts does not make an impact on the poor either. Products made by poor people are often only bought by locals or a few tourists, making a maximum of $1 a day for the sellers.

Conversely, most corporations do not make products that are affordable for the majority of the 3rd world citizens. In Leila’s words, ‘we must engage the bottom of the pyramid if we are to eradicate poverty‘.

Samasource’s mission is to connect poor people to work via the internet, it builds tools to allow their customers to work with them without outsourcing to a large company.

According to Leila, “the Internet is not just an information superhighway, its a work superhighway”, we can use the tools and connectivity the internet provides to make a real difference to global unemployment.

Samasource’s key value proposition is the ‘virtual assembly lines’, where companies give work on a per task basis. This allows people to work from local centers and internet cafes, eliminating the need to travel long distances to big cities to find work.

The system, and Leila herself, is inspiring and taught every delegate at Tech4Africa today that we can use the internet to make a difference and we don’t have to be rich or famous to do it. We can only hope that local initiatives like Heather Ford’s ‘GeekRetreat’ can follow in Leila’s footsteps and we, as South African Geeks, can begin to bridge the gap between the connected and the rest.

Heidi Schneigansz