Location Scaling and Herding Cats

News from the conference room: this is a series of blog posts in which blogging experts briefly review key Tech4Africa 2010 talks and panels from Day 1 and 2.

Day 1

Joe Stump (SimpleGeo) on scaling a business and handling developers.

The masses of data created in web 2.0 has started following Moore’s Law (according to Sergy Brin) and with this increase in data as well as real time tracking it the overload of information becomes a supply and demand problem. The more data you have, the less the data is worth.

There is a big drive for location based information, however the value of this information decreases drastically over distance and time. The relevance of this information depends on: What the information is; Who it is about and What information is Virtually nearby or relevent. With the large adoption of services like FourSquare and MyTown it becomes evident that this gaming style of applications providing relevant location based information are becoming very popular. [As a side fact, more people visit MyTown per day than the total number of people who have accessed FourSquare since its inception.] At the end of the day users are looking for relevant information that can enhance their current experience.

Moving onto online scaling, Joe stressed the necessity of automation with the cloud as well as separating data into partitions from the beginning being a must. An important question that needs to be answered is whether to scale Up or Out. Out is normally better when you are on a budget and expand by getting lots of basic storage; whereas Up would be investing in high-end servers that are generally very costly. The specific language that you use is more based on the application and are not really a factor when scaling.

Another crutial point is having set standards and conventions from the beginning is critical for continuity throughout the business. It lowers the barriers of entry for new team members and makes the different components work together more smoothly, as well as helping if you want to publicize code at a later date. (This became an issue at Digg) Providing this continuity was best done through regular communication between the teams and Joe is an advocate for the SCRUM model to ensure constant communication. Testing was also a must, with different components being tested automatically and peer reviewed before being uploaded. He also believes in a “Swiss Cheese” style of coding which leaves holes in the code to be filled up later.

Dealing with developers can often be a tricky task – mostly due to their passion and personality type. As the core generators of online services they need to be carefully lead through expansion and given enough freedom to work their magic.

Roger Norton