Guest post by John Etherton, Ushahidi Liberia's lead tech consultant. John lived in Liberia for three years starting in 2008; during that time he worked with Georgia Tech, the Clinton Foundation, Crisis Management International and USAID. John is now based in Denver, USA and is the Managing Partner of Etherton Technologies, a consulting firm focused on software engineering for developing contexts.
[caption id="attachment_5244" align="alignleft" width="500" caption="Androids running SMSSync at Ushahidi Liberia"]
[/caption]
These days technology just works. There's no magic and very little rocket science; put in some 1s and 0s and get out 1s and 0s. The days are gone when bugs, actual insects, would chew their way through the computers wire and cause mayhem.
Now some of you right now are saying, “Yeah, but I still can't figure out how to make my Facebook profile private.” That's a user interface issue; the underlying technology is working perfectly. The layer that exposes that technology to you may be poorly designed, but that's not what we're here to discuss.
Yes technology hums along gloriously – until certain assumptions no longer hold true. Such assumptions usually include constant electricity and Internet, two things that are not constant in Liberia. Thus what seems like a perfect combination - SMS technology and Liberia - has a few obstacles to overcome before riding off into the sunset happily ever after.
For us at Ushahidi Liberia, we wanted SMS and Liberia hit it off. SMS is a great way for people on the ground to send in reports of what's really happening. All you need is a phone and a cell phone signal. Most places in the world have cell phone coverage, and an increasing number of people have cell phones. The Ushahidi platform even has built-in support for SMS because it has worked well in other deployments. So let’s get to know more about the compatibility of this couple in particular – SMS and Liberia.
SMS
Simple Messaging Service is a GSM standard that uses extra bandwidth in the signaling path that controls call flow. The signaling path is used to tell a cell phone that a new call is coming in, that the call has been hung up, the number of the incoming call and so forth. Since the signaling path isn't used when there is no phone call, phone companies realized they could add a messaging service on top of this unused signaling path – and SMS was born. Because the signaling path is only intended for short messages like, “incoming call +231-6-555-343”, SMS messages can only be 160 characters long.
Liberia
Liberia is a small country in West Africa still recovering from a civil war that devastated the nation’s infrastructure. Despite the lack of power lines, power stations and any kind of hard-wired telecommunications infrastructure, Liberia now has a relatively robust cell phone network. While less than 1% of Liberians have access to the Internet, at least 20% own cell phones and even more have access to shared phones. SMS seems like a natural choice for Liberians to send messages to our Ushahidi Liberia sites, given that even the cheapest cell phones available in-country support SMS.
[caption id="attachment_5249" align="alignright" width="500" caption="Half a powerline: Liberia's damaged infrastructure"]
[/caption]
The Trials of Matchmaking
At this point, things look good for our two lovebirds. A simple global standard for sending short messages and a country where the telecommunications infrastructure is best equipped to handle short messages. In fact, such a partnership has worked so well in other countries that the wonderful people at kiwanja.net created FrontlineSMS. FrontlineSMS is a program that turns your average laptop and a cell phone into an SMS gateway for bulk messaging and routing incoming SMS to the Internet. By connecting a cell phone to a computer via a data cable, FrontlineSMS can intercept incoming SMS and automatically send them to a website of your choosing – an ideal matchmaker for SMS and Liberia.
However, it is here we encountered our first obstacle. Computers need power. You can get around this requirement for a few hours by using a laptop, but the problem persists. Ushahidi Liberia’s first office used a diesel generator for electricity, like most Liberian workplaces that do not have access to a power grid. Diesel isn't cheap in Liberia, so the generator only ran from 9am to 7pm; that leaves 14 hours without power. In other words, 14 hours without the ability to send and receive SMS via a desktop SMS gateway. Because many of the reports sent to our Liberia instances about conflict and instability, we could not afford to be operational for only 10 hours a day.
The second obstacle was an unreliable Internet connection. In Liberia, all Internet connections are via satellite – far slower and more expensive than fiber optic connections. Ushahidi Liberia’s first ISP leased a satellite Internet connection comparable to a slow DSL connection in the US – comparable until the ISP splits that connection among all of its customers, who then split their slice of the connection amongst all the users in the office/home. During peak working hours as many as 500 people were using that one Internet connection, causing it to drop out when bandwidth was exceeded, and causing any user to grind their teeth in frustration when it was working but oh so slow.
When the Internet dropped out in our office and FrontlineSMS received an SMS to forward to Ushahidi servers, the sending would fail and FrontlineSMS would drop the message without resending. At that time, FrontlineSMS did not notify the sender or the receiver that a message failed to send, so it could be days before the person running the FrontlineSMS instance might realize that half the sent messages were missing. FrontlineSMS, like many platforms, is designed with the assumption that the Internet works. Granted, we were using FrontlineSMS as an SMS gateway to the Internet and not solely as a bulk messaging system, so our needs were specific.
SMSSync to the Rescue
Considering these glitches, our couple might not be as compatible as we thought – that is until SMSSync came along. SMSSync is an Android app written by the Ushahidi team that replaces the computer as the intermediary and runs the SMS gateway program on the phone itself. Since Android phones can connect to the Internet via WiFi and GPRS, they can receive an incoming SMS and then send it out over the Internet all by themselves. This solved our first problem of power. Phones can easily run 14 hours, or a couple of days, without recharge; that's what they're made to do. But we still have the issue of unreliable Internet.
[caption id="attachment_5247" align="alignleft" width="500" caption="SMSSync Android app"]
[/caption]
To address the Internet, our team worked closely with SMSSync’s creator, Henry Addo, to incorporate a resend function that repeatedly tries sending a message until it is received by the target URL. Now messages can be received by SMSSync even when the Internet is out and they stay in a holding pattern until the connection returns.
At this point it seems like everything is going to work out for SMS and Liberia, but not so fast.
There's still one more assumption suggested earlier – that humans will interact with technology correctly. Even the most well-intentioned, experienced user will occasionally get it wrong. The civil war held up the arrival of the latest technologies, and as a result many Liberians don't have extensive experience with technology. We noticed that every so often we'd receive a blank message from our users in the field; someone probably hit send prematurely. At first we didn't pay much attention to this, but then we noticed that SMSSync would stop sending messages after receiving blanks.
After a lot of hair-pulling, we realized the glitch was related to the Ushahidi platform itself. It was programmed to reject blank SMSs as erroneous. SMSSync would try to forward the blank SMS to Ushahidi, Ushahidi would reject it, SMSSync would wait 5 minutes and try again, meanwhile all the other messages were waiting in line. We reprogrammed the Ushahidi platoform to accept all messages, blank or otherwise. Messages that appeared to be errors would be marked as such for the users of the Ushahidi platform to decide what to do with them.
We also had a similar problem with promotional SMS from the cell phone companies. They'd send out things like, “Talk free this Saturday” and often these messages wouldn’t be from numbers like, “06-555-123”, but rather from “winBig” or “LonestarCell.” Again, we didn't think much of this, but SMSSync stopped working shortly after receiving these messages. It turns out that the Ushahidi platform is also set to reject SMS from numbers that aren't numbers, sending an app like SMSSync into a never-ending loop. We also fixed this in the Ushahidi platform.
At this point we've accounted for assumptions about electricity, Internet and human users. The final assumption we had to overcome is that the technology will always, from now to eternity, until you tell it otherwise, do what you want. But here’s a twist: in an effort to save battery power, Android phones are programmed to turn off their WiFi radios after a certain period of inactivity. Thus SMSSync would work brilliantly for awhile, but then stop forwarding messages for no apparent reason. We'd look at the phone, see the unsent SMS, and since we were now using the phone the WiFi would come back on and mysteriously work. Much to our embarrassment, this also took a long time to figure out.
It’s the Simple Things
At this point I'd like to make it clear that 99% of the time it's the simple things that get in the way. It wasn't some small manufacturing defect in our phones, it wasn't a rogue bit of code deep in SMSSync, it was just a simple feature of the phones that, when they aren't working as SMS gateways, works to the user’s advantage.
[caption id="attachment_5248" align="alignright" width="252" caption="KeepScreen Android app"]
[/caption]
We fixed the WiFi automatic sleep by installing KeepScreen on the phones. KeepScreen is a free Andoid App that just keeps the screen on all the time. By keeping the screen on, and making the Android think the user is still using the phone, the WiFi also never goes off.
Now our phones work perfectly, day and night, through power and blackouts, high bandwidth bliss and connection timeouts, to send SMSs from our users on the ground to our servers high in the Internet’s metaphorical cloud. And after much hardship, Liberia, a beautiful country, and SMS, a messaging protocol of elegant simplicity, are together at last.
[/caption]
These days technology just works. There's no magic and very little rocket science; put in some 1s and 0s and get out 1s and 0s. The days are gone when bugs, actual insects, would chew their way through the computers wire and cause mayhem.
Now some of you right now are saying, “Yeah, but I still can't figure out how to make my Facebook profile private.” That's a user interface issue; the underlying technology is working perfectly. The layer that exposes that technology to you may be poorly designed, but that's not what we're here to discuss.
Yes technology hums along gloriously – until certain assumptions no longer hold true. Such assumptions usually include constant electricity and Internet, two things that are not constant in Liberia. Thus what seems like a perfect combination - SMS technology and Liberia - has a few obstacles to overcome before riding off into the sunset happily ever after.
For us at Ushahidi Liberia, we wanted SMS and Liberia hit it off. SMS is a great way for people on the ground to send in reports of what's really happening. All you need is a phone and a cell phone signal. Most places in the world have cell phone coverage, and an increasing number of people have cell phones. The Ushahidi platform even has built-in support for SMS because it has worked well in other deployments. So let’s get to know more about the compatibility of this couple in particular – SMS and Liberia.
SMS
Simple Messaging Service is a GSM standard that uses extra bandwidth in the signaling path that controls call flow. The signaling path is used to tell a cell phone that a new call is coming in, that the call has been hung up, the number of the incoming call and so forth. Since the signaling path isn't used when there is no phone call, phone companies realized they could add a messaging service on top of this unused signaling path – and SMS was born. Because the signaling path is only intended for short messages like, “incoming call +231-6-555-343”, SMS messages can only be 160 characters long.
Liberia
Liberia is a small country in West Africa still recovering from a civil war that devastated the nation’s infrastructure. Despite the lack of power lines, power stations and any kind of hard-wired telecommunications infrastructure, Liberia now has a relatively robust cell phone network. While less than 1% of Liberians have access to the Internet, at least 20% own cell phones and even more have access to shared phones. SMS seems like a natural choice for Liberians to send messages to our Ushahidi Liberia sites, given that even the cheapest cell phones available in-country support SMS.
[caption id="attachment_5249" align="alignright" width="500" caption="Half a powerline: Liberia's damaged infrastructure"]
[/caption]
The Trials of Matchmaking
At this point, things look good for our two lovebirds. A simple global standard for sending short messages and a country where the telecommunications infrastructure is best equipped to handle short messages. In fact, such a partnership has worked so well in other countries that the wonderful people at kiwanja.net created FrontlineSMS. FrontlineSMS is a program that turns your average laptop and a cell phone into an SMS gateway for bulk messaging and routing incoming SMS to the Internet. By connecting a cell phone to a computer via a data cable, FrontlineSMS can intercept incoming SMS and automatically send them to a website of your choosing – an ideal matchmaker for SMS and Liberia.
However, it is here we encountered our first obstacle. Computers need power. You can get around this requirement for a few hours by using a laptop, but the problem persists. Ushahidi Liberia’s first office used a diesel generator for electricity, like most Liberian workplaces that do not have access to a power grid. Diesel isn't cheap in Liberia, so the generator only ran from 9am to 7pm; that leaves 14 hours without power. In other words, 14 hours without the ability to send and receive SMS via a desktop SMS gateway. Because many of the reports sent to our Liberia instances about conflict and instability, we could not afford to be operational for only 10 hours a day.
The second obstacle was an unreliable Internet connection. In Liberia, all Internet connections are via satellite – far slower and more expensive than fiber optic connections. Ushahidi Liberia’s first ISP leased a satellite Internet connection comparable to a slow DSL connection in the US – comparable until the ISP splits that connection among all of its customers, who then split their slice of the connection amongst all the users in the office/home. During peak working hours as many as 500 people were using that one Internet connection, causing it to drop out when bandwidth was exceeded, and causing any user to grind their teeth in frustration when it was working but oh so slow.
When the Internet dropped out in our office and FrontlineSMS received an SMS to forward to Ushahidi servers, the sending would fail and FrontlineSMS would drop the message without resending. At that time, FrontlineSMS did not notify the sender or the receiver that a message failed to send, so it could be days before the person running the FrontlineSMS instance might realize that half the sent messages were missing. FrontlineSMS, like many platforms, is designed with the assumption that the Internet works. Granted, we were using FrontlineSMS as an SMS gateway to the Internet and not solely as a bulk messaging system, so our needs were specific.
SMSSync to the Rescue
Considering these glitches, our couple might not be as compatible as we thought – that is until SMSSync came along. SMSSync is an Android app written by the Ushahidi team that replaces the computer as the intermediary and runs the SMS gateway program on the phone itself. Since Android phones can connect to the Internet via WiFi and GPRS, they can receive an incoming SMS and then send it out over the Internet all by themselves. This solved our first problem of power. Phones can easily run 14 hours, or a couple of days, without recharge; that's what they're made to do. But we still have the issue of unreliable Internet.
[caption id="attachment_5247" align="alignleft" width="500" caption="SMSSync Android app"]
[/caption]
To address the Internet, our team worked closely with SMSSync’s creator, Henry Addo, to incorporate a resend function that repeatedly tries sending a message until it is received by the target URL. Now messages can be received by SMSSync even when the Internet is out and they stay in a holding pattern until the connection returns.
At this point it seems like everything is going to work out for SMS and Liberia, but not so fast.
There's still one more assumption suggested earlier – that humans will interact with technology correctly. Even the most well-intentioned, experienced user will occasionally get it wrong. The civil war held up the arrival of the latest technologies, and as a result many Liberians don't have extensive experience with technology. We noticed that every so often we'd receive a blank message from our users in the field; someone probably hit send prematurely. At first we didn't pay much attention to this, but then we noticed that SMSSync would stop sending messages after receiving blanks.
After a lot of hair-pulling, we realized the glitch was related to the Ushahidi platform itself. It was programmed to reject blank SMSs as erroneous. SMSSync would try to forward the blank SMS to Ushahidi, Ushahidi would reject it, SMSSync would wait 5 minutes and try again, meanwhile all the other messages were waiting in line. We reprogrammed the Ushahidi platoform to accept all messages, blank or otherwise. Messages that appeared to be errors would be marked as such for the users of the Ushahidi platform to decide what to do with them.
We also had a similar problem with promotional SMS from the cell phone companies. They'd send out things like, “Talk free this Saturday” and often these messages wouldn’t be from numbers like, “06-555-123”, but rather from “winBig” or “LonestarCell.” Again, we didn't think much of this, but SMSSync stopped working shortly after receiving these messages. It turns out that the Ushahidi platform is also set to reject SMS from numbers that aren't numbers, sending an app like SMSSync into a never-ending loop. We also fixed this in the Ushahidi platform.
At this point we've accounted for assumptions about electricity, Internet and human users. The final assumption we had to overcome is that the technology will always, from now to eternity, until you tell it otherwise, do what you want. But here’s a twist: in an effort to save battery power, Android phones are programmed to turn off their WiFi radios after a certain period of inactivity. Thus SMSSync would work brilliantly for awhile, but then stop forwarding messages for no apparent reason. We'd look at the phone, see the unsent SMS, and since we were now using the phone the WiFi would come back on and mysteriously work. Much to our embarrassment, this also took a long time to figure out.
It’s the Simple Things
At this point I'd like to make it clear that 99% of the time it's the simple things that get in the way. It wasn't some small manufacturing defect in our phones, it wasn't a rogue bit of code deep in SMSSync, it was just a simple feature of the phones that, when they aren't working as SMS gateways, works to the user’s advantage.
[caption id="attachment_5248" align="alignright" width="252" caption="KeepScreen Android app"][/caption]
We fixed the WiFi automatic sleep by installing KeepScreen on the phones. KeepScreen is a free Andoid App that just keeps the screen on all the time. By keeping the screen on, and making the Android think the user is still using the phone, the WiFi also never goes off.
Now our phones work perfectly, day and night, through power and blackouts, high bandwidth bliss and connection timeouts, to send SMSs from our users on the ground to our servers high in the Internet’s metaphorical cloud. And after much hardship, Liberia, a beautiful country, and SMS, a messaging protocol of elegant simplicity, are together at last.