Introduction To Data Scraping With Python


Last Thursday, I gave a talk at PyconKE 2017 titled “Introduction to Scraping using Python”. This was a beginner-level introduction that used three cool Python libraries:

In the talk, I demonstrated how to use these libraries to programmatically access the Kenya Power & Lighting Company’s website and automatically fetch a monthly power bill.

Below is the full presentation and the accompanying code is on Github. Enjoy!

Dynamically Clean Shared Data with Filtered Datasets



A filtered dataset is a subset of submitted data that satisfies certain conditions set by the user. Filtered datasets are helpful when you need to share a dataset with other users, yet want to keep certain fields or records private that are repetitive, irrelevant or sensitive.

Suppose you carried out a baseline survey on households in several drought-stricken states, and you would like to have local analysts access data that’s only from their individual state. In this case, you could create multiple filtered datasets (one from each state) that segment data from each state — no external tools needed.

If the data is sensitive, the filtered datasets can go into unique projects that are selectively shared with specific users. If the data is not sensitive and you just wanted to make the data easier to understand, all of the filtered datasets can go into the same project.

 Continue reading Dynamically Clean Shared Data with Filtered Datasets...

Streaming Ona Data with NiFi, Kafka, Druid, and Superset


A common need across all our projects and partners’ projects is to build up-to-date indicators from stored data. We have built dashboards showing project progress and other stakeholder-relevant information in our malaria spraying project (mSpray), drought response monitoring project in Somalia, and electronic medical record system (OpenSRP). Currently we create indicators on an ad-hoc basis, but we are in the process of building a unified pipeline to move data from heterogenous systems into a data warehouse and build indicators on top of this data.

This need breaks down into the following minimal requirements:

  1. Store data
  2. Store queries relative to the data
  3. Retrieve the results of queries executed against the latest data

 Continue reading Streaming Ona Data with NiFi, Kafka, Druid, and Superset...

An inside look at Zambia’s new electronic immunization registry


The following was originally written for and posted on the BID Initiative blog. It provides an update on the Zambian electronic immunization registry we’re building with Path, which we introduced in a previous post.

Ona Path Zambia
Photo: Ona. Zambia’s paper-based immunization reporting systems of the past, are time-intensive and often result in longer wait times for patients.

Back in February, I wrote about Ona’s visit to Livingstone, Zambia to work with PATH’s BID Initiative, the nurses in the health facilities, and other project stakeholders to collectively define a health information system that would ease the data collection and reporting burden of the nurses, while at the same time making their data more accessible and useful to them during their work. Coming home from that visit, we had a pretty good understanding of what the nurses’ pain points are, mostly around data entry and reporting requirements. After a rapid and iterative development phase, the app’s first release was launched in April and given to nurses from six health facilities in Livingstone District for the first phase of testing.

 Continue reading An inside look at Zambia’s new electronic immunization registry...

Ona This Month May 2017 Newsletter



What’s new?

Help site revamp

We are excited to announce our new help site, which makes finding answers faster with improved search functionality and article categories. The new site also lets you vote on whether articles are helpful so we can improve the content. Please note, you may experience 404 errors if you use links to the old website — in that case, use the search engine to find what you are looking for.

 Continue reading Ona This Month May 2017 Newsletter...