What is Open Data-Have You Heard About It?

Nowadays, We are becoming internet savvy. Most of us are dependent on the data such as the documents like DOCX, PDF, PPT, XSL, images in the extension of JPEG, PNG, audio, video and so on which are published openly and freely available for all of us. We took the advantages of those data in many ways like students, academician, the researchers used it for literature review and for citation. For the statisticians, economist, policymakers data are the main sources of fact-finding and formulating new strategies. All these examples show how important the data is. In this blog, here I will be sharing the ideas on "Open Data" on the basis of the "open data awareness program" organized by Kathford R&D's wing 'KATHFOSS' in the premise of Kathford International College on 2017-11-22. The resource persons for the event were Mr. Nikesh Balami, CEO, Mr.Shubham Ghimire, COO, Mr. Sagar Ghimire, CTO of an emerging organization -Open Knowledge Nepal. 

The event was conducted within three sessions. Among them, Here is the summary of Session I:

A. What is Open Data?

According to Nikesh Balami, CEO of open knowledge Nepal, To be open data there should be at least three criteria:

i. Data should be available on the Internet: 

It should be accessed freely through the websites and data portals.

ii. Data should be machine-readable: 

With the extension of data PDF->XLS-> CSV->KDF->LOD ( also known as 5 star open data)

iii. Data should be open licensed: 

Data that does not explicitly have an open license is not open data.

B. Benefits of Open Data

i. For the government:

It increases the transparency and accountability of the government, hence develops trust to the public since Right to Information (RTI) Act 2007 guarantees that Nepali citizens can access information on the functioning of any ‘public body’ in order to make governance and policymaking more transparent and accountable. National Information Commision (NIC) is responsible for the promotion and implementation of this RTI.

ii. For the students, academicians, researchers, entrepreneurs and startup firms:

It helps for research and doing innovative projects, supports for business growth, E-learning and so on.

C. Principle of Open Data

• Completeness
• Timeliness
• Primacy
• Access
• Machine readability
• Non-discrimination
• Use of commonly owns standards
• Licensing
• Performance
• Usage Cost

Now, Here is the summary of Session II

A. Current Situation in Nepal:

There are few stakeholders in this sector: Civil Society Organization (CSO) like Kathmandu Living Labs, Open Nepal, Open Knowledge Nepal, Freedom Forum, Code for Nepal, Accountability Lab, Bikas Udhyami etc. work in policy research, advocacy, tech, journalism and so on.  

List of CSOs (Source: Open data Manual-Compiled by OPen Knowledge Nepal) 

Kathmandu Living Labs: http://kathmandulivinglabs.org

 Open Nepal: http://opennepal.net

 Open Knowledge Nepal: http://oknp.org

 Freedom Forum: http://freedomforum.org.np

 Code for Nepal: http://codefornepal.org

 Accountability Lab: http://accountabilitylab.org

 Bikas Udhyami: http://bikasudhyami.com

B. Open data source for Nepal

The idea of open data entered Nepal in early 2013.The published data is still not available in open format (most of the data are published in PDF format). In spite of having rights to request and get data through the RTI Act with any government association, the Act does not a have lawful arrangement to pressurize government offices to open up their information. 

List of some government data sources (Source: Open data Manual-Compiled by OPen Knowledge Nepal)

 Official Portal of Government of Nepal: https://www.nepal.gov.np

 National Planning Commission: http://www.npc.gov.np/en

 Central Bureau of Statistics: http://cbs.gov.np/home

 Ministry of Finance: http://mof.gov.np/en/

 Nepal Rastra Bank: https://www.nrb.org.np

 Ministry of Home Affairs: http://www.moha.gov.np/home

 Ministry of Education: http://www.moe.gov.np

 Ministry of Health: http://mohp.gov.np

 Election Commission Nepal: http://election.gov.np

 Office of Company Registrar: http://ocr.gov.np

 List of some international data sources: 

 World Bank: https://goo.gl/wFjYgH

 United Nations: https://goo.gl/UGUoCh

 UN Digital Repository in Nepal: http://un.info.np

 UNICEF Nepal: https://goo.gl/QwHhJL

 World Food Programme: https://goo.gl/2sG2aS

 List of some CSO data sources: 

 Open Nepal: http://data.opennepal.net

 Election Nepal: http://electionnepal.org

 Nepal in Data: https://nepalindata.com

 NepalMap: http://nepalmap.org

This session was followed by the demo of nepalmap.org and nepalindata

Finally, the Session III spread following info:

A. The process of working with data

i. Data Extraction

The process of retrieving data out of non-machine-readable or unstructured data sources ( web pages, emails, pdf documents, scanned documents and so on). Basically, we could not access raw data from these unstructured data. There are followings ways of extracting data from PDF:

•Word/Excel converters to extract text from PDF: https://www.pdftoexcelonline.

com

• Programming, with some libraries existing for Python, Java, and the command line.

• Using Tabula - an offline open-source software specifically designed to get data out of PDF documents.

Other data extraction tools:

a.Basic scraping tools

 Tabula: http://tabula.technology/

 Import.io: https://import.io/

b. Extracting data with Python

 BeautifulSoup: https://pypi.python.org/pypi/beautifulsoup4

 Python Mechanize: https://pypi.python.org/pypi/mechanize/

 Scrapy: http://scrapy.org/

c. Web scraping tools

 ParseHub: https://www.parsehub.com/

 ScraperWiki: https://scraperwiki.com/

 OutWit Hub: https://goo.gl/1Axk88

 Scraper: https://goo.gl/DZRGZd

ii. Data cleaning

It is the process of fixing errors, duplicity, and format/ standard inconsistencies of extracted data. Tools and language: Spreadsheet, open refine, Python

iii. Data analysis

It is the process of examining and exploring datasets in order to generate

required information.

Online/offline open tools of data analysis

 Tableau Public: https://public.tableau.com/s/

 OpenRefine: http://openrefine.org

 RapidMiner: https://sourceforge.net/projects/rapidminer/

 Google Fusion Tables: https://goo.gl/XEFUVB

 KNIME: https://www.knime.com

 Import.io: https://www.import.io

iv. Data visualization

It is the presentation of data in a pictorial and graphical format.

a. Non-Developers Visualization Tools

 Datawrapper: https://datawrapper.de/

 Infogram: https://infogr.am/en

 Tableau Public: https://public.tableau.com/

 Raw: http://raw.densitydesign.org/

 Plotly: https://plot.ly/

 Timeline JS: http://timeline.knightlab.com/

 ChartBlocks: http://www.chartblocks.com/

 Plotly: https://plot.ly/

b. Developers Visualization Tools

 D3.js: http://d3js.org/

 FusionCharts: http://www.fusioncharts.com/

 Chart.js: http://www.chartjs.org/

 Google Charts: https://developers.google.com/chart/

 Highcharts: http://www.highcharts.com/

c. Map-Based Visualization Tools

 Leaflet: http://leafletjs.com

 Mapbox: https://www.mapbox.com

 CARTO: https://carto.com

B. Publishing data

It is a process of releasing data in a published form for use and reuse by others.

Some of the most used open data formats are JavaScript Object Notation (JSON), Extensible Markup Language (XML), Resource Description Framework (RDF), Spreadsheets, Comma Separated Value (CSV) and Plain Text.

Note: Data that is published as an excel table within a PDF document, without an open license, is not open data because it cannot be easily managed or reused.

Recommended Publishing Medium

a. Existing Data Portals

 Open Nepal Data Portal: http://data.opennepal.net

 Open Knowledge Nepal DataHub: https://old.datahub.io/organization/nepal

b. Independent medium

 GitHub: http://github.com

 Google Drive: http://drive.google.com

 DropBox: http://dropbox.com

C. Open Data Licensing

Data that does not explicitly have an open license is not open data. Creative content, such as text, photographs, slides, and so on, should be licensed using

a Creative Commons. Similarly, Open Definition have the lists of recommended conformant licenses used by different countries: http://opendefinition.org/licenses/

Conclusion

Now, we are come to know about many issues and tools on open data. Hope this blog is helpful to explore and make understanding this new topic. I am heartily thankful towards the resource persons of Open Knowledge Nepal.

See Gallery