.png)
Open Source Data
With the existence of the Internet and Big Data comes the emergence of so-called open source data. What is open source data? Can open source data be useful for your business?
With the existence of the Internet and Big Data comes the emergence of so-called open source data. What is open source data? Can open source data be useful for your business? Find out more by reading this short overview.
What is open source data?
Open source data can be defined simply as data that anyone can access, use, and share. What does this mean?
- Anyone can access it - there are no restrictions in accessing the data. Restrictions can include requirements such as official requests that have a chance to be rejected and file formats that are not commonly used or are not up to industry standards.
- Anyone can use it - governments, industries, and individuals can use the data for any desired purpose. This also means that open data excludes sensitive data that can be used by competition.
- Anyone can share it - the data can be used, reused, and shared by other users.
Data is not free to host, so it is often government agencies and nonprofit organizations that take the initiative to host open source data. Open source data can also include licenses such as Creative Commons that do not restrict how you can use the data but specify how to properly attribute the source of the data.
Why should we use open source data?
Besides the fact that open source data is free to access, use, and share, here are some of the other benefits:
- Increased engagement with the market and the community - many open source data initiatives have a community that supports them. This is a good opportunity to place your brand more prominently in the community by supporting them for free.
- Increased transparency - because the data included in open source data often concerns governance, publishing relevant open source data helps increase transparency. How can your business benefit from this? The open source data, while not containing sensitive information, can include important economic data relevant to your industry. This will help you get a bird’s eye view of your market and plan your next marketing campaigns.
- More ways of interpreting the data - because anyone can interpret the data, they can choose different methods to highlight different aspects of the data that you may not have recognized. Additionally, analysts who chose the same method can serve as your benchmarks when you analyze the data yourself using the same method they use.
What should you remember before using open source data?
Open Data Indicator Europe lists four things one should consider when using open datasets.
First impressions matter
The first impression you get from the page containing the dataset matters! It reflects the amount of effort put into preparing the dataset. A high-quality open source dataset also has the following qualities:
- Easily accessibility
- Well-structured
- Clearly documented
These describe what you see first before downloading the dataset. Usually, the reputation of the source is enough to assure us of the dataset quality, but websites that host open data can also secure open data certificates to declare that the data they host is of high quality.
Datasets can have problems in its content
Even the datasets from reputable sources may contain issues in its content. Some of them are the following:
- Issues in the implementation of its schema (schema is the specification for a certain data format used)
- Contains invalid or incorrect values
- Missing data
- Precision problems that can snowball when processed through data analysis tools
These are not insurmountable problems, but you will need to spend time cleaning up the dataset so it can be properly processed. The process is called data munging.
Datasets need to be fixed before being analyzed
As we have stated in the previous section, we have introduced the concept of data munging. Data munging can spot possible issues in the implementation of the schema of the dataset, invalid values, and missing data.
For other errors such as incorrect values, they may require cross-checking with other sources such as master lists and standard sources to ensure their correctness.
Good quality sometimes depends on your needs
Finally, always remember: Perfection is the enemy of the good. Sometimes, you may feel the urge to dig through the Internet for the best dataset for your needs. However, these datasets may prove to be an overkill, usually because they contain extra data that may not be needed but can slow down the subsequent data analysis. These extra data can be safely scrubbed through data mugging.
What are some free open data sources over the Internet?
Here are some of the free open data sources that you can access over the Internet.
Global and regional data
- World Bank Open Data - contains both economic and development indicators data for most countries. You can access the list of indicators here as a starting point.
- World Health Organization’s Global Health Observatory (GHO) - contains health-related information for most countries.
- Registry of Open Data on AWS - contains hundreds of databases that are hosted on AWS, one of the biggest cloud service providers in the world. While most of the databases hosted are for purely scientific research, some of them are relevant for business applications. Each of them have their own documentation for accessing the information via APIs.
- data.europa.eu - contains more than a million datasets about European countries. Their catalog can be accessed here.
- UNICEF Data - contains worldwide data for the state of development of women and children. The dataset list can be accessed here.
National data
- United States Census Data - contains a wide range of data and statistics about the United States. The website allows you to view portions of datasets without downloading them in tabular, map, and page form.
- Data.gov - contains a wider variety of data and statistics about the United States.
- Open Data Network - contains a wide variety of data for selected geographical regions of the United States.
- Data.gov.uk and UK Data Service - contains a wide variety of data and statistics about the United Kingdom.
Business and financial data
- UN Comtrade Database - contains worldwide trade data.
- International Monetary Fund - contains worldwide economic and financial data.
- Financial Times Markets data - contains daily financial data from different markets worldwide.
- Nasdaq Data Link - contains US financial and market data
- OpenCorporates - contains corporate data from thousands of companies and corporations worldwide. Their data became useful in uncovering illicit activities worldwide.
- U.S. Bureau of Labor Statistics - contains economic and employment data.
Social trends data
- Socialmention - tracks trends real-time over the social networks.
- Google Trends - tracks Google search trends over time. Data is available by country.
- Pew Research Datasets - contains Pew Research survey data on social trends over the years.
- BuzzFeed News · GitHub - contains data used by Buzzfeed in their articles.
- Data | FiveThirtyEight - contains datasets for political trends
Open data for data science, machine learning, and app development
- Yelp Open Dataset - contains business-related data and reviews data for more than 150,000 businesses. You can use it to train algorithms or as sample data for app development.
- Kaggle Datasets - contains thousands of datasets for training for data science skills, machine learning and AI development, or for sample data for app development.
- UCI Machine Learning Repository - contains hundreds of datasets for machine learning training. Businesses nowadays take advantage of machine learning to improve their competitive edge.
- freeCodeCamp/open-data - sample data used by freeCodeCamp in their coding projects.
- OpenML Data Set - contains sample datasets for machine learning training
References
Four things you should know about open data quality
The Pros and Cons of Open Data
Open Data: What Is It and Why Should You Care?
6 Major Benefits of Open Data – ProWebScraper
These Are The Best Free Open Data Sources Anyone Can Use
50 Best Open Data Sources Ready to be Used Right Now
