Dear friends!
Without data, you’re just another person with an opinion. Every company is becoming more and more data-centric in order to identify actionable insights and stay competitive in the marketplace. Therefore, we are going to take a hot minute to discuss strategies and best practices to approach the problem of data sourcing. The goal is to equip you with an effective and robust data processing framework and lay a solid foundation for your decision-making process! Learn more about open source projects and alternative data collection in our ➡️next article Tracking Down Compelling Open Source Projects.
What is Data Sourcing?
Collecting information-rich data plays a pivotal role in the ecosystem of a company. The process to accomplish that is data landscaping and sourcing which includes multiple systematic steps to identify, evaluate, prioritize, extract, and store copious amounts of data from multiple internal or external sources that will be used for your data analysis. The insights generated can significantly affect the growth of the business and also easily become the Achilles’ heel of your organization if disregarded.
Data can come in a huge variety of structured or unstructured formats — everything from strictly formed relational databases to your last reel on Instagram. It might be extracted directly from the primary data source through interviews, surveys, etc. or pulled from a secondary data source such as your sales or accounting department. You will most likely also work with external providers such as data management providers, government institutions, or news channels. Primary data is raw while secondary data is usually collected for a specific purpose and therefore prepared and transformed accordingly.
Data sourcing is a multi-stage process that can be overwhelming and complicated. This strategy will provide you some guidance:
- Identify and prioritize data needs: Always start data sourcing activities by asking what you want to achieve and which sourcing activities have the expected ability to meet your organizational needs. Focus your sourcing activities on datasets that have the most impact on your desired business outcome. What data is needed to achieve this specific goal? What level of granularity is needed? What sample size is required for statistical robustness? What operational factors such as cost, data quality, control, and future collection must be considered?
- Find providers and evaluate quality: Profile your data in terms of structure, validity, age, frequency, granularity, and availability so that you can address these critical questions with your internal or external data provider. High data quality should be the top priority. Do as much due diligence as possible. Understand who the provider is, where they got the data from, and if/how they manipulated the data in any way.
- Select method of collection: Many providers will allow you to access their data through APIs or offer other convenient data formats suitable at a certain price. Web scraping, the technological process of extracting available web data in a structured format, can be a valid alternative to making the data accessible but requires an additional level of expertise, time, and effort. Manual data collection might seem cheaper but is very time-consuming and frequently introduces risks to data quality.
Data Sourcing Best Practices. To source data skillfully and unleash its maximum potential, we suggest the following best data sourcing practices routinely:
- Data Quality 1, limit manuals: Be cautious and try to minimize manual data extraction processes to limit quality issues, breaches, and inconsistencies. Explore options to switch to automatic solutions early on to allow scaling. Collect data as close as possible to the source in order to reduce uncontrolled and obscure manual intervention.
- Data Quality 2, the 1–10–100 impact rule: The further data moves across the supply chain, the more transformations take place, and the greater the cost of possible inaccuracy and failures. Therefore, it is crucial that you develop a clear understanding of what decisions are made based on the data you provide. Define and implement data quality metrics for each source and put action plans and possible alternative data sources in place in proportion to the business impact. Integrate quality checks into every single step of the data process.
- Data Quality 3, decay: Data decay refers to the degradation in the quality of data due to poor database maintenance, old or invalid records, mergers & acquisitions, or new systems and trends. Prepare for changes in your data landscape that force you to adapt. Discuss expectations, formulate tolerance levels, and implement trigger alerts that create an ever-evolving data processing system and ultimately allow you to stay ahead. According to Gartner, around 3% of the data decays every month globally.
- Collaboration: Collaboration with data suppliers is inevitable if you want to scale. Intra- and cross-company data collaboration is vital since it can help drive valuable insight. However, too many cooks (can) spoil the broth. Be aware of the greater data quality risk resulting from human error. Introduce version control, audit trails, and clear documentation guidelines to avoid uncontrolled data manipulation.
- Simplicity: Most datasets these days are too big and complex to analyze in their native form. Apply simplifications where possible. Define and apply systematic processes to your data (e.g. standardization) to enable fast queries, ease of visualization, and innovative adaptations.
The demand for data is exponentially increasing and data providers catching up at a high cost. There are, however, multiple free resources available you can use to build meaningful projects and tackle challenging real-world data. Here is a list of the most well-known:
- Kaggle — interesting and large variety of datasets and competitions!
- UCI Machine Learning Repository — great datasets to start with, easy to access
- Data.gov — the main repository of the US government’s open data project in the field of energy, climate, ecosystems, etc.
- AWS Open Data Registry — valuable datasets and a good way to start your cloud computing (EC2) journey
- Google Cloud Public Datasets — 100+ datasets hosted by BigQuery and Cloud Storage
- data.world — GitHub for data, lots of user contributed datasets. Additionally, the built tools to make working with data easier (SQL, R, Python)
- reddit — A unstructured mess but some interesting datasets!
Data sourcing is gaining significant interesting from data-centric corporations and most entrepreneurs. Properly sourced data saves organizations time, increases productivity across the board, and allows analytics to guide businesses in the right direction. But it requires a high level of accuracy, validity, completeness, reliability of communication and utilization which ultimately creates a single point of truth for the analytics team or other entities that will process the information. Data quality should not be taken lightly. In most cases, there aren’t any alarms that go off when bad data is used. Silent failures like this most likely directly impact your business’s bottom line and could even turn into a very loud problem and PR disaster. In summary, integrate clearly defined standards and metrics to identify the data quality issues at every single step of the data build process.
