Big Data is the term that is circling everywhere in the field of analytics in the modern era. The rise of this term came about as the result of the enormous volume of unstructured data that is getting generated from a plethora of sources. Such voluminous unstructured data carries huge information which if mined properly could help a business achieve groundbreaking results.
Hence, it’s a wide range of applications has made Big Data popular among masses and everyone wants to master the skill associated with it to embrace the lucrative career opportunities that lie ahead. For the data professionals, many companies have various open positions in the job market and the number is only going to increase in the future.
Reason of the craze behind Big Data
The opportunities in the domain of Big Data is diverse and hence its craze is spreading rapidly among professionals from different fields like Banking, Manufacturing, Insurance, Healthcare, E-Commerce, and so on. Below are some of the reasons why its demand keeps on rising.
Talent shortage in Big data – Despite its every increasing opportunity, there is a significant shortage in the number of professionals who are actually trained to work in this field. Those who work in IT are generally accustomed to software development or testing, while people from other fields are familiar with spreadsheets, databases and so on.
However, the required skill to load and mine Big Data is missing significantly which makes it the job which is up for grabs for anyone who could master the skills. Business Analysts and managers along with the engineers need to be familiar with the skills required to work with Big data.
Variety in the types of jobs available – The term Big Data is somewhat holistic and could be misleading in defining the job descriptions for an open position. Even many people use this term in several situations without actually understanding the meaning behind its implementation.
There could be several job types available in the market which has the term Big Data in it. The domain of work could vary from Data analytics to Business analysis to Predictive analytics. It makes easier for one to choose among the various types and train oneself accordingly. Companies like Platform, Teradata, Opera, etc., have many opportunities in big data for their different business needs.
Lucrative salary – One of the major reasons why professionals are hopping onto the big data ecosystem is the salary that it offers. As it’s a niche skill, hence companies are ready to offer competitive packages to the employees. Those who want a learning curve and sharp growth in their career, big data could prove to be the perfect option for them.
As mentioned before, there are a variety of roles which requires big data expertise. Below are the opportunities based on the roles in the field of big data.
Big Data Analyst – One of the most sought after roles in Big Data is that of a Big Data Analyst. To interpret data and extract meaningful information from it which could help the business grow and influence the decision-making process is the work that a big data analyst does.
The professional also needs to have an understanding of tools such as Hadoop, Pig, Hive, etc. Basic statistics and algorithms knowledge along with the analytics skills is required for this role. For the analysis of data, domain knowledge is another important factor needed. To flourish in this role some of the qualities that are expected from a professional are –
Reporting packages and data model experience.
The ability to analyze both structured and unstructured data sets.
The skill to generate reports that could be presented to the clients.
Strong written and verbal communication skills.
An inclination towards problem-solving and an analytical mind.
Providing attention to detail.
The job description for a Big Data analyst includes –
Interpretation and the collection of data.
To the relevant business members, reporting the findings.
Identification of trends and patterns in the data sets.
Working alongside the management team or business to meet business needs.
Coming up with new analysis and data collection process.
Big Data Engineer – The design of a Big Data solutions architect is built upon by the Big Data engineer. Within the organizations, the development, maintenance, testing, and evaluation of the Big Data solutions is done by the Big Data engineer. They also tend to have experience in Hadoop, Spark, and so on, and hence are involved in designing Big Data solutions. An expert in data warehousing, who builds data processing systems and are comfortable working in the latest technologies.
In addition to this, the understanding of software engineering is also important for someone moving into the Big Data domain. Experience in engineering large-scale data infrastructures and software platforms should be present as well. Some of the programming or scripting languages a Big Data engineer should be familiar with are Java, Linux, Python, C++, and so on. Moreover, the knowledge of database systems like MongoDB is also crucial. Using Python or Java, a Big Data engineer should have a clear sense of building processing systems with Hive and Hadoop.
Data Scientist – Regarded as the sexiest job of the 21st century, a Data Scientist is regarded as the captain of the ship in the analytical Eco space. A Data Scientist is expected to have a plethora of skills stating from Data Analysis to building models to even client presentations.
In traditional organizations, the role of a Data Scientist is getting more importance as the way the old-school organizations used to work are now changing with the advent of Big Data. It’s now easier than ever to decipher the data starting from HR to R&D.
Apart from analyzing the raw data and drawing insights using Python, SQL, Excel, etc., a Data Scientist should also be familiar with building predictive models using Machine Learning, Deep Learning, and so on. Those models could save time and money for a business.
Business Intelligence Analyst – This role revolves around gathering data via different sources and also compare that with a competitor’s data. A picture of the company’s competitiveness would be developed by a Business Intelligence Analyst compared to other players in the market. Some of the responsibilities of a Business Intelligence Analyst are –
Managing BI solutions.
Through the applications lifecycle, provide reports and Excel VBA applications.
Analyze the requirements and the business process.
Requirements, design, and user manual documentations.
Identifying the opportunities with technology solutions to improve strategies and processes.
Identifying the needs to streamline and improve operations.
Machine Learning Engineer – A software engineer specialized in machine learning fulfils the role of a Machine Learning Engineer. Some of the responsibilities that a Machine Learning Engineer carries out are –
Running experiments with machine learning libraries using a programming language.
The production deployment of the predictive models.
Optimizing the performance and the scalability of the applications.
Ensuring a seamless data flow between the database and backend systems.
Analyzing data and coming up with new use cases.
Global Job Market of Big Data
Businesses and organizations have now put special attention to the full potential of Big Data. India has a large concentration of the jobs available in the Big Data market. Below are some of the notable points related to the job market of Big Data.
It is estimated that by 2020, the number would be approximately seven lakhs for the opportunities surrounding the role of Data Engineers, Big Data Developers, Data Scientists., and so on.
The average time for which an analytics job stays in the market is longer than the other jobs. The compensation for Big Data professionals is also 40%t more than other IT skills.
Apache Spark, Machine Learning, Hadoop, etc., are some of the skills in the Big Data domain which are the most lucrative. However, hiring such professionals require higher cost and hence it is necessary that better training programs are provided.
Retail, manufacturing, IT, finance is some of the industries which hire Big data expertise people.
People with relevant Big Data skills are a rarity and hence there is a gap between demand and supply. Hence, the average salary is high for people who are working in this field which is more than 98% than in general.
How to be job-ready?
Despite the rising opportunities in Big Data, there is still a lack of relevant skills among the professionals. Hence, it is necessary to get your basics right. You should be familiar with the tools and technique coupled up with the domain knowledge would certainly put you in the driving seat.
Tools like Hive, Hadoop, SQL, Python, Spark are mostly used in this space and hence you should know most of them. Moreover, one should get their hands dirty and work in as many productions based projects as possible to tackle any kind of issues faced during analysis.
There is a huge opportunity for Big Data and now is the best time than ever to keep on learning and improving your skills.
If you are willing to learn more about Big Data or Data Science in general, follow the blogs and courses of Dimensionless.
Follow this link, if you are looking to learn more about data science online!
Additionally, if you are having an interest in learning Data Science, Learnonline Data Science Course to boost your career in Data Science.
Furthermore, if you want to read more about data science, you can read our blogs here
Reports suggest that around 2.5 quintillion bytes of data are generated every single day. As the online usage growth increases at a tremendous rate, there is a need for immediate Data Science professionals who can clean the data, obtain insights from it, visualize it, train model and eventually come up with solutions using Big data for the betterment of the world.
By 2020, experts predict that there will be more than 2.7 million data science and analytics jobs openings. Having a glimpse of the entire Data Science pipeline, it is definitely tiresome for a single human to perform and at the same time excel at all the levels. Hence, Data Science has a plethora of career options that require a spectrum set of skill sets.
Let us explore the top 5 data science career options in 2019 (In no particular order).
1. Data Scientist
Data Scientist is one of the ‘high demand’ job roles. The day to day responsibilities involves the examination of big data. As a result of the analysis of the big data, they also actively perform data cleaning and organize the big data. They are well aware of the machine learning algorithms and understand when to use the appropriate algorithm. During the due course of data analysis and the outcome of machine learning models, patterns are identified in order to solve the business statement.
The reason why this role is so crucial in any organisation is that the company tends to take business decisions with the help of the insights discovered by the Data Scientist to have an edge over the company’s competitors. It is to be noted that the Data Scientist role is inclined more towards the technical domain. As the role demands a wide range of skill set, Data Scientists are one among the highest paid jobs.
Core Skills of a Data Scientist
Database and querying
Data warehousing solutions
Machine learning algorithms
2. Business Intelligence Developer
BI Developer is a job role inclined more towards the Non-Technical domain but has a fair share of Technical responsibilities as well (if required) as a part of their day to day responsibilities. BI developers are responsible for creating and implementing business policies as a result of the insights obtained from the Technical team.
Apart from being a policymaker involving the usage of dedicated (or custom) Business Intelligence analytics tools, they will also have a fair share of coding in order to explore the dataset, present the insights of the dataset in a non-verbal manner. They help in bridging the gap between the technical team that works with the deepest technical understanding and the clients that want the results in the most non-technical manner. They are expected to generate reports from the insights and make it ‘less technical’ for others in the organisation. It is noted that the BI Developers have a deep understanding of Business when compared to Data Scientist.
Core Skills of a Business Analytics Developer
Business model analysis
Design of business workflow
Business Intelligence software integration
3. Machine Learning Engineer
Once the data is clean and ready for analysis, the machine learning engineers work on these big data to train a predictive model that predicts the target variable. These models are used to analyze the trends of the data in the future so that the organisation can take the right business decisions. As the dataset involved in a real-life scenario would involve a lot of dimensions, it is difficult for a human eye to interpret insights from it. This is one of the reasons for training machine learning algorithms as it easily deals with such complex dataset. These engineers carry out a number of tests and analyze the outcomes of the model.
The reason for conducting constant tests on the model using various samples is to test the accuracy of the developed model. Apart from the training models, they also perform exploratory data analysis sometimes in order to understand the dataset completely which will, in turn, help them in training better predictive models.
Core Skills of Machine Learning Engineers
Machine Learning Algorithms
Data Modelling and Evaluation
4. Data Engineer
The pipeline of any data-oriented company begins with the collection of big data from numerous sources. That’s where the data engineers operate in any given project. These engineers integrate data from various sources and optimize them according to the problem statement. The work usually involves writing queries on big data for easy and smooth accessibility. Their day to day responsibility is to provide a streamlined flow of big data from various distributed systems. Data engineering differs from the other data science careers as in, it is concentrated on the system and hardware that aids the company’s data analysis, rather than the analysis of data itself. They provide the organisation with efficient warehousing methods as well.
Core Skills of Data Engineer
Machine Learning algorithm
5. Business Analyst
Business Analyst is one of the most essential roles in the Data Science field. These analysts are responsible for understanding the data and it’s related trend post the decision making about a particular product. They store a good amount of data about various domains of the organisation. These data are really important because if any product of the organisation fails, these analysts work on these big data to understand the reason behind the failure of the project. This type of analysis is vital for all the organisations as it makes them understand the loopholes in the company. The analysts not only backtrack the loophole and in turn provide solutions for the same making sure the organisation takes the right decision in the future. At times, the business analyst act as a bridge between the technical team and the rest of the working community.
Core skills of Business Analyst
The data science career options mentioned above are in no particular order. In my opinion, every career option in Data Science field works complimentary with one another. In any data-driven organization, regardless of the salary, every career role is important at the respective stages in a project.
I have just completed my survey of data (from articles, blogs, white papers, university websites, curated tech websites, and research papers all available online) about predictive analytics.
And I have a reason to believe that we are standing on the brink of a revolution that will transform everything we know about data science and predictive analytics.
But before we go there, you need to know: why the hype about predictive analytics? What is predictive analytics?
Let’s cover that first.
Importance of Predictive Analytics
By PhotoMix Ltd
According to Wikipedia:
Predictive analytics is an area of statistics that deals with extracting information from data and using it to predict trends and behavior patterns. The enhancement of predictive web analytics calculates statistical probabilities of future events online. Predictive analytics statistical techniques include data modeling, machine learning, AI, deep learning algorithms and data mining.
Predictive analytics is why every business wants data scientists. Analytics is not just about answering questions, it is also about finding the right questions to answer. The applications for this field are many, nearly every human endeavor can be listed in the excerpt from Wikipedia that follows listing the applications of predictive analytics:
Predictive analytics is used in actuarial science, marketing, financial services, insurance, telecommunications, retail, travel, mobility, healthcare, child protection, pharmaceuticals, capacity planning, social networking, and a multitude of numerous other fields ranging from the military to online shopping websites, Internet of Things (IoT), and advertising.
In a very real sense, predictive analytics means applying data science models to given scenarios that forecast or generate a score of the likelihood of an event occurring. The data generated today is so voluminous that experts estimate that less than 1% is actually used for analysis, optimization, and prediction. In the case of Big Data, that estimate falls to 0.01% or less.
Common Example Use-Cases of Predictive Analytics
Components of Predictive Analytics
A skilled data scientist can utilize the prediction scores to optimize and improve the profit margin of a business or a company by a massive amount. For example:
If you buy a book for children on the Amazon website, the website identifies that you have an interest in that author and that genre and shows you more books similar to the one you just browsed or purchased.
YouTube also has a very similar algorithm behind its video suggestions when you view a particular video. The site identifies (or rather, the analytics algorithms running on the site identifies) more videos that you would enjoy watching based upon what you are watching now. In ML, this is called a recommender system.
Netflix is another famous example where recommender systems play a massive role in the suggestions for ‘shows you may like’ section, and the recommendations are well-known for their accuracy in most cases
Google AdWords (text ads at the top of every Google Search) that are displayed is another example of a machine learning algorithm whose usage can be classified under predictive analytics.
Departmental stores often optimize products so that common groups are easy to find. For example, the fresh fruits and vegetables would be close to the health foods supplements and diet control foods that weight-watchers commonly use. Coffee/tea/milk and biscuits/rusks make another possible grouping. You might think this is trivial, but department stores have recorded up to 20% increase in sales when such optimal grouping and placement was performed – again, through a form of analytics.
Bank loans and home loans are often approved with the credit scores of a customer. How is that calculated? An expert system of rules, classification, and extrapolation of existing patterns – you guessed it – using predictive analytics.
Allocating budgets in a company to maximize the total profit in the upcoming year is predictive analytics. This is simple at a startup, but imagine the situation in a company like Google, with thousands of departments and employees, all clamoring for funding. Predictive Analytics is the way to go in this case as well.
IoT (Internet of Things) smart devices are one of the most promising applications of predictive analytics. It will not be too long before the sensor data from aircraft parts use predictive analytics to tell its operators that it has a high likelihood of failure. Ditto for cars, refrigerators, military equipment, military infrastructure and aircraft, anything that uses IoT (which is nearly every embedded processing device available in the 21st century).
Fraud detection, malware detection, hacker intrusion detection, cryptocurrency hacking, and cryptocurrency theft are all ideal use cases for predictive analytics. In this case, the ML system detects anomalous behavior on an interface used by the hackers and cybercriminals to identify when a theft or a fraud is taking place, has taken place, or will take place in the future. Obviously, this is a dream come true for law enforcement agencies.
So now you know what predictive analytics is and what it can do. Now let’s come to the revolutionary new technology.
End-to-End Predictive Analytics Product – for non-tech users!
In a remarkable first, a research team at MIT, USA have created a new science called social physics, or sociophysics. Now, much about this field is deliberately kept highly confidential, because of its massive disruptive power as far as data science is concerned, especially predictive analytics. The only requirement of this science is that the system being modeled has to be a human-interaction based environment. To keep the discussion simple, we shall explain the entire system in points.
All systems in which human beings are involved follow scientific laws.
These laws have been identified, verified experimentally and derived scientifically.
Bylaws we mean equations, such as (just an example) Newton’s second law: F = m.a (Force equals mass times acceleration)
These equations establish laws of invariance – that are the same regardless of which human-interaction system is being modeled.
Hence the term social physics – like Maxwell’s laws of electromagnetism or Newton’s theory of gravitation, these laws are a new discovery that are universal as long as the agents interacting in the system are humans.
The invariance and universality of these laws have two important consequences:
The need for large amounts of data disappears – Because of the laws, many of the predictive capacities of the model can be obtained with a minimal amount of data. Hence small companies now have the power to use analytics that was mostly used by the FAMGA(Facebook, Amazon, Microsoft, Google, Apple) set of companies since they were the only ones with the money to maintain Big Data warehouses and data lakes.
There is no need for data cleaning. Since the model being used is canonical, it is independent of data problems like outliers, missing data, nonsense data, unavailable data, and data corruption. This is due to the orthogonality of the model ( a Knowledge Sphere) being constructed and the data available.
Performance is superior to deep learning, Google TensorFlow, Python, R, Julia, PyTorch, and scikit-learn. Consistently, the model has outscored the latter models in Kaggle competitions, without any data pre-processing or data preparation and cleansing!
Data being orthogonal to interpretation and manipulation means that encrypted data can be used as-is. There is no need to decrypt encrypted data to perform a data science task or experiment. This is significant because the independence of the model functioning even for encrypted data opens the door to blockchain technology and blockchain data to be used in standard data science tasks. Furthermore, this allows hashing techniques to be used to hide confidential data and perform the data mining task without any knowledge of what the data indicates.
Are You Serious?
That’s a valid question given these claims! And that is why I recommend everyone who has the slightest or smallest interest in data science to visit and completely read and explore the following links:
Now when I say completely read, I mean completely read. Visit every section and read every bit of text that is available on the three sites above. You will soon understand why this is such a revolutionary idea.
These links above are articles about the social physics book and about the science of sociophysics in general.
For more details, please visit the following articles on Medium. These further document Endor.coin, a cryptocurrency built around the idea of sharing data with the public and getting paid for using the system and usage of your data. Preferably, read all, if busy, at least read Article No, 1.
Upon every data set, the first action performed by the Endor Analytics Platform is clustering, also popularly known as automatic classification. Endor constructs what is known as a Knowledge Sphere, a canonical representation of the data set which can be constructed even with 10% of the data volume needed for the same project when deep learning was used.
Creation of the Knowledge Sphere takes 1-4 hours for a billion records dataset (which is pretty standard these days).
Now an explanation of the mathematics behind social physics is beyond our scope, but I will include the change in the data science process when the Endor platform was compared to a deep learning system built to solve the same problem the traditional way (with a 6-figure salary expert data scientist).
From Appendix A: Social Physics Explained, Section 3.1, pages 28-34 (some material not included):
Prediction Demonstration using the Endor System:
Data: The data that was used in this example originated from a retail financial investment platform and contained the entire investment transactions of members of an investment community. The data was anonymized and made public for research purposes at MIT (the data can be shared upon request).
Summary of the dataset: – 7 days of data – 3,719,023 rows – 178,266 unique users
Automatic Clusters Extraction: Upon first analysis of the data the Endor system detects and extracts “behavioral clusters” – groups of users whose data dynamics violates the mathematical invariances of the Social Physics. These clusters are based on all the columns of the data, but is limited only to the last 7 days – as this is the data that was provided to the system as input.
Behavioural Clusters Summary
Number of clusters:268,218 Clusters sizes: 62 (Mean), 15 (Median), 52508 (Max), 5 (Min) Clusters per user:164 (Mean), 118 (Median), 703 (Max), 2 (Min) Users in clusters: 102,770 out of the 178,266 users Records per user: 6 (Median), 33 (Mean): applies only to users in clusters
Prediction Queries The following prediction queries were defined: 1. New users to become “whales”: users who joined in the last 2 weeks that will generate at least $500 in commission in the next 90 days 2. Reducing activity : users who were active in the last week that will reduce activity by 50% in the next 30 days (but will not churn, and will still continue trading) 3. Churn in “whales”: currently active “whales” (as defined by their activity during the last 90 days), who were active in the past week, to become inactive for the next 30 days 4. Will trade in Apple share for the first time: users who had never invested in Apple share, and would buy it for the first time in the coming 30 days
Knowledge Sphere Manifestation of Queries It is again important to note that the definition of the search queries is completely orthogonal to the extraction of behavioral clusters and the generation of the Knowledge Sphere, which was done independently of the queries definition.
Therefore, it is interesting to analyze the manifestation of the queries in the clusters detected by the system: Do the clusters contain information that is relevant to the definition of the queries, despite the fact that:
1. The clusters were extracted in a fully automatic way, using no semantic information about the data, and –
2. The queries were defined after the clusters were extracted, and did not affect this process.
This analysis is done by measuring the number of clusters that contain a very high concentration of “samples”; In other words, by looking for clusters that contain “many more examples than statistically expected”.
A high number of such clusters (provided that it is significantly higher than the amount received when randomly sampling the same population) proves the ability of this process to extract valuable relevant semantic insights in a fully automatic way.
Comparison to Google TensorFlow
In this section a comparison between prediction process of the Endor system and Google’s TensorFlow is presented. It is important to note that TensorFlow, like any other Deep Learning library, faces some difficulties when dealing with data similar to the one under discussion:
1. An extremely uneven distribution of the number of records per user requires some canonization of the data, which in turn requires:
2. Some manual work, done by an individual who has at least some understanding of data science.
3. Some understanding of the semantics of the data, that requires an investment of time, as well as access to the owner or provider of the data
4. A single-class classification, using an extremely uneven distribution of positive vs. negative samples, tends to lead to the overfitting of the results and require some non-trivial maneuvering.
This again necessitates the involvement of an expert in Deep Learning (unlike the Endor system which can be used by Business, Product or Marketing experts, with no perquisites in Machine Learning or Data Science).
An expert in Deep Learning spent 2 weeks crafting a solution that would be based on TensorFlow and has sufficient expertise to be able to handle the data. The solution that was created used the following auxiliary techniques:
1.Trimming the data sequence to 200 records per customer, and padding the streams for users who have less than 200 records with neutral records.
2.Creating 200 training sets, each having 1,000 customers (50% known positive labels, 50% unknown) and then using these training sets to train the model.
3.Using sequence classification (RNN with 128 LSTMs) with 2 output neurons (positive, negative), with the overall result being the difference between the scores of the two.
Observations (all statistics available in the white paper – and it’s stunning)
1.Endor outperforms Tensor Flow in 3 out of 4 queries, and results in the same accuracy in the 4th . 2.The superiority of Endor is increasingly evident as the task becomes “more difficult” – focusing on the top-100 rather than the top-500.
3.There is a clear distinction between “less dynamic queries” (becoming a whale, churn, reduce activity” – for which static signals should likely be easier to detect) than the “Who will trade in Apple for the first time” query, which are (a) more dynamic, and (b) have a very low baseline, such that for the latter, Endor is 10x times more accurate!
4.As previously mentioned – the Tensor Flow results illustrated here employ 2 weeks of manual improvements done by a Deep Learning expert, whereas the Endor results are 100% automatic and the entire prediction process in Endor took 4 hours.
Clearly, the path going forward for predictive analytics and data science is Endor, Endor, and Endor again!
Predictions for the Future
Personally, one thing has me sold – the robustness of the Endor system to handle noise and missing data. Earlier, this was the biggest bane of the data scientist in most companies (when data engineers are not available). 90% of the time of a professional data scientist would go into data cleaning and data preprocessing since our ML models were acutely sensitive to noise. This is the first solution that has eliminated this ‘grunt’ level work from data science completely.
The second prediction: the Endor system works upon principles of human interaction dynamics. My intuition tells me that data collected at random has its own dynamical systems that appear clearly to experts in complexity theory. I am completely certain that just as this tool developed a prediction tool with human society dynamical laws, data collected in general has its own laws of invariance. And the first person to identify these laws and build another Endor-style platform on them will be at the top of the data science pyramid – the alpha unicorn.
Final prediction – democratizing data science means that now data scientists are not required to have six-figure salaries. The success of the Endor platform means that anyone can perform advanced data science without resorting to TensorFlow, Python, R, Anaconda, etc. This platform will completely disrupt the entire data science technological sector. The first people to master it and build upon it to formalize the rules of invariance in the case of general data dynamics will for sure make a killing.
It is an exciting time to be a data science researcher!
Data Science is a broad field and it would require quite a few things to learn to master all these skills.
Good data management practices are essential for ensuring that research data are of high quality, findable, accessible and have high validity. You can then share data ensuring their sustainability and accessibility in the long-term, for new research and policy or to replicate and validate existing research and policy. It is important that researchers extend these practices to their work with all types of data, be it big (large or complex) data or smaller, more ‘curatable’ datasets.
In this blog, we are going to understand about the data curation. Furthermore, we will be looking into many other advantages which data curation will bring to the big data table.
What is Data Curation?
Curation is the end-to-end process of creating good data through the identification and formation of resources with long-term value. In information technology, it refers mainly to the management of data throughout its lifecycle, from creation and initial storage to the time when it is archived for future research and analysis, or becomes obsolete and is deleted. The goal of data curation in the enterprise is twofold: to ensure compliance and that data can be retrieved for future research or reuse
Why Do You Need Data Curation?
Organizations invest heavily in big data analytics — $44 billion in 2014 alone, according to Gartner; yet, studies show that most organizations use only about 10% of their collected data, data that remains scattered in silos and varied sources across the organization. With data volumes growing exponentially, along with the increasing variety and heterogeneity of data sources, getting the data you need ready for analysis has become a costly and time-consuming process. Multiple data sets from different sources must first be catalogued and connected before they can be used by various analytics tools. Duplicate data and blank fields need to be eliminated, misspellings fixed, columns split or reshaped, and data need to be enriched with data from additional or third party sources to provide more context.
Effective Machine Learning
Machine Learning algorithms have made great strides towards understanding the consumer space. AI consisting of “neural networks” collaborate, and can use Deep Learning to recognize patterns. However, Humans need to intervene, at least initially, to direct algorithmic behavior towards effective learning. Curations are about where the humans can actually add their knowledge to what the machine has automated. This results in prepping for intelligent self-service processes, setting up organizations up for insights.
Dealing with Data Swamps
A Data Lake strategy allows users to easily access raw data, to consider multiple data attributes at once, and the flexibility to ask ambiguous business driven questions. But Data Lakes can end up Data Swamps where finding business value becomes like a quest to find the Holy Grail. Such Data swamps minus well be a Data graveyard. Well data curation here can save your data lakes from becoming the data yards
Ensuring Data Quality
Data Curators clean and undertake actions to ensure the long undertake actions to ensure the long-term preservation and retention of the authoritative nature of digital objects.
Steps in Data Curation
Data curation is the process of turning independently created data sources (structured and semi-structured data) into unified data sets ready for analytics, using domain experts to guide the process. It involves:
One needs to identify different data sources of interest (whether from inside or outside the enterprise) before they start working on a problem statement. Identification of the dataset is as important a thing as solving a problem. Many people underestimate the value of data identification. But, when one does data identification the right way, one can save on a lot of time wastage which can happen while optimizing the solution of the problem
Once you have some data at hand, one needs to clean the data. The incoming data may have a lot of anomalies like spelling errors, missing values, improper entries etc. Most of the data is always dirty and you need to clean it before you can start working with it. Cleaning data is one of the most important tasks under data curation. There is almost 200% value addition once data is in the right format
Data transformation is the process of converting data or information from one format to another, usually from the format of a source system into the required format of a new destination system. The usual process involves converting documents, but data conversions sometimes involve the conversion of a program from one computer language to another to enable the program to run on a different platform. The usual reason for this data migration is the adoption of a new system that’s totally different from the previous one. Data curation also takes care of the data transformation
The more data you need to curate for analytics and other business purposes, the more costly and complex curation becomes — mostly because humans (domain experts, or data owners) aren’t scalable. As such, most enterprises are “tearing their hair out” as they try to cope with data curation at scale.
Roles of a Data Curator
In practice, data curation is more concerned with maintaining and managing the metadata rather than the database itself and, to that end, a large part of the process of data curation revolves around ingesting metadata such as schema, table and column popularity, usage popularity, top joins/filters/queries. Data curators not only create, manage, and maintain data, but may also determine best practices for working with that data. They often present the data in a visual format such as a chart, dashboard or report.
Data curation starts with the “data set.” These data sets are the atoms of data curation. Determining which of these data sets are the most useful or relevant is the job of the data curator. Being able to present the data in an effective manner is also extremely important. While some rules of thumb and best practices apply, the data curator must make an educated decision about which data assets are appropriate to use.
It’s important to know the context of the data before it can be trusted. Data curation uses such arbiters of modern taste as lists, popularity rankings, annotations, relevance feeds, comments, articles and the upvoting or downvoting of data assets to determine their relevancy.
How to Start with Data Curation?
First, companies can inject additional data assessments into their reviews of data with end users that evaluate how data can be used or redirected. One way this can be done is by making data retention reviews a collaborative process across business functions. The collaboration enables users who ordinarily wouldn’t be exposed to some types of data to evaluate if there are ways that this data can be plugged in and used in their own departmental analytics processes.
Second, IT and the business should articulate rules governing data purges. Presently, there is a fear of discarding any data, no matter how useless.
Third, companies should consider adding a data curator, which is a librarian-like curation function, to their big data and analytics staffs.
Data sets are reusable components — anyone conducting analysis should share and expect data sets that they create to be re-used. Re-usability is key to self-service at scale. Companies such as GoDaddy and eBay have already embraced this approach to harvesting and distributing data for re-use, allowing any user to become a curator of data knowledge and resulting in higher productivity.
Data curation observes the use of data, focusing on how context, narrative, and meaning can be collected around a reusable data set. It creates trust in data by tracking the social network and social bonds between users of data. By employing lists, popularity rankings, annotations, relevance feeds, comments, articles and the upvoting or downvoting of data assets, curation takes organizations beyond data documentation to creating trust in data across the enterprise.
Follow this link, if you are looking to learn more about data science online!
There are a huge number of ML algorithms out there. Trying to classify them leads to the distinction being made in types of the training procedure, applications, the latest advances, and some of the standard algorithms used by ML scientists in their daily work. There is a lot to cover, and we shall proceed as given in the following listing:
1. Statistical Algorithms
Statistics is necessary for every machine learning expert. Hypothesis testing and confidence intervals are some of the many statistical concepts to know if you are a data scientist. Here, we consider here the phenomenon of overfitting. Basically, overfitting occurs when an ML model learns so many features of the training data set that the generalization capacity of the model on the test set takes a toss. The tradeoff between performance and overfitting is well illustrated by the following illustration:
Overfitting – from Wikipedia
Here, the black curve represents the performance of a classifier that has appropriately classified the dataset into two categories. Obviously, training the classifier was stopped at the right time in this instance. The green curve indicates what happens when we allow the training of the classifier to ‘overlearn the features’ in the training set. What happens is that we get an accuracy of 100%, but we lose out on performance on the test set because the test set will have a feature boundary that is usually similar but definitely not the same as the training set. This will result in a high error level when the classifier for the green curve is presented with new data. How can we prevent this?
Cross-Validation is the killer technique used to avoid overfitting. How does it work? A visual representation of the k-fold cross-validation process is given below:
The entire dataset is split into equal subsets and the model is trained on all possible combinations of training and testing subsets that are possible as shown in the image above. Finally, the average of all the models is combined. The advantage of this is that this method eliminates sampling error, prevents overfitting, and accounts for bias. There are further variations of cross-validation like non-exhaustive cross-validation and nested k-fold cross validation (shown above). For more on cross-validation, visit the following link.
There are many more statistical algorithms that a data scientist has to know. Some examples include the chi-squared test, the Student’s t-test, how to calculate confidence intervals, how to interpret p-values, advanced probability theory, and many more. For more, please visit the excellent article given below:
Classification refers to the process of categorizing data input as a member of a target class. An example could be that we can classify customers into low-income, medium-income, and high-income depending upon their spending activity over a financial year. This knowledge can help us tailor the ads shown to them accurately when they come online and maximises the chance of a conversion or a sale. There are various types of classification like binary classification, multi-class classification, and various other variants. It is perhaps the most well known and most common of all data science algorithm categories. The algorithms that can be used for classification include:
Support Vector Machines
Linear Discriminant Analysis
and many more. A short illustration of a binary classification visualization is given below:
For more information on classification algorithms, refer to the following excellent links:
Regression is similar to classification, and many algorithms used are similar (e.g. random forests). The difference is that while classification categorizes a data point, regression predicts a continuous real-number value. So classification works with classes while regression works with real numbers. And yes – many algorithms can be used for both classification and regression. Hence the presence of logistic regression in both lists. Some of the common algorithms used for regression are
Support Vector Regression
Partial Least-Squares Regression
For more on regression, I suggest that you visit the following link for an excellent article:
Both articles have a remarkably clear discussion of the statistical theory that you need to know to understand regression and apply it to non-linear problems. They also have source code in Python and R that you can use.
Clustering is an unsupervised learning algorithm category that divides the data set into groups depending upon common characteristics or common properties. A good example would be grouping the data set instances into categories automatically, the process being used would be any of several algorithms that we shall soon list. For this reason, clustering is sometimes known as automatic classification. It is also a critical part of exploratory data analysis (EDA). Some of the algorithms commonly used for clustering are:
Hierarchical Clustering – Agglomerative
Hierarchical Clustering – Divisive
K-Nearest Neighbours Clustering
EM (Expectation Maximization) Clustering
Principal Components Analysis Clustering (PCA)
An example of a common clustering problem visualization is given below:
The above visualization clearly contains three clusters.
Another excellent article on clustering refer the link
Dimensionality Reduction is an extremely important tool that should be completely clear and lucid for any serious data scientist. Dimensionality Reduction is also referred to as feature selection or feature extraction. This means that the principal variables of the data set that contains the highest covariance with the output data are extracted and the features/variables that are not important are ignored. It is an essential part of EDA (Exploratory Data Analysis) and is nearly always used in every moderately or highly difficult problem. The advantages of dimensionality reduction are (from Wikipedia):
It reduces the time and storage space required.
Removal of multi-collinearity improves the interpretation of the parameters of the machine learning model.
It becomes easier to visualize the data when reduced to very low dimensions such as 2D or 3D.
It avoids the curse of dimensionality.
The most commonly used algorithm for dimensionality reduction is Principal Components Analysis or PCA. While this is a linear model, it can be converted to a non-linear model through a kernel trick similar to that used in a Support Vector Machine, in which case the technique is known as Kernel PCA. Thus, the algorithms commonly used are:
Ensembling means combining multiple ML learners together into one pipeline so that the combination of all the weak learners makes an ML application with higher accuracy than each learner taken separately. Intuitively, this makes sense, since the disadvantages of using one model would be offset by combining it with another model that does not suffer from this disadvantage. There are various algorithms used in ensembling machine learning models. The three common techniques usually employed in practice are:
Simple/Weighted Average/Voting: Simplest one, just takes the vote of models in Classification and average in Regression.
Bagging: We train models (same algorithm) in parallel for random sub-samples of data-set with replacement. Eventually, take an average/vote of obtained results.
Boosting: In this models are trained sequentially, where (n)th model uses the output of (n-1)th model and works on the limitation of the previous model, the process stops when result stops improving.
Stacking: We combine two or more than two models using another machine learning algorithm.
(from Amardeep Chauhan on Medium.com)
In all four cases, the combination of the different models ends up having the better performance that one single learner. One particular ensembling technique that has done extremely well on data science competitions on Kaggle is the GBRT model or the Gradient Boosted Regression Tree model.
We include the source code from the scikit-learn module for Gradient Boosted Regression Trees since this is one of the most popular ML models which can be used in competitions like Kaggle, HackerRank, and TopCoder.
In the last decade, there has been a renaissance of sorts within the Machine Learning community worldwide. Since 2002, neural networks research had struck a dead end as the networks of layers would get stuck in local minima in the non-linear hyperspace of the energy landscape of a three layer network. Many thought that neural networks had outlived their usefulness. However, starting with Geoffrey Hinton in 2006, researchers found that adding multiple layers of neurons to a neural network created an energy landscape of such high dimensionality that local minima were statistically shown to be extremely unlikely to occur in practice. Today, in 2019, more than a decade of innovation later, this method of adding addition hidden layers of neurons to a neural network is the classical practice of the field known as deep learning.
Deep Learning has truly taken the computing world by storm and has been applied to nearly every field of computation, with great success. Now with advances in Computer Vision, Image Processing, Reinforcement Learning, and Evolutionary Computation, we have marvellous feats of technology like self-driving cars and self-learning expert systems that perform enormously complex tasks like playing the game of Go (not to be confused with the Go programming language). The main reason these feats are possible is the success of deep learning and reinforcement learning (more on the latter given in the next section below). Some of the important algorithms and applications that data scientists have to be aware of in deep learning are:
Long Short term Memories (LSTMs) for Natural Language Processing
Recurrent Neural Networks (RNNs) for Speech Recognition
Convolutional Neural Networks (CNNs) for Image Processing
Deep Neural Networks (DNNs) for Image Recognition and Classification
Hybrid Architectures for Recommender Systems
Autoencoders (ANNs) for Bioinformatics, Wearables, and Healthcare
Deep Learning Networks typically have millions of neurons and hundreds of millions of connections between neurons. Training such networks is such a computationally intensive task that now companies are turning to the 1) Cloud Computing Systems and 2) Graphical Processing Unit (GPU) Parallel High-Performance Processing Systems for their computational needs. It is now common to find hundreds of GPUs operating in parallel to train ridiculously high dimensional neural networks for amazing applications like dreaming during sleep and computer artistry and artistic creativity pleasing to our aesthetic senses.
Artistic Image Created By A Deep Learning Network. From blog.kadenze.com.
For more on Deep Learning, please visit the following links:
In the recent past and the last three years in particular, reinforcement learning has become remarkably famous for a number of achievements in cognition that were earlier thought to be limited to humans. Basically put, reinforcement learning deals with the ability of a computer to teach itself. We have the idea of a reward vs. penalty approach. The computer is given a scenario and ‘rewarded’ with points for correct behaviour and ‘penalties’ are imposed for wrong behaviour. The computer is provided with a problem formulated as a Markov Decision Process, or MDP. Some basic types of Reinforcement Learning algorithms to be aware of are (some extracts from Wikipedia):
Q-Learning is a model-free reinforcement learning algorithm. The goal of Q-learning is to learn a policy, which tells an agent what action to take under what circumstances. It does not require a model (hence the connotation “model-free”) of the environment, and it can handle problems with stochastic transitions and rewards, without requiring adaptations. For any finite Markov decision process (FMDP), Q-learning finds a policy that is optimal in the sense that it maximizes the expected value of the total reward over any and all successive steps, starting from the current state. Q-learning can identify an optimal action-selection policy for any given FMDP, given infinite exploration time and a partly-random policy. “Q” names the function that returns the reward used to provide the reinforcement and can be said to stand for the “quality” of an action taken in a given state.
State–action–reward–state–action (SARSA) is an algorithm for learning a Markov decision process policy. This name simply reflects the fact that the main function for updating the Q-value depends on the current state of the agent “S1“, the action the agent chooses “A1“, the reward “R” the agent gets for choosing this action, the state “S2” that the agent enters after taking that action, and finally the next action “A2” the agent choose in its new state. The acronym for the quintuple (st, at, rt, st+1, at+1) is SARSA.
3.Deep Reinforcement Learning
This approach extends reinforcement learning by using a deep neural network and without explicitly designing the state space. The work on learning ATARI games by Google DeepMind increased attention to deep reinforcement learning or end-to-end reinforcement learning. Remarkably, the computer agent DeepMind has achieved levels of skill higher than humans at playing computer games. Even a complex game like DOTA 2 was won by a deep reinforcement learning network based upon DeepMind and OpenAI Gym environments that beat human players 3-2 in a tournament of best of five matches.
For more information, go through the following links:
If reinforcement learning was cutting edge data science, AutoML is bleeding edge data science. AutoML (Automated Machine Learning) is a remarkable project that is open source and available on GitHub at the following link that, remarkably, uses an algorithm and a data analysis approach to construct an end-to-end data science project that does data-preprocessing, algorithm selection,hyperparameter tuning, cross-validation and algorithm optimization to completely automate the ML process into the hands of a computer. Amazingly, what this means is that now computers can handle the ML expertise that was earlier in the hands of a few limited ML practitioners and AI experts.
AutoML has found its way into Google TensorFlow through AutoKeras, Microsoft CNTK, and Google Cloud Platform, Microsoft Azure, and Amazon Web Services (AWS). Currently it is a premiere paid model for even a moderately sized dataset and is free only for tiny datasets. However, one entire process might take one to two or more days to execute completely. But at least, now the computer AI industry has come full circle. We now have computers so complex that they are taking the machine learning process out of the hands of the humans and creating models that are significantly more accurate and faster than the ones created by human beings!
The basic algorithm used by AutoML is Network Architecture Search and its variants, given below:
Network Architecture Search (NAS)
PNAS (Progressive NAS)
ENAS (Efficient NAS)
The functioning of AutoML is given by the following diagram:
If you’ve stayed with me till now, congratulations; you have learnt a lot of information and cutting edge technology that you must read up on, much, much more. You could start with the links in this article, and of course, Google is your best friend as a Machine Learning Practitioner. Enjoy machine learning!