1. What is a Data Analyst's role?
Data Analyst Role:-
" create policies and procedures for record management
" identify areas for process improvement and automation setup and upkeep automated data processes
" identify, evaluate, and implement external data validation and cleansing services and tools
" create and monitor key performance indicators
" create and support reporting procedures
" quality control and auditing of data
" communicate with internal and external clients to fully comprehend data content
" Using appropriate tools and techniques, collect, comprehend, and document detailed business requirements.
" survey design and execution, as well as survey data analysis
" manipulating, analysing, and interpreting complex data sets pertaining to the employer's business
" Using business analytics reporting tools, create reports for internal and external audiences.
" design data dashboards, graphs, and visualisations
" provide benchmarking data for the sector and competitors
The steps required in analytics Project:-
1. Define a goal/Business Understanding
2. Getting Data/ Understanding Data
3. Cleaning Data/Data Preparations
4. Exploiting Data/Getting Insight
5. Deploying Machine Learning/Iterating
6. Validating
7. Visualizing and Presenting
The process of repairing or removing incorrect, distorted, improperly formatted, redundant, or imperfect data from a dataset is known as data cleaning.
The goal behind kNN approaches is to find 'k' samples in a dataset that is similar or close in space. The 'k' samples will then be used to estimate the value of the missing data points. The missing values in each sample are imputed using the mean value of the 'k'-neighbors found in the dataset.
Data mining is the process of evaluating obtained information and gathering insights , information and statistics about it.
Data profiling is the process of examining and summarizing important information about data from an existing source.
Develop a report format with details on all questionable data. Data validation criteria that were not met, as well as the date and time of the occurrence, should be recorded.
The suspicious data should be examined by experienced experts to determine their acceptability.
Invalid data should be issued a validation code and replaced.
To work with missing data, the optimal use of analysis strategies such as deletion, single imputation, model-based techniques, and so on should be followed.
Hierarchical clustering, also known as hierarchical cluster analysis or HCA, is an unsupervised machine learning technique that is used to sort unlabelled datasets into clusters.
The dendrogram is a tree-shaped structure that we use to create the hierarchy of clusters in this approach.
The Kmeans algorithm is an iterative technique that attempts to partition a dataset into K separate non-overlapping subgroups (clusters), with each data point belonging to only one of these groups. It aims to make intra-cluster data points as comparable as possible while maintaining clusters as distinct (far) as possible. It distributes data points to clusters so that the sum of the squared distances between them and the cluster's centroid is as small as possible.
Collaborative Filtering is a Machine Learning approach for identifying data correlations. This method is often used in recommender systems to find similarities between user data and items.
If Users A and B both prefer Item X, and User B also prefers Item Y, the system may suggest Product Y to User A.
A time series is an ordered sequence of observations with regard to time periods. In other words, a time series is a sequential grouping of data based on the time of occurrence.
A time series data set is a collection of measurements taken over a fixed period of time, with time acting as the independent variable and the goal as the dependent variables.
Imputation is a method to fill missing value to build a complete data matrix that can be evaluated using traditional methods
Imputation Technique for following Type of Data are:-
1. Numerical Variable :- Mean, Median , Mode, End of Tail, Arbitrary Value Imputation
2. Categorical Variable :- Frequent Category, Adding Missing Imputation
MapReduce is a programming model that enables parallel and distributed processing of massive datasets.
MapReduce is made up of two different tasks - Map and Reduce.
The reduction phase occurs after the mapper phase has been completed, as the term MapReduce implies.
The first is the map task, which reads and processes a block of data to generate key-value pairs as intermediate outputs.
The Reducer receives the output from a Mapper or map task (key-value pairs).
Multiple map tasks send the key-value pair to the reducer.
The reducer then aggregates those intermediate data tuples into a smaller collection of tuples or key-value pairs that constitutes the final output.
Big Data tools include:
" Hadoop " Hive " Pig " Flume " Mahout " Sqoop
" Tableau " RapidMiner " OpenRefine " KNIME " Google Search Term " Solver " NodeXL " io " Alpha Wolfram
Domain knowledge: The first step is to understand how to extract the various features from the data and learn more about the data that we are dealing with. It has more to do with the type of domain we are dealing with and familiarizing the system with it in order to learn more about it.
Feature Selection: This step is more concerned with the feature that we are selecting from the set of available features. Sometimes there are a lot of features, and we have to make an intelligent decision about which type of feature we want to use to move forward with our machine learning endeavor.
Algorithm: This is a critical step because the algorithms we choose will have a significant impact on the entire machine learning process. You have the option of using either the linear or nonlinear algorithms. Support Vector Machines, Decision Trees, Naive Bayes, K-Means Clustering, and other algorithms are used.
Training: This is the most significant aspect of machine learning and where it differs from traditional programming. The training is based on the data we have and includes additional real-world experiences. With each subsequent training phase, the machine improves and becomes wiser, allowing it to make better judgments.
Evaluation: In this stage, we review the machine's decisions to see whether or not they are appropriate. There are several metrics involved in this process, and we must closely deploy each of them to determine the efficacy of the entire machine learning endeavor.
Optimization : This the process of enhancing the performance of the machine learning process via the use of various optimization approaches. Optimization of machine learning is one of the most important components in which the algorithm's performance is substantially increased. The best aspect about optimization strategies is that machine learning not only consumes optimization approaches but also generates new optimization ideas.
Testing: Various tests are performed here, some of which are previously unseen sets of test cases. The data is divided into two sets: test and training. There are different tests available.
Data modeling is the initial phase in the creation of a database. Data modeling is the process of developing a conceptual model based on the relationships between distinct data models. The procedure entails progressing from the conceptual stage to the logical model and finally to the physical schema. It entails a methodical approach to using data modeling approaches.
The process of creating a database is known as Database Design. The database design generates an output that is a comprehensive database data model. Database design, strictly speaking, contains the full logical model of a database, but it can also include physical design options and storage characteristics.
It is a model validation approach used to determine how well the results of a statistical study would generalize to a different data set. Typically employed in situations when the goal is to forecast and one wants to assess how correctly a model will perform in practice. The purpose of cross-validation is to define a data set to test the model on during the training phase in order to limit issues such as overfitting and get insight into how the model will generalize to an independent data set.
Univariate or any other graphical analysis approach can be used to identify outlier values. If the number of outlier values is small, they can be evaluated separately; however, if the number of outliers is considerable, the values can be substituted with either the 99th or 1st percentile values. Outlier values are not all extreme values. The most popular methods for dealing with outlier values are:
1) changing the value and bringing it into a range
2) Simply remove the value.
Measures of recall "How many of the real true samples did we label as true"
Precision is defined as "how many of all the samples we categorized as true are truly true."
The validation set might be regarded as a subset of the training set because it is used to choose parameters and avoid overfitting of the model being developed. A test set, on the other hand, is used to test or evaluate the performance of a trained machine learning model.
In a nutshell, the distinctions are as follows:
The purpose of the Training Set is to suit the criteria, such as weights.
The purpose of the Test Set is to evaluate the model's performance, namely its predictive power and generalization.
The validation set is used to fine-tune the settings.
Sensitivity is widely used to verify a classifier's accuracy . Sensitivity may be defined as "predicted TRUE events/total events." True events are those that occurred and were expected to occur by the model.
Senstivity is easy to calculate: Senstivity = True Positives / Positives in Actual Dependent Variable
True positives are Positive occurrences that have been appropriately identified as Positives.
The ROC curve is a graphical depiction of the difference between true positive and false positive rates at certain thresholds. It is frequently used to represent the trade-off between sensitivity (true positive rate) and false positive rate.
Basic measurements generated from the confusion matrix are:
1. Error Rate = (FP+FN)/(P+N)
2. Accuracy = (TP+TN)/(P+N)
3. Sensitivity (Recall or True positive rate) = TP/P
4. Specificity (True negative rate) = TN/N
5. Precision (Positive predicted value) = TP/(TP+FP)
6. F-Score (Harmonic mean of accuracy and recall) = (1+b) (PREC.REC)/(b2PREC+REC), where b is generally 0.5, 1, 2.
Support vector machine (SVM) is a supervised machine learning technique that may be used for both regression and classification. If your training dataset has n features, SVM attempts to plot them in n-dimensional space, with the value of each feature being the value of a certain coordinate.
SVM employs hyperplanes to divide distinct classes based on the kernel function supplied.
When we remove sub-nodes from a decision node, this is referred to as pruning, or the opposite process of splitting.
Random forest is a flexible machine learning approach that can do both regression and classification problems. It is also used for dimension reduction, missing value treatment, and outlier value treatment. It is a sort of ensemble learning approach in which a set of weak models merge to generate a powerful model.
In Random Forest, we develop numerous trees rather than a single tree. Each tree provides a classifier to categorise a new item based on properties. The forest selects the classification with the most votes, and in the case of regression, it takes the average of outputs from various trees.
Data is often dispersed in a variety of ways, with a bias to the left or right, or it might all be mixed together. However, there is a probability that data will be dispersed about a central value with no bias to the left or right and will approach normal distribution in the form of a bell-shaped curve. A symmetrical bell-shaped curve is used to disperse the random variables.
Regularization is the process of introducing tuning parameters into a model in order to produce smoothness and avoid overfitting. Adding a constant multiple to an existing weight vector is the most common way to accomplish this. This constant is frequently the L1 (Lasso) or L2 (Lasso) constant (ridge). Following that, the model predictions should minimize the loss function derived on the regularised training set.
Data Science is a collection of tools, algorithms, and machine learning methods that aim to uncover hidden patterns in raw data.
" identify areas for process improvement and automation setup and upkeep automated data processes
" identify, evaluate, and implement external data validation and cleansing services and tools
" create and monitor key performance indicators
" create and support reporting procedures
" quality control and auditing of data
" communicate with internal and external clients to fully comprehend data content
" Using appropriate tools and techniques, collect, comprehend, and document detailed business requirements.
" survey design and execution, as well as survey data analysis
" manipulating, analysing, and interpreting complex data sets pertaining to the employer's business
" Using business analytics reporting tools, create reports for internal and external audiences.
" design data dashboards, graphs, and visualisations
" provide benchmarking data for the sector and competitors
2. Describe the stages in an analytics project
The steps required in analytics Project:-
1. Define a goal/Business Understanding
2. Getting Data/ Understanding Data
3. Cleaning Data/Data Preparations
4. Exploiting Data/Getting Insight
5. Deploying Machine Learning/Iterating
6. Validating
7. Visualizing and Presenting
3. Explain data cleaning
The process of repairing or removing incorrect, distorted, improperly formatted, redundant, or imperfect data from a dataset is known as data cleaning.
4. Define logistic regression
Logistic regression is a classification analysis methodology that utilizes prior observations of a data set to predict a binary outcome, such as yes or no. A logistic regression model forecasts a dependent data variable by exploring the relationship between one or more pre-existing independent variables
5. What is KNN imputation?
The goal behind kNN approaches is to find 'k' samples in a dataset that is similar or close in space. The 'k' samples will then be used to estimate the value of the missing data points. The missing values in each sample are imputed using the mean value of the 'k'-neighbors found in the dataset.
6. Distinguish between data mining and data profiling
Data mining is the process of evaluating obtained information and gathering insights , information and statistics about it.
Data profiling is the process of examining and summarizing important information about data from an existing source.
7. What should you do if you have data that is suspicious or missing?
Develop a report format with details on all questionable data. Data validation criteria that were not met, as well as the date and time of the occurrence, should be recorded.
The suspicious data should be examined by experienced experts to determine their acceptability.
Invalid data should be issued a validation code and replaced.
To work with missing data, the optimal use of analysis strategies such as deletion, single imputation, model-based techniques, and so on should be followed.
8. Define Hierarchical Clustering Algorithm.
Hierarchical clustering, also known as hierarchical cluster analysis or HCA, is an unsupervised machine learning technique that is used to sort unlabelled datasets into clusters.
The dendrogram is a tree-shaped structure that we use to create the hierarchy of clusters in this approach.
9. Define K-mean Algorithm.
The Kmeans algorithm is an iterative technique that attempts to partition a dataset into K separate non-overlapping subgroups (clusters), with each data point belonging to only one of these groups. It aims to make intra-cluster data points as comparable as possible while maintaining clusters as distinct (far) as possible. It distributes data points to clusters so that the sum of the squared distances between them and the cluster's centroid is as small as possible.
10 Define the term "collaborative filtering".
Collaborative Filtering is a Machine Learning approach for identifying data correlations. This method is often used in recommender systems to find similarities between user data and items.
If Users A and B both prefer Item X, and User B also prefers Item Y, the system may suggest Product Y to User A.
11. What is Time series analysis?
A time series is an ordered sequence of observations with regard to time periods. In other words, a time series is a sequential grouping of data based on the time of occurrence.
A time series data set is a collection of measurements taken over a fixed period of time, with time acting as the independent variable and the goal as the dependent variables.
12. Define imputation. What are the various imputation techniques?
Imputation is a method to fill missing value to build a complete data matrix that can be evaluated using traditional methods
Imputation Technique for following Type of Data are:-
1. Numerical Variable :- Mean, Median , Mode, End of Tail, Arbitrary Value Imputation
2. Categorical Variable :- Frequent Category, Adding Missing Imputation
13. Explain Map Reduce.
MapReduce is a programming model that enables parallel and distributed processing of massive datasets.
MapReduce is made up of two different tasks - Map and Reduce.
The reduction phase occurs after the mapper phase has been completed, as the term MapReduce implies.
The first is the map task, which reads and processes a block of data to generate key-value pairs as intermediate outputs.
The Reducer receives the output from a Mapper or map task (key-value pairs).
Multiple map tasks send the key-value pair to the reducer.
The reducer then aggregates those intermediate data tuples into a smaller collection of tuples or key-value pairs that constitutes the final output.
14. Describe the tools used in Big Data.
Big Data tools include:
" Hadoop " Hive " Pig " Flume " Mahout " Sqoop
15. What are the best data analysis tools?
" Tableau " RapidMiner " OpenRefine " KNIME " Google Search Term " Solver " NodeXL " io " Alpha Wolfram
16. What Are the Different Elements of a Machine Learning Process?
Domain knowledge: The first step is to understand how to extract the various features from the data and learn more about the data that we are dealing with. It has more to do with the type of domain we are dealing with and familiarizing the system with it in order to learn more about it.
Feature Selection: This step is more concerned with the feature that we are selecting from the set of available features. Sometimes there are a lot of features, and we have to make an intelligent decision about which type of feature we want to use to move forward with our machine learning endeavor.
Algorithm: This is a critical step because the algorithms we choose will have a significant impact on the entire machine learning process. You have the option of using either the linear or nonlinear algorithms. Support Vector Machines, Decision Trees, Naive Bayes, K-Means Clustering, and other algorithms are used.
Training: This is the most significant aspect of machine learning and where it differs from traditional programming. The training is based on the data we have and includes additional real-world experiences. With each subsequent training phase, the machine improves and becomes wiser, allowing it to make better judgments.
Evaluation: In this stage, we review the machine's decisions to see whether or not they are appropriate. There are several metrics involved in this process, and we must closely deploy each of them to determine the efficacy of the entire machine learning endeavor.
Optimization : This the process of enhancing the performance of the machine learning process via the use of various optimization approaches. Optimization of machine learning is one of the most important components in which the algorithm's performance is substantially increased. The best aspect about optimization strategies is that machine learning not only consumes optimization approaches but also generates new optimization ideas.
Testing: Various tests are performed here, some of which are previously unseen sets of test cases. The data is divided into two sets: test and training. There are different tests available.
17. What Is the Difference Between Data Modeling and Database Design?
Data modeling is the initial phase in the creation of a database. Data modeling is the process of developing a conceptual model based on the relationships between distinct data models. The procedure entails progressing from the conceptual stage to the logical model and finally to the physical schema. It entails a methodical approach to using data modeling approaches.
The process of creating a database is known as Database Design. The database design generates an output that is a comprehensive database data model. Database design, strictly speaking, contains the full logical model of a database, but it can also include physical design options and storage characteristics.
18. Describe cross-validation
It is a model validation approach used to determine how well the results of a statistical study would generalize to a different data set. Typically employed in situations when the goal is to forecast and one wants to assess how correctly a model will perform in practice. The purpose of cross-validation is to define a data set to test the model on during the training phase in order to limit issues such as overfitting and get insight into how the model will generalize to an independent data set.
19. How should outlier values be handled?
Univariate or any other graphical analysis approach can be used to identify outlier values. If the number of outlier values is small, they can be evaluated separately; however, if the number of outliers is considerable, the values can be substituted with either the 99th or 1st percentile values. Outlier values are not all extreme values. The most popular methods for dealing with outlier values are:
1) changing the value and bringing it into a range
2) Simply remove the value.
20. What do you mean by "recall" and "precision"?
Measures of recall "How many of the real true samples did we label as true"
Precision is defined as "how many of all the samples we categorized as true are truly true."
21. Can you describe the distinction between a Test Set and a Validation Set?
The validation set might be regarded as a subset of the training set because it is used to choose parameters and avoid overfitting of the model being developed. A test set, on the other hand, is used to test or evaluate the performance of a trained machine learning model.
In a nutshell, the distinctions are as follows:
The purpose of the Training Set is to suit the criteria, such as weights.
The purpose of the Test Set is to evaluate the model's performance, namely its predictive power and generalization.
The validation set is used to fine-tune the settings.
22. What is the statistical power of sensitivity, and how is it calculated?
Sensitivity is widely used to verify a classifier's accuracy . Sensitivity may be defined as "predicted TRUE events/total events." True events are those that occurred and were expected to occur by the model.
Senstivity is easy to calculate: Senstivity = True Positives / Positives in Actual Dependent Variable
True positives are Positive occurrences that have been appropriately identified as Positives.
23. How does a ROC curve work?
The ROC curve is a graphical depiction of the difference between true positive and false positive rates at certain thresholds. It is frequently used to represent the trade-off between sensitivity (true positive rate) and false positive rate.
24. What are Basic Measures Derived from Confusion Matrix?
Basic measurements generated from the confusion matrix are:
1. Error Rate = (FP+FN)/(P+N)
2. Accuracy = (TP+TN)/(P+N)
3. Sensitivity (Recall or True positive rate) = TP/P
4. Specificity (True negative rate) = TN/N
5. Precision (Positive predicted value) = TP/(TP+FP)
6. F-Score (Harmonic mean of accuracy and recall) = (1+b) (PREC.REC)/(b2PREC+REC), where b is generally 0.5, 1, 2.
25. Explain in detail the SVM machine learning algorithm.
Support vector machine (SVM) is a supervised machine learning technique that may be used for both regression and classification. If your training dataset has n features, SVM attempts to plot them in n-dimensional space, with the value of each feature being the value of a certain coordinate.
SVM employs hyperplanes to divide distinct classes based on the kernel function supplied.
26. What exactly is pruning in a Decision Tree?
When we remove sub-nodes from a decision node, this is referred to as pruning, or the opposite process of splitting.
27. What do you mean by Random Forest? How does it work?
Random forest is a flexible machine learning approach that can do both regression and classification problems. It is also used for dimension reduction, missing value treatment, and outlier value treatment. It is a sort of ensemble learning approach in which a set of weak models merge to generate a powerful model.
In Random Forest, we develop numerous trees rather than a single tree. Each tree provides a classifier to categorise a new item based on properties. The forest selects the classification with the most votes, and in the case of regression, it takes the average of outputs from various trees.
28. What do you mean when you say "Normal Distribution"?
Data is often dispersed in a variety of ways, with a bias to the left or right, or it might all be mixed together. However, there is a probability that data will be dispersed about a central value with no bias to the left or right and will approach normal distribution in the form of a bell-shaped curve. A symmetrical bell-shaped curve is used to disperse the random variables.
29. Explain the concept of regularisation and why it is useful.
Regularization is the process of introducing tuning parameters into a model in order to produce smoothness and avoid overfitting. Adding a constant multiple to an existing weight vector is the most common way to accomplish this. This constant is frequently the L1 (Lasso) or L2 (Lasso) constant (ridge). Following that, the model predictions should minimize the loss function derived on the regularised training set.
30. What exactly is data science?
Data Science is a collection of tools, algorithms, and machine learning methods that aim to uncover hidden patterns in raw data.
