⏳ Mining deep into Data Mining - Statistics - PART I ⏳

 Why do we have to know statistics?πŸ€”

As mentioned in the previous posts, we live in the world of data from which we can derive insightful information. Thus, Statistics play a vital role in processing and analyzing the data to make decisions and predictions.

What actually is statistics? πŸ‘€ Let's get more technical 

Statistics is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. There are two types or classes of statistics.
            πŸ‘‰  Descriptive statistics
            πŸ‘‰  Inferential statistics

Descriptive Statistics πŸ˜€

            πŸ‘‰ Descriptive statistics focuses more on analyzing, summarizing, and organizing data in the form of numbers or graphs.

            πŸ‘‰ Bar plots, histograms, pie-charts are used in visualizing descriptive data and determining PDF (probability density function), CDF (cumulative distribution function), Normal distribution.

            πŸ‘‰ Measure of central tendency is determined by mean, median, and mode.
    
            πŸ‘‰ Measure of variance - variance and standard deviation.

How to differentiate descriptive and inferential statistics? πŸ€”

            A suitable example would be the Election Exit poll data (ie) The poll of voters taken after they have exited the polling stations to predict or pre-determine the chances of the winning party. 


πŸ‘‰ Consider any state in India, say XYZ with a population of 7 million people. Once the voting process commences, the exit polls provide predictive data about the performance of each party.

πŸ‘‰ Instead of considering the entire 7 million population, a sample mass of people is considered say 10,000 in each area and the data is collected.

πŸ‘‰ From the data, say party_1 has a majority in area_1, party_2 has a majority in area_2  (ie) we infer a piece of information by testing the sample data to obtain the information. 

πŸ‘‰ In inferential statistics, Confidence interval plays an important role. It denotes the probability of any parameter (in our case, voting count) will fall between a pair of values around the mean. 

πŸ‘‰ Hypothesis testings - such as Z test, T-test, Chi-square tests are performed here. 

Population vs Sample πŸ’₯

Let us consider a problem that is determined to find the Average height of all the people of a state XYZ.  Let the total population count of the state be 1 million. 

πŸ‘‰ Population is defined to be the total observations present (ie) the entire 1 million data. It is denoted by N.

πŸ‘‰ Population mean (ΞΌ) is the average of the group of observations in the population. In our case, it is the average height of all the people.
                                        
Let i denote every instance or observation and h denotes the height of an observation.

πŸ‘‰ It would be tedious to handle the entire population data, thus a sample of data is considered for better observation and analytics. 

πŸ‘‰ Sample is any subset of the population data. In our example, out of the 1 million population, a sample of 50,000 observations can be considered. It is generally denoted by 'n'.

πŸ‘‰ Sample Mean ( x̅) is the average of the group of observations in the sample. In our case, it is the average height of people in the sample data. 
                                    
Let 'i' denote every instance or observation and 'h' denote the height of an observation.

Random Variables

First, let's understand what a variable is. In programming languages, a variable is a name given to a memory location or we can call it a placeholder to store some values.

Similarly, a random variable is the one that is used to hold a value and the value is initially unknown. Simple as that !!.

Types of Random variable
            
                πŸ‘‰ Discrete random variable - It is a variable that holds a whole number or a finite floating-point value. It must not be continuously finite.

Example: 
                πŸ‘‰ The total number of employees in a company, the average salary of an employee, floating values such as 0.3, 0.0025 are discrete.
                πŸ‘‰ 3.142356.... cannot be defined as discrete. 

                πŸ‘‰ Continuous random variable - Any value within the given range is acceptable.  Be it a whole number or a decimal number.

Example: 
                πŸ‘‰ The height of a person may range from  150 to 170 cm. So we may assume any values such as 168.98, 159, 144.345 cms.


Let's explore more about distributions in the successive posts.

                

                           πŸ’‘ Let's keep mining deeper  !!! πŸ’‘


Comments

Post a Comment

Articles by Hemapriya

Comprehending the state-of-art Digit Recognizer dataset using machine learning

Image
   Handwriting Recognition Handwritten text recognition has been a challenge since the first automatic machines were required to identify individual characters in handwritten texts.Consider the five-digit ZIP codes on letters at the post office and the automation used to identify them.To sort mail automatically and efficiently, perfect understanding of these codes is required. Included among the other applications that may come to mind is OCR (Optical Character Recognition) software. OCR software must read handwritten text, or pages of printed books, for general electronic documents in which each character is well defined. But the problem of handwriting recognition goes farther back in time, more precisely to the early 20th Century (1920s), when Emanuel Goldberg (1881–1970) began his studies regarding this issue and suggested that a statistical approach would be an optimal choice.  To address this issue in Python, the scikit-learn library provides a good example to better understand th
Read more

Performance Analysis of Weather Data using Machine Learning

Image
   Weather and Climate Observations Weather data is primarily important for determining the climate of a region. Climate is determined by a number of factors.The formation and advancement of storm systems, the amount of precipitation an area gets, and the number of cloudy days are all influenced by air pressure, temperature, and humidity at various altitudes.These influences affect the environment on a local, international, and global scale over time. Why performance analysis of weather data is important ? The value of weather data analytics in human life is immense. Accurate weather forecasting is beneficial to the agricultural industry, tourism, and preparing for natural disasters such as floods and droughts. Weather forecasting has a lot of economic appeal in news organisations, government agencies, and industrial agriculture. Performance analysis of meteorological data: we can use weather and climate datasets to better understand and forecast the effect on shipping and logistics pr
Read more

Market Basket Analysis using Association Rule-Mining in R language

Image
Association mining is usually done on transactions data from a retail market or from an online e-commerce store. Since most transactions data is large, the apriori algorithm makes it easier to find these patterns or rules quickly. Association Rules are widely used to analyze retail basket or transaction data, and are intended to identify strong rules discovered in transaction data using measures of interestingness, based on the concept of strong rules.  Apriori uses a "bottom up" approach, where frequent subsets are extended one item at a time (a step known as candidate generation), and groups of candidates are tested against the data. The algorithm terminates when no further successful extensions are found. DATASET:  Groceries_dataset Let's code and analyse the algorithm πŸ’ͺ πŸ‘‰ Import the groceries dataset πŸ‘‰Explore the data πŸ‘‰ Perform data preparation such as checking the Null values, normalising the format of data to numeric values and group the data of similar values
Read more