How to Estimate Parameter? [1] - MoM, MLE

If you live in Canada, you definitely know the Roll up the Rim! Tim Horton’s, coffee and doughnuts company in Canada, runs “Roll up the Rim to Win” marketing campaign every February. Customers know prizes such as free cup, store products even vehicles by unrolling the rim on their paper coffee cups.

Me? I drank Tim Horton’s coffee almost every day during the campaign, but I never won this year :( If you want to know what the odds of winning (probability of the first winning) actually are, first you should collect the data. For example you can ask your friends how many cups of coffee you bought until your first win? Suppose you’ve got 2, 10, 8, 9, 1, 20, 4, 3, 6, and 5 from 10 friends. These are real-valued random variable. Now you can estimate the odds of winning from this data.

This is a statistical inference. When the parameter (p=?) is unknown and you wish to estimate its value, you collect the data and treat this data as a random variable. But how? How can we formally compute the estimate? There are two ways; 1) MoM (Methods of Moments) and 2) MLE (Maximum Likelihood Estimate).

Assume we calculated the estimated parameter, from my sample. If I ask another 10 friends, I might get the different random variables so we will get the different estimated parameter, among different estimated parameters, which one is most accurate? How good our estimate will be another question. 

The main point is that an estimated parameter is a random variable with probability distribution known as the sampling distribution.

[1] Method of Moments (MoM)

The idea is to build a system of equations to solve for the parameter value. If we want to know one parameter, we only need one equation. What if we want to know two parameters, two equations are needed, which means two moments are required to find the mean and variance.

The Steps for finding the MoM are following; 1) Calculate low order moments in terms of parameters, 2) Invert the expression in terms of moments and 3) finally simply put the hat on the expression in step 2 to obtain estimator of the parameters. 

Remark

$\star$ The K^th moment: M^(K)(0)=$\mu_{k}$= E(X^k) 

$\star$ $\hat{\mu_{k}}$ can be treated as an unbiased estimate of $\mu_{k}$_.

Example) Mathematical Statistics and Data Analysis 3RD Edition Chapter 8, Q4  

Suppose that X is discrete random variable with P(X=0)=$\frac{2}{3}\theta$, P(X=1)=$\frac{1}{3}\theta$, P(X=2)=$\frac{2}{3}(1-\theta)$, P(X=3)=$\frac{1}{3}(1-\theta)$. The following 10 independent observations were taken from such a distribution. (3, 0, 2, 1, 3, 2, 1, 0, 2, 1)

a) Find the method of moment estimate of $\theta$.

b) Find the approximate standard error for your estimate.

Solution??!!

Example) Mathematical Statistics and Data Analysis 3RD Edition Chapter 8, 54 (a)

Let X1 X2...Xn be iid uniform on [0,  $\theta$]

a) Find the method of moments estimate of $\theta$ and its mean and variance.

Solution??!!

[2] Maximum Likelihood Estimate (MLE) 

What's the likelihood? 

If X₁, X₂, …, X_n are random variable with joint density f(X₁, X₂,…, X_n|$\theta$), then given observed values X_i, …, X_n like ($\theta$) = $L(\theta)$ = $L(\theta)=\prod_{i}^{n}f(X_{i}|\theta)$ 

* $\prod_{i}^{n}$ is a product notation form i to n. 

What's the MLE? 
We want to find the parameter that MAXIMIZE our probability of getting the data we obtained. That means what parameter of $\theta$ give us the most probable chance of getting the data we obtained?!! $MLE \hat{\theta }=L(\hat{\theta })= max L(\theta )$ 

So how can we find the MLE? 
By using log likelihood, as we can take a log transformation without changing the maximum value. So, $l(\theta)=log(L(\theta))=log(\prod_{i}^{n}f(X_{i}|\theta))= \sum_{i}^{n}(f(X_{i}(\theta )))$
And then take a derivative, then set the equation to equal to 0! 

Example) Parameter Estimate Example: MoM, MLE (1)

Let Y1, Y2, …, Yn denotes random sample. f(y|$\theta$)= $(\theta+1)y^\theta$, 0$<$y$<$1, -1$<\theta$, and f(y|$\theta$)=0 otherwise. 

a) $\hat{\theta}_{MoM}$ 

b) $\hat{\theta}_{MLE}$? 

Solution??!!