Mrthfx

Can anyone tell me why this program fails?

Why must traveling waves have the same amplitude to form a standing wave?

Meaning of "SEVERA INDEOVI VAS" from 3rd Century slab

How to deal with a cynical class?

What is the greatest age difference between a married couple in Tanach?

Making a sword in the stone, in a medieval world without magic

What are some nice/clever ways to introduce the tonic's dominant seventh chord?

PlotLabels with equations not expressions

Bash: What does "masking return values" mean?

Brexit - No Deal Rejection

Will a pinhole camera work with instant film?

Unreachable code, but reachable with exception

The use of "touch" and "touch on" in context

Happy pi day, everyone!

Know when to turn notes upside-down(eighth notes, sixteen notes, etc.)

Be in awe of my brilliance!

What are the possible solutions of the given equation?

Define, (actually define) the "stability" and "energy" of a compound

Does splitting a potentially monolithic application into several smaller ones help prevent bugs?

Is it possible to upcast ritual spells?

Instead of Universal Basic Income, why not Universal Basic NEEDS?

At what level can a dragon innately cast its spells?

How do anti-virus programs start at Windows boot?

Good allowance savings plan?

Distribution of Maximum Likelihood Estimator

Maximum likelihood of function of the mean on a restricted parameter spaceMaximum Likelihood Estimator (MLE)Maximum Likelihood Estimator of the exponential function parameter based on Order StatisticsFind maximum likelihood estimateMaximum Likelihood Estimation in case of some specific uniform distributionsMaximum likelihood estimate for a univariate gaussianFind the Maximum Likelihood Estimator given two pdfsMaximum Likelihood Estimate for a likelihood defined by partsVariance of distribution for maximum likelihood estimatorHow is $P(D;theta) = P(D|theta)$?

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)

Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$

$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)

Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$

$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)

Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$

$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

1 Answer
1

active

oldest

votes

3

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f397619%2fdistribution-of-maximum-likelihood-estimator%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

3

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 3 hours ago

dlnBdlnB

3916