• The sample size is large enough to ensure that $np>5$ and $n(1-p)>5$.
• The other conditions for before and after the feature is turned on are similar
• The are data from the samples before and after are independent of one another
• The samples are random, representative and non-bias.

• H₀: p₁− p₂ = 0
• H_A: p₁− p₂ < 0

To determine which tail we are working with (left or right), we need to think about what we are looking to prove: the click-through with the feature enabled (p₂) is higher than the click-through rate without the feature enabled (p₁). This gives the following:

• Click-through with feature (p₂) is higher than the rate without the feature (p₁)
• This gives: p₂ > p₁
• Moving p₂ to the right side gives: 0 > p₁− p₂
• Flipping the inequality (read it right to left) gives: p₁−p₂ < 0
• If we want to prove that p₁−p₂ < 0, we are performing a left-tailed test.

This is a left-tailed test. If p₁ is much lower than p₂, then their difference will be negative and the test statistic (z_test) will be low enough to be in the rejection region on the far left of the graph:

First, let us calculate the sample proportions:

• $x_1 = 56$ and $n_1 = 2946$ gives $\bar{p_1} = \frac{56}{2946}= 0.0190$
• $x_2 = 77$ and $n_2 = 3054$ gives $\bar{p_2} = \frac{77}{3054}= 0.0252$
• $\bar{p} = \frac{x_1+x_2}{n_1+n_2} = \frac{56+77}{2946+3054} = 0.02217$

We can now calculate the test statistic:

$z_{test} = \frac{\bar{p_1}-\bar{p_2}}{\sqrt{\bar{p}(1-\bar{p})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}} = \frac{0.0191-0.0252}{\sqrt{0.02217(1-0.02217)\left(\frac{1}{2946}+\frac{1}{3054}\right)}} = -1.6318$

We are performing a left-tailed and use Excel’s NORM.S.DIST function:

$p\text{-value}=\text{NORM.S.DIST}(-1.6318, 1) = 0.0514$

Because the p-value = 0.0514 (5.14%) is greater than (>) the level of significance of 5% (0.05), we fail to reject H₀.

Because we failed to reject H₀, there is not sufficient evidence to conclude that the click-through rate is higher when the feature is turned on than when it’s turned off.

In differences between Two Proportions testing (this example):

• We are testing if the two populations are different (before and after the feature is enabled).
• We are assuming that we don’t know the ‘true’ proportion but have two different sample proportions (with and without the feature enabled) recorded.
• We use both sample proportions to come up with a ‘best guess’ for the population proportion, which we call the pooled sample proportion (p̄).

In One-Sample Proportion Hypothesis Testing (Example 58.1):

• We are testing the true proportion (the click-through rate) has changed from its historical value (the rate before the feature was enabled)
• We treat the historical value as our population proportion
• We compare to this amount to see if there is any difference

We want to prove that the rate after the feature is enabled is higher than without the feature being enabled. If we set the following to be true:

• p₁ = the true click-through rate after the feature is enabled
• p₂ = the true click-through rate after the feature is enabled

Then, when proving that the click-through rate is higher with the feature enabled, we get:

• H_A: p₁ > p₂ or H_A: p₁ − p₂ > 0
• This is a right-tailed test

From the previous section, we can determine the null and alternate hypotheses:

• H₀: p₁ = p₂ or H_A: p₁ − p₂ = 0
• H_A: p₁ > p₂ or H_A: p₁ − p₂ > 0

Let us first calculate the sample proportions:

• $x_1 = 77$, $n_1 = 3054$, $p_1 = \frac{77}{3054}=0.0252$
• $x_2 = 56$, $n_2 = 2946$, $p_2 = \frac{56}{2946}=0.0190$
• $\bar{p}=\frac{x_1+x_2}{n_1+n_2}=\frac{77+56}{3054+2946} = 0.02217$

We can now use those proportions to calculate the test statistic:

$z_{test} = \frac{\bar{p_1}-\bar{p_2}}{\sqrt{\bar{p}(1-\bar{p})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}} = \frac{0.0252-0.0191}{\sqrt{0.02217(1-0.02217)\left(\frac{1}{3054}+\frac{1}{2946}\right)}} = 1.6318$

We are performing a right-tailed and use Excel’s NORM.S.DIST function:

$p\text{-value}=1-\text{NORM.S.DIST}(1.6318, 1) = 0.0514$

Because the p-value = 0.0514 (5.14%) is greater than (>) the level of significance of 5% (0.05), we fail to reject H₀.

Because we failed to reject H₀, there is not sufficient evidence to conclude that the click-through rate is higher when the feature is turned on than when it’s turned off.