How accurate is our data?

Observed Data

Data/Metric	Calculation	Expected Accuracy	Why useful?
Ad Copy	For each search term captured for you or a competitor, we record every single advert we have seen over the period, whether run by you or a relevant competitor.	100%	See the offers and messaging competitors use.
Ad duration	Calculated by recording the first and last date we saw an advert. (This does assume the advert was running without any breaks).	64% -100%	Determine how frequently competitors are changing their ad copy and when they are A/B testing.
Ad granularity	Probability that we have seen every search term that triggered an ad.		Benchmark yourself. Smaller ad groups with tailored ad copy tend to have higher QS, CTR and so Avg Pos. Higher QS means lower CPCs.
Frequency	Of all the times we searched a term during a specific period, how many times did a particular competitor appear?	70% - 99%	Is a competitor’s strategy to appear less frequently in a higher position or vice versa?
Position (Best/Avg)	What is the best position and average position a competitor appears in for a search term? If looking over multiple terms, weight the position by search volume.	70% - 99%	Which competitors most often appear in top positions? Does this fluctuate or generally remain static?
Search Volume	Google provides an average monthly search volume for every search term.		Able to weight terms, i.e. to give a term with twice the searches double the importance.
Search Term Overlap	For which search terms do you and a competitor both appear?	70% - 99%	If two competitors appear on a high-volume term, but you do not, then it is likely to be worth investigating.

 

Estimated Data

Data/Metric	Calculation	Expected Accuracy	Comments
Impressions	Take search volume for every search term / 30 (days in a month) X Display Length (you have selected) X Frequency (which is weighted by search volume).	†† (see below)	Google rounds its search volume number and it does not fully take into account events that cause large spikes.
CTR	Estimated by the data model and updated nightly.	12-17% error	On average the error in the predictions is between 12% and 17%, († see below).
Clicks	Take the estimated impressions X CTR.	†† (see below)	Accuracy is affected by the above, but most important is consistency, so that you can see your trends over time and how your competitor’s behaviour affects your share of clicks.
CPC	Estimated by the data model and updated nightly.	12-17% error	On average the error in the predictions is between 12% and 17% († see below)
Share of Spend	Take the estimated clicks X CPC for one competitor and divide it by the total for all competitors.		Accuracy is affected by all of the above, but consistency is most important is  so that you can see your trends over time and how your competitor’s behaviour affects your share of spend.

 

† Accuracy of Click-Through-Rate and Cost-Per-Click Estimates (CTR & CPC)

These vary from client to client, as well as between our clients and their competitors. We measure the accuracy of the estimated client CTR and CPC on a daily basis. At present the mean absolute percentage error of estimates for CTR is roughly 12%, while the mean absolute percentage error of the estimates for cost-per-click is approximately 17%. Measuring the accuracy of click-through-rate and cost-per-click estimates is a more difficult problem and is currently under work.

†† Accuracy of Impressions and Clicks

To estimate the number of impressions that an advertiser obtains for a given search term, we take the search volume from Google for that term* and then multiply it by the frequency with which the advertiser’s advert is displayed. We use the number of times we observe a specific advert to estimate the frequency with which an advertiser’s advert is displayed on that search term. To estimate the number of clicks we take the estimate for the number of impressions that an advertiser will receive on a search term, then multiply it by our estimate for the click-through-rate of that advertiser on that search term.

*This figure is provided directly by Google

Observing Competitors in the market

For each search term, how likely are we to see a given advertiser in a day’s worth of scrapes? This depends on various things, such as the number of advertisers in the market, the likelihood that a given advertiser will appear on the search term and the number of times we scrape the search term in a given day.

If all advertisers on the market are equally likely to appear on the given search term, then the likelihood that we will see a particular advertiser on that search term is given in the following table.

	10 Advertisers	20 Advertisers	50 Advertisers
8 scrapes	99.99%	96.81%	70.00%
10 scrapes	99.99%	98.65%	77.86%
15 scrapes	99.99%	99.84%	89.58%
20 scrapes	99.99%	99.98%	95.10%

 

Alternatively, if an advertiser appears on a search term for a certain percentage of the time, then the likelihood that we will see a particular advertiser on that term is as follows.

Percentage of time advertiser appears	10%	20%	50%	80%
8 scrapes	56.95%	83.22%	99.60%	99.99%
10 scrapes	65.13%	89.26%	99.90%	99.99%
15 scrapes	79.41%	96.48%	99.99%	99.99%
20 scrapes	87.84%	98.84%	99.99%	99.99%

 

Given that Google restricts advertisers’ appearances each hour, depending on their budget, it is crucial to scrape as many times per day as possible to ensure that smaller advertisers are picked up accurately.

Example: if an advertiser appears 50%+ of the time per day on a specific search term then there is a 99% chance that Adthena will see them. However, if they appear only 10% of the time then the likelihood of seeing them drops to less than 60%. If it is a competitive term with more than 10 Advertisers appearing on page 1, then we can increase the scrape frequency to improve the likelihood of seeing the smaller advertiser from 60% to 80%.

Competitors’ Advert Duration

When a competitor runs advertising on a search term, how accurately can we estimate the length of time that they ran the campaign? Again, this depends on factors like the number of competitors in the market, the likelihood that the competitor will successfully appear on the search term and the number of times a day that we scrape that  term.

Assuming we scrape the search term 16 times a day, and that the campaign is running for two weeks, then on average we obtain the following accuracy.

	10 Advertisers	20 Advertisers	50 Advertisers
Exact	99.99%	98.58%	26.89%
To within 1 day	100.00%	99.99%	63.92%
To within 2 days	100.00%	99.99%	87.59%
To within 3 days	100.00%	99.99%	96.90%

 

If we know the likelihood with which an advertiser will appear on the given search term, then we will typically obtain the following accuracy.

Percentage of time advertiser appears	20%	50%	80%
Exact	67.05%	99.97%	99.99%
To within 1 day	94.23%	99.99%	99.99%
To within 2 days	99.35%	99.99%	99.99%
To within 3 days	99.94%	99.99%	99.99%

 

Search Term Overlap

If you and a competitor are bidding on a group of search terms, then on what percentage of these overlapping terms are we likely to see the competitor? Assume that there are 1000 overlapping terms and we scrape each term 12 times a day. In the following table we give the 95% confidence interval of the percentage of the overlap for which we will see the competitor on a given day. The table gives difference confidence intervals dependent on the likelihood that we will see the competitor on a given scrape.

Percentage of time advertiser appears	10%	20%	30%
95% confidence interval	[68.9%, 74.9%]	[91.5%, 94.6%]	[97.8%, 99.2%]