Yeah, the average person just has NO idea how difficult it is to run it like an actual scientific experiment if you're looking for rigorous data.
Esp if effect sizes are small and you got a sample size of you, your cat Jimmy, and your cat's best friend CJ from down the block.
So ultimately, intuition plays a much larger role than people are willing to admit.
(Most of what you see around "digital marketing" is bullshit trying to appear rigorous by sprinkling math on it like saltbae to fool imbeciles.)
That's not to say you can't ever split test anything.
But more that people use statistics without having an inkling of a clue about what they're doing.
People will draw huge conclusions from a lander that got 33% vs 31% on something.
How do you know that's significant?
What about replication?
And ultimately, there's only so much time/money you have and some things simply cannot be split tested, practically or at all.
People should have conviction on the big things and then split test the little things.
(The main reason why is because of the difficulty of accurately detecting true effects in high noise environments where there are many confounding variables that can affect the outcome and where the input and output are far apart with tons of noise in between.
This is not: "I kick the box. Something happens"
This is: "I kick the box. The box falls out the airplane. Lands in a field. A cow tries to eat it. A farmer removes it. He throws it in the bin. It gets picked up. [...] and 10 years later you try to tie the conclusion back to that kick.")
Anyway, about that big/little thing:
E.g. if Five Guys was like "The extra fries, free refills, and free toppings are costing us too much. Let's kill it." It might take months or even years to see that reflected back on the balance sheet.
Brand equity will carry you for a long time. Esp. since the majority of their customerbase, like most companies, are light buyers (0-1 a year).
So even a split test ran for a month would give you meaningless data.
These are valuable questions every entrepreneur or solopreneur MUST ask! Thanks!
- Figure out what people wanna buy.
- Who do you wanna serve?
- Where do they hang out online?
- Who are the thought leaders they're interacting with? What solutions are those guys providing & how are they talking about the problems?
- What are they complaining about?
- What products are they buying?
- What pain points do those products solve?
- How much do those products cost?
- How much is solving the problem worth to your audience?
The more I learned about the math related to A-B testing, the more I realized that almost nobody should be doing it.
Facebook and Google and all of those huge companies lie to the smaller ones.
YOUR sample sizes are just never going to be large enough.
Yeah, the average person just has NO idea how difficult it is to run it like an actual scientific experiment if you're looking for rigorous data.
Esp if effect sizes are small and you got a sample size of you, your cat Jimmy, and your cat's best friend CJ from down the block.
So ultimately, intuition plays a much larger role than people are willing to admit.
(Most of what you see around "digital marketing" is bullshit trying to appear rigorous by sprinkling math on it like saltbae to fool imbeciles.)
That's not to say you can't ever split test anything.
But more that people use statistics without having an inkling of a clue about what they're doing.
People will draw huge conclusions from a lander that got 33% vs 31% on something.
How do you know that's significant?
What about replication?
And ultimately, there's only so much time/money you have and some things simply cannot be split tested, practically or at all.
People should have conviction on the big things and then split test the little things.
(The main reason why is because of the difficulty of accurately detecting true effects in high noise environments where there are many confounding variables that can affect the outcome and where the input and output are far apart with tons of noise in between.
This is not: "I kick the box. Something happens"
This is: "I kick the box. The box falls out the airplane. Lands in a field. A cow tries to eat it. A farmer removes it. He throws it in the bin. It gets picked up. [...] and 10 years later you try to tie the conclusion back to that kick.")
Anyway, about that big/little thing:
E.g. if Five Guys was like "The extra fries, free refills, and free toppings are costing us too much. Let's kill it." It might take months or even years to see that reflected back on the balance sheet.
Brand equity will carry you for a long time. Esp. since the majority of their customerbase, like most companies, are light buyers (0-1 a year).
So even a split test ran for a month would give you meaningless data.