The first lesson I learned about logarithms is not to mention them when speaking to a large crowd. All the other lessons sucked, so I thought I'd create my own.
Let's start by talking about the simplest of linear scales: a ruler.
Everyone understands how rulers work: you place them next to tiny trinkets and you look at the ruler's marks to determine the object's size. You can then use a pen and the ruler's marks to represent the size of these objects on a board.
<img src="/posts/logs/log1.png" alt="">
The problem with rulers is that you can’t use them to measure larger objects, such as trees, park benches, and city blocks. To solve that problem, we'll need a bit of imagination.
Instead of having each of our ruler's marks represent 1 inch, we'll pretend they represent 100 inches. That way, we'll compress larger distances into smaller ones.
<img src="/posts/logs/log2.png" alt="">
Now, we can use the same ruler to represent the size of a park bench, a tree, and even a city block.
<img src="/posts/logs/log3.png" alt="">
Notice how we're not using the ruler for measuring anymore. Instead, we're using it to represent the sizes of different objects.
The way we squished hundreds of inches into one mark on the ruler is called a linear transformation because when you draw it on a graph, it makes a straight line. Each mark on the ruler represents 100 more inches, showing a steady, even increase.
<img src="/posts/logs/log4.png" alt="">
Linear transformations preserve the measurements' relative spaces.
For instance, if the marks for 1 and 5 inches are 4 segments apart on a ruler, then the marks for 100 and 500 inches will also be 4 segments apart when we use our scaled-up ruler. That way, the proportional distances remain the same, even though the actual measurements have increased.
<img src="/posts/logs/log5.png" alt="">
<img src="/posts/logs/log6.png" alt="">
The problem with applying a linear transformation to a ruler is that it can only either measure small trinkets or large objects, not both.
A ruler scaled from 0 to 10 inches isn't big enough to measure a park bench, while a ruler scaled from 0 to 900 inches makes it hard to distinguish the marks for smaller objects — they become too close together.
<img src="/posts/logs/log7.png" alt="">
To solve this problem, we'll need to transform our ruler so that each segment represents a greater increase.
Typically, going from the first mark to the second represents a 1-inch increase, just like there's a 1-inch increase when going from the second mark to the third. In other words, segments in normal rulers represent linear increases.
In our new scale, going from the first mark to the second represents a 10-inch increase, and going from the second to the third mark represents a 100-inch increase. Each subsequent segment represents a distance ten times greater than the previous. In this case, each segment represents a non-linear increase.
<img src="/posts/logs/log8.png" alt="">
This way, with just 10 marks, you can represent objects from 1 inch to 100,000,000 inches. Each step exponentially increases the range we can measure without cluttering our ruler with countless tiny increments.
Now, you can represent tiny stones at the first mark and a mountain just 10 segments further because, by the time you get to the 10th mark, the value this mark represents hasn't just grown tenfold, it's grown 10¹⁰ inches.
Here's another example using our new scale. If you plot a cockroach in the scale's first mark, a badger in the second mark, and a troll in the third, it means that the badger is 10x larger than the cockroach and that the troll is 10x larger than the badger.
Furthermore, it shows that the troll is 100x larger than the cockroach, because they're 2 segments apart, so 100 (10²) times larger.
<img src="/posts/logs/log9.png" alt="">
Going left has the opposite effect, it causes things to become increasingly smaller. If you keep going left from 1, you'll start working with fractions, and they get 10x as small as you move on to the leftward mark.
<img src="/posts/logs/log10.png" alt="">
As we've just demonstrated, this type of scale is great for representing ratios.
Now compare how the represented values increase exponentially as you move up marks. See how it's not a linear growth like in the previous transformation?
<img src="/posts/logs/log11.png" alt="">
As shown above, each mark represents an exponential increase in the value it represents. Still, this type of scale is not an exponential scale. It's a logarithmic scale.
If you were to plot the actual values against their corresponding logarithmic values, you'd notice that it resembles the plot of a logarithmic function. This resemblance occurs because, as the actual values increase, the increase in their logarithmic values becomes increasingly smaller.
<img src="/posts/logs/log12.png" alt="">
An interesting effect of using the logarithmic scale is that small values get relatively more distant, while larger values get relatively closer.
<img src="/posts/logs/log13.png" alt="">
The reason you should care about this change in relative distances is because it may cause unintended distortions, or, in the case of ill-intentioned people, intended distortions.
For example. Assume that a general council of alchemists doesn't want people to think there's a dragon fever pandemic. In that case, they can plot the number of infected people using a log scale, making it seem linear, when, in fact, it's exponential.
<img src="/posts/logs/log14.png" alt="">
However, there might be compelling reasons behind the decision to use a logarithmic transformation for this chart.
Imagine there are two kingdoms, Hyrule and the Mushroom Kingdom, each with vastly different population sizes. Hyrule has 10,000 inhabitants, while the Mushroom Kingdom boasts 10,000,000.
Now, let's say a disease outbreak occurs in both kingdoms, starting with 10 infected individuals in each. On a regular linear scale, the number of infected individuals in Hyrule might seem insignificant compared to the Mushroom Kingdom's numbers.
<img src="/posts/logs/log15.png" alt="">
If we switch to a logarithmic scale, the picture changes. The slope of the curve representing the increase in infected individuals in both kingdoms becomes comparable. So, when the disease spreads from 10 to 100 infected individuals in Hyrule, the curve's slope remains consistent. Similarly, when the Mushroom Kingdom sees its infected population rise from 10,000 to 100,000, the slope of its curve mirrors that of Hyrule's.
<img src="/posts/logs/log16.png" alt="">
This scaling ensures that the threat level posed by the disease is communicated effectively in both kingdoms, regardless of their size. Without it, Hyrule's situation might appear deceptively less severe simply because of its smaller population. By utilizing a logarithmic scale, public health officials can accurately assess and address the risk posed by the disease in each kingdom, enabling more informed decision-making and resource allocation.
In addition to impacting visualizations, logarithmic transformations are also useful for statistical purposes, mostly because of:
<ol>
<li>Symmetry. Logarithmic transformations may make data more balanced, which is essential for certain statistical techniques to work effectively.</li>
<li>Homoscedasticity. Logarithmic transformations may help keep the spread of data consistent, which helps compare effects on different groups.</li>
<li>Linearity. Logarithmic transformations may make it easier to see straight-line patterns in data, simplifying analysis.</li>
</ol>
 
<h2>Symmetry</h2>
A graph is symmetric concerning a line if reflecting the graph over that line leaves the graph unchanged.
This histogram of dragon sizes is highly symmetric, for example.
<img src="/posts/logs/log17.png" alt="">
On the other hand, this graph of magic power levels in a sample of healthy wizards is not very symmetric. Instead, it's said to be positively skewed because of its long tail of results to the right.
<img src="/posts/logs/log18.png" alt="">
Now, pay attention to this magic trick: we can make this graph symmetric by applying a logarithmic transformation to all the magic power levels.
Such transformation makes these values more symmetric because the long tail of large values gets compressed, while the small values concentrated on the left get pushed farther apart.
<img src="/posts/logs/log19.png" alt="">
Symmetry is vital in data analysis for various reasons. For example, when using statistical methods, we typically aim to find a mathematical model that accurately represents the distribution of our data. The choice of an appropriate model often depends on whether the data is symmetric.
Anyway, discussing the details of model selection is a topic for another time.
 
<h2>Homoscedasticity</h2>
In the land of Skyrim, knights rely on magical potions to enhance their strength during battles.
When we graph the effectiveness of these potions based on the initial strength of the knights, we notice greater variability in the potion's impact. This variability is a common occurrence with potions and medicines.
For instance, if a knight initially lifts 100 pounds, the potion might yield effects ranging from no change (100 pounds) to a significant increase (160 pounds, a gain of +60 pounds). Conversely, if a knight can lift 500 pounds, the potion's effects could range from maintaining the initial strength (500 pounds) to a substantial increase (800 pounds, a gain of +300 pounds).
When we plot this data on a linear scale, we notice that the variance increases as the knight's initial strength rises. This phenomenon, known as heteroscedasticity, reflects the heterogeneous nature of the data, where the variability of the potion's effects differs across the range of initial strengths.
<img src="/posts/logs/log20.png" alt="">
When we apply a log transformation to our graphs, it makes the data more homoscedastic, meaning it evens out the variance, especially for stronger knights. In simpler terms, this means that the potion's effects become more consistent across different strength levels.
This transformation reveals that the potion consistently boosts strength by around 0 to 20%, making it easier to predict its benefits for knights at all training levels.
<img src="/posts/logs/log21.png" alt="">
 
<h2>Linearity</h2>
Let's resort to our imagination again.
Let's say someone's aura indicates one's affinity with celestial magic, and Moonbeam Extract is said to degrade this connection.
Imagine a scenario where our kingdom's alchemists seek to understand the relationship between Moonbeam Extract and someone's aura.
When we plot people's aura's brilliance against Moonbeam Extract, we notice a curved pattern, with most data clustered at the lower left corner. That plot gives us some clues about the relationship between people's auras and Moonbeam Extract, but it's not very clear.
When we employ logarithmic transformations on both magical properties, the correlation appears straight and clear.
<img src="/posts/logs/log22.png" alt="">
This linearity simplifies our understanding, allowing us to see how the effects of Moonbeam Extract clearly impact people's magic abilities.
 
<h2>A final note on log bases</h2>
Throughout this post, we've mostly used common logs, where each segment represents a tenfold increase compared to the last (log10).
Still, it's worth noting that we can also use other log bases, such as log2 or natural logarithms (logE). These alternative log bases offer different perspectives and may be better suited for specific situations or data types.
 
<h2>About these visualizations</h2>
We've built all these visualizations using Briefer.
If you want to build reports, presentations, or data apps with visualizations like these, you can sign up for early access to Briefer at <a href="https://briefer.cloud">briefer.cloud</a>.

A brief introduction to logarithmic scales and their applications in data visualization and statistics.

Log scales

This week we've released two major new features that we're very excited about: snapshots and a Slack integration.
<h2>Snapshots</h2>
Snapshots allow you to save the state of your document at any time, and then restore it later.
<img src="/posts/changelog-5/snapshots.gif" alt="">
This is useful when you want to experiment with your code, but you're afraid of losing your progress. With snapshots, you can save your work before making changes, and then restore it if you don't like the changes.
Snapshots are also useful to keep track of how your metrics or visualizations change over time. In Briefer, every scheduled run creates a new snapshot, so you can compare results over time.
To use them, click the "Snapshots" button on the upper-right corner of the editor. Then, click the "Save" button to save the current state of your document. You can also restore a snapshot by clicking the desired snapshot, and then clicking the link to restore it at the top of the page.
Snapshots are only available for users in our professional plan, and their retention period is 90 days.
 
<h2>Slack Integration</h2>
Our Slack integration allows you to receive PDF reports directly in Slack.
<img src="/posts/changelog-5/exports.gif" alt="">
To use it, you need connect your Slack workspace to Briefer. To do that, access the "integrations" menu at the bottom of our sidebar. Then, click the "Add" button and select "Slack".
After connecting your Slack workspace, add a scheduled run to your document, and, as you're doing it, click the "Add" button right next to the notification settings.
There, you will be able to select the Slack channel where you want to receive the PDF report.
Then, click the "Add Slack" button, and follow the instructions to connect your Slack account to Briefer.
The Slack integration is only available for users in our professional plan.
Important: You need to invite the Briefer bot to the desired channel before it can send notifications (<code>/invite @Briefer</code>). You won't receive notifications if your page fails to execute.
 
<h2>General Improvements and Bug Fixes</h2>
<ul>
<li>We will automatically execute previous blocks after your environment restarts. That way, you will not face issues with variables not existing anymore.</li>
<li>You can now expand your document horizontally.</li>
<li>Visualizations take 4x less time to load.</li>
<li>We've improved the layout of text blocks. Text will look better now.</li>
<li>We've fixed a few performance issues while switching tabs with multiple editors.</li>
</ul>

We've just released snapshots. They allow you to save the state of your document at any time, and then restore it later. We've also added a Slack integration, so you can receive PDF reports directly in Slack.

Changelog #5: Snapshots and a Slack integration

When we got into YC, we were building a product for companies to spin up development environments in the cloud. The idea was that, as software grows, it inevitably becomes too difficult to run on a single machine. At that point, people would come to us and we'd help each developer an environment in the cloud.
It didn't work.
During our batch, we amassed <a href="https://github.com/briefercloud/layerform">a thousand stars on GitHub</a>, <a href="https://news.ycombinator.com/item?id=37134293">got to HackerNews' front page</a>, and even had a customer pay for our product. Despite the short-term achievements, we soon realized that long-term success was probably not in the cards.
After the batch, we pivoted away from building infrastructure products.
Here's how we made that decision, the lessons we learned, and how I'd approach it differently if I had to do it again.
 
<h2>Too simple to be useful, or too useful to be simple</h2>
It's extremely difficult to build an infra product that's both simple and flexible enough.
When building an infrastructure product you usually have to choose whether you want to serve small or large companies first. That's because these two types of companies care about diametrically opposite things.
Large companies want flexible products that can support their peculiar infrastructure needs, even if it takes a bit longer to set up. On the other hand, small companies would rather have a simple piece of software that takes only a few minutes to set up, even if it doesn't support esoteric network settings, for example.
If you decide to build an infra product for large companies, it will almost certainly be too complex for small companies to use. Conversely, designing an infra product for small companies will make it too simple for large companies to use.
<img src="/posts/pivoting-away-from-infra/complexity-quadrants.png" alt="">
In an ideal world, you'd be able to hide complexity from small companies, while still supporting the needs of large companies. But in practice, this is nearly impossible to achieve - that's why so few infrastructure products are successful.
 
<h2>Small companies don't need it</h2>
We knew that selling infrastructure to large companies was going to be difficult, so we decided to focus on building a tiny product that was useful for small companies first.
<img src="/posts/pivoting-away-from-infra/our-quadrant.png" alt="">
Building a small product would allow us to iterate faster because we'd have customers early on, and we'd be able to get feedback from them. Then, we'd be able to raise capital to expand the product and go after larger companies.
At the time, we thought that pull-request previews were a good candidate for an MVP. We could start by deploying preview environments for pull requests, and then we would expand the product so that people could use these environments for development too.
The first version of our product was simple. It used a <code>docker-compose.yml</code> file to deploy pull requests and then posted a comment on the pull request with a link to the preview deployment.
<img src="/posts/pivoting-away-from-infra/pr.png" alt="">
Just a few days after we put a landing page up, a small company found us and started using the product. As soon as they hit the free tier's limits, they reached out to pay.
We then tried to replicate that success with other companies, but it was extremely difficult. That's because a small company's software is usually small. Thus, they can just check out the branch and get it running in a few seconds. Having preview links was convenient, but not game-changing enough to make them pay.
The same principle applies to other infrastructure products. Even if your software is pretty simple, small companies won't care about it unless it's absolutely essential for them. And even then, they might not want to pay for it, and might just default to a free alternative.
 
<h2>Large companies won't trust you</h2>
After our small bet failed, we decided to rebuild our product to serve large companies, but we had a hard time getting them to trust us.
When deciding whether to purchase a product, buyers usually have the following equation in mind:
<img src="/posts/pivoting-away-from-infra/value-equation.png" alt="">
The higher the value, the more likely they are to buy.
The problem with infrastructure products is that the time-to-value and implementation effort variables are usually too high, while the perceived probability of success is too low. Consequently, no matter how amazing the dream outcome is, the final value is usually too low for buyers to move forward.
Now, let's dissect each of these variables and explain why the perceived probability of success is usually too low, and why time-to-value and implementation effort are usually too high.
 
<h3>Perceived probability of success</h3>
<img src="/posts/pivoting-away-from-infra/perceived-probability-of-success.png" alt="">
Imagine you're running a multi-billion dollar company. Would you trust a scrappy 2-person startup to handle the infrastructure that keeps your engineers productive? I wouldn't.
Sometimes, not even large providers like AWS or Google Cloud can support these companies without a lot of hand-holding. So, why would they trust a 2-person startup?
At large companies, not rocking the boat is usually more important than getting the latest and greatest technology. Consequently, they won't trust you unless you have solid proof that you're at least 10x better than the current alternative.
 
<h3>Effort</h3>
<img src="/posts/pivoting-away-from-infra/effort.png" alt="">
Creating an infrastructure product that demands little effort to implement is extremely difficult because of the very nature of the problem you're solving. You're building a product that has to be flexible enough to support all the unique needs of large companies, and that's usually complex, as we've seen earlier.
Just for engineers to understand how your product works, they'll have to spend a lot of time reading documentation, watching videos, and maybe even attending training sessions.
Furthermore, it's difficult to break down the problem into smaller pieces that can be solved independently. That's because the problems you're solving are usually interconnected, and solving one might create new problems that need to be solved.
Finally, considering all this effort, they'll probably default to having their own engineers build it, instead of buying your product. That's because they'll have more control over the final product, and they'll be able to customize it to their unique needs.
Ah, did I mention they hired tons of engineers who need to justify their next promotion?
 
<h3>Time-to-value</h3>
<img src="/posts/pivoting-away-from-infra/time-to-value.png" alt="">
In addition to all the time engineers will have to spend learning how to use your product and set it up, it will take a while for them to see the productivity gains you promised.
That's because infrastructure products tend to deliver compounding value over time instead of delivering acute value upfront.
Take CI/CD products, for example. Testing and deploying your software for a month isn't that painful and doesn't take that much time. However, if you have to do it for a year, it becomes a huge pain point, and you'll want to automate it.
 
<h2>Slow feedback loops</h2>
The more feedback you have, the smaller the bets you have to make, and thus, the less risk you have to take.
Think about buying 3-digit lottery tickets, for example. Paying $1 for each digit as numbers are drawn is better than paying $3 for all three digits at once.
If you have to buy all three digits at once, the likelihood of guessing all three is only 1%, and each attempt costs you $3.
<img src="/posts/pivoting-away-from-infra/lottery-3-digits.png" alt="">
On the other hand, if you pay $1 per digit, you can decide whether to continue as soon as the first number is drawn. If you get it right, you can pay for the next; if not, you can give up sooner and lose less money.
In this case, there’s only a 10% chance you’ll buy the second number and a 1% chance you’ll buy the third. Therefore, each failed attempt will cost you less, but the overall likelihood of winning is still the same.
<img src="/posts/pivoting-away-from-infra/lottery-1-digit.png" alt="">
Building an infra product is like buying all three digits at once. You have to make a large bet upfront, and you won't know if you're building the right thing until you get your first customer. That's because it's difficult for large companies to trust you, and, even when they do, slow sales cycles and long implementation times will elongate the feedback loop.
Additionally, it will take longer to have a large enough sample size to know if your product is any good, and to understand whether a request is a one-off or a recurring need for most customers.
 
<h2>How I would approach it differently if I had to do it again</h2>
When picking a startup idea, you're also picking a set of problems that come with it. Ultimately, all problems are solvable, but some are more difficult than others for a given team.
Even though I'm glad we pivoted away from infrastructure products, I still believe they're a great opportunity for the right team, so here's how I'd approach it differently if I had to do it again:
<ul>
<li>I'd insist on selling to small companies first. Instead of investing time trying to sell to large companies, I'd invest time in rethinking the product so that it's useful for small companies. That's a piece of advice we got from PG himself, and it was spot on.</li>
<li>I'd accept having to make larger bets and act accordingly. Given the slow feedback loop, I'd be more inclined to think long-term and make larger bets, like building more features upfront and building a community around the product, even if they're not paying customers, and probably won't be for a while (or ever).</li>
<li>I'd cut scope as much as possible and avoid being seen as a replacement for existing solutions. I think this point is valid for all startups, but it's especially true for infrastructure products because switching infrastructure usually takes longer than anything else. Therefore, I'd think harder about whether I can build something small and less critical to go alongside existing solutions.</li>
</ul>
 
<h2>How these learnings helped us pick our next idea</h2>
The worst part of building an infrastructure product was the slow feedback loop, so our first criteria for picking our next idea was to find a problem that we could offer a packaged solution for, and that we could sell to small and medium companies first. That solution also had to be easy to use, and the time to value had to be as short as possible.
Considering these criteria, we then brainstormed a few hypotheses and ran approximately fifty customer interviews to validate them.
We then landed on what we're doing today: a product that helps companies generate unique insights by using code.
<img src="/posts/pivoting-away-from-infra/demo-short.gif" alt="">
The reason Briefer turned out to be a great fit for us is that:
<ul>
<li>Time-to-value is extremely short. Instead of spending days reading documentation and setting up the product, customers can just log in and start generating insights.</li>
<li>It appeals to both small and large companies. Small companies usually love Briefer because their engineers can write code to generate insights, and they don't have to wait for a data team to do it for them. Large companies love it because they can use it to build more advanced analyses, data apps, and predictive models.</li>
<li>We don't have to replace existing solutions. While some companies replace their BI tools with Briefer, most use it alongside them because traditional BI tools can't generate the insights Briefer can, nor can they be used to build data apps or predictive models.</li>
</ul>
In the end, we're glad we went through the infrastructure journey, even though it was tough. It taught us a lot about building products, and it helped us land on a better idea.
I'm extremely thankful for the YC partners, our batchmates, and the YC alumni who helped us along the way. Many of these learnings wouldn't have been possible without their help.
Finally, if you want to try Briefer, or even if you're not sure it's the right choice, please feel free to <a href="https://briefer.cloud">visit our website</a> and <a href="https://calendly.com/lucasfcosta/briefer-intro-call">book an intro call with us</a>. We'd love to hear from you.

Here's the story of how we pivoted after our YC batch, the lessons we learned building in the infrastructure space, how I'd approach it differently if I had to do it again, and how these learnings helped us build Briefer.

What we learned from a failed infrastructure DevTool

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.
<img src="/posts/p-values/honeyberry-method.png" alt="">
 
<h2>Step 1: Understanding p-values so you can misuse them</h2>
Your first step to riches is to create a technique that you can claim to improve any aspect of your customers' lives and that you can write books about.
It doesn't matter if your technique actually works. What matters is that you can convince people that it does, and that's what I'm here to teach you.
Let's say you've created a new study technique and you believe it helps students score better on exams, for example.
To test your hypothesis, you set up an experiment with two groups of students:
<ul>
<li>Control Group: This group studies using traditional methods.</li>
<li>Experimental Group: This group uses your revolutionary study technique.</li>
</ul>
Then, you give both groups the same exam and compare their scores. If the experimental group scores significantly higher, some people might conclude that your study technique is effective.
<img src="/posts/p-values/two-groups.png" alt="">
Now, that's where amateurs usually stop. But you're not an amateur, you're a business person. You need to convince even the nerdiest of your customers that your technique is backed by science.
These people like to think they are smart, so they'll definitely ask how do you know whether the difference in scores is real or just due to chance.
That's where p-values come in. The p-value is a statistical measure that determines how likely it is that the improvements in scores were due to random chance. In other words, p-values represent the chance of the same or more extreme scores happening even if your new study technique doesn't actually work.
A small p-value means that a difference in scores as extreme as the one you observed is unlikely to have occurred by mere luck. Thus, a small p-value suggests that your study technique might be effective.
For example, a p-value of 0.001 means there's only a 0.1% chance that the score improvement was just random. This suggests that the better scores could be due to your new study technique, not just chance.
On the other hand, a large p-value means the difference in scores could easily have happened by chance, and your study technique might not be effective.
For example, a p-value of 0.15 means there's a 15% chance that the score improvement was random. Having such extreme results 15% of times would lead nerds to say that your results are not "statistically significant", because these events are actually not that rare. In other words, they mean that the difference in scores could easily have happened by chance, and your study technique might not be effective.
<blockquote>
The way that classical statistics works is by comparing study data to what is expected when there is nothing. If the data are not typical of what is seen when there is nothing, there must be something!
— Dallal, Gerard. The Little Handbook of Statistical Practice.
</blockquote>
Having learned what p-values are, you probably figured that what we're after is a small p-value. So let's go ahead and make sure we get one.
 
<h2>Step 2: Ensuring that <code>p</code> is small enough</h2>
Let's be honest: the chances of your revolutionary study technique actually working are slim. Consequently, your experiment will probably fail to demonstrate that the new study technique is much different from just doing whatever everyone is already doing.
Once your study fails, you have two options. The first is to repeat the experiment until it works. The second is to analyze the data in multiple ways until you find another measure that looks significant. Then you can rebrand your technique as being effective for that measure, like "improving memory retention" or "increasing focus".
<ul>
<li>Pro-tip 1: if none of the measures look significant, you can always try to find a subgroup of the population that benefits from your technique. For example, "our revolutionary study technique is particularly effective for left-handed students born in the summer".</li>
<li>Pro-tip 2: you can also find transformations of the data that make it look significant, like the sum of outcomes squared or the log of the difference between scores.</li>
</ul>
<img src="/posts/p-values/python-honeyberries.png" alt="">
This step is pretty easy to memorize because both alternatives rely on the fact that if you look around enough, you'll eventually find something that looks like a significant result. Another way to put it is that "the chances of winning the lottery are small, yet often there's a winner".
 
<h3>A crucial detail: make sure p is smaller than 0.05</h3>
There's a convention among scientists that says that a p-value below 0.05 is said to be "statistically significant". So that's what you should aim for, at least most of the time.
There are many theories about why statistical significance is set at 0.05, but, as Gerard Dallal points in his book "The Little Handbook of Statistical Practice", all of them trace back to Karl Pearson's 1914 book "Tables for Statisticians and Biometricians".
<img src="/posts/p-values/karl-pearson-book.jpeg" alt="">
Pearson figured out how likely different outcomes were in statistics. Then Fisher made it easier by creating simple tables in his book "Statistical Methods for Research Workers" in 1925. These tables just showed basic probabilities for certain outcomes. Fisher's idea caught on, and he did the same thing with Frank Yates in 1938. People still use Fisher's tables in statistics texts today.
Fisher's tables were shorter and simpler than Pearson's. Instead of giving lots of details, they just gave rough probabilities. Fisher also came up with a neat way to show how sure we are about results—using stars. One star meant a result observed about 5% of the time, two stars meant it was observed about 1% of the time, and sometimes three stars meant it was observed about 0.1% of the time.
Fisher also said that when these extreme results would only happen 5% of the time without an intervention, we should take notice. He thought this level was good because it helped us find real results without getting too many false alarms. And this 5% rule has stuck around, shaping how we understand data and what we consider important in science.
In one sense, 5% is just a number, and it's not a magic number. But it's a good rule of thumb. Still, some people think it's too high, and that we should use lower levels like 1% or 0.1%.
To make matters simpler, I'd also recommend that you keep this excellent table from the scientific website <code>xkcd</code> handy:
<a href="https://xkcd.com/1478"><img src="/posts/p-values/xkcd-1478.png" alt=""></a>
 
<h2>Step 3: Publishing your results</h2>
Now that you have a small p-value, you need to call journalists and tell them about your revolutionary study technique. They'll be thrilled to write about it, especially if you can provide them with a scientific paper that includes the words "p-value" and "statistically significant".
The important thing here is not to mention that you've run your experiment 100 times until you got a significant result, nor that you've written a python script to test all possible variables until you found one that looks significant.
<img src="/posts/p-values/honeyberry-news.png" alt="">
Another important thing is to explain p-values to them in a way that doesn't sound egregious, but that is not entirely right either.
One incorrect way to explain p-values is to say that they represent the probability that the "null hypothesis" (your study technique not working) is true.
Let's say you've found a p-value of 0.001, for example. In that case, make sure to tell them that there's only a 0.1% chance that your study technique is ineffective. Although that's incorrect, it's much more convincing than saying "if the new study technique didn't work, we would still see the results we did approximately 0.1% of the time".
Another technique that you can use is to invert the logic. Instead of saying that a p-value of 0.001 means there's a 0.1% chance that the score improvement was random, they say that there's a 99.9% chance that the new study technique is effective. That's not correct either, but it's looks better in a headline.
 
<h2>Step 4: Rebuking the skeptics</h2>
Once your revolutionary study technique is published, you'll probably face some skepticism. That's normal, and you should be prepared to deal with it.
The scientists are the most dangerous skeptics. They'll probably try to replicate your experiment and fail to find the same results.
If it's only one or two scientists, you can always say that "failing to find an effect is different from showing that there is no effect". They won't be able to argue with that because you're technically correct.
Now, if quite a few scientists fail to replicate your results, then you're in deep trouble. <a href="https://en.wikipedia.org/wiki/Replication_crisis">That's why it's best to design experiments that are hard to replicate, like those that require a lot of data or that are too vague to be disproven</a>.
In general, keep in mind that the best way to deal with skepticism is to prevent it from happening in the first place.
 
<h2>Putting it all together</h2>
<ul>
<li>P-values are a measure of how likely it is that you'd observe the results you did if your intervention didn't actually work.</li>
<li>The smaller your p-value, the less likely you are to have observed the results you did by mere luck.</li>
<li>Conversely, the larger your p-value, the more likely it is that you'd observe the results you did by mere luck.</li>
<li>A p-value is not the probability that the null hypothesis is true. The opposite is also true: a p-value is not the probability that your intervention is effective.</li>
<li>Just because you haven't observed an effect, it doesn't mean there isn't one.</li>
<li>If you run the same experiment many times, you'll eventually see a p-value below 0.05, even if the null hypothesis is true.</li>
<li>Looking for an effect until you find one is an effective way to make sure you get a small p-value, but it's bad practice unless you're looking for an easy way to sell books.</li>
<li>A p-value below 0.05 is usually considered "statistically significant". Although it's not a magic number, it's a rule of thumb that has been around for a while.</li>
</ul>
 
<h2>Before we go</h2>
In case you didn't notice, this post is a satire, and any parallels to the real world are, obviously, coincidence (no pun intended).
The post was inspired by the excellent work of <a href="https://xkcd.com">Randall Munroe</a>, in particular his comic <a href="https://xkcd.com/882">xkcd 882</a>:
<a href="https://xkcd.com/882"><img src="/posts/p-values/xkcd-882.png" alt=""></a>
Finally, if you're a data-scientist who loves jupyter notebooks or data apps, but finds them hard to share your work with others or too much hassle to deploy, you should check out <a href="https://briefer.cloud">Briefer</a>.

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.

What p-values are, what they are not, and how to sell books with bad statistics

This changelog is the most important one we've shipped so far - that's why it took more than two weeks for us to post it.
In these last few weeks, we've completely rebuilt our editor so it's more understandable, more reliable, and more flexible.
We've also introduced environment variables, which allow you to store information like settings, API keys, and database credentials securely. Then, you can reference these variables in your code without exposing them.
 
<h2>Our new editor and a ton of related improvements</h2>
Before this change, our editor was more unstable than we'd like. Sometimes the editor would show different states than what was happening in the back-end. Refreshing the page also caused the editor to lose part of its state, like which blocks were running, or undergoing AI suggestions. All these issues are now gone, and we're glad to have received lots of positive feedback about the new editor.
The new editor we're rolling out for everyone today is more reliable, more flexible, and more understandable. We now manage and reconcile every single piece of state in the back-end. Consequently, the editor's state will be consistent across refreshes, and for all users and devices.
<img src="/posts/changelog-4/v2editor.gif" alt="">
This change also comes with a bunch of very important improvements. Here are some of them:
<ul>
<li>When using data apps, each viewer will have their own state. That means that if you're viewing a document and someone else is viewing it at the same time, you won't see each other's inputs and results.</li>
<li>The "Run All" button now shows the number of blocks that have already run, and how many are left.</li>
<li>We explicitly display enqueueing and running states for each block.</li>
<li>When a scheduled run is happening as you view a document, you'll see its results in real time.</li>
<li>You can now duplicate whole blocks or individual tabs by clicking the drag-handle menu on the left side of the block.</li>
<li>You can run an entire block from left to right by clicking the "Run all tabs" button that appears within the drag-handle menu for multi-tab blocks.</li>
<li>Data apps will only be updated when you click the "Update Data App" button. This way, you can make multiple changes to the inputs and dropdown fields of a data app and only publish them when you're ready.</li>
<li>Adding new blocks is now more intuitive. You can now add blocks by clicking the "+" button that appears when you hover over gaps between blocks.</li>
<li>You can now group all types of blocks, not only Python, SQL, and Visualizations.</li>
<li>If you create a new dataframe that only exists in a Python block, it will be immediately available within visualization blocks.</li>
<li>You don't have to uncollapse a block to run it anymore. You can now run a block by clicking the "Run" button that appears when you hover over the block.</li>
<li>We'll now persist the open/close state of the document tree on the left as you navigate between documents. This way, you won't have to keep opening nested folders every time you navigate to a different document.</li>
</ul>
 
<h2>Environment variables and how to use them</h2>
Environment variables are a way to store information like API keys, database credentials, and other settings securely. That way, you can then reference these variables in your code without exposing them to viewers.
These environment variables are useful when you want your Python blocks to perform authenticated API requests, like when doing writebacks to your database, for example.
To create an environment variable, click the "Environment Variables" button on the bottom-left corner of the editor. Then, click the button at the bottom of the list to add a new variable. Once you're done, click the save button and restart your environment for the new environment variables to be available.
You can then reference this variable in your Python code by using the <code>os.getenv</code> function.
<img src="/posts/changelog-4/v2vars.gif" alt="">
By the way, we also added a button on the bottom right of the editor so you can restart your environment manually if you need that.
 
<h2>What's next?</h2>
We know it's been somewhat annoying to lose your in-memory state when your environment restarts. We're working on a way to automatically re-run dependencies or persist this state across restarts, so you won't lose your dataframes, variables, and other in-memory state when your environment restarts.
Expect that by the next changelog.

This changelog is the most important one we've shipped so far - it includes a brand new editor and support for environment variables.

Changelog #4: Environment variables, and a brand new editor

Besides the usual bug fixes, we're shipping a major new feature this week: data apps.
You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.
<img src="/posts/changelog-3/data_apps.gif" alt="">
Please notice data apps are only available for users in the professional tier.
 
<h1>When to use data apps features</h1>
You should use data apps features whenever you want viewers to be able to provide input to your reports, models, or analyses.
In a nutshell, data apps are interactive documents that allow viewers provide input to the code you've written.
Typical examples include:
<ul>
<li>Comparing scenarios: Allow viewers to change assumptions and see how the results change.</li>
<li>Exploring data: Allow viewers to change the data they're looking at.</li>
<li>Filtering data: Allow viewers to filter data by date, region, or any other dimension.</li>
</ul>
 
<h1>How data apps work</h1>
To create a data app, you must first add an input or select field to your document by using the <code>/input</code> or <code>/select</code> commands.
For each of those blocks, you can specify a label. The variable name is the one you see on the top right corner of the block in a green box.
Then, you can reference the variable name in your Python code in the same way you would reference a variable you created yourself.
To interpolate the value of the input or select field in SQL blocks, use the <code>{{variable_name}}</code> syntax.
<blockquote>
Please notice that you can now interpolate any variables in your SQL blocks, not only the ones created by input or select fields.
</blockquote>
Finally, use the "Make it a data app" toggle on the top right corner of your document, under the share menu, to enable the data app mode.
Toggling the data app mode allows viewers to provide input to your document, but they still can't edit the document's content.
 
<h2>Other fixes and improvements</h2>
<ul>
<li>We fixed a bug where users would not be automatically logged out once their session expired. Now, users will be imediatelly logged out and redirected to the login page when their session expires.</li>
<li>Document snapshots: we now keep snapshots for users in the professional tier. These snapshots are taken every five minutes and will be available for the last 5 days. The last 10 snapshots are always available, regardless of how old they are.</li>
<li>We've done updates to the visualizations so they support data types they used to not support.</li>
</ul>

This week, we're introducing data apps. You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.

Changelog #3: Introducing data apps

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements.
If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.
 
<h1>General Updates</h1>
Here are a few small improvements we've done to the editor, and to the overall performance of Briefer.
<h2>Editor performance improvements</h2>
We noticed that performance degraded as documents got larger. We've made a few changes to the way we handle large documents.
First, we've eliminated lots of unnecessary re-renders for graphs and tables. This should make the editor feel snappier, especially for large documents.
<h2>Support for larger datasets</h2>
Some users dealing with millions of rows of data were experiencing performance issues.
To fix that, we now run all the queries in your own compute instance and stream the results back to the editor. This change means that we can now support much larger datasets without performance issues. It also means that the users who pay for larger compute will experience more benefits from it.
<h2>Improvements</h2>
Besides the performance improvements we shipped a few minor UI updates. Besides all the small UI tweaks we've done, here are a few highlights:
<h3>In/Not-in filters</h3>
We've added support for <code>in</code> and <code>not-in</code> filters in visualization blocks. This is useful when you want to filter by multiple values at once.
<img src="/posts/changelog-2/in_not_in_filters.png" alt="">
<h3>New query on visualizations</h3>
You can now create queries more easily when using visualizations. Just click on the "New query" option the dropdown, or in the tooltip that appears when you create an empty visualization.
<img src="/posts/changelog-2/new_query_option.png" alt="">
You can also create new queries or tabs by using the <code>+</code> button in the top right corner of the block.
<img src="/posts/changelog-2/plus-tab-button.png" alt="">
<h3>Deleting tabs or entire blocks</h3>
You can now delete tabs or entire blocks by clicking on the drag-handle and selecting the desired delete option.
<img src="/posts/changelog-2/delete_tab_or_block.png" alt="">
<h3>New users are viwers by default</h3>
Previously, when allowing anyone from <code>@yourdomain.com</code> to access Briefer, new users were added as editors by default. We've changed this so that new users are added as viewers by default. This is to prevent new users from accidentally making changes to the workspace.
 
<h1>Bug fixes</h1>
Bug fixes are the biggest part of this release. We've fixed a ton of bugs, but here are a few highlights:
<ul>
<li>Fixed a bug where there was a race condition when creating tokens for compute instances. This was causing some users to experience multiple "something went wrong" alerts that would only go away after the environment got restarted.</li>
<li>When changing the filters or settings for visualizations multiple times, we'd enqueue each update. This caused visualizations to be slow to update. We've fixed this so that only the latest update is enqueued and previous updates are discarded.</li>
<li>Our AI suggestions were sometimes suggesting overwriting dataframes using <code>[Object object]</code> dataframes. We've not only fixed this bug, but we also improved the LLM's context.</li>
<li>Creating new tabs was causing an empty block to appear in the editor until you switched to another tab and back. This was a visual bug that didn't affect the actual document, but it was annoying - and now it's fixed.</li>
<li>Dragging and blocks into one another was not working properly. Sometimes, the editor would try to drag the wrong block, or it would not detect you wanted to group different blocks. This bug is now fixed.</li>
<li>Editing AI suggestions and then accepting them was causing the edited diff not to be applied. Instead, it applied the raw AI suggestion. We fixed this so that any changes you do in the diff editor will now persist in case the suggestion is accepted.</li>
<li>Display of titles was showing incorrect background colors during loading and disabled states. They were also not taking the appropriate space. We've done fixes to their appearance so they behave correctly in all cases.</li>
<li>Visualizations were not being updated in scheduled runs. This was a bug that was causing some users to receive outdated visualizations in their scheduled reports. This is now fixed.</li>
</ul>

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements. Here are some of the highlights.

Changelog #2: Blazing fast performance and tons of bug fixes

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently.
This is a non-exhaustive list of everything that's changed. As we publish more updates, we'll increase the level of detail here.
If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.
 
<h1>New features</h1>
This week: no-code visualizations, tabs (grouping blocks), naming blocks, workspace settings, and exports.
 
<h2>1. No-code visualizations 📊</h2>
The most important change comes first: you can now create graphs without code. For that, type <code>/visualization</code> and add a visualization block.
<img src="/posts/changelog-1/visualizations.gif" alt="">
There, you'll be able to select the query to plot, types of graphs, filters, and more.
Once you're done with your visualization you can click the double-chevron icon to toggle between edit and visualization modes. In visualization mode, only the graph and filters appear.
Please notice that histograms and pie charts are not available yet.
 
<h2>2. Grouping blocks 🫂</h2>
You can now group multiple blocks into one to create multi-tabbed blocks.
Grouping blocks is useful when you need multiple python blocks to clean data, but only want a single python block with the results to show.
<img src="/posts/changelog-1/grouping.gif" alt="">
You can also drag-and-drop tabs to change their order. To ungroup tabs, drag the desired tab out of the tabbed block.
 
<h2>3. Naming blocks 🪄</h2>
Now that you can group blocks, it's important that you name them so that you know what's in each tab.
Naming blocks is also helpful for presenting reports - it helps stakeholders understand what they're seeing.
<img src="/posts/changelog-1/naming.gif" alt="">
To name blocks, just click the title of each block and input whatever title you want.
 
<h2>4. Workspace settings ⚙️</h2>
We heard a few people wanted to change their workspace names or settings after the workspace had been created. That wasn't possible before.
Now, you can rename your workspace and enable everyone with the same email address to join.
<img src="/posts/changelog-1/workspace-settings.gif" alt="">
 
<h2>5. PDF exports 📑</h2>
We know many of you were asking for this, so we have shipped it.
You can export PDF files by opening the share menu on the top right corner and then clicking the PDF export option.
For now, PDF exports are only enabled for users in the professional tier.
<img src="/posts/changelog-1/pdf-exports.gif" alt="">

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently. This week: no-code visualizations, tabs, and exports.

blog

What p-values are, what they are not, and how to sell books with bad statistics

Recent posts

Log scales

Changelog #5: Snapshots and a Slack integration

What we learned from a failed infrastructure DevTool

What p-values are, what they are not, and how to sell books with bad statistics

Changelog #4: Environment variables, and a brand new editor

Changelog #3: Introducing data apps

Changelog #2: Blazing fast performance and tons of bug fixes

Changelog #1: No-code visualizations, tabs, and exports