A/B testing is everywhere, but good luck explaining p-values to your CFO. When the question is “Which feature wins?”, most stakeholders don’t care about confidence intervals, they want a clear answer.

I’ve always found Bayesian A/B testing more intuitive and far better for communicating results. In this post, we’ll break down why.

Say we want to decide if a green button will perform better than a blue one, so we go ahead and run an A/B test. In the end, people usually want to know just one thing: “Is A better than B?”

![](/posts/ab-testing/pipelineab.png)

After 500 visitors see each button, the numbers are in:

* Green: 27 sign-ups — that’s 5.4%.
* Blue: 23 sign-ups — 4.6%

![](/posts/ab-testing/buttons.png)

This is where many stop. They see the numbers and declare a winner: green beats blue. But we know better than to take numbers at face value.

Think about it: if you flip a coin 10 times and get 7 heads, would you bet your savings that it’s rigged? Probably not. With such a small sample, randomness can easily skew the result.

![](/posts/ab-testing/coin.png)

The same logic applies here. Green got 5.4%, blue got 4.6% so it *looks* like a win, but it’s just a snapshot. Rerun the test, and those numbers could flip. Like the coin toss, there’s an underlying truth we just can’t see yet.

That’s what makes A/B testing hard: It’s about the uncertainty behind the numbers we see. How do we measure it? How do we quantify the risk of acting on too little data?



# Measuring uncertainty the usual way

Using the textbook method to measure uncertainty, we need to know how likely it is that the difference we saw (5.4% for Green and 4.6% for Blue) is actually real and not just a fluke.

Think of it this way: if we ran this exact experiment 1,000 times, how often would Green *actually* outperform Blue? Would it win 900 times? 700? Or just 400? If Green wins less than half the time, then it’s not really a winner, it just got lucky the first time we ran the test.

This is where frequentist statistics hands us a tool: **the p-value**.

To understand it, we start by assuming the *opposite* of what we’re trying to prove: that there’s no real difference between Green and Blue. If they’re equally good, then any difference we observe must be due to random chance.

Now imagine plotting all the differences you’d see across thousands of these random experiments. You’d get a bell-shaped curve centered at zero, because most of the time, chance alone won’t budge the results very far.

That’s the graph you see below. The middle of the curve represents small, harmless fluctuations: the noise we expect when nothing’s really going on. The tails are where things get weird: the rare, extreme results that pop up once in a while.

![](/posts/ab-testing/pvalue.png)

A p-value doesn’t tell you the probability that Green is better than Blue. It tells us:

> If Green and Blue were actually equally as good, how likely is it that we’d see a difference as big as 0.8% (or more) just by random chance?

A low p-value, like 0.01, means that if the buttons were actually equally good, seeing a result like 5.4% for Green and 4.6% for Blue would be rare. It might happen just 1 in 100 times, purely due to chance. It’s like saying: *“If there’s no real difference, this result shouldn’t be showing up... and yet it did.”*

That’s why a low p-value is treated as a signal that something might be going on. But if the p-value is high (say, 0.5) that means a result like 5.4% vs. 4.6% would be totally normal even if the buttons were identical. There’s nothing unusual to explain, therefore strong reason to believe one button is actually better.

![](/posts/ab-testing/pvalues2.png)

A p-value doesn’t tell you how likely Green is actually better, only how much your result challenges the assumption that there’s no real difference.

In frequentist statistics, the hypothesis is fixed and the data is random. The guiding idea is: “There’s a true answer out there; we just observed a noisy sample of it.”

You run an A/B test and get 27 sign-ups for Green and 23 for Blue. A frequentist won’t say “there’s a 70% chance Green is better.” Instead, they ask: “If Green and Blue performed exactly the same, how likely would it be to see a result this extreme just by chance?” Green either is better or isn’t. There’s no room for uncertainty about the hypothesis itself.

That makes communicating results tricky. Try explaining p-values and null hypotheses to a CFO who just wants to know if we should switch to the green button. Frequentist methods are statistically sound, but they don't speak the language of probability most people use.

Bayesian thinking flips that around.

Bayesians treat the data as fixed: 27 vs. 23, no arguing with that. What’s uncertain is the hypothesis: Is Green truly better than Blue? The Bayesian question becomes much more intuitive:

“Given what we observed, how likely is it that Green is better?”

That’s an answer you can take to a decision-maker.



# Measuring uncertainty the Bayesian way

The Bayesian approach starts with a different question:

> Given what I’ve seen, what’s the full range of possible conversion rates and how likely is each one?

Instead of giving you a binary verdict or a vague range, it shows you a distribution of beliefs: a spectrum of plausible values, shaped by your actual data. This more aligned with how we naturally think: not in black-and-white outcomes, but in shades of confidence.

To model uncertainty the Bayesian way, we build a probability distribution, that is, a curve that represents what we **believe** the true conversion rate could be, based on the data we have.

That’s where Bayes’ Rule comes in. It gives us the mathematical machinery to **update** our beliefs based on new evidence. Here's what it looks like:

![](/posts/ab-testing/formula1.png)
In words:



![](/posts/ab-testing/formula2.png)

The **prior** is what we believed about the conversion rate before we saw any data. Maybe from past experiments or gut feel, we expect sign-up rates to usually fall somewhere between 3% and 7%. That’s our starting guess.

Then comes the **likelihood**, which asks: Given what we just saw, how well does each possible conversion rate explain it? If Green got 27 sign-ups out of 500 users, then rates like 5% or 6% make a lot of sense. A rate of 1% or 15%? Not really. The likelihood rewards values that make the data unsurprising.

Now we combine those two (the prior and the likelihood) to get the posterior. This is our new belief, updated with the evidence. It’s not a single number, but a full distribution that tells us how plausible each conversion rate is after seeing the results. This is what we actually care about.

Finally, there’s the evidence, technically called the marginal likelihood. It’s just the math that makes sure everything adds up to 100%. You can mostly ignore it unless you’re comparing models.



![](/posts/ab-testing/formula3.png)

This tells us: What we now believe = What we believed × What the data tells us

To figure out *what we now believe* about a button’s true conversion rate, we first need to model what we believed before the test. This means we need a mathematical tool that can express that belief.

What do we already know about our site’s conversion rates? Are they usually around 5%? Closer to 10%? How likely is it that a new feature converts higher than 6%?

We capture that belief with a **prior** distribution: a curve that assigns different degrees of plausibility to different conversion rates. It doesn’t predict what will happen. It simply quantifies what we consider likely before we collect new evidence.

We know each visitor either converts or doesn’t (success or failure, yes or no). When we run an experiment with a fixed number of visitors and record how many convert, we’re counting the number of **successes** across a series of independent trials. That’s precisely the kind of process the **binomial distribution** is meant to describe: it tells us the probability of observing a certain number of conversions, given a fixed number of trials and a known conversion rate.

Tossing a coin is the textbook example of a binomial process. We have two possible outcomes: heads or tails. If the coin is fair, the probability of tails is 0.5.

Flip it 100 times, count the number of tails, and the binomial distribution tells us what kind of results to expect. It looks like this:

![](/posts/ab-testing/binomial.png)

The highest probabilities cluster around 50 tails, with the odds dropping off as you move toward the extremes. That’s what the binomial distribution gives you: **what to expect when you already know the odds.**

But in A/B testing, we don’t know the odds, we’re trying to infer it from the data.

To do that, we need a different kind of curve, one that represents our **uncertainty** about the conversion rate itself. Something that shows all the possible values the rate could take, and how plausible each one is based on the data we have.

That’s exactly what the **Beta distribution** gives us.

It’s defined between 0 and 1, making it perfect for modeling probabilities. But more importantly, it’s flexible. It can represent total uncertainty (where every rate is equally likely) or strong convictions, (where most of the probability mass is concentrated around a narrow range).

In Bayesian A/B testing, we use the Beta distribution to track our evolving belief about the conversion rate. Before the experiment, it reflects our prior assumptions. After we collect data, we update it to reflect the evidence.

Depending on what we know (or don’t), the shape of that belief curve can look very different. Here are a few examples:

![](/posts/ab-testing/betafunction.png)

The beta works naturally with the binomial distribution. When we update a **Beta prior** with binomial data, we get a **Beta posterior**. This property, called conjugacy, makes the updating process mathematically seamless. Our beliefs can change but the form of our model stays the same.

Say your experience as a product manager tells you that most new features tend to "convert" at around 10% of users. Rather than discarding that knowledge, Bayesian inference lets you incorporate it directly into your model by shaping your prior distribution to reflect it.

To incorporate that knowledge into our model, we use the Beta, which is controlled by two parameters: α and β. These parameters shape the curve and determine how confident we are in different conversion rates. The mean of a Beta distribution is:



![](/posts/ab-testing/alpha.png)

So if we believe the true rate is likely around 10%, we can choose values of α and β that center the distribution at 0.10. For example, Beta(2,18) has a mean of 2/(2+18) = 0.10.

This is a way of saying: "Before seeing any data, I believe a 10% conversion rate is most likely, and I'm about as confident in that belief as if I had already seen 20 users, 2 of whom converted."

The notation Beta(α,β) simply tells us how that belief is distributed across the 0-1 range. Higher values of α and β mean stronger prior confidence; smaller values mean more uncertainty.

What the data tells us is that 27 out of 500 visitors convert for the Green button. To update our prior belief, represented by Beta(2,18), we simply incorporate the new evidence.

We do that adding 27 to α (for the conversions) and 473 to β (for the non-conversions). This gives us an updated posterior distribution:

Posterior = Beta(2+27, 18+473) = Beta(29,491)

Here's what that the graph looks like :



![](/posts/ab-testing/posterior.png)

This posterior distribution combines what we believed before the test with the new evidence from the data. It now represents our current belief about the true conversion rate of the Green button.

We model the Blue button the same way.

Starting with the same prior belief, that Beta(2, 18), centered at a 10% conversion rate, we now update it based on the observed data: 23 conversions out of 500 visitors.

Adding 23 to α for the conversions and 477 to β for the non-conversions gives us the updated posterior for the Blue button:

Posterior = Beta(2+23, 18+477) = Beta(25,495)

Graphically:



![](/posts/ab-testing/posterior2.png)

This new distribution reflects what we now believe about the Blue button’s true conversion rate, combining both our prior knowledge and the test data.

At this point, we’ve done something more powerful than produce two conversion rates. We’ve built two full probability distributions—one for each button. Each curve shows not just a single estimate, but a range of possible conversion rates, weighted by how plausible each one is, given both our prior belief and the observed data.

So instead of asking, “Is 5.4% greater than 4.6%?”, we’re asking a richer question: What do we actually believe about the true rate behind each button?

This isn’t something we can answer by comparing averages. Each distribution captures uncertainty, and those uncertainties overlap. What we really want to know is: **if we drew one plausible value from each distribution, how often would Green outperform Blue?**

There’s no simple formula for that, but the logic is straightforward. We can simulate the scenario: draw 10,000 samples from each distribution, compare them one by one, and count the fraction of times Green wins:



![](/posts/ab-testing/equation.png)

In our case:



![](/posts/ab-testing/equation2.png)
Graphically:

![](/posts/ab-testing/simulation.png)

This gives you something far more useful than a p-value: Based on the data and your prior, there’s a 71% chance that Green truly outperforms Blue.

Now you’re not just picking a winner: you’re quantifying confidence, weighing upside, and sizing up the risk.

Which is the actual job.

Why Bayesian methods can be more interesting and intuitive than frequentist ones when it comes to A/B testing.

Why Use Bayesian Methods for A/B Testing

It's not easy to find equilibrium between complex and simple when training a Machine Learning model.
Darts-based explanations about Bias-Variance tradeoff never convinced me, so I decided to break it down: what’s really behind this dilemma?

![](/posts/bias/darts.png)

Imagine everyday lots of dogs and cats pass through your window. Some of them are big, some are small ones.
You decide to classify them by height and weight, so you take measures for a bunch of animals. Cats are usually smaller and lighter. Dogs are generally bigger or heavier.

![](/posts/bias/scatter.png)

To classify these animals we can use Machine Learning. These models generalize from seeing some examples. Generalizing means that when presented a new animal, the model will be able to classify between cats and dogs. The same way a child see cats and learn what a cat is, computers see data and do the same. But choosing an model is not an easy task. We can use one line, lots of lines or even curves to separate dogs and cats.

![](/posts/bias/training.png)

The traced curves are trained on the dataset to classify every new point *below* it as a cat, and every point *above* it as a dog. The line does not separate all dogs and cats correctly while the squigly curve separate all dogs and cats correctly.
To compute the performance of an model, we count it's errors on the measures recorded. A well trained model should be able to classify animals it has never seen before correctly.\
When unseen data is presented, the traced curves below perform differently. The line has some errors, while the curvy line performs poorly.

![](/posts/bias/testing.png)

This task in Machine Learning is called classification, where the learned function outputs categories.
The model is just a function which takes examples of the input space - dogs and cats information - and return labels `cat` and `dog` whether it is a cat or a dog respectively.

There exists lots of functions that receive an animal and return a {`cat`, `dog`} label. They can look like lines, curves or horrors behind our compreehension. We are searching for a classifier that can correctly distinguish between cats and dogs. Below I drew the space of all functions that take an animal and return a class. Each green point in this space is a function.

![](/posts/bias/all.png)

To measure how good each function is, we input information of *all* animals, compare with the labels and count how many times the function mislabeled the pet. This measure is called the function risk. The bigger the risk, the worse the function is, so we are searching for a small risk function.

That is a problem. To compute the true risk we need all dogs and cats information. All pets that lived, those which are alive right now and the ones which are still not born. Poetic - but unrealistic. An ideal model would need data from every pet, past, present and future, in order to classify with precision whether a pet is a dog or a cat.
But we cannot have such big dataset. Instead, we use a smaller one - some dogs and cats from our neighboorhood we measured.

![](/posts/bias/classification.png)

The set of all pets hides a probability function. We cannot see in more three dimensions, so to exemplify, consider only the height and class of the animals `cat` and `dog`. A point in the three dimensional graph has height, class and probability. It says which is the probability of finding a pet with this class (`cat` or `dog`) and this height.

![](/posts/bias/prob.png)

By selecting a height value in this distribution, we obtain another distribution which tells us exactly whether an animal this tall has more chance of being a cat or a dog.

![](/posts/bias/marginal.png)

Unfortunately, we don't have acess to this distribution. If we had, we could skip the learning part.
And since we don't have all images nor the data distribution, we cannot compute the real risk. Our only option is to use our dataset, a set of samples from the real distribution, to compute an approximation of the true risk named *empirical risk*.

![](/posts/bias/empirical_risk.png) 

Many functions exist - each with its own risk. Of the many functions that exist, one has the smallest possible risk. This magical function is sometimes called Oracle.

![](/posts/bias/oracle.png)

The Oracle wins because it cheats by consulting the probability distribution to classify an image. If the image has greater chance of being classified as dog, the Oracle classify as a dog. Otherwise, as a cat. No wonder why it is so good.

![](/posts/bias/oracle_working.png)

Our task as engineers and scientists is to get closer to the Oracle function as possible. Our best try is a function that lives happily in the space, trained on whatever dataset we found.

![](/posts/bias/estimated.png)

The space of functions that take pets and classify as cats or dogs is very diverse. Many functions - linear, quadratic, step functions, discontinuous and other ones - live there. All of them classify animals as cats and dogs.

![](/posts/bias/functions.png)

Nested inside this vast space are subspaces - and each subspace contains a smaller portion of all functions.
Some of these subspaces are bigger than the others. Take the quadratic space, a space where the quadratic functions live.
Linear functions are a subset of quadratics where the quadratic term is zero.
Thus, the quadratic functions space also contains the linear functions space.

![](/posts/bias/linear_quadratic.png)

In Machine Learning, when choosing between Random Forests, Logistic Regressors or Neural Networks to classify a dataset, we also choose a subspace.
If we choose a linear model, like Logistic Regression, we will use a function that lies on the linear functions subspace. Choosing a class of functions means instead of looking through the whole space of functions that take animals as inputs and classify them, we will search over the subspace.

The challenge is choosing which subspace to search.

![](/posts/bias/subspace1.png)

We have no clue whether the Oracle function lives or not in the subspace.
The only confidence is that one function in this subspace has the smallest risk.
During training, when the model learns to classify labels based on images from the dataset, we aim to find this lowest-risk function.
The best we can do is learn a function as close as possible to it using our dataset.

![](/posts/bias/subspace2.png)

We also need to measure how well functions in our subspace can approximate any function in the full space.
Thus, we shall also minimize the distance between the best function in the subspace and the Oracle.

![](/posts/bias/subspace3.png)

At this point we have a more precise view of the learning problem. There are functions that take animals images and return labels; we approximated each function true risk using the empirical risk. An ultimately good function - the Oracle - exists. And to learn we should choose a class of functions. We are asked, what class of functions shall we use?

In one form or another, this problem has long worried engineers and scientists. This is the widely known problem in statistics entitled bias-variance tradeoff.

If we choose to separate classes using a line, every relation we ever find would be linear. But this would not come from data. Instead, it will be a bias imposed by us.
It will have high risk, making some mistakes and will not correctly separate the animals.

![](/posts/bias/linear.png)

In our example, tracing a line to classify dogs and cats is choosing the reduced subspace of linear functions to search on.
A small space means the distance between the better function we can achieve and the Oracle will be very big.
Eventhough, it's easier to find a good estimate for the function inside the subspace since it's too small.

![](/posts/bias/bias.png)

Conversely, if we choose a very curvy line to separate each dataset, we would always be able to separate pets from the dataset correctly.
However, the model would fluctuate wildly, depending on how accurate our dataset reflects the real world.

![](/posts/bias/squigly.png)

In a large subspace, the best achievable function is close to the Oracle (which might even reside there).
Although, an overly large subspace makes the model sensitive to dataset changes.
We could find a function with big risk, small risk or medium risk - the model would suffer from large variance.

![](/posts/bias/variance.png)

In Machine Learning, it's common to explain this duality using the terms underfitting and overfitting. When the model suffers with high variance, we say it *overfitted*. Elseway, models with high bias suffer from *underfitting*.
The ideal model - the one with lowest risk - lies somewhere between those two worlds.

![](/posts/bias/dilemma.png)

To classify correctly the pets in our dataset, a quadratic function would be ideal. Not simple as a line, nor too complicated as the high degree polynomial.

![](/posts/bias/quadratic.png)

Thus, we find the perfectly equilibrated subspace to estimate our function. But the journey of a scientist extends beyond quadratics. The toolbox of a good one should include regularizers.

Darts-based explanations about Bias-Variance tradeoff never convinced me, so I decided to break it down: what’s really behind this dilemma?

Bias-Variance tradeoff: understanding model complexity

Recommendation systems don’t understand preferences, they just measure behavior.

Every click, scroll, and rating becomes a vector. Then it’s all math: compare those vectors, spot patterns, and guess what comes next.

What feels like intuition is really correlation. Strip away the complexity, and it’s just the machinery of judgment: quantifying behavior, comparing signals, and betting on what comes next.

This post is about those measurements: how we encode behavior into numbers, and the different ways we can decide when two things are "similar enough" to drive a recommendation.

 

# Finding Simularities in Data Through Distances

To see how this works in practice, let’s look at something familiar: restaurant ratings.

Imagine we’ve recorded how three different people scored a hippie café and the fast-food place next door. The ratings go from zero to five. Zero means they hated it, and five means they thought it was excellent.

![](/posts/recommendation-systems/table01.png)

Now, let’s say Peter enters the picture. We’ve got his ratings for the same two restaurants, and we’d like to use that to suggest one he hasn’t visited yet.

![](/posts/recommendation-systems/table02.png)

We can approach this by visualizing everyone's restaurant preferences, including Peter's, on a graph, where each person is represented by a point based on their ratings.

![](/posts/recommendation-systems/graph01.png)

The key is to find people whose tastes are similar to Peter’s. On the graph, that means looking for points near his. The closer two points are, the more alike their ratings.

One way to do this is by summing the vertical and horizontal steps it would take to get from one point to the other, as if we could only travel in straight lines, up or across.

![](/posts/recommendation-systems/graph02.png)

This method is called Manhattan distance (or [`cityblock()` in SciPy\`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cityblock.html)).

It is fast, which comes in handy when theres a lot of data to process. Sometimes fast is good enough.

Another option is to draw a straight line between the points. That’s Euclidean distance (SciPy’s [`euclidean()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.euclidean.html)).

![](/posts/recommendation-systems/graph03.png)

![](/posts/recommendation-systems/table03.png)

By identifying Alice as Peter's nearest neighbor, we've narrowed our search. Her high ratings now serve as the most relevant data for predicting what Peter might enjoy next.

Both Euclidean and Manhattan distances are, in fact, variations of a broader mathematical idea known as the **Minkowski distance**.

Rather than being a single formula, [Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) is a family of distance metrics that flexes based on a parameter,\_ p\_. This parameter determines how we combine the differences across each dimension to calculate the total distance.

Mathematically, it looks like this:

![](/posts/recommendation-systems/equation.png)

At `p = 1` (Manhattan distance), the metric treats all differences equally. Each coordinate's difference counts linearly, like walking city blocks: a few small steps in many directions matter as much as one big leap in one.

At `p = 2` (Euclidean distance), the metric begins to emphasize larger differences more. This is the familiar "straight-line" distance, where bigger gaps have an outsized influence.

As p increases beyond 2 the metric becomes even more sensitive to the largest differences in any coordinate. In the limit, as `p → ∞`, only the single largest difference matters. This is known as Chebyshev distance, where the metric effectively says, "I only care about the worst-case deviation."

By tuning p, we can tailor how sensitive our similarity measure is to outliers, clusters, or particular behaviors.

These algorithms deliver speed, certainly. But before trusting the output, we might ask whether everyone’s measuring with the same ruler or if some are holding theirs at a slant.

 

# The Pearsons Correlation Coefficient

Up to this point, we’ve treated ratings as if they’re uniform: a 4 from one person means the same as a 4 from another.

But one may only rate 4s and 5s. Another stays below 3. A third switches between 2 and 5, with no middle ground.

How do we compare ratings when the same number reflects different standards?

The Pearson correlation coefficient helps us cut through that subjectivity by comparing patterns, not scores. Consider the table of ratings below:

![](/posts/recommendation-systems/table04.png)

Alice and Bob both rated four restaurants. They agreed that Restaurant C was the best, followed by A, then B, and finally D.

The numbers they assigned differ. Alice sticks to the 4–5 range, Bob is more liberal with the lows, but their ranking is identical.

Plot Alice's ratings on one axis, Bob's on the other, and the points align neatly along an upward-sloping line. This is what **perfect agreement** looks like—not in absolute values, but in pattern.

![](/posts/recommendation-systems/graph04.png)

If Alice gave high marks to a place Bob hasn't visited, it's likely he'd feel the same.

Pearson correlation turns less obvious patterns into useful insights. It doesn't matter whether someone is generous or stingy with their scores.

It measures is how tightly two sets of numbers track each other's ups and downs: are they moving together linearly, even if not in the same register?

In this case, even though Bob’s scores dip lower than Alice's do, their preferences move in sync. High for Alice means high for Bob. Low means low. The Pearson coefficient for their ratings is near +1. That's a strong positive correlation (SciPy's [`pearsonr()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html) lets us compute it directly).

Now reverse Bob's ratings: what Alice loves, Bob hates, and vice versa. The plot now tilts downward, forming a straight line in the opposite direction. That's a Pearson coefficient of -1. That's perfect *disagreement*.

![](/posts/recommendation-systems/graph05.png)

Finally, imagine a scatterplot of ratings with no pattern at all. Alice's highs are Bob's lows, mids, or anything in between. There’s no consistent relationship. The Pearson correlation here hovers around 0, signaling no meaningful alignment.

![](/posts/recommendation-systems/graph06.png)

Pearson captures the strength of a single relationship. But good recommendations usually come from patterns across a crowd, not just one person.

That’s where k-nearest neighbors steps in. 

# K-nearest neighbor

Consider two users that might look identical on paper: both rate horror films highly and avoid animation and binge thrillers. Pearson and the distance metrics treat them as the same profile, so recommending one user’s favorite to the other seems reasonable.

But then one gives five stars to a romcom buried in the catalog and we recommend Notting Hill to the other. It doesn’t go well.

Anomalies like this are inevitable at the individual level. The group, however, tends to converge on something stable.

We can harness that stability by leaning on what we already know: use Euclidean or Manhattan distance to identify a group of K users who rate content in roughly similar ways. That gives us proximity, which is a behavioral neighborhood.

Next, we bring in Pearson correlation to evaluate how closely each user’s taste aligns with the target’s preferences.

Now we have both pieces: who’s nearby, and how much their opinion should count.

That's the core idea behind k-nearest neighbors.

Assume Alice and Bob are the two users most similar to our target Peter, to whom we're trying to recommend a new restaurant. Their Pearson correlations with Peter’s ratings for the hippie restaurant are:

![](/posts/recommendation-systems/table05.png)

Their inputs aren't equal. Alice's correlation with Peter is stronger, so her opinion is weighted at 60%, while Bob's accounts for 40%. This weighting reflects the degree of alignment.

We compute the recommendation by aggregating these two inputs proportionally.

![](/posts/recommendation-systems/piechart.png)

It’s a clever balance: multiple perspectives, weighted by similarity.

But there's no such thing as a free lunch: the more neighbors we include, the more time we spend calculating correlations. And the more neighbors we include, the more likely we are to include outliers or dilute the signal, which can lead to more generic recommendations.

 

# Knowing Your Data Matters

How you measure similarity shapes what you recommend. The better the fit to your data, the better the recommendations that follow.

Collaborative filtering is a solid foundation. Real-world systems often go further, blending these tools with deeper models of behavior and context.

When most values are present and their magnitude is meaningful, distance metrics like Euclidean or Manhattan can capture genuine gaps in behavior.

But when users apply the same scale differently—some rate high by default, others hold back—Pearson correlation is better. It ignores magnitude and focuses on whether preferences move in the same direction.

In a k-nearest neighbors setup, your choice of metric defines who qualifies as "nearest." Distance gives you behavioral closeness. Pearson gives you preference alignment. Combine both, and you get not just who’s similar—but how much their opinion should count.

No single metric fits every case. What matters is understanding your data and what similarity should actually mean.

Building an intuitive comprehention on recommendation based on distances and the Pearson coefficient

The geometry of taste: putting numbers on preferences

Learning, to a computer, is just turning bad guesses into better ones. In this post, we’ll see how that starts with a straight line: how linear regression makes the first guess, and gradient descent keeps improving it.

Let's start with something familiar: house prices. Bigger houses tend to cost more; smaller ones, less. It's the kind of pattern you can almost see without thinking: more space, more money.

When we plot it, the shape is clear: a loose upward slope, with some noise but a definite trend.

![](/posts/least-squares/scatter.png)

As you can see, price and size move together in a way that feels predictable. Not in fixed steps or categories, but on a sliding scale. A house might go for $180,000, $305,500, or anything in between.

Now imagine you're selling your own house. It's 1,850 square feet—larger than average, but not a mansion. You've seen what homes go for in your area, but the prices are scattered. What's a fair number to list it at?

One option is to text your real estate friend and get a half-baked guess. A better option is to look at the pattern in past sales and sketch a line that seems to match it. You grab a ruler, hold it up to the scatterplot, and draw something that feels about right. Then you find your square footage on the x-axis, trace upward to your line, and read off the predicted price.

![](/posts/least-squares/scatter_line.png)

Whatever line you draw, it'll be a steady upward slope. Bigger homes, higher prices. It might not be perfect, but it gives you a way to turn square footage into a price that kind of makes sense.

And while the lines you might draw vary, they all follow the same formula. Each one takes the area of a house (the explanatory variable), multiplies it by a number (the `slope`), and adds another number (the `intercept`).

![](/posts/least-squares/linearFormula.png)

Notice that this isn't just a house-pricing formula. It's how every straight line works. One number sets the tilt (the `slope`), and the other shifts the line up or down (the `intercept`). That's all it takes to draw a line: scale, then slide.

![](/posts/least-squares/slope.png)

In our case, the slope is the "price-per-square-foot". That's the amount each extra foot adds to the total. If the slope is 150, then a 1,000 square foot home would land at $150,000 just from size alone. A steeper slope means prices rise faster as homes get bigger.

![](/posts/least-squares/intercept.png)

The intercept is where the line starts. It's the predicted price of a home with zero square feet. That doesn't mean much in isolation (nobody's buying a 0-square-foot house), but it sets the baseline, like the minimum price you'd expect even for the tiniest studio. If two neighborhoods increase in price at the same rate per square foot, the one with the higher intercept starts from a pricier floor.

Now that we know each line connects an explanatory variable to a prediction using just a slope and an intercept, the question is: **which line should we trust to price our house**?

To answer that, we need a way to measure how well a line fits the data we already have.

Take one of the houses in our dataset. The line says it should've sold for $350,000. But we know the actual price was $375,000. That $25,000 difference is an "error" that measures how far off the line was for that point. Price it too low and you leave money on the table or it too high and the house might not sell.

![](/posts/least-squares/error.png)

A good line keeps the gaps between predictions and actual values small. The simplest way to measure how well a line fits is to add up those gaps—the errors.

Of course, there's more than one way to measure error. And how we choose to combine them is what turns guessing into regression.

One simple option is to use absolute error, just like we did in the previous example. You take the difference between the predicted price and the actual price (no matter which one is higher) and treat it as a distance. If the house sold for $375,000 and one line guesses $350,000, the error is $25,000. Another line guesses $360,000? That one's off by $15,000. Smaller is better.

To compare lines, you just add up the absolute errors across all the houses. The line with the smallest total error is the winner.

If you're trying to price your own home, this makes intuitive sense: you'd rather be off by $10,000 than $30,000.

But absolute error treats all mistakes the same—two medium errors count the same as one big one. That can hide problems.

If your predictions are always a little off, you're consistently close. But if one estimate nails the price and another misses by $100,000, it's hard to know if the model is solid or just lucky. Consistency matters more than the occasional hit, especially when prices shift over time.

Take home prices, for example. Imagine checking listings in your neighborhood every week. Most of the time, estimates are a little off—maybe $10,000 high or low. Nothing to worry about. But then, one day, a house just like yours is priced $150,000 below market. The next week, another is listed way too high. Suddenly, you're not just surprised—you stop trusting the estimates altogether.

To recover trust in the estimates, we might want big mistakes to count more—not just equally. One way to do that is to make the penalty grow faster as the error increases. In effect, we're plotting the errors on a curve, not a straight line. The further out you go, the steeper the penalty becomes. That's what it means to use a non-linear scale: small misses barely move the needle, but big ones explode.

![](/posts/least-squares/comparing.png)

Instead of measuring error directly, we square it. The effect is clear in the plot above: a $20,000 miss isn't just twice as bad as $10,000—it's four times worse. A $50,000 error? Twenty-five times the cost. Bigger mistakes grow fast. Squaring the errors pushes us toward a line that stays consistently close, not one that swings between lucky guesses and huge misses.

Everything we've done—measuring errors, choosing how to weigh them, and adding them up—comes together into a single idea: the *error function*. It's just a rule for scoring how well a line fits the data. Squared error is a popular option. Absolute error is another, but they are not the only ones.

Different error functions reflect different priorities. For example, the [Deming regression](https://en.wikipedia.org/wiki/Deming_regression) uses a different error function that accounts for errors in both variables—not just the one we're trying to predict. It's useful when both measurements are noisy, like when comparing lab results from two different instruments.

Many lines are close to the data points. We don't want any good line, but the best one (the one with smallest error). No matter which error function you choose, the goal stays the same: find the line that minimizes the error function.

But that raises the next question: where do these lines come from? One option is brute force. Get a set of every possible combination of slope and intercept, one by one. Then, calculate the error and keep the best one.

It works. At least in theory. But there's a problem: there are way too many possibilities. Infinite ones. Testing them all would take forever. We need a more sensible algorithm than computing the error of every possible line.

To understand what we're up against, imagine a plot where each point represents a different line: the slope and intercept are the coordinates, and the vertical axis shows how bad the line is—its error. In this landscape of errors, high points mean worse lines; low points mean better ones. Our goal is simple: find the lowest point. We won't actually draw this plot (there are too many possibilities), but picturing it helps make the task clear.

For example, if you look at the absolute error surface, it's a bit of a mess. It's not smooth like a hill or a valley. Instead, it's more like a weird origami sculpture, with sharp edges and folds.

![](/posts/least-squares/absolute3d.png)

In some cases, like in the example below, every point along the fold gives the same lowest error. If that happens, it means there's not *one* best line, but *many*.

![](/posts/least-squares/folded3d.png)

You can see this clearly in a 2D slice: fix the intercept at zero, and a range of slopes yield the same lowest error.

![](/posts/least-squares/absolute2d.png)

This ambiguity matters because if multiple lines fit the data equally well, then we won't have a single best answer—just a range of equally good ones. Imagine trying to sell a house whose price could reasonably be $100,000, $500,000, or $1,000,000, depending on which line you pick. Not ideal.

This situation happens when the data points are symmetric: tilting the line slightly brings it closer to some points while moving it away from others, without changing the total error. The plot below shows an example. Any line that passes through the green area (like the red one: is a valid solution.

![](/posts/least-squares/multiplesolutions.png)

This kind of ambiguity happens to absolute error only. If we were using squared error instead, the plot would look different: smooth and bowl-shaped, with a single lowest point. One best line, stable and predictable.

![](/posts/least-squares/squared3d.png)

If you were standing anywhere on that surface, you'd just need to walk downhill to find the best line. Pick the direction where the slope is steepest, take a step, and keep moving downhill. Eventually, you'll reach the bottom.

You don't have to explore in every direction or worry about getting stuck. No matter where you start, as long as you follow the steepest descent, you'll get to the best solution. It might take 100, 1,000, or more steps, but you'll get there.

Of course, we're not *actually* hiking across hills. In our case, walking downhill means tweaking numbers: the slope and intercept of the line. Each point on the error surface corresponds to a specific line. Nearby points represent lines with similar parameters. By nudging the slope and intercept step by step, we move through this space—chasing lower and lower error until we land on the best fit.

At that point, we can stop and look at the slope and intercept of the line we found. That's our best guess for the price of your house.

To get a better feel for what's happening as you step down the valley, you can zoom in on just one dimension—the slope. The plot below shows how the error changes as we adjust the slope (while keeping the intercept fixed). Notice how the curve is smooth and bowl-shaped, making it clear how to keep stepping downhill.

![](/posts/least-squares/squared2d.png)

At every step, we need to decide which way to move: left or right. To do that, we measure how steep the curve is at our current point. This measurement (the steepness of the curve) is called the derivative.

If the derivative is positive, it means the error increases if we move right, so we should go left. If the derivative is negative, it means the error increases if we move left, so we should go right.

In short: the derivative points uphill. Since we want to go downhill, we move in the opposite direction.

We keep stepping like this—checking the derivative, flipping directions if needed—until we land at a point where the slope is flat. That's when the derivative becomes zero, and we've reached the minimum.

![](/posts/least-squares/gradient.png)

> When using least squares, a zero derivative always marks a minimum. But that's *not* true in general. It could also be a maximum or an inflection point. To tell the difference between a minimum and a maximum, you'd need to look at the [second derivative](https://en.wikipedia.org/wiki/Second_derivative). If the second derivative is positive, it's a minimum. If it's negative, it's a maximum. If it's zero, we're at an inflection point.

This process of iteratively minimizing a function using derivatives is called **gradient descent**.

The gradient descent is an intuitive and efficient algorithm—and squared error plays especially nice with it.

Squaring the errors keeps everything smooth: no sharp corners, no sudden jumps. That matters because gradient descent needs to measure slopes—and if a function isn't smooth, you can't always define a clear slope to follow.

This wasn't always obvious. In the 1800s, mathematicians like Karl Weierstrass showed that you can have a perfectly continuous curve that's still so jagged that it has no slope at all—not even at a single point. It was a wake-up call: just because something looks smooth from far away doesn't mean it actually is.

Minimizing absolute error runs into a smaller version of this problem. The absolute value function has a sudden bend at zero—right where you most need a clean slope. You can work around it with special tricks, but it's messier and less natural.

Squared error, on the other hand, glides along smoothly. Its derivative is clean, continuous, and geometrically meaningful—it not only points you downhill, but also tells you how big a step to take. No hacks required.

**This is pretty much the real reason why everyone squares their errors instead of taking the absolute value, even when it might not be the most appropriate pick. Squared error makes optimization smooth and easy *and*, as we saw earlier, it also guarantees a single, stable best solution.**

As a side note, there are many ways to find a function's minimum, but Gradient Descent rose to fame thanks to its handsome cousin: the [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent), which is the algorithm of choice for training neural networks.

Underneath all the complexity, deep learning still runs on the same basic idea: adjusting parameters to minimize error, step by step, just like we do with Linear Regression and Squared Errors.

It's simple math, sharpened by persistence. The kind of quiet hustle that helps you put a price on a house and actually trust it.


Building an intuitive understanding of how Linear Regression works and how it leads to Gradient Descent

How a straight line teaches machines to learn

**Today, we're launching Briefer's AI anayst.**

Briefer's AI analyst is an intelligent agent that helps *anyone* on your team turn data into clear, instant insights.

<iframe width="640" height="360" src="https://www.youtube.com/embed/VBXmDJCqOGk" title="" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Notice that this isn't just another text-to-SQL chatbot. Instead, it's an AI agent running inside our powerful data workspace (the one technical users already love).

Our agent can write SQL, plot graphs, and even use Python for deeper analysis.

You can also upload your own business context and metadata, making the agent even smarter.

And because it operates directly in Briefer's notebook interface, you'll see exactly how it thinks—every query, every action, every line of code. That means full transparency and auditability for every insight.

**If you're serious about making your team truly data-driven, [book a demo using this link](https://calendly.com/lucasfcosta/briefer-intro)**.

Today, we're launching Briefer's AI anayst—our intelligent agent that helps anyone on your team turn data into clear, instant insights.

Launching Briefer's AI analyst

When I was writing my first book, my publisher taught me that "books with plenty of diagrams sell better". People flip through, see the visuals, and *feel* like they'll understand it. They don't read a word. They just *know*, and they buy your book.

Same way you'll scroll through this post and think, "yeah, I'll get this."

That's the trick. Visuals feel easy. Your brain sees shapes and goes "I get this." You don't need to decode a table or read a paragraph. **Patterns just pop out.**

**That's exactly why you should plot your data before doing anything else**. Not for decoration. For *understanding*. You'll spot weird outliers, duplicated trends, clumps of activity, empty gaps—things you'd never notice in raw numbers.

In this post, we'll look at the best plots for getting a feel for your data and what to look for in each one.

 

# Histograms

Let's begin with something boring: histograms. They're the fastest way to see the shape of your data—where it piles up, where it thins out, and whether anything looks off. You'll spot skew, gaps, fat tails, or the classic "oh no, this is just one giant spike." It's not fancy, but it tells you what you're dealing with. Before fitting models or running tests, just plot that damn histogram.

![](/posts/look-the-data/histogram.png)

Start with the obvious: outliers.

That one bin way off to the side can stretch your axis, distort your stats, and confuse your model. It might be a bug. Or it might be the most interesting person in the data, like the Rollercoaster Tycoon dev who hand-wrote 50,000 lines of assembly. Either way, it's shaping your results, whether you notice or not.

![](/posts/look-the-data/outlier.png)

Then look at the shape of the distribution. Some datasets don't clump around a single peak. Instead, they split.

![](/posts/look-the-data/multiple_peaks.png)

Two humps, maybe three. That's not noise, that's signal. If your histogram has multiple peaks, don't just shrug. Go figure out if you're looking at different groups mixed together.

Say you're tracking how often people use the terminal: one peak for folks who barely touch it, another for people who live in it. Dig deeper, and you realize Windows and Linux users were hiding in the same chart.

![](/posts/look-the-data/multiple_peaks_groups.png)

That kind of split is called a multimodal distribution, and it's a warning: your data isn't telling one story. It's telling several.

Spotting that kind of split matters. It changes how you model, how you segment, how you interpret the results. Otherwise, you're averaging across groups that don't belong together, and whatever you build next will be based on fiction.

Finally, watch out for skew. A distribution is said to be "skewed" if one tail stretches out longer than the other.

A lot of real-world data behaves this way. Take salaries: most people cluster toward the bottom, but a few stretch the scale so far you start questioning the system. That long tail on the right? That's a right-skewed distribution.

![](/posts/look-the-data/skewed_histogram.png)

Skew isn't the same as outliers. Outliers are a few rare, isolated points. Skew is *structural*. It's the shape of the whole distribution. Most users get crammed into one end of the range, while a long tail quietly stretches out and distorts everything else.

Think of a response time metric for an app. Most users get a response in under a second, but a few unlucky ones wait 5, 10, even 20 seconds. That long tail skews the whole distribution.

Now, say you build an alert that fires if response time goes above the 95th percentile. Sounds smart, until you realize that, for most users, that alert never triggers, even when things feel slow. Picking a sensible "default timeout" is just as tricky because the average is skewed by a few extreme cases, so you end up setting a default that's too generous for most users, and too short for the long tail.

Left skew is the opposite: the long tail drags left. Either way, skew can mess with your stats and assumptions. But in many cases, a simple transformation can help straighten things out.

If you see heavy skew, try a simple transformation (like a [log scale](https://briefer.cloud/blog/posts/logarithms/)). It'll compress the extremes, pull the bulk apart, and make patterns easier to see and model.

![](/posts/look-the-data/normalized_histogram.png)

 

# Quantiles

Quantiles slice your data into equal-sized chunks. They give you a structured way to understand the spread without getting bogged down in raw numbers. Instead of eyeballing two histograms and guessing which one's "more spread out," you can just compare their quantiles. Are the medians far apart? Is one dataset more compressed in the middle but stretched at the edges? Quantiles help you spot those patterns. Therefore, plotting quantiles right on the histogram makes those patterns easier to see and easier to explain.

Some of the most familiar quantiles are percentiles and quartiles. Percentiles slice your data into 100 pieces, quartiles into 4. They show up everywhere, from test scores ("you're in the 90th percentile") to dashboards ("p95 latency"). The middle quartile (the 50th percentile) is your median, and it's often a better summary than the average, especially when your data's skewed or full of outliers.

![](/posts/look-the-data/quantiles.png)

Let's say you're tracking API response times. You plot the histogram and it's a mess: most requests are fast, but a few are way out to the right. Now add vertical lines for the median, 95th, and 99th percentiles. Suddenly, the picture sharpens. The median says "half our users get a response in under 200ms". Great. But the 95th percentile shows some are waiting 2 seconds, and the 99th is worse. Those tails matter. They're the users calling support, churning, or tweeting screenshots. Plotting quantiles doesn't just show spread. It tells you who's going mad.

 

# Box-and-whiskers

Quantiles are great for understanding spread but they're still just lines on a histogram. What if you want to compare distributions side by side, across groups or categories? That's where box-and-whisker plots come in. They take those same quantiles and turn them into a compact, visual summary.

The box shows the middle 50% of your data, from the 25th to the 75th percentile. The lines, called whiskers, usually extend to the smallest and largest values that fall within 1.5 times the height of the box. Anything beyond that is flagged as a potential outlier and plotted separately. That's the standard in most tools like Python's `matplotlib` and `seaborn`, or R's `ggplot2`, and it gives you a quick way to see both the bulk of your data and what's unusually far from it.

![](/posts/look-the-data/boxplot_iqr.png)

For example, let's go back to our terminal usage data. This time, we plot a box-and-whiskers chart comparing Linux and Windows users. Now the difference jumps out: the least active Linux user still uses the terminal as much as the median Windows user. The entire Linux distribution is shifted higher. You could see the peaks in the histogram, but now the comparison is concrete. You can point to the medians, the spread, the outliers. It's no longer just a feeling. It's a fact.

![](/posts/look-the-data/boxplots.png)

 

# Violin plots

Box plots are great for comparing groups, but they hide one important detail: the actual shape of the distribution. You get the quartiles, the median, and some outliers, but not whether the data is clumped, stretched, or split into separate groups. It can make everything look smooth and simple, even when the reality is a lot messier.

Violin plots fix that. They keep the summary from the box plot, but overlay it with a smoothed curve that shows the full distribution (essentially, a rotated and mirrored histogram). That way, you get the best of both worlds: clean comparison and full context.

![](/posts/look-the-data/violin.png)

But violin plots only work when you're comparing distributions, not when you're tracking values over time or looking at raw counts. And visually, they can get messy and overcomplicate simple data. So use them when you need to compare distributions - but for other types of comparisons, you'll want different tools.

 

# Bar plots

If you're comparing a lot of groups, violin and box plots also start to fall apart. Too many shapes, too little clarity. In those cases, it's better to aggregate the data (using the mean, median, max, etc.) and switch to a bar plot. Bar plots make it easy to scan and compare lots of categories side by side.

And just a quick reminder: **avoid pie charts**. They make it almost impossible to compare values accurately, especially when the slices are close in size. Your brain isn't wired to judge angles or arc lengths, but it's great at comparing bar heights. Pie charts look friendly, but they hide the data.

To bring it home, here's a simple example. I pulled some of the most popular programming languages from the Stack Overflow 2024 survey. The bigger the bar, the more people use it. No smoothing, no quartiles. Just raw counts, stacked side by side.

![](/posts/look-the-data/bar.png)

And yes, unfortunately, JavaScript is still there. Some truths you just have to accept.

 

# Scatter plots

Bar plots are great for comparing quantities across several categories-one variable, maybe broken down by groups or periods. But they can't show how two things change together, like whether developers who sleep more write fewer bugs. That's where scatter plots come in.

Scatter plots let you see relationships between two continuous variables. Want to know if more debugging leads to less happiness? Or if people who drink more coffee ship more features? Plot the points. Each one is a pair (X and Y) and the shape they form tells the story.

Here's a classic example: happiness vs. time spent debugging. Each red dot is a programmer. The trend is clear: the more time you spend debugging, the less happy you are.

![](/posts/look-the-data/scatter.png)

So what should you look for in a scatter plot? Start with the shape. If the points stretch into a line or an ellipse, there's probably a real relationship. If they form a vague cloud or a neat circle, the variables likely aren't connected, like happiness and Pluto's position. Unless, of course, you believe mood swings are caused by retrograde motion.

![](/posts/look-the-data/no_relation_scatter.png)

You might also spot clusters: groups of points that behave differently from the rest. Maybe there's a bunch of users who are never happy, no matter how little they debug. Plotting them separately can reveal what's hiding underneath. Turns out, they're all Java developers still stuck writing getters and setters.

![](/posts/look-the-data/scatter_groups.png)

And don't forget about outliers. Scatter plots are great at making them stand out. Like Jonathan, the programmer who loves debugging. While everyone else sinks, he soars. Maybe he's an error. Maybe he's a genius. Either way, he breaks the pattern.

![](/posts/look-the-data/scatter_outlier.png)

Sometimes that's all you need: a clear line, a weird outlier, a tight little cluster. But other times, the pattern hides in the shape. Not every relationship forms a neat straight line. A lot of traditional stats assume it does, but real data often bends, flattens, or takes strange turns. That's exactly where plotting helps. It shows you when things are drifting from expectations before you build the wrong mental model.

Say you're looking at how coffee affects programmer productivity. Plot the data, and you might notice something interesting: productivity rises at first, but then levels off. That kind of pattern is nonlinear: it curves instead of forming a straight line.

![](/posts/look-the-data/nonlinear.png)

Why does that matter? Because if you assume a straight line when the relationship curves, your predictions will be way off, especially at the edges. You'll think the tenth cup of coffee makes you twice as productive, when in reality it just gives you a minor panic attack.

Sometimes, a simple fix, like applying a log scale to the axis, can help straighten things out. Plots like this don't just show you what's happening; they show you when simple assumptions break down, and when you might need to look at the data a little differently.

 

# Heatmaps

Scatter plots are great for seeing how two variables relate. But what if you need to add a third variable? That's when you turn to heatmaps.

Instead of points scattered on a plane, a heatmap uses color to represent that extra dimension, making complex patterns easy to spot.

A classic example is GitHub's contribution heatmap. The two axes show months and weekdays, while the color shows how many commits someone made each day. Take a look at the one below. That's Linus Torvalds' contribution history. At a glance, you can see when he's most active, and when things cool down.

![](/posts/look-the-data/heatmap.png)

Choosing the right color palette matters because it shapes how people interpret your data. Usually, darker colors feel like "more" and lighter colors like "less," but that's not a fixed rule, it depends on context.

Imagine flipping GitHub's heatmap colors: intense, dark squares for zero activity would feel weird, right? When your data has a meaningful midpoint, like temperatures centered around zero, it's best to use a diverging color palette, where two colors meet in the middle. Cold goes one way, hot the other, and zero stays neutral.

![](/posts/look-the-data/palettes.png)

So what should you look for in a heatmap? Patterns and contrasts.

Look for clusters, where values pile up in one area, and gaps, places where things consistently go quiet. Pay attention to streaks-rows or columns that stand out-and sudden color shifts that might point to meaningful events or changes. Heatmaps are all about spotting these trends at a glance, so trust your eyes first, then dig deeper.

 

# Line plots

Bar charts shout. Pie charts charm. Line Plots reveal motion. They show how things change across time, categories and variables. The points represents average over time and the lines are maximum and minimum measurements.

Take Brazilian product sales over time. Here, the y-axis is units sold over years. Each line is a product: Watermelons, Easter eggs, and [Panetone](https://pt.wikipedia.org/wiki/Panetone), an Italian Christmas cake that's actually more popular in Brazil than in Italy. In a way, it's the Brazilian version of British mince pies.

![](/posts/look-the-data/line.png)

Here you can see that Panetone flatlines for most of the year and then skyrockets in December. Watermelon peaks in the hottest months, January and February, then cools off as summer fades. Chocolate eggs show a clear spike around Easter, then, they're gone like they never existed.

That's the magic of line plots: they don't just track numbers. They reveal timing, behavior, and opportunity, all in one continuous story.

 

# Putting it all together

We've seen a lot of plots-each with their own strengths. But across all of them, the questions you're asking are very similar.

Whether it's a histogram, a scatter plot, or a heatmap, the goal is to get a feel for the data. To see what stands out, what's typical, and where things behave differently than expected.

Here's what to look for-no matter what plot you're using:

* **What's typical?** What values are common, where does the data cluster, and where does it thin out?
* **What's unusual?** Are there outliers-points that stand far from the rest? Are they errors, or something worth investigating?
* **Is the data shaped how I expect?** Look for skew, gaps, and multiple peaks. Real-world data is rarely smooth.
* **Are there clear groups?** Do values naturally split into clusters? Can you tell them apart, or are they overlapping?
* **How do variables move together?** Does one go up as the other goes down? Is it linear, curved, or noisy?
* **What's missing?** Are there empty spots where data should be? Do any categories or ranges look oddly quiet?

The more you practice spotting these, the faster they jump out-long before you run any numbers. That's the real power of plotting: it gives your brain something to lock onto before your tools even get involved.

By the way, remember how I said visuals *feel* easy? How your brain skips the words and just says, "yeah, I get this"?

You just did.

Which plots to use to get a feel for your data and what to look for in each of them.

How to get a feel for the data

Today, we're launching [Briefer's open-source offering](https://github.com/briefercloud/briefer)

Briefer is a platform for people to build notebooks and dashboards in a single place. It's much more flexible than traditional BI tools because it allows you to write code and use any library you want to manipulate data and create visualizations.

<video autoplay loop muted playsinline class="w-full rounded-lg shadow-lg mt-3 mb-4 overflow-hidden border border-gray-100">
 <source src="/posts/launching-briefer-oss/full-demo-oss.mp4" type="video/mp4">
</video>

From today, you can use Briefer for free on your machine, or deploy it to your own infrastructure so that your whole team can use it and eliminate your data silos.

 

## What's included in the open-source offering?

![](/posts/launching-briefer-oss/oss-vs-paid.png)

Briefer's open-source offering includes everything you need to build notebooks and dashboards, including Markdown, SQL, Python, point-and-click visualizations, scheduled runs, writebacks, and more.

Also, given you'll be running Briefer on your own machines, there's no limit on how many users you can invite, how much memory and compute you can give to your notebooks, and how many pages you can schedule.

The only things that are *not* in Briefer's open-source offering are PDF exports, granular permissions, and email and Slack integrations. These are available in the paid version of Briefer, for which you can sign up [here](https://briefer.cloud).

 

## Why open-source?

We want people to have zero barriers to using Briefer. We believe that the best way to do that is to make it open-source.

By making Briefer open-source, we're giving you the freedom to run it on your own infrastructure, customize it to your needs, and contribute to its development.

Furthermore, given Briefer will be running on your own machines, you don't have to worry about your data being stored in a third-party service. You have full control over your data.

If you want an in-depth explanation of our open-source strategy, [you can read it here](/blog/posts/open-source-strategy).

 

## How do I get started?

You can get started by following the instructions in our [GitHub repository](https://github.com/briefercloud/briefer). While you're there, please don't forget to give us a star ⭐.

If you have any questions, please send me an email at lucas.costa\[at]briefer.cloud. I respond to every email I receive from users.

Now, it's time to build.

Best, LdC.

Launching Briefer's open-source offering

This changelog is all about quality-of-life improvements. We've added search, keyboard shortcuts, pivot tables, and support for Trino as a data source.

 

## Search

The more people use Briefer, the more pages they create. This can make it hard to find the page you're looking for. That's why we've added a search bar to the pages list.

![](/posts/changelog-9/search.gif)

To toggle the search bar, click the magnifying glass icon on the top right of the pages list. Then, you can type the name of the page you're looking for, and the list will be filtered to show only the pages that match your query. Alternatively, you can use the `CMD (or Ctrl) + K` keyboard shortcut to focus the search bar.

Please notice that this is not a full-text search. It only searches for page names for now.

 

## Keyboard Shortcuts

Many people coming from Jupyter Notebooks are used to using keyboard shortcuts to navigate their notebooks. That's why we've added a few keyboard shortcuts to Briefer.

![](/posts/changelog-9/shortcuts.gif)

Briefer's keyboard shortcuts have two modes: command mode and edit mode. You can toggle between them by pressing `ESC` and `Enter` (or focusing on the block), respectively.

In command mode, you can move between blocks using the arrow keys or `h`, `j`, `k`, and `l`, just like in Vim. You can also delete blocks with `dd`.

To add new blocks, you can use `a` to add a block above the current one, or `b` to add a block below it, followed by the block type you want to add: `m` for Markdown, `p` for Python, and `q` for query.

When editing blocks, you can use `CMD (or Ctrl) + Enter` to run the block, `Shift + Enter` to run the block and focus on the one below it, and `Alt + Enter` to run the block and add a new one below it. You can also `ESC` to exit edit mode.

**To view all these keyboard shortcuts within the platform, you can open `Keyboard Shortcuts` within the three dots menu on the top right of the screen when you're inside a notebook.**

 

## Pivot Tables

Pivot tables are a powerful way to summarize and analyze data. They allow you to group data by one or more columns and apply an aggregation function to the values in another column.

![](/posts/changelog-9/pivot-tables.gif)

To use pivot tables in Briefer, you can create a query block and write a SQL query to act as the source of the pivot table. Then, you can click the `Pivot Table` option when adding a new block to create a pivot table from the query results.

 

## Trino support

Some of our users have been asking for Trino support, and we're happy to announce that it's now available in Briefer.

To add a Trino data source, you can go to the data sources page and click the `Add Data Source` button. Then, you can select `Trino` from the list of available data sources and fill in the connection details.

We've made plenty of quality-of-life improvements to the platform this month. We've added a search bar to the pages list, keyboard shortcuts, pivot tables, and support for Trino as a data source.

Changelog #9: Search, keyboard shortcuts, pivot tables and Trino support

Today, we're launching [a free and open-source offering](/blog/posts/launching-briefer-oss). But I'll be honest with you: Briefer is a VC-backed company, and it must make money.

It's ironic to me that many *for-profit* companies can't say that out loud. Maybe it's because they're afraid that money will get in the way of making users happy. Or maybe they know it won't, but they're afraid of people thinking it will.

We don't think that way.

We think that doing right by your users is the best way to make money, and making money is the best way to do right by your users. That's because money allows you to invest in improving your product and offer a better service to more people, some of whom will use it for free, and that's fine.

![](/posts/open-source-strategy/oss-cycle.png)

Being clear about our strategy and intentions is part of doing right by our users and it's why I'm writing this post.

In it, I'll explain how companies can structure their open-source offerings, and how we structured ours in a way that's good for users and good for business. I'll also explain why we chose the AGPL license, how it benefits users, and how it benefits us too. Finally, I'll also explain some of the things we won't do, including changing our license.

 
 

# Structuring an open-source offering

![](/posts/open-source-strategy/summary1.png)

When companies open-source their software it means [they can't *really* charge for the software itself](https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowRequireFee)¹. Thus, if you must make money, you need to sell something else, which is usually:

1. Support and professional services
2. Compute and infrastructure
3. A non-open-source version of the software

In the first case, companies scale as a function of how many support and consulting staff they can hire. It's a solid model but it doesn't scale because hiring people is costly.

As a result, this approach often falls short of [investors' expectations for significant returns](https://www.saastr.com/vcs-need-you-to-be-a-potential-fund-returner-when-they-invest-but-after-expectations-there-are-more-nuanced), which makes it hard to raise funding and thus prevents most founders from being able to go full-time on their project. Consequently, it becomes difficult to build enough traction to generate leads for the professional services they would like to sell, especially considering there's no marketing budget.

The second case, selling compute and infrastructure, usually comes in the form of a cloud-hosted version of the software. This model works well for companies building complex and mission-critical software, like databases and managed Kubernetes clusters. That's because it's a bet that the company can provide a cheaper, more convenient, and more reliable service than customers could provide themselves.

The problem with this model is that it incentivizes companies to make it hard for customers to run the software on their own. This dysfunctional pattern manifests itself in many ways, like when companies insist on overly complex helm charts (*ugh*), withhold documentation on how to run the software, or require customers to operate a dozen pods when one would work just as well.

The third case, selling a non-open-source version of the software, is fair in the sense that companies will still give value away while carving out a subset of features upon which they can charge. That way, they can still make money to maintain and improve the open-source version, while profiting from the extra features in the non-open-source version.

The problem with this last model is that it creates a conflict of interest between the company and its customers. Customers want the open-source version to have more features, while the company wants the open-source version to have *fewer* features so that customers will pay for the non-open-source version.

 
 

# Our model

![](/posts/open-source-strategy/summary2.png)

Briefer's approach is a mix of the second and third models. We will sell a freemium cloud-hosted version of our software and maintain a generous AGPL-licensed version with fewer features.

Some people call our strategy ["open-core"](https://about.gitlab.com/blog/2016/07/20/gitlab-is-open-core-github-is-closed-source/) and that's technically right. Still, I'd rather say that we have two pieces of software: one that is open-source and another that is not. I think that's more direct because we're not trying to hide the fact that we're selling a non-open-source version of our software. Anyway, open-core is fine too, and there's nothing wrong with it.

We chose to do it this way because we want everyone to use Briefer. Therefore, choosing a service-based model would not be viable because it limits our access to funding, which is necessary to scale Briefer and pay our bills as we build it.

We then considered picking the second model, selling compute and infrastructure, but I was afraid it would corner us into having to eventually change licenses.

Another reason why we rejected that model is because we refused to compromise the software's quality just to profit from users' struggles to run and operate it. Making it ugly, inefficient, and poorly documented was not an option. That would completely defeat the purpose of having an open-source offering, and, as an engineer, it would cause me physical pain.

That's when we decided to mix the second and third models.

This choice allows us to have a small subset of features for which we can charge so we don't have to worry about how easy it is to run Briefer.

That way, there will be an incentive to make the open-source version as easy as possible to run because that helps us increase our user base, thus increasing the number of people who might eventually pay for the cloud-hosted version.

This incentive to increase the user base (our top-of-funnel) also means we can't simply reserve the best features for the non-open-source version. Instead, we have to find a balance between an open-source version that's competitive and attractive enough for people to use it and a cloud-hosted version that's valuable enough for people to pay for it.

In our case, we decided to charge only for the features that larger companies usually need (>100 employees) or those necessary to directly make money from Briefer, like embedding charts into your own application.

![](/posts/open-source-strategy/oss-vs-paid.png)

We thought that was a fair heuristic because companies with more than 100 employees usually have more resources to pay for software, and they usually get more value from it, so it makes sense to take a small cut.

Conversely, small companies are usually strapped for cash, and it wouldn't make sense to charge them pennies. We'd rather help them now and have them eventually pay for the cloud-hosted version when they're thriving and getting more value out of our software. That way, we can help them grow and make enough money to maintain, improve, and distribute both the free and paid versions of Briefer. That's a win-win.

 
 

# Why we chose the AGPL license

![](/posts/open-source-strategy/summary3.png)

We licensed Briefer's open-source version under AGPL for three reasons.

The first is that it's a well-known license, so most people will be fully aware of what they can and can't do with our software.

The second is that it's a great way to ensure that the open-source software will remain open-source forever.

The third is that the AGPL still allows users to implement any features they need, but disincentivizes competitors from copying our software and selling it as their own.

If they were to do that, they would have to open-source all their changes and commoditize their work, so it's not worth it for the vast majority of companies.

Here's where the difference between AGPL and GPL comes into play. We chose AGPL because it closes [the "network loophole"](https://www.revenera.com/blog/software-composition-analysis/understanding-the-saas-loophole-in-gpl/) that exists with the GPL. Under GPL, it was possible to modify the software and use it over a network without releasing the changes. But with AGPL, if competitors modify our software and offer it as a service over a network—like a cloud application—they must make those changes public under the same AGPL license.

We never considered using a license that's not in [the OSI-approved list](https://opensource.org/licenses) because those are, obviously, not open-source. Additionally, we have already made a compromise in not open-sourcing the whole codebase, so I thought it would be fair to pick the "freest" license of them all.

 
 

# Our open-source strategy

![](/posts/open-source-strategy/summary4.png)

The main reason we created an open-source offering is that it's the best way to align our user's interests with the company's financial interests.

On one side, more people can use Briefer because it's free and open-source. These people can implement the features they need, and distribute them to others.

On the other side, it's a competitive advantage for us because it increases our top of funnel, helps us improve our activation rates, and gives us a competitive edge against incumbents.

Here's each of these points in more detail.

 

### Taking on incumbents

Briefer's main competitors are Salesforce's Tableau and Microsoft's Power BI. At this point, these companies aren't just companies anymore. They're institutions. They have a lot of money, a lot of engineers, and they're not going to go away anytime soon.

On the other hand, Briefer is a small company that's doing well, but that's just starting. We don't have the same brand recognition, the same number of customers, or the same amount of money. That's why people ask us questions like: "How do I know you'll be around tomorrow?" and "How are you different?"

Open source is a great way to answer these questions.

That's because open-source software outlives companies. If Briefer disappears tomorrow, people can still use the software, implement the features they need, and distribute them to others. That's a great way to assure everyone that using our software is a safe bet too.

Open-source software is also a great differentiator because it helps us build a strong community. These people can contribute features, fix bugs, and report issues, which makes our software better, faster, and more reliable.

Finally, by going open-source we commoditize our competitors' core functionality. This means they now have to compete against us in terms of innovative features, performance, and price, all of which are usually not their strong suits, let's be honest.

 

### Helping more people while increasing our top-of-funnel

Making our software open-source means more people can use it because they won't have to "hop on a quick call" or swipe their credit card to get started.

Instead, people can just clone the repository and run a single command to get started.

That's great for people because they can use Briefer for free to study, work on their personal projects, or grow their business. It's also great for us because the lower barrier to entry increases our top-of-funnel.

Then, if they like our software, we win, regardless of whether they pay for the cloud-hosted version. That's because non-paying users who like our software will certainly tell their friends about it, and some of them will eventually pay for the cloud-hosted version.

As I mentioned earlier, this incentive to increase the user base also means we can't simply reserve the best features for the non-open-source version. Otherwise, we'd be shooting ourselves in the foot because no one would use it.

Furthermore, if we're successful, it's likely that more companies will create competing open-source offerings. That will push us to open-source more features so that we can stay competitive, which is better for all users, even the ones not using our software.

 

### Improving activation rates

Most people out there don't go around pasting their database credentials into every SaaS product they find. Sometimes they don't trust a company they've never heard of, sometimes they can't punch a hole in their firewall, and sometimes they simply don't want to go through the hassle of getting management's approval.

This behavior makes it more difficult for people to see value in Briefer because they won't have real data from which they can derive valuable insights. As a result, they will churn.

We tried to solve this problem in many ways, including:

1. Creating demo datasets that people can use to get started
2. Supporting file uploads so people can upload their data without having to connect to a database
3. Devising a thorough document detailing all our security practices in an attempt to make people trust us

None of that worked well. All of these initiatives were helpful, but none truly solved the problem.

Looking back, we're now confident that the only effective solution is to make our software open-source. That way, people can run it locally, with their own data, without having to connect to the internet, and behind a VPN if necessary.

This change will increase activation rates, which will then lead to higher retention rates, which leads to more feedback, which leads to a better product, which leads to more users and more conversions.

 

### Crowdsourcing innovation

It's difficult to figure out what people want, and it's even more difficult to figure out how to prioritize everything they want.

Open-source helps us manage Briefer's roadmap along with our users because there will be more of them, and because they'll have access to the source code. That way, they can help us figure out where to go, and help us get there by implementing what they need.

 

### It makes us happier

The first time I worked with my co-founder was when we were maintaining an open-source project called [Chai.js](https://github.com/chaijs/chai). It was the most fun I've ever had working on a project, and I want every day at Briefer to feel like that.

That may sound like a selfish reason, but it's not.

Building a company takes many years, and the only way to keep going is to have fun along the way. That's because building a company is hard, and it's easy to give up when things get tough, but not when you're having fun.

When you're having fun you don't mind the late-night coding sessions, all the "no's" you get from investors, or the bugs that keep popping up. You just keep going because you're having fun, and time will pass anyway, so you might as well spend it wisely and build something meaningful.

 
 

# Things we won't do

![](/posts/open-source-strategy/summary5.png)

Finally, I want to make sure I address some things we will *not* do.

The first is to change our license.

One of the main reasons why Briefer has two separate pieces of software is so that we'll always have a significant number of money-making levers to pull in the non-open-source version.

I think having separate licenses is a better compromise than risking our users' trust by changing the license of the open-source version in exchange for short-term profits.

The second thing we won't do is to stop maintaining the open-source version.

Open-source is a core-part of our strategy. It's the right thing to do for our users, and, consequently, it's the right thing to do for us because it makes people happy, and because it helps Briefer succeed in all the ways I mentioned earlier.

The third and final thing we won't do is to stop being transparent.

I want to build Briefer as a company that people can trust, and that means being transparent about our intentions, our strategy, and our roadmap. That's the right thing to do for our users and, honestly, it's good for business. Therefore, it is the right thing for us too.

If you have questions, email me at lucas.costa\[at]briefer.cloud and I'll be happy to answer them personally.

Best, LdC.

## Footnotes

\[1] [You *can* charge for people for copy of GPL-licensed software if they're getting it from you](https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowMoney). So, *technically*, you could make money selling software itself. The reason that wouldn't work in practice is that [others could then redistribute the software for free](https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowRequireFee). Therefore, it would be nearly impossible to compete with them.

Going open-source as a VC-backed company

This month is packed with new features and improvements. We've been working hard to make Briefer more powerful and easier to use, and we're excited to show you what we've been up to.

 

## Publishing notebooks

Now, whenever you open a notebook, it'll be shown in "view" mode. This means you can't edit the notebook until you click the "Edit" button on the top right corner of the screen. If a notebook has never been published, opening it will automatically put you in "edit" mode.

Whenever you edit a notebook, you must click the "Publish" button to "push" your changes for viewing. This way, you can be sure that your changes are saved and that you won't accidentally modify a notebook while others are viewing it or when a schedule is about to run.

![](/posts/changelog-8/publishing.png)

The same applies to the dashboard view. You can't edit a dashboard until you click the "Edit" button on the top right corner of the screen. If a dashboard has never been published, opening it will automatically put you in "edit" mode.

Please notice that now the schedule button will run the latest published version of the notebook, not the one you're currently editing. This is to prevent accidental changes to the notebook while the schedule is running.

 

### Hiding and showing blocks from the published version

Another cool feature that comes with publishing is the ability to hide and show blocks from the published version of the notebook.

This is great for hiding blocks that are only useful for debugging or development purposes.

![](/posts/changelog-8/hideblocks.gif)

To hide a block, click the eye icon on the left side of the block. To show it again, click the same icon. If you want to hide all tabs at once, you can click the "Hide all" option within the drag-handle menu on the left.

 

## Dynamic dropdowns

You can now use a dataframe's column to populate a dropdown in your notebook. This is great for when your filters depend on a query's result instead of hard-coded values.

![](/posts/changelog-8/dynamic-dropdowns.gif)

To make a dropdown dynamic, you need to select the `Dynamic` option at the bottom of the dropdown configuration. Then, you can select the dataframe and column you want to use to populate the dropdown.

To hide the dropdown's configurations, click the gear icon next to its variable name.

 

## Date-time Inputs

If you want to use date-time inputs in your notebooks, you can now do so by clicking `Inputs > Date` when adding a new block.

![](/posts/changelog-8/dates.gif)

Every date input has a two pieces of configuration:

1. Whether the input is a date or a date-time field.
2. The timezone that the input should be displayed in.

After you've configured the input, you can use it in your code by referencing the variable name you've given it, just like the other inputs.

When referencing the input in your Python code, it will be a `datetime` object, which you can use to filter dataframes, create new columns, or perform any other operation you'd like.

In SQL, you can interpolate the input using the `{{var_name}}` syntax, which will be replaced by the input's value when the query is executed.

 

## Writebacks

Before writebacks, our users had to write Python code to push data back to their databases. This was cumbersome and error-prone, so we decided to make it easier.

Writebacks are a low-code way to write data back to your database. You can use them to update tables, create new ones, or even delete data.

We recommend using writebacks in conjunction with the scheduling feature to create data pipelines that run automatically. These writeback blocks are also great for testing and experimenting with data before committing to a final solution.

![](/posts/changelog-8/writebacks.gif)

To use writebacks, you need to select the `Writeback` block from the block list. Then, you can select which dataframe you want to write back to your database, and configure the writeback settings, like whether you want to append or replace the data, and what to do when there are conflicts.

Please notice that writeback blocks will only appear for users with a Postgres or BigQuery database connected to their workspace. Writing to other databases is not supported at the moment.

 

## Math typesetting (LaTeX)

You can now use LaTeX to write math formulas in your notebooks. This is great for writing scientific papers, creating educational content, or just showing off your math skills.

![](/posts/changelog-8/math.gif)

To use LaTeX, you need to wrap your math formulas in `$` symbols. For example, `$\int_0^\infty e^{-x^2} dx$`.

 

## Performance Improvements

I don't want to dwell too much on this, but we've made several performance improvements to the platform to accommodate the huge number of sign ups we've had in the past few weeks.

Those performance improvements include better client and server-side caching, as well as optimizations to the way our CRDT libraries write data.

We also fixed leaks which would cause long-running tabs to consume more memory than necessary.

 

## General Improvements and Bug Fixes

* We've improved the way data sources pings are handled, which should make the platform more reliable.
* Whenever data source pings fail, we now show a warning message in the UI.
* We fixed a bug in the data source creation which would cause it to create multiple data sources with the same name.
* We've added 13GiB of example data to a demo database that gets attached to every workspace. That should help new users get started with Briefer.
* PDF exports now won't have page breaks. That prevents visualizations from being split across pages.
* Fixed a bug where SQL blocks would not show data sources added after the block was created.
* Fixed plenty of visual bugs in the dashboards.
* Minor aesthetic improvements to dashboard graph displays.

Changelog #8: Publishing, writebacks, date inputs, and dynamic dropdowns

At Briefer, we are building multiplayer notebooks. That means many users can edit and run blocks **in the same notebook at the same time**.

![](/posts/multiplayer-notebooks/multiplayer.gif)

In this post, I'll explain how we got to multiplayer notebooks that actually work.

First, I'll describe Jupyter Notebook's architecture, and how we built a single-player `v0` on top of it.

Then, I'll explain what CRDTs are, how they handle conflicts and collaborative editing, and how we used them to build a multiplayer `v1` that work *reasonably* well.

Next, I'll walk you through the problems we had with `v1`, and show how we solved them in `v2` by communicating changes directly through CRDTs.

Finally, I'll share a few lingering issues we have with `v2` and what we're planning to do next.

 

## v0: Building a notebook

Most people think Jupyter Notebooks are monolithic, but they're actually a collection of three major components that work together: the [Jupyter Notebook *client*](https://github.com/jupyter/notebook), the [Jupyter Server](https://github.com/jupyter-server/jupyter_server), and the [IPython kernel](https://ipython.org/).

When most people talk about Jupyter Notebooks, they're referring to the Jupyter Notebook *client*. That client is a JavaScript application that talks to a Python application called the Jupyter *Server* over a WebSocket.

Whenever you run a block of code in a Jupyter Notebook, your browser writes to the WebSocket from which the Jupyter Server is reading. The Jupyter Server then responds by writing the output back to the WebSocket, which the client reads and renders.

![](/posts/multiplayer-notebooks/jupyter-nb-websocket.png)

Still, the Jupyter Server does *not* run the code itself. Instead, it sends the code to an IPython kernel through a [ZeroMQ](https://zeromq.org/) socket using the [Jupyter messaging protocol](https://jupyter-client.readthedocs.io/en/stable/messaging.html).

It's that IPython kernel process that *actually* runs the code. Then, it sends the output back to the Jupyter Server, which, in turn, sends it back to the Jupyter Notebook client. If necessary, the server can also write to and read from the notebook's `.ipynb` file.

![](/posts/multiplayer-notebooks/jupyter-nb-overview.png)

This decoupled architecture makes it easier to build new types of notebooks without reinventing the wheel. Instead of reimplementing everything from scratch, people can just build clients that talk to the Jupyter Server and render outputs.

For example, if you like pain and text-based interfaces as much as I do, you could use the [Qt console](https://qtconsole.readthedocs.io/en/stable/) and earn some extra street cred (or lose some, depending on who you ask).

![](/posts/multiplayer-notebooks/qtconsole.png)

In the same way, you could build a better notebook client that works in the cloud. You could, for example, wrap SQL code in a Python function and offer native SQL support in your notebooks. That's what we did at Briefer.

Our `v0` was a single-page app that talked to our API which then relayed commands to the Jupyter Notebook server for that user's workspace. For each notebook in a workspace, the Jupyter Notebook server would maintain a separate IPython session.

In addition to that application, we also had a database where the notebooks' contents were stored.

![](/posts/multiplayer-notebooks/briefer-overview.png)

Whenever users opened a notebook, the Briefer client sent a `GET` request to the backend and render the results. As users edited the notebooks, we'd write the changes to a websocket to keep the backend in sync.

Finally, whenever someone wanted to run a block of code, they'd send a `POST` request to the backend, which would run the code and return the output. Once the client received the output, it would write it to the notebook and flush the result back to the server.

![](/posts/multiplayer-notebooks/briefer-flow.png)

This architecture worked great, except for one thing: only one person could edit the notebook at a time. Whenever two people tried to edit the same notebook, the last person to save their changes would overwrite the other person's changes.

Let's say users A (latency = 50ms) and B (latency = 250ms) wanted to edit a `print("Hello world!")` block, for example. In that case, if user A changed it to `print("Hello Mars!")` and user B simultaneously changed it to `print("Hello world! Existence is pain")`, the end-result would be `print("Hello world! Existence is pain")` because the server would receive user B's changes last.

![](/posts/multiplayer-notebooks/users-overwrite.png)

At the time, we solved that problem by locking the notebook whenever someone started editing it. That way, only one person could edit the notebook at a time.

It was a dirty solution, but it helped us ship `v0` in only a few days.

![](/posts/multiplayer-notebooks/users-lock.png)

In a way, this is similar to how a typical mutex works in a multi-threaded application. Whenever a thread wants to access a shared resource, it locks the mutex. If another thread tries to access the same resource, it has to wait until the first thread releases the mutex.

Even though it was a suboptimal solution, it worked well enough for our first few users. These users just needed a cloud environment to write and run code in, and other quality of life features, like native SQL support, automatically turning query results into Pandas DataFrames, and a prettier interface to make their work more presentable to non-technical stakeholders.

That suboptimal solution also bought us time to work on a proper `v1` with multiplayer support as users tested `v0` and gave us feedback on a bunch of other things.

 

## v1: Multiplayer notebooks that worked *reasonably* well

Let's start by exploring a conceptual solution to the multiplayer problem. Then, we'll dive into how we implemented it in `v1`.

 

### A conceptual solution to the multiplayer problem

If we wanted notebooks to be truly multiplayer, we'd have to throw away our mutex-like approach and allow multiple people to edit the same piece of state at the same time.

The problem with simultaneous edits is that conflicts may happen. If two people edit the same piece of state at the same time, someone or something (preferrably) has to decide what the final state should be.

One way of solving this issue would be to break our notebooks' state into blocks. Then, instead of saving the entire notebook every time someone made a change, we saved the block that was edited only.

That way, multiple people could edit *different* blocks at the same time without conflicts. Additionally, we could sync each block's changes in real-time to everyone editing the notebook.

![](/posts/multiplayer-notebooks/users-block-lock.png)

The problem with that approach was that it still wasn't possible to edit the same block at the same time. If I edited the notebook's first block and you edited the second, we wouldn't have any conflicts. However, a third person wouldn't be able to edit either block until we were done.

In a way, breaking the notebook's state into individual blocks simply meant that we moved our "mutex" from the notebook to the block level.

This is where CRDTs come in.

CRDT stands for Conflict-free Replicated Data Type. It's a type of data structure that is capable of *automatically* merging changes made by multiple users to the same piece of state.

Take that `print("Hello world!")` example from before. If user A changes it to `print("Hello Mars!")` and user B changes it to `print("Hello world! Existence is pain")`, the CRDT merges the changes automatically into `print("Hello Mars! Existence is pain")`.

It does so because it knows that replacing `world` with `Mars` happened right after `Hello ` and before `!`, while adding `Existence is pain` happened after `!`. That way, it can apply both changes in order and merge them automatically.

![](/posts/multiplayer-notebooks/users-crdt.png)

This approach is similar to what Git does when people edit different parts of the same file. If you change the first line of code and I change the last, Git will merge the changes automatically because its algorithm is programmed to do so.

![](/posts/multiplayer-notebooks/git-auto-merge.png)

The difference between CRDTs and Git is that CRDTs can merge changes automatically even if they happen in the same line or in the same position within a line.

**This way, if we turned each block's state into a CRDT, multiple people could edit the same block and the CRDT would merge the changes automatically, regardless of where they happened.**

Finally, one last thing to notice about CRDTs as they do not care whether a merge's results are correct. They simply merge the changes automatically and ensure that the final state is consistent across all users.

Consequently, it's possible for two users editing the same block to end up with a block that doesn't make sense. For example, if user A changes `print("Hello world!")` to `print("Hello Mars!")` and user B simultaneously changes it to `print("Hello Venus!")`, the CRDT would merge the changes into `print("Hello MarsVenus!")`.

![](/posts/multiplayer-notebooks/users-crdt-simultaneous.png)

Still, that conflict-resolution mechanism is okay for our notebooks because people see each other's changes in real-time. That way, they can avoid getting in the way of each other.

Furthermore, it's unlikely for two people to edit the same position within block at the same time. In comparison, Git can't tolerate these types of merges because people change files in isolation and only see other people's changes when they `git pull`.

That's all to say that CRDTs are *not* a good fit for *every* type of application, but they're a great fit for real-time collaborative applications like our notebooks.

 

### v1 architecture

Now that we have a conceptual solution to the multiplayer problem, I'll dive into how we implemented it in `v1`.

At Briefer, we're huge nerds, but not *that* much. We didn't want to roll our own CRDT implementations. Instead, we had to find an implementation that worked-well and that was easy to integrate with the editor libraries we using in our text and code blocks, [TipTap](https://www.tiptap.dev/) and [Monaco](https://microsoft.github.io/monaco-editor/).

Ideally, whichever library we chose would also be straightforward to integrate with our backend because we didn't want to write our own custom protocol either.

Considering these criteria, the obvious choice was to go with [Yjs](https://github.com/yjs/yjs).

Yjs is a CRDT library written in JavaScript that implements a variety of data structures, like arrays, maps, and text. Each of these data structures is capable of handling conflicts automatically, as I described above.

Additionally, both TipTap and Monaco have built-in support for Yjs, and the Yjs team also maintains `y-websocket`, a WebSocket provider that makes it easy to sync Yjs data between multiple clients and a server.

We then wired up Yjs to our frontend and backend.

First, we moved over all notebook's state to a `Y.Doc` object containing two-top level keys: a `layout` array and a `blocks` map. The `layout` array references the blocks in the order they appear in the notebook, while the `blocks` map contains the blocks' content indexed by their IDs.

In each block, we also include at least one `Y.XmlFragment` that we pass to TipTap or Monaco. It's that `Y.XmlFragment` that these libraries write to and read from.

```Typescript
// This is a simplified example
type Block = Y.XmlElement<{
 id: string;
 title: string;
 type: "sql" | "python" | "text";
 content: Y.XmlFragment;
 result: Y.XmlFragment;
}>;

type BrieferNotebook = Y.Doc<{
 blocks: Y.Map<Block>;
 // In reality, layout is an array of arrays
 // because a single layout row can have multiple
 // tabs (one for each block)
 layout: Y.Array<string>;
};
```

Finally, we used `y-websocket` to sync the `Y.Doc` object between the clients and the server. That way, whenever someone made a change to the notebook, Yjs would automatically merge the changes and sync them to everyone editing the notebook.

![](/posts/multiplayer-notebooks/client-vs-server.png)

This architecture worked *reasonably* well, except for one thing: the way we handled block executions.

Whenever you ran a block of code in `v1`, the client would send a `POST` request to the backend, which would run the code and return the output. Once the client received the output, it would write it to the `Y.Doc` object and flush the result back to the server.

![](/posts/multiplayer-notebooks/v1-flow.png)

The problem with this approach is that it led to race conditions. If two people ran the same block at the same time, the last person to save their output would overwrite everyone else's output, even if they ran the block first.

Let's say user A and user B had a block that incremented a counter by 1 every time it ran. If user A ran the block and user B ran it right after, the server would run the block twice (correctly). But then if user A's response (`1`) arrived later than user B's (`2`), the server would overwrite user B's response with user A's (an incorrect `1`).

This problem gets even worse when you realize that both clients will display `1` as the block's output, even though the Python runtime state is `2`.

![](/posts/multiplayer-notebooks/incorrect-state.png)

Additionally, people didn't know whether someone else was running a block because the "loading" state was only visible to the person who ran the block.

Sometimes, people would also close their browsers before the block finished executing. In that case, the code execution would alter the notebook's state, but the client wouldn't show it on the UI because it never got the response back and thus didn't update the `Y.Doc` object.

 

## v2: When we finally got it right

In `v2`, we finally cracked multiplayer notebooks.

The way we did that was by moving all the communication between the client and the server to the `Y.Doc` object.

Instead of sending `POST` requests to the backend, the front-end would change the `Y.Doc`'s state and the back-end would react accordingly.

Assume that you want to run a Python block, for example.

In `v1`, your browser would send a `POST` request to the backend, which would run the code and return the output. Once your browser received the output, it would write it to the `Y.Doc` object and flush the result back to the server.

In `v2`, your browser would change the block's state to `run-requested`. Then, the back-end would change the block's state to `running` and kick off the code execution. Once the code execution finished, the back-end itself would change the block's state back to `idle` and **write the output directly to the `Y.Doc` object**, causing all connected clients to see the output in real-time.

![](/posts/multiplayer-notebooks/v2-flow.png)

This approach eliminates race conditions because the updates to the `Y.Doc` are processed in order, as well as IPython executions. Here's what happens when users A and B try to run the same `increment_counter` block at the exact same time:

1. Both clients set the block's state to `run-requested`.
2. The server processes whichever `run-requested` event arrives first (let's say A's)
 2.1 The server tells IPython to run the code
 2.2 The server changes the block's state to `running`
3. The server processes the second `run-requested` event (B's)
 3.1 The server tells IPython to run the code
 3.2 The server changes the block's state to `running`
 4 . IPython finishes running the code for A (IPython only runs one piece of code at a time)
 4.1 The server writes the output to the `Y.Doc` object (`1`)
4. IPython finishes running the code for B
 5.1 The server writes the output to the `Y.Doc` object (`2`)

This way, the front-end would always be in sync with the back-end. If you ran a block and closed your browser before the execution finished, the back-end would still write the output to the `Y.Doc` object and sync it to everyone editing the notebook.

Additionally, if you and I were editing the same notebook and you ran a block, I'd see the block's state change to `running` because that's dictated by the `Y.Doc` object, not by the local browser state.

Although this approach works great in the vast majority of cases, it still has one issue: it's not resilient to back-end failures.

If you run a block and the back-end crashes *during* the execution, the block's state will be stuck in `running` forever. That's because the back-end mistakenly believes the block is still running, when it has actually completed; however, the crash prevents it from receiving confirmation from the Jupyter Server to transition the block's state back to idle.

![](/posts/multiplayer-notebooks/limbo-updates.png)

Although the back-end rarely ever crashes, it's still possible that we have to restart it when deploying new versions or even just scaling up the infrastructure.

For now, we solved this issue by timeouting the block's state and observing Jupyter Server's state, but it's a patchy solution that we're not entirely happy with.

![](/posts/multiplayer-notebooks/recovered-block-state.png)

 

## v3: What comes next

In `v3`, we're planning on implementing a queue system to handle block executions.

That way, if the server crashes during the execution, the block won't get stuck in `running` forever. Instead, once the server comes back up, it will continue executing the block from the queue.

We don't yet know where this queue will live, but we're considering adding it to the `Y.Doc` object itself. That way, the queue will always be in sync with the notebook's state and any worker assigned to that document can pick up the next block to execute.

If you have ideas, suggestions, or feedback, please reach out to us at <founders@briefer.cloud>.

 

## Try these multiplayer notebooks yourself

If you want to try these multiplayer notebooks yourself, you can sign up for Briefer at [briefer.cloud](https://briefer.cloud).

Briefer is free, and we won't ask you for a credit card. We're also not going to send you spammy emails with generic PDF-ebooks, and our sales team definitely won't call you to ask if you want to upgrade to a paid plan (I *am* the sales team, so I can guarantee that).

Worst case scenario I'll ask you for feedback, but that's about it.

Stay hacky,

LdC

At Briefer, we are building multiplayer notebooks. That means many users can edit and run blocks in the same notebook at the same time. In this post, I'll explain how we got to multiplayer notebooks that actually work.

Notebooks and CRDTs

People usually ask me what's wrong with Jupyter Notebooks and why we're building a tool to replace them. They also ask us how we're making notebooks better.

I usually repeat the same things over and over, so I thought I'd write them down here so I can link to it.

 

## 1. Too messy too quickly

The first thing I don't like about notebooks is that they get too messy too quickly. One morning you have 10 blocks, and by the end of the day you have 192, but only six of them are meant to be seen by other people and the rest is just you jiu-jitsuing with the data.

We solved that problem by doing three things.

First, we allowed people to group their blocks into tabs. This way, you can have an "API Call" tab to fetch data, a "Data Cleaning" tab to clean it, and a "Visualization" tab to plot it.

![](/posts/improving-notebooks/tabs.gif)

Then, we created the concept of "publishing" a notebook and added buttons for people to hide certain blocks from the published version.

![](/posts/improving-notebooks/edit-vs-publish-blocks.png)

Finally, we realized that vertical scrolling is not always the best way to navigate a notebook. It wastes a lot of screen real estate and makes it hard to see the "big picture" of what's going on.

We fixed that by adding a "dashboard" view with a grid in which you can drag and drop blocks.

![](/posts/improving-notebooks/notebooks-to-dashboards.gif)

Today, most published notebooks use the dashboard view instead of the traditional vertical scrolling.

 

## 2. Difficult to find and explore

The second thing I don't like about notebooks is that they go to GitHub and then die because no one ever opens them again.

I wanted notebooks to be "live" artifacts that everyone can find, open, and explore, including non-technical people.

To solve that problem we created a Notion-like file tree.

![](/posts/improving-notebooks/the-file-tree.gif)

The file tree makes notebooks more approachable because it allows you to create top-level notebooks like "Sales Team" and put all your sales-related notebooks inside it, like sales forecasts and customer segmentation.

I know the icon selection is a bit cheesy, but it makes things more approachable for non-technical people. Once we launched this feature, people complained that it included 288 icons "only" and that they wanted more, so we added an extra 1487. Today, most notebooks have a custom icon.

 

## 3. Too much friction for querying databases

The third thing I don't like about notebooks is that they require a lot of boilerplate code to query databases.

We fixed that by adding a "query" block that allows you to write SQL and see the results in a table.

![](/posts/improving-notebooks/sql.png)

We also know that queries are useless unless you can wrangle the data, so each query's results are automatically converted into a pandas dataframe.

![](/posts/improving-notebooks/sql-to-dataframe.png)

Then we went full circle and allowed people to write SQL queries against their pandas dataframes. We did that because sometimes the declarative style of SQL makes it a better choice than Python.

![](/posts/improving-notebooks/select-from-dataframe.png)

 

## 4. Not collaborative

The fourth thing I don't like about notebooks is that I can't edit them at the same time as someone else. I also don't like it that I have to push my changes to GitHub and then someone else has to pull these changes to see what I'm doing.

We fixed that by using [Yjs](https://yjs.dev/) to store the notebook's state.

Whenever you create a block, delete a block, or move a block, we send that information to Yjs, and then Yjs sends it to everyone else who's viewing the notebook.

If two people try to edit the same block at the same time, Yjs knows how to merge their changes, so no one loses their work.

 

## 5. Outdated results

The fifth thing I don't like about notebooks is that they get stale, and you have to re-run them manually to get the latest results.

Let's say you have a sales forecast notebook built on top of the latest CRM data. In that case, Jamie from the sales team will probably ask you to re-run the notebook every week or so to get the latest forecast.

We fixed that by adding a "schedule" button that allows you to run a notebook at a specific time, like every Monday at 9 am.

![](/posts/improving-notebooks/scheduled-runs.png)

If the notebook fails to run, we can send you an email or a Slack message. Otherwise, we can send you the complete notebook in PDF format on those same channels.

I love the notifications feature, but some people overuse it. They schedule their notebooks to run every few minutes and then get mad when they get too many notifications. We don't know how to solve that problem yet.

 

## 6. Not interactive

The sixth thing I don't like about notebooks is that they're not interactive.

We fixed that by adding a few native text fields and dropdown blocks that allow you to ask for user input.

![](/posts/improving-notebooks/normal-inputs.png)

We know that users sometimes want their inputs to be dynamic, like when they want to use values from their databases as options in a dropdown, so we did that too.

![](/posts/improving-notebooks/dynamic-inputs.png)

We do know about `ipywidgets`, but we don't like them because they get lost among the other blocks, and their styling is not consistent with the rest of the notebook.

 

## 7. Friction to plot graphs

The seventh (and final) thing I don't like about notebooks is that plotting simple graphs requires more code than it should.

Whenever I need a simple line chart, I have to write a few lines of Python code, and then I have to do all the styling manually. Otherwise, the chart looks like it was made in Excel.

Additionally, I keep having to look up the documentation to remember how to plot anything other than basic charts.

We solved that by adding "visualization blocks" that allow people to plot data by clicking buttons instead.

![](/posts/improving-notebooks/vis.png)

These blocks look good by default (at least that's what we think) and you can use them to plot data from any dataframe.

Some people complained about having to filter data using Python before plotting it, so we added a "filter" button to the visualization blocks too.

![](/posts/improving-notebooks/vis-filters.png)

Still, sometimes people need to use Python to plot more complex graphs, like when they want to plot a sankey diagram or a network graph. In that case, they can click "edit with AI" and ask the LLM to plot the graph for them. That way, they don't have to read the documentation every time they need to plot something new.

![](/posts/improving-notebooks/ai-sankey.png)

People usually ask me what's wrong with Jupyter Notebooks and why we're building a tool to replace them. I usually repeat the same things over and over, so I thought I'd write them down here.

Making notebooks better

I used to love two things about computers. The first was that they did exactly what you tell them to do. The second was that they did the exact same thing every time.

Humans shattered that magic. We now have AI assistants who do what you tell them to do, but they also do a lot of other things that you didn't tell them to do, and they do it differently every time.

When I started implementing AI assistants in our notebooks, I hated that. At times, the assistant would perfectly execute complex tasks like developing a churn prediction model but then stumble over something as simple as plotting a revenue line chart from a CSV of financial data.

At first, I thought I'd have to do something fancier than just RAG and "prompt engineering" to get the AI assistant to work. Then, I realized that I could get better results with a few UI tweaks, and this is what this blog post is about.

![](/posts/code-gen-for-notebooks/ai-assistant-15fps.gif)

 

## Setting good examples

The most useful mental model for implementing AI assistants is to think of yourself as a manager onboarding a new senior employee.

You wouldn't expect the new senior employee to know everything about your business on their first day, but you would expect them to be smart and capable of learning quickly. In other words, you assume zero knowledge and infinite intelligence.

I like that approach, and that's what most people do. Let's say they're building an assistant to help fifth-graders with their homework. In that case, most people would just do some kind of RAG over fifth-grade textbooks, and that would be it.

In our case, we did something similar. Initially, we fed the LLM with the content of the current SQL or Python block, the user's database schema, and a list of data frames, their columns, and their types.

That approach worked well, but one common complaint was that the LLM's output didn't follow the user's coding standards. For example, the user's previous code used `plotly` but the LLM used `matplotlib`. Or the user's code used a specific color palette, but the LLM would just default to the standard one.

To fix that, we started including a few code snippets from the notebook so that the AI assistant could see whether we were using `matplotlib` or `plotly`, and what colors users prefer.

Oddly enough, one of the most impactful changes we made was to include the document's title and the content of relevant text blocks if we had any. That way, the AI assistant could see what the user was trying to do, and suggest something that made sense in that context.

![](/posts/code-gen-for-notebooks/ai-context-examples.png)

In a way, this approach mimics what a senior engineer does when they're submitting their first few pull requests. They look at the existing codebase, see what the team is doing, and try to follow the same patterns.

 

## Encouraging small steps

Pull request sizes are usually inversely proportional to the wrinkles on a programmer's forehead. The more seasoned the programmer, the smaller the pull request.

That's because seasoned programmers know that the more code they write before getting feedback, the more code they'll have to erase if something is wrong.

Additionally, working on small steps at a time makes tasks easier to wrap your head around, and easier to debug.

The same principles apply to AI assistants. You should encourage users to give the AI assistant small tasks at a time and ask the user for feedback on each task before the assistant moves on to the next one.

In our case, we used to have an input for users to ask the AI to perform an *entire* analysis on their behalf. After a month or so, we removed it because we found that tasks got too large for the AI to handle at once, and too intricate for users to describe accurately.

![](/posts/code-gen-for-notebooks/ai-user-flow.png)

Now, we include AI assistants in individual blocks. That way, users end up asking the AI to do a single step at a time.

Consequently, user's prompts usually get more accurate, because they don't have as many intricacies to describe and thus tend to devote more time to crafting a better prompt.

A further benefit of encouraging small steps and getting feedback on each is that the user gets to vet each suggestion before moving on to the next one. That way, the assistant's corpus gets more accurate over time, and the user gets more comfortable with the assistant's suggestions.

 

## Optimizing for feedback

When asking users to take small steps, you must make each step as fast and pleasant as possible. Otherwise, users will get frustrated and give up.

In our case, for example, we used to have the AI replace the user's code with its suggestions, and we'd do it all at once. That was a terrible idea because users couldn't tell whether the AI was working until it was done, and they couldn't see what the AI was doing until it was too late.

To fix that, we did three things.

First, we started streaming the AI's response instead of emitting it all at once. That way, users could see the AI's suggestions as they were being generated, and they could stop the AI at any time if they didn't like what they were seeing.

After that, we started showing the user a diff of what the AI assistant suggests. Then, they could accept or reject the changes. That way, users could clearly see what the AI assistant was suggesting versus what they wrote.

Finally, we added a button that allowed users to try the AI assistant's suggestion without accepting it. That way, users could try the suggestions before accepting them, and they could also modify the diff in case something was slightly off.

![](/posts/code-gen-for-notebooks/accept-reject-try-buttons.png)

Something else we're working on is to allow users to prompt the AI assistant's suggestions without having to accept them.

In addition to making each step more seamless, these diffs also prevent users from running undesirable code as it puts on them the burden of reviewing each change.

 

## Encouraging action

Clicking a button is quicker than writing a prompt, and it's also more intuitive. Consequently, you should consider adding a "quick trigger" button whenever there's an action that the assistant can take without the user having to prompt it.

In our case, errors are a prime example of that.

Whenever someone writes invalid SQL or makes a typo in their Python code, the error that comes up usually has all the necessary information for the AI assistant to fix it. Therefore, there's no need to ask the user to prompt the AI assistant to fix it.

Instead, we show a button that says "Fix with AI". When the user clicks it, we feed the LLM with the error message and a bit of context, and we ask it to try to fix it.

![](/posts/code-gen-for-notebooks/fix-with-ai.gif)

These types of scenarios are more common than you'd think, and they're a great opportunity to show users the value of the AI assistant. People will trigger the assistant more often if they don't have to think about what to do with it.

 

## Models usually take care of themselves

Finally, one last piece of advice is to assume that new LLMs will only get better, faster, and cheaper over time.

Take GPT-4o, for example. When it was released, it solved quite a few problems we had with the previous versions. All of a sudden, our default assistant got way faster and more accurate.

The trick is that we didn't have to do anything to make that happen. Had we spent months fine-tuning models and squeezing every last drop of performance out of them, we would've wasted a lot of time.

Instead, we focused on making the user's experience as good as possible, and we let the LLM's performance take care of itself.

Then, when a new model came along, we could just plug it in and get better results without having to change anything else.

In comparison, everyone else who spent months working on their models had an assistant that was just as good as ours, but without all the UI improvements we had made.

 

## Putting it all together

Perceived performance is the only performance that actually matters. If users *perceive* your assistant to the better, it doesn't matter whether its underlying model is the one with the highest scores on standard benchmarks.

In a way, making AI assistants better is as much about improving the models as it is about improving the user's experience when using them.

Sure, there's a lot of value in creating better models, but there's a lot more innovation you can do if you focus on the specific problem you're trying to solve.

Consequently, if you're adding AI features to your product, you should probably start with the low-hanging fruit: UI/UX improvements.

Here's what we found helpful:

1. **Set good examples**: instead of including only raw data, augment generation with user-generated examples and specific context.
2. **Encourage small steps**: ask users to prompt the AI assistant on small tasks at a time, and ask for feedback on each task before moving on to the next one. Then, use each step's result to improve future suggestions.
3. **Optimize for feedback**: make each step as fast and pleasant as possible, and make it easy for the user to adjust the assistant's suggestions.
4. **Encourage action**: add a "quick trigger" button whenever there's an action that the assistant can take without the user having to prompt it.

Improving your user's experience with AI assistants without messing with the models.

Copilot UIs

In the past few weeks, we've got plenty of requests for "quality of life" features, so we delivered them.

This week, we're glad to show you how to use new dashboard features, connect MySQL databases, and query uploaded files using plain SQL.

Many people asked for context-aware autocomplete in SQL and Python blocks, so we shipped that too.

 

## Querying files

You can now write SQL queries against files and dataframes. This means you can upload a CSV file and query it using SQL, for example.

To do that, add a "File Upload" block, drag your files into the block, and then click the "query" button to generate an example `SELECT * FROM` query.

Feel free to change that query to do whatever you want with the data in the file.

![](/posts/changelog-7/query-files.gif)

In addition to querying files, you can also query any pandas dataframes, as you can see in the example below.

![](/posts/changelog-7/query-dataframes.png)

As a general note, you should select the `Files (DuckDB)` data source whenever you want to query anything other than a database.

 

## Better dashboards

We did tons of changes to the aesthetics of dashboards, but we're not done yet.

In the past few weeks, we cleaned up the visualization display, added support for data labels, and also improved the way all blocks look in the dashboard.

Additionally, blocks now show their titles on the top left corner, and the sidebar has been completely revamped so all blocks look nice within it, and so you can see their titles. That way, it's easy to identify what each block is doing at a glance.

![](/posts/changelog-7/dashboards-tour.gif)

 

### Dashboard headings

Headings can help you organize your dashboard's content. You can add a headding to any block by clicking the "Add Heading" button on the top right corner of the block list.

These headings will only appear in the dashboard view, not in the notebook.

![](/posts/changelog-7/dashboard-headings.gif)

 

### New visualization types

In addition to the aesthetics, we've also added a few new visualization types, such as "trend" and "numbers".

The "trend" visualization is great for showing trends over time, while the "numbers" visualization is perfect for highlighting a single metric in a large font.

![](/posts/changelog-7/numbers-and-trends.png)

 

## Autocomplete

We've added autocomplete to SQL and Python blocks. This means you can now get suggestions for column names, table names, variable names, and functions while writing your queries.

![](/posts/changelog-7/autocomplete.png)

Please note that for the SQL autocomplete to work, the permissions of the user you're using to connect to the database must allow for querying the information schema.

 

## MySQL support

We've added support for MySQL databases. You can now connect to a MySQL database and query it using SQL blocks.

 

## General Improvements and Bug Fixes

* It's now possible to undo AI suggestions and try them before accepting the diff.
* There's a new onboarding document that will help you get started with Briefer when creating new workspaces.
* Visualizations now support multiple Y-axes.
* Document's titles will now appear in the browser's tab.
* We fixed a bug in which the line chart would offset ticks in the X-axis.
* We fixed a bug in which the "in/not in" filters would not work for dataframes.
* We fixed a bug in which the blocks would be repositioned when opening the sidebar.
* We fixed a bug in which the progress indication would show incorrect values when running entire notebooks.
* We fixed a bug where invalid visualizations would fail silently and not show any error messages.

In the past few weeks, we've got plenty of requests for 'quality of life' features, so we delivered them. This week, we're glad to show you how to use new dashboard features, connect MySQL databases, and query uploaded files using plain SQL.

Changelog #7: Querying files and dataframes, better dashboards, autocomplete, and MySQL support

Sales pitches are the only place where "self-serve dashboards" work. In the real world, it's a different story.

That story usually starts with an engineer or data scientist who's frustrated because they spend too much time writing queries and preparing dashboards for business people. They think that if they make BI easy enough, everyone will be able to "self-serve", but that rarely ever happens.

What actually happens is that engineers and data scientists end up being the only people using the "self-serve" BI tool, which, ironically, makes it not self-serve at all.

 

## Why self-serve BI doesn't work

**SQL is the only self-serve BI tool**. Still, most "self-serve" BI vendors don't want to admit it, so they find ways to reinvent the wheel and disguise SQL as something else. Maybe that's what customers want to hear, or maybe they genuinely believe it, I'm not sure.

Either way, the problem with most approaches to "self-serve" BI is that they consider SQL to be the only barrier to business stakeholders querying data. That's not true.

Let's say you could wave a magic wand and make writing SQL queries as easy as writing an email. Do you think business stakeholders would suddenly start querying data? The answer is no.

Even if these people knew how to write SQL queries, they wouldn't understand the semantics of the data they're querying. They wouldn't know what the data means, where it comes from, or how it's calculated. They wouldn't know how to interpret the results, or how to validate them. In other words, writing the SQL query is actually the easy part.

Now, let's look at two attempts to make BI "self-serve" and see whether the hypothesis above holds.

 

### Attempt 1: The conventional "dropdowns and checkboxes" approach

First, let's take a look at an example of a conventional "self-serve" BI interface which uses dropdowns and checkboxes as a way to make queries more accessible.

![](/posts/self-serve-bi/sql-by-mouse.png)

The first thing to notice is that this interface is just an attempt at what I call "SQL-by-mouse". I might be just an old grumpy person, but I don't see how this is any better than writing SQL. In fact, it's worse because it's slower, less reliable, more limited, and not generalizable to other tools.

In any case, I'll concede one thing: some non-technical people are just scared of any monospaced writing with syntax highlight, so this interface might be a good first step.

Anyway, let's ignore all these problems and just assume that this interface is the best thing since [htmx](https://htmx.org/). Let's assume that it's 100x easier to use than SQL.

Even if that were the case, your CFO, for example, wouldn't just go ahead and use these dropdowns and checkboxes to query data. The reason they wouldn't use it is that they don't have the context to understand the data they're querying, and they'll probably be insecure about the results, which they won't know how to validate because they don't understand how the data got there in the first place.

Now, I've got a question for you: if your CFO doesn't use this interface, who will? That's right, *you* will. And guess what's the first thing you'll do when you get to these dropdowns? Correct, you'll immediately look for somewhere to type in SQL. Oh, the irony.

 

### Attempt 2: The text-to-SQL approach

Remember that magic wand that makes writing SQL queries as easy as writing an email? That's what text-to-SQL tools are. Nonetheless, they're not enough, just as I explained earlier.

Again, the problem is not the technology itself. In fact, LLMs are almost too effective at translating natural language into SQL. They will find a way to generate a query for any question you ask, even if that question doesn't make sense, which is exactly the problem.

On the other hand, a technical person would notice that the question doesn't make sense, and they would ask for more context. They would ask for details about the business person's hypothesis and the problem at hand. Then, they would explain what type of data is available, and work with the business person to formulate a precise and useful question.

Again, SQL is not the problem.

In any case, I do think LLMs could be the actual solution to self-serve BI, but not in its current form. For them to work, they'd need to be fed with more context, and [they need to get better at expressing uncertainty](https://allenpike.com/2024/llms-trained-on-internet) and asking for more information.

 

## What actually works

If we assume that the problem with self-serve BI is not SQL, but the context and semantics of the data, then it follows that the solution is to teach people about the data they're querying, regardless of interface.

The problem with this type of training is that it takes time, and most business stakeholders need answers this week, not next quarter.

Even in the best-case scenario, where companies train business stakeholders to understand the data they're querying, these people will need time to keep up with the changes in the database's schema, data models, and ETL processes.

Additionally, documenting all this knowledge generates significant overhead for the technical team, and it quickly gets out-of-date.

So what's the true solution to self-serve BI? The answer is simple: not to make BI self-serve for non-technical people. Instead, the solution is to make technical people support business stakeholders and help them do it more efficiently, using better tools.

The definition of a better tool may vary, but I have a few suggestions.

The first suggestion is to give LLMs to technical people, not business stakeholders. Even though text-to-SQL is not quite there in terms of understanding context and semantics, technical people *already* have context and semantics, so why not let *them* use it? That way, a single technical person can serve more business stakeholders, and do it faster.

The second suggestion is to give technical people more flexible tools. Instead of giving them dropdowns, or a SQL-only interface, let them play around with data using Python, R, or any other tool they're comfortable with. This way, they don't need to be passing data back and forth between different tools, and they're not limited by the capabilities of a BI tool.

The third and final suggestion is to make it easier for technical people to share their work. Notebooks and internal data applications are notoriously bad at that because they require others to deal with containers, dependencies, and infrastructure. It's a lot of overhead for a business stakeholder who just wants to see a chart. Instead, we need tools that have all the building blocks of a data application, but that are as easy to share as a Google Doc.

I'm obviously biased because I'm building [a data tool for technical people](https://briefer.cloud), and it does exactly these three things. But what other choice do I have? I'd rather build tools that work than tools that don't. And, most importantly, I'd rather not pretend that the problem is something it's not.

Sales pitches are the only place where self-serve BI works. In the real world, it's a different story.

Self-serve dashboards

Since the very first day we launched Briefer, we had a strong hunch that dashboards would be something people would want. We were right. We've been getting a lot of requests for this feature, and we're happy to announce that it's finally here.

Along with dashboards, we've also made some improvements to our visualizations and AI assistant. Here's what's new.

 

## Dashboards

You can now turn your notebooks into dashboards. Dashboards are a great way to keep track of your most important metrics and KPIs in one place. **You can add multiple visualizations to a dashboard, rearrange them, and resize them to fit your needs.**

At Briefer, we care a lot about empowering users to do whatever they want with their data, that's why our dashboards support Python outputs. **You can use Plotly, Seaborn, or any other Python library to create your visualizations and add them to your dashboards.**

 

![](/posts/changelog-6/dashboards-light.gif)

To create a dashboard, click the "Dashboards" button on the upper-right corner of the editor. Then, drag and drop the blocks you want to add to the dashboard.

You can also resize each block by dragging the bottom-right corner of the visualization.

Once you're done, click the "Publish" button to save your dashboard. You can access it at any time by clicking the "Dashboards" button on the upper-right corner of the editor.

Please notice that you will need to republish your dashboard every time you make changes to the blocks if you want to see them reflected in the dashboard.

 

## Better visualizations

To go along with the new dashboards feature, we've also made some improvements to our visualizations.

You can now rename each axis in your visualizations, format the value displays, and choose whether to display the labels or not.

We've also added support for histograms.

![](/posts/changelog-6/new-vis.gif)

 

## Better AI

Our SQL and Python AI assistant has also received some updates. It's now better at understanding your questions and providing relevant answers.

We've also had a few users ask for a way to change the foundational model of the AI assistant. Now, you can do that by clicking the "Settings" button on the lower-left corner of the editor, and then selecting the desired foundational model to use.

 

## General Improvements and Bug Fixes

* We now support Athena and Oracle databases as data sources.
* There's now a complete onboarding guide for new workspaces. It includes instructional videos and tips to help you get started.
* We've improved the Slack integration so that it's easier to set up, especially when it comes to channel selection.
* The schedules list now shows the notification settings for each scheduled run.
* We've updated the notification settings so that it can send PDF attachments containing your documents.
* Copying and pasting an image into a text block will now work as expected.
* There's a clearer indication of each user's tier right next to the logo on the top-left corner of the app.
* There's now a "Create visualization" button on the right of each SQL block. Clicking it will create a new visualization block with the results of the SQL block.

Besides those changes, we've done a variety of bug fixes and small performance improvements.

We recently launched dashboards, a feature many of you have been asking for. With that, we also improved our visualizations and AI assistant. Here's what's new.

Changelog #6: Dashboards, better visualizations, and better AI

The first lesson I learned about logarithms is not to mention them when speaking to a large crowd. All the other lessons sucked, so I thought I'd create my own.

Let's start by talking about the simplest of linear scales: a ruler.

Everyone understands how rulers work: you place them next to tiny trinkets and you look at the ruler's marks to determine the object's size. You can then use a pen and the ruler's marks to represent the size of these objects on a board.

![](/posts/logs/log1.png)

The problem with rulers is that you can't use them to measure larger objects, such as trees, park benches, and city blocks. To solve that problem, we'll need a bit of imagination.

Instead of having each of our ruler's marks represent 1 inch, we'll pretend they represent 100 inches. That way, we'll compress larger distances into smaller ones.

![](/posts/logs/log2.png)

Now, we can use the same ruler to represent the size of a park bench, a tree, and even a city block.

![](/posts/logs/log3.png)

Notice how we're not using the ruler for measuring anymore. Instead, we're using it to *represent* the sizes of different objects.

**The way we squished hundreds of inches into one mark on the ruler is called a linear transformation because when you draw it on a graph, it makes a straight line**. Each mark on the ruler represents 100 more inches, showing a steady, even increase.

![](/posts/logs/log4.png)

**Linear transformations preserve the measurements' relative spaces.**

For instance, if the marks for 1 and 5 inches are 4 segments apart on a ruler, then the marks for 100 and 500 inches will also be 4 segments apart when we use our scaled-up ruler. That way, the proportional distances remain the same, even though the actual measurements have increased.

![](/posts/logs/log5.png)

![](/posts/logs/log6.png)

**The problem with applying a linear transformation to a ruler is that it can only either measure small trinkets or large objects, not both.**

A ruler scaled from 0 to 10 inches isn't big enough to measure a park bench, while a ruler scaled from 0 to 900 inches makes it hard to distinguish the marks for smaller objects — they become too close together.

![](/posts/logs/log7.png)

To solve this problem, we'll need to transform our ruler so that each segment represents a greater increase.

Typically, going from the first mark to the second represents a 1-inch increase, just like there's a 1-inch increase when going from the second mark to the third. In other words, segments in normal rulers represent linear increases.

In our new scale, going from the first mark to the second represents a 10-inch increase, and going from the second to the third mark represents a 100-inch increase. Each subsequent segment represents a distance ten times greater than the previous. In this case, each segment represents a *non-linear* increase.

![](/posts/logs/log8.png)

This way, with just 10 marks, you can represent objects from 1 inch to 100,000,000 inches. Each step exponentially increases the range we can measure without cluttering our ruler with countless tiny increments.

Now, you can represent tiny stones at the first mark and a mountain just 10 segments further because, by the time you get to the 10th mark, the value this mark represents hasn't just grown tenfold, it's grown 10¹⁰ inches.

Here's another example using our new scale. If you plot a cockroach in the scale's first mark, a badger in the second mark, and a troll in the third, it means that the badger is 10x larger than the cockroach and that the troll is 10x larger than the badger.

Furthermore, it shows that the troll is 100x larger than the cockroach, because they're 2 segments apart, so 100 (10²) times larger.

![](/posts/logs/log9.png)

Going left has the opposite effect, it causes things to become increasingly smaller. If you keep going left from 1, you'll start working with fractions, and they get 10x as small as you move on to the leftward mark.

![](/posts/logs/log10.png)

As we've just demonstrated, this type of scale is great for representing ratios.

Now compare how the represented values increase exponentially as you move up marks. See how it's not a linear growth like in the previous transformation?

![](/posts/logs/log11.png)

As shown above, each mark represents an exponential increase in the value it represents. Still, this type of scale is *not* an exponential scale. It's a *logarithmic* scale.

If you were to plot the actual values against their corresponding logarithmic values, you'd notice that it resembles the plot of a logarithmic function. This resemblance occurs because, as the actual values increase, the increase in their logarithmic values becomes increasingly smaller.

![](/posts/logs/log12.png)

**An interesting effect of using the logarithmic scale is that small values get relatively more distant, while larger values get relatively closer.**

![](/posts/logs/log13.png)

The reason you should care about this change in relative distances is because it may cause unintended distortions, or, in the case of ill-intentioned people, *intended* distortions.

For example. Assume that a general council of alchemists doesn't want people to think there's a dragon fever pandemic. In that case, they can plot the number of infected people using a log scale, making it seem linear, when, in fact, it's exponential.

![](/posts/logs/log14.png)

However, there might be compelling reasons behind the decision to use a logarithmic transformation for this chart.

Imagine there are two kingdoms, Hyrule and the Mushroom Kingdom, each with vastly different population sizes. Hyrule has 10,000 inhabitants, while the Mushroom Kingdom boasts 10,000,000.

Now, let's say a disease outbreak occurs in both kingdoms, starting with 10 infected individuals in each. On a regular linear scale, the number of infected individuals in Hyrule might seem insignificant compared to the Mushroom Kingdom's numbers.

![](/posts/logs/log15.png)

If we switch to a logarithmic scale, the picture changes. The slope of the curve representing the increase in infected individuals in both kingdoms becomes comparable. So, when the disease spreads from 10 to 100 infected individuals in Hyrule, the curve's slope remains consistent. Similarly, when the Mushroom Kingdom sees its infected population rise from 10,000 to 100,000, the slope of its curve mirrors that of Hyrule's.

![](/posts/logs/log16.png)

This scaling ensures that the threat level posed by the disease is communicated effectively in both kingdoms, regardless of their size. Without it, Hyrule's situation might appear deceptively less severe simply because of its smaller population. By utilizing a logarithmic scale, public health officials can accurately assess and address the risk posed by the disease in each kingdom, enabling more informed decision-making and resource allocation.

In addition to impacting visualizations, logarithmic transformations are also useful for statistical purposes, mostly because of:

1. **Symmetry**. Logarithmic transformations may make data more balanced, which is essential for certain statistical techniques to work effectively.
2. **Homoscedasticity**. Logarithmic transformations may help keep the spread of data consistent, which helps compare effects on different groups.
3. **Linearity**. Logarithmic transformations may make it easier to see straight-line patterns in data, simplifying analysis.

 

## Symmetry

A graph is symmetric concerning a line if reflecting the graph over that line leaves the graph unchanged.

This histogram of dragon sizes is highly symmetric, for example.

![](/posts/logs/log17.png)

On the other hand, this graph of magic power levels in a sample of healthy wizards is not very symmetric. Instead, it's said to be positively skewed because of its long tail of results to the right.

![](/posts/logs/log18.png)

Now, pay attention to this magic trick: we can make this graph symmetric by applying a logarithmic transformation to all the magic power levels.

Such transformation makes these values more symmetric because the long tail of large values gets compressed, while the small values concentrated on the left get pushed farther apart.

![](/posts/logs/log19.png)

Symmetry is vital in data analysis for various reasons. For example, when using statistical methods, we typically aim to find a mathematical model that accurately represents the distribution of our data. The choice of an appropriate model often depends on whether the data is symmetric.

Anyway, discussing the details of model selection is a topic for another time.

 

## Homoscedasticity

In the land of Skyrim, knights rely on magical potions to enhance their strength during battles.

When we graph the effectiveness of these potions based on the initial strength of the knights, we notice greater variability in the potion's impact. This variability is a common occurrence with potions and medicines.

For instance, if a knight initially lifts 100 pounds, the potion might yield effects ranging from no change (100 pounds) to a significant increase (160 pounds, a gain of +60 pounds). Conversely, if a knight can lift 500 pounds, the potion's effects could range from maintaining the initial strength (500 pounds) to a substantial increase (800 pounds, a gain of +300 pounds).

When we plot this data on a linear scale, we notice that the variance increases as the knight's initial strength rises. This phenomenon, known as heteroscedasticity, reflects the heterogeneous nature of the data, where the variability of the potion's effects differs across the range of initial strengths.

![](/posts/logs/log20.png)

When we apply a log transformation to our graphs, it makes the data more homoscedastic, meaning it evens out the variance, especially for stronger knights. In simpler terms, this means that the potion's effects become more consistent across different strength levels.

This transformation reveals that the potion consistently boosts strength by around 0 to 20%, making it easier to predict its benefits for knights at all training levels.

![](/posts/logs/log21.png)

 

## Linearity

Let's resort to our imagination again.

Let's say someone's aura indicates one's affinity with celestial magic, and Moonbeam Extract is said to degrade this connection.

Imagine a scenario where our kingdom's alchemists seek to understand the relationship between Moonbeam Extract and someone's aura.

When we plot people's aura's brilliance against Moonbeam Extract, we notice a curved pattern, with most data clustered at the lower left corner. That plot gives us some clues about the relationship between people's auras and Moonbeam Extract, but it's not very clear.

When we employ logarithmic transformations on both magical properties, the correlation appears straight and clear.

![](/posts/logs/log22.png)

This linearity simplifies our understanding, allowing us to see how the effects of Moonbeam Extract clearly impact people's magic abilities.

 

## A final note on log bases

Throughout this post, we've mostly used common logs, where each segment represents a tenfold increase compared to the last (log10).

Still, it's worth noting that we can also use other log bases, such as log2 or natural logarithms (logE). These alternative log bases offer different perspectives and may be better suited for specific situations or data types.

 

## About these visualizations

We've built all these visualizations using Briefer.

If you want to build reports, presentations, or data apps with visualizations like these, you can sign up for early access to Briefer at [briefer.cloud](https://briefer.cloud).

A brief introduction to logarithmic scales and their applications in data visualization and statistics.

Log scales

This week we've released two major new features that we're very excited about: snapshots and a Slack integration.

## Snapshots

Snapshots allow you to save the state of your document at any time, and then restore it later.

![](/posts/changelog-5/snapshots.gif)

This is useful when you want to experiment with your code, but you're afraid of losing your progress. With snapshots, you can save your work before making changes, and then restore it if you don't like the changes.

Snapshots are also useful to keep track of how your metrics or visualizations change over time. In Briefer, every scheduled run creates a new snapshot, so you can compare results over time.

To use them, click the "Snapshots" button on the upper-right corner of the editor. Then, click the "Save" button to save the current state of your document. You can also restore a snapshot by clicking the desired snapshot, and then clicking the link to restore it at the top of the page.

**Snapshots are only available for users in our professional plan, and their retention period is 90 days**.

 

## Slack Integration

Our Slack integration allows you to receive PDF reports directly in Slack.

![](/posts/changelog-5/exports.gif)

To use it, you need connect your Slack workspace to Briefer. To do that, access the "integrations" menu at the bottom of our sidebar. Then, click the "Add" button and select "Slack".

After connecting your Slack workspace, add a scheduled run to your document, and, as you're doing it, click the "Add" button right next to the notification settings.

There, you will be able to select the Slack channel where you want to receive the PDF report.

Then, click the "Add Slack" button, and follow the instructions to connect your Slack account to Briefer.

The Slack integration is only available for users in our professional plan.

**Important**: You need to invite the Briefer bot to the desired channel before it can send notifications (`/invite @Briefer`). You won't receive notifications if your page fails to execute.

 

## General Improvements and Bug Fixes

* We will automatically execute previous blocks after your environment restarts. That way, you will not face issues with variables not existing anymore.
* You can now expand your document horizontally.
* Visualizations take 4x less time to load.
* We've improved the layout of text blocks. Text will look better now.
* We've fixed a few performance issues while switching tabs with multiple editors.

We've just released snapshots. They allow you to save the state of your document at any time, and then restore it later. We've also added a Slack integration, so you can receive PDF reports directly in Slack.

Changelog #5: Snapshots and a Slack integration

When we got into YC, we were building a product for companies to spin up development environments in the cloud. The idea was that, as software grows, it inevitably becomes too difficult to run on a single machine. At that point, people would come to us and we'd help each developer an environment in the cloud.

**It didn't work**.

During our batch, we amassed [a thousand stars on GitHub](https://github.com/briefercloud/layerform), [got to HackerNews' front page](https://news.ycombinator.com/item?id=37134293), and even had a customer pay for our product. Despite the short-term achievements, we soon realized that long-term success was probably not in the cards.

After the batch, **we pivoted away from building infrastructure products**.

Here's how we made that decision, the lessons we learned, and how I'd approach it differently if I had to do it again.

 

## Too simple to be useful, or too useful to be simple

It's extremely difficult to build an infra product that's both simple *and* flexible enough.

When building an infrastructure product you *usually* have to choose whether you want to serve small or large companies first. That's because these two types of companies care about diametrically opposite things.

Large companies want flexible products that can support their peculiar infrastructure needs, even if it takes a bit longer to set up. On the other hand, small companies would rather have a simple piece of software that takes only a few minutes to set up, even if it doesn't support esoteric network settings, for example.

If you decide to build an infra product for large companies, it will almost certainly be too complex for small companies to use. Conversely, designing an infra product for small companies will make it too simple for large companies to use.

![](/posts/pivoting-away-from-infra/complexity-quadrants.png)

In an ideal world, you'd be able to hide complexity from small companies, while still supporting the needs of large companies. But in practice, this is nearly impossible to achieve - that's why so few infrastructure products are successful.

 

## Small companies don't need it

We knew that selling infrastructure to large companies was going to be difficult, so we decided to focus on building a tiny product that was useful for small companies first.
![](/posts/pivoting-away-from-infra/our-quadrant.png)

Building a small product would allow us to iterate faster because we'd have customers early on, and we'd be able to get feedback from them. Then, we'd be able to raise capital to expand the product and go after larger companies.

At the time, we thought that pull-request previews were a good candidate for an MVP. We could start by deploying preview environments for pull requests, and then we would expand the product so that people could use these environments for development too.

The first version of our product was simple. It used a `docker-compose.yml` file to deploy pull requests and then posted a comment on the pull request with a link to the preview deployment.

![](/posts/pivoting-away-from-infra/pr.png)

Just a few days after we put a landing page up, a small company found us and started using the product. As soon as they hit the free tier's limits, they reached out to pay.

We then tried to replicate that success with other companies, but it was extremely difficult. That's because a small company's software is usually small. Thus, they can just check out the branch and get it running in a few seconds. Having preview links was convenient, but not game-changing enough to make them pay.

The same principle applies to other infrastructure products. Even if your software is pretty simple, small companies won't care about it unless it's absolutely essential for them. And even then, they might not want to pay for it, and might just default to a free alternative.

 

## Large companies won't trust you

After our small bet failed, we decided to rebuild our product to serve large companies, but we had a hard time getting them to trust us.

When deciding whether to purchase a product, buyers usually have the following equation in mind:

![](/posts/pivoting-away-from-infra/value-equation.png)

The higher the value, the more likely they are to buy.

The problem with infrastructure products is that the time-to-value and implementation effort variables are usually too high, while the perceived probability of success is too low. Consequently, no matter how amazing the dream outcome is, the final value is usually too low for buyers to move forward.

Now, let's dissect each of these variables and explain why the perceived probability of success is usually too low, and why time-to-value and implementation effort are usually too high.

 

### Perceived probability of success

![](/posts/pivoting-away-from-infra/perceived-probability-of-success.png)

Imagine you're running a multi-billion dollar company. Would you trust a scrappy 2-person startup to handle the infrastructure that keeps your engineers productive? I wouldn't.

Sometimes, not even large providers like AWS or Google Cloud can support these companies without a lot of hand-holding. So, why would they trust a 2-person startup?

At large companies, not rocking the boat is usually more important than getting the latest and greatest technology. Consequently, they won't trust you unless you have solid proof that you're at least 10x better than the current alternative.

 

### Effort

![](/posts/pivoting-away-from-infra/effort.png)

Creating an infrastructure product that demands little effort to implement is extremely difficult because of the very nature of the problem you're solving. You're building a product that has to be flexible enough to support all the unique needs of large companies, and that's usually complex, as we've seen earlier.

Just for engineers to understand how your product works, they'll have to spend a lot of time reading documentation, watching videos, and maybe even attending training sessions.

Furthermore, it's difficult to break down the problem into smaller pieces that can be solved independently. That's because the problems you're solving are usually interconnected, and solving one might create new problems that need to be solved.

Finally, considering all this effort, they'll probably default to having their own engineers build it, instead of buying your product. That's because they'll have more control over the final product, and they'll be able to customize it to their unique needs.

Ah, did I mention they hired tons of engineers who need to justify their next promotion?

 

### Time-to-value

![](/posts/pivoting-away-from-infra/time-to-value.png)

In addition to all the time engineers will have to spend learning how to use your product and set it up, it will take a while for them to see the productivity gains you promised.

That's because infrastructure products tend to deliver compounding value over time instead of delivering acute value upfront.

Take CI/CD products, for example. Testing and deploying your software for a month isn't that painful and doesn't take that much time. However, if you have to do it for a year, it becomes a huge pain point, and you'll want to automate it.

 

## Slow feedback loops

The more feedback you have, the smaller the bets you have to make, and thus, the less risk you have to take.

Think about buying 3-digit lottery tickets, for example. Paying $1 for each digit as numbers are drawn is better than paying $3 for all three digits at once.

If you have to buy all three digits at once, the likelihood of guessing all three is only 1%, and each attempt costs you $3.

![](/posts/pivoting-away-from-infra/lottery-3-digits.png)

On the other hand, if you pay $1 per digit, you can decide whether to continue as soon as the first number is drawn. If you get it right, you can pay for the next; if not, you can give up sooner and lose less money.

In this case, there's only a 10% chance you'll buy the second number and a 1% chance you'll buy the third. Therefore, each failed attempt will cost you less, but the overall likelihood of winning is still the same.

![](/posts/pivoting-away-from-infra/lottery-1-digit.png)

Building an infra product is like buying all three digits at once. You have to make a large bet upfront, and you won't know if you're building the right thing until you get your first customer. That's because it's difficult for large companies to trust you, and, even when they do, slow sales cycles and long implementation times will elongate the feedback loop.

Additionally, it will take longer to have a large enough sample size to know if your product is any good, and to understand whether a request is a one-off or a recurring need for most customers.

 

## How I would approach it differently if I had to do it again

When picking a startup idea, you're also picking a set of problems that come with it. Ultimately, all problems are solvable, but some are more difficult than others for a given team.

Even though I'm glad we pivoted away from infrastructure products, I still believe they're a great opportunity for the right team, so here's how I'd approach it differently if I had to do it again:

* **I'd insist on selling to small companies first.** Instead of investing time trying to sell to large companies, I'd invest time in rethinking the product so that it's useful for small companies. That's a piece of advice we got from PG himself, and it was spot on.
* **I'd accept having to make larger bets and act accordingly.** Given the slow feedback loop, I'd be more inclined to think long-term and make larger bets, like building more features upfront and building a community around the product, even if they're not paying customers, and probably won't be for a while (or ever).
* **I'd cut scope as much as possible and avoid being seen as a replacement for existing solutions.** I think this point is valid for all startups, but it's especially true for infrastructure products because switching infrastructure usually takes longer than anything else. Therefore, I'd think harder about whether I can build something small and less critical to go alongside existing solutions.

 

## How these learnings helped us pick our next idea

The worst part of building an infrastructure product was the slow feedback loop, so our first criteria for picking our next idea was to find a problem that we could offer a packaged solution for, and that we could sell to small and medium companies first. That solution also had to be easy to use, and the time to value had to be as short as possible.

Considering these criteria, we then brainstormed a few hypotheses and ran approximately fifty customer interviews to validate them.

We then landed on what we're doing today: a product that helps companies generate unique insights by using code.

![](/posts/pivoting-away-from-infra/demo-short.gif)

The reason Briefer turned out to be a great fit for us is that:

* **Time-to-value is extremely short**. Instead of spending days reading documentation and setting up the product, customers can just log in and start generating insights.
* **It appeals to both small and large companies**. Small companies usually love Briefer because their engineers can write code to generate insights, and they don't have to wait for a data team to do it for them. Large companies love it because they can use it to build more advanced analyses, data apps, and predictive models.
* **We don't have to replace existing solutions**. While some companies replace their BI tools with Briefer, most use it alongside them because traditional BI tools can't generate the insights Briefer can, nor can they be used to build data apps or predictive models.

In the end, we're glad we went through the infrastructure journey, even though it was tough. It taught us a lot about building products, and it helped us land on a better idea.

I'm extremely thankful for the YC partners, our batchmates, and the YC alumni who helped us along the way. Many of these learnings wouldn't have been possible without their help.

Finally, if you want to try Briefer, or even if you're not sure it's the right choice, please feel free to [visit our website](https://briefer.cloud) and [book an intro call with us](https://calendly.com/lucasfcosta/briefer-intro-call). We'd love to hear from you.

Here's the story of how we pivoted after our YC batch, the lessons we learned building in the infrastructure space, how I'd approach it differently if I had to do it again, and how these learnings helped us build Briefer.

What we learned from a failed infrastructure DevTool

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.

![](/posts/p-values/honeyberry-method.png)

 

## Step 1: Understanding p-values so you can misuse them

Your first step to riches is to create a technique that you can claim to improve any aspect of your customers' lives and that you can write books about.

It doesn't matter if your technique actually works. What matters is that you can convince people that it does, and that's what I'm here to teach you.

Let's say you've created a new study technique and you believe it helps students score better on exams, for example.

To test your hypothesis, you set up an experiment with two groups of students:

* Control Group: This group studies using traditional methods.
* Experimental Group: This group uses your revolutionary study technique.

Then, you give both groups the same exam and compare their scores. If the experimental group scores significantly higher, some people might conclude that your study technique is effective.

![](/posts/p-values/two-groups.png)

Now, that's where amateurs usually stop. But you're not an amateur, you're a business person. You need to convince even the nerdiest of your customers that your technique is backed by science.

These people like to think they are smart, so they'll definitely ask how do you know whether the difference in scores is real or just due to chance.

That's where p-values come in. The p-value is a statistical measure that determines how likely it is that the improvements in scores were due to random chance. In other words, p-values represent the chance of the same or more extreme scores happening even if your new study technique doesn't actually work.

A small p-value means that a difference in scores as extreme as the one you observed is unlikely to have occurred by mere luck. Thus, a small p-value suggests that your study technique might be effective.

For example, a p-value of 0.001 means there's only a 0.1% chance that the score improvement was just random. This suggests that the better scores could be due to your new study technique, not just chance.

On the other hand, a large p-value means the difference in scores could easily have happened by chance, and your study technique might not be effective.

For example, a p-value of 0.15 means there's a 15% chance that the score improvement was random. Having such extreme results 15% of times would lead nerds to say that your results are not "statistically significant", because these events are actually not that rare. In other words, they mean that the difference in scores could easily have happened by chance, and your study technique might not be effective.

> The way that classical statistics works is by comparing study data to what is expected when there is nothing. If the data are not typical of what is seen when there is nothing, there must be something!
>
> — Dallal, Gerard. The Little Handbook of Statistical Practice.

Having learned what p-values are, you probably figured that what we're after is a small p-value. So let's go ahead and make sure we get one.

 

## Step 2: Ensuring that `p` is small enough

Let's be honest: the chances of your revolutionary study technique actually working are slim. Consequently, your experiment will probably fail to demonstrate that the new study technique is much different from just doing whatever everyone is already doing.

Once your study fails, you have two options. The first is to repeat the experiment until it works. The second is to analyze the data in multiple ways until you find another measure that looks significant. Then you can rebrand your technique as being effective for that measure, like "improving memory retention" or "increasing focus".

* Pro-tip 1: if none of the measures look significant, you can always try to find a subgroup of the population that benefits from your technique. For example, "our revolutionary study technique is particularly effective for left-handed students born in the summer".
* Pro-tip 2: you can also find transformations of the data that make it look significant, like the sum of outcomes squared or the log of the difference between scores.

![](/posts/p-values/python-honeyberries.png)

This step is pretty easy to memorize because both alternatives rely on the fact that if you look around enough, you'll eventually find something that looks like a significant result. Another way to put it is that "the chances of winning the lottery are small, yet often there's a winner".

 

### A crucial detail: make sure p is smaller than 0.05

There's a convention among scientists that says that a p-value below 0.05 is said to be "statistically significant". So that's what you should aim for, at least most of the time.

There are many theories about why statistical significance is set at 0.05, but, as Gerard Dallal points in his book "The Little Handbook of Statistical Practice", all of them trace back to Karl Pearson's 1914 book "Tables for Statisticians and Biometricians".

![](/posts/p-values/karl-pearson-book.jpeg)

Pearson figured out how likely different outcomes were in statistics. Then Fisher made it easier by creating simple tables in his book "Statistical Methods for Research Workers" in 1925. These tables just showed basic probabilities for certain outcomes. Fisher's idea caught on, and he did the same thing with Frank Yates in 1938. People still use Fisher's tables in statistics texts today.

Fisher's tables were shorter and simpler than Pearson's. Instead of giving lots of details, they just gave rough probabilities. Fisher also came up with a neat way to show how sure we are about results—using stars. One star meant a result observed about 5% of the time, two stars meant it was observed about 1% of the time, and sometimes three stars meant it was observed about 0.1% of the time.

Fisher also said that when these extreme results would only happen 5% of the time without an intervention, we should take notice. He thought this level was good because it helped us find real results without getting too many false alarms. And this 5% rule has stuck around, shaping how we understand data and what we consider important in science.

In one sense, 5% is just a number, and it's not a magic number. But it's a good rule of thumb. Still, some people think it's too high, and that we should use lower levels like 1% or 0.1%.

To make matters simpler, I'd also recommend that you keep this excellent table from the scientific website `xkcd` handy:

[![](/posts/p-values/xkcd-1478.png)](https://xkcd.com/1478)

 

## Step 3: Publishing your results

Now that you have a small p-value, you need to call journalists and tell them about your revolutionary study technique. They'll be thrilled to write about it, especially if you can provide them with a scientific paper that includes the words "p-value" and "statistically significant".

The important thing here is not to mention that you've run your experiment 100 times until you got a significant result, nor that you've written a python script to test all possible variables until you found one that looks significant.

![](/posts/p-values/honeyberry-news.png)

Another important thing is to explain p-values to them in a way that doesn't *sound* egregious, but that is not entirely right either.

One incorrect way to explain p-values is to say that they represent the probability that the "null hypothesis" (your study technique not working) is true.

Let's say you've found a p-value of 0.001, for example. In that case, make sure to tell them that there's only a 0.1% chance that your study technique is ineffective. Although that's incorrect, it's much more convincing than saying "if the new study technique didn't work, we would still see the results we did approximately 0.1% of the time".

Another technique that you can use is to invert the logic. Instead of saying that a p-value of 0.001 means there's a 0.1% chance that the score improvement was random, they say that there's a 99.9% chance that the new study technique is effective. That's not correct either, but it's looks better in a headline.

 

## Step 4: Rebuking the skeptics

Once your revolutionary study technique is published, you'll probably face some skepticism. That's normal, and you should be prepared to deal with it.

The scientists are the most dangerous skeptics. They'll probably try to replicate your experiment and fail to find the same results.

If it's only one or two scientists, you can always say that "failing to find an effect is different from showing that there is no effect". They won't be able to argue with that because you're technically correct.

Now, if quite a few scientists fail to replicate your results, then you're in deep trouble. [That's why it's best to design experiments that are hard to replicate, like those that require a lot of data or that are too vague to be disproven](https://en.wikipedia.org/wiki/Replication_crisis).

In general, keep in mind that the best way to deal with skepticism is to prevent it from happening in the first place.

 

## Putting it all together

* P-values are a measure of how likely it is that you'd observe the results you did if your intervention didn't actually work.
* The smaller your p-value, the less likely you are to have observed the results you did by mere luck.
* Conversely, the larger your p-value, the more likely it is that you'd observe the results you did by mere luck.
* A p-value is *not* the probability that the null hypothesis is true. The opposite is also true: a p-value is *not* the probability that your intervention is effective.
* Just because you haven't observed an effect, it doesn't mean there isn't one.
* If you run the same experiment many times, you'll eventually see a p-value below 0.05, even if the null hypothesis is true.
* Looking for an effect until you find one is an effective way to make sure you get a small p-value, but it's bad practice unless you're looking for an easy way to sell books.
* A p-value below 0.05 is usually considered "statistically significant". Although it's not a magic number, it's a rule of thumb that has been around for a while.

 

## Before we go

In case you didn't notice, this post is a satire, and any parallels to the real world are, obviously, coincidence (no pun intended).

The post was inspired by the excellent work of [Randall Munroe](https://xkcd.com), in particular his comic [xkcd 882](https://xkcd.com/882):

[![](/posts/p-values/xkcd-882.png)](https://xkcd.com/882)

Finally, if you're a data-scientist who loves jupyter notebooks or data apps, but finds them hard to share your work with others or too much hassle to deploy, you should check out [Briefer](https://briefer.cloud).

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.

What p-values are, what they are not, and how to sell books with bad statistics

This changelog is the most important one we've shipped so far - that's why it took more than two weeks for us to post it.

In these last few weeks, we've completely rebuilt our editor so it's more understandable, more reliable, and more flexible.

We've also introduced environment variables, which allow you to store information like settings, API keys, and database credentials securely. Then, you can reference these variables in your code without exposing them.

 

## Our new editor and a ton of related improvements

Before this change, our editor was more unstable than we'd like. Sometimes the editor would show different states than what was happening in the back-end. Refreshing the page also caused the editor to lose part of its state, like which blocks were running, or undergoing AI suggestions. **All these issues are now gone, and we're glad to have received lots of positive feedback about the new editor.**

The new editor we're rolling out for everyone today is more reliable, more flexible, and more understandable. We now manage and reconcile every single piece of state in the back-end. Consequently, the editor's state will be consistent across refreshes, and for all users and devices.

![](/posts/changelog-4/v2editor.gif)

This change also comes with a bunch of very important improvements. Here are some of them:

* When using data apps, each viewer will have their own state. That means that if you're viewing a document and someone else is viewing it at the same time, you won't see each other's inputs and results.
* The "Run All" button now shows the number of blocks that have already run, and how many are left.
* We explicitly display enqueueing and running states for each block.
* When a scheduled run is happening as you view a document, you'll see its results in real time.
* You can now duplicate whole blocks or individual tabs by clicking the drag-handle menu on the left side of the block.
* You can run an entire block from left to right by clicking the "Run all tabs" button that appears within the drag-handle menu for multi-tab blocks.
* Data apps will only be updated when you click the "Update Data App" button. This way, you can make multiple changes to the inputs and dropdown fields of a data app and only publish them when you're ready.
* Adding new blocks is now more intuitive. You can now add blocks by clicking the "+" button that appears when you hover over gaps between blocks.
* You can now group all types of blocks, not only Python, SQL, and Visualizations.
* If you create a new dataframe that only exists in a Python block, it will be immediately available within visualization blocks.
* You don't have to uncollapse a block to run it anymore. You can now run a block by clicking the "Run" button that appears when you hover over the block.
* We'll now persist the open/close state of the document tree on the left as you navigate between documents. This way, you won't have to keep opening nested folders every time you navigate to a different document.

 

## Environment variables and how to use them

Environment variables are a way to store information like API keys, database credentials, and other settings securely. That way, you can then reference these variables in your code without exposing them to viewers.

These environment variables are useful when you want your Python blocks to perform authenticated API requests, like when doing writebacks to your database, for example.

To create an environment variable, click the "Environment Variables" button on the bottom-left corner of the editor. Then, click the button at the bottom of the list to add a new variable. Once you're done, click the save button and restart your environment for the new environment variables to be available.

You can then reference this variable in your Python code by using the `os.getenv` function.

![](/posts/changelog-4/v2vars.gif)

By the way, we also added a button on the bottom right of the editor so you can restart your environment manually if you need that.

 

## What's next?

We know it's been somewhat annoying to lose your in-memory state when your environment restarts. We're working on a way to automatically re-run dependencies or persist this state across restarts, so you won't lose your dataframes, variables, and other in-memory state when your environment restarts.

Expect that by the next changelog.

This changelog is the most important one we've shipped so far - it includes a brand new editor and support for environment variables.

Changelog #4: Environment variables, and a brand new editor

Besides the usual bug fixes, we're shipping a major new feature this week: **data apps**.

You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.

![](/posts/changelog-3/data_apps.gif)

Please notice data apps are only available for users in the professional tier.

 

# When to use data apps features

You should use data apps features whenever you want viewers to be able to provide input to your reports, models, or analyses.

In a nutshell, data apps are interactive documents that allow viewers provide input to the code you've written.

Typical examples include:

* **Comparing scenarios**: Allow viewers to change assumptions and see how the results change.
* **Exploring data**: Allow viewers to change the data they're looking at.
* **Filtering data**: Allow viewers to filter data by date, region, or any other dimension.

 

# How data apps work

To create a data app, you must first add an input or select field to your document by using the `/input` or `/select` commands.

For each of those blocks, you can specify a label. The variable name is the one you see on the top right corner of the block in a green box.

Then, you can reference the variable name in your Python code in the same way you would reference a variable you created yourself.

To interpolate the value of the input or select field in SQL blocks, use the `{{variable_name}}` syntax.

> Please notice that you can now interpolate *any* variables in your SQL blocks, not only the ones created by input or select fields.

Finally, use the "Make it a data app" toggle on the top right corner of your document, under the share menu, to enable the data app mode.

Toggling the data app mode allows viewers to provide input to your document, but they still can't edit the document's content.

 

## Other fixes and improvements

* We fixed a bug where users would not be automatically logged out once their session expired. Now, users will be imediatelly logged out and redirected to the login page when their session expires.
* Document snapshots: we now keep snapshots for users in the professional tier. These snapshots are taken every five minutes and will be available for the last 5 days. The last 10 snapshots are always available, regardless of how old they are.
* We've done updates to the visualizations so they support data types they used to not support.

This week, we're introducing data apps. You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.

Changelog #3: Introducing data apps

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements.

If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.

 

# General Updates

Here are a few small improvements we've done to the editor, and to the overall performance of Briefer.

## Editor performance improvements

We noticed that performance degraded as documents got larger. We've made a few changes to the way we handle large documents.

First, we've eliminated lots of unnecessary re-renders for graphs and tables. This should make the editor feel snappier, especially for large documents.

## Support for larger datasets

Some users dealing with millions of rows of data were experiencing performance issues.

To fix that, we now run all the queries in your own compute instance and stream the results back to the editor. This change means that we can now support much larger datasets without performance issues. It also means that the users who pay for larger compute will experience more benefits from it.

## Improvements

Besides the performance improvements we shipped a few minor UI updates. Besides all the small UI tweaks we've done, here are a few highlights:

### In/Not-in filters

We've added support for `in` and `not-in` filters in visualization blocks. This is useful when you want to filter by multiple values at once.

![](/posts/changelog-2/in_not_in_filters.png)

### New query on visualizations

You can now create queries more easily when using visualizations. Just click on the "New query" option the dropdown, or in the tooltip that appears when you create an empty visualization.

![](/posts/changelog-2/new_query_option.png)

You can also create new queries or tabs by using the `+` button in the top right corner of the block.

![](/posts/changelog-2/plus-tab-button.png)

### Deleting tabs or entire blocks

You can now delete tabs or entire blocks by clicking on the drag-handle and selecting the desired delete option.

![](/posts/changelog-2/delete_tab_or_block.png)

### New users are viwers by default

Previously, when allowing anyone from `@yourdomain.com` to access Briefer, new users were added as editors by default. We've changed this so that new users are added as viewers by default. This is to prevent new users from accidentally making changes to the workspace.

 

# Bug fixes

Bug fixes are the biggest part of this release. We've fixed a ton of bugs, but here are a few highlights:

* Fixed a bug where there was a race condition when creating tokens for compute instances. This was causing some users to experience multiple "something went wrong" alerts that would only go away after the environment got restarted.
* When changing the filters or settings for visualizations multiple times, we'd enqueue each update. This caused visualizations to be slow to update. We've fixed this so that only the latest update is enqueued and previous updates are discarded.
* Our AI suggestions were sometimes suggesting overwriting dataframes using `[Object object]` dataframes. We've not only fixed this bug, but we also improved the LLM's context.
* Creating new tabs was causing an empty block to appear in the editor until you switched to another tab and back. This was a visual bug that didn't affect the actual document, but it was annoying - and now it's fixed.
* Dragging and blocks into one another was not working properly. Sometimes, the editor would try to drag the wrong block, or it would not detect you wanted to group different blocks. This bug is now fixed.
* Editing AI suggestions and then accepting them was causing the edited diff not to be applied. Instead, it applied the raw AI suggestion. We fixed this so that any changes you do in the diff editor will now persist in case the suggestion is accepted.
* Display of titles was showing incorrect background colors during loading and disabled states. They were also not taking the appropriate space. We've done fixes to their appearance so they behave correctly in all cases.
* Visualizations were not being updated in scheduled runs. This was a bug that was causing some users to receive outdated visualizations in their scheduled reports. This is now fixed.

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements. Here are some of the highlights.

Changelog #2: Blazing fast performance and tons of bug fixes

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently.

This is a non-exhaustive list of everything that's changed. As we publish more updates, we'll increase the level of detail here.

If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.

 

# New features

This week: no-code visualizations, tabs (grouping blocks), naming blocks, workspace settings, and exports.

 

## 1. No-code visualizations 📊

The most important change comes first: you can now create graphs without code. For that, type `/visualization` and add a visualization block.

![](/posts/changelog-1/visualizations.gif)

There, you'll be able to select the query to plot, types of graphs, filters, and more.

Once you're done with your visualization you can click the double-chevron icon to toggle between edit and visualization modes. In visualization mode, only the graph and filters appear.

Please notice that histograms and pie charts are not available yet.

 

## 2. Grouping blocks 🫂

You can now group multiple blocks into one to create multi-tabbed blocks.

Grouping blocks is useful when you need multiple python blocks to clean data, but only want a single python block with the results to show.

![](/posts/changelog-1/grouping.gif)

You can also drag-and-drop tabs to change their order. To ungroup tabs, drag the desired tab out of the tabbed block.

 

## 3. Naming blocks 🪄

Now that you can group blocks, it's important that you name them so that you know what's in each tab.

Naming blocks is also helpful for presenting reports - it helps stakeholders understand what they're seeing.

![](/posts/changelog-1/naming.gif)

To name blocks, just click the title of each block and input whatever title you want.

 

## 4. Workspace settings ⚙️

We heard a few people wanted to change their workspace names or settings *after* the workspace had been created. That wasn't possible before.

Now, you can rename your workspace and enable everyone with the same email address to join.

![](/posts/changelog-1/workspace-settings.gif)

 

## 5. PDF exports 📑

We know many of you were asking for this, so we have shipped it.

You can export PDF files by opening the share menu on the top right corner and then clicking the PDF export option.

**For now, PDF exports are only enabled for users in the professional tier.**

![](/posts/changelog-1/pdf-exports.gif)

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently. This week: no-code visualizations, tabs, and exports.

Blog

Why Use Bayesian Methods for A/B Testing

More articles

Why Use Bayesian Methods for A/B Testing

Bias-Variance tradeoff: understanding model complexity

The geometry of taste: putting numbers on preferences

How a straight line teaches machines to learn

How to get a feel for the data

Going open-source as a VC-backed company