At Briefer, we are building multiplayer notebooks. That means many users can edit and run blocks in the same notebook at the same time. In this post, I'll explain how we got to multiplayer notebooks that actually work.

Multiplayer notebooks

People usually ask me what's wrong with Jupyter Notebooks and why we're building a tool to replace them. They also ask us how we're making notebooks better.
I usually repeat the same things over and over, so I thought I'd write them down here so I can link to it.
<h2>1. Too messy too quickly</h2>
The first thing I don't like about notebooks is that they get too messy too quickly. One morning you have 10 blocks, and by the end of the day you have 192, but only six of them are meant to be seen by other people and the rest is just you jiu-jitsuing with the data.
We solved that problem by doing three things.
First, we allowed people to group their blocks into tabs. This way, you can have an "API Call" tab to fetch data, a "Data Cleaning" tab to clean it, and a "Visualization" tab to plot it.
<img src="/posts/improving-notebooks/tabs.gif" alt="">
Then, we created the concept of "publishing" a notebook and added buttons for people to hide certain blocks from the published version.
<img src="/posts/improving-notebooks/edit-vs-publish-blocks.png" alt="">
Finally, we realized that vertical scrolling is not always the best way to navigate a notebook. It wastes a lot of screen real estate and makes it hard to see the "big picture" of what's going on.
We fixed that by adding a "dashboard" view with a grid in which you can drag and drop blocks.
<img src="/posts/improving-notebooks/notebooks-to-dashboards.gif" alt="">
Today, most published notebooks use the dashboard view instead of the traditional vertical scrolling.
<h2>2. Difficult to find and explore</h2>
The second thing I don't like about notebooks is that they go to GitHub and then die because no one ever opens them again.
I wanted notebooks to be "live" artifacts that everyone can find, open, and explore, including non-technical people.
To solve that problem we created a Notion-like file tree.
<img src="/posts/improving-notebooks/the-file-tree.gif" alt="">
The file tree makes notebooks more approachable because it allows you to create top-level notebooks like "Sales Team" and put all your sales-related notebooks inside it, like sales forecasts and customer segmentation.
I know the icon selection is a bit cheesy, but it makes things more approachable for non-technical people. Once we launched this feature, people complained that it included 288 icons "only" and that they wanted more, so we added an extra 1487. Today, most notebooks have a custom icon.
<h2>3. Too much friction for querying databases</h2>
The third thing I don't like about notebooks is that they require a lot of boilerplate code to query databases.
We fixed that by adding a "query" block that allows you to write SQL and see the results in a table.
<img src="/posts/improving-notebooks/sql.png" alt="">
We also know that queries are useless unless you can wrangle the data, so each query's results are automatically converted into a pandas dataframe.
<img src="/posts/improving-notebooks/sql-to-dataframe.png" alt="">
Then we went full circle and allowed people to write SQL queries against their pandas dataframes. We did that because sometimes the declarative style of SQL makes it a better choice than Python.
<img src="/posts/improving-notebooks/select-from-dataframe.png" alt="">
<h2>4. Not collaborative</h2>
The fourth thing I don't like about notebooks is that I can't edit them at the same time as someone else. I also don't like it that I have to push my changes to GitHub and then someone else has to pull these changes to see what I'm doing.
We fixed that by using <a href="https://yjs.dev/">Yjs</a> to store the notebook's state.
Whenever you create a block, delete a block, or move a block, we send that information to Yjs, and then Yjs sends it to everyone else who's viewing the notebook.
If two people try to edit the same block at the same time, Yjs knows how to merge their changes, so no one loses their work.
<h2>5. Outdated results</h2>
The fifth thing I don't like about notebooks is that they get stale, and you have to re-run them manually to get the latest results.
Let's say you have a sales forecast notebook built on top of the latest CRM data. In that case, Jamie from the sales team will probably ask you to re-run the notebook every week or so to get the latest forecast.
We fixed that by adding a "schedule" button that allows you to run a notebook at a specific time, like every Monday at 9 am.
<img src="/posts/improving-notebooks/scheduled-runs.png" alt="">
If the notebook fails to run, we can send you an email or a Slack message. Otherwise, we can send you the complete notebook in PDF format on those same channels.
I love the notifications feature, but some people overuse it. They schedule their notebooks to run every few minutes and then get mad when they get too many notifications. We don't know how to solve that problem yet.
<h2>6. Not interactive</h2>
The sixth thing I don't like about notebooks is that they're not interactive.
We fixed that by adding a few native text fields and dropdown blocks that allow you to ask for user input.
<img src="/posts/improving-notebooks/normal-inputs.png" alt="">
We know that users sometimes want their inputs to be dynamic, like when they want to use values from their databases as options in a dropdown, so we did that too.
<img src="/posts/improving-notebooks/dynamic-inputs.png" alt="">
We do know about <code>ipywidgets</code>, but we don't like them because they get lost among the other blocks, and their styling is not consistent with the rest of the notebook.
<h2>7. Friction to plot graphs</h2>
The seventh (and final) thing I don't like about notebooks is that plotting simple graphs requires more code than it should.
Whenever I need a simple line chart, I have to write a few lines of Python code, and then I have to do all the styling manually. Otherwise, the chart looks like it was made in Excel.
Additionally, I keep having to look up the documentation to remember how to plot anything other than basic charts.
We solved that by adding "visualization blocks" that allow people to plot data by clicking buttons instead.
<img src="/posts/improving-notebooks/vis.png" alt="">
These blocks look good by default (at least that's what we think) and you can use them to plot data from any dataframe.
Some people complained about having to filter data using Python before plotting it, so we added a "filter" button to the visualization blocks too.
<img src="/posts/improving-notebooks/vis-filters.png" alt="">
Still, sometimes people need to use Python to plot more complex graphs, like when they want to plot a sankey diagram or a network graph. In that case, they can click "edit with AI" and ask the LLM to plot the graph for them. That way, they don't have to read the documentation every time they need to plot something new.
<img src="/posts/improving-notebooks/ai-sankey.png" alt="">

People usually ask me what's wrong with Jupyter Notebooks and why we're building a tool to replace them. I usually repeat the same things over and over, so I thought I'd write them down here.

Making notebooks better

I used to love two things about computers. The first was that they did exactly what you tell them to do. The second was that they did the exact same thing every time.
Humans shattered that magic. We now have AI assistants who do what you tell them to do, but they also do a lot of other things that you didn't tell them to do, and they do it differently every time.
When I started implementing AI assistants in our notebooks, I hated that. At times, the assistant would perfectly execute complex tasks like developing a churn prediction model but then stumble over something as simple as plotting a revenue line chart from a CSV of financial data.
At first, I thought I'd have to do something fancier than just RAG and "prompt engineering" to get the AI assistant to work. Then, I realized that I could get better results with a few UI tweaks, and this is what this blog post is about.
<img src="/posts/code-gen-for-notebooks/ai-assistant-15fps.gif" alt="">
<h2>Setting good examples</h2>
The most useful mental model for implementing AI assistants is to think of yourself as a manager onboarding a new senior employee.
You wouldn't expect the new senior employee to know everything about your business on their first day, but you would expect them to be smart and capable of learning quickly. In other words, you assume zero knowledge and infinite intelligence.
I like that approach, and that's what most people do. Let's say they're building an assistant to help fifth-graders with their homework. In that case, most people would just do some kind of RAG over fifth-grade textbooks, and that would be it.
In our case, we did something similar. Initially, we fed the LLM with the content of the current SQL or Python block, the user's database schema, and a list of data frames, their columns, and their types.
That approach worked well, but one common complaint was that the LLM's output didn't follow the user's coding standards. For example, the user's previous code used <code>plotly</code> but the LLM used <code>matplotlib</code>. Or the user's code used a specific color palette, but the LLM would just default to the standard one.
To fix that, we started including a few code snippets from the notebook so that the AI assistant could see whether we were using <code>matplotlib</code> or <code>plotly</code>, and what colors users prefer.
Oddly enough, one of the most impactful changes we made was to include the document's title and the content of relevant text blocks if we had any. That way, the AI assistant could see what the user was trying to do, and suggest something that made sense in that context.
<img src="/posts/code-gen-for-notebooks/ai-context-examples.png" alt="">
In a way, this approach mimics what a senior engineer does when they're submitting their first few pull requests. They look at the existing codebase, see what the team is doing, and try to follow the same patterns.
<h2>Encouraging small steps</h2>
Pull request sizes are usually inversely proportional to the wrinkles on a programmer's forehead. The more seasoned the programmer, the smaller the pull request.
That's because seasoned programmers know that the more code they write before getting feedback, the more code they'll have to erase if something is wrong.
Additionally, working on small steps at a time makes tasks easier to wrap your head around, and easier to debug.
The same principles apply to AI assistants. You should encourage users to give the AI assistant small tasks at a time and ask the user for feedback on each task before the assistant moves on to the next one.
In our case, we used to have an input for users to ask the AI to perform an entire analysis on their behalf. After a month or so, we removed it because we found that tasks got too large for the AI to handle at once, and too intricate for users to describe accurately.
<img src="/posts/code-gen-for-notebooks/ai-user-flow.png" alt="">
Now, we include AI assistants in individual blocks. That way, users end up asking the AI to do a single step at a time.
Consequently, user's prompts usually get more accurate, because they don't have as many intricacies to describe and thus tend to devote more time to crafting a better prompt.
A further benefit of encouraging small steps and getting feedback on each is that the user gets to vet each suggestion before moving on to the next one. That way, the assistant's corpus gets more accurate over time, and the user gets more comfortable with the assistant's suggestions.
<h2>Optimizing for feedback</h2>
When asking users to take small steps, you must make each step as fast and pleasant as possible. Otherwise, users will get frustrated and give up.
In our case, for example, we used to have the AI replace the user's code with its suggestions, and we'd do it all at once. That was a terrible idea because users couldn't tell whether the AI was working until it was done, and they couldn't see what the AI was doing until it was too late.
To fix that, we did three things.
First, we started streaming the AI's response instead of emitting it all at once. That way, users could see the AI's suggestions as they were being generated, and they could stop the AI at any time if they didn't like what they were seeing.
After that, we started showing the user a diff of what the AI assistant suggests. Then, they could accept or reject the changes. That way, users could clearly see what the AI assistant was suggesting versus what they wrote.
Finally, we added a button that allowed users to try the AI assistant's suggestion without accepting it. That way, users could try the suggestions before accepting them, and they could also modify the diff in case something was slightly off.
<img src="/posts/code-gen-for-notebooks/accept-reject-try-buttons.png" alt="">
Something else we're working on is to allow users to prompt the AI assistant's suggestions without having to accept them.
In addition to making each step more seamless, these diffs also prevent users from running undesirable code as it puts on them the burden of reviewing each change.
<h2>Encouraging action</h2>
Clicking a button is quicker than writing a prompt, and it's also more intuitive. Consequently, you should consider adding a "quick trigger" button whenever there's an action that the assistant can take without the user having to prompt it.
In our case, errors are a prime example of that.
Whenever someone writes invalid SQL or makes a typo in their Python code, the error that comes up usually has all the necessary information for the AI assistant to fix it. Therefore, there's no need to ask the user to prompt the AI assistant to fix it.
Instead, we show a button that says "Fix with AI". When the user clicks it, we feed the LLM with the error message and a bit of context, and we ask it to try to fix it.
<img src="/posts/code-gen-for-notebooks/fix-with-ai.gif" alt="">
These types of scenarios are more common than you'd think, and they're a great opportunity to show users the value of the AI assistant. People will trigger the assistant more often if they don't have to think about what to do with it.
<h2>Models usually take care of themselves</h2>
Finally, one last piece of advice is to assume that new LLMs will only get better, faster, and cheaper over time.
Take GPT-4o, for example. When it was released, it solved quite a few problems we had with the previous versions. All of a sudden, our default assistant got way faster and more accurate.
The trick is that we didn't have to do anything to make that happen. Had we spent months fine-tuning models and squeezing every last drop of performance out of them, we would've wasted a lot of time.
Instead, we focused on making the user's experience as good as possible, and we let the LLM's performance take care of itself.
Then, when a new model came along, we could just plug it in and get better results without having to change anything else.
In comparison, everyone else who spent months working on their models had an assistant that was just as good as ours, but without all the UI improvements we had made.
<h2>Putting it all together</h2>
Perceived performance is the only performance that actually matters. If users perceive your assistant to the better, it doesn't matter whether its underlying model is the one with the highest scores on standard benchmarks.
In a way, making AI assistants better is as much about improving the models as it is about improving the user's experience when using them.
Sure, there's a lot of value in creating better models, but there's a lot more innovation you can do if you focus on the specific problem you're trying to solve.
Consequently, if you're adding AI features to your product, you should probably start with the low-hanging fruit: UI/UX improvements.
Here's what we found helpful:
<ol>
<li>Set good examples: instead of including only raw data, augment generation with user-generated examples and specific context.</li>
<li>Encourage small steps: ask users to prompt the AI assistant on small tasks at a time, and ask for feedback on each task before moving on to the next one. Then, use each step's result to improve future suggestions.</li>
<li>Optimize for feedback: make each step as fast and pleasant as possible, and make it easy for the user to adjust the assistant's suggestions.</li>
<li>Encourage action: add a "quick trigger" button whenever there's an action that the assistant can take without the user having to prompt it.</li>
</ol>

Improving your user's experience with AI assistants without messing with the models.

Copilot UIs

In the past few weeks, we've got plenty of requests for "quality of life" features, so we delivered them.
This week, we're glad to show you how to use new dashboard features, connect MySQL databases, and query uploaded files using plain SQL.
Many people asked for context-aware autocomplete in SQL and Python blocks, so we shipped that too.
<h2>Querying files</h2>
You can now write SQL queries against files and dataframes. This means you can upload a CSV file and query it using SQL, for example.
To do that, add a "File Upload" block, drag your files into the block, and then click the "query" button to generate an example <code>SELECT * FROM</code> query.
Feel free to change that query to do whatever you want with the data in the file.
<img src="/posts/changelog-7/query-files.gif" alt="">
In addition to querying files, you can also query any pandas dataframes, as you can see in the example below.
<img src="/posts/changelog-7/query-dataframes.png" alt="">
As a general note, you should select the <code>Files (DuckDB)</code> data source whenever you want to query anything other than a database.
<h2>Better dashboards</h2>
We did tons of changes to the aesthetics of dashboards, but we're not done yet.
In the past few weeks, we cleaned up the visualization display, added support for data labels, and also improved the way all blocks look in the dashboard.
Additionally, blocks now show their titles on the top left corner, and the sidebar has been completely revamped so all blocks look nice within it, and so you can see their titles. That way, it's easy to identify what each block is doing at a glance.
<img src="/posts/changelog-7/dashboards-tour.gif" alt="">
<h3>Dashboard headings</h3>
Headings can help you organize your dashboard's content. You can add a headding to any block by clicking the "Add Heading" button on the top right corner of the block list.
These headings will only appear in the dashboard view, not in the notebook.
<img src="/posts/changelog-7/dashboard-headings.gif" alt="">
<h3>New visualization types</h3>
In addition to the aesthetics, we've also added a few new visualization types, such as "trend" and "numbers".
The "trend" visualization is great for showing trends over time, while the "numbers" visualization is perfect for highlighting a single metric in a large font.
<img src="/posts/changelog-7/numbers-and-trends.png" alt="">
<h2>Autocomplete</h2>
We've added autocomplete to SQL and Python blocks. This means you can now get suggestions for column names, table names, variable names, and functions while writing your queries.
<img src="/posts/changelog-7/autocomplete.png" alt="">
Please note that for the SQL autocomplete to work, the permissions of the user you're using to connect to the database must allow for querying the information schema.
<h2>MySQL support</h2>
We've added support for MySQL databases. You can now connect to a MySQL database and query it using SQL blocks.
<h2>General Improvements and Bug Fixes</h2>
<ul>
<li>It's now possible to undo AI suggestions and try them before accepting the diff.</li>
<li>There's a new onboarding document that will help you get started with Briefer when creating new workspaces.</li>
<li>Visualizations now support multiple Y-axes.</li>
<li>Document's titles will now appear in the browser's tab.</li>
<li>We fixed a bug in which the line chart would offset ticks in the X-axis.</li>
<li>We fixed a bug in which the "in/not in" filters would not work for dataframes.</li>
<li>We fixed a bug in which the blocks would be repositioned when opening the sidebar.</li>
<li>We fixed a bug in which the progress indication would show incorrect values when running entire notebooks.</li>
<li>We fixed a bug where invalid visualizations would fail silently and not show any error messages.</li>
</ul>

In the past few weeks, we've got plenty of requests for 'quality of life' features, so we delivered them. This week, we're glad to show you how to use new dashboard features, connect MySQL databases, and query uploaded files using plain SQL.

Changelog #7: Querying files and dataframes, better dashboards, autocomplete, and MySQL support

Sales pitches are the only place where “self-serve dashboards" work. In the real world, it's a different story.
That story usually starts with an engineer or data scientist who's frustrated because they spend too much time writing queries and preparing dashboards for business people. They think that if they make BI easy enough, everyone will be able to “self-serve”, but that rarely ever happens.
What actually happens is that engineers and data scientists end up being the only people using the "self-serve" BI tool, which, ironically, makes it not self-serve at all.
<h2>Why self-serve BI doesn't work</h2>
SQL is the only self-serve BI tool. Still, most "self-serve" BI vendors don't want to admit it, so they find ways to reinvent the wheel and disguise SQL as something else. Maybe that's what customers want to hear, or maybe they genuinely believe it, I'm not sure.
Either way, the problem with most approaches to "self-serve" BI is that they consider SQL to be the only barrier to business stakeholders querying data. That's not true.
Let's say you could wave a magic wand and make writing SQL queries as easy as writing an email. Do you think business stakeholders would suddenly start querying data? The answer is no.
Even if these people knew how to write SQL queries, they wouldn't understand the semantics of the data they're querying. They wouldn't know what the data means, where it comes from, or how it's calculated. They wouldn't know how to interpret the results, or how to validate them. In other words, writing the SQL query is actually the easy part.
Now, let's look at two attempts to make BI "self-serve" and see whether the hypothesis above holds.
<h3>Attempt 1: The conventional "dropdowns and checkboxes" approach</h3>
First, let's take a look at an example of a conventional "self-serve" BI interface which uses dropdowns and checkboxes as a way to make queries more accessible.
<img src="/posts/self-serve-bi/sql-by-mouse.png" alt="">
The first thing to notice is that this interface is just an attempt at what I call "SQL-by-mouse". I might be just an old grumpy person, but I don't see how this is any better than writing SQL. In fact, it's worse because it's slower, less reliable, more limited, and not generalizable to other tools.
In any case, I'll concede one thing: some non-technical people are just scared of any monospaced writing with syntax highlight, so this interface might be a good first step.
Anyway, let's ignore all these problems and just assume that this interface is the best thing since <a href="https://htmx.org/">htmx</a>. Let's assume that it's 100x easier to use than SQL.
Even if that were the case, your CFO, for example, wouldn't just go ahead and use these dropdowns and checkboxes to query data. The reason they wouldn't use it is that they don't have the context to understand the data they're querying, and they'll probably be insecure about the results, which they won't know how to validate because they don't understand how the data got there in the first place.
Now, I've got a question for you: if your CFO doesn't use this interface, who will? That's right, you will. And guess what's the first thing you'll do when you get to these dropdowns? Correct, you'll immediately look for somewhere to type in SQL. Oh, the irony.
<h3>Attempt 2: The text-to-SQL approach</h3>
Remember that magic wand that makes writing SQL queries as easy as writing an email? That's what text-to-SQL tools are. Nonetheless, they're not enough, just as I explained earlier.
Again, the problem is not the technology itself. In fact, LLMs are almost too effective at translating natural language into SQL. They will find a way to generate a query for any question you ask, even if that question doesn't make sense, which is exactly the problem.
On the other hand, a technical person would notice that the question doesn't make sense, and they would ask for more context. They would ask for details about the business person's hypothesis and the problem at hand. Then, they would explain what type of data is available, and work with the business person to formulate a precise and useful question.
Again, SQL is not the problem.
In any case, I do think LLMs could be the actual solution to self-serve BI, but not in its current form. For them to work, they'd need to be fed with more context, and <a href="https://allenpike.com/2024/llms-trained-on-internet">they need to get better at expressing uncertainty</a> and asking for more information.
<h2>What actually works</h2>
If we assume that the problem with self-serve BI is not SQL, but the context and semantics of the data, then it follows that the solution is to teach people about the data they're querying, regardless of interface.
The problem with this type of training is that it takes time, and most business stakeholders need answers this week, not next quarter.
Even in the best-case scenario, where companies train business stakeholders to understand the data they're querying, these people will need time to keep up with the changes in the database's schema, data models, and ETL processes.
Additionally, documenting all this knowledge generates significant overhead for the technical team, and it quickly gets out-of-date.
So what's the true solution to self-serve BI? The answer is simple: not to make BI self-serve for non-technical people. Instead, the solution is to make technical people support business stakeholders and help them do it more efficiently, using better tools.
The definition of a better tool may vary, but I have a few suggestions.
The first suggestion is to give LLMs to technical people, not business stakeholders. Even though text-to-SQL is not quite there in terms of understanding context and semantics, technical people already have context and semantics, so why not let them use it? That way, a single technical person can serve more business stakeholders, and do it faster.
The second suggestion is to give technical people more flexible tools. Instead of giving them dropdowns, or a SQL-only interface, let them play around with data using Python, R, or any other tool they're comfortable with. This way, they don't need to be passing data back and forth between different tools, and they're not limited by the capabilities of a BI tool.
The third and final suggestion is to make it easier for technical people to share their work. Notebooks and internal data applications are notoriously bad at that because they require others to deal with containers, dependencies, and infrastructure. It's a lot of overhead for a business stakeholder who just wants to see a chart. Instead, we need tools that have all the building blocks of a data application, but that are as easy to share as a Google Doc.
I'm obviously biased because I'm building <a href="https://briefer.cloud">a data tool for technical people</a>, and it does exactly these three things. But what other choice do I have? I'd rather build tools that work than tools that don't. And, most importantly, I'd rather not pretend that the problem is something it's not.

Sales pitches are the only place where self-serve BI works. In the real world, it's a different story.

Self-serve dashboards

Since the very first day we launched Briefer, we had a strong hunch that dashboards would be something people would want. We were right. We've been getting a lot of requests for this feature, and we're happy to announce that it's finally here.
Along with dashboards, we've also made some improvements to our visualizations and AI assistant. Here's what's new.
<h2>Dashboards</h2>
You can now turn your notebooks into dashboards. Dashboards are a great way to keep track of your most important metrics and KPIs in one place. You can add multiple visualizations to a dashboard, rearrange them, and resize them to fit your needs.
At Briefer, we care a lot about empowering users to do whatever they want with their data, that's why our dashboards support Python outputs. You can use Plotly, Seaborn, or any other Python library to create your visualizations and add them to your dashboards.
<img src="/posts/changelog-6/dashboards-light.gif" alt="">
To create a dashboard, click the "Dashboards" button on the upper-right corner of the editor. Then, drag and drop the blocks you want to add to the dashboard.
You can also resize each block by dragging the bottom-right corner of the visualization.
Once you're done, click the "Publish" button to save your dashboard. You can access it at any time by clicking the "Dashboards" button on the upper-right corner of the editor.
Please notice that you will need to republish your dashboard every time you make changes to the blocks if you want to see them reflected in the dashboard.
<h2>Better visualizations</h2>
To go along with the new dashboards feature, we've also made some improvements to our visualizations.
You can now rename each axis in your visualizations, format the value displays, and choose whether to display the labels or not.
We've also added support for histograms.
<img src="/posts/changelog-6/new-vis.gif" alt="">
<h2>Better AI</h2>
Our SQL and Python AI assistant has also received some updates. It's now better at understanding your questions and providing relevant answers.
We've also had a few users ask for a way to change the foundational model of the AI assistant. Now, you can do that by clicking the "Settings" button on the lower-left corner of the editor, and then selecting the desired foundational model to use.
<h2>General Improvements and Bug Fixes</h2>
<ul>
<li>We now support Athena and Oracle databases as data sources.</li>
<li>There's now a complete onboarding guide for new workspaces. It includes instructional videos and tips to help you get started.</li>
<li>We've improved the Slack integration so that it's easier to set up, especially when it comes to channel selection.</li>
<li>The schedules list now shows the notification settings for each scheduled run.</li>
<li>We've updated the notification settings so that it can send PDF attachments containing your documents.</li>
<li>Copying and pasting an image into a text block will now work as expected.</li>
<li>There's a clearer indication of each user's tier right next to the logo on the top-left corner of the app.</li>
<li>There's now a "Create visualization" button on the right of each SQL block. Clicking it will create a new visualization block with the results of the SQL block.</li>
</ul>
Besides those changes, we've done a variety of bug fixes and small performance improvements.

We recently launched dashboards, a feature many of you have been asking for. With that, we also improved our visualizations and AI assistant. Here's what's new.

Changelog #6: Dashboards, better visualizations, and better AI

The first lesson I learned about logarithms is not to mention them when speaking to a large crowd. All the other lessons sucked, so I thought I'd create my own.
Let's start by talking about the simplest of linear scales: a ruler.
Everyone understands how rulers work: you place them next to tiny trinkets and you look at the ruler's marks to determine the object's size. You can then use a pen and the ruler's marks to represent the size of these objects on a board.
<img src="/posts/logs/log1.png" alt="">
The problem with rulers is that you can’t use them to measure larger objects, such as trees, park benches, and city blocks. To solve that problem, we'll need a bit of imagination.
Instead of having each of our ruler's marks represent 1 inch, we'll pretend they represent 100 inches. That way, we'll compress larger distances into smaller ones.
<img src="/posts/logs/log2.png" alt="">
Now, we can use the same ruler to represent the size of a park bench, a tree, and even a city block.
<img src="/posts/logs/log3.png" alt="">
Notice how we're not using the ruler for measuring anymore. Instead, we're using it to represent the sizes of different objects.
The way we squished hundreds of inches into one mark on the ruler is called a linear transformation because when you draw it on a graph, it makes a straight line. Each mark on the ruler represents 100 more inches, showing a steady, even increase.
<img src="/posts/logs/log4.png" alt="">
Linear transformations preserve the measurements' relative spaces.
For instance, if the marks for 1 and 5 inches are 4 segments apart on a ruler, then the marks for 100 and 500 inches will also be 4 segments apart when we use our scaled-up ruler. That way, the proportional distances remain the same, even though the actual measurements have increased.
<img src="/posts/logs/log5.png" alt="">
<img src="/posts/logs/log6.png" alt="">
The problem with applying a linear transformation to a ruler is that it can only either measure small trinkets or large objects, not both.
A ruler scaled from 0 to 10 inches isn't big enough to measure a park bench, while a ruler scaled from 0 to 900 inches makes it hard to distinguish the marks for smaller objects — they become too close together.
<img src="/posts/logs/log7.png" alt="">
To solve this problem, we'll need to transform our ruler so that each segment represents a greater increase.
Typically, going from the first mark to the second represents a 1-inch increase, just like there's a 1-inch increase when going from the second mark to the third. In other words, segments in normal rulers represent linear increases.
In our new scale, going from the first mark to the second represents a 10-inch increase, and going from the second to the third mark represents a 100-inch increase. Each subsequent segment represents a distance ten times greater than the previous. In this case, each segment represents a non-linear increase.
<img src="/posts/logs/log8.png" alt="">
This way, with just 10 marks, you can represent objects from 1 inch to 100,000,000 inches. Each step exponentially increases the range we can measure without cluttering our ruler with countless tiny increments.
Now, you can represent tiny stones at the first mark and a mountain just 10 segments further because, by the time you get to the 10th mark, the value this mark represents hasn't just grown tenfold, it's grown 10¹⁰ inches.
Here's another example using our new scale. If you plot a cockroach in the scale's first mark, a badger in the second mark, and a troll in the third, it means that the badger is 10x larger than the cockroach and that the troll is 10x larger than the badger.
Furthermore, it shows that the troll is 100x larger than the cockroach, because they're 2 segments apart, so 100 (10²) times larger.
<img src="/posts/logs/log9.png" alt="">
Going left has the opposite effect, it causes things to become increasingly smaller. If you keep going left from 1, you'll start working with fractions, and they get 10x as small as you move on to the leftward mark.
<img src="/posts/logs/log10.png" alt="">
As we've just demonstrated, this type of scale is great for representing ratios.
Now compare how the represented values increase exponentially as you move up marks. See how it's not a linear growth like in the previous transformation?
<img src="/posts/logs/log11.png" alt="">
As shown above, each mark represents an exponential increase in the value it represents. Still, this type of scale is not an exponential scale. It's a logarithmic scale.
If you were to plot the actual values against their corresponding logarithmic values, you'd notice that it resembles the plot of a logarithmic function. This resemblance occurs because, as the actual values increase, the increase in their logarithmic values becomes increasingly smaller.
<img src="/posts/logs/log12.png" alt="">
An interesting effect of using the logarithmic scale is that small values get relatively more distant, while larger values get relatively closer.
<img src="/posts/logs/log13.png" alt="">
The reason you should care about this change in relative distances is because it may cause unintended distortions, or, in the case of ill-intentioned people, intended distortions.
For example. Assume that a general council of alchemists doesn't want people to think there's a dragon fever pandemic. In that case, they can plot the number of infected people using a log scale, making it seem linear, when, in fact, it's exponential.
<img src="/posts/logs/log14.png" alt="">
However, there might be compelling reasons behind the decision to use a logarithmic transformation for this chart.
Imagine there are two kingdoms, Hyrule and the Mushroom Kingdom, each with vastly different population sizes. Hyrule has 10,000 inhabitants, while the Mushroom Kingdom boasts 10,000,000.
Now, let's say a disease outbreak occurs in both kingdoms, starting with 10 infected individuals in each. On a regular linear scale, the number of infected individuals in Hyrule might seem insignificant compared to the Mushroom Kingdom's numbers.
<img src="/posts/logs/log15.png" alt="">
If we switch to a logarithmic scale, the picture changes. The slope of the curve representing the increase in infected individuals in both kingdoms becomes comparable. So, when the disease spreads from 10 to 100 infected individuals in Hyrule, the curve's slope remains consistent. Similarly, when the Mushroom Kingdom sees its infected population rise from 10,000 to 100,000, the slope of its curve mirrors that of Hyrule's.
<img src="/posts/logs/log16.png" alt="">
This scaling ensures that the threat level posed by the disease is communicated effectively in both kingdoms, regardless of their size. Without it, Hyrule's situation might appear deceptively less severe simply because of its smaller population. By utilizing a logarithmic scale, public health officials can accurately assess and address the risk posed by the disease in each kingdom, enabling more informed decision-making and resource allocation.
In addition to impacting visualizations, logarithmic transformations are also useful for statistical purposes, mostly because of:
<ol>
<li>Symmetry. Logarithmic transformations may make data more balanced, which is essential for certain statistical techniques to work effectively.</li>
<li>Homoscedasticity. Logarithmic transformations may help keep the spread of data consistent, which helps compare effects on different groups.</li>
<li>Linearity. Logarithmic transformations may make it easier to see straight-line patterns in data, simplifying analysis.</li>
</ol>
<h2>Symmetry</h2>
A graph is symmetric concerning a line if reflecting the graph over that line leaves the graph unchanged.
This histogram of dragon sizes is highly symmetric, for example.
<img src="/posts/logs/log17.png" alt="">
On the other hand, this graph of magic power levels in a sample of healthy wizards is not very symmetric. Instead, it's said to be positively skewed because of its long tail of results to the right.
<img src="/posts/logs/log18.png" alt="">
Now, pay attention to this magic trick: we can make this graph symmetric by applying a logarithmic transformation to all the magic power levels.
Such transformation makes these values more symmetric because the long tail of large values gets compressed, while the small values concentrated on the left get pushed farther apart.
<img src="/posts/logs/log19.png" alt="">
Symmetry is vital in data analysis for various reasons. For example, when using statistical methods, we typically aim to find a mathematical model that accurately represents the distribution of our data. The choice of an appropriate model often depends on whether the data is symmetric.
Anyway, discussing the details of model selection is a topic for another time.
<h2>Homoscedasticity</h2>
In the land of Skyrim, knights rely on magical potions to enhance their strength during battles.
When we graph the effectiveness of these potions based on the initial strength of the knights, we notice greater variability in the potion's impact. This variability is a common occurrence with potions and medicines.
For instance, if a knight initially lifts 100 pounds, the potion might yield effects ranging from no change (100 pounds) to a significant increase (160 pounds, a gain of +60 pounds). Conversely, if a knight can lift 500 pounds, the potion's effects could range from maintaining the initial strength (500 pounds) to a substantial increase (800 pounds, a gain of +300 pounds).
When we plot this data on a linear scale, we notice that the variance increases as the knight's initial strength rises. This phenomenon, known as heteroscedasticity, reflects the heterogeneous nature of the data, where the variability of the potion's effects differs across the range of initial strengths.
<img src="/posts/logs/log20.png" alt="">
When we apply a log transformation to our graphs, it makes the data more homoscedastic, meaning it evens out the variance, especially for stronger knights. In simpler terms, this means that the potion's effects become more consistent across different strength levels.
This transformation reveals that the potion consistently boosts strength by around 0 to 20%, making it easier to predict its benefits for knights at all training levels.
<img src="/posts/logs/log21.png" alt="">
<h2>Linearity</h2>
Let's resort to our imagination again.
Let's say someone's aura indicates one's affinity with celestial magic, and Moonbeam Extract is said to degrade this connection.
Imagine a scenario where our kingdom's alchemists seek to understand the relationship between Moonbeam Extract and someone's aura.
When we plot people's aura's brilliance against Moonbeam Extract, we notice a curved pattern, with most data clustered at the lower left corner. That plot gives us some clues about the relationship between people's auras and Moonbeam Extract, but it's not very clear.
When we employ logarithmic transformations on both magical properties, the correlation appears straight and clear.
<img src="/posts/logs/log22.png" alt="">
This linearity simplifies our understanding, allowing us to see how the effects of Moonbeam Extract clearly impact people's magic abilities.
<h2>A final note on log bases</h2>
Throughout this post, we've mostly used common logs, where each segment represents a tenfold increase compared to the last (log10).
Still, it's worth noting that we can also use other log bases, such as log2 or natural logarithms (logE). These alternative log bases offer different perspectives and may be better suited for specific situations or data types.
<h2>About these visualizations</h2>
We've built all these visualizations using Briefer.
If you want to build reports, presentations, or data apps with visualizations like these, you can sign up for early access to Briefer at <a href="https://briefer.cloud">briefer.cloud</a>.

A brief introduction to logarithmic scales and their applications in data visualization and statistics.

Log scales

This week we've released two major new features that we're very excited about: snapshots and a Slack integration.
<h2>Snapshots</h2>
Snapshots allow you to save the state of your document at any time, and then restore it later.
<img src="/posts/changelog-5/snapshots.gif" alt="">
This is useful when you want to experiment with your code, but you're afraid of losing your progress. With snapshots, you can save your work before making changes, and then restore it if you don't like the changes.
Snapshots are also useful to keep track of how your metrics or visualizations change over time. In Briefer, every scheduled run creates a new snapshot, so you can compare results over time.
To use them, click the "Snapshots" button on the upper-right corner of the editor. Then, click the "Save" button to save the current state of your document. You can also restore a snapshot by clicking the desired snapshot, and then clicking the link to restore it at the top of the page.
Snapshots are only available for users in our professional plan, and their retention period is 90 days.
<h2>Slack Integration</h2>
Our Slack integration allows you to receive PDF reports directly in Slack.
<img src="/posts/changelog-5/exports.gif" alt="">
To use it, you need connect your Slack workspace to Briefer. To do that, access the "integrations" menu at the bottom of our sidebar. Then, click the "Add" button and select "Slack".
After connecting your Slack workspace, add a scheduled run to your document, and, as you're doing it, click the "Add" button right next to the notification settings.
There, you will be able to select the Slack channel where you want to receive the PDF report.
Then, click the "Add Slack" button, and follow the instructions to connect your Slack account to Briefer.
The Slack integration is only available for users in our professional plan.
Important: You need to invite the Briefer bot to the desired channel before it can send notifications (<code>/invite @Briefer</code>). You won't receive notifications if your page fails to execute.
<h2>General Improvements and Bug Fixes</h2>
<ul>
<li>We will automatically execute previous blocks after your environment restarts. That way, you will not face issues with variables not existing anymore.</li>
<li>You can now expand your document horizontally.</li>
<li>Visualizations take 4x less time to load.</li>
<li>We've improved the layout of text blocks. Text will look better now.</li>
<li>We've fixed a few performance issues while switching tabs with multiple editors.</li>
</ul>

We've just released snapshots. They allow you to save the state of your document at any time, and then restore it later. We've also added a Slack integration, so you can receive PDF reports directly in Slack.

Changelog #5: Snapshots and a Slack integration

When we got into YC, we were building a product for companies to spin up development environments in the cloud. The idea was that, as software grows, it inevitably becomes too difficult to run on a single machine. At that point, people would come to us and we'd help each developer an environment in the cloud.
It didn't work.
During our batch, we amassed <a href="https://github.com/briefercloud/layerform">a thousand stars on GitHub</a>, <a href="https://news.ycombinator.com/item?id=37134293">got to HackerNews' front page</a>, and even had a customer pay for our product. Despite the short-term achievements, we soon realized that long-term success was probably not in the cards.
After the batch, we pivoted away from building infrastructure products.
Here's how we made that decision, the lessons we learned, and how I'd approach it differently if I had to do it again.
<h2>Too simple to be useful, or too useful to be simple</h2>
It's extremely difficult to build an infra product that's both simple and flexible enough.
When building an infrastructure product you usually have to choose whether you want to serve small or large companies first. That's because these two types of companies care about diametrically opposite things.
Large companies want flexible products that can support their peculiar infrastructure needs, even if it takes a bit longer to set up. On the other hand, small companies would rather have a simple piece of software that takes only a few minutes to set up, even if it doesn't support esoteric network settings, for example.
If you decide to build an infra product for large companies, it will almost certainly be too complex for small companies to use. Conversely, designing an infra product for small companies will make it too simple for large companies to use.
<img src="/posts/pivoting-away-from-infra/complexity-quadrants.png" alt="">
In an ideal world, you'd be able to hide complexity from small companies, while still supporting the needs of large companies. But in practice, this is nearly impossible to achieve - that's why so few infrastructure products are successful.
<h2>Small companies don't need it</h2>
We knew that selling infrastructure to large companies was going to be difficult, so we decided to focus on building a tiny product that was useful for small companies first.
<img src="/posts/pivoting-away-from-infra/our-quadrant.png" alt="">
Building a small product would allow us to iterate faster because we'd have customers early on, and we'd be able to get feedback from them. Then, we'd be able to raise capital to expand the product and go after larger companies.
At the time, we thought that pull-request previews were a good candidate for an MVP. We could start by deploying preview environments for pull requests, and then we would expand the product so that people could use these environments for development too.
The first version of our product was simple. It used a <code>docker-compose.yml</code> file to deploy pull requests and then posted a comment on the pull request with a link to the preview deployment.
<img src="/posts/pivoting-away-from-infra/pr.png" alt="">
Just a few days after we put a landing page up, a small company found us and started using the product. As soon as they hit the free tier's limits, they reached out to pay.
We then tried to replicate that success with other companies, but it was extremely difficult. That's because a small company's software is usually small. Thus, they can just check out the branch and get it running in a few seconds. Having preview links was convenient, but not game-changing enough to make them pay.
The same principle applies to other infrastructure products. Even if your software is pretty simple, small companies won't care about it unless it's absolutely essential for them. And even then, they might not want to pay for it, and might just default to a free alternative.
<h2>Large companies won't trust you</h2>
After our small bet failed, we decided to rebuild our product to serve large companies, but we had a hard time getting them to trust us.
When deciding whether to purchase a product, buyers usually have the following equation in mind:
<img src="/posts/pivoting-away-from-infra/value-equation.png" alt="">
The higher the value, the more likely they are to buy.
The problem with infrastructure products is that the time-to-value and implementation effort variables are usually too high, while the perceived probability of success is too low. Consequently, no matter how amazing the dream outcome is, the final value is usually too low for buyers to move forward.
Now, let's dissect each of these variables and explain why the perceived probability of success is usually too low, and why time-to-value and implementation effort are usually too high.
<h3>Perceived probability of success</h3>
<img src="/posts/pivoting-away-from-infra/perceived-probability-of-success.png" alt="">
Imagine you're running a multi-billion dollar company. Would you trust a scrappy 2-person startup to handle the infrastructure that keeps your engineers productive? I wouldn't.
Sometimes, not even large providers like AWS or Google Cloud can support these companies without a lot of hand-holding. So, why would they trust a 2-person startup?
At large companies, not rocking the boat is usually more important than getting the latest and greatest technology. Consequently, they won't trust you unless you have solid proof that you're at least 10x better than the current alternative.
<h3>Effort</h3>
<img src="/posts/pivoting-away-from-infra/effort.png" alt="">
Creating an infrastructure product that demands little effort to implement is extremely difficult because of the very nature of the problem you're solving. You're building a product that has to be flexible enough to support all the unique needs of large companies, and that's usually complex, as we've seen earlier.
Just for engineers to understand how your product works, they'll have to spend a lot of time reading documentation, watching videos, and maybe even attending training sessions.
Furthermore, it's difficult to break down the problem into smaller pieces that can be solved independently. That's because the problems you're solving are usually interconnected, and solving one might create new problems that need to be solved.
Finally, considering all this effort, they'll probably default to having their own engineers build it, instead of buying your product. That's because they'll have more control over the final product, and they'll be able to customize it to their unique needs.
Ah, did I mention they hired tons of engineers who need to justify their next promotion?
<h3>Time-to-value</h3>
<img src="/posts/pivoting-away-from-infra/time-to-value.png" alt="">
In addition to all the time engineers will have to spend learning how to use your product and set it up, it will take a while for them to see the productivity gains you promised.
That's because infrastructure products tend to deliver compounding value over time instead of delivering acute value upfront.
Take CI/CD products, for example. Testing and deploying your software for a month isn't that painful and doesn't take that much time. However, if you have to do it for a year, it becomes a huge pain point, and you'll want to automate it.
<h2>Slow feedback loops</h2>
The more feedback you have, the smaller the bets you have to make, and thus, the less risk you have to take.
Think about buying 3-digit lottery tickets, for example. Paying $1 for each digit as numbers are drawn is better than paying $3 for all three digits at once.
If you have to buy all three digits at once, the likelihood of guessing all three is only 1%, and each attempt costs you $3.
<img src="/posts/pivoting-away-from-infra/lottery-3-digits.png" alt="">
On the other hand, if you pay $1 per digit, you can decide whether to continue as soon as the first number is drawn. If you get it right, you can pay for the next; if not, you can give up sooner and lose less money.
In this case, there’s only a 10% chance you’ll buy the second number and a 1% chance you’ll buy the third. Therefore, each failed attempt will cost you less, but the overall likelihood of winning is still the same.
<img src="/posts/pivoting-away-from-infra/lottery-1-digit.png" alt="">
Building an infra product is like buying all three digits at once. You have to make a large bet upfront, and you won't know if you're building the right thing until you get your first customer. That's because it's difficult for large companies to trust you, and, even when they do, slow sales cycles and long implementation times will elongate the feedback loop.
Additionally, it will take longer to have a large enough sample size to know if your product is any good, and to understand whether a request is a one-off or a recurring need for most customers.
<h2>How I would approach it differently if I had to do it again</h2>
When picking a startup idea, you're also picking a set of problems that come with it. Ultimately, all problems are solvable, but some are more difficult than others for a given team.
Even though I'm glad we pivoted away from infrastructure products, I still believe they're a great opportunity for the right team, so here's how I'd approach it differently if I had to do it again:
<ul>
<li>I'd insist on selling to small companies first. Instead of investing time trying to sell to large companies, I'd invest time in rethinking the product so that it's useful for small companies. That's a piece of advice we got from PG himself, and it was spot on.</li>
<li>I'd accept having to make larger bets and act accordingly. Given the slow feedback loop, I'd be more inclined to think long-term and make larger bets, like building more features upfront and building a community around the product, even if they're not paying customers, and probably won't be for a while (or ever).</li>
<li>I'd cut scope as much as possible and avoid being seen as a replacement for existing solutions. I think this point is valid for all startups, but it's especially true for infrastructure products because switching infrastructure usually takes longer than anything else. Therefore, I'd think harder about whether I can build something small and less critical to go alongside existing solutions.</li>
</ul>
<h2>How these learnings helped us pick our next idea</h2>
The worst part of building an infrastructure product was the slow feedback loop, so our first criteria for picking our next idea was to find a problem that we could offer a packaged solution for, and that we could sell to small and medium companies first. That solution also had to be easy to use, and the time to value had to be as short as possible.
Considering these criteria, we then brainstormed a few hypotheses and ran approximately fifty customer interviews to validate them.
We then landed on what we're doing today: a product that helps companies generate unique insights by using code.
<img src="/posts/pivoting-away-from-infra/demo-short.gif" alt="">
The reason Briefer turned out to be a great fit for us is that:
<ul>
<li>Time-to-value is extremely short. Instead of spending days reading documentation and setting up the product, customers can just log in and start generating insights.</li>
<li>It appeals to both small and large companies. Small companies usually love Briefer because their engineers can write code to generate insights, and they don't have to wait for a data team to do it for them. Large companies love it because they can use it to build more advanced analyses, data apps, and predictive models.</li>
<li>We don't have to replace existing solutions. While some companies replace their BI tools with Briefer, most use it alongside them because traditional BI tools can't generate the insights Briefer can, nor can they be used to build data apps or predictive models.</li>
</ul>
In the end, we're glad we went through the infrastructure journey, even though it was tough. It taught us a lot about building products, and it helped us land on a better idea.
I'm extremely thankful for the YC partners, our batchmates, and the YC alumni who helped us along the way. Many of these learnings wouldn't have been possible without their help.
Finally, if you want to try Briefer, or even if you're not sure it's the right choice, please feel free to <a href="https://briefer.cloud">visit our website</a> and <a href="https://calendly.com/lucasfcosta/briefer-intro-call">book an intro call with us</a>. We'd love to hear from you.

Here's the story of how we pivoted after our YC batch, the lessons we learned building in the infrastructure space, how I'd approach it differently if I had to do it again, and how these learnings helped us build Briefer.

What we learned from a failed infrastructure DevTool

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.
<img src="/posts/p-values/honeyberry-method.png" alt="">
<h2>Step 1: Understanding p-values so you can misuse them</h2>
Your first step to riches is to create a technique that you can claim to improve any aspect of your customers' lives and that you can write books about.
It doesn't matter if your technique actually works. What matters is that you can convince people that it does, and that's what I'm here to teach you.
Let's say you've created a new study technique and you believe it helps students score better on exams, for example.
To test your hypothesis, you set up an experiment with two groups of students:
<ul>
<li>Control Group: This group studies using traditional methods.</li>
<li>Experimental Group: This group uses your revolutionary study technique.</li>
</ul>
Then, you give both groups the same exam and compare their scores. If the experimental group scores significantly higher, some people might conclude that your study technique is effective.
<img src="/posts/p-values/two-groups.png" alt="">
Now, that's where amateurs usually stop. But you're not an amateur, you're a business person. You need to convince even the nerdiest of your customers that your technique is backed by science.
These people like to think they are smart, so they'll definitely ask how do you know whether the difference in scores is real or just due to chance.
That's where p-values come in. The p-value is a statistical measure that determines how likely it is that the improvements in scores were due to random chance. In other words, p-values represent the chance of the same or more extreme scores happening even if your new study technique doesn't actually work.
A small p-value means that a difference in scores as extreme as the one you observed is unlikely to have occurred by mere luck. Thus, a small p-value suggests that your study technique might be effective.
For example, a p-value of 0.001 means there's only a 0.1% chance that the score improvement was just random. This suggests that the better scores could be due to your new study technique, not just chance.
On the other hand, a large p-value means the difference in scores could easily have happened by chance, and your study technique might not be effective.
For example, a p-value of 0.15 means there's a 15% chance that the score improvement was random. Having such extreme results 15% of times would lead nerds to say that your results are not "statistically significant", because these events are actually not that rare. In other words, they mean that the difference in scores could easily have happened by chance, and your study technique might not be effective.
<blockquote>
The way that classical statistics works is by comparing study data to what is expected when there is nothing. If the data are not typical of what is seen when there is nothing, there must be something!
— Dallal, Gerard. The Little Handbook of Statistical Practice.
</blockquote>
Having learned what p-values are, you probably figured that what we're after is a small p-value. So let's go ahead and make sure we get one.
<h2>Step 2: Ensuring that <code>p</code> is small enough</h2>
Let's be honest: the chances of your revolutionary study technique actually working are slim. Consequently, your experiment will probably fail to demonstrate that the new study technique is much different from just doing whatever everyone is already doing.
Once your study fails, you have two options. The first is to repeat the experiment until it works. The second is to analyze the data in multiple ways until you find another measure that looks significant. Then you can rebrand your technique as being effective for that measure, like "improving memory retention" or "increasing focus".
<ul>
<li>Pro-tip 1: if none of the measures look significant, you can always try to find a subgroup of the population that benefits from your technique. For example, "our revolutionary study technique is particularly effective for left-handed students born in the summer".</li>
<li>Pro-tip 2: you can also find transformations of the data that make it look significant, like the sum of outcomes squared or the log of the difference between scores.</li>
</ul>
<img src="/posts/p-values/python-honeyberries.png" alt="">
This step is pretty easy to memorize because both alternatives rely on the fact that if you look around enough, you'll eventually find something that looks like a significant result. Another way to put it is that "the chances of winning the lottery are small, yet often there's a winner".
<h3>A crucial detail: make sure p is smaller than 0.05</h3>
There's a convention among scientists that says that a p-value below 0.05 is said to be "statistically significant". So that's what you should aim for, at least most of the time.
There are many theories about why statistical significance is set at 0.05, but, as Gerard Dallal points in his book "The Little Handbook of Statistical Practice", all of them trace back to Karl Pearson's 1914 book "Tables for Statisticians and Biometricians".
<img src="/posts/p-values/karl-pearson-book.jpeg" alt="">
Pearson figured out how likely different outcomes were in statistics. Then Fisher made it easier by creating simple tables in his book "Statistical Methods for Research Workers" in 1925. These tables just showed basic probabilities for certain outcomes. Fisher's idea caught on, and he did the same thing with Frank Yates in 1938. People still use Fisher's tables in statistics texts today.
Fisher's tables were shorter and simpler than Pearson's. Instead of giving lots of details, they just gave rough probabilities. Fisher also came up with a neat way to show how sure we are about results—using stars. One star meant a result observed about 5% of the time, two stars meant it was observed about 1% of the time, and sometimes three stars meant it was observed about 0.1% of the time.
Fisher also said that when these extreme results would only happen 5% of the time without an intervention, we should take notice. He thought this level was good because it helped us find real results without getting too many false alarms. And this 5% rule has stuck around, shaping how we understand data and what we consider important in science.
In one sense, 5% is just a number, and it's not a magic number. But it's a good rule of thumb. Still, some people think it's too high, and that we should use lower levels like 1% or 0.1%.
To make matters simpler, I'd also recommend that you keep this excellent table from the scientific website <code>xkcd</code> handy:
<a href="https://xkcd.com/1478"><img src="/posts/p-values/xkcd-1478.png" alt=""></a>
<h2>Step 3: Publishing your results</h2>
Now that you have a small p-value, you need to call journalists and tell them about your revolutionary study technique. They'll be thrilled to write about it, especially if you can provide them with a scientific paper that includes the words "p-value" and "statistically significant".
The important thing here is not to mention that you've run your experiment 100 times until you got a significant result, nor that you've written a python script to test all possible variables until you found one that looks significant.
<img src="/posts/p-values/honeyberry-news.png" alt="">
Another important thing is to explain p-values to them in a way that doesn't sound egregious, but that is not entirely right either.
One incorrect way to explain p-values is to say that they represent the probability that the "null hypothesis" (your study technique not working) is true.
Let's say you've found a p-value of 0.001, for example. In that case, make sure to tell them that there's only a 0.1% chance that your study technique is ineffective. Although that's incorrect, it's much more convincing than saying "if the new study technique didn't work, we would still see the results we did approximately 0.1% of the time".
Another technique that you can use is to invert the logic. Instead of saying that a p-value of 0.001 means there's a 0.1% chance that the score improvement was random, they say that there's a 99.9% chance that the new study technique is effective. That's not correct either, but it's looks better in a headline.
<h2>Step 4: Rebuking the skeptics</h2>
Once your revolutionary study technique is published, you'll probably face some skepticism. That's normal, and you should be prepared to deal with it.
The scientists are the most dangerous skeptics. They'll probably try to replicate your experiment and fail to find the same results.
If it's only one or two scientists, you can always say that "failing to find an effect is different from showing that there is no effect". They won't be able to argue with that because you're technically correct.
Now, if quite a few scientists fail to replicate your results, then you're in deep trouble. <a href="https://en.wikipedia.org/wiki/Replication_crisis">That's why it's best to design experiments that are hard to replicate, like those that require a lot of data or that are too vague to be disproven</a>.
In general, keep in mind that the best way to deal with skepticism is to prevent it from happening in the first place.
<h2>Putting it all together</h2>
<ul>
<li>P-values are a measure of how likely it is that you'd observe the results you did if your intervention didn't actually work.</li>
<li>The smaller your p-value, the less likely you are to have observed the results you did by mere luck.</li>
<li>Conversely, the larger your p-value, the more likely it is that you'd observe the results you did by mere luck.</li>
<li>A p-value is not the probability that the null hypothesis is true. The opposite is also true: a p-value is not the probability that your intervention is effective.</li>
<li>Just because you haven't observed an effect, it doesn't mean there isn't one.</li>
<li>If you run the same experiment many times, you'll eventually see a p-value below 0.05, even if the null hypothesis is true.</li>
<li>Looking for an effect until you find one is an effective way to make sure you get a small p-value, but it's bad practice unless you're looking for an easy way to sell books.</li>
<li>A p-value below 0.05 is usually considered "statistically significant". Although it's not a magic number, it's a rule of thumb that has been around for a while.</li>
</ul>
<h2>Before we go</h2>
In case you didn't notice, this post is a satire, and any parallels to the real world are, obviously, coincidence (no pun intended).
The post was inspired by the excellent work of <a href="https://xkcd.com">Randall Munroe</a>, in particular his comic <a href="https://xkcd.com/882">xkcd 882</a>:
<a href="https://xkcd.com/882"><img src="/posts/p-values/xkcd-882.png" alt=""></a>
Finally, if you're a data-scientist who loves jupyter notebooks or data apps, but finds them hard to share your work with others or too much hassle to deploy, you should check out <a href="https://briefer.cloud">Briefer</a>.

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.

What p-values are, what they are not, and how to sell books with bad statistics

This changelog is the most important one we've shipped so far - that's why it took more than two weeks for us to post it.
In these last few weeks, we've completely rebuilt our editor so it's more understandable, more reliable, and more flexible.
We've also introduced environment variables, which allow you to store information like settings, API keys, and database credentials securely. Then, you can reference these variables in your code without exposing them.
<h2>Our new editor and a ton of related improvements</h2>
Before this change, our editor was more unstable than we'd like. Sometimes the editor would show different states than what was happening in the back-end. Refreshing the page also caused the editor to lose part of its state, like which blocks were running, or undergoing AI suggestions. All these issues are now gone, and we're glad to have received lots of positive feedback about the new editor.
The new editor we're rolling out for everyone today is more reliable, more flexible, and more understandable. We now manage and reconcile every single piece of state in the back-end. Consequently, the editor's state will be consistent across refreshes, and for all users and devices.
<img src="/posts/changelog-4/v2editor.gif" alt="">
This change also comes with a bunch of very important improvements. Here are some of them:
<ul>
<li>When using data apps, each viewer will have their own state. That means that if you're viewing a document and someone else is viewing it at the same time, you won't see each other's inputs and results.</li>
<li>The "Run All" button now shows the number of blocks that have already run, and how many are left.</li>
<li>We explicitly display enqueueing and running states for each block.</li>
<li>When a scheduled run is happening as you view a document, you'll see its results in real time.</li>
<li>You can now duplicate whole blocks or individual tabs by clicking the drag-handle menu on the left side of the block.</li>
<li>You can run an entire block from left to right by clicking the "Run all tabs" button that appears within the drag-handle menu for multi-tab blocks.</li>
<li>Data apps will only be updated when you click the "Update Data App" button. This way, you can make multiple changes to the inputs and dropdown fields of a data app and only publish them when you're ready.</li>
<li>Adding new blocks is now more intuitive. You can now add blocks by clicking the "+" button that appears when you hover over gaps between blocks.</li>
<li>You can now group all types of blocks, not only Python, SQL, and Visualizations.</li>
<li>If you create a new dataframe that only exists in a Python block, it will be immediately available within visualization blocks.</li>
<li>You don't have to uncollapse a block to run it anymore. You can now run a block by clicking the "Run" button that appears when you hover over the block.</li>
<li>We'll now persist the open/close state of the document tree on the left as you navigate between documents. This way, you won't have to keep opening nested folders every time you navigate to a different document.</li>
</ul>
<h2>Environment variables and how to use them</h2>
Environment variables are a way to store information like API keys, database credentials, and other settings securely. That way, you can then reference these variables in your code without exposing them to viewers.
These environment variables are useful when you want your Python blocks to perform authenticated API requests, like when doing writebacks to your database, for example.
To create an environment variable, click the "Environment Variables" button on the bottom-left corner of the editor. Then, click the button at the bottom of the list to add a new variable. Once you're done, click the save button and restart your environment for the new environment variables to be available.
You can then reference this variable in your Python code by using the <code>os.getenv</code> function.
<img src="/posts/changelog-4/v2vars.gif" alt="">
By the way, we also added a button on the bottom right of the editor so you can restart your environment manually if you need that.
<h2>What's next?</h2>
We know it's been somewhat annoying to lose your in-memory state when your environment restarts. We're working on a way to automatically re-run dependencies or persist this state across restarts, so you won't lose your dataframes, variables, and other in-memory state when your environment restarts.
Expect that by the next changelog.

This changelog is the most important one we've shipped so far - it includes a brand new editor and support for environment variables.

Changelog #4: Environment variables, and a brand new editor

Besides the usual bug fixes, we're shipping a major new feature this week: data apps.
You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.
<img src="/posts/changelog-3/data_apps.gif" alt="">
Please notice data apps are only available for users in the professional tier.
<h1>When to use data apps features</h1>
You should use data apps features whenever you want viewers to be able to provide input to your reports, models, or analyses.
In a nutshell, data apps are interactive documents that allow viewers provide input to the code you've written.
Typical examples include:
<ul>
<li>Comparing scenarios: Allow viewers to change assumptions and see how the results change.</li>
<li>Exploring data: Allow viewers to change the data they're looking at.</li>
<li>Filtering data: Allow viewers to filter data by date, region, or any other dimension.</li>
</ul>
<h1>How data apps work</h1>
To create a data app, you must first add an input or select field to your document by using the <code>/input</code> or <code>/select</code> commands.
For each of those blocks, you can specify a label. The variable name is the one you see on the top right corner of the block in a green box.
Then, you can reference the variable name in your Python code in the same way you would reference a variable you created yourself.
To interpolate the value of the input or select field in SQL blocks, use the <code>{{variable_name}}</code> syntax.
<blockquote>
Please notice that you can now interpolate any variables in your SQL blocks, not only the ones created by input or select fields.
</blockquote>
Finally, use the "Make it a data app" toggle on the top right corner of your document, under the share menu, to enable the data app mode.
Toggling the data app mode allows viewers to provide input to your document, but they still can't edit the document's content.
<h2>Other fixes and improvements</h2>
<ul>
<li>We fixed a bug where users would not be automatically logged out once their session expired. Now, users will be imediatelly logged out and redirected to the login page when their session expires.</li>
<li>Document snapshots: we now keep snapshots for users in the professional tier. These snapshots are taken every five minutes and will be available for the last 5 days. The last 10 snapshots are always available, regardless of how old they are.</li>
<li>We've done updates to the visualizations so they support data types they used to not support.</li>
</ul>

This week, we're introducing data apps. You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.

Changelog #3: Introducing data apps

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements.
If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.
<h1>General Updates</h1>
Here are a few small improvements we've done to the editor, and to the overall performance of Briefer.
<h2>Editor performance improvements</h2>
We noticed that performance degraded as documents got larger. We've made a few changes to the way we handle large documents.
First, we've eliminated lots of unnecessary re-renders for graphs and tables. This should make the editor feel snappier, especially for large documents.
<h2>Support for larger datasets</h2>
Some users dealing with millions of rows of data were experiencing performance issues.
To fix that, we now run all the queries in your own compute instance and stream the results back to the editor. This change means that we can now support much larger datasets without performance issues. It also means that the users who pay for larger compute will experience more benefits from it.
<h2>Improvements</h2>
Besides the performance improvements we shipped a few minor UI updates. Besides all the small UI tweaks we've done, here are a few highlights:
<h3>In/Not-in filters</h3>
We've added support for <code>in</code> and <code>not-in</code> filters in visualization blocks. This is useful when you want to filter by multiple values at once.
<img src="/posts/changelog-2/in_not_in_filters.png" alt="">
<h3>New query on visualizations</h3>
You can now create queries more easily when using visualizations. Just click on the "New query" option the dropdown, or in the tooltip that appears when you create an empty visualization.
<img src="/posts/changelog-2/new_query_option.png" alt="">
You can also create new queries or tabs by using the <code>+</code> button in the top right corner of the block.
<img src="/posts/changelog-2/plus-tab-button.png" alt="">
<h3>Deleting tabs or entire blocks</h3>
You can now delete tabs or entire blocks by clicking on the drag-handle and selecting the desired delete option.
<img src="/posts/changelog-2/delete_tab_or_block.png" alt="">
<h3>New users are viwers by default</h3>
Previously, when allowing anyone from <code>@yourdomain.com</code> to access Briefer, new users were added as editors by default. We've changed this so that new users are added as viewers by default. This is to prevent new users from accidentally making changes to the workspace.
<h1>Bug fixes</h1>
Bug fixes are the biggest part of this release. We've fixed a ton of bugs, but here are a few highlights:
<ul>
<li>Fixed a bug where there was a race condition when creating tokens for compute instances. This was causing some users to experience multiple "something went wrong" alerts that would only go away after the environment got restarted.</li>
<li>When changing the filters or settings for visualizations multiple times, we'd enqueue each update. This caused visualizations to be slow to update. We've fixed this so that only the latest update is enqueued and previous updates are discarded.</li>
<li>Our AI suggestions were sometimes suggesting overwriting dataframes using <code>[Object object]</code> dataframes. We've not only fixed this bug, but we also improved the LLM's context.</li>
<li>Creating new tabs was causing an empty block to appear in the editor until you switched to another tab and back. This was a visual bug that didn't affect the actual document, but it was annoying - and now it's fixed.</li>
<li>Dragging and blocks into one another was not working properly. Sometimes, the editor would try to drag the wrong block, or it would not detect you wanted to group different blocks. This bug is now fixed.</li>
<li>Editing AI suggestions and then accepting them was causing the edited diff not to be applied. Instead, it applied the raw AI suggestion. We fixed this so that any changes you do in the diff editor will now persist in case the suggestion is accepted.</li>
<li>Display of titles was showing incorrect background colors during loading and disabled states. They were also not taking the appropriate space. We've done fixes to their appearance so they behave correctly in all cases.</li>
<li>Visualizations were not being updated in scheduled runs. This was a bug that was causing some users to receive outdated visualizations in their scheduled reports. This is now fixed.</li>
</ul>

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements. Here are some of the highlights.

Changelog #2: Blazing fast performance and tons of bug fixes

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently.
This is a non-exhaustive list of everything that's changed. As we publish more updates, we'll increase the level of detail here.
If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.
<h1>New features</h1>
This week: no-code visualizations, tabs (grouping blocks), naming blocks, workspace settings, and exports.
<h2>1. No-code visualizations 📊</h2>
The most important change comes first: you can now create graphs without code. For that, type <code>/visualization</code> and add a visualization block.
<img src="/posts/changelog-1/visualizations.gif" alt="">
There, you'll be able to select the query to plot, types of graphs, filters, and more.
Once you're done with your visualization you can click the double-chevron icon to toggle between edit and visualization modes. In visualization mode, only the graph and filters appear.
Please notice that histograms and pie charts are not available yet.
<h2>2. Grouping blocks 🫂</h2>
You can now group multiple blocks into one to create multi-tabbed blocks.
Grouping blocks is useful when you need multiple python blocks to clean data, but only want a single python block with the results to show.
<img src="/posts/changelog-1/grouping.gif" alt="">
You can also drag-and-drop tabs to change their order. To ungroup tabs, drag the desired tab out of the tabbed block.
<h2>3. Naming blocks 🪄</h2>
Now that you can group blocks, it's important that you name them so that you know what's in each tab.
Naming blocks is also helpful for presenting reports - it helps stakeholders understand what they're seeing.
<img src="/posts/changelog-1/naming.gif" alt="">
To name blocks, just click the title of each block and input whatever title you want.
<h2>4. Workspace settings ⚙️</h2>
We heard a few people wanted to change their workspace names or settings after the workspace had been created. That wasn't possible before.
Now, you can rename your workspace and enable everyone with the same email address to join.
<img src="/posts/changelog-1/workspace-settings.gif" alt="">
<h2>5. PDF exports 📑</h2>
We know many of you were asking for this, so we have shipped it.
You can export PDF files by opening the share menu on the top right corner and then clicking the PDF export option.
For now, PDF exports are only enabled for users in the professional tier.
<img src="/posts/changelog-1/pdf-exports.gif" alt="">

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently. This week: no-code visualizations, tabs, and exports.

blog

Multiplayer notebooks

Recent posts