| home 	| blog 	| resume 	| tags 	|
|:---:	|:---:	|:---:	|:---:	|


Improving your user's experience with AI assistants without messing with the models.

Table

You’re staring at a messy dataset, deadlines looming, and no bandwidth for deep analysis. Sound familiar? With generative AI for data visualization, you can skip the grunt work and get straight to the patterns that matter. This post walks through a full analysis using natural language prompts, with no scripts or BI dashboards in sight.

## Why Generative AI Is a Game Changer for Data Visualization

Most data workflows are optimized for people who code, not people with questions. Even simple questions require wrangling CSVs, writing Python, or clicking through dashboard builders. Generative AI offers a faster path: type what you want to know, and get a clear answer plus the chart to prove it.

We used Briefer for this walkthrough. It's a data workspace with a built-in agent that lets you talk to your dataset. Ask a question like "How does sleep affect exam scores?" and the agent returns both an explanation and a chart. The whole process feels more like pair programming with your data than traditional analysis.

## What follows is a series of questions and responses based on a real dataset of 80,000 students. Each answer comes with a generated visualization and just enough interpretation to guide the next step.

## Exploring the Dataset

We began with a simple instruction:

### Describe the dataset

![Agent prompt](/posts/generative-ai-for-data-analysis/agent.webp)

Here's what the agent returned:

![Agent answer](/posts/generative-ai-for-data-analysis/agent_answer.webp)

> The enhanced student habits performance dataset is a comprehensive collection of data for 80,000 students, containing 31 variables with no missing values. The dataset encompasses various aspects of student life and academic performance, including demographic information (such as gender and age), lifestyle habits (including sleep patterns, exercise frequency, social media usage, and screen time), study behaviors (like study hours and consistency), and academic performance metrics (measured through grades, test scores, and GPA). This rich dataset enables analysis of relationships between students' daily habits and their academic outcomes, facilitating insights into which behaviors most strongly correlate with educational success.

This overview gave us a strong foundation: a balanced dataset with rich behavioral variables, ready to be explored from multiple angles.

We began with a simple but essential question about study habits:

### What is the average exam score based on study hours per day?

![The average exam score based on study hours per day.](/posts/generative-ai-for-data-analysis/study_hours.webp)

> The visualization shows average exam scores for each major across different study hour categories. Similar to the gender breakdown, all majors show a consistent positive correlation between study hours and exam performance. Engineering students tend to achieve the highest scores in the higher study hour categories, while Arts majors show slightly lower scores compared to other majors. However, the pattern of improved performance with increased study time is consistent across all academic disciplines.

This confirmed what we might expect—but seeing the trend hold across all majors gave us more confidence. It also hinted at the strength of discipline and focus in academic performance.

With screen time often blamed for distraction, we posed the next question with skepticism:

### How does screen time correlate with exam scores?

![Correlation between screen time and exam scores.](/posts/generative-ai-for-data-analysis/screen_time.webp)

> I've analyzed the relationship between screen time and exam scores in the student dataset. The data shows a positive but moderate correlation (r = 0.17) between screen time and exam performance. Students with higher screen time tend to have slightly higher exam scores, with those in the 8-10 hour range scoring highest (90.1 points on average). The difference between the lowest and highest performing groups is about 11%. This is somewhat counterintuitive, as one might expect excessive screen time to negatively impact academic performance. It's possible that screen time in this dataset includes educational content, online learning, or research activities that contribute positively to academic outcomes. It could also suggest that students who are more engaged with technology may have better access to resources or more efficient study methods.

Here, generative AI helped not just with correlation but interpretation—pointing out possible reasons behind the surprising trend, which we might have missed without the assistive narrative.

Curious about another lifestyle factor, we turned to rest:

### Analyze how sleep hours affects exam score

![Optimal sleep duration.](/posts/generative-ai-for-data-analysis/optimal_sleep_duration.webp)

> In this analysis, I focused on common sleep durations (4–10 hours) and found that 8.5 hours of sleep is associated with the highest exam scores (about 89 points on average) among common sleep durations. The visualization shows a clear trend where performance improves as sleep duration increases from 4 to 8.5 hours, then begins to decline slightly with longer sleep durations.

![how sleep hours affects exam scores.](/posts/generative-ai-for-data-analysis/sleep_duration.webp)

> Most students in the dataset get between 6–7 hours of sleep, which is below the optimal amount for academic performance. This suggests that encouraging students to get closer to 8 hours of sleep could improve their exam scores. The difference between sleeping 6 hours and 8.5 hours corresponds to approximately 2–3 points on exam scores, which could be significant for grade boundaries.

The recommendation was subtle, yet actionable: more rest, better results—up to a point. To better understand this relationship, we followed up:

### Make a quick comment about the causality in this relation

> While our analysis shows a clear relationship between sleep duration and exam performance, with students sleeping 8–8.5 hours achieving the highest scores, it's important to note that correlation doesn't necessarily imply causation. The observed pattern could result from multiple mechanisms: sleep directly enhancing cognitive functions essential for academic performance; confounding variables where students with good sleep habits also have other beneficial study habits; or common underlying factors affecting both sleep quality and academic preparedness.

> Although we can't definitively establish causation without randomized controlled experiments, the consistent pattern observed across thousands of students aligns with extensive neuroscience research on sleep's role in memory consolidation and cognitive function. This suggests that adequate sleep likely contributes to academic success, even if it's part of a more complex relationship involving multiple factors that collectively influence student performance.

This caveat grounded the insight in real-world nuance—and hinted at the importance of thoughtful interpretation when reading data relationships.

To close the loop, we asked a broader question:

### What are the top three predictors of exam score?

> Based on my analysis of the student dataset, the top three predictors of exam scores are:

> 1. Previous GPA: With a very strong correlation of 0.93, this is by far the strongest predictor of exam performance. This indicates that past academic achievement is highly indicative of future success.
> 2. Study Hours Per Day: With a correlation of 0.25, time spent studying shows a moderate positive relationship with exam scores. Students who dedicate more hours to studying tend to achieve better results.
> 3. Motivation Level: With a correlation of 0.25, a student's self-reported motivation level is equally important as study hours in predicting academic performance.

![Top predictors of exam score.](/posts/generative-ai-for-data-analysis/top_predictors.webp)

> The regression analysis confirms these findings and shows that these three factors, along with other variables, explain approximately 87% of the variance in exam scores. This suggests that academic success is largely determined by a combination of past performance (GPA), effort (study hours), and intrinsic factors (motivation level).

![Regression analysis by predictors.](/posts/generative-ai-for-data-analysis/top3.webp)

The result was both validating and enlightening. GPA dominance was expected—but motivation and study time had similar weight, reinforcing the role of mindset alongside effort.

## From Curiosity to Clarity

Working through this dataset with a conversational agent made analysis feel lightweight. Every time we asked a question, we got both a thoughtful explanation and a [solid visualization]($https://briefer.cloud/blog/posts/look-the-data/$). There was no context-switching between environments, no writing boilerplate code, and no time spent setting up plots that don’t scale.

The result was faster understanding, better focus, and surprisingly practical takeaways. This approach doesn’t replace traditional methods for deep statistical work, but it’s an excellent way to unblock early exploration and move ideas forward without friction.

If you're ready to explore your own data with this kind of clarity and speed, try it yourself. Create clean analysis and dashboards 10x faster. **[Try Briefer for free](https://briefer.cloud/pricing/)**


Discover how generative AI for data visualization turns exploration easier than ever. Ask questions, get plot and build dashboards instantly.

Generative AI for Data Visualization: See What You’ve Been Missing

Modern data engineering workflows turn chaotic pipelines into cohesive systems. Without them, teams find themselves tangled in brittle scripts, chasing mysterious data bugs, and responding to yet another "why is this dashboard blank?" message at midnight. This post unpacks how to design workflows that are reliable, maintainable, and most importantly, give your team some peace of mind.

## Why data engineering workflows matter

The explosion of data sources, formats, and consumer expectations has made informal, ad-hoc pipelines an unsustainable strategy. Gone are the days when nightly ETL jobs into a single data warehouse sufficed. Today’s workflows must contend with SaaS apps, semi-structured data, real-time streaming, and machine learning pipelinesm, all under pressure to deliver accurate insights rapidly.

A well-designed workflow doesn’t just move data from one place to another; it defines the velocity, reliability, and ultimately the credibility of data-driven decisions within the organization. Without structure, you’re not running a workflow: you’re automating chaos.

## Anatomy of a modern data engineering workflow

![Data Engineering Workflow with Orchestration](/posts/data-engineering-workflow/image.webp)

### Ingestion: where the battle begins

At the core of this structure lies ingestion, the process of getting data from various systems into a centralized environment. This is where the battle against entropy begins. For many teams, managed ingestion tools offer the most pragmatic path forward. <a href="https://airbyte.com/" target="_blank">Airbyte</a>, for instance, provides an open-source alternative that balances flexibility with ease of use. Its pre-built connectors and modular design let teams standardize ingestion without locking into a proprietary system.

While some organizations turn to streaming platforms like Apache Kafka for low-latency event processing, it’s worth pausing to ask whether you really need to operate such heavyweight infrastructure. In most cases, simpler batch ingestion suffices and keeps complexity in check.

### Storage: where data lives

Once ingested, data needs a reliable and scalable home. This is where the modern cloud data warehouse steps in, offering separation of storage and compute, automated scaling, and SQL compatibility for broad accessibility. Snowflake exemplifies this new paradigm, making it possible for multiple teams to work concurrently without bottlenecks or performance degradation.

Google BigQuery offers a similar model, but the choice between them often comes down to cloud alignment and cost structures. Regardless of the platform, the goal remains the same: store raw and transformed data in a way that supports both analytical flexibility and operational efficiency.

### Transformation: unlocking analytical value

With data securely stored, transformation becomes the next crucial phase, arguably where the real analytical value is unlocked. Historically, transformations occurred before loading, but modern workflows embrace ELT, performing transformations inside the warehouse itself.

This shift has propelled tools like dbt to the forefront. <a href="https://docs.getdbt.com/" target="_blank">dbt enables modular, version-controlled SQL transformations</a> that are rigorously tested and automatically documented. By promoting a software engineering approach to data modeling, dbt reduces the risk of rogue SQL proliferating across the organization, ensuring transformations are both reproducible and trustworthy.

For teams dealing with truly massive datasets, distributed processing frameworks like Apache Spark still hold relevance, but they introduce a layer of operational overhead that is often avoidable for most analytics workflows.

### Orchestration: keeping it all on track

As transformations grow in complexity and interdependence, orchestration emerges as the unsung hero of the workflow. Orchestration ensures that tasks execute in the right order, with dependencies managed, failures retried, and workflows monitored.

Apache Airflow has become the default solution for this, providing a DAG-based interface that can model complex workflows across diverse systems. While its verbosity and sometimes finicky configuration are common gripes, Airflow’s flexibility keeps it firmly entrenched in data engineering stacks.

Newer contenders like Dagster promise improved developer experience and type safety, but broad adoption is still catching up. Regardless of the orchestration tool, the goal is consistent: prevent your workflow from becoming an untraceable spaghetti mess of cron jobs and manual triggers.

![Airflow DAG showing sequential tasks for a data pipeline with dependencies](/posts/data-engineering-workflow/dag.webp)

### Delivery: getting data to the people

The final mile of the workflow is where data is delivered to those who need it most. This is no longer limited to dashboards and reports but increasingly involves feeding data back into operational systems, informing marketing campaigns, personalization engines, and customer support tools.

Reverse ETL platforms like Hightouch make this seamless by syncing data from the warehouse to external applications. However, the core of delivery for most organizations remains business intelligence and analytics.

Here, Briefer offers a powerful solution: a modern data workspace with a built-in AI analyst that empowers anyone on the team to explore data through SQL, Python, or intuitive visualizations. Briefer’s unified interface eliminates the friction of juggling multiple tools, making it easier to transform raw data into actionable insights efficiently and collaboratively.

## Final thoughts

The stages of ingestion, storage, transformation, orchestration, and delivery form the backbone of a modern data engineering workflow. The challenge is not merely selecting the right tools but designing a system where each component fits cohesively with the others.

Over-engineering leads to brittle complexity; under-engineering leaves you firefighting. The goal is to strike a balance that supports current needs while providing enough flexibility to evolve with future demands.

Ultimately, designing an effective data engineering workflow is about more than technology. It’s about creating a reliable, maintainable system that enables your team to focus on delivering value rather than patching broken pipelines. Structure should serve as a scaffold for efficiency and innovation, not as a bureaucratic straitjacket.

Create clean analysis and dashboards 10x faster. <a href="https://briefer.cloud/pricing/">Try Briefer for free!</a>

Building an intuitive understanding of how Data Engineering Workflows connect ingestion, transformation, and delivery.

Data Engineering Workflows: From Chaos to Cohesion

Data engineering often feels like a grind of repetitive tasks, fragile pipelines, and endless monitoring. The good news is that AI is starting to turn these pains into smoother, more automated workflows. Instead of replacing the heavy hitters—Airbyte, Dagster, and dbt—AI is weaving into their fabric to improve reliability, speed, and engineer productivity. Understanding these integrations reveals how data teams can work smarter without losing control.

![Data engineering workflow enhanced by AI across ingestion, orchestration, and transformation.](/posts/ai-for-data-engineering/integration.webp)

## AI Empowering Data Ingestion and Integration

Airbyte’s core strength is flexible, scalable data ingestion, but manual schema mapping and handling schema drift have historically slowed teams down. <a href="https://airbyte.com/data-engineering-resources/ai-data-integration" target="_blank">AI integration helps by automatically detecting schemas and mapping source fields to destinations without constant human intervention</a>. This reduces onboarding friction when adding new data sources, which traditionally required painstaking manual setup.

Beyond schema detection, AI also continuously monitors for schema changes or anomalies, alerting engineers before a pipeline breaks. This proactive approach reduces downtime and avoids scrambling to fix broken jobs after the fact. By offloading these error-prone tasks to AI, data engineers can focus on designing meaningful data flows rather than babysitting connectors.

## Smarter Orchestration and Workflow Management

Dagster, as a modern orchestration platform, benefits from AI in several key ways that enhance pipeline robustness and visibility. Predictive failure detection is a standout: AI models analyze historical runs and detect warning signs of potential failures, allowing teams to intervene proactively instead of reacting after outages.

AI also optimizes resource management by dynamically adjusting task parallelism and scheduling based on workload patterns and system health, improving cluster utilization without manual tuning. On the observability front, AI-powered alert triaging helps engineers cut through alert noise by highlighting critical failures and filtering out false positives. This means teams spend less time firefighting and more time iterating on data products.

## AI-Assisted Data Transformation and Testing

dbt has long been the go-to tool for SQL-based transformations, and its AI enhancements aim to speed up development while maintaining quality. <a href="https://www.getdbt.com/blog/ai-data-engineering" target="_blank">AI-assisted code generation helps engineers</a> by suggesting SQL snippets tailored to the project context, reducing boilerplate and speeding iteration cycles.

Testing and documentation also see AI support: smart recommendations for tests catch common anomalies before they propagate, and AI-generated documentation updates keep project knowledge fresh without extra manual work. However, the human engineer remains central to validate and tune these outputs, ensuring transformations meet business needs with precision.

![AI-assisted dbt workflow.](/posts/ai-for-data-engineering/dbt.webp)

## Beyond the Pipeline: Metadata, Governance, and Collaboration

AI’s impact extends well beyond individual pipeline steps, especially in metadata management and data governance. Automatic tagging and lineage inference simplify understanding complex data landscapes, reducing the time needed to trace data origins and dependencies.
By continuously monitoring data quality metrics using AI, teams can detect issues early and ensure compliance with organizational standards. This not only boosts confidence in data products but also smooths collaboration by making datasets more discoverable and trustworthy across business units.

## Conclusion: AI is a Force Multiplier, Not a Replacement

The promise of AI in data engineering isn’t to automate humans out of the loop but to lift tedious work off their plates. Integrations with Airbyte, Dagster, and dbt show that AI can enhance reliability, speed, and developer experience without sacrificing control or flexibility. Embracing these advances means your data team can focus on strategic initiatives rather than firefighting, building a more resilient and efficient data stack.
Create clean analysis and dashboards 10x faster. <a href="https://briefer.cloud/pricing/">Try Briefer for free!</a>

Explore how AI integrations with traditional tools like Airbyte, Dagster, and dbt help data engineers automate workflows, improve quality, and focus on strategic work.

AI for Data Engineering: Smarter Workflows with Airbyte, Dagster, and dbt

If your data engineering team feels overwhelmed, with unclear roles and constant firefighting, chances are the team structure is broken. Modern data challenges demand more than just throwing bodies at problems. Getting the right structure unlocks faster delivery, better data quality, and less frustration.

Real-world experience and foundational texts like *Fundamentals of Data Engineering* emphasize that clear role definitions and collaboration models are essential. From platform engineers who build and maintain infrastructure, to analytics engineers focused on transforming data, and data product owners who keep business needs front and center—the right division of labor is the backbone of an effective team.

## Why structure matters

Expecting one team or person to own everything, from pipelines and data lakes to modeling and stakeholder alignment, quickly leads to chaos. As your data landscape grows, so does the need to specialize. Trying to do it all under a generic “data engineer” label creates friction and bottlenecks.

Good structure clarifies who builds the pipes, who curates the data, and who prioritizes work. It’s also a communication accelerator: when analytics engineers are embedded with product teams, they get faster feedback and produce more relevant insights. Platform engineers, freed from context-switching, focus on reliability and scalability.

In addition, clear team boundaries help manage growth. Without structure, doubling your data volume or adding new business units can cause a collapse of workflows. Structure provides a framework to onboard new team members efficiently and reduce duplicated efforts.

## Proven patterns in the wild

<a href="https://www.getdbt.com/blog/data-team-structure" target="_blank">dbt’s team model</a> champions a hybrid approach: a central platform team builds reusable data infrastructure, while analytics engineers work embedded within business units. This avoids both over-centralization and duplication of effort. Embedded engineers stay close to the product and understand business context deeply, while the platform team enforces standards, builds tools, and maintains reliability.

<a href="https://shopify.engineering/how-to-structure-data-teams" target="_blank">Shopify Engineering’s</a> journey mirrors this. They initially had a centralized data team but realized this created bottlenecks. By splitting into platform and embedded roles, they improved delivery speed and trust from stakeholders. The platform team focuses on scalable pipelines and tooling, and embedded engineers tailor data models for specific products and teams.

Other successful teams echo this balance. Too much centralization means slow turnaround and frustrated business users. Too little, and you risk inconsistency, duplicated work, and fragmented data quality.

![Data Engineering Team Structure: choose the right one](/posts/data-engineering-team-structure/image.webp)

## Common pitfalls to avoid

A classic trap is ambiguous roles where “data engineer” means everything. This overloads teams and blurs accountability. Infrastructure gets neglected, pipelines break, and delivery slows. Engineers end up firefighting infrastructure problems instead of enabling data consumers.

Another frequent failure is over-centralization. When one team is gatekeeper for all data, requests pile up, and the organization loses trust. Business teams start going around the data group, creating shadow analytics and splintered data sources.

Unclear ownership also worsens technical debt. When no one is clearly responsible for data quality or pipeline health, fixes get delayed, bugs accumulate, and scalability suffers. This creates a fragile system where people hesitate to make changes, fearing they’ll break something essential.

On top of that, failing to embed analytics engineers near business units slows feedback loops and reduces the relevance of insights. The communication overhead and context gaps frustrate both data teams and stakeholders. Add to this the chaos of skipping tooling standardization, where teams reinvent pipelines, skip testing, and struggle with deployment: it’s clear how quickly progress can stall.

## Best practices for your data engineering team structure

Start by defining roles clearly, matching them to your company’s scale and needs. Platform engineers own infrastructure and pipelines, ensuring reliability, scalability, and automation. Analytics engineers focus on transformation, modeling, and business-facing data products. Data product owners prioritize requests and keep business alignment tight.

Embedding analytics engineers within product teams is a powerful accelerator. It improves communication and responsiveness, allowing data products to evolve quickly and accurately. Meanwhile, a centralized platform team focuses on core infrastructure, tooling, and enforcing standards, preventing fragmentation and duplication.

Standardizing on tools like dbt for data transformations ensures version control, testing, and documentation—all essential for scaling. Orchestration tools like Airflow automate pipeline scheduling and monitoring, improving reliability.

Here’s a simple example Airflow DAG running dbt daily, showing how teams can automate reliable workflows:

```python
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('daily_dbt_run', start_date=datetime(2025, 1, 1), schedule_interval='@daily') as dag:
 dbt_run = BashOperator(
 task_id='dbt_run',
 bash_command='cd /usr/app && dbt run'
 )
```

Monitoring and alerting should be baked into pipelines early, so teams catch issues before stakeholders do. Regularly reviewing team structure as the company grows helps spot emerging bottlenecks.

## Moving forward

Data engineering teams don’t need to reinvent the wheel but do need to rethink how they organize. Clear roles, a hybrid structure, and standardized tooling reduce friction and technical debt. That lets teams focus on what matters: delivering timely, trusted data that powers your business decisions.

Invest in platform reliability first, then scale analytics embedding. Foster a culture of ownership and continuous improvement. Remember: structure is a tool, not a constraint. Use it to empower your team, not box them in. With the right setup, you can create clean analysis and dashboards 10x faster. <a href="https://briefer.cloud/pricing/">Try Briefer for free!</a>

Building an intuitive understanding of effective data engineering team structures to boost delivery and reduce chaos.

Data Engineering Team Structure: Building High-Impact Teams

Every data engineer knows the pain of managing chaotic pipelines, brittle scripts, and never-ending integration issues. The best data engineering tools offer a way out—turning fragile setups into scalable, reliable systems that can handle everything from ELT to orchestration, streaming, and analytics.

Modern data stacks are no longer about monoliths but about assembling the right tools for each job, often combining open-source flexibility with cloud convenience.

![Modern data workflow with the best tools on data engineering](/posts/best-data-engineering-tools/image.webp)

## Data storage: structured, unstructured, and everything in between

For companies with primarily structured data, relational databases remain the go-to choice. Enterprise players often lean on stalwarts like **Oracle** or **SQL Server**, typically deployed in hybrid environments blending on-premise infrastructure with the cloud. However, the pragmatic choice for many modern organizations is **PostgreSQL**: a robust, open-source relational database that’s evolved into a highly capable, production-grade system over the last decade.

When data becomes less structured—media files, logs, or semi-structured formats like JSON—cloud storage solutions take over. **AWS S3**, **Azure Blob Storage**, and **Google Cloud Storage** dominate this space, offering scalable and cost-effective blob storage. These platforms pair well with serverless query engines such as **AWS Athena**, **Google BigQuery**, or **Azure Synapse**—though Synapse is rumored to be heading toward retirement, with **Microsoft Fabric** taking its place as the all-in-one analytics solution.

## Data ingestion: getting data where it needs to be

Extracting and loading data has come a long way from bespoke Python scripts cobbled together at midnight. Open-source tools like **Airbyte** have made a splash by offering a modular, connector-based approach to ELT, letting engineers quickly integrate APIs, databases, and SaaS platforms without reinventing the wheel. Airbyte's community-driven connector model means it's constantly expanding, catering to niche and mainstream needs alike.

For organizations willing to trade control for convenience, **Fivetran** offers a managed ELT platform that shines in reliability and breadth of pre-built connectors. It’s often the preferred choice for companies with sufficient budget and a desire to minimize maintenance overhead.

Meanwhile, **Meltano** continues to champion the open-source ethos, giving data teams more flexibility and control over their pipelines, particularly in development-centric environments. Cloud-native tools like **AWS Glue** and **Azure Data Factory** also provide serverless ingestion pipelines, though they often lag behind in API connector diversity.

## Orchestration: keeping pipelines under control

Once the data starts flowing, orchestration becomes paramount. <a href="https://airflow.apache.org/use-cases/etl_analytics/" target="_blank">**Apache Airflow** has cemented itself as the industry standard</a> for managing complex workflows, offering fine-grained control, extensibility, and a thriving ecosystem of plugins and integrations. Its directed acyclic graph (DAG) paradigm lets teams model dependencies clearly, although it can introduce operational overhead.

```python
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('etl_pipeline', start_date=datetime(2025, 1, 1), schedule_interval='@daily') as dag:
 extract = BashOperator(task_id='extract', bash_command='python extract.py')
 transform = BashOperator(task_id='transform', bash_command='python transform.py')
 load = BashOperator(task_id='load', bash_command='python load.py')

 extract >> transform >> load
```

For teams prioritizing developer experience and type safety, **Dagster** has emerged as a compelling alternative. Built with modern software engineering practices in mind, Dagster encourages testability and structured config management, making it particularly appealing for teams that treat data pipelines as production-grade software.

In some cases, managed services like **AWS Glue Workflows** or **Azure Data Factory** may be sufficient, especially when sticking within a single cloud ecosystem, but they often lack the flexibility and ecosystem breadth of open-source tools.

## Processing and transformation: making data usable

Batch and stream processing used to be synonymous with **Hadoop**, but that era has largely passed. Today, **Apache Spark** reigns supreme, capable of handling both batch jobs and real-time streams with its powerful distributed computation engine. Spark is often deployed on managed platforms like **Databricks**, which has led the evolution of the <a href="https://www.databricks.com/product/data-lakehouse" target="_blank">**lakehouse** architecture</a>, blending low-cost object storage with structured, scalable query layers via **Delta Lake**.

Databricks' lakehouse model offers teams the ability to process massive datasets efficiently while enforcing data reliability through ACID transactions. Originally tied to **Azure Databricks**, the platform has since expanded to **AWS**, making it more cloud-agnostic and appealing to a broader user base.

Challenging Databricks in the lakehouse space is **Dremio**, an open-source SQL query engine that uses **Apache Iceberg** or Delta Lake as table formats. Dremio offers the ability to store data in S3 or other cloud storage solutions while providing performant, ANSI SQL-compliant queries without the need to load data into a traditional warehouse. Its focus on open formats and cost-effective compute makes it attractive to organizations wary of vendor lock-in.

When it comes to transformation, **dbt (data build tool)** has become indispensable. By popularizing SQL-based, version-controlled transformations, dbt bridges the gap between data engineers and analysts, promoting maintainability and transparency in transformation logic. Its support for testing, documentation, and modular development patterns has redefined how modern teams approach data modeling.

```sql
-- models/orders_transformed.sql

with raw_orders as (
 select * from {{ ref('raw_orders') }}
)

select
 order_id,
 customer_id,
 total_amount,
 order_date::date
from raw_orders
where status = 'completed'
```

## Analytics and dashboards

All these engineering feats ultimately support one goal: turning raw data into insights. This is where analytics platforms and dashboards step in. Traditional business intelligence tools offer robust reporting capabilities, but teams seeking to simplify the analytics process, especially when working with Python and AI, <a href="https://briefer.cloud/blog/posts/simple-changes-make-notebooks-better/">turn to **Briefer**</a>.

Briefer enables data teams to rapidly transform datasets into clean, shareable dashboards and analysis—bridging the gap between data engineering and business outcomes. Its Python-native design makes it particularly well-suited for data scientists and engineers who prefer working in code-first environments but need to deliver polished outputs.

***

## Conclusion

The modern data engineering toolkit is a far cry from its humble beginnings, now consisting of specialized tools optimized for distinct layers of the data stack. Whether it’s orchestrating complex workflows with **Airflow**, transforming datasets with **dbt**, or building scalable lakehouses with **Databricks** or **Dremio**, the **best data engineering tools** empower teams to build reliable, scalable, and maintainable systems.

Turn complex data pipelines into clean, actionable insights. Try **Briefer** today and simplify your analytics, accelerate your impact. <a href="https://briefer.cloud/pricing/">Get started for free!</a>

Discover the best data engineering tools to build scalable pipelines, automate workflows, and manage modern data architectures.

Best Data Engineering Tools: build reliable Pipelines in 2025

Building reliable data pipelines can feel like assembling furniture with missing pieces: frustrating, time-consuming, and prone to structural collapse. Many data engineers struggle with piecing together knowledge from scattered blogs and documentation. This guide compiles the **best books on data engineering** to provide structured, expert-backed resources for mastering the field.

Professionals across industries rely on these books to move beyond ad hoc solutions toward scalable, maintainable systems. Each title here offers distinct value, whether it’s grounding in core principles, practical tooling guidance, or ensuring data quality and governance.

![Modern data workflow with the best books on data engineering](/posts/best-data-engineering-books/image.webp)

## *Fundamentals of Data Engineering* by Joe Reis and Matt Housley

Widely regarded as the modern reference for aspiring and practicing data engineers, *Fundamentals of Data Engineering* offers a comprehensive overview of the data engineering lifecycle. From data ingestion and storage to orchestration and observability, the book connects theoretical concepts to real-world systems with practical clarity.

Readers frequently highlight its value in demystifying the often overwhelming ecosystem of tools and frameworks. The authors provide guidance on when and why to use popular technologies such as Apache Kafka, Airflow, and dbt, avoiding the trap of recommending tools without context.

Although some newcomers might find the book dense, it’s particularly beneficial for those looking to build a strategic understanding of modern data architectures. By emphasizing trade-offs in design decisions, it equips readers to construct robust and scalable pipelines, rather than falling into the trap of over-engineering solutions.

## *Data Quality Fundamentals* by Barr Moses, Lior Gavish and Molly Vorwerck

Bad data can silently erode the integrity of entire analytics systems. *Data Quality Fundamentals* addresses this critical challenge, making it a must-read among the **best books on data engineering**. Authored by industry veterans who specialize in data reliability, the book introduces frameworks for measuring and improving data quality.

Key concepts such as <a href="https://www.ibm.com/think/topics/data-sla" target="_blank">data SLAs</a>, observability, and lineage are presented with actionable examples. The book helps shift the mindset of data engineers from merely moving data to ensuring its reliability, fostering greater trust in data-driven decision-making across organizations.

While it is particularly valuable for mid-level engineers and team leads responsible for maintaining production pipelines, the structured approaches to detecting and mitigating data issues make it relevant for anyone working in data engineering. The emphasis on proactive monitoring and root cause analysis aligns well with modern practices in data reliability engineering.

## *The Data Warehouse Toolkit* by Margy Ross and Ralph Kimball

A cornerstone of data warehouse design, *The Data Warehouse Toolkit* continues to be essential reading despite the evolution of data platforms. Ralph Kimball’s methodologies, particularly dimensional modeling, have shaped how enterprises structure their analytical systems for decades.

The book meticulously details techniques such as star schemas, snowflake schemas, and handling slowly changing dimensions. While some may consider its approach traditional, these frameworks remain deeply embedded in modern cloud data warehouses and business intelligence tools.

For data engineers tasked with creating or maintaining analytics-friendly data models, Kimball’s work offers time-tested strategies to ensure consistency, performance, and scalability. Its systematic approach to data warehouse design ensures that even as technologies evolve, the underlying architectural principles remain sound and applicable.

## *DAMA-DMBOK: Data Management Body of Knowledge*

The *DAMA-DMBOK* is the definitive reference for data management professionals. Comprehensive and methodical, it encompasses everything from data governance and architecture to metadata management and security, making it one of the **best books on data engineering** for understanding the broader context of data work.

Rather than a narrative-driven guide, this book serves as an extensive framework for establishing and maintaining data management practices. Organizations seeking to implement or refine their data governance policies will find it particularly useful, as it aligns with global standards and best practices.

Though not focused solely on engineering topics like pipelines or distributed systems, the *DAMA-DMBOK* provides foundational knowledge essential for roles involving data architecture, compliance, and enterprise data strategy. Its structured approach makes it indispensable for those aiming to mature their organization’s data capabilities beyond technical implementation alone.

## *Designing Data-Intensive Applications* by Martin Kleppmann

A seminal work on distributed systems, *Designing Data-Intensive Applications* is frequently cited as one of the **best books on data engineering**. Martin Kleppmann’s writing excels at breaking down complex topics such as consistency models, distributed logs, and stream processing into accessible yet rigorous explanations.

The book is praised for its deep exploration of system design trade-offs, particularly regarding consistency, availability, and partition tolerance, key components of the <a href="https://www.ibm.com/think/topics/cap-theorem" target="_blank">CAP theorem</a>. Through detailed examples and diagrams, it clarifies concepts like replication, partitioning, and event sourcing, which are critical for designing reliable, scalable data systems.

While it requires a solid technical foundation, this book is indispensable for senior engineers and architects responsible for creating high-throughput, resilient platforms. Kleppmann’s synthesis of academic theory with practical insights makes it a cornerstone text for anyone serious about mastering modern data architectures.

## Conclusion

The **best books on data engineering** bridge the gap between improvised fixes and building systems that scale reliably under pressure. Covering everything from core methodologies and essential tooling to governance frameworks and distributed architectures, these titles equip data engineers with the skills needed at every career stage.

Studying these resources helps professionals design robust pipelines, improve data quality, and navigate the complexities of modern data ecosystems with confidence.

Ready to streamline your data workflows? Briefer makes it simple and fast to transform raw data into clean, actionable insights. <a href="https://briefer.cloud/pricing/">Try for free now!</a>

Discover the best books on data engineering to master scalable pipelines, data quality, and modern architectures.

Best Books on Data Engineering: the essential reads for 2025

If you're comfortable writing `SELECT` statements and building Python scripts, you might wonder: *should I analyze my data directly in SQL or load it into Python?* The answer depends on your goals, dataset size, and the complexity of the analysis. Let's break down when to stay in SQL—and when it's smarter to switch to Python.

## When to Analyze Directly in SQL

SQL is optimized for **working within databases**. It’s fast, declarative, and designed to process **structured data** efficiently without moving it elsewhere.

### Use SQL for:

* **Filtering and aggregating large datasets:** SQL engines like PostgreSQL or MySQL handle operations like `GROUP BY`, `SUM()`, and `JOIN` more efficiently than Python, especially on large tables.

* **Data summarization:** When your analysis is descriptive (e.g., total sales by region), SQL is ideal.

* **Ensuring data integrity:** SQL enforces constraints and types at the database level, preventing bad data from propagating.

* **Reducing data transfer:** If your dataset is huge (think millions of rows), running aggregations in SQL and exporting only the summarized result to Python is far more efficient.

### Calculating monthly sales trends directly in SQL

```sql
SELECT EXTRACT(MONTH FROM sale_date) AS month, SUM(amount) AS total_sales
FROM sales
GROUP BY month;
```

**Key takeaway:** Use SQL for **data reduction** and **clean joins** before moving data elsewhere.

![SQL query reducing large dataset into summarized table](/posts/python-vs-sql/sql.webp)

## When to Transfer Data to Python

SQL’s analytical capabilities stop short when you need **<a href="https://scikit-learn.org/" target="_blank">complex modeling</a>**, **<a href="https://scipy.org/" target="_blank">custom calculations</a>**, or **<a href="https://briefer.cloud/blog/posts/look-the-data/" target="_blank">visualizations</a>**. That’s where Python excels.

### Use Python for:

* **Statistical analysis:** SQL lacks native functions for linear regression, hypothesis testing, or advanced time-series forecasting. Python offers libraries like `statsmodels` and `scikit-learn` for these.

* **Data visualization:** While SQL can return numbers, it doesn’t make charts. Python’s `matplotlib` or `seaborn` can visualize trends effectively.

* **Machine learning and AI:** For clustering, classification, or neural networks, you’ll need tools like `scikit-learn`, `PyTorch`, or `TensorFlow`.

* **Automated workflows:** Python scripts can automate data ingestion, analysis, and reporting—something SQL alone isn’t built for.

### Loading a SQL query result into Pandas and running a predictive model

```python
import pandas as pd
from sqlalchemy import create_engine

engine = create_engine('postgresql://user:pass@host/db')
df = pd.read_sql('SELECT * FROM customer_data', engine)

from sklearn.linear_model import LogisticRegression
model = LogisticRegression().fit(df[['age', 'income']], df['churn'])
```

**Key takeaway:** Use Python for **complex logic**, **modeling**, and **visual outputs**.

![Python workflow pulling data from SQL and running machine learning.](/posts/python-vs-sql/python.webp)

## When to Combine Python and SQL for Data Analysis

The most productive data workflows balance **Python vs SQL for data analysis**, using both together:

1. **Extract** and **pre-filter** data in SQL.
2. **Analyze** and **model** it in Python.
3. Optionally **store results** back in SQL for reporting.

This hybrid approach optimizes performance and capability.

Briefer is a powerful data workspace with a built-in AI analyst that makes this even easier. It helps anyone on your team turn data into insights through SQL, Python, and point-and-click visualizations. In Briefer, queries naturally turn into Pandas DataFrames, streamlining the workflow from extraction to analysis.

## Pitfalls to Watch Out For

* **Overloading Python:** Don’t pull the entire database into Pandas “just in case.” Filter in SQL first.

* **Premature optimization:** Simple summaries? Stay in SQL. Don’t complicate things by exporting to Python unnecessarily.

* **Data type mismatches:** SQL types (`DECIMAL`, `DATE`) don’t always map cleanly to Python types (`float`, `datetime`). Watch out when transferring.

* **Security concerns:** Pulling data from production databases for local Python analysis can pose risks. Always follow data governance policies.

## FAQ

### Should I always analyze data in SQL first?

Yes, start with SQL for initial filtering and aggregation. If SQL alone suffices, stop there. Otherwise, transfer the reduced dataset to Python.

### Is Python slower than SQL for data analysis?

For **basic aggregations**—yes. SQL is usually faster within the database. But for **complex analytics**, Python's ecosystem outpaces SQL.

### How do I combine SQL and Python effectively?

Use SQL to prepare data, then transfer via `pandas.read_sql()` or `SQLAlchemy` for further Python-based processing, visualization, or modeling.

## Streamline Your Data Analysis and Reporting with Briefer

If you're looking to streamline your entire analysis-to-insight pipeline with faster data analysis, <a href="https://briefer.cloud/pricing/">start using Briefer for free</a>
and see how it can transform your reporting process.

Understand the trade-offs of Python vs SQL for data analysis, with clear guidelines on when to use each tool and practical examples.

Python vs SQL for Data Analysis

Data professionals often struggle with fragmented datasets spread across multiple systems: databases, APIs, cloud services, and real-time streams. This fragmentation leads to inconsistent analyses, slow decision-making, and increased maintenance overhead. **Data integration strategies** help solve this by combining these sources into a unified view, enabling seamless analysis and operational efficiency. With the rise of big data, effective integration isn’t just a technical necessity—it’s a competitive advantage. According to Gartner, <a href="https://www.gartner.com/en/data-analytics/topics/data-quality" target="_blank">poor data integration costs organizations an average of $12.9 million annually in lost productivity and bad decisions</a>.

![Multiple data sources funnel into a single integrated data warehouse.](/posts/data-integration/integration.webp)

## Why Data Integration Is Essential

Data integration provides a consolidated foundation for analytics, reporting, and machine learning. Key techniques include:

* **ETL (Extract, Transform, Load)**: Ideal for structured batch processing.
* **ELT (Extract, Load, Transform)**: Suited for modern cloud data warehouses like Snowflake.
* **Data Virtualization**: Offers real-time access without physically moving data.

Each approach solves different integration challenges. As Forrester Research puts it: <a href="https://www.forrester.com/report/vendor-landscape-integration-strategy-and-delivery-service-providers/RES121347" target="_blank">“A fit-for-purpose integration strategy can drive operational efficiency and fuel business innovation.”</a>.

## Star Media Group Unifies Data for Smarter Insights

Star Media Group, a leading media company in Southeast Asia, faced challenges with fragmented data from multiple sources like digital platforms, subscriber systems, and advertising channels. To overcome this, they implemented a modern data integration strategy using **Google Cloud’s BigQuery** and **Dataflow** to create a unified, scalable data warehouse.

This approach enabled Star Media to consolidate diverse data streams into a single platform, allowing real-time analysis and reporting across departments. By integrating their data seamlessly, they improved audience targeting and optimized ad revenue with data-driven campaigns. According to Google Cloud, this integration <a href="https://cloud.google.com/customers/star-media-group/" target="_blank">led to faster data access and empowered their teams to make smarter, timely decisions</a>.

This case highlights how combining cloud-based ETL pipelines and data warehousing can transform complex data landscapes into actionable business intelligence.

![Unified data integration pipeline used by Star Media.](/posts/data-integration/starmedia.webp)

## Pitfalls to Avoid in Data Integration

While implementing **data integration strategies** can be transformative, there are pitfalls to watch for:

* **Latency issues**: Not all use cases require real-time data; forcing it can strain resources unnecessarily. Sometimes, batch processing with ETL/ELT tools like **Fivetran** or **Airbyte** is sufficient and more efficient.
* **Data inconsistency**: Without rigorous validation and proper transformation, integrated data stored in lakes or warehouses such as **Google BigQuery** or **Amazon Redshift** can lead to misleading analytics.
* **Overcomplicating pipelines**: Adding too many steps or tools increases maintenance costs and complexity. Using modeling tools like **dbt** helps keep transformations clear and manageable.
* **Ignoring data governance**: Poor metadata management and unclear data lineage reduce trust in the system and make troubleshooting difficult.

Real-time integration is powerful, but always align your method to specific business needs.

## Best Practices for Seamless Data Integration

To maximize the benefits of data integration, follow these guidelines:

1. **Start with clear objectives**: Define the business insights your integration should support.
2. **Leverage proven tools for each step**:

 * **ETL/ELT**: Use automated connectors like **Fivetran** or **Airbyte** to reliably extract and load data.
 * **Data lakes and warehouses**: Store and query your consolidated data in platforms like **Google BigQuery** or **Amazon Redshift**.
 * **Modeling**: Use **dbt** to transform raw data into clean, analytics-ready datasets.
3. **Prioritize data quality**: Implement regular audits and automated validation checks.
4. **Embrace data governance**: Establish clear policies for metadata, access control, and data lineage to build trust and transparency.

For a deeper dive, see <a href="https://www.fivetran.com/use-case/operations/infrastructure-modernization" target="_blank">Fivetran’s resources and best practices</a>.

## Future Trends in Data Integration Strategies

The future of **data integration strategies** will be shaped by automation and AI. Emerging tools are increasingly automating data mapping and transformation. The rise of **data fabrics** offers unified architectures that simplify integration across hybrid and multi-cloud environments. Additionally, edge computing is processing data closer to its source, cutting latency for IoT and real-time applications.

Gartner predicts that by 2026, <a href="https://www.gartner.com/en/documents/6503971" target="_blank">50% of organizations will adopt data fabric architecture to streamline integration across disparate systems</a>. Staying ahead of these trends ensures your data strategy remains scalable and competitive.

## FAQ

### What are the most effective data integration strategies?

The choice depends on your needs:

* **ETL** for structured batch jobs
* **ELT** for cloud-native scalability
* **Data Virtualization** for real-time insights without data movement

### How can document management systems support data integration strategies?

Document management systems streamline the organization and storage of unstructured data, such as PDFs, contracts, or reports, which are often critical for analysis. By using a <a href="https://www.dokmee.com/About/Blog/ArticleID/349/best-document-management-system" target="_blank">best document management system</a>, businesses can centralize documents, making it easier to extract metadata or content for integration into data pipelines. For example, structured data from documents can be fed into ETL processes or cloud data warehouses like Google BigQuery, enhancing insights from combined sources.

### Which tools are best for seamless data integration?

Top picks:

* **Apache Kafka**: Real-time data streams
* **Talend**: Comprehensive ETL
* **Informatica**: Enterprise-grade automation
* **Fivetran**: Zero-maintenance connectors

## Unlock Better Insights with Seamless Data Integration

Effective data integration lays the foundation for meaningful analysis and better decisions. Briefer makes it easy to bring your data together and turn it into clear, actionable dashboards. <a href="https://briefer.cloud/pricing/">Explore how Briefer can help</a>.

Master data integration strategies and learn how to combine diverse data sources for streamlined, reliable insights.

Data Integration Strategies: Combining Sources for Seamless Analysis

Over the last decade, data analytics teams have shifted dramatically, from isolated technical groups to strategic partners embedded across organizations. Once primarily focused on reporting and data collection, these teams are now integral to shaping business strategy and innovation. This evolution is fueled by the growing recognition that data-driven insights enable faster, smarter decisions. Modern analytics teams must be adaptable, structured to align with changing business needs, and capable of driving cross-functional initiatives that deliver measurable value.

## Core Roles That Drive Analytics Success

A high-performing analytics team relies on a range of specialized roles working in concert:

* **Data Engineers** design and maintain pipelines, ensuring reliable data flow.
* **Data Analysts** interpret data and generate actionable business insights.
* **Data Scientists** develop models to forecast trends and optimize processes.

Increasingly, roles like <a href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/analytics-translator" target="_blank">Analytics Translators</a>
, professionals who bridge technical findings and business objectives, are recognized as essential for maximizing impact.

The effectiveness of these roles hinges on collaboration. Cross-functional synergy transforms isolated data points into cohesive, strategic narratives that drive growth.

## Choosing the Right Team Model: Centralized, Decentralized, or Hybrid?

When structuring a data analytics team, organizations typically choose between centralized, decentralized, or hybrid models. Each approach offers distinct advantages and challenges, depending on business priorities, scale, and culture.

A **centralized model** consolidates all data expertise within a single team. This fosters consistency in data governance, standardizes tools and methodologies, and often reduces redundant efforts across departments. However, it can also create bottlenecks when departments need specialized insights quickly, as all requests must pass through a central team.

In contrast, a **decentralized model** embeds data analysts within individual business units. This structure enables teams to respond rapidly to specific needs, offering tailored insights aligned with departmental goals. The downside is potential inconsistency in data practices, tools, and metrics, which can hinder organization-wide data alignment.

Many organizations are now opting for a **hybrid model** that blends centralized oversight with decentralized execution.
Research by Gartner indicates that <a href="https://www.gartner.com/en/documents/3891681" target="_blank">hybrid models are increasingly popular</a>
, offering scalability without sacrificing agility.
A core data team manages governance, infrastructure, and standards, while embedded analysts within departments ensure agility and domain expertise. This model balances consistency with responsiveness, and it’s often cited as the most effective structure for large, complex organizations.

![Centralized, decentralized, and hybrid data team structures](/posts/data-team-structure/teams.webp)

## Building an Effective Data Analytics Hierarchy

A clear hierarchy ensures accountability and facilitates collaboration. Typically:

* A **Head of Data or Chief Data Officer** sets strategy.
* **Analytics Managers** coordinate daily operations.
* **Technical Leads** oversee engineers, analysts, and scientists.

Successful companies often emphasize flat hierarchies to encourage innovation while retaining enough structure to ensure focus and alignment with business priorities.

For instance, <a href="https://engineering.atspotify.com/2014/09/squad-health-check-model" target="_blank">Spotify’s data teams operate with autonomous squads</a>
aligned to specific product areas, fostering both ownership and agility.

## Collaboration Best Practices for Analytics Teams

Effective collaboration is at the heart of any successful data analytics team. Regular cross-functional meetings help ensure that analytics efforts remain aligned with broader business objectives. These meetings foster transparency, enabling stakeholders from various departments to share their needs and provide feedback on ongoing projects.

Building a culture of openness is equally important. A <a href="https://hbr.org/2016/11/why-diverse-teams-are-smarter" target="_blank">Harvard study</a>
suggests that when team members feel comfortable sharing ideas, asking questions, and offering constructive critiques, innovation thrives. Encouraging psychological safety within the team helps to surface diverse perspectives and novel solutions to complex data challenges.

Finally, continuous learning should be embedded into the team culture. Supporting professional development through workshops, conferences, or peer-led training ensures that team members stay current with evolving data technologies and methodologies. This investment in learning not only strengthens individual skill sets but also enriches the team’s collective capability to tackle new problems.

## Case Studies: How Top Companies Structure Their Teams

* **Netflix** employs a hybrid model with <a href="https://netflixtechblog.com/part-1-a-survey-of-analytics-engineering-work-at-netflix-d761cfd551ee?gi=4a9d505b7d48" target="_blank">centralized data engineering and decentralized analytics</a>
 embedded within product teams. This allows for both standardized tooling and specialized analysis.
* **Airbnb** developed its own data science framework (Airflow) to empower decentralized analytics teams while maintaining data quality through centralized engineering support.

These examples show that the most successful structures are those tailored to an organization’s specific needs and culture.

![In this schema, centralized engineering feeds decentralized analytics pods](/posts/data-team-structure/example.webp)

## Looking Ahead: The Future of Data Analytics Teams

Data analytics teams are on the cusp of another transformation, shaped by:

* **Automation and AI**: Automating routine data tasks frees up analysts for higher-value work.
* **DataOps and MLOps**: Embedding software engineering principles to streamline analytics and machine learning workflows.
* **Remote collaboration**: As hybrid work becomes standard, asynchronous tools and global teams redefine how analytics projects are managed.

Companies that invest in adaptable structures and upskill their teams for these emerging paradigms will be best positioned for future success.

## FAQ

### What roles are essential in a modern data analytics team?

Key roles include data engineers, analysts, scientists, and increasingly, analytics translators who bridge technical and business functions.

### What’s the most effective team model?

Hybrid models often offer the best of both worlds: centralized governance with decentralized agility. Although, the ideal structure depends on organizational goals.

### How do companies measure analytics team performance?

By combining technical metrics (like model accuracy) with business impact metrics (like time-to-insight and decision-making improvements).

## Empower Your Team with the Right Tools for Data and Collaboration

Bad data derails even well structured analytics teams and weakens collaboration. Every unseen anomaly warps reports and slows decisions. <a href="https://www.siffletdata.com/" target="_blank">Sifflet</a> applies AI-powered observability to keep data clean and lineage clear, so analysts stay focused on insights instead of firefighting.

Strong data analytics teams rely on both structure and tools. Briefer helps unify your data and create intuitive dashboards, so your team can focus on insights instead of wrangling spreadsheets. <a href="https://briefer.cloud/pricing/">**Try Briefer for free**</a>.

Learn how to design an effective data analytics team structure, covering key roles, collaboration best practices, and performance optimization.

Data Analytics Team Structure: Key Roles and Best Practices

If you've ever battled with spreadsheets, you know the challenges: endless rows, confusing formulas, and the constant fear of crashing your computer. These frustrations are common among data scientists who rely on spreadsheets for data analysis. However, **Python data science libraries** offer robust tools that transform chaotic spreadsheets into streamlined, insightful workflows. Say goodbye to clunky interfaces and hello to Python's elegant simplicity.

## Why Python Data Science Libraries Matter

Handling large datasets and complex analyses requires more than spreadsheets. Libraries like **<a href="https://numpy.org/doc" target="_blank">NumPy</a>**, **<a href="https://pandas.pydata.org/" target="_blank">Pandas</a>**, and **<a href="https://matplotlib.org/" target="_blank">Matplotlib</a>** provide foundational support for numerical operations, data manipulation, and <a href="https://briefer.cloud/blog/posts/look-the-data/" target="_blank">visualization</a>, making data science more efficient and scalable.

## Retail Sales Boost with Python Libraries

Imagine a retail team aiming to boost sales by better understanding customer purchase patterns. They start by using Pandas to clean and manipulate messy sales spreadsheets: filtering transactions, handling missing data, and grouping by customer segments with just a few lines of code. Next, NumPy helps them efficiently perform numerical calculations, like computing average purchase values and seasonal trends.

For visual insights, Matplotlib and Seaborn come into play, creating clear charts that highlight peak buying times and popular products. Meanwhile, Scikit-learn powers predictive models that forecast future sales based on historical data, helping the team plan inventory smartly.

```python
import pandas as pd
import numpy as np

# Load dataset
df = pd.read_csv("sales_data.csv")

# Basic data manipulation
df['Total'] = df['Quantity'] * df['Price']

# Summary stats with NumPy
mean_sales = np.mean(df['Total'])
print(f"Average sales: {mean_sales:.2f}")
```

To speed up certain data-heavy calculations, they use Numba to just-in-time compile Python functions, improving performance without rewriting code. And if the project requires automation, (e.g. scraping competitor pricing data) they rely on Selenium to control browsers and extract information automatically.

This combination of libraries allows the retail team to move from raw spreadsheet chaos to actionable insights swiftly, demonstrating why Python’s data science ecosystem is essential for real-world projects.

![Workflow with Python data science libraries](/posts/top-libraries/pipeline.webp)

## Pitfalls to Avoid When Choosing Libraries

Choosing the right Python libraries can be tricky. Python’s inherent speed limitations mean that for performance-critical tasks, relying solely on pure Python libraries might slow you down. That’s why libraries like Numba (which uses just-in-time compilation) or PySpark (which distributes computation) are crucial for scaling, but they also add complexity and require learning new paradigms.

Not considering the learning curve is a mistake. Some powerful libraries (e.g., FastAPI for web APIs or PyTorch for deep learning) require time investment, so evaluate whether the team’s skill set aligns with the library before adopting it. Balancing performance, simplicity, and maintainability is key when selecting Python data science tools.

## The Essential Libraries

![Python data science libraries core ecosystem](/posts/top-libraries/core.webp)

* **Core data manipulation & analysis**:
 * *NumPy*: Efficient numerical computing
 * *Pandas*: Powerful data manipulation
 * *Matplotlib* & *Seaborn*: Data visualization and statistical plots

* **Machine learning & AI**:
 * *Scikit-learn*: Classic ML algorithms
 * *PyTorch*: Deep learning and neural networks
 * *Numba*: Speeding up Python code via JIT compilation

* **Web & Automation tools**:
 * *FastAPI*: Modern web APIs
 * *Selenium*: Browser automation

* **Data handling & NLP**:
 * *SQLAlchemy*: Database toolkit
 * *SpaCy*: Advanced NLP

* **Big data & computer vision**:
 * *PySpark*: Scalable big data processing
 * *OpenCV*: Computer vision tasks

## FAQ

### How do I set up a Python data science environment?

The easiest way is to install Python via **Anaconda**, which comes bundled with most essential libraries like **NumPy**, **Pandas**, and **Matplotlib**. For interactive coding and visualization, **Jupyter Notebook** is a popular choice. If you run into issues, resources like official documentation and **Stack Overflow** provide step-by-step troubleshooting guides.

### Which Python data science libraries do users recommend the most?

According to user feedback, **Pandas** and **Scikit-learn** are highly praised for their ease of use and robust functionality. One data scientist remarked, “These libraries transformed our workflow and saved hours weekly.” Additionally, benchmark reports consistently rank **NumPy** as the top performer for numerical operations.

### What are the future trends in Python data science libraries?

The integration of **AI** and **<a href="https://briefer.cloud/blog/posts/least-squares/" target="_blank">machine learning</a>** is driving rapid advancements in Python libraries. Tools like **PyTorch** and **FastAPI** are expected to gain even more features and adoption. Staying current with these evolving libraries ensures you can leverage the latest innovations in your data science projects.

## Streamline Collaboration and Turn Insights into Action

Managing tools and sharing insights across teams can be a significant challenge, often leading to inefficiencies and delays in decision-making. Without a streamlined process, teams can struggle to collaborate effectively, leading to disjointed workflows and a lack of transparency. This can result in missed opportunities and slowed project timelines. <a href="https://briefer.cloud/pricing/">Try Briefer for free</a>
and turn your Python-powered insights into collaborative, interactive reports with ease.

Explore the top Python data science libraries that transform spreadsheet chaos into powerful, efficient analysis workflows.

Top Python Data Science Libraries for Efficient Analysis

Struggling to reproduce your last experiment because you can't recall which dataset version you used? Or finding collaboration tricky as datasets evolve? That’s the pain of lacking **data versioning**. As many experts note, modern machine learning success depends more on **increasing data use** than endlessly tweaking models. Without structured **data versioning**, managing growing datasets becomes chaotic and unreliable.

This guide shows how **data versioning** empowers teams to handle data growth, ensuring reproducibility, efficiency, and smooth collaboration.

## Why Data Versioning Is Essential

Data is often treated as static, but real-world datasets are constantly updated, cleaned, and augmented. Traditional methods, like renaming files or relying on manual notes, don’t scale as datasets balloon to gigabytes or terabytes.

![Data versioning DAG showing dataset and model evolution.](/posts/data-versioning/dag.webp)

Modern ML practitioners emphasize that models gain more from expanding datasets than from minor architectural changes. This underscores the need for robust **data versioning** to systematically manage evolving datasets, models, and experiments.

Without it, experiments are difficult to reproduce, mistakes creep in, and valuable insights are lost in the shuffle.

## How Data Versioning Works

With **data versioning**, datasets and models become first-class citizens in your workflow. Here’s how you’d version a dataset using <a href="https://dvc.org" target="_blank">**DVC**</a>:

```bash
dvc add data/raw_dataset.csv
git add data/.gitignore data/raw_dataset.csv.dvc
git commit -m "Add versioned raw dataset"
```

Each dataset version links to a specific state of your project, ensuring results are reproducible.

Behind the scenes, **data versioning** tools construct a **Directed Acyclic Graph (DAG)** connecting datasets, models, and experiments.

```bash
$ dvc dag
 +---------+
 | prepare |
 +---------+
 *
 *
 *
 +-----------+
 | featurize |
 +-----------+
 ** **
 ** *
 * **
+-------+ *
| train | **
+-------+ *
 ** **
 ** **
 * *
 +----------+
 | evaluate |
 +----------+
```

## Will Data Versioning Slow You Down?

A common concern is that **data versioning** will reduce velocity by duplicating massive datasets. Fortunately, modern tools mitigate this by:

* **Deduplication**: storing only differences between versions.
* **Remote storage**: keeping large files out of Git repositories.
* **Lazy loading**: fetching only required data for experiments.

These optimizations mean that your team can maintain full version histories without compromising performance.

## Best Practices for Data Versioning

To get the most out of **data versioning**, apply these practices:

* Version datasets, models, and experiment configurations consistently.
* Use dedicated tools like **DVC** for seamless integration with Git.
* Maintain metadata to ensure reproducibility.
* Visualize dependencies with DAGs.
* Train your team in versioning workflows to avoid errors.

For structured learning, check out the <a href="https://dvc.org/doc/tutorials" target="_blank">official DVC course</a>.

## FAQ

### Why is data versioning important in modern data projects?

Data versioning ensures teams can track changes to datasets and models over time, making experiments reproducible and collaboration more efficient across complex projects.

### How does data versioning integrate with other tools?

Modern versioning systems, like DVC, integrate seamlessly with source control and data workflows, forming a DAG (Directed Acyclic Graph) that reflects data and model evolution without compromising performance.

## Streamline Data Analysis with Version Control

Briefer is a tool for data analysis that integrates seamlessly with data versioning systems, helping teams manage datasets efficiently while creating reproducible and shareable dashboards with ease. <a href="https://briefer.cloud/pricing/">Start your free trial now</a> and simplify your data workflows.

Learn how version control for data boosts reproducibility, collaboration, and dataset management without slowing your workflow.

Why Data Versioning Is Essential for Modern Data Teams

From spreadsheet frustration to programming power: many data scientists hit a wall with spreadsheets. As data grows, errors multiply and workflows slow down. That’s when languages like R and Python transform the process, offering precision and scalability that spreadsheets can’t match. If you’re weighing R vs Python for data science, this guide breaks down how each handles typical challenges so you can pick the right tool.

![Data science workflow evolving from spreadsheets to R, Python and advanced Machine Learning.](/posts/r-python-data-science/image.webp)

## R for Data Science: Precision in Statistics

R is the statistician’s best friend. It’s built for statistical analysis and visualization, with packages like **ggplot2** and **dplyr** leading the way. For example, you can visualize complex datasets quickly:

```R
library(ggplot2)
ggplot(data = mtcars, aes(x = wt, y = mpg)) + 
 geom_point() + 
 geom_smooth(method = "lm")
```

R’s ecosystem handles complex models and advanced data manipulation with ease. For official resources, see <a href="https://www.r-project.org/doc/" target="_blank">R's documentation</a>.

## Python for Data Science: Flexibility and Scale

Python is the versatile contender. With libraries like **Pandas** for data wrangling, <a href="https://briefer.cloud/blog/posts/look-the-data/" target="_blank">**Matplotlib** for visualization</a>, and **TensorFlow** for machine learning, Python adapts to almost any task. Here's a simple example of data manipulation in Python:

```python
import pandas as pd

df = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df['C'] = df['A'] + df['B']
print(df)
```

One benchmark shows Python’s rapid growth in data science, driven by its flexibility and massive community support.

## Choosing Between R and Python

Use cases for R vs Python in data analysis vary. R dominates in exploratory analysis and reporting, while Python excels in data preprocessing and integration. For instance, **NumPy** and **SciPy** make Python ideal for numerical tasks, while R’s **dplyr** simplifies data wrangling.

| Feature | R | Python |
| ----------- | -------------------- | --------------------- |
| Strength | Statistical analysis | General-purpose |
| Key library | `ggplot2`, `dplyr` | `Pandas`, `TensorFlow` |
| Best for | Academic research | Industry applications |

Industry trends favor Python in commercial roles, but R remains strong in academia and research where statistical rigor is key.

## R vs Python for Machine Learning

In machine learning, it’s a close race. R’s **caret** simplifies access to multiple algorithms:

```R
library(caret)
model <- train(Species ~ ., data = iris, method = "rpart")
print(model)
```

Python’s **scikit-learn** and **TensorFlow** power everything from predictive models to neural networks:

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
clf = RandomForestClassifier()
clf.fit(iris.data, iris.target)
```

One project combined both: R for fine-tuning and Python for model deployment, showing their complementary strengths.

## Best Practices: Picking the Right Language

Both communities offer robust support. Python’s global network ensures quick answers and extensive tutorials. R’s passionate base focuses on statistical rigor. Many data scientists choose based on the community that aligns with their goals and learning style.

*Pro tip*: Join both. It's common to use R for specialized statistical work and Python for deployment or integration.

## Performance of R vs Python

Performance matters. Python is fast at data processing, especially with **NumPy** and **Cython**. R is efficient in statistical computations, boosted by **data.table**.

Benchmark studies show:

* For general tasks, Python wins on speed.
* For specialized statistics, R holds its ground.

*Example*: Sorting a large dataset with R's **data.table**:

```R
library(data.table)
DT <- data.table(x = rnorm(1e6))
setkey(DT, x)
```

## FAQ: R vs Python for Data Science

### What’s the main difference between R and Python for data science?

R specializes in statistical analysis and visualization. Python covers broader applications like automation, web development, and machine learning.

### Which is better for data analysis?

R leads in statistical reporting; Python shines in data preprocessing and integration.

### How do they compare in machine learning?

R offers robust modeling with **caret**. Python provides extensive machine learning libraries like **scikit-learn** and **TensorFlow**.

## Accelerate Exploratory Data Analysis with Briefer

Make exploratory data analysis 10× faster with Briefer’s AI-powered notebook. <a href="https://briefer.cloud/pricing/"> Start using Briefer for free.</a>

Explore the strengths, use cases, and benchmarks of R vs Python for Data Science to choose the best tool for your projects.

R vs Python for Data Science: Make the Right Choice

Ever found your organization tangled in data confusion, unsure whether sensitive data should be anonymized, duplicated, or even shared? Without structure, data gets mishandled, risking compliance breaches and inefficiency. That’s where data governance tools step in, helping organizations manage data effectively, securely, and responsibly. This guide explores how these tools create order out of chaos, focusing on practical benefits for your team.

## Why Data Governance

Data governance is the capability that enables your organization to manage data securely, efficiently, and responsibly. This includes decisions like whether data should be anonymized, how it’s distributed internally, and ensuring regulatory compliance.

![Data governance tools transforming raw data into compliant assets](/posts/data-governance/assets.webp)

## Defining the Data Governance Framework

A data governance framework structures how your organization manages its critical data assets. At its core, it includes tools that enforce policies, apply rules, and classify data correctly.

### Data Taxonomy: Structuring Data for Clarity

Effective governance starts with a solid data taxonomy: a system that defines data types, sensitivity levels, and handling protocols. For example:

* Public data → freely shareable
* Internal data → staff-restricted
* Confidential data → access-controlled, possibly anonymized

These classifications guide how data flows through the organization and ensure consistency.

## Benefits of Using Data Governance Tools

### Better Decision-Making

With structured data, decision-makers gain accurate, reliable insights, reducing guesswork and driving strategic outcomes.

### Improved Cost Control

By eliminating redundant data storage and streamlining processes, organizations cut costs and optimize resource allocation.

### Easier Risk Management

Governance tools help enforce compliance and security policies, reducing the likelihood of costly data breaches or regulatory penalties.

### Enhanced Team Access

Clearly structured and classified data ensures that the right people have timely access, promoting collaboration and efficiency across departments.

## Compliance Risks Without Proper Tools

Neglecting compliance features in your **data governance tools** can expose your organization to legal risks. GDPR violations alone have led to <a href="https://www.dlapiper.com/en/insights/publications/2025/01/dla-piper-gdpr-fines-and-data-breach-survey-january-2025" target="_blank">fines exceeding €1.6 billion since 2018</a>. Compliance tools with built-in audit trails, like **Collibra**, help ensure adherence to regulations and protect your reputation.

Automation simplifies governance by minimizing manual processes. For instance, automated data lineage tracking ensures traceability across systems. Proprietary studies show automation reduces data handling errors by up to 60%. Tools like **Atlan** offer automated cataloging, saving time and boosting reliability.

## The Value of Compliance

As the GDPR and other regulations enforce stricter data practices, compliance becomes critical. According to <a href="https://www.iso.org/standard/62816.html" target="_blank">ISO/IEC 38505-1</a>, data governance is essential for managing data-related risks and ensuring accountability.

## Trends in Data Governance Tools

Emerging trends include AI-powered metadata management and predictive compliance monitoring. Research from **Gartner** predicts that <a href="https://www.gartner.com/en/newsroom/press-releases/2023-10-11-gartner-says-more-than-80-percent-of-enterprises-will-have-used-generative-ai-apis-or-deployed-generative-ai-enabled-applications-by-2026" target="_blank">by 2026, 70% of enterprises will adopt AI-enhanced **data governance tools** to stay competitive</a>. Staying ahead means investing in solutions that evolve with technological shifts.

![How to choose the best data governance tools based on organizational needs](/posts/data-governance/integration.webp)

## FAQ

### What are the best data governance tools for effective data management?

Tools offering automated data quality, compliance, and security features, such as Collibra, Informatica, or Atlan.

### How do data governance tools maintain data quality?

Through profiling, cleansing, enrichment, and monitoring functionalities that ensure accuracy and consistency.

### How do data governance tools protect sensitive data?

By enforcing encryption, access controls, and maintaining audit logs to secure information.

## Connecting Data Governance Tools to Dashboards

With strong data governance in place, Briefer dashboards unlock even greater value. Governance tools ensure that the data feeding your dashboards is accurate, secure, and compliant—making insights not just faster, but also more reliable. If you're looking for ad-hoc, shareable dashboards powered by trustworthy data, <a href="https://briefer.cloud/pricing/">**try Briefer for free**</a>.

Discover how top data governance tools improve compliance, security, and data quality while cutting operational risks.

How Data Governance Tools Prevent Compliance Nightmares

Your team just pulled in a massive dataset from multiple sources. But now what? Without clear collaboration between data engineers and data analysts, that raw data stays just that: raw. In modern data projects, **data engineers** and **data analysts** play distinct yet connected roles to transform data into actionable dashboards and insights.

## Why Data Engineers Lay the Groundwork First

Every data project starts with the data infrastructure. The **data engineer** ensures that data from various systems, like web traffic logs, CRM exports or transactional databases, flows smoothly into centralized storage like a data warehouse. Using tools like <a href="https://airflow.apache.org/use-cases/etl_analytics/" target="_blank">Apache Airflow</a> or <a href="https://www.getdbt.com/product/what-is-dbt" target="_blank">dbt</a>, engineers build **automated pipelines** to clean, transform, and load this data, ensuring it's reliable and up-to-date.

![Data engineer building automated pipelines to centralize and clean data.](/posts/data-analyst-engineer/engineer.webp)

In a real project, the data engineer sets up a daily process that pulls transactional data from the production system into a cleaned dataset, making it ready for the analysts to explore without the need for tedious, manual data wrangling.

## How Data Analysts Turn Data Into Insights

Once the pipelines are running, the <a href="https://briefer.cloud/blog/posts/look-the-data/" target="_blank">**data analyst**</a> steps in. Analysts query this prepared data, applying business logic to answer specific questions. In the same project, the analyst might investigate trends like “Which products are seeing increased sales week over week?” They build **interactive dashboards** that allow stakeholders to monitor KPIs in real-time and adjust strategies accordingly.

![Data analyst creating dashboards from clean, structured datasets.](/posts/data-analyst-engineer/analyst.webp)

Through this flow, data engineers ensure data is accessible and reliable, while analysts transform it into meaningful insights.

## Misaligned Data Expectations

Without proper coordination, analysts might work with incomplete or misaligned data. For example, if the engineer changes pipeline logic but doesn’t communicate it, the analyst’s dashboards could show misleading trends or even break entirely. That’s why clear **data contracts** and shared documentation are essential to keep workflows stable and reliable.

## Best Practices for Data Collaboration

* **Shared version control**: Both engineers and analysts can use Git to track changes to pipelines, queries, and dashboards.
* **Regular syncs**: Weekly standups help teams align on schema changes and business priorities.
* **Integrated tools**: Using platforms like Briefer helps teams connect their pipelines directly with dashboards seamlessly.

These practices help teams avoid duplicated effort and maintain **data reproducibility**.

## From Raw Data to Executive Dashboard

Consider a project where a fintech company wants to monitor user churn. The engineer creates pipelines that aggregate user activity logs, applying transformations to flag churn-prone behaviors. The analyst then uses this dataset to build a **churn risk dashboard** that executives consult weekly.

From engineering pipelines to analytical insights, this real-world flow demonstrates how collaboration drives impactful data products.

## FAQ

### What’s the main difference between a Data Engineer and a Data Analyst?

Data Engineers build and maintain the infrastructure for data collection and processing, while Data Analysts interpret that data to generate insights and support business decisions.

### How do Data Engineers and Analysts collaborate in real projects?

Data Engineers ensure data is clean, structured, and accessible. Data Analysts then use that data for analysis, creating reports and dashboards that inform strategy and operations.

### How does Briefer help Data Engineers and Analysts?

Briefer enables both roles to transform data into shareable dashboards and visualizations quickly. It integrates with modern data workflows, streamlining collaboration and accelerating insight delivery.

## From Raw Data to Collaborative Insights

By bridging the work of data engineers and analysts, organizations can streamline their data workflows from infrastructure to insight. With tools like Briefer, teams can quickly transform raw data into shareable dashboards and actionable visualizations, reducing friction between engineering and analysis. <a href="https://briefer.cloud/pricing/">Start a free trial now</a> and see how seamless collaboration enhances your data projects.

Discover how data engineers and analysts collaborate on real-world projects, building pipelines, models, and dashboards together.

Data Engineer vs Data Analyst: How They Collaborate on Real Projects

## A dumb solution

To get home from São Carlos to Araxá, I pass through Ribeirão Preto. It’s the shortest and most convenient route. Going through Brasília, Brazilian capital, is a bad idea - I’d still get home, but I’d walk many kilometers more.

![](/posts/regularization/map.png)

Not only those two paths exist, but many more. Similarly, there exists **many** functions that passes through all points in a graph.

![](/posts/regularization/graph.png)

But some are inefficient, just like going through Brasília. Regularizers help us find simpler solutions.

Take polynomial functions, for example. Their empirical risk is the sum of squared errors - the distances between predicted values and actual ones.
Essentially, the empirical risk measures how good the curve fits the data. In the plot below, both models have similar empirical risk. Which one should we choose?

![](/posts/regularization/curves.png)

The squiggly line might perform well on the training dataset, but it is just like a student that memorizes homework questions instead of learning how to solve each one. When taking the test, the student is shocked to learn that not a single question is identical to the homework.

Likewise, the wiggly curve fails on new examples. The smoother one, however, generalizes better and handles unseen data more reliably.

![](/posts/regularization/testing.png)

In Machine Learning, we say the wiggly curve *overfitted*
Overfitting happens when there are too many possibilities to choose a model from.

There are [better ways of visualizing overfitting than thinking about squiggly lines](https://briefer.cloud/blog/posts/bias/).
Think of the space of functions - all possible mappings from $x$ to $y$. We should pick linear functions, polynomials, or neural networks to train our model - a smaller subspace within the space of all functions.

During training, we search for functions inside this subspace - shown as green points - that are closest to the Oracle, marked in purple. The Oracle is a hypothetical function that perfectly captures the true relationship between $x$ and $y$.
We then pick the function with the lowest empirical risk.

When we choose a complex model, like a neural network or a high-order polynomial, the subspace becomes too large - and finding the best model is like finding a needle in a haystack.

![](/posts/regularization/subspace.png)

The final model depends heavily on the training data - it can end up with low, medium, or high true risk depending on the dataset. In other words, it has high variance.

![](/posts/regularization/variance.png)

Choosing a simpler model is not always an options, since they can be too simple.

![](/posts/regularization/simple_model.png)

The idea behind regularizers is to restrict this big subspace in such way that dumb solutions like passing through Brasília are no longer an option.

![](/posts/regularization/limited_subspace.png)

## Creating a regularizer

We didn’t like the wiggly functions because they overfit.
A good regularizer should penalize the parts of those functions that lead to overfitting.

![](/posts/regularization/testing.png)

A first-order polynomial predicts values using a line: $y = f(x) = ax + b$, where $a$ and $b$ are the coefficients. A second-order polynomial adds a quadratic term: $y = ax^2 + bx + c$. Adding more terms creates higher-order polynomials. Our squiggly curve is one of these high-order polynomials with coefficients $a, b, c, d...$. We can use the whole alphabet if needed.

The problem with high-order polynomials is their large coefficients. When coefficients get extreme, the function can vary wildly and fit every point perfectly.

* High Coefficients Values: $f(x) = 30x^3-23x^2-16x+10$
* Low Coefficients Values: $f(x) = 0.4x^3-1x^2-0.8x+10$

We’ve seen that curvy lines and big coefficients cause overfitting. Intuitively, smaller coefficients - which create smoother curves - work better.

This intuition is called prior knowledge. For example, it’s why we avoid routes with many cities when traveling-the more cities, the longer the trip.

Here, the prior is: "most coefficients should be close to zero."
We express this using a bell-curve (Normal) probability distribution, which favors coefficients near zero.

![](/posts/regularization/prior.png)

To restrict the large function space and reduce variance, we add prior knowledge to the usual regression optimization.

Now, we minimize

$$
\text{regularized error}=\text{empirical risk + penalty(coefficients)}
$$

The penalty term is the standard norm of the coefficients $||\text{coefficients}||^2$ which increases as the coefficients get larger. A norm is just a function that take a vector outputs a number. Here, the standard norm is the square root of the sums of the coefficients squared.

<details style="border: 1px solid #ccc; border-radius: 6px; padding: 10px; margin: 12px 0;">
 <summary style="font-weight: bold; cursor: pointer;"> 
 The maths of adding the Prior to the Error Equation
 </summary>
 
During training we minimize the squared error plus a penalty term.

$$
\text{regularized error}=\text{empirical risk + penalty(coefficients)}
$$

Now we develop the penalty given it is a Normal curve. It's a probability distribution on the vector of $k$ coefficients $\theta$. We want a small penalty on high probability coefficients, so a negative sign is used.

$$
\text{penalty(coefficients)} = - \log p(\theta)
$$

The coefficients are distributed using a Normal distribution. It reads $\theta$ is distributed according to a Normal distribution with mean $0$:

$$
p(\theta)=
\mathcal{N}(\theta | 0, I)=
\dfrac{1}{(2\pi)^{2/D}}
\exp(
-\dfrac{\theta^T \theta}{2}
)
$$

Taking the logarithm of the distribution $p\theta$:
$$
\log p(\theta)=
-\log (2\pi)^{2/D}
-\dfrac{\theta^T \theta}{2}
$$

We keep only terms that are related to \theta. Finally, our regularization term is the sum of the coeficients squared. A little complicated. This is usually written using the norm.

$$
\theta^T \theta =
||\theta||^2
$$

We can come back to our original error expression using the found prior.

</details>

$$
\text{regularized error}=\text{empirical risk} + ||\text{coefficients}||^2
$$

In traditional regression, we find coefficients that minimize empirical risk. The plot shows how the error changes with different coefficients - the minimal risk is at the lowest point.

![](/posts/regularization/3dplot.png)

Now, we search coefficients that **minimize our risk** and are **close to zero**.
With this constraint, visualizing in two dimensions makes it easier to understand.

![](/posts/regularization/slicing.png)

The 3D plot is represented by ellipses, which are contours showing different error levels. Greener ellipses mean smaller errors. The circle at the origin represents the regularizer. We want to find [coefficients that minimize the error and stay inside this circle](https://en.wikipedia.org/wiki/Lagrange_multiplier).

![](/posts/regularization/penalization.png)

We can solve this optimization using Gradient Descent. This algorithm moves downhill, following the steepest descent until it reaches a minimum. Here, that minimum is where the error curve meets the regularizer’s circle.

![](/posts/regularization/solution.png)

This minimum gives us coefficients that minimize squared error while staying close to zero.

Thus, the Normal Prior over the coefficients of the model gives this regularizer. Can we ajust the regularizer for different problems?

## Tweaking the Regularizer

I ommited an important factor all along - one of the most important ones.

If the function space is very large or complex, we might want stronger regularization. Otherwise, a smaller penalty usually works fine.

This is controlled by a term called lambda ($\lambda$), which sets the regularization strength. When lambda is close to zero, regularization is weak.

![](/posts/regularization/weak_regularization.png)

As lambda increases, regularization gets stronger, pushing weights closer to zero.

![](/posts/regularization/larger_regularization.png)

This lambda is commonly called an **hyperparameter** of the model.
A hyperparameter is one parameter of the model engineers and scientists should set to make sure training goes right. And there are many techniques for it.

This is only one way of regularizing machine learnig models. At least we are not going through Brasília now.

Exploring how regularizers restrict the search space for a problem in such way that dumb solutions are no longer an option.

How Regularizers Fix Dumb Solutions

Speed kills rigor. In startups, the pressure to ship fast pushes teams to report *anything* that looks like an improvement. That’s how p-hacking happens. This piece breaks down three common cases—and how to avoid them.



# Example 01: Multiple comparisons without correction

Imagine you're a product manager trying to optimize your website’s dashboard. Your goal is to increase user signups. Your team designs four different layouts: A, B, C, and D.

<img src="/posts/p-hacking/ab.png" width="500" height="300" />

You run an A/B/n test. Users are randomly assigned to one of the four layouts and you track their activity. Your hypothesis is: *layout influences signup behavior*.

You plan ship the winner if the p-value for one of the layout choices falls below the conventional threshold of 0.05.

Then you check the results:

<img src="/posts/p-hacking/mislead.png" width="400" height="300" />

Option B looks best. p = 0.041. It floats to the top as if inviting action. The team is satisfied and ships it.

But the logic beneath the 0.05 cutoff is more fragile than it appears. That threshold assumes you’re testing a single variant. But you tested four. That alone **increases** the odds of a false positive.

Let’s look at what that actually means.

Setting a p-value threshold of 0.05 is equivalent to saying: *"I’m willing to accept a 5% chance of shipping something that only looked good by chance."*

So the probability that one test **doesn’t** result in a false positive is:

$$1−0.05=0.95$$

Now, if you run 4 independent tests, the probability that none of them produce a false positive is:

$$0.95 \times 0.95 \times 0.95 \times 0.95 = 0.8145$$

That means the probability that **at least one test** gives you a false positive is:

$$1 − 0.8145 = 0.1855$$

So instead of working with a 5% false positive rate, you’re actually closer to 18.5%: nearly a 1 in 5 risk that you're shipping something based on a fluke.

And that risk scales quickly. The more variants you test, the higher the odds that something looks like a win just by coincidence. Statistically, the probability of at least one false positive increases with each additional test, converging toward 1 as the number of comparisons grows:

<img src="/posts/p-hacking/numberoftests.png" width="700" height="300" />

Bottom line: you ran a four-arm experiment but interpreted it like a one-arm test. You never adjusted your cutoff to account for multiple comparisons. Which means the p-value you relied on doesn’t mean what you think it does.

This is p-hacking. You looked at the results, picked the one that cleared the bar, and ignored the fact that the bar was never calibrated for this setup.

## How to avoid this: adjusting the threshold

The Bonferroni Correction is one way to avoid using the wrong cutoff when testing for multiple options. It's straightforward: you account for the number of hypotheses *k* by adjusting the acceptable p-value for significance:

$$ \text{adjusted threshold}= \frac{0.05}{k}$$

In our dashboard test with 4 variants, that’s:

$$\frac{0.05}{4} = 0.0125$$

Under this correction, only p-values below 0.0125 should be considered significant. Your p = 0.041 result? It no longer qualifies.

<img src="/posts/p-hacking/bonferoni.png" width="500" height="300" />

Fewer results will pass the bar. That can feel frustrating in fast-moving product teams. But now you're actually measuring something you can trust.



# Example 02: Reframing the metric after the results are in

Back to the dashboard experiment: after you applied the Bonferroni correction you got... nothing. None of your dashboard variants significantly improved user signup rates.

<img src="/posts/p-hacking/squareone.png" width="500" height="300" />

This is frustrating. You've invested weeks in the redesign, and you're facing a product review with no wins to show. Nobody likes arriving empty-handed to leadership meetings.

So you dig deeper. The data's already collected so why not explore other insights? Maybe signups didn't improve, but what about **retention**? You check for retention rates and discover something interesting:

<img src="/posts/p-hacking/retention.png" width="400" height="300" />

Option B shows slightly higher retention than the rest, with p = 0.034. Suddenly your narrative shifts: "This experiment was really about improving retention all along!"

You pivot the story and now it’s tempting to call it a win for B and ship it. But each extra metric you check is another chance for randomness to sneak in. If you check 20 metrics, the odds that at least one will look like a winner by pure chance shoot up to about two in three. That’s because the probability that none of the 20 metrics show a false positive is:

$$1−(1−0.05)^9 =64%$$

Graphically, it looks like this:

<img src="/posts/p-hacking/metrics.png" width="700" height="300" />

That promising retention improvement? It’s just the kind of anomaly you’d expect to find after enough digging.

## The pre-registration solution

Each time you add a new metric, you increase the chance of finding a false positive. A single test with a threshold of p < 0.05 implies a 5 percent risk of error. But the more tests you run, the more that risk accumulates. In the limit, it approaches certainty.

Pre-registration prevents this. By stating in advance which metric will count as evidence, you fix the false positive rate at its intended level. The p-value retains its meaning. You are testing one hypothesis, not several in disguise.

Decide your success metrics before running the test. Document them explicitly and stick to them.

This isn't academic nit-picking. It's how medical research works when lives are on the line. Your startup's growth deserves the same rigor.



# Example 03: Running experiments until we get a hit

Even if you’ve accounted for multiple variants and didn't the temptation to shift metrics, one bias remains: impatience. The lure of early results is difficult to ignore and can lead to bad decisions

Now you're running an A/B test of two button styles, scheduled for two weeks.

<img src="/posts/p-hacking/ab2.png" width="500" height="300" />

Each day, you check the dashboard, just in case. On the ninth day, the p-value for button B dips to 0.048:

<img src="/posts/p-hacking/9days.png" width="700" height="300" />

Should you stop the test and ship B? At this point, you know a win shouldn’t come that easily. A p-value only works if you set your stopping rule in advance. Peeking the p-value each day during nine days is like you’re running nine experiments. Each day is a new opportunity for randomness to look like signal.

After 9 peeks, the probability that at least one p-value dips below 0.05 is:

$$1−(1−0.05)^9 =37%$$

<img src="/posts/p-hacking/peek.png" width="700" height="300" />

And there’s another subtle trap: by not waiting for the experiment to finish, you’re watching the p-value bounce around as new data arrives. That "significant" result on day 9 might be nothing more than a lucky swing, gone by day 14:

<img src="/posts/p-hacking/15days.png" width="700" height="300" />

Shipping on an early p-value is like betting on a horse halfway around the track.

## How to properly peek

If you absolutely must make early stopping decisions, here’s how to do it responsibly using **sequential testing**.

Let’s go back to our button test. You planned to collect data for 2 weeks. Instead of using a flat $$p<0.05$$ the whole time, sequential testing adjusts the threshold depending on when you stop:

* Week 1: Only stop if p < 0.01 (super strict)
* Day 10: Only stop if p < 0.025 (still strict)
* Day 14: Normal p < 0.05 threshold

This approach controls the overall false positive rate, even if you peek multiple times.

Remember our day 9 result where p = 0.048? Under sequential testing, that wouldn't qualify. You'd need p < 0.01 to stop in week 1. So you'd keep running the test and probably discover it wasn't actually significant:

<img src="/posts/p-hacking/alpha.png" width="700" height="300" />

It works like "spending" your false positive budget gradually instead of all at once.

So yes, you can peek with discipline. But for most teams, the simpler and safer move is still the right one: wait the damn two weeks.



# In summary

Your next experiment will be more reliable if you:

* Pre-register hypotheses and metrics
* Avoid digging through metrics post hoc
* Use corrections when testing multiple variants
* Apply proper thresholds if you peek early
* Celebrate definitive negative results (might be controversial)

The irony is that better statistical practices actually accelerate learning. Instead of shipping noise and wondering why your metrics plateau, you'll build genuine understanding of what drives user behavior. That's worth slowing down for.

When agile experimentation at startups becomes a p-hacking trap

P-Hacking in Startups

Confidence intervals help us know more about the world, even when we can’t see all of it.

Say ou want to answer a simple question: **how many slices of pizza can a person eat in 30 minutes?**

How would you figure that out on your own?

Sure, you *could* make every person on the planet eat pizza for 30 minutes and calculate the global average. But unless you’ve got a few billion dollars and an industrial cheese pipeline, that’s not happening.

Instead you take a sample. You gather a group of people, feed them pizza, and measure what happens. So you try it. You randomly select 25 people, give them pizza, and record the results. The average comes out to 3 slices per person. Here's what that sample looks like:

![](/posts/confidence-intervals/histo1.png)

To be sure, you run the experiment again. You take a new group of 25 people, give them the same 30-minute pizza challenge, and this time, the average comes out to 2.8 slices:

![](/posts/confidence-intervals/histo2.png)

Before running the experiment a third time, you stop and think: **Am I even close to the truth?** Those two averages (3 and 2.8) seem reasonable. But are they *right*? Can a single group of just 25 people really tell us something meaningful about the entire population? And if so, under what conditions does that actually work?

To start answering that, let’s look at how the averages (the sample means) behave. So far, we’ve got two of them, each from a different group:

![](/posts/confidence-intervals/samplemean1.png)

The averages are landing in a similar range. That seems reassuring.

But how do we know they’re actually good estimates? After all, each one is based on just a handful of people. What if we just got lucky? What if the average is inflated because we happened to sample a group full of mukbang YouTubers?

That’s a fair concern, and it’s why *randomness* matters. We're assuming our samples are **random**, meaning everyone in the world had an equal chance of being picked. That’s the starting point. Without randomness, the whole experiment is off course before it begins. Randomness is what keeps the selection fair and protects us from bias.

Still, you might wonder: *couldn’t a sample land at an extreme just by chance?*

Technically, yes. But let’s think about what that would take.

To get a sample average as high as the most extreme pizza eater on Earth, your entire group would need to be made up of mukbang stars. Not one or two — all 25. Every single person in the sample would have to be a professional eater. That’s not impossible. But it’s *very* unlikely. The result? A histogram with a sky-high average. Something like this:

![](/posts/confidence-intervals/histo3.png)

This sample gives us an average of 5.6 slices per person. Where does that number sit in the bigger picture?

If we repeated this process 10,000 times, each time randomly picking 25 people and calculating their average, most of those results would fall near the center. This is what that distribution would look like in our case:

![](/posts/confidence-intervals/samplemean2.png)

When you repeat the sampling process over and over, the typical results win. They happen again and again. They show up more often. The rare stuff? Not so lucky.

You might randomly draw a sample with 25 mukbangers once (maybe twice if you're cursed) but those extremes don’t get reinforced. They don't pile up. They stay stranded out on the edges of the distribution.

The same goes for the other extreme. To get a very low average, you'd need to randomly select only people who barely eat pizza. Again, very rare.

The extremes cancel each other out, and that tug-of-war pulls the average toward the middle.

This is where this outliers fall into in the distribution:

![](/posts/confidence-intervals/samplemean3.png)

You might have noticed the distribution of the sample means forms a shape. It’s smooth, bell-like, and centered. That’s the **normal distribution curve**.

But where does that shape come from? Does the distribution of everyone’s pizza-eating ability also look like that?

**Not necessarily**. The bell curve shows up because we’re averaging. It’s a feature of how sample means behave, not a mirror of how individuals actually eat.

To see the difference, imagine we had data on all 8 billion people on Earth. If we plotted how many slices each person eats in 30 minutes, we’d get the population distribution: the full spectrum of eating habits, from nibblers to mukbang stars.

It looks something like this:

![](/posts/confidence-intervals/pop.png)

The distribution is far from tidy. It’s right-skewed, with most people clustered around 2 or 3 slices, and a long tail stretching toward the outliers, the few who manage 8, 10, even 14 slices. And yet, when we take random samples from this messy population, something remarkable happens:
the sample means arrange themselves into a clean, symmetrical curve.

Even when the population is messy, the distribution of sample means still forms a clean, bell-shaped curve. In other words: **no matter how unpredictable the population is, the averages behave predictably.**

Not only that, the sample means we observe **vary around the true population mean**. In our case, that mean is $$t=3$$ slices.

This isn’t just a fluke. No matter what the population looks like, the distribution of sample means forms a normal curve, *centered on the value we want to estimate*. This is the **Central Limit Theorem**, a foundational idea in statistics. It was observed long before it was formally proven: take enough random samples, and the averages begin to behave predictably.

If we place the population distribution and the distribution of sample means side by side, here’s what we see:

![](/posts/confidence-intervals/pop2.png)

We can see the sample means clustering around the population mean. In other words, we know that, most of the time, our sample mean lands within some distance of the true value we’re trying to estimate.

And since the sample means follow a normal distribution, we can apply all the properties that come with it. Specifically:

* About 68% of the time, our sample mean will be a distance *d* from the true mean.

* About 95% of the time, our sample mean will fall within 2 times this distance.

* About 99% of the time, our sample mean will fall within 2.57 this distance

![](/posts/confidence-intervals/pop3.png)

This distance is the typical amount the sample mean strays from the population mean. In our pizza contest, it’s how far the average from a group of 25 people might wander from the true average you'd get if you could feed everyone on Earth.

It depends on two things: how wildly appetites vary, and how many people you include in your sample.

When it comes to people's appetites, the more variation there is, the farther the sample mean can stray. If some people eat 1 slice and others devour 100, the average can swing wildly. But if nearly everyone eats between 2 and 4 slices, the averages settle down. The distance can be shorter (green threshold).
But if nearly everyone eats between 2 and 4 slices, the averages settle down. The distance can be shorter (green threshold).

![](/posts/confidence-intervals/sem1.png)

When it comes to the number of people in your sample, the more you include, the less the sample mean wiggles around.

On the other hand, with a smaller group, say 25 tasters, the sample mean can land high or low depending on who ends up in the group (red threshold). With 100 tasters, the randomness evens out. The extreme eaters and light eaters balance each other better, so the sample mean doesn't move around as much from sample to sample (green threshold).

![](/posts/confidence-intervals/sem3.png)

That distance is called the **Standard Error of the Mean**. It tells you how much you should expect your sample mean to wiggle just by chance. It’s a way to measure how much your sample mean might differ from the true mean you’re trying to find.

So once you calculate the mean of your sample and know how far it tends to stray (the standard error) from the **true** mean, you can say:

> **I’m 95% confident that the true population mean lies within this range.**

This **doesn’t mean** there’s a 95% chance the true mean is in this interval. The true mean is fixed: it either is or isn’t in there. A confidence interval tells you how **reliable** your method is.

If you repeated the process many times: took new random samples and asked how many slices of pizza each person eats; then 95% of the intervals you build would **contain** the true average, which in this case is 3 slices. Visually:

![](/posts/confidence-intervals/cis.png)

Thanks to the Central Limit Theorem, we know how to find the true mean, because we understand how the sample means behave. No matter how messy the data is, the sample means form a normal curve, and they tend to cluster around the true mean.

We also have a way to measure *how far* the sample means usually wanders: the Standard Error of the Mean tells us how much sample means typically vary from the value we’re trying to find.



# There's a tradeoff

Now you might be thinking: If a confidence interval only reflects the reliability of the method, and not whether this specific interval captures the true value, then what’s stopping me from claiming anything I want with 99.999% confidence?

Sure. You could say: "I’m 95% confident people eat between 1 and 1,000 slices of pizza in 30 minutes."

Technically, that’s valid. The interval is so wide, it almost guarantees the true mean is inside. But it’s also useless. It tells you nothing.

There’s a tradeoff.

You can always make the interval wider to boost confidence, but the wider it gets, the less helpful it becomes. On the other hand, if you make the interval too narrow, you might miss the truth entirely.

**A good confidence interval strikes a balance.**
It’s wide enough to be reliable, but narrow enough to be meaningful. That's when a confidence interval becomes a tool for reasoning; one that reflects both *what* you know, and *how well* you know it.



# The Dog, the Boy, and the Mean

To sum up: **if you want to understand confidence intervals, picture a boy and his dog.**

The boy walks the dog every day through the neighborhood. The dog is obedient, leash-free, and curious. He drifts, but is never too far. You begin to notice a pattern: roughly 68% of the time, he’s within a certain range; 95% of the time, within twice that range.

![](/posts/confidence-intervals/dog1.png)
![](/posts/confidence-intervals/dog2.png)

So when you see the boy, it’s easy to guess the dog is nearby. The boy is the center, and the dog loops loosely around him.

![](/posts/confidence-intervals/radius.png)

Now flip the scene.

One day, you don’t see the boy. You see only the dog.

![](/posts/confidence-intervals/dog3.png)

You pause, look around, and say: "The boy must be right about... there." The catch is: you never had to see the boy at all.

By watching the dog again and again, you learned how far he tends to stray. So now, even in a single moment, a lone observation tells you something meaningful about what you can’t see.

This is the logic behind a confidence interval.

What we’re really after is the real average, the number you’d get if you could somehow ask every person on Earth how much pizza they eat in 30 minutes. That number exists, but we can't measure it directly. **That number is the boy**. And right now, we can’t see him.

**So instead, we watch the dog**. Each time we randomly pick 25 people and calculate their average, it's like taking a snapshot of where the dog is. That number (3.0 or 2.8) is a single frame. A moment in time.

Knowing where the dog is help us find the boy thanks to the Central Limit Theorem, which tells us how he drifts from the boy. Visually:

![](/posts/confidence-intervals/pop4.png)

Confidence intervals are valuable because we usually have to reason with limited information. You can’t see the boy (the true mean) directly. But if you understand how the dog (your sample mean) tends to move, you can make solid guesses about where the boy is.

That’s what inference is: You don’t need the exact truth, just how far off you might be.

A confidence interval gives you that range.
It tells you: Here’s what I observed, here’s how much it might vary, and how sure I am about it.

Between the dog and the boy, or between the data and the truth, that’s where statistics does its work.

How can we know anything about the world if we can’t see all of it? Confidence intervals help us reason with limited information, giving us a range where we can expect the true value to lie.

Confidence intervals

A/B testing is everywhere, but good luck explaining p-values to your CFO. When the question is “Which feature wins?”, most stakeholders don’t care about confidence intervals, they want a clear answer.

I’ve always found Bayesian A/B testing more intuitive and far better for communicating results. In this post, we’ll break down why.

Say we want to decide if a green button will perform better than a blue one, so we go ahead and run an A/B test. In the end, people usually want to know just one thing: “Is A better than B?”

![](/posts/ab-testing/pipelineab.png)

After 500 visitors see each button, the numbers are in:

* Green: 27 sign-ups — that’s 5.4%.
* Blue: 23 sign-ups — 4.6%

![](/posts/ab-testing/buttons.png)

This is where many stop. They see the numbers and declare a winner: green beats blue. But we know better than to take numbers at face value.

Think about it: if you flip a coin 10 times and get 7 heads, would you bet your savings that it’s rigged? Probably not. With such a small sample, randomness can easily skew the result.

![](/posts/ab-testing/coin.png)

The same logic applies here. Green got 5.4%, blue got 4.6% so it *looks* like a win, but it’s just a snapshot. Rerun the test, and those numbers could flip. Like the coin toss, there’s an underlying truth we just can’t see yet.

That’s what makes A/B testing hard: It’s about the uncertainty behind the numbers we see. How do we measure it? How do we quantify the risk of acting on too little data?



# Measuring uncertainty the usual way

Using the textbook method to measure uncertainty, we need to know how likely it is that the difference we saw (5.4% for Green and 4.6% for Blue) is actually real and not just a fluke.

Think of it this way: if we ran this exact experiment 1,000 times, how often would Green *actually* outperform Blue? Would it win 900 times? 700? Or just 400? If Green wins less than half the time, then it’s not really a winner, it just got lucky the first time we ran the test.

This is where frequentist statistics hands us a tool: **the p-value**.

To understand it, we start by assuming the *opposite* of what we’re trying to prove: that there’s no real difference between Green and Blue. If they’re equally good, then any difference we observe must be due to random chance.

Now imagine plotting all the differences you’d see across thousands of these random experiments. You’d get a bell-shaped curve centered at zero, because most of the time, chance alone won’t budge the results very far.

That’s the graph you see below. The middle of the curve represents small, harmless fluctuations: the noise we expect when nothing’s really going on. The tails are where things get weird: the rare, extreme results that pop up once in a while.

![](/posts/ab-testing/pvalue.png)

A p-value doesn’t tell you the probability that Green is better than Blue. It tells us:

> If Green and Blue were actually equally as good, how likely is it that we’d see a difference as big as 0.8% (or more) just by random chance?

A low p-value, like 0.01, means that if the buttons were actually equally good, seeing a result like 5.4% for Green and 4.6% for Blue would be rare. It might happen just 1 in 100 times, purely due to chance. It’s like saying: *“If there’s no real difference, this result shouldn’t be showing up... and yet it did.”*

That’s why a low p-value is treated as a signal that something might be going on. But if the p-value is high (say, 0.5) that means a result like 5.4% vs. 4.6% would be totally normal even if the buttons were identical. There’s nothing unusual to explain, therefore strong reason to believe one button is actually better.

![](/posts/ab-testing/pvalues2.png)

A p-value doesn’t tell you how likely Green is actually better, only how much your result challenges the assumption that there’s no real difference.

In frequentist statistics, the hypothesis is fixed and the data is random. The guiding idea is: “There’s a true answer out there; we just observed a noisy sample of it.”

You run an A/B test and get 27 sign-ups for Green and 23 for Blue. A frequentist won’t say “there’s a 70% chance Green is better.” Instead, they ask: “If Green and Blue performed exactly the same, how likely would it be to see a result this extreme just by chance?” Green either is better or isn’t. There’s no room for uncertainty about the hypothesis itself.

That makes communicating results tricky. Try explaining p-values and null hypotheses to a CFO who just wants to know if we should switch to the green button. Frequentist methods are statistically sound, but they don't speak the language of probability most people use.

Bayesian thinking flips that around.

Bayesians treat the data as fixed: 27 vs. 23, no arguing with that. What’s uncertain is the hypothesis: Is Green truly better than Blue? The Bayesian question becomes much more intuitive:

“Given what we observed, how likely is it that Green is better?”

That’s an answer you can take to a decision-maker.



# Measuring uncertainty the Bayesian way

The Bayesian approach starts with a different question:

> Given what I’ve seen, what’s the full range of possible conversion rates and how likely is each one?

Instead of giving you a binary verdict or a vague range, it shows you a distribution of beliefs: a spectrum of plausible values, shaped by your actual data. This more aligned with how we naturally think: not in black-and-white outcomes, but in shades of confidence.

To model uncertainty the Bayesian way, we build a probability distribution, that is, a curve that represents what we **believe** the true conversion rate could be, based on the data we have.

That’s where Bayes’ Rule comes in. It gives us the mathematical machinery to **update** our beliefs based on new evidence. Here's what it looks like:

![](/posts/ab-testing/formula1.png)
In words:



![](/posts/ab-testing/formula2.png)

The **prior** is what we believed about the conversion rate before we saw any data. Maybe from past experiments or gut feel, we expect sign-up rates to usually fall somewhere between 3% and 7%. That’s our starting guess.

Then comes the **likelihood**, which asks: Given what we just saw, how well does each possible conversion rate explain it? If Green got 27 sign-ups out of 500 users, then rates like 5% or 6% make a lot of sense. A rate of 1% or 15%? Not really. The likelihood rewards values that make the data unsurprising.

Now we combine those two (the prior and the likelihood) to get the posterior. This is our new belief, updated with the evidence. It’s not a single number, but a full distribution that tells us how plausible each conversion rate is after seeing the results. This is what we actually care about.

Finally, there’s the evidence, technically called the marginal likelihood. It’s just the math that makes sure everything adds up to 100%. You can mostly ignore it unless you’re comparing models.



![](/posts/ab-testing/formula3.png)

This tells us: What we now believe = What we believed × What the data tells us

To figure out *what we now believe* about a button’s true conversion rate, we first need to model what we believed before the test. This means we need a mathematical tool that can express that belief.

What do we already know about our site’s conversion rates? Are they usually around 5%? Closer to 10%? How likely is it that a new feature converts higher than 6%?

We capture that belief with a **prior** distribution: a curve that assigns different degrees of plausibility to different conversion rates. It doesn’t predict what will happen. It simply quantifies what we consider likely before we collect new evidence.

We know each visitor either converts or doesn’t (success or failure, yes or no). When we run an experiment with a fixed number of visitors and record how many convert, we’re counting the number of **successes** across a series of independent trials. That’s precisely the kind of process the **binomial distribution** is meant to describe: it tells us the probability of observing a certain number of conversions, given a fixed number of trials and a known conversion rate.

Tossing a coin is the textbook example of a binomial process. We have two possible outcomes: heads or tails. If the coin is fair, the probability of tails is 0.5.

Flip it 100 times, count the number of tails, and the binomial distribution tells us what kind of results to expect. It looks like this:

![](/posts/ab-testing/binomial.png)

The highest probabilities cluster around 50 tails, with the odds dropping off as you move toward the extremes. That’s what the binomial distribution gives you: **what to expect when you already know the odds.**

But in A/B testing, we don’t know the odds, we’re trying to infer it from the data.

To do that, we need a different kind of curve, one that represents our **uncertainty** about the conversion rate itself. Something that shows all the possible values the rate could take, and how plausible each one is based on the data we have.

That’s exactly what the **Beta distribution** gives us.

It’s defined between 0 and 1, making it perfect for modeling probabilities. But more importantly, it’s flexible. It can represent total uncertainty (where every rate is equally likely) or strong convictions, (where most of the probability mass is concentrated around a narrow range).

In Bayesian A/B testing, we use the Beta distribution to track our evolving belief about the conversion rate. Before the experiment, it reflects our prior assumptions. After we collect data, we update it to reflect the evidence.

Depending on what we know (or don’t), the shape of that belief curve can look very different. Here are a few examples:

![](/posts/ab-testing/betafunction.png)

The beta works naturally with the binomial distribution. When we update a **Beta prior** with binomial data, we get a **Beta posterior**. This property, called conjugacy, makes the updating process mathematically seamless. Our beliefs can change but the form of our model stays the same.

Say your experience as a product manager tells you that most new features tend to "convert" at around 10% of users. Rather than discarding that knowledge, Bayesian inference lets you incorporate it directly into your model by shaping your prior distribution to reflect it.

To incorporate that knowledge into our model, we use the Beta, which is controlled by two parameters: α and β. These parameters shape the curve and determine how confident we are in different conversion rates. The mean of a Beta distribution is:



![](/posts/ab-testing/alpha.png)

So if we believe the true rate is likely around 10%, we can choose values of α and β that center the distribution at 0.10. For example, Beta(2,18) has a mean of 2/(2+18) = 0.10.

This is a way of saying: "Before seeing any data, I believe a 10% conversion rate is most likely, and I'm about as confident in that belief as if I had already seen 20 users, 2 of whom converted."

The notation Beta(α,β) simply tells us how that belief is distributed across the 0-1 range. Higher values of α and β mean stronger prior confidence; smaller values mean more uncertainty.

What the data tells us is that 27 out of 500 visitors convert for the Green button. To update our prior belief, represented by Beta(2,18), we simply incorporate the new evidence.

We do that adding 27 to α (for the conversions) and 473 to β (for the non-conversions). This gives us an updated posterior distribution:

Posterior = Beta(2+27, 18+473) = Beta(29,491)

Here's what that the graph looks like :



![](/posts/ab-testing/posterior.png)

This posterior distribution combines what we believed before the test with the new evidence from the data. It now represents our current belief about the true conversion rate of the Green button.

We model the Blue button the same way.

Starting with the same prior belief, that Beta(2, 18), centered at a 10% conversion rate, we now update it based on the observed data: 23 conversions out of 500 visitors.

Adding 23 to α for the conversions and 477 to β for the non-conversions gives us the updated posterior for the Blue button:

Posterior = Beta(2+23, 18+477) = Beta(25,495)

Graphically:



![](/posts/ab-testing/posterior2.png)

This new distribution reflects what we now believe about the Blue button’s true conversion rate, combining both our prior knowledge and the test data.

At this point, we’ve done something more powerful than produce two conversion rates. We’ve built two full probability distributions—one for each button. Each curve shows not just a single estimate, but a range of possible conversion rates, weighted by how plausible each one is, given both our prior belief and the observed data.

So instead of asking, “Is 5.4% greater than 4.6%?”, we’re asking a richer question: What do we actually believe about the true rate behind each button?

This isn’t something we can answer by comparing averages. Each distribution captures uncertainty, and those uncertainties overlap. What we really want to know is: **if we drew one plausible value from each distribution, how often would Green outperform Blue?**

There’s no simple formula for that, but the logic is straightforward. We can simulate the scenario: draw 10,000 samples from each distribution, compare them one by one, and count the fraction of times Green wins:



![](/posts/ab-testing/equation.png)

In our case:



![](/posts/ab-testing/equation2.png)
Graphically:

![](/posts/ab-testing/simulation.png)

This gives you something far more useful than a p-value: Based on the data and your prior, there’s a 71% chance that Green truly outperforms Blue.

Now you’re not just picking a winner: you’re quantifying confidence, weighing upside, and sizing up the risk.

Which is the actual job.

Why Bayesian methods can be more interesting and intuitive than frequentist ones when it comes to A/B testing.

Why Use Bayesian Methods for A/B Testing

It's not easy to find equilibrium between complex and simple when training a Machine Learning model.
Darts-based explanations about Bias-Variance tradeoff never convinced me, so I decided to break it down: what’s really behind this dilemma?

![](/posts/bias/darts.png)

Imagine everyday lots of dogs and cats pass through your window. Some of them are big, some are small ones.
You decide to classify them by height and weight, so you take measures for a bunch of animals. Cats are usually smaller and lighter. Dogs are generally bigger or heavier.

![](/posts/bias/scatter.png)

To classify these animals we can use Machine Learning. These models generalize from seeing some examples. Generalizing means that when presented a new animal, the model will be able to classify between cats and dogs. The same way a child see cats and learn what a cat is, computers see data and do the same. But choosing an model is not an easy task. We can use one line, lots of lines or even curves to separate dogs and cats.

![](/posts/bias/training.png)

The traced curves are trained on the dataset to classify every new point *below* it as a cat, and every point *above* it as a dog. The line does not separate all dogs and cats correctly while the squigly curve separate all dogs and cats correctly.
To compute the performance of an model, we count it's errors on the measures recorded. A well trained model should be able to classify animals it has never seen before correctly.\
When unseen data is presented, the traced curves below perform differently. The line has some errors, while the curvy line performs poorly.

![](/posts/bias/testing.png)

This task in Machine Learning is called classification, where the learned function outputs categories.
The model is just a function which takes examples of the input space - dogs and cats information - and return labels `cat` and `dog` whether it is a cat or a dog respectively.

There exists lots of functions that receive an animal and return a {`cat`, `dog`} label. They can look like lines, curves or horrors behind our compreehension. We are searching for a classifier that can correctly distinguish between cats and dogs. Below I drew the space of all functions that take an animal and return a class. Each green point in this space is a function.

![](/posts/bias/all.png)

To measure how good each function is, we input information of *all* animals, compare with the labels and count how many times the function mislabeled the pet. This measure is called the function risk. The bigger the risk, the worse the function is, so we are searching for a small risk function.

That is a problem. To compute the true risk we need all dogs and cats information. All pets that lived, those which are alive right now and the ones which are still not born. Poetic - but unrealistic. An ideal model would need data from every pet, past, present and future, in order to classify with precision whether a pet is a dog or a cat.
But we cannot have such big dataset. Instead, we use a smaller one - some dogs and cats from our neighboorhood we measured.

![](/posts/bias/classification.png)

The set of all pets hides a probability function. We cannot see in more three dimensions, so to exemplify, consider only the height and class of the animals `cat` and `dog`. A point in the three dimensional graph has height, class and probability. It says which is the probability of finding a pet with this class (`cat` or `dog`) and this height.

![](/posts/bias/prob.png)

By selecting a height value in this distribution, we obtain another distribution which tells us exactly whether an animal this tall has more chance of being a cat or a dog.

![](/posts/bias/marginal.png)

Unfortunately, we don't have acess to this distribution. If we had, we could skip the learning part.
And since we don't have all images nor the data distribution, we cannot compute the real risk. Our only option is to use our dataset, a set of samples from the real distribution, to compute an approximation of the true risk named *empirical risk*.

![](/posts/bias/empirical_risk.png) 

Many functions exist - each with its own risk. Of the many functions that exist, one has the smallest possible risk. This magical function is sometimes called Oracle.

![](/posts/bias/oracle.png)

The Oracle wins because it cheats by consulting the probability distribution to classify an image. If the image has greater chance of being classified as dog, the Oracle classify as a dog. Otherwise, as a cat. No wonder why it is so good.

![](/posts/bias/oracle_working.png)

Our task as engineers and scientists is to get closer to the Oracle function as possible. Our best try is a function that lives happily in the space, trained on whatever dataset we found.

![](/posts/bias/estimated.png)

The space of functions that take pets and classify as cats or dogs is very diverse. Many functions - linear, quadratic, step functions, discontinuous and other ones - live there. All of them classify animals as cats and dogs.

![](/posts/bias/functions.png)

Nested inside this vast space are subspaces - and each subspace contains a smaller portion of all functions.
Some of these subspaces are bigger than the others. Take the quadratic space, a space where the quadratic functions live.
Linear functions are a subset of quadratics where the quadratic term is zero.
Thus, the quadratic functions space also contains the linear functions space.

![](/posts/bias/linear_quadratic.png)

In Machine Learning, when choosing between Random Forests, Logistic Regressors or Neural Networks to classify a dataset, we also choose a subspace.
If we choose a linear model, like Logistic Regression, we will use a function that lies on the linear functions subspace. Choosing a class of functions means instead of looking through the whole space of functions that take animals as inputs and classify them, we will search over the subspace.

The challenge is choosing which subspace to search.

![](/posts/bias/subspace1.png)

We have no clue whether the Oracle function lives or not in the subspace.
The only confidence is that one function in this subspace has the smallest risk.
During training, when the model learns to classify labels based on images from the dataset, we aim to find this lowest-risk function.
The best we can do is learn a function as close as possible to it using our dataset.

![](/posts/bias/subspace2.png)

We also need to measure how well functions in our subspace can approximate any function in the full space.
Thus, we shall also minimize the distance between the best function in the subspace and the Oracle.

![](/posts/bias/subspace3.png)

At this point we have a more precise view of the learning problem. There are functions that take animals images and return labels; we approximated each function true risk using the empirical risk. An ultimately good function - the Oracle - exists. And to learn we should choose a class of functions. We are asked, what class of functions shall we use?

In one form or another, this problem has long worried engineers and scientists. This is the widely known problem in statistics entitled bias-variance tradeoff.

If we choose to separate classes using a line, every relation we ever find would be linear. But this would not come from data. Instead, it will be a bias imposed by us.
It will have high risk, making some mistakes and will not correctly separate the animals.

![](/posts/bias/linear.png)

In our example, tracing a line to classify dogs and cats is choosing the reduced subspace of linear functions to search on.
A small space means the distance between the better function we can achieve and the Oracle will be very big.
Eventhough, it's easier to find a good estimate for the function inside the subspace since it's too small.

![](/posts/bias/bias.png)

Conversely, if we choose a very curvy line to separate each dataset, we would always be able to separate pets from the dataset correctly.
However, the model would fluctuate wildly, depending on how accurate our dataset reflects the real world.

![](/posts/bias/squigly.png)

In a large subspace, the best achievable function is close to the Oracle (which might even reside there).
Although, an overly large subspace makes the model sensitive to dataset changes.
We could find a function with big risk, small risk or medium risk - the model would suffer from large variance.

![](/posts/bias/variance.png)

In Machine Learning, it's common to explain this duality using the terms underfitting and overfitting. When the model suffers with high variance, we say it *overfitted*. Elseway, models with high bias suffer from *underfitting*.
The ideal model - the one with lowest risk - lies somewhere between those two worlds.

![](/posts/bias/dilemma.png)

To classify correctly the pets in our dataset, a quadratic function would be ideal. Not simple as a line, nor too complicated as the high degree polynomial.

![](/posts/bias/quadratic.png)

Thus, we find the perfectly equilibrated subspace to estimate our function. But the journey of a scientist extends beyond quadratics. The toolbox of a good one should include regularizers.

Darts-based explanations about Bias-Variance tradeoff never convinced me, so I decided to break it down: what’s really behind this dilemma?

Bias-Variance tradeoff: understanding model complexity

Recommendation systems don’t understand preferences, they just measure behavior.

Every click, scroll, and rating becomes a vector. Then it’s all math: compare those vectors, spot patterns, and guess what comes next.

What feels like intuition is really correlation. Strip away the complexity, and it’s just the machinery of judgment: quantifying behavior, comparing signals, and betting on what comes next.

This post is about those measurements: how we encode behavior into numbers, and the different ways we can decide when two things are "similar enough" to drive a recommendation.

 

# Finding Simularities in Data Through Distances

To see how this works in practice, let’s look at something familiar: restaurant ratings.

Imagine we’ve recorded how three different people scored a hippie café and the fast-food place next door. The ratings go from zero to five. Zero means they hated it, and five means they thought it was excellent.

![](/posts/recommendation-systems/table01.png)

Now, let’s say Peter enters the picture. We’ve got his ratings for the same two restaurants, and we’d like to use that to suggest one he hasn’t visited yet.

![](/posts/recommendation-systems/table02.png)

We can approach this by visualizing everyone's restaurant preferences, including Peter's, on a graph, where each person is represented by a point based on their ratings.

![](/posts/recommendation-systems/graph01.png)

The key is to find people whose tastes are similar to Peter’s. On the graph, that means looking for points near his. The closer two points are, the more alike their ratings.

One way to do this is by summing the vertical and horizontal steps it would take to get from one point to the other, as if we could only travel in straight lines, up or across.

![](/posts/recommendation-systems/graph02.png)

This method is called Manhattan distance (or [`cityblock()` in SciPy\`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cityblock.html)).

It is fast, which comes in handy when theres a lot of data to process. Sometimes fast is good enough.

Another option is to draw a straight line between the points. That’s Euclidean distance (SciPy’s [`euclidean()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.euclidean.html)).

![](/posts/recommendation-systems/graph03.png)

![](/posts/recommendation-systems/table03.png)

By identifying Alice as Peter's nearest neighbor, we've narrowed our search. Her high ratings now serve as the most relevant data for predicting what Peter might enjoy next.

Both Euclidean and Manhattan distances are, in fact, variations of a broader mathematical idea known as the **Minkowski distance**.

Rather than being a single formula, [Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) is a family of distance metrics that flexes based on a parameter,\_ p\_. This parameter determines how we combine the differences across each dimension to calculate the total distance.

Mathematically, it looks like this:

![](/posts/recommendation-systems/equation.png)

At `p = 1` (Manhattan distance), the metric treats all differences equally. Each coordinate's difference counts linearly, like walking city blocks: a few small steps in many directions matter as much as one big leap in one.

At `p = 2` (Euclidean distance), the metric begins to emphasize larger differences more. This is the familiar "straight-line" distance, where bigger gaps have an outsized influence.

As p increases beyond 2 the metric becomes even more sensitive to the largest differences in any coordinate. In the limit, as `p → ∞`, only the single largest difference matters. This is known as Chebyshev distance, where the metric effectively says, "I only care about the worst-case deviation."

By tuning p, we can tailor how sensitive our similarity measure is to outliers, clusters, or particular behaviors.

These algorithms deliver speed, certainly. But before trusting the output, we might ask whether everyone’s measuring with the same ruler or if some are holding theirs at a slant.

 

# The Pearsons Correlation Coefficient

Up to this point, we’ve treated ratings as if they’re uniform: a 4 from one person means the same as a 4 from another.

But one may only rate 4s and 5s. Another stays below 3. A third switches between 2 and 5, with no middle ground.

How do we compare ratings when the same number reflects different standards?

The Pearson correlation coefficient helps us cut through that subjectivity by comparing patterns, not scores. Consider the table of ratings below:

![](/posts/recommendation-systems/table04.png)

Alice and Bob both rated four restaurants. They agreed that Restaurant C was the best, followed by A, then B, and finally D.

The numbers they assigned differ. Alice sticks to the 4–5 range, Bob is more liberal with the lows, but their ranking is identical.

Plot Alice's ratings on one axis, Bob's on the other, and the points align neatly along an upward-sloping line. This is what **perfect agreement** looks like—not in absolute values, but in pattern.

![](/posts/recommendation-systems/graph04.png)

If Alice gave high marks to a place Bob hasn't visited, it's likely he'd feel the same.

Pearson correlation turns less obvious patterns into useful insights. It doesn't matter whether someone is generous or stingy with their scores.

It measures is how tightly two sets of numbers track each other's ups and downs: are they moving together linearly, even if not in the same register?

In this case, even though Bob’s scores dip lower than Alice's do, their preferences move in sync. High for Alice means high for Bob. Low means low. The Pearson coefficient for their ratings is near +1. That's a strong positive correlation (SciPy's [`pearsonr()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html) lets us compute it directly).

Now reverse Bob's ratings: what Alice loves, Bob hates, and vice versa. The plot now tilts downward, forming a straight line in the opposite direction. That's a Pearson coefficient of -1. That's perfect *disagreement*.

![](/posts/recommendation-systems/graph05.png)

Finally, imagine a scatterplot of ratings with no pattern at all. Alice's highs are Bob's lows, mids, or anything in between. There’s no consistent relationship. The Pearson correlation here hovers around 0, signaling no meaningful alignment.

![](/posts/recommendation-systems/graph06.png)

Pearson captures the strength of a single relationship. But good recommendations usually come from patterns across a crowd, not just one person.

That’s where k-nearest neighbors steps in. 

# K-nearest neighbor

Consider two users that might look identical on paper: both rate horror films highly and avoid animation and binge thrillers. Pearson and the distance metrics treat them as the same profile, so recommending one user’s favorite to the other seems reasonable.

But then one gives five stars to a romcom buried in the catalog and we recommend Notting Hill to the other. It doesn’t go well.

Anomalies like this are inevitable at the individual level. The group, however, tends to converge on something stable.

We can harness that stability by leaning on what we already know: use Euclidean or Manhattan distance to identify a group of K users who rate content in roughly similar ways. That gives us proximity, which is a behavioral neighborhood.

Next, we bring in Pearson correlation to evaluate how closely each user’s taste aligns with the target’s preferences.

Now we have both pieces: who’s nearby, and how much their opinion should count.

That's the core idea behind k-nearest neighbors.

Assume Alice and Bob are the two users most similar to our target Peter, to whom we're trying to recommend a new restaurant. Their Pearson correlations with Peter’s ratings for the hippie restaurant are:

![](/posts/recommendation-systems/table05.png)

Their inputs aren't equal. Alice's correlation with Peter is stronger, so her opinion is weighted at 60%, while Bob's accounts for 40%. This weighting reflects the degree of alignment.

We compute the recommendation by aggregating these two inputs proportionally.

![](/posts/recommendation-systems/piechart.png)

It’s a clever balance: multiple perspectives, weighted by similarity.

But there's no such thing as a free lunch: the more neighbors we include, the more time we spend calculating correlations. And the more neighbors we include, the more likely we are to include outliers or dilute the signal, which can lead to more generic recommendations.

 

# Knowing Your Data Matters

How you measure similarity shapes what you recommend. The better the fit to your data, the better the recommendations that follow.

Collaborative filtering is a solid foundation. Real-world systems often go further, blending these tools with deeper models of behavior and context.

When most values are present and their magnitude is meaningful, distance metrics like Euclidean or Manhattan can capture genuine gaps in behavior.

But when users apply the same scale differently—some rate high by default, others hold back—Pearson correlation is better. It ignores magnitude and focuses on whether preferences move in the same direction.

In a k-nearest neighbors setup, your choice of metric defines who qualifies as "nearest." Distance gives you behavioral closeness. Pearson gives you preference alignment. Combine both, and you get not just who’s similar—but how much their opinion should count.

No single metric fits every case. What matters is understanding your data and what similarity should actually mean.

Building an intuitive comprehention on recommendation based on distances and the Pearson coefficient

The geometry of taste: putting numbers on preferences

Learning, to a computer, is just turning bad guesses into better ones. In this post, we’ll see how that starts with a straight line: how linear regression makes the first guess, and gradient descent keeps improving it.

Let's start with something familiar: house prices. Bigger houses tend to cost more; smaller ones, less. It's the kind of pattern you can almost see without thinking: more space, more money.

When we plot it, the shape is clear: a loose upward slope, with some noise but a definite trend.

![](/posts/least-squares/scatter.png)

As you can see, price and size move together in a way that feels predictable. Not in fixed steps or categories, but on a sliding scale. A house might go for $180,000, $305,500, or anything in between.

Now imagine you're selling your own house. It's 1,850 square feet—larger than average, but not a mansion. You've seen what homes go for in your area, but the prices are scattered. What's a fair number to list it at?

One option is to text your real estate friend and get a half-baked guess. A better option is to look at the pattern in past sales and sketch a line that seems to match it. You grab a ruler, hold it up to the scatterplot, and draw something that feels about right. Then you find your square footage on the x-axis, trace upward to your line, and read off the predicted price.

![](/posts/least-squares/scatter_line.png)

Whatever line you draw, it'll be a steady upward slope. Bigger homes, higher prices. It might not be perfect, but it gives you a way to turn square footage into a price that kind of makes sense.

And while the lines you might draw vary, they all follow the same formula. Each one takes the area of a house (the explanatory variable), multiplies it by a number (the `slope`), and adds another number (the `intercept`).

![](/posts/least-squares/linearFormula.png)

Notice that this isn't just a house-pricing formula. It's how every straight line works. One number sets the tilt (the `slope`), and the other shifts the line up or down (the `intercept`). That's all it takes to draw a line: scale, then slide.

![](/posts/least-squares/slope.png)

In our case, the slope is the "price-per-square-foot". That's the amount each extra foot adds to the total. If the slope is 150, then a 1,000 square foot home would land at $150,000 just from size alone. A steeper slope means prices rise faster as homes get bigger.

![](/posts/least-squares/intercept.png)

The intercept is where the line starts. It's the predicted price of a home with zero square feet. That doesn't mean much in isolation (nobody's buying a 0-square-foot house), but it sets the baseline, like the minimum price you'd expect even for the tiniest studio. If two neighborhoods increase in price at the same rate per square foot, the one with the higher intercept starts from a pricier floor.

Now that we know each line connects an explanatory variable to a prediction using just a slope and an intercept, the question is: **which line should we trust to price our house**?

To answer that, we need a way to measure how well a line fits the data we already have.

Take one of the houses in our dataset. The line says it should've sold for $350,000. But we know the actual price was $375,000. That $25,000 difference is an "error" that measures how far off the line was for that point. Price it too low and you leave money on the table or it too high and the house might not sell.

![](/posts/least-squares/error.png)

A good line keeps the gaps between predictions and actual values small. The simplest way to measure how well a line fits is to add up those gaps—the errors.

Of course, there's more than one way to measure error. And how we choose to combine them is what turns guessing into regression.

One simple option is to use absolute error, just like we did in the previous example. You take the difference between the predicted price and the actual price (no matter which one is higher) and treat it as a distance. If the house sold for $375,000 and one line guesses $350,000, the error is $25,000. Another line guesses $360,000? That one's off by $15,000. Smaller is better.

To compare lines, you just add up the absolute errors across all the houses. The line with the smallest total error is the winner.

If you're trying to price your own home, this makes intuitive sense: you'd rather be off by $10,000 than $30,000.

But absolute error treats all mistakes the same—two medium errors count the same as one big one. That can hide problems.

If your predictions are always a little off, you're consistently close. But if one estimate nails the price and another misses by $100,000, it's hard to know if the model is solid or just lucky. Consistency matters more than the occasional hit, especially when prices shift over time.

Take home prices, for example. Imagine checking listings in your neighborhood every week. Most of the time, estimates are a little off—maybe $10,000 high or low. Nothing to worry about. But then, one day, a house just like yours is priced $150,000 below market. The next week, another is listed way too high. Suddenly, you're not just surprised—you stop trusting the estimates altogether.

To recover trust in the estimates, we might want big mistakes to count more—not just equally. One way to do that is to make the penalty grow faster as the error increases. In effect, we're plotting the errors on a curve, not a straight line. The further out you go, the steeper the penalty becomes. That's what it means to use a non-linear scale: small misses barely move the needle, but big ones explode.

![](/posts/least-squares/comparing.png)

Instead of measuring error directly, we square it. The effect is clear in the plot above: a $20,000 miss isn't just twice as bad as $10,000—it's four times worse. A $50,000 error? Twenty-five times the cost. Bigger mistakes grow fast. Squaring the errors pushes us toward a line that stays consistently close, not one that swings between lucky guesses and huge misses.

Everything we've done—measuring errors, choosing how to weigh them, and adding them up—comes together into a single idea: the *error function*. It's just a rule for scoring how well a line fits the data. Squared error is a popular option. Absolute error is another, but they are not the only ones.

Different error functions reflect different priorities. For example, the [Deming regression](https://en.wikipedia.org/wiki/Deming_regression) uses a different error function that accounts for errors in both variables—not just the one we're trying to predict. It's useful when both measurements are noisy, like when comparing lab results from two different instruments.

Many lines are close to the data points. We don't want any good line, but the best one (the one with smallest error). No matter which error function you choose, the goal stays the same: find the line that minimizes the error function.

But that raises the next question: where do these lines come from? One option is brute force. Get a set of every possible combination of slope and intercept, one by one. Then, calculate the error and keep the best one.

It works. At least in theory. But there's a problem: there are way too many possibilities. Infinite ones. Testing them all would take forever. We need a more sensible algorithm than computing the error of every possible line.

To understand what we're up against, imagine a plot where each point represents a different line: the slope and intercept are the coordinates, and the vertical axis shows how bad the line is—its error. In this landscape of errors, high points mean worse lines; low points mean better ones. Our goal is simple: find the lowest point. We won't actually draw this plot (there are too many possibilities), but picturing it helps make the task clear.

For example, if you look at the absolute error surface, it's a bit of a mess. It's not smooth like a hill or a valley. Instead, it's more like a weird origami sculpture, with sharp edges and folds.

![](/posts/least-squares/absolute3d.png)

In some cases, like in the example below, every point along the fold gives the same lowest error. If that happens, it means there's not *one* best line, but *many*.

![](/posts/least-squares/folded3d.png)

You can see this clearly in a 2D slice: fix the intercept at zero, and a range of slopes yield the same lowest error.

![](/posts/least-squares/absolute2d.png)

This ambiguity matters because if multiple lines fit the data equally well, then we won't have a single best answer—just a range of equally good ones. Imagine trying to sell a house whose price could reasonably be $100,000, $500,000, or $1,000,000, depending on which line you pick. Not ideal.

This situation happens when the data points are symmetric: tilting the line slightly brings it closer to some points while moving it away from others, without changing the total error. The plot below shows an example. Any line that passes through the green area (like the red one: is a valid solution.

![](/posts/least-squares/multiplesolutions.png)

This kind of ambiguity happens to absolute error only. If we were using squared error instead, the plot would look different: smooth and bowl-shaped, with a single lowest point. One best line, stable and predictable.

![](/posts/least-squares/squared3d.png)

If you were standing anywhere on that surface, you'd just need to walk downhill to find the best line. Pick the direction where the slope is steepest, take a step, and keep moving downhill. Eventually, you'll reach the bottom.

You don't have to explore in every direction or worry about getting stuck. No matter where you start, as long as you follow the steepest descent, you'll get to the best solution. It might take 100, 1,000, or more steps, but you'll get there.

Of course, we're not *actually* hiking across hills. In our case, walking downhill means tweaking numbers: the slope and intercept of the line. Each point on the error surface corresponds to a specific line. Nearby points represent lines with similar parameters. By nudging the slope and intercept step by step, we move through this space—chasing lower and lower error until we land on the best fit.

At that point, we can stop and look at the slope and intercept of the line we found. That's our best guess for the price of your house.

To get a better feel for what's happening as you step down the valley, you can zoom in on just one dimension—the slope. The plot below shows how the error changes as we adjust the slope (while keeping the intercept fixed). Notice how the curve is smooth and bowl-shaped, making it clear how to keep stepping downhill.

![](/posts/least-squares/squared2d.png)

At every step, we need to decide which way to move: left or right. To do that, we measure how steep the curve is at our current point. This measurement (the steepness of the curve) is called the derivative.

If the derivative is positive, it means the error increases if we move right, so we should go left. If the derivative is negative, it means the error increases if we move left, so we should go right.

In short: the derivative points uphill. Since we want to go downhill, we move in the opposite direction.

We keep stepping like this—checking the derivative, flipping directions if needed—until we land at a point where the slope is flat. That's when the derivative becomes zero, and we've reached the minimum.

![](/posts/least-squares/gradient.png)

> When using least squares, a zero derivative always marks a minimum. But that's *not* true in general. It could also be a maximum or an inflection point. To tell the difference between a minimum and a maximum, you'd need to look at the [second derivative](https://en.wikipedia.org/wiki/Second_derivative). If the second derivative is positive, it's a minimum. If it's negative, it's a maximum. If it's zero, we're at an inflection point.

This process of iteratively minimizing a function using derivatives is called **gradient descent**.

The gradient descent is an intuitive and efficient algorithm—and squared error plays especially nice with it.

Squaring the errors keeps everything smooth: no sharp corners, no sudden jumps. That matters because gradient descent needs to measure slopes—and if a function isn't smooth, you can't always define a clear slope to follow.

This wasn't always obvious. In the 1800s, mathematicians like Karl Weierstrass showed that you can have a perfectly continuous curve that's still so jagged that it has no slope at all—not even at a single point. It was a wake-up call: just because something looks smooth from far away doesn't mean it actually is.

Minimizing absolute error runs into a smaller version of this problem. The absolute value function has a sudden bend at zero—right where you most need a clean slope. You can work around it with special tricks, but it's messier and less natural.

Squared error, on the other hand, glides along smoothly. Its derivative is clean, continuous, and geometrically meaningful—it not only points you downhill, but also tells you how big a step to take. No hacks required.

**This is pretty much the real reason why everyone squares their errors instead of taking the absolute value, even when it might not be the most appropriate pick. Squared error makes optimization smooth and easy *and*, as we saw earlier, it also guarantees a single, stable best solution.**

As a side note, there are many ways to find a function's minimum, but Gradient Descent rose to fame thanks to its handsome cousin: the [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent), which is the algorithm of choice for training neural networks.

Underneath all the complexity, deep learning still runs on the same basic idea: adjusting parameters to minimize error, step by step, just like we do with Linear Regression and Squared Errors.

It's simple math, sharpened by persistence. The kind of quiet hustle that helps you put a price on a house and actually trust it.


Building an intuitive understanding of how Linear Regression works and how it leads to Gradient Descent

How a straight line teaches machines to learn

**Today, we're launching Briefer's AI anayst.**

Briefer's AI analyst is an intelligent agent that helps *anyone* on your team turn data into clear, instant insights.

<iframe width="640" height="360" src="https://www.youtube.com/embed/VBXmDJCqOGk" title="" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Notice that this isn't just another text-to-SQL chatbot. Instead, it's an AI agent running inside our powerful data workspace (the one technical users already love).

Our agent can write SQL, plot graphs, and even use Python for deeper analysis.

You can also upload your own business context and metadata, making the agent even smarter.

And because it operates directly in Briefer's notebook interface, you'll see exactly how it thinks—every query, every action, every line of code. That means full transparency and auditability for every insight.

**If you're serious about making your team truly data-driven, [book a demo using this link](https://calendly.com/lucasfcosta/briefer-intro)**.

Today, we're launching Briefer's AI anayst—our intelligent agent that helps anyone on your team turn data into clear, instant insights.

Launching Briefer's AI analyst

When I was writing my first book, my publisher taught me that "books with plenty of diagrams sell better". People flip through, see the visuals, and *feel* like they'll understand it. They don't read a word. They just *know*, and they buy your book.

Same way you'll scroll through this post and think, "yeah, I'll get this."

That's the trick. Visuals feel easy. Your brain sees shapes and goes "I get this." You don't need to decode a table or read a paragraph. **Patterns just pop out.**

**That's exactly why you should plot your data before doing anything else**. Not for decoration. For *understanding*. You'll spot weird outliers, duplicated trends, clumps of activity, empty gaps—things you'd never notice in raw numbers.

In this post, we'll look at the best plots for getting a feel for your data and what to look for in each one.

 

# Histograms

Let's begin with something boring: histograms. They're the fastest way to see the shape of your data—where it piles up, where it thins out, and whether anything looks off. You'll spot skew, gaps, fat tails, or the classic "oh no, this is just one giant spike." It's not fancy, but it tells you what you're dealing with. Before fitting models or running tests, just plot that damn histogram.

![](/posts/look-the-data/histogram.png)

Start with the obvious: outliers.

That one bin way off to the side can stretch your axis, distort your stats, and confuse your model. It might be a bug. Or it might be the most interesting person in the data, like the Rollercoaster Tycoon dev who hand-wrote 50,000 lines of assembly. Either way, it's shaping your results, whether you notice or not.

![](/posts/look-the-data/outlier.png)

Then look at the shape of the distribution. Some datasets don't clump around a single peak. Instead, they split.

![](/posts/look-the-data/multiple_peaks.png)

Two humps, maybe three. That's not noise, that's signal. If your histogram has multiple peaks, don't just shrug. Go figure out if you're looking at different groups mixed together.

Say you're tracking how often people use the terminal: one peak for folks who barely touch it, another for people who live in it. Dig deeper, and you realize Windows and Linux users were hiding in the same chart.

![](/posts/look-the-data/multiple_peaks_groups.png)

That kind of split is called a multimodal distribution, and it's a warning: your data isn't telling one story. It's telling several.

Spotting that kind of split matters. It changes how you model, how you segment, how you interpret the results. Otherwise, you're averaging across groups that don't belong together, and whatever you build next will be based on fiction.

Finally, watch out for skew. A distribution is said to be "skewed" if one tail stretches out longer than the other.

A lot of real-world data behaves this way. Take salaries: most people cluster toward the bottom, but a few stretch the scale so far you start questioning the system. That long tail on the right? That's a right-skewed distribution.

![](/posts/look-the-data/skewed_histogram.png)

Skew isn't the same as outliers. Outliers are a few rare, isolated points. Skew is *structural*. It's the shape of the whole distribution. Most users get crammed into one end of the range, while a long tail quietly stretches out and distorts everything else.

Think of a response time metric for an app. Most users get a response in under a second, but a few unlucky ones wait 5, 10, even 20 seconds. That long tail skews the whole distribution.

Now, say you build an alert that fires if response time goes above the 95th percentile. Sounds smart, until you realize that, for most users, that alert never triggers, even when things feel slow. Picking a sensible "default timeout" is just as tricky because the average is skewed by a few extreme cases, so you end up setting a default that's too generous for most users, and too short for the long tail.

Left skew is the opposite: the long tail drags left. Either way, skew can mess with your stats and assumptions. But in many cases, a simple transformation can help straighten things out.

If you see heavy skew, try a simple transformation (like a [log scale](https://briefer.cloud/blog/posts/logarithms/)). It'll compress the extremes, pull the bulk apart, and make patterns easier to see and model.

![](/posts/look-the-data/normalized_histogram.png)

 

# Quantiles

Quantiles slice your data into equal-sized chunks. They give you a structured way to understand the spread without getting bogged down in raw numbers. Instead of eyeballing two histograms and guessing which one's "more spread out," you can just compare their quantiles. Are the medians far apart? Is one dataset more compressed in the middle but stretched at the edges? Quantiles help you spot those patterns. Therefore, plotting quantiles right on the histogram makes those patterns easier to see and easier to explain.

Some of the most familiar quantiles are percentiles and quartiles. Percentiles slice your data into 100 pieces, quartiles into 4. They show up everywhere, from test scores ("you're in the 90th percentile") to dashboards ("p95 latency"). The middle quartile (the 50th percentile) is your median, and it's often a better summary than the average, especially when your data's skewed or full of outliers.

![](/posts/look-the-data/quantiles.png)

Let's say you're tracking API response times. You plot the histogram and it's a mess: most requests are fast, but a few are way out to the right. Now add vertical lines for the median, 95th, and 99th percentiles. Suddenly, the picture sharpens. The median says "half our users get a response in under 200ms". Great. But the 95th percentile shows some are waiting 2 seconds, and the 99th is worse. Those tails matter. They're the users calling support, churning, or tweeting screenshots. Plotting quantiles doesn't just show spread. It tells you who's going mad.

 

# Box-and-whiskers

Quantiles are great for understanding spread but they're still just lines on a histogram. What if you want to compare distributions side by side, across groups or categories? That's where box-and-whisker plots come in. They take those same quantiles and turn them into a compact, visual summary.

The box shows the middle 50% of your data, from the 25th to the 75th percentile. The lines, called whiskers, usually extend to the smallest and largest values that fall within 1.5 times the height of the box. Anything beyond that is flagged as a potential outlier and plotted separately. That's the standard in most tools like Python's `matplotlib` and `seaborn`, or R's `ggplot2`, and it gives you a quick way to see both the bulk of your data and what's unusually far from it.

![](/posts/look-the-data/boxplot_iqr.png)

For example, let's go back to our terminal usage data. This time, we plot a box-and-whiskers chart comparing Linux and Windows users. Now the difference jumps out: the least active Linux user still uses the terminal as much as the median Windows user. The entire Linux distribution is shifted higher. You could see the peaks in the histogram, but now the comparison is concrete. You can point to the medians, the spread, the outliers. It's no longer just a feeling. It's a fact.

![](/posts/look-the-data/boxplots.png)

 

# Violin plots

Box plots are great for comparing groups, but they hide one important detail: the actual shape of the distribution. You get the quartiles, the median, and some outliers, but not whether the data is clumped, stretched, or split into separate groups. It can make everything look smooth and simple, even when the reality is a lot messier.

Violin plots fix that. They keep the summary from the box plot, but overlay it with a smoothed curve that shows the full distribution (essentially, a rotated and mirrored histogram). That way, you get the best of both worlds: clean comparison and full context.

![](/posts/look-the-data/violin.png)

But violin plots only work when you're comparing distributions, not when you're tracking values over time or looking at raw counts. And visually, they can get messy and overcomplicate simple data. So use them when you need to compare distributions - but for other types of comparisons, you'll want different tools.

 

# Bar plots

If you're comparing a lot of groups, violin and box plots also start to fall apart. Too many shapes, too little clarity. In those cases, it's better to aggregate the data (using the mean, median, max, etc.) and switch to a bar plot. Bar plots make it easy to scan and compare lots of categories side by side.

And just a quick reminder: **avoid pie charts**. They make it almost impossible to compare values accurately, especially when the slices are close in size. Your brain isn't wired to judge angles or arc lengths, but it's great at comparing bar heights. Pie charts look friendly, but they hide the data.

To bring it home, here's a simple example. I pulled some of the most popular programming languages from the Stack Overflow 2024 survey. The bigger the bar, the more people use it. No smoothing, no quartiles. Just raw counts, stacked side by side.

![](/posts/look-the-data/bar.png)

And yes, unfortunately, JavaScript is still there. Some truths you just have to accept.

 

# Scatter plots

Bar plots are great for comparing quantities across several categories-one variable, maybe broken down by groups or periods. But they can't show how two things change together, like whether developers who sleep more write fewer bugs. That's where scatter plots come in.

Scatter plots let you see relationships between two continuous variables. Want to know if more debugging leads to less happiness? Or if people who drink more coffee ship more features? Plot the points. Each one is a pair (X and Y) and the shape they form tells the story.

Here's a classic example: happiness vs. time spent debugging. Each red dot is a programmer. The trend is clear: the more time you spend debugging, the less happy you are.

![](/posts/look-the-data/scatter.png)

So what should you look for in a scatter plot? Start with the shape. If the points stretch into a line or an ellipse, there's probably a real relationship. If they form a vague cloud or a neat circle, the variables likely aren't connected, like happiness and Pluto's position. Unless, of course, you believe mood swings are caused by retrograde motion.

![](/posts/look-the-data/no_relation_scatter.png)

You might also spot clusters: groups of points that behave differently from the rest. Maybe there's a bunch of users who are never happy, no matter how little they debug. Plotting them separately can reveal what's hiding underneath. Turns out, they're all Java developers still stuck writing getters and setters.

![](/posts/look-the-data/scatter_groups.png)

And don't forget about outliers. Scatter plots are great at making them stand out. Like Jonathan, the programmer who loves debugging. While everyone else sinks, he soars. Maybe he's an error. Maybe he's a genius. Either way, he breaks the pattern.

![](/posts/look-the-data/scatter_outlier.png)

Sometimes that's all you need: a clear line, a weird outlier, a tight little cluster. But other times, the pattern hides in the shape. Not every relationship forms a neat straight line. A lot of traditional stats assume it does, but real data often bends, flattens, or takes strange turns. That's exactly where plotting helps. It shows you when things are drifting from expectations before you build the wrong mental model.

Say you're looking at how coffee affects programmer productivity. Plot the data, and you might notice something interesting: productivity rises at first, but then levels off. That kind of pattern is nonlinear: it curves instead of forming a straight line.

![](/posts/look-the-data/nonlinear.png)

Why does that matter? Because if you assume a straight line when the relationship curves, your predictions will be way off, especially at the edges. You'll think the tenth cup of coffee makes you twice as productive, when in reality it just gives you a minor panic attack.

Sometimes, a simple fix, like applying a log scale to the axis, can help straighten things out. Plots like this don't just show you what's happening; they show you when simple assumptions break down, and when you might need to look at the data a little differently.

 

# Heatmaps

Scatter plots are great for seeing how two variables relate. But what if you need to add a third variable? That's when you turn to heatmaps.

Instead of points scattered on a plane, a heatmap uses color to represent that extra dimension, making complex patterns easy to spot.

A classic example is GitHub's contribution heatmap. The two axes show months and weekdays, while the color shows how many commits someone made each day. Take a look at the one below. That's Linus Torvalds' contribution history. At a glance, you can see when he's most active, and when things cool down.

![](/posts/look-the-data/heatmap.png)

Choosing the right color palette matters because it shapes how people interpret your data. Usually, darker colors feel like "more" and lighter colors like "less," but that's not a fixed rule, it depends on context.

Imagine flipping GitHub's heatmap colors: intense, dark squares for zero activity would feel weird, right? When your data has a meaningful midpoint, like temperatures centered around zero, it's best to use a diverging color palette, where two colors meet in the middle. Cold goes one way, hot the other, and zero stays neutral.

![](/posts/look-the-data/palettes.png)

So what should you look for in a heatmap? Patterns and contrasts.

Look for clusters, where values pile up in one area, and gaps, places where things consistently go quiet. Pay attention to streaks-rows or columns that stand out-and sudden color shifts that might point to meaningful events or changes. Heatmaps are all about spotting these trends at a glance, so trust your eyes first, then dig deeper.

 

# Line plots

Bar charts shout. Pie charts charm. Line Plots reveal motion. They show how things change across time, categories and variables. The points represents average over time and the lines are maximum and minimum measurements.

Take Brazilian product sales over time. Here, the y-axis is units sold over years. Each line is a product: Watermelons, Easter eggs, and [Panetone](https://pt.wikipedia.org/wiki/Panetone), an Italian Christmas cake that's actually more popular in Brazil than in Italy. In a way, it's the Brazilian version of British mince pies.

![](/posts/look-the-data/line.png)

Here you can see that Panetone flatlines for most of the year and then skyrockets in December. Watermelon peaks in the hottest months, January and February, then cools off as summer fades. Chocolate eggs show a clear spike around Easter, then, they're gone like they never existed.

That's the magic of line plots: they don't just track numbers. They reveal timing, behavior, and opportunity, all in one continuous story.

 

# Putting it all together

We've seen a lot of plots-each with their own strengths. But across all of them, the questions you're asking are very similar.

Whether it's a histogram, a scatter plot, or a heatmap, the goal is to get a feel for the data. To see what stands out, what's typical, and where things behave differently than expected.

Here's what to look for-no matter what plot you're using:

* **What's typical?** What values are common, where does the data cluster, and where does it thin out?
* **What's unusual?** Are there outliers-points that stand far from the rest? Are they errors, or something worth investigating?
* **Is the data shaped how I expect?** Look for skew, gaps, and multiple peaks. Real-world data is rarely smooth.
* **Are there clear groups?** Do values naturally split into clusters? Can you tell them apart, or are they overlapping?
* **How do variables move together?** Does one go up as the other goes down? Is it linear, curved, or noisy?
* **What's missing?** Are there empty spots where data should be? Do any categories or ranges look oddly quiet?

The more you practice spotting these, the faster they jump out-long before you run any numbers. That's the real power of plotting: it gives your brain something to lock onto before your tools even get involved.

By the way, remember how I said visuals *feel* easy? How your brain skips the words and just says, "yeah, I get this"?

You just did.

Which plots to use to get a feel for your data and what to look for in each of them.

How to get a feel for the data

Today, we're launching [Briefer's open-source offering](https://github.com/briefercloud/briefer)

Briefer is a platform for people to build notebooks and dashboards in a single place. It's much more flexible than traditional BI tools because it allows you to write code and use any library you want to manipulate data and create visualizations.

<video autoplay loop muted playsinline class="w-full rounded-lg shadow-lg mt-3 mb-4 overflow-hidden border border-gray-100">
 <source src="/posts/launching-briefer-oss/full-demo-oss.mp4" type="video/mp4">
</video>

From today, you can use Briefer for free on your machine, or deploy it to your own infrastructure so that your whole team can use it and eliminate your data silos.

 

## What's included in the open-source offering?

![](/posts/launching-briefer-oss/oss-vs-paid.png)

Briefer's open-source offering includes everything you need to build notebooks and dashboards, including Markdown, SQL, Python, point-and-click visualizations, scheduled runs, writebacks, and more.

Also, given you'll be running Briefer on your own machines, there's no limit on how many users you can invite, how much memory and compute you can give to your notebooks, and how many pages you can schedule.

The only things that are *not* in Briefer's open-source offering are PDF exports, granular permissions, and email and Slack integrations. These are available in the paid version of Briefer, for which you can sign up [here](https://briefer.cloud).

 

## Why open-source?

We want people to have zero barriers to using Briefer. We believe that the best way to do that is to make it open-source.

By making Briefer open-source, we're giving you the freedom to run it on your own infrastructure, customize it to your needs, and contribute to its development.

Furthermore, given Briefer will be running on your own machines, you don't have to worry about your data being stored in a third-party service. You have full control over your data.

If you want an in-depth explanation of our open-source strategy, [you can read it here](/blog/posts/open-source-strategy).

 

## How do I get started?

You can get started by following the instructions in our [GitHub repository](https://github.com/briefercloud/briefer). While you're there, please don't forget to give us a star ⭐.

If you have any questions, please send me an email at lucas.costa\[at]briefer.cloud. I respond to every email I receive from users.

Now, it's time to build.

Best, LdC.

Launching Briefer's open-source offering

This changelog is all about quality-of-life improvements. We've added search, keyboard shortcuts, pivot tables, and support for Trino as a data source.

 

## Search

The more people use Briefer, the more pages they create. This can make it hard to find the page you're looking for. That's why we've added a search bar to the pages list.

![](/posts/changelog-9/search.gif)

To toggle the search bar, click the magnifying glass icon on the top right of the pages list. Then, you can type the name of the page you're looking for, and the list will be filtered to show only the pages that match your query. Alternatively, you can use the `CMD (or Ctrl) + K` keyboard shortcut to focus the search bar.

Please notice that this is not a full-text search. It only searches for page names for now.

 

## Keyboard Shortcuts

Many people coming from Jupyter Notebooks are used to using keyboard shortcuts to navigate their notebooks. That's why we've added a few keyboard shortcuts to Briefer.

![](/posts/changelog-9/shortcuts.gif)

Briefer's keyboard shortcuts have two modes: command mode and edit mode. You can toggle between them by pressing `ESC` and `Enter` (or focusing on the block), respectively.

In command mode, you can move between blocks using the arrow keys or `h`, `j`, `k`, and `l`, just like in Vim. You can also delete blocks with `dd`.

To add new blocks, you can use `a` to add a block above the current one, or `b` to add a block below it, followed by the block type you want to add: `m` for Markdown, `p` for Python, and `q` for query.

When editing blocks, you can use `CMD (or Ctrl) + Enter` to run the block, `Shift + Enter` to run the block and focus on the one below it, and `Alt + Enter` to run the block and add a new one below it. You can also `ESC` to exit edit mode.

**To view all these keyboard shortcuts within the platform, you can open `Keyboard Shortcuts` within the three dots menu on the top right of the screen when you're inside a notebook.**

 

## Pivot Tables

Pivot tables are a powerful way to summarize and analyze data. They allow you to group data by one or more columns and apply an aggregation function to the values in another column.

![](/posts/changelog-9/pivot-tables.gif)

To use pivot tables in Briefer, you can create a query block and write a SQL query to act as the source of the pivot table. Then, you can click the `Pivot Table` option when adding a new block to create a pivot table from the query results.

 

## Trino support

Some of our users have been asking for Trino support, and we're happy to announce that it's now available in Briefer.

To add a Trino data source, you can go to the data sources page and click the `Add Data Source` button. Then, you can select `Trino` from the list of available data sources and fill in the connection details.

We've made plenty of quality-of-life improvements to the platform this month. We've added a search bar to the pages list, keyboard shortcuts, pivot tables, and support for Trino as a data source.

Changelog #9: Search, keyboard shortcuts, pivot tables and Trino support

Today, we're launching [a free and open-source offering](/blog/posts/launching-briefer-oss). But I'll be honest with you: Briefer is a VC-backed company, and it must make money.

It's ironic to me that many *for-profit* companies can't say that out loud. Maybe it's because they're afraid that money will get in the way of making users happy. Or maybe they know it won't, but they're afraid of people thinking it will.

We don't think that way.

We think that doing right by your users is the best way to make money, and making money is the best way to do right by your users. That's because money allows you to invest in improving your product and offer a better service to more people, some of whom will use it for free, and that's fine.

![](/posts/open-source-strategy/oss-cycle.png)

Being clear about our strategy and intentions is part of doing right by our users and it's why I'm writing this post.

In it, I'll explain how companies can structure their open-source offerings, and how we structured ours in a way that's good for users and good for business. I'll also explain why we chose the AGPL license, how it benefits users, and how it benefits us too. Finally, I'll also explain some of the things we won't do, including changing our license.

 
 

# Structuring an open-source offering

![](/posts/open-source-strategy/summary1.png)

When companies open-source their software it means [they can't *really* charge for the software itself](https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowRequireFee)¹. Thus, if you must make money, you need to sell something else, which is usually:

1. Support and professional services
2. Compute and infrastructure
3. A non-open-source version of the software

In the first case, companies scale as a function of how many support and consulting staff they can hire. It's a solid model but it doesn't scale because hiring people is costly.

As a result, this approach often falls short of [investors' expectations for significant returns](https://www.saastr.com/vcs-need-you-to-be-a-potential-fund-returner-when-they-invest-but-after-expectations-there-are-more-nuanced), which makes it hard to raise funding and thus prevents most founders from being able to go full-time on their project. Consequently, it becomes difficult to build enough traction to generate leads for the professional services they would like to sell, especially considering there's no marketing budget.

The second case, selling compute and infrastructure, usually comes in the form of a cloud-hosted version of the software. This model works well for companies building complex and mission-critical software, like databases and managed Kubernetes clusters. That's because it's a bet that the company can provide a cheaper, more convenient, and more reliable service than customers could provide themselves.

The problem with this model is that it incentivizes companies to make it hard for customers to run the software on their own. This dysfunctional pattern manifests itself in many ways, like when companies insist on overly complex helm charts (*ugh*), withhold documentation on how to run the software, or require customers to operate a dozen pods when one would work just as well.

The third case, selling a non-open-source version of the software, is fair in the sense that companies will still give value away while carving out a subset of features upon which they can charge. That way, they can still make money to maintain and improve the open-source version, while profiting from the extra features in the non-open-source version.

The problem with this last model is that it creates a conflict of interest between the company and its customers. Customers want the open-source version to have more features, while the company wants the open-source version to have *fewer* features so that customers will pay for the non-open-source version.

 
 

# Our model

![](/posts/open-source-strategy/summary2.png)

Briefer's approach is a mix of the second and third models. We will sell a freemium cloud-hosted version of our software and maintain a generous AGPL-licensed version with fewer features.

Some people call our strategy ["open-core"](https://about.gitlab.com/blog/2016/07/20/gitlab-is-open-core-github-is-closed-source/) and that's technically right. Still, I'd rather say that we have two pieces of software: one that is open-source and another that is not. I think that's more direct because we're not trying to hide the fact that we're selling a non-open-source version of our software. Anyway, open-core is fine too, and there's nothing wrong with it.

We chose to do it this way because we want everyone to use Briefer. Therefore, choosing a service-based model would not be viable because it limits our access to funding, which is necessary to scale Briefer and pay our bills as we build it.

We then considered picking the second model, selling compute and infrastructure, but I was afraid it would corner us into having to eventually change licenses.

Another reason why we rejected that model is because we refused to compromise the software's quality just to profit from users' struggles to run and operate it. Making it ugly, inefficient, and poorly documented was not an option. That would completely defeat the purpose of having an open-source offering, and, as an engineer, it would cause me physical pain.

That's when we decided to mix the second and third models.

This choice allows us to have a small subset of features for which we can charge so we don't have to worry about how easy it is to run Briefer.

That way, there will be an incentive to make the open-source version as easy as possible to run because that helps us increase our user base, thus increasing the number of people who might eventually pay for the cloud-hosted version.

This incentive to increase the user base (our top-of-funnel) also means we can't simply reserve the best features for the non-open-source version. Instead, we have to find a balance between an open-source version that's competitive and attractive enough for people to use it and a cloud-hosted version that's valuable enough for people to pay for it.

In our case, we decided to charge only for the features that larger companies usually need (>100 employees) or those necessary to directly make money from Briefer, like embedding charts into your own application.

![](/posts/open-source-strategy/oss-vs-paid.png)

We thought that was a fair heuristic because companies with more than 100 employees usually have more resources to pay for software, and they usually get more value from it, so it makes sense to take a small cut.

Conversely, small companies are usually strapped for cash, and it wouldn't make sense to charge them pennies. We'd rather help them now and have them eventually pay for the cloud-hosted version when they're thriving and getting more value out of our software. That way, we can help them grow and make enough money to maintain, improve, and distribute both the free and paid versions of Briefer. That's a win-win.

 
 

# Why we chose the AGPL license

![](/posts/open-source-strategy/summary3.png)

We licensed Briefer's open-source version under AGPL for three reasons.

The first is that it's a well-known license, so most people will be fully aware of what they can and can't do with our software.

The second is that it's a great way to ensure that the open-source software will remain open-source forever.

The third is that the AGPL still allows users to implement any features they need, but disincentivizes competitors from copying our software and selling it as their own.

If they were to do that, they would have to open-source all their changes and commoditize their work, so it's not worth it for the vast majority of companies.

Here's where the difference between AGPL and GPL comes into play. We chose AGPL because it closes [the "network loophole"](https://www.revenera.com/blog/software-composition-analysis/understanding-the-saas-loophole-in-gpl/) that exists with the GPL. Under GPL, it was possible to modify the software and use it over a network without releasing the changes. But with AGPL, if competitors modify our software and offer it as a service over a network—like a cloud application—they must make those changes public under the same AGPL license.

We never considered using a license that's not in [the OSI-approved list](https://opensource.org/licenses) because those are, obviously, not open-source. Additionally, we have already made a compromise in not open-sourcing the whole codebase, so I thought it would be fair to pick the "freest" license of them all.

 
 

# Our open-source strategy

![](/posts/open-source-strategy/summary4.png)

The main reason we created an open-source offering is that it's the best way to align our user's interests with the company's financial interests.

On one side, more people can use Briefer because it's free and open-source. These people can implement the features they need, and distribute them to others.

On the other side, it's a competitive advantage for us because it increases our top of funnel, helps us improve our activation rates, and gives us a competitive edge against incumbents.

Here's each of these points in more detail.

 

### Taking on incumbents

Briefer's main competitors are Salesforce's Tableau and Microsoft's Power BI. At this point, these companies aren't just companies anymore. They're institutions. They have a lot of money, a lot of engineers, and they're not going to go away anytime soon.

On the other hand, Briefer is a small company that's doing well, but that's just starting. We don't have the same brand recognition, the same number of customers, or the same amount of money. That's why people ask us questions like: "How do I know you'll be around tomorrow?" and "How are you different?"

Open source is a great way to answer these questions.

That's because open-source software outlives companies. If Briefer disappears tomorrow, people can still use the software, implement the features they need, and distribute them to others. That's a great way to assure everyone that using our software is a safe bet too.

Open-source software is also a great differentiator because it helps us build a strong community. These people can contribute features, fix bugs, and report issues, which makes our software better, faster, and more reliable.

Finally, by going open-source we commoditize our competitors' core functionality. This means they now have to compete against us in terms of innovative features, performance, and price, all of which are usually not their strong suits, let's be honest.

 

### Helping more people while increasing our top-of-funnel

Making our software open-source means more people can use it because they won't have to "hop on a quick call" or swipe their credit card to get started.

Instead, people can just clone the repository and run a single command to get started.

That's great for people because they can use Briefer for free to study, work on their personal projects, or grow their business. It's also great for us because the lower barrier to entry increases our top-of-funnel.

Then, if they like our software, we win, regardless of whether they pay for the cloud-hosted version. That's because non-paying users who like our software will certainly tell their friends about it, and some of them will eventually pay for the cloud-hosted version.

As I mentioned earlier, this incentive to increase the user base also means we can't simply reserve the best features for the non-open-source version. Otherwise, we'd be shooting ourselves in the foot because no one would use it.

Furthermore, if we're successful, it's likely that more companies will create competing open-source offerings. That will push us to open-source more features so that we can stay competitive, which is better for all users, even the ones not using our software.

 

### Improving activation rates

Most people out there don't go around pasting their database credentials into every SaaS product they find. Sometimes they don't trust a company they've never heard of, sometimes they can't punch a hole in their firewall, and sometimes they simply don't want to go through the hassle of getting management's approval.

This behavior makes it more difficult for people to see value in Briefer because they won't have real data from which they can derive valuable insights. As a result, they will churn.

We tried to solve this problem in many ways, including:

1. Creating demo datasets that people can use to get started
2. Supporting file uploads so people can upload their data without having to connect to a database
3. Devising a thorough document detailing all our security practices in an attempt to make people trust us

None of that worked well. All of these initiatives were helpful, but none truly solved the problem.

Looking back, we're now confident that the only effective solution is to make our software open-source. That way, people can run it locally, with their own data, without having to connect to the internet, and behind a VPN if necessary.

This change will increase activation rates, which will then lead to higher retention rates, which leads to more feedback, which leads to a better product, which leads to more users and more conversions.

 

### Crowdsourcing innovation

It's difficult to figure out what people want, and it's even more difficult to figure out how to prioritize everything they want.

Open-source helps us manage Briefer's roadmap along with our users because there will be more of them, and because they'll have access to the source code. That way, they can help us figure out where to go, and help us get there by implementing what they need.

 

### It makes us happier

The first time I worked with my co-founder was when we were maintaining an open-source project called [Chai.js](https://github.com/chaijs/chai). It was the most fun I've ever had working on a project, and I want every day at Briefer to feel like that.

That may sound like a selfish reason, but it's not.

Building a company takes many years, and the only way to keep going is to have fun along the way. That's because building a company is hard, and it's easy to give up when things get tough, but not when you're having fun.

When you're having fun you don't mind the late-night coding sessions, all the "no's" you get from investors, or the bugs that keep popping up. You just keep going because you're having fun, and time will pass anyway, so you might as well spend it wisely and build something meaningful.

 
 

# Things we won't do

![](/posts/open-source-strategy/summary5.png)

Finally, I want to make sure I address some things we will *not* do.

The first is to change our license.

One of the main reasons why Briefer has two separate pieces of software is so that we'll always have a significant number of money-making levers to pull in the non-open-source version.

I think having separate licenses is a better compromise than risking our users' trust by changing the license of the open-source version in exchange for short-term profits.

The second thing we won't do is to stop maintaining the open-source version.

Open-source is a core-part of our strategy. It's the right thing to do for our users, and, consequently, it's the right thing to do for us because it makes people happy, and because it helps Briefer succeed in all the ways I mentioned earlier.

The third and final thing we won't do is to stop being transparent.

I want to build Briefer as a company that people can trust, and that means being transparent about our intentions, our strategy, and our roadmap. That's the right thing to do for our users and, honestly, it's good for business. Therefore, it is the right thing for us too.

If you have questions, email me at lucas.costa\[at]briefer.cloud and I'll be happy to answer them personally.

Best, LdC.

## Footnotes

\[1] [You *can* charge for people for copy of GPL-licensed software if they're getting it from you](https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowMoney). So, *technically*, you could make money selling software itself. The reason that wouldn't work in practice is that [others could then redistribute the software for free](https://www.gnu.org/licenses/gpl-faq.en.html#DoesTheGPLAllowRequireFee). Therefore, it would be nearly impossible to compete with them.

Going open-source as a VC-backed company

This month is packed with new features and improvements. We've been working hard to make Briefer more powerful and easier to use, and we're excited to show you what we've been up to.

 

## Publishing notebooks

Now, whenever you open a notebook, it'll be shown in "view" mode. This means you can't edit the notebook until you click the "Edit" button on the top right corner of the screen. If a notebook has never been published, opening it will automatically put you in "edit" mode.

Whenever you edit a notebook, you must click the "Publish" button to "push" your changes for viewing. This way, you can be sure that your changes are saved and that you won't accidentally modify a notebook while others are viewing it or when a schedule is about to run.

![](/posts/changelog-8/publishing.png)

The same applies to the dashboard view. You can't edit a dashboard until you click the "Edit" button on the top right corner of the screen. If a dashboard has never been published, opening it will automatically put you in "edit" mode.

Please notice that now the schedule button will run the latest published version of the notebook, not the one you're currently editing. This is to prevent accidental changes to the notebook while the schedule is running.

 

### Hiding and showing blocks from the published version

Another cool feature that comes with publishing is the ability to hide and show blocks from the published version of the notebook.

This is great for hiding blocks that are only useful for debugging or development purposes.

![](/posts/changelog-8/hideblocks.gif)

To hide a block, click the eye icon on the left side of the block. To show it again, click the same icon. If you want to hide all tabs at once, you can click the "Hide all" option within the drag-handle menu on the left.

 

## Dynamic dropdowns

You can now use a dataframe's column to populate a dropdown in your notebook. This is great for when your filters depend on a query's result instead of hard-coded values.

![](/posts/changelog-8/dynamic-dropdowns.gif)

To make a dropdown dynamic, you need to select the `Dynamic` option at the bottom of the dropdown configuration. Then, you can select the dataframe and column you want to use to populate the dropdown.

To hide the dropdown's configurations, click the gear icon next to its variable name.

 

## Date-time Inputs

If you want to use date-time inputs in your notebooks, you can now do so by clicking `Inputs > Date` when adding a new block.

![](/posts/changelog-8/dates.gif)

Every date input has a two pieces of configuration:

1. Whether the input is a date or a date-time field.
2. The timezone that the input should be displayed in.

After you've configured the input, you can use it in your code by referencing the variable name you've given it, just like the other inputs.

When referencing the input in your Python code, it will be a `datetime` object, which you can use to filter dataframes, create new columns, or perform any other operation you'd like.

In SQL, you can interpolate the input using the `{{var_name}}` syntax, which will be replaced by the input's value when the query is executed.

 

## Writebacks

Before writebacks, our users had to write Python code to push data back to their databases. This was cumbersome and error-prone, so we decided to make it easier.

Writebacks are a low-code way to write data back to your database. You can use them to update tables, create new ones, or even delete data.

We recommend using writebacks in conjunction with the scheduling feature to create data pipelines that run automatically. These writeback blocks are also great for testing and experimenting with data before committing to a final solution.

![](/posts/changelog-8/writebacks.gif)

To use writebacks, you need to select the `Writeback` block from the block list. Then, you can select which dataframe you want to write back to your database, and configure the writeback settings, like whether you want to append or replace the data, and what to do when there are conflicts.

Please notice that writeback blocks will only appear for users with a Postgres or BigQuery database connected to their workspace. Writing to other databases is not supported at the moment.

 

## Math typesetting (LaTeX)

You can now use LaTeX to write math formulas in your notebooks. This is great for writing scientific papers, creating educational content, or just showing off your math skills.

![](/posts/changelog-8/math.gif)

To use LaTeX, you need to wrap your math formulas in `$` symbols. For example, `$\int_0^\infty e^{-x^2} dx$`.

 

## Performance Improvements

I don't want to dwell too much on this, but we've made several performance improvements to the platform to accommodate the huge number of sign ups we've had in the past few weeks.

Those performance improvements include better client and server-side caching, as well as optimizations to the way our CRDT libraries write data.

We also fixed leaks which would cause long-running tabs to consume more memory than necessary.

 

## General Improvements and Bug Fixes

* We've improved the way data sources pings are handled, which should make the platform more reliable.
* Whenever data source pings fail, we now show a warning message in the UI.
* We fixed a bug in the data source creation which would cause it to create multiple data sources with the same name.
* We've added 13GiB of example data to a demo database that gets attached to every workspace. That should help new users get started with Briefer.
* PDF exports now won't have page breaks. That prevents visualizations from being split across pages.
* Fixed a bug where SQL blocks would not show data sources added after the block was created.
* Fixed plenty of visual bugs in the dashboards.
* Minor aesthetic improvements to dashboard graph displays.

Changelog #8: Publishing, writebacks, date inputs, and dynamic dropdowns

At Briefer, we are building multiplayer notebooks. That means many users can edit and run blocks **in the same notebook at the same time**.

![](/posts/multiplayer-notebooks/multiplayer.gif)

In this post, I'll explain how we got to multiplayer notebooks that actually work.

First, I'll describe Jupyter Notebook's architecture, and how we built a single-player `v0` on top of it.

Then, I'll explain what CRDTs are, how they handle conflicts and collaborative editing, and how we used them to build a multiplayer `v1` that work *reasonably* well.

Next, I'll walk you through the problems we had with `v1`, and show how we solved them in `v2` by communicating changes directly through CRDTs.

Finally, I'll share a few lingering issues we have with `v2` and what we're planning to do next.

 

## v0: Building a notebook

Most people think Jupyter Notebooks are monolithic, but they're actually a collection of three major components that work together: the [Jupyter Notebook *client*](https://github.com/jupyter/notebook), the [Jupyter Server](https://github.com/jupyter-server/jupyter_server), and the [IPython kernel](https://ipython.org/).

When most people talk about Jupyter Notebooks, they're referring to the Jupyter Notebook *client*. That client is a JavaScript application that talks to a Python application called the Jupyter *Server* over a WebSocket.

Whenever you run a block of code in a Jupyter Notebook, your browser writes to the WebSocket from which the Jupyter Server is reading. The Jupyter Server then responds by writing the output back to the WebSocket, which the client reads and renders.

![](/posts/multiplayer-notebooks/jupyter-nb-websocket.png)

Still, the Jupyter Server does *not* run the code itself. Instead, it sends the code to an IPython kernel through a [ZeroMQ](https://zeromq.org/) socket using the [Jupyter messaging protocol](https://jupyter-client.readthedocs.io/en/stable/messaging.html).

It's that IPython kernel process that *actually* runs the code. Then, it sends the output back to the Jupyter Server, which, in turn, sends it back to the Jupyter Notebook client. If necessary, the server can also write to and read from the notebook's `.ipynb` file.

![](/posts/multiplayer-notebooks/jupyter-nb-overview.png)

This decoupled architecture makes it easier to build new types of notebooks without reinventing the wheel. Instead of reimplementing everything from scratch, people can just build clients that talk to the Jupyter Server and render outputs.

For example, if you like pain and text-based interfaces as much as I do, you could use the [Qt console](https://qtconsole.readthedocs.io/en/stable/) and earn some extra street cred (or lose some, depending on who you ask).

![](/posts/multiplayer-notebooks/qtconsole.png)

In the same way, you could build a better notebook client that works in the cloud. You could, for example, wrap SQL code in a Python function and offer native SQL support in your notebooks. That's what we did at Briefer.

Our `v0` was a single-page app that talked to our API which then relayed commands to the Jupyter Notebook server for that user's workspace. For each notebook in a workspace, the Jupyter Notebook server would maintain a separate IPython session.

In addition to that application, we also had a database where the notebooks' contents were stored.

![](/posts/multiplayer-notebooks/briefer-overview.png)

Whenever users opened a notebook, the Briefer client sent a `GET` request to the backend and render the results. As users edited the notebooks, we'd write the changes to a websocket to keep the backend in sync.

Finally, whenever someone wanted to run a block of code, they'd send a `POST` request to the backend, which would run the code and return the output. Once the client received the output, it would write it to the notebook and flush the result back to the server.

![](/posts/multiplayer-notebooks/briefer-flow.png)

This architecture worked great, except for one thing: only one person could edit the notebook at a time. Whenever two people tried to edit the same notebook, the last person to save their changes would overwrite the other person's changes.

Let's say users A (latency = 50ms) and B (latency = 250ms) wanted to edit a `print("Hello world!")` block, for example. In that case, if user A changed it to `print("Hello Mars!")` and user B simultaneously changed it to `print("Hello world! Existence is pain")`, the end-result would be `print("Hello world! Existence is pain")` because the server would receive user B's changes last.

![](/posts/multiplayer-notebooks/users-overwrite.png)

At the time, we solved that problem by locking the notebook whenever someone started editing it. That way, only one person could edit the notebook at a time.

It was a dirty solution, but it helped us ship `v0` in only a few days.

![](/posts/multiplayer-notebooks/users-lock.png)

In a way, this is similar to how a typical mutex works in a multi-threaded application. Whenever a thread wants to access a shared resource, it locks the mutex. If another thread tries to access the same resource, it has to wait until the first thread releases the mutex.

Even though it was a suboptimal solution, it worked well enough for our first few users. These users just needed a cloud environment to write and run code in, and other quality of life features, like native SQL support, automatically turning query results into Pandas DataFrames, and a prettier interface to make their work more presentable to non-technical stakeholders.

That suboptimal solution also bought us time to work on a proper `v1` with multiplayer support as users tested `v0` and gave us feedback on a bunch of other things.

 

## v1: Multiplayer notebooks that worked *reasonably* well

Let's start by exploring a conceptual solution to the multiplayer problem. Then, we'll dive into how we implemented it in `v1`.

 

### A conceptual solution to the multiplayer problem

If we wanted notebooks to be truly multiplayer, we'd have to throw away our mutex-like approach and allow multiple people to edit the same piece of state at the same time.

The problem with simultaneous edits is that conflicts may happen. If two people edit the same piece of state at the same time, someone or something (preferrably) has to decide what the final state should be.

One way of solving this issue would be to break our notebooks' state into blocks. Then, instead of saving the entire notebook every time someone made a change, we saved the block that was edited only.

That way, multiple people could edit *different* blocks at the same time without conflicts. Additionally, we could sync each block's changes in real-time to everyone editing the notebook.

![](/posts/multiplayer-notebooks/users-block-lock.png)

The problem with that approach was that it still wasn't possible to edit the same block at the same time. If I edited the notebook's first block and you edited the second, we wouldn't have any conflicts. However, a third person wouldn't be able to edit either block until we were done.

In a way, breaking the notebook's state into individual blocks simply meant that we moved our "mutex" from the notebook to the block level.

This is where CRDTs come in.

CRDT stands for Conflict-free Replicated Data Type. It's a type of data structure that is capable of *automatically* merging changes made by multiple users to the same piece of state.

Take that `print("Hello world!")` example from before. If user A changes it to `print("Hello Mars!")` and user B changes it to `print("Hello world! Existence is pain")`, the CRDT merges the changes automatically into `print("Hello Mars! Existence is pain")`.

It does so because it knows that replacing `world` with `Mars` happened right after `Hello ` and before `!`, while adding `Existence is pain` happened after `!`. That way, it can apply both changes in order and merge them automatically.

![](/posts/multiplayer-notebooks/users-crdt.png)

This approach is similar to what Git does when people edit different parts of the same file. If you change the first line of code and I change the last, Git will merge the changes automatically because its algorithm is programmed to do so.

![](/posts/multiplayer-notebooks/git-auto-merge.png)

The difference between CRDTs and Git is that CRDTs can merge changes automatically even if they happen in the same line or in the same position within a line.

**This way, if we turned each block's state into a CRDT, multiple people could edit the same block and the CRDT would merge the changes automatically, regardless of where they happened.**

Finally, one last thing to notice about CRDTs as they do not care whether a merge's results are correct. They simply merge the changes automatically and ensure that the final state is consistent across all users.

Consequently, it's possible for two users editing the same block to end up with a block that doesn't make sense. For example, if user A changes `print("Hello world!")` to `print("Hello Mars!")` and user B simultaneously changes it to `print("Hello Venus!")`, the CRDT would merge the changes into `print("Hello MarsVenus!")`.

![](/posts/multiplayer-notebooks/users-crdt-simultaneous.png)

Still, that conflict-resolution mechanism is okay for our notebooks because people see each other's changes in real-time. That way, they can avoid getting in the way of each other.

Furthermore, it's unlikely for two people to edit the same position within block at the same time. In comparison, Git can't tolerate these types of merges because people change files in isolation and only see other people's changes when they `git pull`.

That's all to say that CRDTs are *not* a good fit for *every* type of application, but they're a great fit for real-time collaborative applications like our notebooks.

 

### v1 architecture

Now that we have a conceptual solution to the multiplayer problem, I'll dive into how we implemented it in `v1`.

At Briefer, we're huge nerds, but not *that* much. We didn't want to roll our own CRDT implementations. Instead, we had to find an implementation that worked-well and that was easy to integrate with the editor libraries we using in our text and code blocks, [TipTap](https://www.tiptap.dev/) and [Monaco](https://microsoft.github.io/monaco-editor/).

Ideally, whichever library we chose would also be straightforward to integrate with our backend because we didn't want to write our own custom protocol either.

Considering these criteria, the obvious choice was to go with [Yjs](https://github.com/yjs/yjs).

Yjs is a CRDT library written in JavaScript that implements a variety of data structures, like arrays, maps, and text. Each of these data structures is capable of handling conflicts automatically, as I described above.

Additionally, both TipTap and Monaco have built-in support for Yjs, and the Yjs team also maintains `y-websocket`, a WebSocket provider that makes it easy to sync Yjs data between multiple clients and a server.

We then wired up Yjs to our frontend and backend.

First, we moved over all notebook's state to a `Y.Doc` object containing two-top level keys: a `layout` array and a `blocks` map. The `layout` array references the blocks in the order they appear in the notebook, while the `blocks` map contains the blocks' content indexed by their IDs.

In each block, we also include at least one `Y.XmlFragment` that we pass to TipTap or Monaco. It's that `Y.XmlFragment` that these libraries write to and read from.

```Typescript
// This is a simplified example
type Block = Y.XmlElement<{
 id: string;
 title: string;
 type: "sql" | "python" | "text";
 content: Y.XmlFragment;
 result: Y.XmlFragment;
}>;

type BrieferNotebook = Y.Doc<{
 blocks: Y.Map<Block>;
 // In reality, layout is an array of arrays
 // because a single layout row can have multiple
 // tabs (one for each block)
 layout: Y.Array<string>;
};
```

Finally, we used `y-websocket` to sync the `Y.Doc` object between the clients and the server. That way, whenever someone made a change to the notebook, Yjs would automatically merge the changes and sync them to everyone editing the notebook.

![](/posts/multiplayer-notebooks/client-vs-server.png)

This architecture worked *reasonably* well, except for one thing: the way we handled block executions.

Whenever you ran a block of code in `v1`, the client would send a `POST` request to the backend, which would run the code and return the output. Once the client received the output, it would write it to the `Y.Doc` object and flush the result back to the server.

![](/posts/multiplayer-notebooks/v1-flow.png)

The problem with this approach is that it led to race conditions. If two people ran the same block at the same time, the last person to save their output would overwrite everyone else's output, even if they ran the block first.

Let's say user A and user B had a block that incremented a counter by 1 every time it ran. If user A ran the block and user B ran it right after, the server would run the block twice (correctly). But then if user A's response (`1`) arrived later than user B's (`2`), the server would overwrite user B's response with user A's (an incorrect `1`).

This problem gets even worse when you realize that both clients will display `1` as the block's output, even though the Python runtime state is `2`.

![](/posts/multiplayer-notebooks/incorrect-state.png)

Additionally, people didn't know whether someone else was running a block because the "loading" state was only visible to the person who ran the block.

Sometimes, people would also close their browsers before the block finished executing. In that case, the code execution would alter the notebook's state, but the client wouldn't show it on the UI because it never got the response back and thus didn't update the `Y.Doc` object.

 

## v2: When we finally got it right

In `v2`, we finally cracked multiplayer notebooks.

The way we did that was by moving all the communication between the client and the server to the `Y.Doc` object.

Instead of sending `POST` requests to the backend, the front-end would change the `Y.Doc`'s state and the back-end would react accordingly.

Assume that you want to run a Python block, for example.

In `v1`, your browser would send a `POST` request to the backend, which would run the code and return the output. Once your browser received the output, it would write it to the `Y.Doc` object and flush the result back to the server.

In `v2`, your browser would change the block's state to `run-requested`. Then, the back-end would change the block's state to `running` and kick off the code execution. Once the code execution finished, the back-end itself would change the block's state back to `idle` and **write the output directly to the `Y.Doc` object**, causing all connected clients to see the output in real-time.

![](/posts/multiplayer-notebooks/v2-flow.png)

This approach eliminates race conditions because the updates to the `Y.Doc` are processed in order, as well as IPython executions. Here's what happens when users A and B try to run the same `increment_counter` block at the exact same time:

1. Both clients set the block's state to `run-requested`.
2. The server processes whichever `run-requested` event arrives first (let's say A's)
 2.1 The server tells IPython to run the code
 2.2 The server changes the block's state to `running`
3. The server processes the second `run-requested` event (B's)
 3.1 The server tells IPython to run the code
 3.2 The server changes the block's state to `running`
 4 . IPython finishes running the code for A (IPython only runs one piece of code at a time)
 4.1 The server writes the output to the `Y.Doc` object (`1`)
4. IPython finishes running the code for B
 5.1 The server writes the output to the `Y.Doc` object (`2`)

This way, the front-end would always be in sync with the back-end. If you ran a block and closed your browser before the execution finished, the back-end would still write the output to the `Y.Doc` object and sync it to everyone editing the notebook.

Additionally, if you and I were editing the same notebook and you ran a block, I'd see the block's state change to `running` because that's dictated by the `Y.Doc` object, not by the local browser state.

Although this approach works great in the vast majority of cases, it still has one issue: it's not resilient to back-end failures.

If you run a block and the back-end crashes *during* the execution, the block's state will be stuck in `running` forever. That's because the back-end mistakenly believes the block is still running, when it has actually completed; however, the crash prevents it from receiving confirmation from the Jupyter Server to transition the block's state back to idle.

![](/posts/multiplayer-notebooks/limbo-updates.png)

Although the back-end rarely ever crashes, it's still possible that we have to restart it when deploying new versions or even just scaling up the infrastructure.

For now, we solved this issue by timeouting the block's state and observing Jupyter Server's state, but it's a patchy solution that we're not entirely happy with.

![](/posts/multiplayer-notebooks/recovered-block-state.png)

 

## v3: What comes next

In `v3`, we're planning on implementing a queue system to handle block executions.

That way, if the server crashes during the execution, the block won't get stuck in `running` forever. Instead, once the server comes back up, it will continue executing the block from the queue.

We don't yet know where this queue will live, but we're considering adding it to the `Y.Doc` object itself. That way, the queue will always be in sync with the notebook's state and any worker assigned to that document can pick up the next block to execute.

If you have ideas, suggestions, or feedback, please reach out to us at <founders@briefer.cloud>.

 

## Try these multiplayer notebooks yourself

If you want to try these multiplayer notebooks yourself, you can sign up for Briefer at [briefer.cloud](https://briefer.cloud).

Briefer is free, and we won't ask you for a credit card. We're also not going to send you spammy emails with generic PDF-ebooks, and our sales team definitely won't call you to ask if you want to upgrade to a paid plan (I *am* the sales team, so I can guarantee that).

Worst case scenario I'll ask you for feedback, but that's about it.

Stay hacky,

LdC

At Briefer, we are building multiplayer notebooks. That means many users can edit and run blocks in the same notebook at the same time. In this post, I'll explain how we got to multiplayer notebooks that actually work.

Notebooks and CRDTs

People usually ask me what's wrong with Jupyter Notebooks and why we're building a tool to replace them. They also ask us how we're making notebooks better.

I usually repeat the same things over and over, so I thought I'd write them down here so I can link to it.

 

## 1. Too messy too quickly

The first thing I don't like about notebooks is that they get too messy too quickly. One morning you have 10 blocks, and by the end of the day you have 192, but only six of them are meant to be seen by other people and the rest is just you jiu-jitsuing with the data.

We solved that problem by doing three things.

First, we allowed people to group their blocks into tabs. This way, you can have an "API Call" tab to fetch data, a "Data Cleaning" tab to clean it, and a "Visualization" tab to plot it.

![](/posts/improving-notebooks/tabs.gif)

Then, we created the concept of "publishing" a notebook and added buttons for people to hide certain blocks from the published version.

![](/posts/improving-notebooks/edit-vs-publish-blocks.png)

Finally, we realized that vertical scrolling is not always the best way to navigate a notebook. It wastes a lot of screen real estate and makes it hard to see the "big picture" of what's going on.

We fixed that by adding a "dashboard" view with a grid in which you can drag and drop blocks.

![](/posts/improving-notebooks/notebooks-to-dashboards.gif)

Today, most published notebooks use the dashboard view instead of the traditional vertical scrolling.

 

## 2. Difficult to find and explore

The second thing I don't like about notebooks is that they go to GitHub and then die because no one ever opens them again.

I wanted notebooks to be "live" artifacts that everyone can find, open, and explore, including non-technical people.

To solve that problem we created a Notion-like file tree.

![](/posts/improving-notebooks/the-file-tree.gif)

The file tree makes notebooks more approachable because it allows you to create top-level notebooks like "Sales Team" and put all your sales-related notebooks inside it, like sales forecasts and customer segmentation.

I know the icon selection is a bit cheesy, but it makes things more approachable for non-technical people. Once we launched this feature, people complained that it included 288 icons "only" and that they wanted more, so we added an extra 1487. Today, most notebooks have a custom icon.

 

## 3. Too much friction for querying databases

The third thing I don't like about notebooks is that they require a lot of boilerplate code to query databases.

We fixed that by adding a "query" block that allows you to write SQL and see the results in a table.

![](/posts/improving-notebooks/sql.png)

We also know that queries are useless unless you can wrangle the data, so each query's results are automatically converted into a pandas dataframe.

![](/posts/improving-notebooks/sql-to-dataframe.png)

Then we went full circle and allowed people to write SQL queries against their pandas dataframes. We did that because sometimes the declarative style of SQL makes it a better choice than Python.

![](/posts/improving-notebooks/select-from-dataframe.png)

 

## 4. Not collaborative

The fourth thing I don't like about notebooks is that I can't edit them at the same time as someone else. I also don't like it that I have to push my changes to GitHub and then someone else has to pull these changes to see what I'm doing.

We fixed that by using <a href="https://yjs.dev/" target="_blank">Yjs</a> to store the notebook's state.

Whenever you create a block, delete a block, or move a block, we send that information to Yjs, and then Yjs sends it to everyone else who's viewing the notebook.

If two people try to edit the same block at the same time, Yjs knows how to merge their changes, so no one loses their work.

 

## 5. Outdated results

The fifth thing I don't like about notebooks is that they get stale, and you have to re-run them manually to get the latest results.

Let's say you have a <a href="https://www.meetrecord.com/blog/12-sales-forecasting-methods-you-need-to-know" target="_blank">sales forecast</a> notebook built on top of the latest CRM data. In that case, Jamie from the sales team will probably ask you to re-run the notebook every week or so to get the latest forecast.

We fixed that by adding a "schedule" button that allows you to run a notebook at a specific time, like every Monday at 9 am.

![](/posts/improving-notebooks/scheduled-runs.png)

If the notebook fails to run, we can send you an email or a Slack message. Otherwise, we can send you the complete notebook in PDF format on those same channels.

I love the notifications feature, but some people overuse it. They schedule their notebooks to run every few minutes and then get mad when they get too many notifications. We don't know how to solve that problem yet.

 

## 6. Not interactive

The sixth thing I don't like about notebooks is that they're not interactive.

We fixed that by adding a few native text fields and dropdown blocks that allow you to ask for user input.

![](/posts/improving-notebooks/normal-inputs.png)

We know that users sometimes want their inputs to be dynamic, like when they want to use values from their databases as options in a dropdown, so we did that too.

![](/posts/improving-notebooks/dynamic-inputs.png)

We do know about `ipywidgets`, but we don't like them because they get lost among the other blocks, and their styling is not consistent with the rest of the notebook.

 

## 7. Friction to plot graphs

The seventh (and final) thing I don't like about notebooks is that plotting simple graphs requires more code than it should.

Whenever I need a simple line chart, I have to write a few lines of Python code, and then I have to do all the styling manually. Otherwise, the chart looks like it was made in Excel.

Additionally, I keep having to look up the documentation to remember how to plot anything other than basic charts.

We solved that by adding "visualization blocks" that allow people to plot data by clicking buttons instead.

![](/posts/improving-notebooks/vis.png)

These blocks look good by default (at least that's what we think) and you can use them to plot data from any dataframe.

Some people complained about having to filter data using Python before plotting it, so we added a "filter" button to the visualization blocks too.

![](/posts/improving-notebooks/vis-filters.png)

Still, sometimes people need to use Python to plot more complex graphs, like when they want to plot a sankey diagram or a network graph. In that case, they can click "edit with AI" and ask the LLM to plot the graph for them. That way, they don't have to read the documentation every time they need to plot something new.

![](/posts/improving-notebooks/ai-sankey.png)

People usually ask me what's wrong with Jupyter Notebooks and why we're building a tool to replace them. I usually repeat the same things over and over, so I thought I'd write them down here.

Making notebooks better

I used to love two things about computers. The first was that they did exactly what you tell them to do. The second was that they did the exact same thing every time.

Humans shattered that magic. We now have AI assistants who do what you tell them to do, but they also do a lot of other things that you didn't tell them to do, and they do it differently every time.

When I started implementing AI assistants in our notebooks, I hated that. At times, the assistant would perfectly execute complex tasks like developing a churn prediction model but then stumble over something as simple as plotting a revenue line chart from a CSV of financial data.

At first, I thought I'd have to do something fancier than just RAG and "prompt engineering" to get the AI assistant to work. Then, I realized that I could get better results with a few UI tweaks, and this is what this blog post is about.

![](/posts/code-gen-for-notebooks/ai-assistant-15fps.gif)

 

## Setting good examples

The most useful mental model for implementing AI assistants is to think of yourself as a manager onboarding a new senior employee.

You wouldn't expect the new senior employee to know everything about your business on their first day, but you would expect them to be smart and capable of learning quickly. In other words, you assume zero knowledge and infinite intelligence.

I like that approach, and that's what most people do. Let's say they're building an assistant to help fifth-graders with their homework. In that case, most people would just do some kind of RAG over fifth-grade textbooks, and that would be it.

In our case, we did something similar. Initially, we fed the LLM with the content of the current SQL or Python block, the user's database schema, and a list of data frames, their columns, and their types.

That approach worked well, but one common complaint was that the LLM's output didn't follow the user's coding standards. For example, the user's previous code used `plotly` but the LLM used `matplotlib`. Or the user's code used a specific color palette, but the LLM would just default to the standard one.

To fix that, we started including a few code snippets from the notebook so that the AI assistant could see whether we were using `matplotlib` or `plotly`, and what colors users prefer.

Oddly enough, one of the most impactful changes we made was to include the document's title and the content of relevant text blocks if we had any. That way, the AI assistant could see what the user was trying to do, and suggest something that made sense in that context.

![](/posts/code-gen-for-notebooks/ai-context-examples.png)

In a way, this approach mimics what a senior engineer does when they're submitting their first few pull requests. They look at the existing codebase, see what the team is doing, and try to follow the same patterns.

 

## Encouraging small steps

Pull request sizes are usually inversely proportional to the wrinkles on a programmer's forehead. The more seasoned the programmer, the smaller the pull request.

That's because seasoned programmers know that the more code they write before getting feedback, the more code they'll have to erase if something is wrong.

Additionally, working on small steps at a time makes tasks easier to wrap your head around, and easier to debug.

The same principles apply to AI assistants. You should encourage users to give the AI assistant small tasks at a time and ask the user for feedback on each task before the assistant moves on to the next one.

In our case, we used to have an input for users to ask the AI to perform an *entire* analysis on their behalf. After a month or so, we removed it because we found that tasks got too large for the AI to handle at once, and too intricate for users to describe accurately.

![](/posts/code-gen-for-notebooks/ai-user-flow.png)

Now, we include AI assistants in individual blocks. That way, users end up asking the AI to do a single step at a time.

Consequently, user's prompts usually get more accurate, because they don't have as many intricacies to describe and thus tend to devote more time to crafting a better prompt.

A further benefit of encouraging small steps and getting feedback on each is that the user gets to vet each suggestion before moving on to the next one. That way, the assistant's corpus gets more accurate over time, and the user gets more comfortable with the assistant's suggestions.

 

## Optimizing for feedback

When asking users to take small steps, you must make each step as fast and pleasant as possible. Otherwise, users will get frustrated and give up.

In our case, for example, we used to have the AI replace the user's code with its suggestions, and we'd do it all at once. That was a terrible idea because users couldn't tell whether the AI was working until it was done, and they couldn't see what the AI was doing until it was too late.

To fix that, we did three things.

First, we started streaming the AI's response instead of emitting it all at once. That way, users could see the AI's suggestions as they were being generated, and they could stop the AI at any time if they didn't like what they were seeing.

After that, we started showing the user a diff of what the AI assistant suggests. Then, they could accept or reject the changes. That way, users could clearly see what the AI assistant was suggesting versus what they wrote.

Finally, we added a button that allowed users to try the AI assistant's suggestion without accepting it. That way, users could try the suggestions before accepting them, and they could also modify the diff in case something was slightly off.

![](/posts/code-gen-for-notebooks/accept-reject-try-buttons.png)

Something else we're working on is to allow users to prompt the AI assistant's suggestions without having to accept them.

In addition to making each step more seamless, these diffs also prevent users from running undesirable code as it puts on them the burden of reviewing each change.

 

## Encouraging action

Clicking a button is quicker than writing a prompt, and it's also more intuitive. Consequently, you should consider adding a "quick trigger" button whenever there's an action that the assistant can take without the user having to prompt it.

In our case, errors are a prime example of that.

Whenever someone writes invalid SQL or makes a typo in their Python code, the error that comes up usually has all the necessary information for the AI assistant to fix it. Therefore, there's no need to ask the user to prompt the AI assistant to fix it.

Instead, we show a button that says "Fix with AI". When the user clicks it, we feed the LLM with the error message and a bit of context, and we ask it to try to fix it.

![](/posts/code-gen-for-notebooks/fix-with-ai.gif)

These types of scenarios are more common than you'd think, and they're a great opportunity to show users the value of the AI assistant. People will trigger the assistant more often if they don't have to think about what to do with it.

 

## Models usually take care of themselves

Finally, one last piece of advice is to assume that new LLMs will only get better, faster, and cheaper over time.

Take GPT-4o, for example. When it was released, it solved quite a few problems we had with the previous versions. All of a sudden, our default assistant got way faster and more accurate.

The trick is that we didn't have to do anything to make that happen. Had we spent months fine-tuning models and squeezing every last drop of performance out of them, we would've wasted a lot of time.

Instead, we focused on making the user's experience as good as possible, and we let the LLM's performance take care of itself.

Then, when a new model came along, we could just plug it in and get better results without having to change anything else.

In comparison, everyone else who spent months working on their models had an assistant that was just as good as ours, but without all the UI improvements we had made.

 

## Putting it all together

Perceived performance is the only performance that actually matters. If users *perceive* your assistant to the better, it doesn't matter whether its underlying model is the one with the highest scores on standard benchmarks.

In a way, making AI assistants better is as much about improving the models as it is about improving the user's experience when using them.

Sure, there's a lot of value in creating better models, but there's a lot more innovation you can do if you focus on the specific problem you're trying to solve.

Consequently, if you're adding AI features to your product, you should probably start with the low-hanging fruit: UI/UX improvements.

Here's what we found helpful:

1. **Set good examples**: instead of including only raw data, augment generation with user-generated examples and specific context.
2. **Encourage small steps**: ask users to prompt the AI assistant on small tasks at a time, and ask for feedback on each task before moving on to the next one. Then, use each step's result to improve future suggestions.
3. **Optimize for feedback**: make each step as fast and pleasant as possible, and make it easy for the user to adjust the assistant's suggestions.
4. **Encourage action**: add a "quick trigger" button whenever there's an action that the assistant can take without the user having to prompt it.

Copilot UIs

In the past few weeks, we've got plenty of requests for "quality of life" features, so we delivered them.

This week, we're glad to show you how to use new dashboard features, connect MySQL databases, and query uploaded files using plain SQL.

Many people asked for context-aware autocomplete in SQL and Python blocks, so we shipped that too.

 

## Querying files

You can now write SQL queries against files and dataframes. This means you can upload a CSV file and query it using SQL, for example.

To do that, add a "File Upload" block, drag your files into the block, and then click the "query" button to generate an example `SELECT * FROM` query.

Feel free to change that query to do whatever you want with the data in the file.

![](/posts/changelog-7/query-files.gif)

In addition to querying files, you can also query any pandas dataframes, as you can see in the example below.

![](/posts/changelog-7/query-dataframes.png)

As a general note, you should select the `Files (DuckDB)` data source whenever you want to query anything other than a database.

 

## Better dashboards

We did tons of changes to the aesthetics of dashboards, but we're not done yet.

In the past few weeks, we cleaned up the visualization display, added support for data labels, and also improved the way all blocks look in the dashboard.

Additionally, blocks now show their titles on the top left corner, and the sidebar has been completely revamped so all blocks look nice within it, and so you can see their titles. That way, it's easy to identify what each block is doing at a glance.

![](/posts/changelog-7/dashboards-tour.gif)

 

### Dashboard headings

Headings can help you organize your dashboard's content. You can add a headding to any block by clicking the "Add Heading" button on the top right corner of the block list.

These headings will only appear in the dashboard view, not in the notebook.

![](/posts/changelog-7/dashboard-headings.gif)

 

### New visualization types

In addition to the aesthetics, we've also added a few new visualization types, such as "trend" and "numbers".

The "trend" visualization is great for showing trends over time, while the "numbers" visualization is perfect for highlighting a single metric in a large font.

![](/posts/changelog-7/numbers-and-trends.png)

 

## Autocomplete

We've added autocomplete to SQL and Python blocks. This means you can now get suggestions for column names, table names, variable names, and functions while writing your queries.

![](/posts/changelog-7/autocomplete.png)

Please note that for the SQL autocomplete to work, the permissions of the user you're using to connect to the database must allow for querying the information schema.

 

## MySQL support

We've added support for MySQL databases. You can now connect to a MySQL database and query it using SQL blocks.

 

## General Improvements and Bug Fixes

* It's now possible to undo AI suggestions and try them before accepting the diff.
* There's a new onboarding document that will help you get started with Briefer when creating new workspaces.
* Visualizations now support multiple Y-axes.
* Document's titles will now appear in the browser's tab.
* We fixed a bug in which the line chart would offset ticks in the X-axis.
* We fixed a bug in which the "in/not in" filters would not work for dataframes.
* We fixed a bug in which the blocks would be repositioned when opening the sidebar.
* We fixed a bug in which the progress indication would show incorrect values when running entire notebooks.
* We fixed a bug where invalid visualizations would fail silently and not show any error messages.

In the past few weeks, we've got plenty of requests for 'quality of life' features, so we delivered them. This week, we're glad to show you how to use new dashboard features, connect MySQL databases, and query uploaded files using plain SQL.

Changelog #7: Querying files and dataframes, better dashboards, autocomplete, and MySQL support

Sales pitches are the only place where "self-serve dashboards" work. In the real world, it's a different story.

That story usually starts with an engineer or data scientist who's frustrated because they spend too much time writing queries and preparing dashboards for business people. They think that if they make BI easy enough, everyone will be able to "self-serve", but that rarely ever happens.

What actually happens is that engineers and data scientists end up being the only people using the "self-serve" BI tool, which, ironically, makes it not self-serve at all.

 

## Why self-serve BI doesn't work

**SQL is the only self-serve BI tool**. Still, most "self-serve" BI vendors don't want to admit it, so they find ways to reinvent the wheel and disguise SQL as something else. Maybe that's what customers want to hear, or maybe they genuinely believe it, I'm not sure.

Either way, the problem with most approaches to "self-serve" BI is that they consider SQL to be the only barrier to business stakeholders querying data. That's not true.

Let's say you could wave a magic wand and make writing SQL queries as easy as writing an email. Do you think business stakeholders would suddenly start querying data? The answer is no.

Even if these people knew how to write SQL queries, they wouldn't understand the semantics of the data they're querying. They wouldn't know what the data means, where it comes from, or how it's calculated. They wouldn't know how to interpret the results, or how to validate them. In other words, writing the SQL query is actually the easy part.

Now, let's look at two attempts to make BI "self-serve" and see whether the hypothesis above holds.

 

### Attempt 1: The conventional "dropdowns and checkboxes" approach

First, let's take a look at an example of a conventional "self-serve" BI interface which uses dropdowns and checkboxes as a way to make queries more accessible.

![](/posts/self-serve-bi/sql-by-mouse.png)

The first thing to notice is that this interface is just an attempt at what I call "SQL-by-mouse". I might be just an old grumpy person, but I don't see how this is any better than writing SQL. In fact, it's worse because it's slower, less reliable, more limited, and not generalizable to other tools.

In any case, I'll concede one thing: some non-technical people are just scared of any monospaced writing with syntax highlight, so this interface might be a good first step.

Anyway, let's ignore all these problems and just assume that this interface is the best thing since [htmx](https://htmx.org/). Let's assume that it's 100x easier to use than SQL.

Even if that were the case, your CFO, for example, wouldn't just go ahead and use these dropdowns and checkboxes to query data. The reason they wouldn't use it is that they don't have the context to understand the data they're querying, and they'll probably be insecure about the results, which they won't know how to validate because they don't understand how the data got there in the first place.

Now, I've got a question for you: if your CFO doesn't use this interface, who will? That's right, *you* will. And guess what's the first thing you'll do when you get to these dropdowns? Correct, you'll immediately look for somewhere to type in SQL. Oh, the irony.

 

### Attempt 2: The text-to-SQL approach

Remember that magic wand that makes writing SQL queries as easy as writing an email? That's what text-to-SQL tools are. Nonetheless, they're not enough, just as I explained earlier.

Again, the problem is not the technology itself. In fact, LLMs are almost too effective at translating natural language into SQL. They will find a way to generate a query for any question you ask, even if that question doesn't make sense, which is exactly the problem.

On the other hand, a technical person would notice that the question doesn't make sense, and they would ask for more context. They would ask for details about the business person's hypothesis and the problem at hand. Then, they would explain what type of data is available, and work with the business person to formulate a precise and useful question.

Again, SQL is not the problem.

In any case, I do think LLMs could be the actual solution to self-serve BI, but not in its current form. For them to work, they'd need to be fed with more context, and [they need to get better at expressing uncertainty](https://allenpike.com/2024/llms-trained-on-internet) and asking for more information.

 

## What actually works

If we assume that the problem with self-serve BI is not SQL, but the context and semantics of the data, then it follows that the solution is to teach people about the data they're querying, regardless of interface.

The problem with this type of training is that it takes time, and most business stakeholders need answers this week, not next quarter.

Even in the best-case scenario, where companies train business stakeholders to understand the data they're querying, these people will need time to keep up with the changes in the database's schema, data models, and ETL processes.

Additionally, documenting all this knowledge generates significant overhead for the technical team, and it quickly gets out-of-date.

So what's the true solution to self-serve BI? The answer is simple: not to make BI self-serve for non-technical people. Instead, the solution is to make technical people support business stakeholders and help them do it more efficiently, using better tools.

The definition of a better tool may vary, but I have a few suggestions.

The first suggestion is to give LLMs to technical people, not business stakeholders. Even though text-to-SQL is not quite there in terms of understanding context and semantics, technical people *already* have context and semantics, so why not let *them* use it? That way, a single technical person can serve more business stakeholders, and do it faster.

The second suggestion is to give technical people more flexible tools. Instead of giving them dropdowns, or a SQL-only interface, let them play around with data using Python, R, or any other tool they're comfortable with. This way, they don't need to be passing data back and forth between different tools, and they're not limited by the capabilities of a BI tool.

The third and final suggestion is to make it easier for technical people to share their work. Notebooks and internal data applications are notoriously bad at that because they require others to deal with containers, dependencies, and infrastructure. It's a lot of overhead for a business stakeholder who just wants to see a chart. Instead, we need tools that have all the building blocks of a data application, but that are as easy to share as a Google Doc.

I'm obviously biased because I'm building [a data tool for technical people](https://briefer.cloud), and it does exactly these three things. But what other choice do I have? I'd rather build tools that work than tools that don't. And, most importantly, I'd rather not pretend that the problem is something it's not.

Sales pitches are the only place where self-serve BI works. In the real world, it's a different story.

Self-serve dashboards

Since the very first day we launched Briefer, we had a strong hunch that dashboards would be something people would want. We were right. We've been getting a lot of requests for this feature, and we're happy to announce that it's finally here.

Along with dashboards, we've also made some improvements to our visualizations and AI assistant. Here's what's new.

 

## Dashboards

You can now turn your notebooks into dashboards. Dashboards are a great way to keep track of your most important metrics and KPIs in one place. **You can add multiple visualizations to a dashboard, rearrange them, and resize them to fit your needs.**

At Briefer, we care a lot about empowering users to do whatever they want with their data, that's why our dashboards support Python outputs. **You can use Plotly, Seaborn, or any other Python library to create your visualizations and add them to your dashboards.**

 

![](/posts/changelog-6/dashboards-light.gif)

To create a dashboard, click the "Dashboards" button on the upper-right corner of the editor. Then, drag and drop the blocks you want to add to the dashboard.

You can also resize each block by dragging the bottom-right corner of the visualization.

Once you're done, click the "Publish" button to save your dashboard. You can access it at any time by clicking the "Dashboards" button on the upper-right corner of the editor.

Please notice that you will need to republish your dashboard every time you make changes to the blocks if you want to see them reflected in the dashboard.

 

## Better visualizations

To go along with the new dashboards feature, we've also made some improvements to our visualizations.

You can now rename each axis in your visualizations, format the value displays, and choose whether to display the labels or not.

We've also added support for histograms.

![](/posts/changelog-6/new-vis.gif)

 

## Better AI

Our SQL and Python AI assistant has also received some updates. It's now better at understanding your questions and providing relevant answers.

We've also had a few users ask for a way to change the foundational model of the AI assistant. Now, you can do that by clicking the "Settings" button on the lower-left corner of the editor, and then selecting the desired foundational model to use.

 

## General Improvements and Bug Fixes

* We now support Athena and Oracle databases as data sources.
* There's now a complete onboarding guide for new workspaces. It includes instructional videos and tips to help you get started.
* We've improved the Slack integration so that it's easier to set up, especially when it comes to channel selection.
* The schedules list now shows the notification settings for each scheduled run.
* We've updated the notification settings so that it can send PDF attachments containing your documents.
* Copying and pasting an image into a text block will now work as expected.
* There's a clearer indication of each user's tier right next to the logo on the top-left corner of the app.
* There's now a "Create visualization" button on the right of each SQL block. Clicking it will create a new visualization block with the results of the SQL block.

Besides those changes, we've done a variety of bug fixes and small performance improvements.

We recently launched dashboards, a feature many of you have been asking for. With that, we also improved our visualizations and AI assistant. Here's what's new.

Changelog #6: Dashboards, better visualizations, and better AI

The first lesson I learned about logarithms is not to mention them when speaking to a large crowd. All the other lessons sucked, so I thought I'd create my own.

Let's start by talking about the simplest of linear scales: a ruler.

Everyone understands how rulers work: you place them next to tiny trinkets and you look at the ruler's marks to determine the object's size. You can then use a pen and the ruler's marks to represent the size of these objects on a board.

![](/posts/logs/log1.png)

The problem with rulers is that you can't use them to measure larger objects, such as trees, park benches, and city blocks. To solve that problem, we'll need a bit of imagination.

Instead of having each of our ruler's marks represent 1 inch, we'll pretend they represent 100 inches. That way, we'll compress larger distances into smaller ones.

![](/posts/logs/log2.png)

Now, we can use the same ruler to represent the size of a park bench, a tree, and even a city block.

![](/posts/logs/log3.png)

Notice how we're not using the ruler for measuring anymore. Instead, we're using it to *represent* the sizes of different objects.

**The way we squished hundreds of inches into one mark on the ruler is called a linear transformation because when you draw it on a graph, it makes a straight line**. Each mark on the ruler represents 100 more inches, showing a steady, even increase.

![](/posts/logs/log4.png)

**Linear transformations preserve the measurements' relative spaces.**

For instance, if the marks for 1 and 5 inches are 4 segments apart on a ruler, then the marks for 100 and 500 inches will also be 4 segments apart when we use our scaled-up ruler. That way, the proportional distances remain the same, even though the actual measurements have increased.

![](/posts/logs/log5.png)

![](/posts/logs/log6.png)

**The problem with applying a linear transformation to a ruler is that it can only either measure small trinkets or large objects, not both.**

A ruler scaled from 0 to 10 inches isn't big enough to measure a park bench, while a ruler scaled from 0 to 900 inches makes it hard to distinguish the marks for smaller objects — they become too close together.

![](/posts/logs/log7.png)

To solve this problem, we'll need to transform our ruler so that each segment represents a greater increase.

Typically, going from the first mark to the second represents a 1-inch increase, just like there's a 1-inch increase when going from the second mark to the third. In other words, segments in normal rulers represent linear increases.

In our new scale, going from the first mark to the second represents a 10-inch increase, and going from the second to the third mark represents a 100-inch increase. Each subsequent segment represents a distance ten times greater than the previous. In this case, each segment represents a *non-linear* increase.

![](/posts/logs/log8.png)

This way, with just 10 marks, you can represent objects from 1 inch to 100,000,000 inches. Each step exponentially increases the range we can measure without cluttering our ruler with countless tiny increments.

Now, you can represent tiny stones at the first mark and a mountain just 10 segments further because, by the time you get to the 10th mark, the value this mark represents hasn't just grown tenfold, it's grown 10¹⁰ inches.

Here's another example using our new scale. If you plot a cockroach in the scale's first mark, a badger in the second mark, and a troll in the third, it means that the badger is 10x larger than the cockroach and that the troll is 10x larger than the badger.

Furthermore, it shows that the troll is 100x larger than the cockroach, because they're 2 segments apart, so 100 (10²) times larger.

![](/posts/logs/log9.png)

Going left has the opposite effect, it causes things to become increasingly smaller. If you keep going left from 1, you'll start working with fractions, and they get 10x as small as you move on to the leftward mark.

![](/posts/logs/log10.png)

As we've just demonstrated, this type of scale is great for representing ratios.

Now compare how the represented values increase exponentially as you move up marks. See how it's not a linear growth like in the previous transformation?

![](/posts/logs/log11.png)

As shown above, each mark represents an exponential increase in the value it represents. Still, this type of scale is *not* an exponential scale. It's a *logarithmic* scale.

If you were to plot the actual values against their corresponding logarithmic values, you'd notice that it resembles the plot of a logarithmic function. This resemblance occurs because, as the actual values increase, the increase in their logarithmic values becomes increasingly smaller.

![](/posts/logs/log12.png)

**An interesting effect of using the logarithmic scale is that small values get relatively more distant, while larger values get relatively closer.**

![](/posts/logs/log13.png)

The reason you should care about this change in relative distances is because it may cause unintended distortions, or, in the case of ill-intentioned people, *intended* distortions.

For example. Assume that a general council of alchemists doesn't want people to think there's a dragon fever pandemic. In that case, they can plot the number of infected people using a log scale, making it seem linear, when, in fact, it's exponential.

![](/posts/logs/log14.png)

However, there might be compelling reasons behind the decision to use a logarithmic transformation for this chart.

Imagine there are two kingdoms, Hyrule and the Mushroom Kingdom, each with vastly different population sizes. Hyrule has 10,000 inhabitants, while the Mushroom Kingdom boasts 10,000,000.

Now, let's say a disease outbreak occurs in both kingdoms, starting with 10 infected individuals in each. On a regular linear scale, the number of infected individuals in Hyrule might seem insignificant compared to the Mushroom Kingdom's numbers.

![](/posts/logs/log15.png)

If we switch to a logarithmic scale, the picture changes. The slope of the curve representing the increase in infected individuals in both kingdoms becomes comparable. So, when the disease spreads from 10 to 100 infected individuals in Hyrule, the curve's slope remains consistent. Similarly, when the Mushroom Kingdom sees its infected population rise from 10,000 to 100,000, the slope of its curve mirrors that of Hyrule's.

![](/posts/logs/log16.png)

This scaling ensures that the threat level posed by the disease is communicated effectively in both kingdoms, regardless of their size. Without it, Hyrule's situation might appear deceptively less severe simply because of its smaller population. By utilizing a logarithmic scale, public health officials can accurately assess and address the risk posed by the disease in each kingdom, enabling more informed decision-making and resource allocation.

In addition to impacting visualizations, logarithmic transformations are also useful for statistical purposes, mostly because of:

1. **Symmetry**. Logarithmic transformations may make data more balanced, which is essential for certain statistical techniques to work effectively.
2. **Homoscedasticity**. Logarithmic transformations may help keep the spread of data consistent, which helps compare effects on different groups.
3. **Linearity**. Logarithmic transformations may make it easier to see straight-line patterns in data, simplifying analysis.

 

## Symmetry

A graph is symmetric concerning a line if reflecting the graph over that line leaves the graph unchanged.

This histogram of dragon sizes is highly symmetric, for example.

![](/posts/logs/log17.png)

On the other hand, this graph of magic power levels in a sample of healthy wizards is not very symmetric. Instead, it's said to be positively skewed because of its long tail of results to the right.

![](/posts/logs/log18.png)

Now, pay attention to this magic trick: we can make this graph symmetric by applying a logarithmic transformation to all the magic power levels.

Such transformation makes these values more symmetric because the long tail of large values gets compressed, while the small values concentrated on the left get pushed farther apart.

![](/posts/logs/log19.png)

Symmetry is vital in data analysis for various reasons. For example, when using statistical methods, we typically aim to find a mathematical model that accurately represents the distribution of our data. The choice of an appropriate model often depends on whether the data is symmetric.

Anyway, discussing the details of model selection is a topic for another time.

 

## Homoscedasticity

In the land of Skyrim, knights rely on magical potions to enhance their strength during battles.

When we graph the effectiveness of these potions based on the initial strength of the knights, we notice greater variability in the potion's impact. This variability is a common occurrence with potions and medicines.

For instance, if a knight initially lifts 100 pounds, the potion might yield effects ranging from no change (100 pounds) to a significant increase (160 pounds, a gain of +60 pounds). Conversely, if a knight can lift 500 pounds, the potion's effects could range from maintaining the initial strength (500 pounds) to a substantial increase (800 pounds, a gain of +300 pounds).

When we plot this data on a linear scale, we notice that the variance increases as the knight's initial strength rises. This phenomenon, known as heteroscedasticity, reflects the heterogeneous nature of the data, where the variability of the potion's effects differs across the range of initial strengths.

![](/posts/logs/log20.png)

When we apply a log transformation to our graphs, it makes the data more homoscedastic, meaning it evens out the variance, especially for stronger knights. In simpler terms, this means that the potion's effects become more consistent across different strength levels.

This transformation reveals that the potion consistently boosts strength by around 0 to 20%, making it easier to predict its benefits for knights at all training levels.

![](/posts/logs/log21.png)

 

## Linearity

Let's resort to our imagination again.

Let's say someone's aura indicates one's affinity with celestial magic, and Moonbeam Extract is said to degrade this connection.

Imagine a scenario where our kingdom's alchemists seek to understand the relationship between Moonbeam Extract and someone's aura.

When we plot people's aura's brilliance against Moonbeam Extract, we notice a curved pattern, with most data clustered at the lower left corner. That plot gives us some clues about the relationship between people's auras and Moonbeam Extract, but it's not very clear.

When we employ logarithmic transformations on both magical properties, the correlation appears straight and clear.

![](/posts/logs/log22.png)

This linearity simplifies our understanding, allowing us to see how the effects of Moonbeam Extract clearly impact people's magic abilities.

 

## A final note on log bases

Throughout this post, we've mostly used common logs, where each segment represents a tenfold increase compared to the last (log10).

Still, it's worth noting that we can also use other log bases, such as log2 or natural logarithms (logE). These alternative log bases offer different perspectives and may be better suited for specific situations or data types.

 

## About these visualizations

We've built all these visualizations using Briefer.

If you want to build reports, presentations, or data apps with visualizations like these, you can sign up for early access to Briefer at [briefer.cloud](https://briefer.cloud).

A brief introduction to logarithmic scales and their applications in data visualization and statistics.

Log scales

This week we've released two major new features that we're very excited about: snapshots and a Slack integration.

## Snapshots

Snapshots allow you to save the state of your document at any time, and then restore it later.

![](/posts/changelog-5/snapshots.gif)

This is useful when you want to experiment with your code, but you're afraid of losing your progress. With snapshots, you can save your work before making changes, and then restore it if you don't like the changes.

Snapshots are also useful to keep track of how your metrics or visualizations change over time. In Briefer, every scheduled run creates a new snapshot, so you can compare results over time.

To use them, click the "Snapshots" button on the upper-right corner of the editor. Then, click the "Save" button to save the current state of your document. You can also restore a snapshot by clicking the desired snapshot, and then clicking the link to restore it at the top of the page.

**Snapshots are only available for users in our professional plan, and their retention period is 90 days**.

 

## Slack Integration

Our Slack integration allows you to receive PDF reports directly in Slack.

![](/posts/changelog-5/exports.gif)

To use it, you need connect your Slack workspace to Briefer. To do that, access the "integrations" menu at the bottom of our sidebar. Then, click the "Add" button and select "Slack".

After connecting your Slack workspace, add a scheduled run to your document, and, as you're doing it, click the "Add" button right next to the notification settings.

There, you will be able to select the Slack channel where you want to receive the PDF report.

Then, click the "Add Slack" button, and follow the instructions to connect your Slack account to Briefer.

The Slack integration is only available for users in our professional plan.

**Important**: You need to invite the Briefer bot to the desired channel before it can send notifications (`/invite @Briefer`). You won't receive notifications if your page fails to execute.

 

## General Improvements and Bug Fixes

* We will automatically execute previous blocks after your environment restarts. That way, you will not face issues with variables not existing anymore.
* You can now expand your document horizontally.
* Visualizations take 4x less time to load.
* We've improved the layout of text blocks. Text will look better now.
* We've fixed a few performance issues while switching tabs with multiple editors.

We've just released snapshots. They allow you to save the state of your document at any time, and then restore it later. We've also added a Slack integration, so you can receive PDF reports directly in Slack.

Changelog #5: Snapshots and a Slack integration

When we got into YC, we were building a product for companies to spin up development environments in the cloud. The idea was that, as software grows, it inevitably becomes too difficult to run on a single machine. At that point, people would come to us and we'd help each developer an environment in the cloud.

**It didn't work**.

During our batch, we amassed [a thousand stars on GitHub](https://github.com/briefercloud/layerform), [got to HackerNews' front page](https://news.ycombinator.com/item?id=37134293), and even had a customer pay for our product. Despite the short-term achievements, we soon realized that long-term success was probably not in the cards.

After the batch, **we pivoted away from building infrastructure products**.

Here's how we made that decision, the lessons we learned, and how I'd approach it differently if I had to do it again.

 

## Too simple to be useful, or too useful to be simple

It's extremely difficult to build an infra product that's both simple *and* flexible enough.

When building an infrastructure product you *usually* have to choose whether you want to serve small or large companies first. That's because these two types of companies care about diametrically opposite things.

Large companies want flexible products that can support their peculiar infrastructure needs, even if it takes a bit longer to set up. On the other hand, small companies would rather have a simple piece of software that takes only a few minutes to set up, even if it doesn't support esoteric network settings, for example.

If you decide to build an infra product for large companies, it will almost certainly be too complex for small companies to use. Conversely, designing an infra product for small companies will make it too simple for large companies to use.

![](/posts/pivoting-away-from-infra/complexity-quadrants.png)

In an ideal world, you'd be able to hide complexity from small companies, while still supporting the needs of large companies. But in practice, this is nearly impossible to achieve - that's why so few infrastructure products are successful.

 

## Small companies don't need it

We knew that selling infrastructure to large companies was going to be difficult, so we decided to focus on building a tiny product that was useful for small companies first.
![](/posts/pivoting-away-from-infra/our-quadrant.png)

Building a small product would allow us to iterate faster because we'd have customers early on, and we'd be able to get feedback from them. Then, we'd be able to raise capital to expand the product and go after larger companies.

At the time, we thought that pull-request previews were a good candidate for an MVP. We could start by deploying preview environments for pull requests, and then we would expand the product so that people could use these environments for development too.

The first version of our product was simple. It used a `docker-compose.yml` file to deploy pull requests and then posted a comment on the pull request with a link to the preview deployment.

![](/posts/pivoting-away-from-infra/pr.png)

Just a few days after we put a landing page up, a small company found us and started using the product. As soon as they hit the free tier's limits, they reached out to pay.

We then tried to replicate that success with other companies, but it was extremely difficult. That's because a small company's software is usually small. Thus, they can just check out the branch and get it running in a few seconds. Having preview links was convenient, but not game-changing enough to make them pay.

The same principle applies to other infrastructure products. Even if your software is pretty simple, small companies won't care about it unless it's absolutely essential for them. And even then, they might not want to pay for it, and might just default to a free alternative.

 

## Large companies won't trust you

After our small bet failed, we decided to rebuild our product to serve large companies, but we had a hard time getting them to trust us.

When deciding whether to purchase a product, buyers usually have the following equation in mind:

![](/posts/pivoting-away-from-infra/value-equation.png)

The higher the value, the more likely they are to buy.

The problem with infrastructure products is that the time-to-value and implementation effort variables are usually too high, while the perceived probability of success is too low. Consequently, no matter how amazing the dream outcome is, the final value is usually too low for buyers to move forward.

Now, let's dissect each of these variables and explain why the perceived probability of success is usually too low, and why time-to-value and implementation effort are usually too high.

 

### Perceived probability of success

![](/posts/pivoting-away-from-infra/perceived-probability-of-success.png)

Imagine you're running a multi-billion dollar company. Would you trust a scrappy 2-person startup to handle the infrastructure that keeps your engineers productive? I wouldn't.

Sometimes, not even large providers like AWS or Google Cloud can support these companies without a lot of hand-holding. So, why would they trust a 2-person startup?

At large companies, not rocking the boat is usually more important than getting the latest and greatest technology. Consequently, they won't trust you unless you have solid proof that you're at least 10x better than the current alternative.

 

### Effort

![](/posts/pivoting-away-from-infra/effort.png)

Creating an infrastructure product that demands little effort to implement is extremely difficult because of the very nature of the problem you're solving. You're building a product that has to be flexible enough to support all the unique needs of large companies, and that's usually complex, as we've seen earlier.

Just for engineers to understand how your product works, they'll have to spend a lot of time reading documentation, watching videos, and maybe even attending training sessions.

Furthermore, it's difficult to break down the problem into smaller pieces that can be solved independently. That's because the problems you're solving are usually interconnected, and solving one might create new problems that need to be solved.

Finally, considering all this effort, they'll probably default to having their own engineers build it, instead of buying your product. That's because they'll have more control over the final product, and they'll be able to customize it to their unique needs.

Ah, did I mention they hired tons of engineers who need to justify their next promotion?

 

### Time-to-value

![](/posts/pivoting-away-from-infra/time-to-value.png)

In addition to all the time engineers will have to spend learning how to use your product and set it up, it will take a while for them to see the productivity gains you promised.

That's because infrastructure products tend to deliver compounding value over time instead of delivering acute value upfront.

Take CI/CD products, for example. Testing and deploying your software for a month isn't that painful and doesn't take that much time. However, if you have to do it for a year, it becomes a huge pain point, and you'll want to automate it.

 

## Slow feedback loops

The more feedback you have, the smaller the bets you have to make, and thus, the less risk you have to take.

Think about buying 3-digit lottery tickets, for example. Paying $1 for each digit as numbers are drawn is better than paying $3 for all three digits at once.

If you have to buy all three digits at once, the likelihood of guessing all three is only 1%, and each attempt costs you $3.

![](/posts/pivoting-away-from-infra/lottery-3-digits.png)

On the other hand, if you pay $1 per digit, you can decide whether to continue as soon as the first number is drawn. If you get it right, you can pay for the next; if not, you can give up sooner and lose less money.

In this case, there's only a 10% chance you'll buy the second number and a 1% chance you'll buy the third. Therefore, each failed attempt will cost you less, but the overall likelihood of winning is still the same.

![](/posts/pivoting-away-from-infra/lottery-1-digit.png)

Building an infra product is like buying all three digits at once. You have to make a large bet upfront, and you won't know if you're building the right thing until you get your first customer. That's because it's difficult for large companies to trust you, and, even when they do, slow sales cycles and long implementation times will elongate the feedback loop.

Additionally, it will take longer to have a large enough sample size to know if your product is any good, and to understand whether a request is a one-off or a recurring need for most customers.

 

## How I would approach it differently if I had to do it again

When picking a startup idea, you're also picking a set of problems that come with it. Ultimately, all problems are solvable, but some are more difficult than others for a given team.

Even though I'm glad we pivoted away from infrastructure products, I still believe they're a great opportunity for the right team, so here's how I'd approach it differently if I had to do it again:

* **I'd insist on selling to small companies first.** Instead of investing time trying to sell to large companies, I'd invest time in rethinking the product so that it's useful for small companies. That's a piece of advice we got from PG himself, and it was spot on.
* **I'd accept having to make larger bets and act accordingly.** Given the slow feedback loop, I'd be more inclined to think long-term and make larger bets, like building more features upfront and building a community around the product, even if they're not paying customers, and probably won't be for a while (or ever).
* **I'd cut scope as much as possible and avoid being seen as a replacement for existing solutions.** I think this point is valid for all startups, but it's especially true for infrastructure products because switching infrastructure usually takes longer than anything else. Therefore, I'd think harder about whether I can build something small and less critical to go alongside existing solutions.

 

## How these learnings helped us pick our next idea

The worst part of building an infrastructure product was the slow feedback loop, so our first criteria for picking our next idea was to find a problem that we could offer a packaged solution for, and that we could sell to small and medium companies first. That solution also had to be easy to use, and the time to value had to be as short as possible.

Considering these criteria, we then brainstormed a few hypotheses and ran approximately fifty customer interviews to validate them.

We then landed on what we're doing today: a product that helps companies generate unique insights by using code.

![](/posts/pivoting-away-from-infra/demo-short.gif)

The reason Briefer turned out to be a great fit for us is that:

* **Time-to-value is extremely short**. Instead of spending days reading documentation and setting up the product, customers can just log in and start generating insights.
* **It appeals to both small and large companies**. Small companies usually love Briefer because their engineers can write code to generate insights, and they don't have to wait for a data team to do it for them. Large companies love it because they can use it to build more advanced analyses, data apps, and predictive models.
* **We don't have to replace existing solutions**. While some companies replace their BI tools with Briefer, most use it alongside them because traditional BI tools can't generate the insights Briefer can, nor can they be used to build data apps or predictive models.

In the end, we're glad we went through the infrastructure journey, even though it was tough. It taught us a lot about building products, and it helped us land on a better idea.

I'm extremely thankful for the YC partners, our batchmates, and the YC alumni who helped us along the way. Many of these learnings wouldn't have been possible without their help.

Finally, if you want to try Briefer, or even if you're not sure it's the right choice, please feel free to [visit our website](https://briefer.cloud) and [book an intro call with us](https://calendly.com/lucasfcosta/briefer-intro-call). We'd love to hear from you.

Here's the story of how we pivoted after our YC batch, the lessons we learned building in the infrastructure space, how I'd approach it differently if I had to do it again, and how these learnings helped us build Briefer.

What we learned from a failed infrastructure DevTool

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.

![](/posts/p-values/honeyberry-method.png)

 

## Step 1: Understanding p-values so you can misuse them

Your first step to riches is to create a technique that you can claim to improve any aspect of your customers' lives and that you can write books about.

It doesn't matter if your technique actually works. What matters is that you can convince people that it does, and that's what I'm here to teach you.

Let's say you've created a new study technique and you believe it helps students score better on exams, for example.

To test your hypothesis, you set up an experiment with two groups of students:

* Control Group: This group studies using traditional methods.
* Experimental Group: This group uses your revolutionary study technique.

Then, you give both groups the same exam and compare their scores. If the experimental group scores significantly higher, some people might conclude that your study technique is effective.

![](/posts/p-values/two-groups.png)

Now, that's where amateurs usually stop. But you're not an amateur, you're a business person. You need to convince even the nerdiest of your customers that your technique is backed by science.

These people like to think they are smart, so they'll definitely ask how do you know whether the difference in scores is real or just due to chance.

That's where p-values come in. The p-value is a statistical measure that determines how likely it is that the improvements in scores were due to random chance. In other words, p-values represent the chance of the same or more extreme scores happening even if your new study technique doesn't actually work.

A small p-value means that a difference in scores as extreme as the one you observed is unlikely to have occurred by mere luck. Thus, a small p-value suggests that your study technique might be effective.

For example, a p-value of 0.001 means there's only a 0.1% chance that the score improvement was just random. This suggests that the better scores could be due to your new study technique, not just chance.

On the other hand, a large p-value means the difference in scores could easily have happened by chance, and your study technique might not be effective.

For example, a p-value of 0.15 means there's a 15% chance that the score improvement was random. Having such extreme results 15% of times would lead nerds to say that your results are not "statistically significant", because these events are actually not that rare. In other words, they mean that the difference in scores could easily have happened by chance, and your study technique might not be effective.

> The way that classical statistics works is by comparing study data to what is expected when there is nothing. If the data are not typical of what is seen when there is nothing, there must be something!
>
> — Dallal, Gerard. The Little Handbook of Statistical Practice.

Having learned what p-values are, you probably figured that what we're after is a small p-value. So let's go ahead and make sure we get one.

 

## Step 2: Ensuring that `p` is small enough

Let's be honest: the chances of your revolutionary study technique actually working are slim. Consequently, your experiment will probably fail to demonstrate that the new study technique is much different from just doing whatever everyone is already doing.

Once your study fails, you have two options. The first is to repeat the experiment until it works. The second is to analyze the data in multiple ways until you find another measure that looks significant. Then you can rebrand your technique as being effective for that measure, like "improving memory retention" or "increasing focus".

* Pro-tip 1: if none of the measures look significant, you can always try to find a subgroup of the population that benefits from your technique. For example, "our revolutionary study technique is particularly effective for left-handed students born in the summer".
* Pro-tip 2: you can also find transformations of the data that make it look significant, like the sum of outcomes squared or the log of the difference between scores.

![](/posts/p-values/python-honeyberries.png)

This step is pretty easy to memorize because both alternatives rely on the fact that if you look around enough, you'll eventually find something that looks like a significant result. Another way to put it is that "the chances of winning the lottery are small, yet often there's a winner".

 

### A crucial detail: make sure p is smaller than 0.05

There's a convention among scientists that says that a p-value below 0.05 is said to be "statistically significant". So that's what you should aim for, at least most of the time.

There are many theories about why statistical significance is set at 0.05, but, as Gerard Dallal points in his book "The Little Handbook of Statistical Practice", all of them trace back to Karl Pearson's 1914 book "Tables for Statisticians and Biometricians".

![](/posts/p-values/karl-pearson-book.jpeg)

Pearson figured out how likely different outcomes were in statistics. Then Fisher made it easier by creating simple tables in his book "Statistical Methods for Research Workers" in 1925. These tables just showed basic probabilities for certain outcomes. Fisher's idea caught on, and he did the same thing with Frank Yates in 1938. People still use Fisher's tables in statistics texts today.

Fisher's tables were shorter and simpler than Pearson's. Instead of giving lots of details, they just gave rough probabilities. Fisher also came up with a neat way to show how sure we are about results—using stars. One star meant a result observed about 5% of the time, two stars meant it was observed about 1% of the time, and sometimes three stars meant it was observed about 0.1% of the time.

Fisher also said that when these extreme results would only happen 5% of the time without an intervention, we should take notice. He thought this level was good because it helped us find real results without getting too many false alarms. And this 5% rule has stuck around, shaping how we understand data and what we consider important in science.

In one sense, 5% is just a number, and it's not a magic number. But it's a good rule of thumb. Still, some people think it's too high, and that we should use lower levels like 1% or 0.1%.

To make matters simpler, I'd also recommend that you keep this excellent table from the scientific website `xkcd` handy:

[![](/posts/p-values/xkcd-1478.png)](https://xkcd.com/1478)

 

## Step 3: Publishing your results

Now that you have a small p-value, you need to call journalists and tell them about your revolutionary study technique. They'll be thrilled to write about it, especially if you can provide them with a scientific paper that includes the words "p-value" and "statistically significant".

The important thing here is not to mention that you've run your experiment 100 times until you got a significant result, nor that you've written a python script to test all possible variables until you found one that looks significant.

![](/posts/p-values/honeyberry-news.png)

Another important thing is to explain p-values to them in a way that doesn't *sound* egregious, but that is not entirely right either.

One incorrect way to explain p-values is to say that they represent the probability that the "null hypothesis" (your study technique not working) is true.

Let's say you've found a p-value of 0.001, for example. In that case, make sure to tell them that there's only a 0.1% chance that your study technique is ineffective. Although that's incorrect, it's much more convincing than saying "if the new study technique didn't work, we would still see the results we did approximately 0.1% of the time".

Another technique that you can use is to invert the logic. Instead of saying that a p-value of 0.001 means there's a 0.1% chance that the score improvement was random, they say that there's a 99.9% chance that the new study technique is effective. That's not correct either, but it's looks better in a headline.

 

## Step 4: Rebuking the skeptics

Once your revolutionary study technique is published, you'll probably face some skepticism. That's normal, and you should be prepared to deal with it.

The scientists are the most dangerous skeptics. They'll probably try to replicate your experiment and fail to find the same results.

If it's only one or two scientists, you can always say that "failing to find an effect is different from showing that there is no effect". They won't be able to argue with that because you're technically correct.

Now, if quite a few scientists fail to replicate your results, then you're in deep trouble. [That's why it's best to design experiments that are hard to replicate, like those that require a lot of data or that are too vague to be disproven](https://en.wikipedia.org/wiki/Replication_crisis).

In general, keep in mind that the best way to deal with skepticism is to prevent it from happening in the first place.

 

## Putting it all together

* P-values are a measure of how likely it is that you'd observe the results you did if your intervention didn't actually work.
* The smaller your p-value, the less likely you are to have observed the results you did by mere luck.
* Conversely, the larger your p-value, the more likely it is that you'd observe the results you did by mere luck.
* A p-value is *not* the probability that the null hypothesis is true. The opposite is also true: a p-value is *not* the probability that your intervention is effective.
* Just because you haven't observed an effect, it doesn't mean there isn't one.
* If you run the same experiment many times, you'll eventually see a p-value below 0.05, even if the null hypothesis is true.
* Looking for an effect until you find one is an effective way to make sure you get a small p-value, but it's bad practice unless you're looking for an easy way to sell books.
* A p-value below 0.05 is usually considered "statistically significant". Although it's not a magic number, it's a rule of thumb that has been around for a while.

 

## Before we go

In case you didn't notice, this post is a satire, and any parallels to the real world are, obviously, coincidence (no pun intended).

The post was inspired by the excellent work of [Randall Munroe](https://xkcd.com), in particular his comic [xkcd 882](https://xkcd.com/882):

[![](/posts/p-values/xkcd-882.png)](https://xkcd.com/882)

Finally, if you're a data-scientist who loves jupyter notebooks or data apps, but finds them hard to share your work with others or too much hassle to deploy, you should check out [Briefer](https://briefer.cloud).

In this blog post, I'll teach you how to sell books by abusing statistics. Along the way, you will learn what p-values are, what they are not, and why you should care about them.

What p-values are, what they are not, and how to sell books with bad statistics

This changelog is the most important one we've shipped so far - that's why it took more than two weeks for us to post it.

In these last few weeks, we've completely rebuilt our editor so it's more understandable, more reliable, and more flexible.

We've also introduced environment variables, which allow you to store information like settings, API keys, and database credentials securely. Then, you can reference these variables in your code without exposing them.

 

## Our new editor and a ton of related improvements

Before this change, our editor was more unstable than we'd like. Sometimes the editor would show different states than what was happening in the back-end. Refreshing the page also caused the editor to lose part of its state, like which blocks were running, or undergoing AI suggestions. **All these issues are now gone, and we're glad to have received lots of positive feedback about the new editor.**

The new editor we're rolling out for everyone today is more reliable, more flexible, and more understandable. We now manage and reconcile every single piece of state in the back-end. Consequently, the editor's state will be consistent across refreshes, and for all users and devices.

![](/posts/changelog-4/v2editor.gif)

This change also comes with a bunch of very important improvements. Here are some of them:

* When using data apps, each viewer will have their own state. That means that if you're viewing a document and someone else is viewing it at the same time, you won't see each other's inputs and results.
* The "Run All" button now shows the number of blocks that have already run, and how many are left.
* We explicitly display enqueueing and running states for each block.
* When a scheduled run is happening as you view a document, you'll see its results in real time.
* You can now duplicate whole blocks or individual tabs by clicking the drag-handle menu on the left side of the block.
* You can run an entire block from left to right by clicking the "Run all tabs" button that appears within the drag-handle menu for multi-tab blocks.
* Data apps will only be updated when you click the "Update Data App" button. This way, you can make multiple changes to the inputs and dropdown fields of a data app and only publish them when you're ready.
* Adding new blocks is now more intuitive. You can now add blocks by clicking the "+" button that appears when you hover over gaps between blocks.
* You can now group all types of blocks, not only Python, SQL, and Visualizations.
* If you create a new dataframe that only exists in a Python block, it will be immediately available within visualization blocks.
* You don't have to uncollapse a block to run it anymore. You can now run a block by clicking the "Run" button that appears when you hover over the block.
* We'll now persist the open/close state of the document tree on the left as you navigate between documents. This way, you won't have to keep opening nested folders every time you navigate to a different document.

 

## Environment variables and how to use them

Environment variables are a way to store information like API keys, database credentials, and other settings securely. That way, you can then reference these variables in your code without exposing them to viewers.

These environment variables are useful when you want your Python blocks to perform authenticated API requests, like when doing writebacks to your database, for example.

To create an environment variable, click the "Environment Variables" button on the bottom-left corner of the editor. Then, click the button at the bottom of the list to add a new variable. Once you're done, click the save button and restart your environment for the new environment variables to be available.

You can then reference this variable in your Python code by using the `os.getenv` function.

![](/posts/changelog-4/v2vars.gif)

By the way, we also added a button on the bottom right of the editor so you can restart your environment manually if you need that.

 

## What's next?

We know it's been somewhat annoying to lose your in-memory state when your environment restarts. We're working on a way to automatically re-run dependencies or persist this state across restarts, so you won't lose your dataframes, variables, and other in-memory state when your environment restarts.

Expect that by the next changelog.

This changelog is the most important one we've shipped so far - it includes a brand new editor and support for environment variables.

Changelog #4: Environment variables, and a brand new editor

Besides the usual bug fixes, we're shipping a major new feature this week: **data apps**.

You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.

![](/posts/changelog-3/data_apps.gif)

Please notice data apps are only available for users in the professional tier.

 

# When to use data apps features

You should use data apps features whenever you want viewers to be able to provide input to your reports, models, or analyses.

In a nutshell, data apps are interactive documents that allow viewers provide input to the code you've written.

Typical examples include:

* **Comparing scenarios**: Allow viewers to change assumptions and see how the results change.
* **Exploring data**: Allow viewers to change the data they're looking at.
* **Filtering data**: Allow viewers to filter data by date, region, or any other dimension.

 

# How data apps work

To create a data app, you must first add an input or select field to your document by using the `/input` or `/select` commands.

For each of those blocks, you can specify a label. The variable name is the one you see on the top right corner of the block in a green box.

Then, you can reference the variable name in your Python code in the same way you would reference a variable you created yourself.

To interpolate the value of the input or select field in SQL blocks, use the `{{variable_name}}` syntax.

> Please notice that you can now interpolate *any* variables in your SQL blocks, not only the ones created by input or select fields.

Finally, use the "Make it a data app" toggle on the top right corner of your document, under the share menu, to enable the data app mode.

Toggling the data app mode allows viewers to provide input to your document, but they still can't edit the document's content.

 

## Other fixes and improvements

* We fixed a bug where users would not be automatically logged out once their session expired. Now, users will be imediatelly logged out and redirected to the login page when their session expires.
* Document snapshots: we now keep snapshots for users in the professional tier. These snapshots are taken every five minutes and will be available for the last 5 days. The last 10 snapshots are always available, regardless of how old they are.
* We've done updates to the visualizations so they support data types they used to not support.

This week, we're introducing data apps. You can now add inputs and select fields to your documents, and allow viewers to provide input to your Python code or SQL queries.

Changelog #3: Introducing data apps

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements.

If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.

 

# General Updates

Here are a few small improvements we've done to the editor, and to the overall performance of Briefer.

## Editor performance improvements

We noticed that performance degraded as documents got larger. We've made a few changes to the way we handle large documents.

First, we've eliminated lots of unnecessary re-renders for graphs and tables. This should make the editor feel snappier, especially for large documents.

## Support for larger datasets

Some users dealing with millions of rows of data were experiencing performance issues.

To fix that, we now run all the queries in your own compute instance and stream the results back to the editor. This change means that we can now support much larger datasets without performance issues. It also means that the users who pay for larger compute will experience more benefits from it.

## Improvements

Besides the performance improvements we shipped a few minor UI updates. Besides all the small UI tweaks we've done, here are a few highlights:

### In/Not-in filters

We've added support for `in` and `not-in` filters in visualization blocks. This is useful when you want to filter by multiple values at once.

![](/posts/changelog-2/in_not_in_filters.png)

### New query on visualizations

You can now create queries more easily when using visualizations. Just click on the "New query" option the dropdown, or in the tooltip that appears when you create an empty visualization.

![](/posts/changelog-2/new_query_option.png)

You can also create new queries or tabs by using the `+` button in the top right corner of the block.

![](/posts/changelog-2/plus-tab-button.png)

### Deleting tabs or entire blocks

You can now delete tabs or entire blocks by clicking on the drag-handle and selecting the desired delete option.

![](/posts/changelog-2/delete_tab_or_block.png)

### New users are viwers by default

Previously, when allowing anyone from `@yourdomain.com` to access Briefer, new users were added as editors by default. We've changed this so that new users are added as viewers by default. This is to prevent new users from accidentally making changes to the workspace.

 

# Bug fixes

Bug fixes are the biggest part of this release. We've fixed a ton of bugs, but here are a few highlights:

* Fixed a bug where there was a race condition when creating tokens for compute instances. This was causing some users to experience multiple "something went wrong" alerts that would only go away after the environment got restarted.
* When changing the filters or settings for visualizations multiple times, we'd enqueue each update. This caused visualizations to be slow to update. We've fixed this so that only the latest update is enqueued and previous updates are discarded.
* Our AI suggestions were sometimes suggesting overwriting dataframes using `[Object object]` dataframes. We've not only fixed this bug, but we also improved the LLM's context.
* Creating new tabs was causing an empty block to appear in the editor until you switched to another tab and back. This was a visual bug that didn't affect the actual document, but it was annoying - and now it's fixed.
* Dragging and blocks into one another was not working properly. Sometimes, the editor would try to drag the wrong block, or it would not detect you wanted to group different blocks. This bug is now fixed.
* Editing AI suggestions and then accepting them was causing the edited diff not to be applied. Instead, it applied the raw AI suggestion. We fixed this so that any changes you do in the diff editor will now persist in case the suggestion is accepted.
* Display of titles was showing incorrect background colors during loading and disabled states. They were also not taking the appropriate space. We've done fixes to their appearance so they behave correctly in all cases.
* Visualizations were not being updated in scheduled runs. This was a bug that was causing some users to receive outdated visualizations in their scheduled reports. This is now fixed.

This week, we've focused on improving what already exists, rather than adding new features. We've shipped a ton of bug fixes and performance improvements. Here are some of the highlights.

Changelog #2: Blazing fast performance and tons of bug fixes

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently.

This is a non-exhaustive list of everything that's changed. As we publish more updates, we'll increase the level of detail here.

If you've got questions, comments, or any feedback, email us at contact@briefer.cloud.

 

# New features

This week: no-code visualizations, tabs (grouping blocks), naming blocks, workspace settings, and exports.

 

## 1. No-code visualizations 📊

The most important change comes first: you can now create graphs without code. For that, type `/visualization` and add a visualization block.

![](/posts/changelog-1/visualizations.gif)

There, you'll be able to select the query to plot, types of graphs, filters, and more.

Once you're done with your visualization you can click the double-chevron icon to toggle between edit and visualization modes. In visualization mode, only the graph and filters appear.

Please notice that histograms and pie charts are not available yet.

 

## 2. Grouping blocks 🫂

You can now group multiple blocks into one to create multi-tabbed blocks.

Grouping blocks is useful when you need multiple python blocks to clean data, but only want a single python block with the results to show.

![](/posts/changelog-1/grouping.gif)

You can also drag-and-drop tabs to change their order. To ungroup tabs, drag the desired tab out of the tabbed block.

 

## 3. Naming blocks 🪄

Now that you can group blocks, it's important that you name them so that you know what's in each tab.

Naming blocks is also helpful for presenting reports - it helps stakeholders understand what they're seeing.

![](/posts/changelog-1/naming.gif)

To name blocks, just click the title of each block and input whatever title you want.

 

## 4. Workspace settings ⚙️

We heard a few people wanted to change their workspace names or settings *after* the workspace had been created. That wasn't possible before.

Now, you can rename your workspace and enable everyone with the same email address to join.

![](/posts/changelog-1/workspace-settings.gif)

 

## 5. PDF exports 📑

We know many of you were asking for this, so we have shipped it.

You can export PDF files by opening the share menu on the top right corner and then clicking the PDF export option.

**For now, PDF exports are only enabled for users in the professional tier.**

![](/posts/changelog-1/pdf-exports.gif)

Welcome to Briefer's first public changelog. Here is a list of the most important changes we've shipped recently. This week: no-code visualizations, tabs, and exports.

Blog

How Regularizers Fix Dumb Solutions

More articles

How Regularizers Fix Dumb Solutions

P-Hacking in Startups

Confidence intervals

Why Use Bayesian Methods for A/B Testing

Bias-Variance tradeoff: understanding model complexity

The geometry of taste: putting numbers on preferences