Why a Modern Data Stack?
In a fast-growing company like Make IT Safe, data is everywhere. It comes from sources as varied as flat files, Excel files, MySQL databases and tools such as Pipedrive. But while data is essential, it quickly becomes unusable if it is not organised, cleaned and ready to be analysed. It is like having a library full of books with no labels and no classification: good luck finding what you are looking for!
Before we set up our Modern Data Stack, we were in a situation where each team worked with its own data, often cut off from everyone else’s. Not ideal! We were wasting time looking for information, and our decisions were based on data that was often missing or incomplete.
To solve this problem, we decided to build a modern, robust infrastructure: our Modern Data Stack. It is designed to unify all our data, make it accessible and, above all, keep it secure.
Our approach: an open-source, secure Modern Data Stack
Our technology choices were guided by three principles:
- Data sovereignty: All our data stays on our servers. Nothing leaves our infrastructure.
- Open source: We want to be in control of the tools we use, without depending on expensive or rigid proprietary solutions.
- Flexibility and security: Every building block of our stack is deployed and configured in a secure environment, in line with market standards.
It is as if we had built our own factory, perfectly suited to our needs, in which every machine (or tool) plays a precise role in turning raw data into analyses we can act on.
The building blocks of our Modern Data Stack
We have structured our stack around four main tools, each playing a key role in the data processing pipeline. Here is an in-depth look at each component.
Setting up our Modern Data Stack was not a spur-of-the-moment decision, nor was it driven by a taste for “new” and “cool” tools. On the contrary, every choice was carefully considered and based on two essential elements: our in-house expertise and the backing of external support.
We already had significant experience with the technologies involved. For example, our team knew MySQL well and had solid foundations in advanced SQL, which naturally steered us towards DBT and ClickHouse so that we could make full use of that skill set. We also knew that data integration and ETL pipelines were challenges we could meet effectively with a tool like Airbyte, which respects our technical and organisational constraints.
However, to make sure the transition took place in the best possible conditions, we did not hesitate to surround ourselves with external experts. These partners helped us configure the tools, optimise the pipelines and avoid common pitfalls, while sharing best practices suited to our needs. This support not only sped up the implementation of our architecture, it also allowed our in-house teams to build up their skills in these new technologies.
By combining our in-house expertise with targeted external support, we were able to build an infrastructure that is not only high-performing but also fully aligned with our strategic objectives and our operational constraints. This approach allowed us to adopt modern tools while remaining pragmatic, without giving in to the mere urge to use fashionable technologies. Each tool was chosen for its ability to meet specific needs while fitting smoothly with the other components of our stack.
Airbyte: the universal connector that does all the dirty work
Airbyte is our “data vacuum cleaner”. It goes and fetches information wherever it is, whatever the source. Whether it is a MySQL database, an API or a CSV file, Airbyte can connect to all of it and deliver the data straight into our warehouse.
But Airbyte does more than extract data. It is also designed to run pipelines reliably. For example, if a task fails (and it happens!), it restarts it automatically.
A concrete example:
Let’s say we want to retrieve data from our Pipedrive CRM to track sales opportunities. With Airbyte, we have set up a connector that synchronises this data with our data warehouse every day. No more manual downloads or laborious imports.
Why we love it:
- It is open source, so it can be fully customised.
- Its connector library is huge: more than 300 integrations available.
- It offers a clear interface for supervising all data flows.
DBT: the conductor of data transformations
If Airbyte is the vacuum cleaner, DBT (Data Build Tool) is the architect. It takes the raw data that has been extracted and turns it into models ready for analysis. The idea? To structure the data so that teams can understand it and use it.
With DBT, our analysts can write models in SQL to clean, aggregate or enrich the data. But that is not all: DBT keeps a record of every transformation carried out, which guarantees complete transparency and auditability.
A concrete example:
Take customer satisfaction data. It arrives in our data warehouse in raw form: a mix of comments, numerical ratings and metadata. With DBT, we have created a model that automatically sorts feedback into categories (positive, neutral, negative) and calculates an overall satisfaction score.
Why it is indispensable:
- Transformations are versioned, just like code, which makes collaborative work easier.
- Errors are easy to trace and correct thanks to the history of the models.
- Our teams save a huge amount of time by reusing existing models.
ClickHouse: the data warehouse built for speed
ClickHouse is the brain of our stack. This is where all the data transformed by DBT is stored, ready to be analysed. Why ClickHouse? Because it is designed to handle huge volumes of data while remaining extremely fast.
Unlike a conventional database, ClickHouse is optimised for analytical queries. This means we can query billions of rows and get an answer in a few milliseconds.
A concrete example:
We use ClickHouse to analyse marketing performance. How many leads were generated this week? Which channels work best? These answers, which used to be laborious to obtain, are now available almost instantly.
Its superpowers:
- Advanced data compression, which reduces storage costs.
- An OLAP (Online Analytical Processing) architecture, ideal for multi-dimensional analysis.
- Scalability: it can handle petabytes of data without breaking a sweat.
Metabase: the interface that gives data meaning
Metabase is our window onto the data. It is the tool we use to visualise and explore all the information consolidated in ClickHouse. One of Metabase’s biggest advantages is its simplicity: you do not need to be a data scientist to use it.
With Metabase, each team can create its own dashboards or ask questions of the data directly, such as “What are our best-selling products?” or “What is the average satisfaction rate this quarter?”.
A concrete example:
Our CSM team uses Metabase to track customer satisfaction. At a glance, the team can see which customers need particular attention and which actions to prioritise.
Why it is great:
- Intuitive, even for non-technical users.
- Dashboards are interactive and customisable.
- API integration to automate the sending of reports.
Security and governance: an absolute priority
For us, data security and governance are not optional. Here is what we have put in place:
- Hosting within our secure network: No tool and no data leaves our infrastructure.
- Strict access control: Only authorised people have access to specific data.
- Full auditability: Every change and every query is logged to guarantee transparency.
We have also adopted internal policies to train our teams in data management, a crucial point for avoiding human error.
Future possibilities: integrating AI and language models
Our Modern Data Stack, as it stands today, provides a solid foundation for collecting, transforming, storing and analysing our data. But we are looking further still. One of the most exciting opportunities lies in integrating artificial intelligence, and more specifically large language models (LLMs), to maximise the impact of our data.
By adding AI agents or LLMs such as Llama, we could go beyond descriptive analysis. For example, these models could not only identify trends in our historical data but also generate projections based on different scenarios. Let’s take a concrete case: an LLM connected to our pipeline could analyse the performance of a marketing campaign and, on that basis, suggest strategic adjustments or even simulate the impact of an additional marketing budget on future sales. Llama stands out as a solution particularly well suited to our needs for several reasons, notably the fact that it is open source and that we can host it on our own infrastructure.
These agents could also play a part in supporting our customers proactively. By combining their predictive analysis capabilities with their conversational skills, we could provide personalised recommendations to each customer. For example, a customer looking to improve its regulatory compliance could receive a list of specific suggestions tailored to its sector of activity, together with a clear, reasoned explanation.
Another promising area is the automation of decision-making tasks. By using LLMs to process structured data in ClickHouse, we could automate complex processes such as cost optimisation or anomaly detection. This would let our teams focus on more strategic tasks, while ensuring that our data is actively and continuously working for us.
However, integrating AI does not come without challenges. We must make sure these tools meet the high standards of security and sovereignty we have set. The data used to train the models, or to interact with them, must remain confidential and entirely under our control. In addition, we must ensure that the recommendations generated by AI are explainable, understandable and aligned with our customers’ specific needs.
All in all, integrating AI into our infrastructure is a natural and ambitious step. It will allow us not only to explore new horizons in data analysis, but also to strengthen our commitment to our customers by offering them even more relevant and personalised solutions. These innovations will form the foundation of our next stage of development.
Conclusion: a stack for today and tomorrow
By building this Modern Data Stack, we have transformed our relationship with data. Not only are we more efficient, we can also support our customers more proactively. This infrastructure is an investment in our future, and it is designed to evolve with our needs.
Acknowledgements
I would also like to extend my warm thanks to Sophie Lohezic for her invaluable support in setting up this stack. Her expertise in data architectures and pipelines was decisive in structuring the infrastructure, optimising certain technical choices and avoiding several classic pitfalls when building a Modern Data Stack.
Beyond the technical side, her advice also helped speed up the move to production and strengthen good practices around data governance and data management.
