LionPact Hub

Know when your data pipeline goes stale.

LionPact Hub is a free, open-source R&D project on streaming data observability and agentic AI. It holds the tools I build and free tutorials on how they work.

Explore LionPact Watch Connect on LinkedIn
A streaming pipeline with one lagging stage Events flow from Kafka to Flink to an Iceberg table and on to Athena. Every link is fresh except Flink to Iceberg, which is 4 minutes behind and flagged for attention. 2s behind 4m 12s behind 9s behind Kafkaorders topic Flinkenrich job Icebergorders_clean Athenadashboards Analystsreports
The question LionPact Watch answers: which stage is behind, and by how much.

Projects

Each project is small, open and free. Each one is built to understand a real problem properly by solving it end to end.

LionPact Watch

In development

Observability for streaming pipelines built on Kafka, Flink and Spark Streaming. It checks data while it moves, so a late or broken stream is caught before it reaches a dashboard, a report or an ML model.

How it works and how to use it

  • FreshnessHow far each stage is behind real time
  • LatencyEnd-to-end time from source event to table
  • Schema contractsBreaking field or type changes, caught at the source
  • AnomaliesVolume and value drift against a learned baseline
  • LineageWhich downstream tables and reports a failure affects
  • AlertsSlack and email, with the stage and the cause

agentkit

Python package

A small Python package for building AI agents. It covers tool calling, a planning loop, memory and evaluation in 14 focused modules, about 1,300 lines in total.

It is written to be read. Each module does one job, so your team can see exactly how an agent decides, calls a tool and checks its own result.

What it does and how teams use it

  • Tool callingPlain Python functions become tools the model can use
  • Planning loopPlan, act, check the result, repeat until done
  • MemoryKeeps the conversation and working state between steps
  • EvaluationTest tasks that check answers and tool use automatically

LionPact Learn

Free tutorials

Hands-on tutorials that explain the ideas behind the projects, from first principles to production patterns.

Tutorials are being moved here from the old site, and new ones on pipeline observability and real-time streaming are being written. Follow on LinkedIn to hear when they go live.

Data engineering Lakehouse with Apache Iceberg Streaming pipelines Pipeline observability MLOps LLMOps RAG and embeddings Agentic AI

About

LionPact Hub is a personal R&D project by Rajesh Kaushik, a data engineer and architect based in Delhi.

I started it to go deeper than day-to-day work allows: to build observability and agent tooling from scratch, and to write down what I learn so other engineers and teams can use it.

Everything here is free and open source, and it will stay that way. There is nothing to buy.

What it is
Open-source projects and free tutorials
Focus
Streaming data observability, lakehouses, agentic AI
Built with
Python, Spark, Kafka, Flink, Apache Iceberg, AWS
Cost
Free, always

Want to know more?

For questions, early access, feature ideas or to follow progress, connect with Rajesh Kaushik on LinkedIn.

Connect on LinkedIn