Week 9 - Capstone

I am taking time to think about the future of data engineering. I want to make sure that I have a good understanding of the current state of the art and the future of data engineering.

From my obeservation of my Discord community, I have notice that there are a lot of people who already have the project done and are looking for feedback. Furthermore, you will easily find a lot of projects that are similar with them to help you get started.

Project Ideas

I include 9 project ideas that I have thought of. I will try to include the project ideas that are similar to the ones that I have already done.

• Big Data Framework, Stack Explore for handing Big Data
• Kappa Data Pipeline aka Realtime using AWS, procesinng streaming data on AWS
• Data Modeling and Analytic Engineering, data modeling and analytic engineering with dbt, airflow and metabase
• Data pipeline with Open Source Mage AI and Clickhouse, low-code tools for data engineering
• AWS Ingestion Pipeline, AWS data ingestion pipeline, one-size-fits-all
• Azure Data Pipeline in 1 hour, challenge to learning and practicing Azure Data Pipeline within 1 hour
• Design ETL Pipeline for Interview Assessment, How to design ETL pipeline for interview assessment
• AWS Local Stack with Terraform and Docker, start development on local with Mock of AWS stack using LocalStack
• Open Data Processing with dbt, data modeling practices, and databricks multi-hop medallion architecture.

Community Projects

• Setting up Open Mordern Data Project by Simon Späti
• Quick Start Data Engineering Skills by Tuan Tran
• List of 7 Data Projects for Data Engineers by Darshil Parmar
• Beginner DE Project by Joseph Machado

Call to Action

• If you have any project ideas? feel free to share them with me on Discord.
• If you have any feedback or suggestions? please feel free to share them with me on Discord.
• If you have any questions or need help? please feel free to ask me on Discord.

I am excited to see what you will build with this project. I hope that you have a great time and that you are able to learn a lot from this project.

Thank you for your time and I look forward to hearing from you soon.

Stay tuned. If you have any questions or know about me. Check out my profile here Long Bui.

Start The Project From Business Needs

Remember that the technology or innovation is for resolving business problems.

To Senior, we are the one who deeply understand the architecture and impact of the changes. Balancing the needs of business and the impact of technology is a challenge. Start from the business needs and build the solution that will solve the problem.

To Mid, we are managing the modules or components of the solution, resopnsibity for changes within the modules. Starting from what is the change and how to improve the ASIS, how to make sure that the change is done right.

To Junior, we are the one who is responsible for the implementation of the solution. We are responsible for the quality of the code, the performance, and the scalability of the solution.

To Fresh or Undergrad, we are new to this and being assisted by the mentors. We are learning the basics of the technology and the best practices. We are also learning how to work with the team and how to collaborate with other team members. Implementing the changes and actively participating the process will help us to learn and grow.

How Roles and Responsibilities in Scrum by Seniority?

I remcommend to read Srum Roles and Responsibilities by Atlassian Writers and all kind of Scrum Activities in Software Development Process.