A decade ago, the position of a data engineer was essentially nonexistent.  Slowly and gradually, the role changed as the field matured. But with the advent of new generative AI technologies, many people fear their jobs are in jeopardy.

Already in the early months of 2024, GenAI is beginning to upend the way data teams think about ingesting, transforming, and surfacing data to consumers. Tasks that were once fundamental to data engineering are now being accomplished by AI—usually faster and sometimes with a higher degree of accuracy.

As familiar workflows evolve, it naturally begs the question: will GenAI replace data engineers?

Thus, if a data engineer has to secure his position, he has to “Go All In”, said Pavan Najundalah, Head of Trendence Studio. 

Speaking at AIM’ Media House’s Data Engineering Summit 2024, Pavan said today’s data engineer needs a beginner’s mindset to adapt to the growing change.

“2023 was the year of copilots from Github to Fabric and Google. 2024 is the year of agents. Which means AI is after your job. The question you should be asking is, will I become obsolete? We saw what happened to DEVIN. The world will change whether you like it or not,” Najundalah said.

“Today’s data engineers should expand left and right. Focus on your core foundational skills because they are not going to change. You don’t need to handcraft your code anymore. We all know that augmented coding is the future,” he added.

Why do we need a change?

Ten years ago, businesses relied on on-premise infrastructure for data storage. At this time, data engineers were more concerned with fine-tuning their machine configuration than with generating business value.

The cloud companies appeared, promising to offer services they would handle on your behalf. You can then concentrate on your business’s needs. This has changed the game.

So, Najundalah asks an important question: If half your job is going to be done through a copilot, what will you do?

“Understand what is the business problem you’re trying to solve. My recommendation to data engineers is to shift left, which is the person who gave you the requirement and probably understands the business a little better. By doing this, you’re now zooming out of a core data engineer’s profile and understanding the business and the longer impact you can create,” Najundalah added.

“That’s not it. Start looking at the right side to see if a data scientist is building a model based on the data you have staged, and figure out how you can start expanding,” he added.

Division Into Schools Of Thoughts

Najundalah divided today’s GenAI into two schools of thought: one that sees everything as “rainbow and sunshine”,” while the other is “gloomy and fears a thunderstorm”.” 

“We have seen rapid changes in AI in such a short time that forces everyone to change. I believe today’s Chief Experience Officer (CXO) should “Bet and Check” while Data Engineering Practitioners should “bet big but in pockets”. 

Although humans losing jobs to robots is a lovely story, it is far from the truth for data engineers. AIM research tells us that data engineers continue to be in high demand. Senior developers working in generative AI draw over INR 1 crore per annum, while an entrant’s salary could easily be around INR 18 lakh per annum, much higher than India’s median income. 

Data engineers are needed to create and manage AI applications. Data engineers are increasingly responsible for how generative AI is integrated into the business, just as they develop and maintain the infrastructure supporting the data stack. AI infrastructure is created and maintained using the advanced data engineering abilities we discussed, including abstract thought, business comprehension, and contextual creation.

Furthermore, incorrect data can occasionally occur even with the most advanced AI. Things malfunction. Shortly, we don’t see an AI engaging in much self-reflection, unlike a human, who can recognise and fix mistakes.

So, when things go wrong, someone needs to be there babysitting the AI to catch it—a “human-in-the-loop,” if you will.

The post Data Engineers: Expand Left and Right: Pavan Najundalah appeared first on AIM.