Senior Data/AI Engineering

Job description

JOB LOCATION: 2900 Plano Pkwy., Plano, TX 75075 DUTIES: Responsible for the development of high performance, distributed computing systems using Big Data technologies for GenAI development. Build scalable multi-threaded Spark clusters using Databricks interfacing with NoSQL, including data mining using various distributed technologies on the Azure Cloud platform. Design and deliver GenAI technologies at scale with real time results using OpenAI and large language models to service chat experience for business partners across the company. Engage with clients to understand business requirements and work with teams to identify architecture needed to deliver necessary features. Deliver insights and improvements in GenAI results and influence development decisions for optimal software delivery. Build auto LLM to automate discovery of the best configuration to be identified in leaderboard results using RAG models. Build truth set auto generator to optimize leaderboard ranking based on standardized questions and answers. Integrate various calculation and models for GenAI truth set results to rank best performing results. Integrate Lang Chain loaders to embed model packages to be able to p...