LLM Devops Engineer

LLM Devops Engineer

Please note that this role does not require you to build LLMs or fine-tune them. For RLHF data generation, you might have to put together or be provided a prompt by the customer, ask the model questions, and evaluate the outputs generated by two versions of the LLM model.

Day-to-day

• Design and develop challenging prompts based on provided source code with good coverage of DevOps and Infrastructure technologies
• Implement the verified code that can be executed to verify whether a model's response to the prompt is correct or not
• Conduct evaluations (Evals) to benchmark model performance and analyze results for continuous improvement.
• Evaluate and rank AI model responses to user queries across diverse domains, ensuring alignment with predefined criteria.
• Develop comprehensive explanations and rationales for evaluations, showcasing excellent reasoning and technical expertise.
• Lead efforts in Supervised Fine-Tuning (SFT), including creating and maintaining high-quality, task-specific datasets.
• Collabo

You bring

• Technical Expertise
• Proven experience with configuration management and infrastructure automation tools such as Ansible, Terraform, CloudFormation and/or similar platforms.
• Strong exposure to AWS cloud platforms with experience in designing and managing multi-cloud environments.
• Hands-on experience with container technologies (Docker) and container orchestration (Kubernetes).
• Proficiency in scripting languages (Bash, Python, etc.) for automation and tool integration.
• Familiarity with CI/CD tools (Jenkins, GitLab CI, CircleCI, etc.) and version control systems (Git).
• Operational Excellence

Pay

• Start: is next week].
• Commitments Required: 40 hours per week, with a 4-hour overlap with PST.
• Contractor position (Note: this role does not include medical/paid leave).