Sr. Site Reliability Engineer, AI Infrastructure (Starshield) Verified today
About the role
SpaceX is deploying Starshield, the world's largest US government satellite constellation, tasked with providing immediate access to critical intelligence and national security data for the US government globally. As a Sr. Site Reliability Engineer focused on Starshield's software and GPU infrastructure, you will design, operate and scale the infrastructure which supports these critical national security missions. You will develop automation to deploy and manage on-premise compute resources, create highly scalable and maintainable software products, and directly collaborate with engineering teams across the organization.
Responsibilities
- Manage GPU/CPU infrastructure deployments to Top Secret data centers
- Manage and provide support for GPU as a service for external customers on bare metal hardware and virtualized platforms
- Design, validate, and productize solutions for AI clusters at 100k+ GPU scale
- Develop automation to deploy and manage on-premise Kubernetes and AI clusters, and operating systems
- Deploy and manage core infrastructure such as databases, monitoring and distributed storage
- Closely collaborate with AI engineers to create highly scalable, operable, and maintainable products
- Engage in the whole lifecycle of services from inception and design, through deployment, operation and refinement
- Implement monitoring and alerting to support high availability systems
- Identify areas for improvement and create innovative solutions that enable high system availability
- Mentor and train junior engineers and lead the team to technical excellence
Basic Qualifications
- Bachelor's degree in computer science, information systems/IT, or an engineering discipline and 5+ years of professional experience with Linux operating systems; OR 7+ years of professional experience in software, DevOps, or site reliability engineering in lieu of a degree
- 5+ years of experience with Kubernetes
- 5+ years of experience managing Linux operating systems
- Experience with Terraform, Ansible, or other infrastructure tools
- Experience with containerization technologies such as OCI containers and Kubernetes
- Experience scripting in Bash, Python, or similar languages
- Development experience in Python, C++, or Go
Preferred Skills and Experience
- 5+ years of experience with Python and Python-based development frameworks
- Experience managing Kubernetes clusters, not just using them
- Knowledge of Linux boot process and systems configuration
- Deep understanding of testing, continuous integration, build, deployment and continuous monitoring
- Understanding of relevant build technologies such as Bazel and Makefiles
- Focus on performance bottlenecks and performance improvement techniques
- Understanding of distributed databases and data modeling
- Experience with automatically managing thousands of servers using Terraform or Ansible
- Strong networking knowledge of TCP/IP
- Cloud virtualization experience as the cloud provider
- Experience with NVIDIA GPU deployment stacks such as Blackwell and Rubin
- Excellent communications skills with ability to communicate with customers, peers, and management in formal and informal situations
- Active Top Secret, Top Secret SCI, or DOE Level Q clearance
Additional Requirements
- Must be willing to work extended hours and weekends as needed
- Must be willing to travel domestically and globally when needed
- This position requires successfully obtaining and maintaining a Top Secret Security Clearance as a condition of employment. While the clearance may not be immediately necessary upon hire, you are encouraged to initiate the application process promptly upon accepting this offer.
Compensation and Benefits
Pay Range: Level 3: $165,000.00 - $265,000.00
Your actual level and base salary will be determined on a case-by-case basis and may vary based on job-related knowledge and skills, education, and experience.
Base salary is one part of your total rewards package. You may also be eligible for long-term incentives in the form of company stock or long-term cash awards, potential discretionary bonuses, and the ability to purchase additional stock at a discount through an Employee Stock Purchase Plan.
Benefits include comprehensive medical, vision, and dental coverage, access to a 401(k) retirement plan, short and long-term disability insurance, life insurance, paid parental leave, and various other discounts and perks. You will accrue 3 weeks of paid vacation and be eligible for 10 or more paid holidays per year. Employees accrue paid sick leave pursuant to Company policy.
Those with an active clearance will receive a 10% differential, up to an additional $20,000 annually, once officially briefed into a classified program.
ITAR Requirements
To conform to U.S. Government export regulations, applicant must be a (i) U.S. citizen or national, (ii) U.S. lawful permanent resident, (iii) Refugee under 8 U.S.C. Section 1157, or (iv) Asylee under 8 U.S.C. Section 1158, or be eligible to obtain the required authorizations from the U.S. Department of State.