Job Description

Black Canyon Consulting (BCC) is actively looking for Senior Linux System Administrator to support our work for the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM), an institute of the National Institutes of Health. This opportunity is full time and onsite at the NIH-NCBI in Bethesda, MD and/or remote work.

NCBI is part of the National Library of Medicine (NLM) at the National Institutes of Health (NIH). NCBI is the world’s premier biomedical center hosting over six million daily users that seek research, clinical, genetic, and other information that directly impacts biomedical research and public health – at NCBI you can literally help to accelerate cures for diseases! NCBI’s wide range of applications, platforms (node, python, Django, C++, you name it) and environments (big data [petabytes], machine learning, multiple clouds) serve more users than almost any other US Government Agency according to https://analytics.usa.gov/

We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement. If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!

Duties & Responsibilities

• Develop and document standard operating procedures for a wide range of tasks, such as systems provisioning, installation, upgrading, and migration; host and service monitoring; configuration management; problem troubleshooting; and other aspect of systems administration.
• Manage and configure services, software, and hardware configurations.
• Take direction from the team lead and work independently within a team-environment.
• Mentor junior team members.
• Learn new technologies and skills.
• Develop and maintain custom tools in Python, Perl, Ruby, and Bash to automate managing the UNIX infrastructure.
• Use Puppet to maintain thousands of physical and virtual Linux CentOS systems and associated applications.
• Develop policies and procedures for standard systems administration tasks (packaging, OS installs, patch management).
• Build open-source software from source code.
• Document procedures, systems, services, and software. Provide ad-hoc training where appropriate.
• Provide UNIX support to NCBI's internal developers and researchers.
• Provide technical solutions to a wide range of problems and user requests through an established trouble ticketing system.
• Troubleshoot, research, design, and implement solutions to complex systems problems while accounting for technical limits, overall cost, and security policies.
• Assist with planning and implementation of major OS upgrade, deployments, and architectural changes.

Requirements:

• Willingness to learn new technologies and skills.
• Demonstrate significant experience in Linux Systems Administration (7+ years).
• Comfortable with Unix command line tools.
• Highly versed in Linux administration and best practices.
• Fluent in programming/scripting skills for automation in at least two of Bash, Ruby, Python, and Perl. (Other languages also helpful, but not required.)
• General understanding of Linux internals (system calls, file systems, processes, etc).
• General understanding of TCP/IP and related protocols (HTTP, FTP, NFS, etc…)
• Comfortable working with users of varying levels of IT knowledge.
• Experience with building open-source software from source code.
• Able to work with minimum supervision on assigned tasks, and work with others both within and outside of the team.
• Strong communication skills, especially with regards to documentation.

Bonus Points (not required!):

• Experience in a complex, dynamic, production, public-facing Linux environment.
• Experience with OpenShift, Kubernetes, GKE, or similar technologies.
• Knowledge of cloud technology such as AWS and GCP.
• Experience with automated configuration management, such as Puppet (preferred), Ansible, Chef, bcfg2, CFEngine, etc.
• Experience with version control. (Git is preferred, but not required.)
• Experience supporting large web application environments and development.
• Experience with enterprise network architectures.
• Experience with monitoring, alerting, and observability tools.