GovCIO is currently hiring for Systems Architect (Senior) of Infrastructure Operations to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.
Responsibilities Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment. Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering.Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.Maintain data flow visibility from source → forwarder → indexer to confirm data completeness and performance.Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults.Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies.Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review.Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations.Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected.Document incidents, RCA findings, and preventive actions for future reference.Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications. QualificationsBachelor's with 10 years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 7 years experience
Required Skills and Experience
Expert skills in Enterprise Security, ITSI, SOAR, and the Slunk product line.Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
Clearance Required: Must be able to obtain and maintain AOUSC Public Trust
Company Overview
GovCIO is a team of transformers--people who are passionate about transforming government IT. Every day, we make a positive impact by delivering innovative IT services and solutions that improve how government agencies operate and serve our citizens.
But we can't do it alone. We need great people to help us do great things - for our customers, our culture, and our ability to attract other great people. We are changing the face of government IT and building a workforce that fuels this mission. Are you ready to be a transformer?
What You Can Expect
Interview & Hiring Process
If you are selected to move forward through the process, here’s what you can expect:
During the Interview ProcessVirtual video interview conducted via video with the hiring manager and/or teamCamera must be onA valid photo ID must be presented during each interviewDuring the Hiring ProcessEnhanced Biometrics ID verification screeningBackground check, to include:Criminal history (past 7 years)Verification of your highest level of educationVerification of your employment history (past 7 years), based on information provided in your applicationEmployee Perks
At GovCIO, we consistently hear that meaningful work and a collaborative team environment are two of the top reasons our employees enjoy working here. In addition, our employees have access to a range of perks and benefits to support their personal and professional well-being, beyond the standard company offered health benefits, including:
Employee Assistance Program (EAP)Corporate DiscountsLearning & Development platform, to include certification preparation contentTraining, Education and Certification Assistance*Referral Bonus ProgramInternal Mobility ProgramPet InsuranceFlexible Work Environment*Available to full-time employees
Our employees’ unique talents and contributions are the driving force behind our success in supporting our customers, which ultimately fuels the success of our company. Join us and be a part of a culture that invests in its people and prioritizes continuous enhancement of the employee experience.
We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, disability, or status as a protected veteran. EOE, including disability/vets.
Posted Pay Range
The posted pay range, if referenced, reflects the range expected for this position at the commencement of employment, however, base pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, education, experience, and internal equity. The total compensation package for this position may also include other compensation elements, to be discussed during the hiring process. If hired, employee will be in an “at-will position” and the GovCIO reserves the right to modify base salary (as well as any other discretionary payment or compensation program) at any time, including for reasons related to individual performance, GovCIO or individual department/team performance, and market factors.
Posted Salary Range USD $105,000.00 - USD $145,000.00 /Yr. Options Apply for this job onlineApplySubmit a referral to this jobRefer Sorry the Share function is not working properly at this moment. Please refresh the page and try again later. Share on your newsfeed Application FAQsSoftware Powered by iCIMS
www.icims.com