Systems Engineer/Senior Data Engineer - Splunk, ServiceNow & AppDynamics

***Security Clearance: Must possess an active TS/SCI with Full Scope Polygraph*** KDA is seeking a highly skilled and experienced Systems Engineer/Senior Data Engineer with a strong background in designing, implementing, and optimizing data pipelines and solutions for critical security and operational platforms, specifically Splunk, ServiceNow, and AppDynamics. The ideal candidate will be a proactive, problem-solving individual with a proven track record of managing complex data ingestion, normalization, and correlation processes within highly visible, operationally driven environments. This role requires an individual who thrives in a collaborative setting, possesses excellent communication skills, and is dedicated to continuous improvement and innovation. Key Responsibilities • Splunk Data Engineering: • Design, engineer, and maintain robust Splunk infrastructures, including clustered environments, for large-scale data ingestion, correlation, and reporting. • Automate complex data ingestion methods (e.g., S3, syslog, JSON, APIs) from diverse sources across multiple enclaves. • Develop and implement methods for data tagging and cataloging to ensure compliance with evolving security standards and facilitate efficient data discovery. • Optimize data ingest performance and efficiency across various network environments. Familiar with Technical Add-ons. • Parse and normalize non-standard data sets to enable comprehensive analysis and correlation within Splunk. • Develop and refine Splunk queries, dashboards, and reports to visualize security events, infrastructure health, and operational metrics. • Collaborate with IT operations and cyber security teams to enrich data sets, ascertain cyber threats, and bolster security posture. • Maintain ITSI and SIEM-like tools and custom content within virtualized environments. • Perform tuning and filtering of events and information, creating custom views and content. • Familiar with UBA and Splunk. • Collaborate with cross-functional teams to design and implement data integrations between various security and operational tools (including Splunk and AppDynamics) and ServiceNow. • Develop and maintain data pipelines to ensure accurate and timely flow of security incidents, alerts, and operational metrics into ServiceNow for incident management, problem management, and reporting. • Assist in defining and implementing data models within ServiceNow to support security operations and compliance initiatives. • Work with third-party services for design review and optimal deployment configuration for enterprise cloud service utilization (relevant to integrations). • Design and implement data collection strategies for AppDynamics, ensuring comprehensive monitoring of application performance and infrastructure. • Integrate AppDynamics data with Splunk for centralized visibility and correlation with other security and operational logs. • Troubleshoot problematic service deployments and data flows, utilizing forensic tools and audit log review (relevant to monitoring and analysis). • Develop methods to leverage AppDynamics data for identifying potential risks and optimizing application performance. General Data Engineering & Systems Expertise: • Collaborate with mission partners to develop long-term enterprise audit solutions and normalize non-standard data sets. • Engineer and maintain secure virtualized and cloud environments for data platforms. • Deploy and harden servers running Linux OS in accordance with CIS and other STIG guidelines. • Develop runbooks, SOPs, and documentation for new processes and systems. • Perform liaison duties between service providers and clients to bridge communication gaps and ensure adherence to SLAs. • Review and evaluate data integrity and develop use cases for various data sets. • Maintain system baselines and configuration management for data engineering tools. • Contribute to the development of plans to safeguard data against unauthorized modification, destruction, or disclosure. • Strong understanding of cyber security principles and experience with various security tools (e.g., Next-Gen Firewalls, IPS/IDS, Tenable Nessus, Rapid7 Nexpose, McAfee EPO, Symantec SEP). Required Skills & Experience • 20+ years of progressive experience in Information Technology and Security, with a strong focus on data engineering and systems integration. • Security Clearance: Candidate must possess an active TS/SCI with Full Scope Polygraph • Demonstrated expertise in engineering and maintaining large-scale Splunk environments, including data ingestion, parsing, normalization, and content development. • Experience with automating complex data ingestion methods (e.g., S3, syslog, JSON, APIs). • Strong understanding of data tagging, cataloging, and data governance best practices. • Proficiency with Linux OS administration and hardening. • Familiarity with cloud security principles and deploying commercial services into protected/secured enclaves (e.g., AWS). • Experience with SIEM solutions and their implementation, configuration, and maintenance. • Strong scripting skills (e.g., BASH, Python, PowerShell). • Excellent collaboration and communication skills, with the ability to work effectively in small teams and large collaborative efforts. • Ability to troubleshoot complex technical issues and perform root cause analysis. • Proven ability to develop and maintain documentation (runbooks, SOPs). Desired Skills (Plus, But Not Required) • Experience with ServiceNow platform administration, development, or integration. • Experience with AppDynamics for application performance monitoring and data collection. • Experience with configuration management tools - Git, Ansible