Abnob Doss

Python Java Big Data DevOps

• Directed a 7-member team in benchmarking infrastructure, achieving a 200% boost in data processing efficiency.
• Piloted 'Spark-as-a-Service' to transition from Hive-on-HDFS to Iceberg-on-S3 tables.
• Integrated Prometheus for real-time infrastructure monitoring and designed Grafana dashboards. This strategic implementation resulted in a 50% reduction in monitoring efforts and significantly decreased hardware-caused outage durations.
• Led a team of 10 in architecting a highly configurable and versatile Java Spark ETL framework, paired with tools for automated HQL generation and Autosys JIL scheduling. This streamlined approach minimized code changes during ETL onboarding and facilitated the efficient integration of over 100 tables.
• Drove the expansion of our toolset to fully meet regulatory archival standards, leveraging the capabilities of S3 retention-enabled buckets.
• Directed the enhancement of a nearing-capacity 40+ TB Oracle database, fortifying it against impending data growth. Identified over 20 pivotal performance tweaks. Faced with challenges from a daily 100+ TB processing reporting engine and mounting SLA misses, I collaborated with big data vendors to devise strategic resolutions, overseeing several proof-of-concept endeavors.

Python Java Big Data DevOps

• Led a 15-member team in a strategic transition from traditional RDBMS to a big data platform, handling tens of terabytes monthly and reducing storage expansion costs by half.
• Achieved a 500% improvement in Spark ETL performance through Spark optimization, facilitating the swift integration of 5 downstream systems and 100+ feeds within a year, all while adhering to strict SLAs.
• Collaborated in the design and implementation of a machine learning model for T+1 materiality prediction, achieving over 70% accuracy and seamlessly integrating it into a live data view.

Python KDB+/q C/C++ Java DevOps

• Led Quality Assurance efforts for an ambitious greenfield project, successfully navigating the complex certification process for market data exchanges, focusing on high-volume energy products, and ensuring the reliability and integrity of data flows.
• Spearheaded DevOps enhancements to deploy market access/data across multiple North American exchanges, executed a seamless infrastructure migration from RHEL6 to RHEL7, and optimized server configurations for reduced latency using Solarflare OpenOnload and PTP.
• Collaborated with network teams to establish multicast connectivity for high-volume exchanges, optimizing server network paths, and deployed a multicast exchange simulator for new exchange market data certifications.
• Managed a KDB+ market data database, handling millions of daily ticks, maintaining 95% uptime, and providing traders with historical data, real-time insights, and vital indicator APIs.
• Collaborated with the trading desk to devise a Python-based machine learning model, forecasting energy product price movements using historical settle prices.
• Designed and implemented a full-stack web dashboard granting business users real-time control over LTA check values and kill switch status, powered by a Java-to-KDB backend and a Bootstrap/JavaScript frontend.