Apache Spark Developer

6+ years

Sponsorship: U.S. Citizens, Green Card Holders, EAD Holders, and H-1B transfer candidates are encouraged to apply. We are unable to sponsor new H-1B visa petitions for this position.

• Experience building enterprise Lakehouse architectures using Databricks or Delta Lake.
• Familiarity with Apache Airflow, Azure Data Factory, AWS Step Functions, or Control-M for workflow orchestration.
• Experience with machine learning workflows using Spark MLlib, MLflow, or feature engineering pipelines.
• Knowledge of Kubernetes, Docker, and containerized Spark deployments.
• Experience implementing Data Quality frameworks using Great Expectations or Deequ.
• Familiarity with Apache NiFi, Apache Flink, Trino, or Presto.
• Experience working with cloud object storage including Amazon S3, Azure Data Lake Storage (ADLS Gen2), or Google Cloud Storage.
• Knowledge of Infrastructure as Code using Terraform or ARM templates.
• Experience with enterprise monitoring tools including Prometheus, Grafana, Datadog, or OpenTelemetry.
• Cloud certifications in Azure, AWS, Databricks, or Apache Spark-related technologies are highly desirable.

Project Environment

You will be joining a modern data engineering team responsible for building cloud-native big data platforms supporting enterprise analytics, AI, and business intelligence initiatives. Current projects include:

• Enterprise data lakehouse implementation using Databricks and Delta Lake
• Real-time streaming analytics processing billions of daily events
• Large-scale customer analytics and behavioral data platforms
• Financial risk modeling and fraud detection pipelines
• Healthcare clinical and operational analytics solutions
• Cloud migration of legacy Hadoop and ETL workloads
• Machine learning feature engineering and model training pipelines
• Enterprise reporting platforms supporting executive dashboards and self-service analytics
• Distributed data processing infrastructure deployed on Azure and AWS

This is a hands-on engineering role where you will contribute to distributed system architecture, Spark application development, cloud migration, performance optimization, production support, and continuous improvement of enterprise-scale data processing platforms.

• Six or more years of professional software or data engineering experience.
• Four or more years of hands-on Apache Spark development experience in enterprise production environments.
• Strong proficiency in PySpark, Scala, or Spark SQL for distributed data processing.
• Deep understanding of Apache Spark architecture including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
• Strong experience with distributed computing concepts including partitioning, shuffling, caching, broadcast joins, and fault tolerance.
• Advanced SQL skills with databases such as SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
• Experience working with Hadoop ecosystem technologies including Hive, HDFS, YARN, and Parquet.
• Experience processing streaming data using Spark Structured Streaming, Apache Kafka, or Event Hubs.
• Hands-on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse Analytics, or Google Dataproc.
• Experience integrating Spark applications with Delta Lake, Apache Iceberg, or Apache Hudi.
• Strong understanding of data warehousing concepts, dimensional modeling, and data lake architecture.
• Experience using Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
• Strong debugging, troubleshooting, and Spark performance tuning skills.
• Experience working in Agile Scrum development environments.