DataStage Administrator

JOB DESCRIPTION

DataStage Administrator (7+ years’ experience)

Location: Miami, FL (Hybrid)- Consultant should be in FL or willing to relocate from Day-1

Long Term

Project Overview:

Objective: To migrate an existing on-premises IBM InfoSphere DataStage environment to the cloud (AWS), ensuring seamless data integration, transformation, and ETL (Extract, Transform, Load) processes for telecom-related data.

Scope:

Assessment: Understand the current on-premises architecture, including the infrastructure, applications, and workflows in use.

Migration Planning: Plan the migration process, including identifying dependencies, data flows, and integration points, as well as ensuring minimal disruption to ongoing telecom operations.

Infrastructure Setup on AWS: Set up the necessary cloud infrastructure on AWS, such as EC2 instances, S3 buckets, RDS databases, Redshift, or other data services, depending on the needs of the telecom environment.

DataStage Migration: Migrate Datastage jobs, configurations, and scripts to AWS. This could involve either re-hosting (moving the current Datastage configuration) or re-platforming (optimizing the environment for AWS).

Testing & Validation: Test the new setup in AWS to ensure that all ETL processes work as expected and that performance and scalability are optimal.

Post-Migration Support: Ensure ongoing monitoring and support after migration to troubleshoot issues and fine-tune performance.

Roles & Responsibilities of a Datastage Administrator in this Migration Project:

Pre-Migration Tasks:

Conduct a comprehensive audit of the current Datastage setup.

Identify and document all data flows and dependencies, particularly those related to the telecom domain (e.g., call data records, billing data, network performance metrics).

Work with other stakeholders (e.g., network engineers, telecom domain experts) to ensure all use cases are understood.

Choose appropriate AWS services for Datastage components, such as:

EC2 for computing

S3 for data storage

RDS/Aurora or Redshift for databases and analytics

Lambda and Glue for serverless processing

Migration Planning:

Create a migration strategy that includes timelines, resource allocation, and risk mitigation steps.

Develop a cutover plan to switch from on-premises to AWS with minimal service disruption.

Identify the migration method: lift-and-shift vs. re-architecture.

Coordinate with telecom domain architects for alignment with business requirements (e.g., billing cycles, data consistency, SLAs).

Infrastructure Setup on AWS:

Provision AWS resources (e.g., EC2 instances with suitable specifications, storage in S3, RDS for databases).

Configure IAM roles for access control and security.

Set up AWS Direct Connect or VPN if private network connectivity is required between on-prem and cloud.

Implement data security and compliance measures in line with telecom industry regulations (e.g., GDPR, HIPAA).

Datastage Migration Execution:

Migrate Datastage ETL jobs, sequences, and configurations to the cloud.

Ensure that existing custom scripts and connectors work in the cloud environment.

Adjust job parallelism and resource allocation to leverage AWS scalability features like Auto Scaling.

Address any issues with file transfer protocols (e.g., FTP, SFTP) between on-prem and AWS.

Testing & Validation:

Unit testing and integration testing of Datastage jobs post-migration.

Verify data accuracy and completeness for telecom datasets.

Test data transformation rules and ensure that they function correctly in the AWS environment.

Conduct performance testing to verify that the new AWS infrastructure can handle the data loads and workloads typical of the telecom domain.

Post-Migration Support:

Provide ongoing monitoring of Datastage jobs and AWS resources using CloudWatch, Datadog, or similar.

Troubleshoot performance issues (e.g., memory, CPU usage) and optimize configurations.

Fine-tune resource allocation to ensure cost efficiency without sacrificing performance.

Work with the security team to ensure data protection and compliance.

Documentation:

Document the new cloud-based Datastage environment and any customizations made.

Update disaster recovery plans to account for cloud-based resources.

Training & Knowledge Transfer:

Conduct training sessions for the internal team on the new AWS environment, tools, and best practices.

Ensure that knowledge is transferred for long-term management of the Datastage system in the cloud.

Key Challenges to Consider:

Data Migration Complexity: Migrating large datasets and ensuring no data loss during the migration.

Performance Tuning: Optimizing Datastage jobs to work efficiently in the cloud environment.

Compatibility: Ensuring that on-prem Datastage jobs, especially those with custom scripts or legacy components, function correctly after migration.

Cost Optimization: Managing AWS costs effectively, especially as telecom data volumes can be significant.

Technologies/Tools You Might Use:

AWS Services: EC2, S3, RDS, Glue, Lambda, Redshift, Kinesis

Datastage (installed on AWS EC2 or via managed services)

Networking: AWS Direct Connect, VPN, VPC Peering

Monitoring: AWS CloudWatch, CloudTrail, Datadog

Security: AWS IAM, KMS, encryption, audit logging
Details Required:

Full Name:

Contact#:

Email ID:

Interview availability:

Availability to join:

Visa status:

Visa Validity:

Current rate/salary:

Expected rate/salary:

Employer details:

Location:

Willing to relocate:

LinkedIn URL:

Visa copy attached Yes/No:

DL copy attached Yes/no: