ETL Performance Optimization and Evaluation of PostgreSQL vs MS SQL as a Data Warehouse Platform
Budget: -
HOURLY / PART_TIME
⭐ 0.00 (0)
Germany
pyspark, data-extraction, microsoft-power-bi, sql-programming, ms-sql, ssis, data-analysis, scrapy-framework, selenium-webdriver, databricks-platform, microsoft-power-bi-data-visualization, selenium, data-scraping
Qualifiche preferite
- Esperienza: Intermedio
We have built a data warehouse based on Microsoft SQL Server 2022 with 16 CPU cores and 256 GB of RAM.
The databases have a total size of approximately 4 TB, distributed across 36 databases. Some of these databases are quite large (up to 1.1 TB), while others are considerably smaller.
The data is physically stored on NVMe drives connected via 10 Gbit networking.
We use ETL jobs developed in Visual Studio, which retrieve data on a daily basis from various source systems, including Microsoft SQL Server, PostgreSQL, Oracle, and MySQL.
We would like to discuss the following topics:
Optimization of ETL Performance
The performance of the implemented ETL processes is, in some cases, not sufficient for our requirements. We would therefore like to explore options for accelerating the data loading processes and are looking for expert support in analyzing and optimizing the current setup.
Evaluation of a Migration from Microsoft SQL Server to PostgreSQL
In addition, we would like to assess the feasibility of migrating our data warehouse from Microsoft SQL Server to PostgreSQL (combined with python jobs). We are interested in understanding the available approaches for high-performance data loading into a PostgreSQL-based data warehouse, as well as the advantages and disadvantages of such a migration.
Should you feel confident in supporting us with either or both of these topics, we would be pleased to hear from you.
Apri su Upwork
AI proposal draft
Generate a short cover letter for this job. Edit before sending.
Sign in to generate an AI proposal draft.
Accedi