Python Software Developer
Full time - Lisbon
I was part of the back end team in the company AMA, in the development of the Portal Mais Transparência as a Python Developer.
Using Python, Pandas and Apache Airflow and taking advantage of existing code, I increased the automation capabilities for collecting data from different entities, validating the collected data and loading it into databases in different development environments.
This entire process was automated, with the different steps orchestrated through Apache Airflow.
I refactored and standardized the existing code for the entire EVTL process, for all the different themes on the site and facilitated the entry of new thematic areas.
With Python and SQL I created the necessary data validations before loading into the final tables, allowing us to confirm that the data we received from the entities is correct and presented in the way we need on the Portal.
Essentially, I was responsible for the entire EVTL process (Extraction, Validation, Transformation and Loading).
Using the Flyway application, I contributed to changing the procedures and work routines, which allowed us to start keeping a history with Git of all the SQL scripts needed for the four different environments, and also facilitate the recreation of the structure of a SQL database from scratch.
Because of the large amount of Excel files made available by different entities and because we need updated these files on the computers of different team members, I created a desktop application with Python and Tkinter, so that the entire team can update the necessary files in a simple way.
This application creates on the user's computer a structure of folders separated by entity providing the data, downloads the files for each entity from the data source API and organizes the files in their respective folders, excluding the oldest ones.
Technologies: Python, Django, PostgreSql, Apache Airflow, Flyway.
