This repository has a collection of utilities for Glue Crawlers. These utilities come in the form of AWS CloudFormation templates or AWS CDK applications.
-
Updated
Dec 21, 2021 - Python
#
aws-glue-crawler
Here are 36 public repositories matching this topic...
This repository contains source code for the AWS Database Blog Post Reduce data archiving costs for compliance by automating RDS snapshot exports to Amazon S3
aws-lambda aws-kms aws-cloudformation amazon-rds amazon-sns amazon-s3 amazon-athena aws-backup aws-glue amazon-eventbridge aws-glue-crawler
-
Updated
Apr 26, 2023
ETL Data pipeline using aws services
-
Updated
Aug 23, 2024 - Python
Terraform configuration that creates several AWS services, uploads data in S3 and starts the Glue Crawler and Glue Job.
-
Updated
Feb 10, 2022 - Python
Automation framework to catalog AWS data sources using Glue
aws typescript aws-s3 dynamodb glue python3 data-catalog rds gdpr pii data-governance aws-cdk aws-glue-workflow aws-glue-crawler aws-glue-data-catalog
-
Updated
May 24, 2024 - Python
AWS Athena, Glue Database, Glue Crawler and S3 buckets deployment through AWS GUI console.
-
Updated
Dec 13, 2022
In this project I have used the Trending YouTube Video Statistics data from Kaggle to analyze and prepare it for usage.
-
Updated
Nov 7, 2022
Unveiling job market trends with Scrapy and AWS
-
Updated
Apr 5, 2024 - Python
Creating an audit table for a DynamoDB table using CloudTrail, Kinesis Data Stream, Lambda, S3, Glue and Athena and CloudFormation
aws-python-lambda aws-iam aws-cloudformation aws-cloudtrail aws-cloudwatch aws-athena aws-cloudwatch-logs aws-kinesis-stream aws-glue-crawler aws-iam-roles aws-iam-policies aws-s3-bucket aws-glue-data-catalog
-
Updated
Jul 6, 2023 - Python
An end-to-end data pipeline built with AWS S3, Glue, Crawler, Athena, Tableau visulization
-
Updated
Mar 27, 2024 - Jupyter Notebook
It is a project build using ETL(Extract, Transform, Load) pipeline using Spotify API on AWS.
python aws aws-lambda aws-s3 spotify-api data-engineering aws-athena data-engineering-pipeline spotipy-library aws-glue-crawler awscloudwatch aws-glue-data-catalog aws-trigger
-
Updated
Dec 3, 2024 - Jupyter Notebook
Analyzed a multicategory e-commerce store using big data techniques on a Kaggle dataset with the help of AWS EC2, AWS S3, PySpark, AWS Glue ETL, AWS Athena, AWS CloudFormation, AWS Lambda and Power BI!
aws big-data aws-lambda power-bi pyspark aws-ec2 aws-cloudformation aws-athena kaggle-dataset aws-services end-to-end-pipeline end-to-end-project aws-glue-crawler aws-s3-bucket
-
Updated
Sep 7, 2024 - Python
AWS Athena, Glue Database, Glue Crawler and S3 buckets deployment through CloudFormation stack on AWS console.
-
Updated
Dec 14, 2022
An end-to-end solution for managing and analyzing YouTube video data from Kaggle, leveraging AWS services and visualized through Quicksight and Tableau
aws marketing youtube aws-lambda aws-s3 youtube-api aws-iam tableau content-strategy aws-athena aws-lambda-python aws-glue quicksight aws-glue-crawler user-insights
-
Updated
Sep 23, 2023 - Python
Implementing data pipeline using AWS services for airlines data
python aws aws-s3 aws-sns aws-redshift step-functions aws-eventbridge aws-glue-workflow aws-glue-crawler
-
Updated
Oct 15, 2024 - Python
Working with Glue Data Catalog and Running the Glue Crawler On Demand
-
Updated
May 11, 2023
Smart City Realtime Data Engineering Project
python aws kafka aws-s3 pyspark spark-streaming aws-ec2 aws-athena aws-redshift aws-glue aws-quicksight aws-glue-crawler aws-glue-data-catalog
-
Updated
May 24, 2024 - Python
The Project aims to establish a robust data pipeline for tracking and analyzing sales performance using various AWS services. The process involves creating a DynamoDB database, implementing Change Data Capture (CDC), utilizing Kinesis streams, and finally, storing and querying the data in Amazon Athena.
python aws-lambda dynamodb s3-bucket kinesis kinesis-firehose aws-athena glue-catalog aws-glue-crawler eventbridge-pipes
-
Updated
Feb 11, 2024 - Python
A pipeline within AWS to capture schema changes in S3 files and to update them in a DB.
aws crawler aws-lambda dynamodb s3 aws-dynamodb aws-cloudwatch-logs aws-lambda-python aws-glue aws-eventbridge glue-catalog aws-glue-crawler
-
Updated
Nov 30, 2021
Este projeto tem como objetivo realizar a coleta, catalogo, governança, processamento e visualização de dados.
-
Updated
Mar 28, 2023
Improve this page
Add a description, image, and links to the aws-glue-crawler topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the aws-glue-crawler topic, visit your repo's landing page and select "manage topics."