Build A StreamSets Pipeline In 5 Minutes

Demonstration on building a realtime data pipeline using Streamsets

create pipeline
name pipeline
empty data pipeline interface
add origin JDBC Query Consumer
JDBC configuration interface
add processor Record Deduplicator
fill in column names in “Fields to Compare” with “Specified Fields” selected under “Compare”
select “All Fields” under “Compare”
allows to convert the preselected column
process every fields and convert those fields with the preselected source type
add destination Kinese Firehose
add in AWS firehose access key and secret and firehose streams name
realtime-time data ingestion and transformation statistics

Writing articles to test own knowledge depth

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store