This part is delicate as several pieces from different services must fit together:
- The data structure, table declaration, and Kinesis Redshift configuration
- The data fields aggregation and subsequent parsing
- The role and associated policies
The fields of the Redshift table that stores the data need to be synchronized in three different places:
- The Redshift table with a proper definition of fields.
- The script that sends the data to Kinesis. Depending on how the record sent to Kinesis is aggregated together and later parsed by Redshift, the script must concatenate the same number of fields in the same order as the ones defined in the Redshift table. For instance, when we write record = ','.join([str(st.id) ...
