[Proposal] Mixed Full Refresh/Incremental sync that captures recent deletes

[LucasSegersFabro]

Hi!

It would be nice if I could:

• sync old data from a stream (let's say, starting from 2020-01-01 to now())
• on subsequent syncs, instead of reading all data again (because I can't know if records are deleted), I'd like to delete all destination data where some cursor field is >= $LOOKBACK_PERIOD_START_DATE (commonly known as {{ stream_interval.start_time }}) right before loading the data
• that would effectively capture deletes observed in the lookback window without having to scan the entire stream or lose old data

The only alternative we have today so that we don't miss deletes is to Full Refresh (effectively reading the whole stream everytime, or have a moving start_date)
Or Full Refresh Append which would complicate data usage

So far, only one person in slack showed interest for google analytics usecase

---

Idea 1: new "mixed" sync mode when Incremental Sync is enabled (native)

I'd like to configure:

• Which "cursor" column I'd like to "delete after"
• Lookback period (which is already an option on Incremental Syncs)

Idea 2: a "pre data commit" hook (hacky)

If I had some way to add logic or an api call right before airbyte tries to insert into the destination and send:

• the cursor column
• the destination stream name
• stream_interval.start_time

I could delete the records myself right before airbyte inserts it

[Luisgustavom1]

rt

[4ndreello]

nice

[pmarcio87]

rt