Example DAGs

This page contains a selection of example DAGs. These demonstrate some common use cases for the IQVIA NLP Airflow provider. The requirements for each DAG are listed in the DAG’s docstring. Feel free to copy or customize these DAGs to match your needs.

I2E

• i2e_query_without_transfers.py

  DAG showing how to use I2E operators to index a source data file in I2E and run a query against the index. This DAG doesn’t upload any source data to I2E, and doesn’t copy the query results to anywhere outside I2E.

• i2e_query.py

  DAG showing how to use I2E operators to upload source data from a local file to I2E, index the source data, run a query against the index, write the query results to a local file, and delete all transient resources from I2E.

• i2e_query_with_multiple_queries.py

  DAG showing how to use I2E operators to upload source data from a local file to I2E, index the source data, run two queries against the index, write the query results to local files, and delete all transient resources from I2E.

• i2e_query_with_i2e_hook.py

  DAG showing how to use the I2EHook directly to upload source data from a local file to I2E, index the source data, run a query against the index, write the query results to a local file, and delete all transient resources from I2E.

• i2e_resource_bundles.py

  DAG showing how to use the S3ToI2EResourceBundleOperator and LocalFilesystemToI2EResourceBundleOperator to retrieve an I2E resource bundle from S3 or a local file, and install the bundle on multiple I2E servers.

• i2e_resource_deletion.py

  DAG showing how to use the I2EDeleteResourcesOperator to delete all source data and indexes from a particular I2E folder.

SQL

• sql_query_transfer.py

  DAG showing how to use the SQLQueryToI2EOperator and I2EToSQLOperator to upload data from an SQL database to I2E, and to write the query results to an SQL database.

• sql_timespan_transfer.py

  DAG showing how to use the SQLTimestampToI2EOperator and I2EToSQLOperator to periodically upload recent data (from the last hour) from an SQL database to I2E, and to write the query results to an SQL database.

Amazon S3

• s3_transfer.py

  DAG showing how to use the S3ToI2EOperator and I2EToS3Operator to upload data from an S3 bucket to I2E, and to write the query results to an S3 bucket.

• s3_timespan_transfer.py

  DAG showing how to use the S3ToI2EOperator and I2EToS3Operator to periodically upload recent data (from the last hour) from an S3 bucket to I2E, and to write the query results to an S3 bucket.

Legacy NLP Data Factory

• i2e_legacy_webhooks.py

  DAG showing how to interact with legacy NLP Data Factory webhooks.