Python API¶

Operators¶

class iqvia_nlp_provider.operators.i2e.I2EMakeIndexOperator(i2e_conn_id: str, i2e_source_data_uri: str, i2e_index_template: str, i2e_index_settings_overrides: dict[str, Any] | None = None, i2e_folder: str | None = None, i2e_wait_timeout: int = 300, **kwargs: Any)¶

Operator that builds an index in I2E.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
i2e_source_data_uri – URI of the source data in I2E to be indexed.
i2e_index_template – Name (not the full URI) of the index template to use.
i2e_index_settings_overrides – Dictionary of settings that will overwrite settings from the i2e_index_template provided.
i2e_folder – If provided, the index will be created in a folder with this name.
i2e_wait_timeout – The maximum time to wait for an I2E task to finish before raising a tenacity.RetryError. It is recommended to increase this value if you are working with large amounts of data.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Returns:

URI of the created index.

Return type:

str

class iqvia_nlp_provider.operators.i2e.I2ERunQueryOperator(i2e_conn_id: str, i2e_index_uri: str, i2e_query_template: str, i2e_query_settings_overrides: dict[str, Any] | None = None, i2e_wait_timeout: int = 60, **kwargs: Any)¶

Operator that runs an I2E query.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
i2e_index_uri – URI of the index in I2E to be queried.
i2e_query_template – Name (not the full URI) of the query template to use.
i2e_query_settings_overrides – Dictionary of settings that will overwrite settings from the i2e_query_template provided.
i2e_wait_timeout – The maximum time to wait for an I2E task to finish before raising a tenacity.RetryError. It is recommended to increase this value if you have a complex I2E query which may take longer to complete.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Returns:

URI of the query results.

Return type:

str

class iqvia_nlp_provider.operators.i2e.I2EPickConnectionIDOperator(strategy: Strategy = Strategy.RANDOM, choices: list[str] | None = None, **kwargs: Any)¶

Operator that picks an I2E connection ID based on a given strategy.

Parameters:

strategy – The strategy to use when picking an I2E connection ID. See Strategy enum for available options.
choices – List of I2E connection IDs to choose from. If not provided, all I2E connection IDs will be used.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Returns:

The chosen I2E connection ID.

Return type:

str

class iqvia_nlp_provider.operators.i2e.I2EDeleteResourcesOperator(i2e_conn_id: str, i2e_resource_uris: list[str] | None = None, i2e_resource_types: list[str] | None = None, i2e_folder: str | None = None, **kwargs: Any)¶

Operator that deletes one or more I2E resources.

There are two different methods for specifying which resources should be deleted:

If i2e_resource_uris is provided then those specific resources will be deleted.
If i2e_resource_types and i2e_folder are provided, then all resources with a matching resource type, in the given I2E folder, will be deleted.

One or both of these methods can be used. For example, it is valid to provide only i2e_resource_uris, or to provide only i2e_resource_types and i2e_folder, or to provide all of the above.

This operator overrides Airflow’s default values for trigger_rule and post_execute. Since this operator may be responsible for deleting sensitive data from I2E, trigger_rule is set to "all_done" to ensure the task always runs regardless of the state of upstream tasks. This operator’s post_execute value ensures that the task will only be marked as successful if all upstream tasks are successful, to prevent upstream failures from being hidden by a successful “cleanup” task. If you’d like to revert back to Airflow’s default behavior, set trigger_rule="all_success" and post_execute=None.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
i2e_resource_uris – A list of URIs of resources in I2E to be deleted. Any None values in this list will be ignored.
i2e_resource_types – All resources in I2E that have a resource type in i2e_resource_types, and are in the i2e_folder folder, will be deleted. Some examples of resource types are “source_data” and “index”. If this argument is provided, then the i2e_folder argument must also be provided.
i2e_folder – All resources in I2E that have a resource type in i2e_resource_types, and are in the i2e_folder folder, will be deleted. If this argument is provided, then the i2e_resource_types argument must also be provided.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

NoneType

class iqvia_nlp_provider.operators.hart.HARTModelsHubPredictionOperator(hart_conn_id: str, models_hub_conn_id: str, review_id: str, doc_ids: str, model_id: str, target_name: str, create_project: bool, hyper_params: PredictionHyperParams, **kwargs: Any)¶

Run model inference on HART documents using a model in Models Hub, and import the results into HART.

Parameters:

hart_conn_id – The connection ID for HART.
models_hub_conn_id – The connection ID for Models Hub.
review_id – The review ID of the HART project.
doc_ids – The IDs of the HART documents.
model_id – The ID of the model in Models Hub.
target_name – The name of the column that will contain the target labels.
create_project – Whether to import model results into a new HART project.
hyper_params – Model prediction hyper params. See PredictionHyperParams for configurable parameters.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

NoneType

class iqvia_nlp_provider.operators.hart.HARTModelsHubFineTuneOperator(models_hub_conn_id: str, hart_conn_id: str, review_id: str, since: str, column_id: str, model_id: str, base_model: str, logical_column_id: str | None, hyper_params: FineTuningHyperParams, labels: List[str], target_name: str, **kwargs: Any)¶

Fetch curated assertion rows from a HART project and use these to fine tune a model in Models Hub.

Parameters:

models_hub_conn_id – The connection ID for Models Hub.
hart_conn_id – The connection ID for HART.
review_id – The review ID of the HART project.
since – The earliest “updated” date of assertion rows to return.
column_id – The ID of the target column.
model_id – The ID of the model in Models Hub.
base_model – The name of the base model to be fine tuned.
logical_column_id – The logical column id of the target column.
hyper_params – Model fine tuning hyper params. See FineTuningHyperParams for configurable parameters.
labels – An array of target labels.
target_name – The name of the column containing the target labels.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Returns:

Fine tuning run results.

Return type:

FineTuningResponse

Transfer operators¶

Transferring source data¶

class iqvia_nlp_provider.transfers.s3.S3ToI2EOperator(aws_conn_id: str, i2e_conn_id: str, aws_bucket_name: str, aws_key: str | None = None, aws_prefix: str | None = None, select_data_added_between: tuple[datetime, datetime] | None = None, i2e_filename: str | None = None, i2e_folder: str | None = None, legacy_before_hook: Webhook | None = None, **kwargs: Any)¶

Operator that retrieves a single file or multiple files from an S3 bucket, and uploads the files to I2E as source data. Multiple files are combined into a zip file.

Parameters:

aws_conn_id – The connection ID used to connect to Amazon S3. The connection type must be an Amazon Web Services Connection type, as described here: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html
i2e_conn_id – The connection ID used to connect to the I2E server.
aws_bucket_name – The S3 bucket name.
aws_key – The S3 key path to a single source data file. If provided, only this file will be selected. Mutually exclusive with aws_prefix. Omit both this and aws_prefix to select all files in the bucket.
aws_prefix – The S3 prefix to select multiple source data files. If provided, all files whose keys begin with this prefix will be selected. Mutually exclusive with aws_key. Omit both this and aws_key to select all files in the bucket.
select_data_added_between – A tuple of two datetime objects representing the start and end dates between which to retrieve source data from a given aws_prefix. If None, then data will be retrieved regardless of date. Otherwise, only data created between these dates will be retrieved and uploaded to I2E. The timespan includes the end date but does not include the start date. For example, a value like (1:00, 2:00) would include data created at exactly 2:00, but not data created at exactly 1:00.
i2e_filename – The name of source data file that will be uploaded to I2E.
i2e_folder – If provided, the source data will be uploaded to this folder in I2E.
legacy_before_hook – Legacy NLP Data Factory before hook configuration. If provided, the source data will be sent to the legacy NLP Data Factory before hook endpoint before being uploaded to I2E.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Raises:

airflow.sdk.exceptions.AirflowSkipException – If no matching files are found in S3.

Returns:

URI of the uploaded source data.

Return type:

str

class iqvia_nlp_provider.transfers.sql.SQLQueryToI2EOperator(sql_conn_id: str, i2e_conn_id: str, sql_query: str, sql_params: dict[str, str] | None = None, exclude_columns: list[str] | None = None, i2e_filename: str = 'source.xml', i2e_folder: str | None = None, legacy_before_hook: Webhook | None = None, **kwargs: Any)¶

Operator that retrieves data from an SQL table by executing an SQL query, and uploads the retrieved data to I2E as source data.

Parameters:

sql_conn_id – The connection ID used to connect to the source SQL database. This can be any connection type accepted by the apache-airflow-providers-common-sql provider, as listed here: https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/supported-database-types.html
i2e_conn_id – The connection ID used to connect to the I2E server.
sql_query – The SQL query to be executed. This must be a SELECT query.
sql_params – The parameters to be substituted into the SQL query at query runtime. Use this field for any user-provided to protect against SQL injection.
exclude_columns – A list of columns to exclude from the uploaded source data.
i2e_filename – The name of source data file that will be uploaded to I2E.
i2e_folder – If provided, the source data will be uploaded to this folder in I2E.
legacy_before_hook – Legacy NLP Data Factory before hook configuration. If provided, the source data will be sent to the legacy NLP Data Factory before hook endpoint before being uploaded to I2E.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Raises:

airflow.sdk.exceptions.AirflowSkipException – If no rows are returned by the SQL query.

Returns:

URI of the uploaded source data.

Return type:

str

class iqvia_nlp_provider.transfers.sql.SQLTimestampToI2EOperator(sql_conn_id: str, i2e_conn_id: str, sql_table: str, sql_timestamp_column: str | None = None, select_data_added_between: tuple[datetime, datetime] | None = None, exclude_timestamp_column: bool = True, i2e_filename: str = 'source.xml', i2e_folder: str | None = None, legacy_before_hook: Webhook | None = None, **kwargs: Any)¶

Operator that retrieves data from an SQL table, based on the value of a timestamp column, and uploads the retrieved data to I2E as source data.

Parameters:

sql_conn_id – The connection ID used to connect to the source SQL database. This can be any connection type accepted by the apache-airflow-providers-common-sql provider, as listed here: https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/supported-database-types.html
i2e_conn_id – The connection ID used to connect to the I2E server.
sql_table – The name of the database table containing the source data. When using a Snowflake connection, you must pass your table name in capital letters.
sql_timestamp_column – The name of the column in the source table that contains the creation timestamp of the row of data in Coordinated Universal Time (UTC). The column must be in a format that can store both a date and time (e.g. timestamp in PostgreSQL/MySQL/MariaDB, or datetime in Microsoft SQL Server). When using a Snowflake connection, you must pass your column name in capital letters.
select_data_added_between – A tuple of two datetime objects representing the start and end dates between which to retrieve source data. If None, then all data in the source table will be retrieved. Otherwise, only data created between these dates will be retrieved and uploaded to I2E. The timespan includes the end date but does not include the start date. For example, a value like (1:00, 2:00) would include data created at exactly 2:00, but not data created at exactly 1:00.
exclude_timestamp_column – Whether to exclude the timestamp column from the uploaded source data.
i2e_filename – The name of source data file that will be uploaded to I2E.
i2e_folder – If provided, the source data will be uploaded to this folder in I2E.
legacy_before_hook – Legacy NLP Data Factory before hook configuration. If provided, the source data will be sent to the legacy NLP Data Factory before hook endpoint before being uploaded to I2E.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Raises:

airflow.sdk.exceptions.AirflowSkipException – If no matching rows are found in the source table.

Returns:

URI of the uploaded source data.

Return type:

str

class iqvia_nlp_provider.transfers.sql.SQLToI2EOperator(sql_conn_id: str, i2e_conn_id: str, sql_table: str, sql_timestamp_column: str | None = None, select_data_added_between: tuple[datetime, datetime] | None = None, exclude_timestamp_column: bool = True, i2e_filename: str = 'source.xml', i2e_folder: str | None = None, legacy_before_hook: Webhook | None = None, **kwargs: Any)¶: Alias for SQLTimestampToI2EOperator, provided for backwards compatibility.

class iqvia_nlp_provider.transfers.local.LocalFilesystemToI2EOperator(i2e_conn_id: str, local_file_path: str, i2e_folder: str | None = None, i2e_filename: str | None = None, legacy_before_hook: Webhook | None = None, **kwargs: Any)¶

Operator that retrieves a local file and uploads it to I2E as source data. The local file must be accessible from the Airflow worker.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
local_file_path – Path to the source data file on the local filesystem.
i2e_folder – If provided, the source data will be uploaded to this folder in I2E.
i2e_filename – The name of source data file that will be uploaded to I2E.
legacy_before_hook – Legacy NLP Data Factory before hook configuration. If provided, the source data will be sent to the legacy NLP Data Factory before hook endpoint before being uploaded to I2E.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Returns:

URI of the uploaded source data.

Return type:

str

Transferring query results¶

class iqvia_nlp_provider.transfers.s3.I2EToS3Operator(i2e_conn_id: str, aws_conn_id: str, i2e_query_results_uri: str, aws_bucket_name: str, aws_prefix: str | None = None, aws_filename: str | None = None, s3_overwrite: bool = False, legacy_after_hook: Webhook | None = None, i2e_source_data_uri: str | None = None, **kwargs: Any)¶

Operator that reads query results from I2E and uploads them to an S3 bucket.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
aws_conn_id – The connection ID used to connect to AWS S3. The connection type must be an Amazon Web Services Connection type, as described here: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html
i2e_query_results_uri – URI of the query results in I2E.
aws_bucket_name – The S3 bucket name.
aws_prefix – The key prefix for storing the file in S3. If provided, this will be prepended to aws_filename with a slash (/).
aws_filename – The name to use for the file that is uploaded to S3. Defaults to the query results filename generated by I2E. Only simple file name is allowed. If path-like value provided that raises an error.
s3_overwrite – The boolean flag which when set to True overrides destination key if it exists. If set to False (which is default) a ValueError is raised if key already exists.
legacy_after_hook – Legacy NLP Data Factory after hook configuration. If provided, the query results will be sent to the legacy NLP Data Factory after hook endpoint before being uploaded to the AWS S3 bucket. When using this argument, the i2e_source_data_uri argument must also be provided.
i2e_source_data_uri – The URI of the source data on the I2E server. This argument is required if legacy_after_hook is provided.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

None

class iqvia_nlp_provider.transfers.sql.I2EToSQLOperator(i2e_conn_id: str, sql_conn_id: str, i2e_query_results_uri: str, sql_table: str, schema_mapping: dict[str, str] | None = None, use_mapped_columns_only: bool = False, sql_batch_size: int = 1000, i2e_results_delimiter: str = '\t', legacy_after_hook: Webhook | None = None, i2e_source_data_uri: str | None = None, sql_audit_columns: dict[str, Any] | None = None, **kwargs: Any)¶

Operator that reads query results from I2E and writes them to an SQL table. Query results must be in a delimiter-separated file format, such as TSV or CSV.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
sql_conn_id – The connection ID used to connect to the target SQL database. This can be any connection type accepted by the apache-airflow-providers-common-sql provider, as listed here: https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/supported-database-types.html
i2e_query_results_uri – URI of the query results in I2E.
sql_table – The name of the database table to write the query results to. When using a Snowflake connection, you must pass your table name in capital letters.
schema_mapping – A dictionary that maps the I2E query result columns to the target table columns, where key is the I2E result column name and value is the corresponding database column. Defaults to an empty dictionary, meaning all result columns are mapped to an auto-generated column name in the target table.
use_mapped_columns_only – Whether to only use the mapped columns in the target table. If set to True, any I2E query result columns that do not have a mapping in the schema_mapping will be omitted in the table.
sql_batch_size – The number of rows to write to the SQL table at a time.
i2e_results_delimiter – The delimiter used in the I2E query results.
legacy_after_hook – Legacy NLP Data Factory after hook configuration. If provided, the query results will be sent to the legacy NLP Data Factory after hook endpoint before being written to the SQL table. When using this argument, the i2e_source_data_uri argument must also be provided.
i2e_source_data_uri – The URI of the source data on the I2E server. This argument is required if legacy_after_hook is provided.
sql_audit_columns – Optional dictionary of extra column names and values to append to every inserted row. Keys are the target SQL column names; values are written as-is for each row. This parameter is templateable, so values can reference Airflow context (for example, {"RUN_ID": "{{ ti.xcom_pull(task_ids='get_unique_id') }}"}). The target SQL table must already contain these columns. Defaults to an empty dictionary.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

NoneType

class iqvia_nlp_provider.transfers.local.I2EToLocalFilesystemOperator(i2e_conn_id: str, i2e_query_results_uri: str, local_folder: str, local_filename: str | None = None, legacy_after_hook: Webhook | None = None, i2e_source_data_uri: str | None = None, **kwargs: Any)¶

Operator that reads query results from I2E and writes them to a local file on the Airflow worker.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
i2e_query_results_uri – URI of the query results in I2E.
local_folder – The folder to write the local file to.
local_filename – The name to use for the local file. Defaults to the query results filename generated by I2E.
legacy_after_hook – Legacy NLP Data Factory after hook configuration. If provided, the query results will be sent to the legacy NLP Data Factory after hook endpoint before being written to the local file. When using this argument, the i2e_source_data_uri argument must also be provided.
i2e_source_data_uri – The URI of the source data on the I2E server. This argument is required if legacy_after_hook is provided.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

NoneType

Transferring resource bundles¶

class iqvia_nlp_provider.transfers.s3.S3ToI2EResourceBundleOperator(aws_conn_id: str, i2e_conn_ids: list[str], aws_bucket_name: str, aws_key: str, i2e_wait_timeout: int = 60, **kwargs: Any)¶

Operator that retrieves an I2E resource bundle stored in S3, and installs the bundle across multiple I2E servers. This operator will wait for the bundle to be installed on all I2E servers before returning.

Parameters:

aws_conn_id – The connection ID used to connect to Amazon S3. The connection type must be an Amazon Web Services Connection type, as described here: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html
i2e_conn_ids – The list of connection IDs used to connect to the I2E servers. The bundle will be installed on every I2E server in this list.
aws_bucket_name – The S3 bucket name.
aws_key – The key path in S3 where the bundle will be retrieved from.
i2e_wait_timeout – The maximum time to wait for an I2E task to finish before raising an AirflowException. It is recommended to increase this value if you are working with large amounts of data.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

NoneType

class iqvia_nlp_provider.transfers.local.LocalFilesystemToI2EResourceBundleOperator(i2e_conn_ids: list[str], local_file_path: str, i2e_wait_timeout: int = 60, **kwargs: Any)¶

Operator that retrieves an I2E resource bundle stored in a local file, and installs the bundle across multiple I2E servers. The local file must be accessible from the Airflow worker. This operator will wait for the bundle to be installed on all I2E servers before returning.

Parameters:

i2e_conn_ids – The list of connection IDs used to connect to the I2E servers. The bundle will be installed on every I2E server in this list.
local_file_path – Path to the I2E resource bundle file on the local filesystem.
i2e_wait_timeout – The maximum time to wait for the I2E bundle installation task to finish before raising an AirflowException. It is recommended to increase this value if you are working with large resource bundles.
**kwargs – Additional keyword arguments to pass to the BaseOperator.

Return type:

NoneType

Hooks¶

class iqvia_nlp_provider.hooks.i2e.I2EHook(i2e_conn_id: str = 'i2e_default', i2e_wait_timeout: int = 60)¶

Hook that interacts with an I2E server.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server.
i2e_wait_timeout – The maximum time to wait for an I2E task to finish before raising an AirflowException. It is recommended to increase this value if you are working with large amounts of data or have a complex I2E query which may take longer to complete.

delete_resource(resource: str | None) → None¶

Delete the resource at the given URI.

Raises:: StatusError – If the resource could not be deleted.
Parameters:: resource – URI of the resource to delete.

delete_resources_in_folder(resource_type: str, folder: str) → list[str]¶

Delete all resources of the given resource type in the given folder.

Parameters:

resource_type – The type of the resource to delete. For example, “source_data” or “index”.
folder – The I2E folder.

Returns:

List of paths of the deleted resources.

get_query_results(query_results_uri: str) → ResponseStream¶

Get the query results from the given URI.

Parameters:: query_results_uri – URI of the query results to get.
Returns:: File-like object containing the query results encoded as bytes.

post_bundle_installation_task(repository_bundle_uri: str, filename: str) → str¶

Post a bundle installation task to I2E.

Parameters:

repository_bundle_uri – URI of the ‘repository_bundle’ resource to install.
filename – Name of the resource bundle to install.

Returns:

URI of the created bundle_installation task.

post_make_index_task(source_data_uri: str, index_template: str | None = None, index_settings_overrides: dict[str, Any] | None = None, folder: str | None = None) → str¶

Post a make index task to I2E.

Parameters:

source_data_uri – URI for the source data used by the make_index task.
index_template – URI for the index template to use. If provided, index_settings_overrides will be merged with the settings from the index template.
index_settings_overrides – Dictionary of settings to use for the make index task. These settings will overwrite any settings from the index template.
folder – If provided, the index will be created in a folder with this name.

Returns:

URI of the created make_index task.

post_query_task(index_uri: str, query_template: str | None = None, query_settings_overrides: dict[str, Any] | None = None) → str¶

Post a query task to I2E.

Parameters:

index_uri – URI for the index to query.
query_template – URI for the query template to use. If provided, query_settings_overrides will be merged with the settings from the query template.
query_settings_overrides – Dictionary of settings to use for the query task. These settings will overwrite any settings from the query template.

Returns:

URI of the created query task.

upload_resource_bundle(filename: str, content: IO[bytes]) → str¶

Upload a resource bundle to I2E.

Parameters:

filename – Name of the resource bundle to upload.
content – File-like object containing the content of the resource bundle to upload.

Returns:

URI of the uploaded resource bundle.

upload_source_data(filename: str, content: IO[bytes] | Iterator[bytes], folder: str | None = None) → str¶

Upload source data to I2E under the given file name.

Parameters:

filename – Name of the file to upload.
content – File-like object containing the content of the file to upload.
folder – If provided, the source data will be uploaded to a folder with this name.

Returns:

URI of the uploaded source data.

validate_connection() → None¶: Validate the connection to the I2E server. This method will raise an exception on failure.

wait_for_bundle_installation_task(bundle_installation_task_uri: str) → None¶

Wait until an I2E bundle_installation task finishes.

Parameters:: bundle_installation_task_uri – URI of the bundle_installation task to wait for.
Raises:: AirflowException – If timed out while waiting.

wait_for_make_index_task(make_index_task_uri: str) → str¶

Wait until an I2E make_index task finishes.

Parameters:: make_index_task_uri – URI of the make_index task to wait for.
Raises:: AirflowException – If timed out while waiting.
Returns:: URI of the created index.

wait_for_query_task(query_task_uri: str) → str¶

Wait until an I2E query task finishes.

Parameters:: query_task_uri – URI of the query task to wait for.
Raises:: AirflowException – If timed out while waiting.
Returns:: URI of the query results.

class iqvia_nlp_provider.hooks.legacy_webhook.LegacyWebhookHook(webhook_conn_id: str = 'webhook_default', result_options: ResultOptions | None = None, encoding: str = 'utf-8')¶

Hook to interact with the legacy NLP Data Factory webhooks.

Parameters:

webhook_conn_id – The connection ID of type “webhook” used to connect to a server hosting legacy NLP Data Factory webhook endpoints.
result_options – Dataclass specifying which results files to include in the webhook call.
encoding – The encoding to use when sending files to and receiving files from the webhook server.

run_after_hook(i2e_conn_id: str, endpoint: str, i2e_source_data_uri: str, i2e_query_results_uri: str) → tuple[Response, str]¶

Send a POST request to the specified endpoint of a legacy NLP Data Factory after hook with the given query results.

Parameters:

i2e_conn_id – The connection ID used to connect to the I2E server, for retrieving the query results.
endpoint – The endpoint to be used to connect to the webhook. For example, a webhook running on https://hostname:8000/afterhook would have the endpoint “afterhook”.
i2e_source_data_uri – URI of the source data in I2E.
i2e_query_results_uri – URI of the query results in I2E.

Returns:

A tuple containing the response from the after hook, and the name for the new query results file.

Raises:

AirflowException – If the response status code has an unexpected value.

run_before_hook(endpoint: str, source_filename: str, source_data: IO[bytes] | Iterator[bytes]) → tuple[str, IO[bytes]]¶

Send a POST request to the specified endpoint of a legacy NLP Data Factory before hook with the given source data.

Parameters:

endpoint – The endpoint to be used to connect to the webhook. For example, a webhook running on https://hostname:8000/before-hook/ would have the endpoint “before-hook/”.
source_filename – Name of the source data file.
source_data – File-like object containing the content of the source data to be sent to the before hook.

Returns:

A tuple containing the name for the new source data file, and an iterator of bytes with the new source data content.

Raises:

AirflowException – If the response status code has an unexpected value.

class iqvia_nlp_provider.hooks.legacy_webhook.ResultOptions(query_results: bool = True, cached_documents: bool = False, highlight: bool = False, highlighted_documents: Literal['no', 'content', 'content_and_stylesheet'] = 'no')¶: Dataclass for specifying which results files to include in the webhook call.

class iqvia_nlp_provider.hooks.hart.HARTHook(hart_conn_id: str = 'hart_default')¶

Interact with a HART server.

Parameters:: hart_conn_id – The connection ID used to connect to the HART server.

convert_assertion_rows_to_fine_tuning_data(assertion_rows: List[AssertionRow], target_column_id: int, labels: List[str]) → FineTuningData¶

Convert assertion rows to training data that can be used to fine tune a text classification model in Models Hub.

Rows must meet the following criteria to be included in the output: - the row must have an ACCEPTED acceptance status - the value of the cell associated with the target column must be in the labels list - the row must have evidence associated with the target column

Parameters:

assertion_rows – assertion rows from a HART project
target_column_id – The ID of the column containing the target labels
labels – The target labels

Returns:

a dict with keys “Text”, “Target”, “Weight”, “Coords”

Return type:

FineTuningData

list_assertion_rows(review_id: int, since: str) → List[AssertionRow]¶

Fetch assertion rows.

Parameters:

review_id – The Review ID of the HART project.
since – The starting date of the annotations to fetch for training.

Returns assertion_rows:

An array of assertion rows.

Raises:

AirflowException – If no assertion rows are returned, or on a non 2xx/3xx response code from HART.

list_documents(review_id: int, doc_ids: str) → List[Document]¶

Fetch documents.

Parameters:

review_id – The Review ID of the HART project.
doc_ids – A comma-separated string of document IDs to be fetched.

Returns documents:

An array of HART documents.

prepare_documents_for_text_classification(documents: List[Document]) → PredictionData¶

Convert an array of HART documents into PredictionData, to be used for model inference.

Documents are split into sentences, and offsets are calculated for each sentence relative to the full document text.

Parameters:: documents – an array of HART documents
Returns:: A dictionary of preprocessed data, that can be sent to Models Hub for model prediction.
Return type:: PredictionData

write_results(review_id: int, name: str, description: str, create_project: bool, target_name: str, json_file: List[UJSONDocument], documents: Dict[str, str]) → None¶

Operator to upload a JSON file to HART.

Parameters:

review_id – The Review ID of the HART project.
name – The name of the HART project.
description – The description of the HART project.
create_project – Create a new project (boolean).
target_name – The name of the column containing the target labels.
json_file – The unified JSON results to import.
documents – The documents to import.

class iqvia_nlp_provider.hooks.models_hub.ModelsHubHook(models_hub_conn_id: str = 'models_hub_default')¶

Hook that interacts with a Models Hub server.

Parameters:: models_hub_conn_id – The connection ID used to connect to the Models Hub server.

convert_prediction_response_to_unified_json(prediction_output: PredictionResponse, column_name: str, labels: list[str], logical_column_id: int, confidence_threshold: float) → List[UJSONDocument]¶: Convert from a Models Hub prediction response to Unified JSON.

create_model(name: str, model_type: str, labels: List[str], base_model: str, model_tags: dict[str, str | None] | None = None, description: str | None = None) → Model¶

Create a model in Models Hub.

Parameters:

name – The unique name for the model. This will be the model_id.
model_type – Type of the model (text_classification, token_classification).
labels – Output labels.
base_model – The base model to use.
model_tags – Tags for the model.
description – A description for the model.

Returns model:

The created model.

fine_tune(run_id: str, model_id: str, hyper_params: FineTuningHyperParams, training_data: FineTuningData | FineTuningDataNER) → TriggerTaskResponse¶

Fine tune a model in Models Hub.

Parameters:

run_id – The dag run id.
model_id – The id of the model.
hyper_params – Model fine tuning hyper params. See FineTuningHyperParams for configurable parameters.
training_data – The training data.

Returns task_id:

The ID of the fine tuning task.

get_model(model_id: str) → Model¶

Get a model in Models Hub from the model_id.

Parameters:: model_id – The ID of the model in Models Hub.
Returns model:: The model.

predict(model_id: str, data: PredictionData, hyper_params: PredictionHyperParams | None = None) → TriggerTaskResponse¶

Predict the labels of some text using an ML classifier.

Parameters:

model_id – The ID of the model in Models Hub.
data – Data that has been preprocessed for model prediction.
hyper_params – Model prediction hyper params. See PredictionHyperParams for configurable parameters.

Returns task_id:

The ID of the prediction task.

wait_for_fine_tuning_task_completion(task_id: str, model_id: str) → FineTuningResponse¶

Continuously poll until a fine tuning task has completed. Will raise an AirflowException on a failed task or a non 2xx/3xx response code.

Parameters:

task_id – The ID of the task.
model_id – The ID of the model.

Raises:

AirflowException – If completed task does not have results.

Returns:

Fine tuning task results.

wait_for_prediction_task_completion(task_id: str, model_id: str) → PredictionResponse¶

Continuously poll until a prediction task has completed. Will raise an AirflowException on a failed task or a non 2xx/3xx response code.

Parameters:

task_id – The ID of the task.
model_id – The ID of the model.

Raises:

AirflowException – If completed task does not have results.

Returns results:

The model prediction output.

Other¶

enum iqvia_nlp_provider.operators.i2e.Strategy(value)¶

Strategies for picking an I2E connection ID. This enum is used by the I2EPickConnectionIDOperator.

Valid values are as follows:

RANDOM = <Strategy.RANDOM: 'random'>¶: Pick a random I2E connection ID from the given list.

ROUND_ROBIN = <Strategy.ROUND_ROBIN: 'round_robin'>¶: Pick an I2E connection ID from the given list, based on the I2E connection ID used in the most recent successful DAG task with the same task ID and DAG as the current task. If this task used an I2E connection ID from the given list, return the next element in the list. For this function to work correctly, it must be called from within an Airflow task that returns the I2E connection ID as its XCom return value.

LEAST_BUSY = <Strategy.LEAST_BUSY: 'least_busy'>¶: Pick an I2E connection from the given list, based on which I2E server has the fewest in-progress tasks. This function is not recommended if you have a large number of I2E servers, since it requires making requests to each I2E server.

class iqvia_nlp_provider.transfers.Webhook(conn_id: str, endpoint: str, result_options: ~iqvia_nlp_provider.hooks.legacy_webhook.ResultOptions = <factory>, webhook_return_file_encoding: str = 'utf-8')¶

Class for storing the connection ID and endpoint information for a legacy NLP Data Factory webhook. This class is used by various transfer operators.

Parameters:

conn_id – The connection ID to be used to connect to the webhook.
endpoint – The endpoint to be used to connect to the webhook. For example, a webhook running on https://hostname:8000/after-hook/ would have the endpoint “after-hook/”.
result_options – The result files to include in the call to the after hook. Defaults to an empty ResultOptions instance which only includes query results.
webhook_return_file_encoding – The encoding used by the webhook’s response. Defaults to “utf-8”.

iqvia_nlp_provider.utils.i2e_connections.get_all_i2e_connection_ids() → list[str]¶

Retrieve all I2E connection IDs present in Airflow.

In Airflow 2 this could read the Connection table via the ORM inside task code. Airflow 3 restricts direct metastore access from tasks, so this lists connections through GET /api/v2/connections (with pagination) instead.

Returns:: List of all I2E connection IDs (sorted).

class iqvia_nlp_provider.hooks.models_hub_typing.PredictionHyperParams¶

A dictionary of model prediction hyper parameters.

batch: str | None¶: The number of items processed together in a single iteration.

confidenceThreshold: str¶: The minimum confidence score required for the model to output the predicted label.

class iqvia_nlp_provider.hooks.models_hub_typing.FineTuningHyperParams¶

A dictionary of model fine tuning hyper parameters.

batch: str¶: The number of items processed together in a single iteration.

epochs: str | None¶: The number of complete passes through the entire training dataset.

lr: str | None¶: The step size at each iteration while moving toward a minimum of a loss function.